Pitch - Swift Tooling Workgroup

We are also interested in benchmarking specifically - will try to take the time to write down our discussions and share.

1 Like

Yeah I would say IMO the charter would be 2 and 3:
Make developing and building Swift applications, scripts and services on any supported platform a great experience

Most of the other points follow on from that

Edit: expanded the charter slightly for my own selfish interest in Swift scripting :sweat_smile:

3 Likes

One other important tool I don't see mentioned in the original list is Swift REPL. The workgroup could drive a lot of improvements in this area, for example:

  1. Making sure that REPL works reliably on all platforms (we've seen issues on Linux and AFAIR it still doesn't work on Windows).
  2. Improving REPL UX: it could benefit from syntax highlighting support, history search, a dedicated command for displaying inferred type of expressions, improving auto-complete, allowing users to surface doc comments for symbols available in scope etc.
  3. Fixing some other known bugs and inconsistencies, for example see missing XCTest support.
10 Likes

Since the documentation workgroup is focused in documentation experience instead of the documentation itself, I think the tooling workgroup should also take care of standardizing and documenting the usage of SwiftPM and other tools. There’re too many undocumented pieces in SwiftPM, which can only be inspected by source code. More user-friendly guides and tutorials will largely improve the user experience of Swift toolchain.

5 Likes

Continuing development on the vs code extension would also be great as a lightweight IDE. But currently, as much as I’d love to switch over from Xcode, the experience still feels less refined.

2 Likes

I’d really like a more refined library for CLIs. Argument parser is great, but it can only help with static CLI. But CLIs can obviously be a lot more interactive (some good examples are homebrew, and the GitHub and firebase CLIs. I know there’s some basic formatting in swift tools support, but that only scratches the surface of what’s possible with ANSI escape codes. Having something like ncurses adapted and modernized to a Swift library would be great and make great Seift-built CLIs even better.

1 Like

Just FYI that this year's argument parser GSoC project by @KeithBirdKTH is working on interactive mode for argument parser: [GSoC] CLI design for interactive mode · Issue #448 · apple/swift-argument-parser · GitHub

In general though yeah it probably is right to "make building small tools in swift simpler" could also be part of the goals here. It plays well with the "make it easy to develop (small cli apps) in swift".

4 Likes

I wasn’t referring to necessarily smaller projects, I just think that simple QoL improvements developers expect from their modern CLI tools should be more accessible. For example, having access to the tty’s dimensions to build more immersive graphics, controlling the inputs buffering behavior to instantly get access to key presses, and even setting the terminal’s title are minor changes that can be unnecessarily complex for Swift CLI apps.

i just want to add this is really constraining library development, because libraries would like to have tools (plugins and executables) that only build on certain platforms but support a wider range of platforms than their own tools support. some of these things are tools like linters and formatters that really aren't necessary for most consumers of the library to build.

because SPM cannot distinguish between source and target platforms, packages end up fragmenting into collections of packages that live together in a repository but have separate manifests, because including the tools in the main package would break cross compile builds.

Do you have an example of this in an open-source repository we could look into to understand how to improve the situation?

1 Like

Ok, here's a short summary with more details coming if there's interest in the community:

4 Likes

certainly. i will use swift-json as an example.

there is a perception that “‘server’ libraries” like swift-json do not give a shit about iOS/watchOS/tvOS users. i have always tried to make my libraries build on as many platforms as is feasible, but SPM makes it weirdly difficult to get any kind of CI set up for these platforms, because SPM thinks of itself as a macOS/linux build system.

the relevant issue ticket here is Packages that contain tools (e.g. swift-markdown) don't seem to build on iOS · Issue #1299 · SwiftPackageIndex/SwiftPackageIndex-Server · GitHub .

the gist of it is that SPM builds all executable and plugin targets on all platforms, even platforms (like iOS) that have no concept of executables or plugins. this automatically fails cross-compilation builds, and the only way to get SPM to stop impaling itself on the executables is to disable them on any source platform that could potentially cross-compile to a deployment-only platform.

practically speaking, this means the only way to meaningfully develop swift-json on macOS is to patch the package manifest for local development, reverting it before submitting changes upstream, because macOS is the cross-compilation source platform for iOS, tvOS, and watchOS.

why #1299 is being tracked as a swiftpackageindex.com bug and not an SPM bug is beyond me, because it affects anyone who has CI for non-desktop platforms, and is also using “modern” SPM features, like snippets and plugins, and apparently, command-line tools.

ever since SPM got snippets, a separate but related problem has also cropped up: packages that adopt snippets are not backwards-compatible with swift <=5.6 (meaning, every currently-available release toolchain). for some reason, SPM 5.6 knows about snippets, but it cannot serialize them properly (it crashes), and it also cannot conditionally ignore them, because the path to the Snippets/ directory is hard-coded.

fortunately, the snippets problem doesn’t affect consumers of the library, because they don’t need to run any of the project’s tools (which crash on SPM <= 5.6). but adding even a single snippet to an SPM project directory effectively means every one of its tools must be gated to swift >= 5.7, including the tools that have nothing to do with snippets, because SPM attempts to serialize the whole package for each tool.

i originally filed the snippets crash as a DocC plugin issue, but have since discovered that it also affects swift-package-catalog and other tools that depend on PackagePlugin.

2 Likes

Here's a snippet of a package.swift that shows the problem:

// swift-tools-version:5.5

import PackageDescription

let dependencies: [PackageDescription.Package.Dependency] = an array of depends
let serverTargetDependencies: [Target.Dependency] = an array of target depends
let serverTargetResources: [Resource] = an array of resources

#if os(macOS)

let package = Package(
    name: "displayserver",
    platforms: [
        .macOS(.v11),
    ],
    products: [
        .executable(
            name: "displayserver",
            targets: [
                "Server",
            ]
        ),
    ],
    dependencies: dependencies,
    targets: [
        .executableTarget(
            name: "Server",
            dependencies: serverTargetDependencies,
            resources: serverTargetResources,
            linkerSettings: []
        ),
        .testTarget(
            name: "ServerTests",
            dependencies: ["Server"]
        ),
    ],
    cxxLanguageStandard: .cxx14
)

#else

let package = Package(
    name: "displayserver",
    products: [
        .executable(
            name: "displayserver",
            targets: [
                "Server",
            ]
        ),
    ],
    dependencies: dependencies,
    targets: [
        .executableTarget(
            name: "Server",
            dependencies: serverTargetDependencies,
            resources: serverTargetResources,
            linkerSettings: [ .linkedLibrary("stdc++") ]
        ),
        .testTarget(
            name: "ServerTests",
            dependencies: ["Server"]
        ),
    ],
    cxxLanguageStandard: .cxx14
)

#endif

Its the linker settings that are the problem in this case. bc I'm building with a destination that is an x-compile to linux, I'm basically out of luck. This Package.swift needs to compile for MacOS, bc that's where the compilation will run, but the compilation settings need to be for linux, in this case, I will be running the gold linker to generate an Arm64 Ubunu 20.04 executable and there is no way to direct SPM to do that.

I don’t think SwiftPM is lacking any functionality given your example. LinkerSetting.linkedLibrary has an optional condition: parameter that can be used to conditionally linking for Linux targets, so the #if os() check doesn’t even have to exist.

1 Like

What about .linkedLibrary("stdc++", .when(platforms: [.linux]))?

EDIT:

@stevapple beat me by a few seconds ;)

1 Like

Is that going to be based on the target triple in the destination? Have to admit I haven't built that package in a while.

This is definitely needed.

Tooling is the one area where swift falls short. And it hinders adoption outside of apples ecosystem.

4 Likes

This part is largely covered by the Swift Platform Work Group initiative. Feel free to discuss about cross-compilation experience there.

2 Likes

+1 on Swift scripting. There's lots of developers interested in using Swift for scripting as well, there's nothing selfish about it. Unfortunately the scripting support in Swift is practically not given (there's no file import or script-dependency definition, for example), but related pitches do exist.

I think it would make a lot of sense for Swift Scripting to be part of a Swift Tooling workgroup as scripting mostly benefits from great tooling and scripts can be seen as tools themselves, given that they are typically run from the command line.

2 Likes

Thanks for bringing this idea up @0xTim, and for everyone providing feedback on this thread.

Over the last few months, the Core Team has been reorganizing the project by creating a number of dedicated workgroups to oversee different areas of the project. That work isn’t done, and the Core Team still sees a lot of areas that would immediately benefit from having this kind of focused leadership, including platform support and portability, productivity tools, toolchain distribution, tools for building and distributing Swift programs, and source editing.

The package ecosystem is another growth area, with new tools and services coming online that would benefit from clear guidance towards building a safe and secured ecosystem.

Over the last few weeks, the core team has been discussing how to best address the needs expressed in this thread in light of these motivations. The exact details are yet to be determined, but it is likely to include the formation of a new workgroup with the charter to lead in across the platform support, productivity tools, and packages ecosystem concerns. As the Core Team continues to work out the organizational structure details, we encourage the community to continue and discuss these ideas. We also invite community members who would like to be part of such initiative to reach out directly to the core team to express such interest.

5 Likes