Pitch - Swift Tooling Workgroup

Evolution Pitches
1

Hi all,

With the recent announcement of the language workgroup and new directive of the core team, I'd like to propose a new workgroup - the Swift Tooling Workgroup. The language workgroups remit covers Swift itself and evolution of the language which leaves a number of tooling areas stuck in a bit of no-man's land with no central group to oversee, drive and direct effort.

There are a number of areas that I feel would fall under the tooling workgroup, with some overlap with the existing workgroups:

  • SwiftPM - whilst language features and the compiler fall under the language workgroup, SwiftPM itself seems to be outside that. There are lots of good ideas for improvements and missing features and this would be central to evolving Swift's adoption
  • SourceKit-LSP - this has seen a load of recent focus thanks to the VSCode extension and would be great to continue
  • The VSCode extension and other potential IDE integrations. Having the same group responsible for all integrations and the LSP would help coordinate effort and sort out priorities, similar to the Rust dev tools workgroup
  • Swift Format - again, falls outside of the language workgroup's remit but needs help being adopted by the ecosystem
  • API Breaking Change checker - I guess this is part of SwiftPM but is useful in its own right and could do with new features and bugs fixing (eg adding new method parameters with default parameters is marked as a breaking change when it's not)
  • Swiftly - this tool is currently being developed by the SSWG to make installing any version of Swift on any supported platform easy. It's a needed tool bigger than just the server space
  • Docker images, Debs and RPMs - related to Swiftly, all part of tooling and build infrastructure, extremely important and currently held up by the awesome work from @mishal_shah and the community
  • DocC - falls under the documentation workgroup but there would be overlap with the plugin, integration with CLI tools etc
  • Any other official plugins designed to make writing and building Swift better, examples could include linting, formatting, licence header generation
  • Outliers like SwiftMarkdown - it's unclear if this tool is the responsibility of the documentation workgroup or not
  • Any work on package registries - this is an important part of the future ecosystem and would require coordination from implementors, security experts, infrastructure teams etc

IMO there are a lot of areas critical to the success of Swift that currently don't have any oversight from a workgroup, outside of the Core Team. From my perspective the Swift Tooling Workgroup is the last major piece of Swift missing a workgroup and would really benefit the language.

Tim

58 Likes
2

Can we add cross-compilers to that list?

11 Likes
3

Love this idea!
I don't know how it works for external contributors (if at all) but I'd love to help if needed :)

2 Likes
4

Yes please! +1 for the cross-compilers tooling to be included in the list.

8 Likes
5

+1, great proposal - agree it is a critical area for the overall success.

1 Like
6

The swift compiler is already a cross-compiler and I personally believe that it belongs within the purview of the Language Workgroup. Furthermore, it is already possible to perform cross-compilation with the compiler when used directly or with tools like CMake. Most of the challenges in cross-compiling I've found tend to be more around the platform SDK packaging and distribution rather than the compiler itself. Improving the ease of cross-compiling with SPM is a worthy goal, but fits quite well into the SPM category which @0xTim already included in the list.

5 Likes
7

Nice pitch! All these are part of the Swift ecosystem and improvements should benefit almost every Swift user.

Generally speaking, the topic of tooling may be “too large”, compared to language and documentation. When speaking of Swift tooling, I would see two main parts: the Swift toolchain and ecosystem around the toolchain. Although there’re still some overlaps and these two subtopics will often affect each other, this can be a way to split and focus a workgroup.

The Swift toolchain experience involves setting up the toolchain (layout, Swiftly, Debs & RPMs, Windows installer), building (SwiftPM, cross-compilation), debugging (LLDB, REPL), testing (CodeCov) and deploying (Docker) a Swift application. I would also mention the experience of building and debugging the toolchain itself, especially on Windows.

The outer ecosystem should involve those tools (and non-server libraries?) that can add to Swift’s usability. Each of the packages is dedicated and I have little idea on who should be in such a workgroup except for the major maintainers of them.

SourceKit-LSP and DocC lie on the edge, but I would suggest classifying them as the latter because they’re mostly independent and the toolchain side already seems big. Library evolution (ABI breaking change checker) is something I’m not familiar with though, so no idea for classification.

2 Likes
8

Other libraries also include the Argument Parser, ProtoBuf and Swift System

4 Likes
9

Gigantic +1 from me, getting some of the sharp edges ironed out of these will make Swift much more approachable. Would love to help out where I can.

1 Like
10

Another big +1.

Tooling specifically is currently a big roadblock to adoption across the ecosystem and should be a major focus to improving the eco-system. The pitch includes examples of tooling that is larger than the Swift Server eco-system (Swiftly, VSCode extension) that are primarily being driven by the Server work group which is probably not ideal. The community having input into other critical aspects of the ecosystem would be beneficial as well.

2 Likes
11

This sounds great, and is an area of interest for me in my day-to-day work. I’d love to be involved in whatever capacity is helpful.

12

@compnerd, you more than anyone would know, and i have immense respect for what you've accomplished with windows, but i'm nowhere near as certain that the language team is the right place for this.

theres a real problem with getting xcompile bug fixes upstreamed. theres just not a process for that and i think that relates to the language development process being too centralized. in my corner of the world we were stuck on 5.2 for two years for that reason.

not all tools needed to xcompile for various linices ship with the swift compiler, gold being the example that springs immediately to mind. i can't even imagine how the arduino guys will deal with their toolchain.

it is true that the biggest problems are library dependencies bc its never really clear what those dependencies are and which are needed at compile time and which at runtime. i'm not sure that the language team will have the bandwidth to manage the somewhat dynamic process of dependency discovery that goes along with specializing an xcompiler to a platform that may not be a major apple concern.

certainly willing to be convinced i'm wrong but i'd think there ought to be some xcompilation facet to this proposed workgroup.

13

Regarding cross-compilation, @stevapple's planned SDK workgroup is ideally where that would go. But since that currently doesn't exist, we may need to support cross-compilation in these other workgroups until it does.

14

Do you have examples of these? I am interested both as cross-compilation is something that I care about but also because I am left wondering if there are pieces that I may have missed with the Windows work.

That is a packaging issue, not really a toolchain issue. Packaging lld is sufficient for a linker, and something we already do on Windows. I would actually be curious what tools are missing on the Linux distributions - the Windows distribution is actually complete enough to build if you have the Windows SDK available (MSVC is only required as the C library is part of MSVC rather than the Windows SDK).

There is nothing that the Swift project can do here IMO. The target libraries and headers are part of the platform SDK. The easiest thing here is to simply do something like readelf -Wd .../usr/lib/*.so | grep NEEDED | sort -u expanding the path to the toolchain image. The DT_NEEDED entries and its transitive closure will tell you what the runtime dependencies are. The build dependencies for the platform are primarily the C and C++ libraries and their transitive closure of dependencies.

15

Heh. Do I. :slight_smile:

I went back to @uraimo's Build Swift for Arm site. He has kept a great list of of the diffs he had to apply to the compiler to build for Arm6/7/8 going all the way back to Swift 3.0.2 The folders of interest are:

  • icu.diffs/aarch32
  • llbuild.diffs/aarch32
  • llvm-project.diffs/aarch32
  • swift-corelibs-libdispatch.diffs
  • swift-tools-support-core.diffs/raspbian
  • swift.diffs
  • swiftpm.diffs/aarch32

You can trace the bugs that prevented Swift on Raspberry Pi (and other Arm platforms) from staying current and how long it took to get the patches in. It's a fascinating history of a small dedicated community that I hadn't looked at in a couple of years.

Plenty of the issues were patches to things like icu and llvm that had to be applied bc swift would use ancient versions and Swift for Arm required the fixes.

Simultaneously @weissi, @Helge_Hess1 and (eventually) I were taking those builds and using (what eventually became) the build script at SwiftCrossCompilers to create Mac -> R/Pi and x86 cross compilers. The history of that stuff is at that site. Lately @krzyzanowskim has been doing yeoman's work keeping it all up-to-date.

Another really good source to find out pain points getting changes upstreamed would be this issue where the community was stuck trying to get 5.2 compiled:

@Finagolfin 's repo for Android cross-compilation is a great resource that details the follow-up to that period.

I should also call out @futurejones's work at swift-arm64 because basically no one's toolchains would work without that. It's an enormous amount of work that he does AFAICT out of the goodness of his heart.

Most recently @colemancda has been absolutely kicking the toolchains for a variety of weird boards that I know nothing about. He has posted several times in these fora about trying to get his changes upstreamed.

The latest list of tools needed are in the build script. Reviewing that for the first time in a while, I see it takes a couple of brew installs and gold to construct the xcompiler chain. And of course it takes that whole ludicrously evolved script.

This was really my point from before. I don't think that the Language group can do much to help, but the proposed Tools group certainly could. My conception of what is needed from the Tools group is to maintain a much more reliable tool chain platform for each community to build its own xcompilers. The bash script which descends from @weissi's original stuff needs to be redone in Swift, for example and one could easily imagine that work being tangentially related to SPM. Because, in the end, SPM really needs some changes itself to facilitate doing xcompiles. SPM just does not distinguish right now between the source and target build systems and that creates a lot of pain. That work rightly, IMHO, belongs in a tools group.

yes that's pretty much what the SwiftCrossCompilers build script does. It's just such a painfully iterative process bc you can't discover the transitive dependencies until you have downloaded the lib you just found out about. This could be much improved. It should also be noted that the discovered dependencies are different for every single version of every single target x-compile OS. and that complicates things a bit.

3 Likes
16

I didn't read the whole thread, but I also have SPMDestinations, which is improving a lot on the original scripts (and splits the process in multiple, distinct parts).
It was originally intended for swift-lambda (cross compiling to x86 Amazon/Ubuntu from M1), but could be enhanced to Raspi Linux as well.

3 Likes
17

Can we add testing and benchmarking?

Testing feels like it has been ignored pretty-much since forever, but it may be too big a task and probably deserves its own workgroup.

Benchmarking can also be difficult when working with an unfamiliar platform. I've been wondering if we could create a SwiftPM plugin with presets for common benchmarking tasks (tracking allocations, etc) on Linux and other platforms where Instruments is not available. I think something like that could possibly fit under a tooling workgroup.

9 Likes
18

I think it makes sense to have a standalone cross-compilation workgroup to investigate the layout and distribution of toolchain and SDK, to improve the overall cross-compilation functionality of the toolchain and to write up cross-compilation terms and guides. Cross-compilation becomes more and more important a thing with our efforts on embedded programming and bootstrapping, and it also ties to the goal of making Swift available on more platforms, no matter as a host or as a target.

I suggest we continue in another thread on the cross-compilation topic, since @0xTim‘s initiative includes far more than that.

4 Likes
19

I would love to see more of a focus on improved tooling for Swift. However, I'd like to understand what the charter of this group would be. I can think of several possible things:

  1. Spur development of more tools
  2. Simplify the "on-ramp" to Swift development
  3. Identity and address the major pain points in existing Swift tools
  4. Develop libraries that makes it easier to build and integrate tools
  5. Manage reviews for SwiftPM proposals

I have my favorites from the above list, but I'm curious what you---and others who would like to be involved---feel should be the focus of such a group.

I tend to agree with @stevapple that cross-compilation belongs elsewhere, because it's a mostly separable problem that involves different expertise.

Doug

2 Likes
20

To me, #2 is the first priority, but that largely encompasses a lot of the other goals. The biggest pain point here in my experience is getting a good development environment and tooling set up - on Mac, people should not have to download 60 GB of Xcode just to be able to successfully develop a Swift project in VS Code (last I checked, swift test was one of the last showstoppers from being able to use a completely standalone toolchain), or on Linux have to resort to third-party package repositories/Docker to get everything installed. I know there's been some smoke on this topic recently under the name swiftly, but details are scarce for now so this is still my biggest want from such a workgroup.

At the end of the day, I'd love for someone to be able to run three commands on any platform and be up and running with something minimal:

  1. git clone {some Swift repo}
  2. brew install/apt install/etc.
  3. swift build

Until we can match that simplicity provided by other languages, there will always be the perception that getting started with Swift is complicated - imagine being a newbie developer trying to get started and being presented with the official Getting Started documentation. Those instructions are correct and provide a lot of good context, but can be daunting - compare this to Rust's similar page that has a one-liner at the top you can run to get fully bootstrapped.

The next thing lacking is "bootstrap documentation" for various specific application domains - Rust has this nailed, and I'd like to see something similar from a first-party source that walks through how to get a simple hello-world program up and running in various areas, even if that list is just a link to some "blessed" third-party sources.

8 Likes