Pitch - Swift Tooling Workgroup

Other libraries also include the Argument Parser, ProtoBuf and Swift System

4 Likes

Gigantic +1 from me, getting some of the sharp edges ironed out of these will make Swift much more approachable. Would love to help out where I can.

1 Like

Another big +1.

Tooling specifically is currently a big roadblock to adoption across the ecosystem and should be a major focus to improving the eco-system. The pitch includes examples of tooling that is larger than the Swift Server eco-system (Swiftly, VSCode extension) that are primarily being driven by the Server work group which is probably not ideal. The community having input into other critical aspects of the ecosystem would be beneficial as well.

2 Likes

This sounds great, and is an area of interest for me in my day-to-day work. I’d love to be involved in whatever capacity is helpful.

@compnerd, you more than anyone would know, and i have immense respect for what you've accomplished with windows, but i'm nowhere near as certain that the language team is the right place for this.

theres a real problem with getting xcompile bug fixes upstreamed. theres just not a process for that and i think that relates to the language development process being too centralized. in my corner of the world we were stuck on 5.2 for two years for that reason.

not all tools needed to xcompile for various linices ship with the swift compiler, gold being the example that springs immediately to mind. i can't even imagine how the arduino guys will deal with their toolchain.

it is true that the biggest problems are library dependencies bc its never really clear what those dependencies are and which are needed at compile time and which at runtime. i'm not sure that the language team will have the bandwidth to manage the somewhat dynamic process of dependency discovery that goes along with specializing an xcompiler to a platform that may not be a major apple concern.

certainly willing to be convinced i'm wrong but i'd think there ought to be some xcompilation facet to this proposed workgroup.

Regarding cross-compilation, @stevapple's planned SDK workgroup is ideally where that would go. But since that currently doesn't exist, we may need to support cross-compilation in these other workgroups until it does.

Do you have examples of these? I am interested both as cross-compilation is something that I care about but also because I am left wondering if there are pieces that I may have missed with the Windows work.

That is a packaging issue, not really a toolchain issue. Packaging lld is sufficient for a linker, and something we already do on Windows. I would actually be curious what tools are missing on the Linux distributions - the Windows distribution is actually complete enough to build if you have the Windows SDK available (MSVC is only required as the C library is part of MSVC rather than the Windows SDK).

There is nothing that the Swift project can do here IMO. The target libraries and headers are part of the platform SDK. The easiest thing here is to simply do something like readelf -Wd .../usr/lib/*.so | grep NEEDED | sort -u expanding the path to the toolchain image. The DT_NEEDED entries and its transitive closure will tell you what the runtime dependencies are. The build dependencies for the platform are primarily the C and C++ libraries and their transitive closure of dependencies.

Heh. Do I. :slight_smile:

I went back to @uraimo's Build Swift for Arm site. He has kept a great list of of the diffs he had to apply to the compiler to build for Arm6/7/8 going all the way back to Swift 3.0.2 The folders of interest are:

  • icu.diffs/aarch32
  • llbuild.diffs/aarch32
  • llvm-project.diffs/aarch32
  • swift-corelibs-libdispatch.diffs
  • swift-tools-support-core.diffs/raspbian
  • swift.diffs
  • swiftpm.diffs/aarch32

You can trace the bugs that prevented Swift on Raspberry Pi (and other Arm platforms) from staying current and how long it took to get the patches in. It's a fascinating history of a small dedicated community that I hadn't looked at in a couple of years.

Plenty of the issues were patches to things like icu and llvm that had to be applied bc swift would use ancient versions and Swift for Arm required the fixes.

Simultaneously @weissi, @Helge_Hess1 and (eventually) I were taking those builds and using (what eventually became) the build script at SwiftCrossCompilers to create Mac -> R/Pi and x86 cross compilers. The history of that stuff is at that site. Lately @krzyzanowskim has been doing yeoman's work keeping it all up-to-date.

Another really good source to find out pain points getting changes upstreamed would be this issue where the community was stuck trying to get 5.2 compiled:

@Buttaface 's repo for Android cross-compilation is a great resource that details the follow-up to that period.

I should also call out @futurejones's work at swift-arm64 because basically no one's toolchains would work without that. It's an enormous amount of work that he does AFAICT out of the goodness of his heart.

Most recently @colemancda has been absolutely kicking the toolchains for a variety of weird boards that I know nothing about. He has posted several times in these fora about trying to get his changes upstreamed.

The latest list of tools needed are in the build script. Reviewing that for the first time in a while, I see it takes a couple of brew installs and gold to construct the xcompiler chain. And of course it takes that whole ludicrously evolved script.

This was really my point from before. I don't think that the Language group can do much to help, but the proposed Tools group certainly could. My conception of what is needed from the Tools group is to maintain a much more reliable tool chain platform for each community to build its own xcompilers. The bash script which descends from @weissi's original stuff needs to be redone in Swift, for example and one could easily imagine that work being tangentially related to SPM. Because, in the end, SPM really needs some changes itself to facilitate doing xcompiles. SPM just does not distinguish right now between the source and target build systems and that creates a lot of pain. That work rightly, IMHO, belongs in a tools group.

yes that's pretty much what the SwiftCrossCompilers build script does. It's just such a painfully iterative process bc you can't discover the transitive dependencies until you have downloaded the lib you just found out about. This could be much improved. It should also be noted that the discovered dependencies are different for every single version of every single target x-compile OS. and that complicates things a bit.

3 Likes

I didn't read the whole thread, but I also have SPMDestinations, which is improving a lot on the original scripts (and splits the process in multiple, distinct parts).
It was originally intended for swift-lambda (cross compiling to x86 Amazon/Ubuntu from M1), but could be enhanced to Raspi Linux as well.

3 Likes

Can we add testing and benchmarking?

Testing feels like it has been ignored pretty-much since forever, but it may be too big a task and probably deserves its own workgroup.

Benchmarking can also be difficult when working with an unfamiliar platform. I've been wondering if we could create a SwiftPM plugin with presets for common benchmarking tasks (tracking allocations, etc) on Linux and other platforms where Instruments is not available. I think something like that could possibly fit under a tooling workgroup.

9 Likes

I think it makes sense to have a standalone cross-compilation workgroup to investigate the layout and distribution of toolchain and SDK, to improve the overall cross-compilation functionality of the toolchain and to write up cross-compilation terms and guides. Cross-compilation becomes more and more important a thing with our efforts on embedded programming and bootstrapping, and it also ties to the goal of making Swift available on more platforms, no matter as a host or as a target.

I suggest we continue in another thread on the cross-compilation topic, since @0xTim‘s initiative includes far more than that.

4 Likes

I would love to see more of a focus on improved tooling for Swift. However, I'd like to understand what the charter of this group would be. I can think of several possible things:

  1. Spur development of more tools
  2. Simplify the "on-ramp" to Swift development
  3. Identity and address the major pain points in existing Swift tools
  4. Develop libraries that makes it easier to build and integrate tools
  5. Manage reviews for SwiftPM proposals

I have my favorites from the above list, but I'm curious what you---and others who would like to be involved---feel should be the focus of such a group.

I tend to agree with @stevapple that cross-compilation belongs elsewhere, because it's a mostly separable problem that involves different expertise.

Doug

2 Likes

To me, #2 is the first priority, but that largely encompasses a lot of the other goals. The biggest pain point here in my experience is getting a good development environment and tooling set up - on Mac, people should not have to download 60 GB of Xcode just to be able to successfully develop a Swift project in VS Code (last I checked, swift test was one of the last showstoppers from being able to use a completely standalone toolchain), or on Linux have to resort to third-party package repositories/Docker to get everything installed. I know there's been some smoke on this topic recently under the name swiftly, but details are scarce for now so this is still my biggest want from such a workgroup.

At the end of the day, I'd love for someone to be able to run three commands on any platform and be up and running with something minimal:

  1. git clone {some Swift repo}
  2. brew install/apt install/etc.
  3. swift build

Until we can match that simplicity provided by other languages, there will always be the perception that getting started with Swift is complicated - imagine being a newbie developer trying to get started and being presented with the official Getting Started documentation. Those instructions are correct and provide a lot of good context, but can be daunting - compare this to Rust's similar page that has a one-liner at the top you can run to get fully bootstrapped.

The next thing lacking is "bootstrap documentation" for various specific application domains - Rust has this nailed, and I'd like to see something similar from a first-party source that walks through how to get a simple hello-world program up and running in various areas, even if that list is just a link to some "blessed" third-party sources.

8 Likes

Personally, I agree. I would love for #2 to define the charter for this group, and its work revolve around that.

Doug

2 Likes

I've done something like that in Colab, using the pseudocode below:

!curl [URL of Swift toolchain] | tar -xz
!mv [name of toolchain] /opt/swift/toolchain
!git clone [URL of package]
!cd [package name] && /opt/swift/toolchain/usr/bin/swift build

But for the interactive Jupyter notebook experience, you do have to rely on an external repository to compile libJupyterKernel.so and facilitate automatic downloading of toolchains. I'm hoping to bring this to local Jupyter notebooks, so that my install_swift.sh script can provide a painless and extremely fast setup experience on all platforms.

Although this is limited in use (geared toward scripting/data science), I'm transforming it into a complex build system that supports compiling Swift packages, testing them, injecting Swift modules into packages, and constructing complex build setups. For a beginner, it should provide enough functionality to learn the language and get accustomed to SwiftPM.

Perhaps we could fix bugs in the Swift package manager that occur in abnormal use cases? I encountered quite a lot while making Swift-Colab's new SwiftPM engine. However, not everyone will stress the core Swift tooling components to the breaking point like I am.

these are both great ideas for workgroups, and could be added to a “project infrastructure” umbrella. i think GitHub Actions workflows are also another important area related to these two topics, and could count as “project infrastructure”.

better CI coverage would make it easier and encourage more package authors to support multiple platforms at once, which is really central to swift’s growth. so far the swift package index has been able to partially fill this need for packages that already support multiple platforms, but it’s much harder to port a package to a new platform since it builds daily.

2 Likes

ICU diffs really should go upstream to the unicode project.
LLVM diffs (which from a quick glance aren't in a mergeable state) should go to llvm.org

The dispatch changes are changes to flags, that can/should be controlled by the CMake invocation.
The majority of the swift changes are build infrastructure, that I don't see why they would not have been accepted. The changes for prespecialization are suspect and that deserves a far more comprehensive approach to resolving the issue which I suspect is code size. The changes to the refcount handling seem odd and possibly hint towards a platform bug rather than a swift issue.
The tools-support-core diffs look like they should be easy to get merged (and they already have been?)
SwiftPM diffs are a workaround, it doesn't have any context on what/why. In fact, it opens up concerns around ODR.

The changes really feel mostly like hasty workarounds rather than patches for cross-compiling. I understand that from their perspective that difference is uninteresting but in order to ensure that the Swift project remains healthy and easy to evolve, those considerations need to be made.

1 Like

Sounds like a great workgroup -- and if the focus indeed were about making it easier to use and develop Swift everywhere that'd be amazing. I think that fits all the mentioned tools -- from formatting, IDEs and getting started with Swiftly etc etc.

I'm also seconding the ask for benchmarking tools actually.

There's:

which exist but are not super active; it'd be great to have a more vibrant benchmarking ecosystem, like what we kicked off in scala/jvm ecosystem with JMH and GitHub - sbt/sbt-jmh: "Trust no one, bench everything." - sbt plugin for JMH (Java Microbenchmark Harness)

So removing all kinds of such missing tools and roadblocks would be awesome for such group.
Focusing on it becoming simple to quickly get to doing all kinds of such things with Swift.

8 Likes

We are also interested in benchmarking specifically - will try to take the time to write down our discussions and share.

1 Like

Yeah I would say IMO the charter would be 2 and 3:
Make developing and building Swift applications, scripts and services on any supported platform a great experience

Most of the other points follow on from that

Edit: expanded the charter slightly for my own selfish interest in Swift scripting :sweat_smile:

3 Likes