I understand this implies an open-sourcing of significant Apple resources, which is already fantastic for the community and the spread of the language.
But: are there any comparisons / benchmarks as to why Swift Build is actually better than what SwiftPM provides today? Is it faster, more correct, etc? What I’m missing is the technical motivation for this - what is „broken“, to put it unflattering - about the status quo?
Just for my understanding: Swift Build is akin to ninja (with an in-Memory PIF representation rather than a ninja manifest) and SwiftPM is akin to make or CMake? Am I getting the distinction right?
First class support for cross-compilation - as Jake mentioned in Evolving SwiftPM Builds with Swift Build - #11 by jakepetroules, Swift Build essentially treats every target as a cross compilation target. Baking this constraint into the build system at a low-level has important correctness benefits, and means that support for new platforms can be added in a largely data-driven manner. We've received a PR from @kateinoigakukun already to add WebAssembly support, which I think demonstrates this modularity well: Sign in to GitHub · GitHub. As a bonus, I think this is also a major stepping stone towards first-class support for Swift SDKs.
A more flexible intermediate representation. Swift Build has very flexible abstractions for defining new build configuration options and product types, for things like compiler and linker settings, libraries, apps, and more. I think there are a lot of opportunities to think about how we could evolve the Package.swift manifest format to take advantage of this flexibility.
More dynamic build rules for Swift. Like SwiftPM's current build system, part of Swift Build's planning process involves generating a static graph of tasks. Unlike the current build system however, Swift Build also has the concept of dynamic task graphs which can be spliced into that static task graph, leading to more of a hybrid model. This has enabled us to integrate with the compiler more effectively using tools like the SwiftDriver library to schedule tasks more efficiently and leverage more detailed knowledge of Swift's compilation model to implement optimizations that aren't possible when we treat the compile of each module as a black box. One example of this - Swift Build will prioritize the emission of a target's swiftmodule over the object files. This allows it to begin compiling dependents of that module before the objects are all finished compiling, which can be a major speedup in heavily modularized projects. This category of optimization is much harder to do in SwiftPM's current build system without a substantial rework of how it constructs the build graph. It's also an area with a lot of opportunity for further work, IMO. For example, it lays some of the foundations to begin looking at interleaving build planning with build execution and unlock more gains.
It's hard to draw a precise analogy to other build systems, but I think a good way of thinking about this is:
SwiftPM remains the user facing component developers interact. When building, it interacts with Swift Build's build service to lower Package.swift to the PIF intermediate representation, and sends it a build request.
Swift Build is responsible for accepting a PIF and a build request, planning a build using the PIF, builtin rules, and user-provided rules.
Swift Build then executes that build graph in coordination with LLBuild
Also, will the new régime incorporate this PR to SPM?
Yes, that feature is still a work in progress BTW. But I did it intentionally at the package load layer so it would work with any build system that may come along. I'm hoping Swift Build will offer better ways of doing it, but I'm still learning it along with everyone else.
Thanks a lot for taking the time to reply @owenv. I'm excited especially about the first-class cross-compilation support, and the optimizations this change unlocks. Thank you for working on this and for being part of the efforts to open source it.
I should note, and I assume it's just growing pains: I recently had some annoying bugs working with cross-platform SwiftPM packages in Xcode, which I didn't have with SwiftPM directly. So I fear those bugs will now move over to SwiftPM as well.
Specifically, I mean opening a Swift Package in Xcode that has e.g. condition: .when(platform: [.android]) but Xcode still wanted to build that target for the one Xcode was set to (e.g. My Mac), which fails.
I also find SwiftPM to work more smoothly with local packages. Xcode does not seem to automatically update after you edit a local Package.swift, and it's necessary to choose File -> Packages -> Reset Package Cache. That seems to delete the entire world and also re-download remote dependencies, so it's very much overkill for what could happen. I don't remember seeing this issue with SwiftPM itself in the last ~5 years.
I'm not sure to what extent either of those issues are directly related to, or will be inherited by, a move by SwiftPM to Swift Build? The silver lining is that these bugs will hopefully be more visible now and more readily fixed? I'm happy to file reproducers for these but I'm not sure where. In the case of editing a local Package.swift, I'm also not sure how.
On the topic of targets: you alluded to Swift Build enabling more fine-grained control over target compilation. We have e.g. ML and DSP code that is functionally useless in -Onone due to its slow execution speed, so one thing that would be a huge QoL improvement for us in SwiftPM would be the ability to override optimization settings per target (without using unsafe flags). Xcode targets allow this. Would Swift Build be the right project to request this functionality from?
Package resolution is outside the scope of the Swift Build project, and remains something handled by either SwiftPM, or the IDE in coordination with SwiftPM. For this issue, I recommend filing a bug using the Apple Feedback Assistant since it's related to IDE functionality
I think there are two parts to enhancements like this:
The underlying build engine needs to support the new config option - This is the domain of Swift Build
User-facing components need to expose the option as part of the tools' UX - This is SwiftPM's domain
So really, it's a collaborative effort for many of these types of changes. In practice if you're filing a feature request, it's ok to make a guess if you're not sure which repository is most relevant, it's quite easy to transfer issues around these days if needed
Fair enough. I would file a bug on Xcode but I've never received a response to or seen resolution of my Apple Feedback reports in the past, so it feels a little futile.
While we can't promise a response to every feedback report or timelines on fixes, I just want to emphasize that we value these reports and we do read them. Please don't be discouraged from reporting issues!