Swift static libraries in binary targets on non-Apple platforms

Using Swift
1

Background

I'm looking for ways to improve build times of packages that depend on moreSwift/swift-winui (i.e. apps that use SwiftCrossUI on Windows). Clean debug builds currently take around 10 minutes and clean release builds currently take around 20 minutes. And that's when I'm building it in a VM on my desktop that has 12 cores and more than enough RAM, which is close to the best case scenario.

I've looked into quite a few different ways to improve build times of the swift-winui family of packages, but all approaches that I've tried have made only marginal improvements, if any at all. In particular, swift-uwp has around 170k lines of generated Swift code, which kind of limits how fast these builds can become without pretty targeted profiling of the Swift compiler.

That's what has led to me looking into distributing pre-built Swift libraries on Windows.

Distributing pre-built Swift libraries on Windows

SE-0482: Binary Static Library Dependencies introduced support for shipping and consuming binary static libraries using SwiftPM on non-Apple platforms, which is almost exactly what I need! Unfortunately, support for Swift static libraries was doomed to the future directions. First, I have a couple of questions;

  • Has anyone already put work into supporting that future direction? I'd like to avoid doubling up work if possible
  • If not, does anyone familiar with those parts of SwiftPM have a vague idea of how complex the implementation might be? My assumption is that much of the complexity will come from updating the auditing tool?

If I did pursue this approach, I'd probably have to set up a separate swift-winui-prebuilts package that contains prebuilt versions of the library products present in the latest swift-winui releases, which isn't ideal because it'd require some wiring within SwiftCrossUI's package manifest to allow people to opt in and out of the WinUI prebuilts. It'd also require developers to take care when explicitly depending on swift-winui alongside swift-cross-ui in their projects.

Alternative approach 1 (band-aid)

An alternative approach that I've been playing around with is extending SwiftPM's Swift Syntax prebuilts system to support WinUI prebuilts. For now, that'd be in a custom fork of SwiftPM that people can use if they want to speed up their SwiftCrossUI app builds on Windows. That's obviously not ideal, because it requires keeping a SwiftPM fork up to date and maintaining builds for people to use, but once developers have the fork installed, it would provide a more convenient experience, because they could opt in and out of prebuilts from the command line, no matter how they depend on WinUI.

Alternative approach 2 (long term)

The better version of this approach long term (imo) would be to design and implement a 'prebuilts' feature that all packages can adapt across all platforms supported by Swift, but that's likely more work than I have time for at the moment (the evolution process can be quite arduous).

Out of the potential designs I have in mind, I think that the most likely ideas to be accepted would either be some sort of extension of the Registry protocol to allow clients to request prebuilts in place of source code when available, or a new protocol akin to the Registry protocol but specifically for requesting prebuilts. I prefer the latter because it leaves the door open to supporting prebuilts for packages published as GitHub repos. Either way, developers would opt-in per machine/workspace using registry/prebuilt configuration files, eliminating some of the security arguments that I could imagine coming up (because if any prebuilts are to end up in a build, the developer must explicitly opt in).

This approach would also allow developers to use prebuilts for packages that don't officially vend any, and it'd allow different developers to use different prebuilts for the same packages; both of which independent developers and companies would appreciate alike.

@dschaefer2, is opening up prebuilts to third party packages on the table?

5 Likes
2

I'm actually working on a new mechanism to integrate binaries into packages. I've been playing with porting one of the "How to Make a 2D Game in C++ and SDL3" YouTube videos to Swift with SDL3. It's really opened my eyes to how hard it is to leverage the Release binaries for a project like this in a Swift package.

So, I'll confess, as you've seen, the swift-syntax prebuilts are heavily hard coded in SwiftPM. I really took advantage of having a good say in how they're laid out on swift.org. It's not necessarily a layout I'd want to force on everyone. And I really didn't provide any mechanism to make that extensible. Not everyone will want to create a fork :slight_smile: .

But there are lessons to be learned there, like how to make sure the downloads are secure with the right signatures and checksums. And the biggest lesson was how to deal with compiler compatibility when including swiftmodule files in the download. And of course how poor our PlatformConditions are at describing Linux.

Though the biggest benefit of the prebuilts is that if SwiftPM can't find the binary for your situation, it does resort back to building from source so it always works. In that sense it is a global build caching mechanism where you get to leverage someone else's build artifacts to speed up your own.

That said, I don't think we should bind ourselves to supporting the current mechanism as API. That's a decision that lasts forever and I don't think we're ready for that yet and can hopefully come up with something better.

I'd rather spend time in the shorter term on a more general solution that also handles cases like I had to deal with in my SDL experiment. There, artifact bundles and binary targets also fell apart. But it really did feel equivalent to the prebuilts experience where I could use the binaries if they were available and build from source when they are not.

The question is how do we specify what binaries are available and how to hook them up in the build system and how do we make sure they're secure. Ideally everything would be in the package manifest but there's a chicken and egg situation where we can't really update the manifest after the release build is done. I've had several versions of some sort of manifest file for prebuilts and that seemed to help. But again those need to be signed for trust.

Or, do we let the packages that add the dependency to your package specify that, like we do with binaryTargets today. Though that makes it much harder to adopt.

I'm driven to solve this as I've started work on being able to build Swift SDKs from packages. They'll have large build times as well that would be solved with prebuilts. A more general mechanism to support prebuilts for any Swift package would certainly help. But let's see if we can solve this with a common solution we only have to build one of and not have it hard code the swift.org download site :slight_smile: .

8 Likes
3

I don't understand what this means: do you mean listing prebuilt Swift SDKs to use in the package manifest, so they are automatically downloaded and installed?

4

I've talked about it on a few working group calls. But it's essentially defining a Swift SDK as a product of a package and then using swift build to build the components and assemble them with whatever manifest files SwiftBuild needs to consume them. You then would want to publish that build output as a prebuilt for that package so not everyone is building Swift SDKs all the time.

I'll write up a fuller post on that subject once I figure out a few more details and understand the problem better. Swift SDKs right now, as you are very familiar with unfortunately @Finagolfin :slight_smile: , are very hard to build but also hard to find and install. This would be an attempt to help with that by making them a first class citizen in the package ecosystem. Lots of work ahead though.

5

Still unclear on this, nobody builds the platform SDKs like the Swift SDK for Android "all the time." It is built once on swift.org for each tag and then everybody downloads and uses that prebuilt platform SDK locally or on CI.

Yes, hard to build, which is why we only do that once at swift.org. You really want users building Swift platform SDKs locally? Note that the stdlib and corelibs builds are only configured by CMake right now.

As for finding and installing the prebuilt platform SDKs, pretty easy to do now, but I have long agreed with you that we could make that easier by tying them into the package manifest somehow, no disagreement there.

6

That's true for the lucky few. But there are environments where developers want to customize how the SDKs are built. WASM is one. We also need to create SDKs for Embedded and get those runtimes out of the toolchain to help reduce its size. And that world is full of different operating systems that it would be nice to have SDKs for. We need to allow for others to be able to produce SDKs. And allowing SDKs in package registries would help a lot with that and help with their discoverability as well.

1 Like
7

But the point of this post is to open the discussion on how to generalize prebuilts and I have started to give that some thought as a key part of that feature and to help big packages like SwiftCrossUI and to leverage existing mechanisms of distributing prebuilts securely.

4 Likes