Hey @Max_Desiatov and all, I'm Pradeep, a Swift developer currently on Linux (Ubuntu). I've been exploring the utils/ directory in swiftlang/swift to understand how the existing Wasm sysroot implementation works before putting together a proposal for this project.
What I've mapped so far:
The entry point utils/build-script is a Python script that parses arguments in two modes — preset and normal — then hands control to BuildScriptInvocation in build_swift/build_script_invocation.py. The execute() method on that class orchestrates the actual pipeline: configuring CMake, building products, and running tests.
Products (stdlib, Foundation, libdispatch, etc.) are each represented as classes under utils/swift_build_support/swift_build_support/products/. My understanding is that the cross-compilation work involves creating a new product class (or extending existing ones) that accepts a sysroot path + target triple and invokes CMake with CMAKE_SYSROOT and a toolchain file set appropriately — generalizing what the Wasm target already does in an ad-hoc way.
Specific questions before I finalize my design:
Flag design: Should --experimental-sysroot be a standalone flag paired with an existing --cross-compile-hosts argument, or should this introduce a new argument group entirely? I see cross_compile_hosts defined in driver_arguments.py — is extending that the right entry point or would you prefer a clean separation?
Product splitting: The idea description says "split out Swift core library builds into separate cross-compilation build products." My reading is that this means a new product class that wraps stdlib + core libs specifically for cross-compile scenarios, rather than modifying each existing product class to conditionally handle cross-compilation. Is that the right architectural direction?
Test bar: For the deliverable "tests pass on target system" — is the expectation QEMU user-mode emulation (qemu-aarch64-static) on the build host, or is there a preferred way the project validates cross-compiled binaries today?
Wasm as first consumer: My plan is to refactor the current Wasm sysroot handling to use the new generic mechanism as its first real-world user, both to validate the abstraction and to avoid regressing existing Wasm builds. Does that align with your vision for the project?
I'm on Ubuntu x86_64 which feels like the right environment for this — cross-compiling to other Linux targets from Linux avoids macOS-specific toolchain complexity. Happy to put together a draft proposal once I have clarity on the design questions above
[edit by @ktoso: the whole post was bold, I have removed the font weight]
The project is about cross-compilation targets, not cross-compilation hosts. I'm not sure how --cross-compile-hosts would be relevant.
The means new product classes for each core library that supports cross-compilation scenarios, rather than using a single big product class that cross-compiles multiple core libraries at once. For example wasmswiftsdk.py product manages XCTest, libxml2, Foundation, and much more, as opposed to each library having its own product.
This completely depends on the target platform, there's no universal answer to this. Wasm is not and can't be tested with QEMU, it's tested with WasmKit. For other cross-compilation platforms it could be the opposite.
Yes, that overall makes sense. It would help if you have some other targets in mind that would help you validate the approach.
It is not clear why you want that: I myself don't understand why you replicated all that logic for Wasm alone, so it might help to explain why you did that.
Thank you both for the clarification — this really helps sharpen the design.
I understand now that --cross-compile-hosts reflects the historical path through which corelib cross-compilation has been triggered, but the correct conceptual framing for this project is target-side rather than host-side. So the new --experimental-sysroot flag should live in its own clean target-facing argument group rather than extending the existing host cross-compilation path.
On product splitting — confirmed: the intent is one product class per core library rather than a single monolithic product. wasmswiftsdk.py would be what this refactor moves away from, not toward.
For additional validation targets beyond Wasm, I'm thinking Android (aarch64/armv7 via NDK sysroot) and embedded Linux with musl libc — these two differ enough architecturally that supporting both should genuinely stress-test the generic mechanism. Does that scope feel right to you, Max?
"host" is the proper term of art. The three terms here are:
build: the machine that you are building on host: the machine that the code will run on target: RARE the host that the code that host generates will run on (e.g. host = aarch64-unknown-windows-msvc and target = aarch64-unknown-linux-android is a Windows ARM64 hosted compiler that generates code for ARM64 android).
Thank you for the precise clarification, Saleem — the build/host/target distinction is really helpful to have clearly defined as I work on the proposal.
--experimental-sysroot and --experimental-sysroot-triple flag infrastructure
SysrootConfig dataclass design
Per-library product class refactor (splitting wasmswiftsdk.py)
Generic CMake integration via -DSWIFT_EXPERIMENTAL_SYSROOT
End-to-end validation plan for Wasm + a second target (Android NDK or musl Linux) @Max_Desiatov — you confirmed the per-library split was the right direction. Would appreciate a quick check that the Phase 4 refactor reflects what you had in mind. @finagolfin — given your point on --cross-compile-hosts being the historical path, would love your take on whether --experimental-sysroot as a clean separation in Phase 1 is the right call, or whether it should integrate with the existing path instead. @compnerd — thanks for the build/host/target clarification — the proposal uses host-side framing throughout to stay consistent with that.
Happy to revise before the submission deadline. Thanks for the guidance so far.
I've never understood the emphasis on the platform C/C++ sysroot, as while it is true that build-script does not have an easy way to pass in such a generic platform sysroot, that is a tiny part of cross-compilation.
The larger issue is putting together all the cross-compilation flags needed for your platform and if you look at the wasi*.py build configuration scripts, most of it is passing in all the various flags they need.
For Android, I was able to cross-compile the corelibs six years ago by adding about a dozen flags and tying into --cross-compile-hosts, so there wasn't much to add. I used some prebuilt Android libraries for Foundation dependencies like libxml2 and libcurl though, as I was just distributing that Android SDK myself, so I wasn't building everything from source like the official wasm SDK does.
The way I see it, there are two separate goals here:
build-script depends on a large bash script called build-script-impl, which is where most of the corelibs are still built, ie they don't have Python build products. We have been slowly translating those over from the bash script to Python over the years, eg see the small cmark transition and the massive LLVM translation, both done by @drexin. The corelibs are closer to cmark in size, luckily for you, and I think everybody agrees that this should be done.
Having a more standardized way to cross-compile Swift libraries from those Python build products, for which the long-gestating support for CMake Toolchain files in build-script would be the natural approach (a grep doesn't seem to show the wasm files using that though), as products like cmark, llvm, and Testing already use it. Obviously, each build product would likely need its own flags too, but you can probably put the core cross-compilation config in a toolchain file for most targets. This aspect is more open-ended and subject to discussion, so you will have to hash it out with some Swift people.
As you might see from my initial Android link, the current support for cross-compiling the corelibs is not great and we can certainly do a lot better, but what's there now works for macOS and Android. Take a look at those links and see what approach seems best to you, then hash it out with some Swift people like @kateinoigakukun, or whatever the GSoC process is.
Based on his feedback, I now understand the project has two distinct parts:
Migrating the corelib products from build-script-impl (bash) to Python product classes — the pattern already established by cmark and LLVM
Standardizing cross-compilation in those new Python classes, for which finagolfin pointed to CMake toolchain files as the natural approach
My question for you specifically: the wasm products (wasmswiftsdk.py, wasmstdlib.py) currently don't use CMake toolchain files, while cmark, LLVM, and Testing already do. Was there a deliberate reason the wasm files diverged from that pattern? And would migrating them to use a toolchain file be the right direction for this project?
This would significantly reshape my current proposal and I want to get the architecture right before the submission deadline.
It makes sense to use CMake toolchain files. But beyond that, we need to keep in mind that each product has its own CMake options, and each platform has its own value to specify for them. For example, WASI platform does not support dynamic linking, so we need to specify some of Swift stdlib specific options. So it would be great if you could include how to standardize them to make it easier to add new platform support in your proposal. (e.g. Rust toolchain's build system has a way to configure options per target platform rust/bootstrap.example.toml at 4cf5f9580233c36f6bc8db76e282ba8a1c1ea491 · rust-lang/rust · GitHub)
Thank you — CMake toolchain files make sense and I'll incorporate that into the proposal.
For per-platform, per-product options: the Rust bootstrap.example.toml link you shared was really helpful. Looking at how Rust handles per-target configuration, I also realized my current --experimental-sysroot flag design only handles one target per invocation — which breaks down when you need to build for multiple targets at once. Rust solves exactly this with its [target.<triple>] config blocks.
So I'm rethinking the flag infrastructure to a --cross-compile-config flag pointing to a targets.json file, where each entry carries a triple, sysroot, toolchain file path, and an options block for platform-specific flags like has_dynamic_linking: false for WASI. Python reads this, constructs a SysrootConfig per target, and each product class queries it for its CMake flags — all in one invocation.
Does this align with what you had in mind, or is there a config format that fits build-script's existing patterns better?
and I think everybody agrees that this should be done.