Separate CMake process into compiler and stdlib

Development Compiler
,
1

Current state

Currently, the build system build stdlibs for multiple SDKs in single cmake invocation.

This design of build system makes it complex to support multiple platforms as scalable because we need to configure them at once.

This design would work well if there is no difference in options between SDKs. However, if you want to change the options for each SDK, you have to either add options for each SDK to the add_library wrapper,

or call add_library for each SDKs using foreach in CMakeLists.txt.

If CMake configuration itself runs for each platforms, CMakeLists.txt can focus only on the platform.

To achieve the desired state, we need to separate stdlib CMake process for each platforms. But currently compiler and stdlib are built at once in single CMake unit. Therefore, we need to separate cmake process into compiler and stdlib at first.

Desired State

Here is an example of build directory structure after this separation.
This follows the manifest of build system. swift/BuildManifesto.md at main · apple/swift · GitHub

The new structure of swit-macosx-x86_64 is compatible with current directory structure, so there shouldn't be any effect for packaging toolchain.

Ninja-ReleaseAssert/
  swift-macosx-x86_64/
    bin/
      swiftc
      swift
      …
    lib/swift
      macos/ (symbolic link to ../../swiftstdlib-macosx-x86_64/lib/swift/macosx)
      iphonesimulator/ (symbolic link to ../../swiftstdlib-iphonesimulator-x86_64/lib/swift/iphonesimulator)
      iphoneos/ (symbolic link to ../../swiftstdlib-iphoneos-arm64/lib/swift/iphoneos)
  swiftstdlib-macosx-x86_64/
    lib/swift/macosx/
      libswiftCore.dylib
      …
    test-macosx-x86_64/...
  swiftstdlib-iphonesimulator-x86_64/
    lib/swift/iphonesimulator/
      libswiftCore.dylib
      …
  swiftstdlib-iphoneos-arm64/
    lib/swift/iphoneos/
      libswiftCore.dylib
      …

Steps

To evolve incrementally and safely, I propose these steps.

Step 1

Move statements related with only stdlib in the root CMakeLists.txt into ./stdlib/CMakeLists.txt

At this phase, compiler and stdlib are built in single cmake process but stdlib CMake variables scope is isolated from compiler CMake unit.

Step 2

Change CMake system to build ./stdlib standalone.

And change build-script to invoke cmake for compiler and stdlib separately.

At this phase, directory structure would be like this


Ninja-ReleaseAssert/
  swift-macosx-x86_64/
    bin/
    lib/swift
      macos/ (symbolic link to ../../swiftstdlib-macosx-x86_64/lib/swift/macosx)
      iphonesimulator/ (symbolic link to ../../swiftstdlib-macosx-x86_64/lib/swift/iphonesimulator)
      iphoneos/ (symbolic link to ../../swiftstdlib-macosx-x86_64/lib/swift/iphoneos)
  swiftstdlib-macosx-x86_64/
    lib/swift/
      macosx/
        libswiftCore.dylib …
      iphonesimulator/
        libswiftCore.dylib ...
      iphoneos/
        libswiftCore.dylib …
    test-macosx-x86_64/

Step 3

Deprecate SWIFT_SDKS and change build-script to invoke cmake for each SDK.

At this phase, directory structure should be the same as the desired state.

Step 4

To evolve further, invoke cmake for each os/arch/platform triple instead of SDK.

Draft PR

Here is a draft change for this separation. (but not cleaned up yet, so still draft)

This branch is in "Step 2" and all test cases including validation-test are passed.

I want to confirm that this separation is worth doing. cc: @compnerd

https://github.com/kateinoigakukun/swift/pull/56/files

2 Likes
2

There is already a bigger plan here to get this to work. Just splitting the cmake process is not sufficient, what we need to do is far more complex. We need to be very careful here in how we do this. I can elaborate more in a bit. That being said, I would definitely want to review/be part of this process.

4 Likes
3

At a high level, I think that having someone else do this would actually be a net reduction in engineering time to the team since it is an involved change that would require me to hand hold/there is additional work that would need to be done to make sure it doesn't break internal stuff. That would at least require me to do part of the work. My overall plan is to hopefully get a chance to do this sometime over the summer.

1 Like
4

Thanks for telling me the plan!

Until your work done, how can I help the project to evolve?
Can moving stdlib options like SWIFT_STDLIB_ASSERTIONS into ./stdlib/CMakeLists.txt conflict your work?

5

I have no opinion on your or other plans in this direction, but I'll note that by cleaning up some of the current CMake config, I was able to do part of what you seem to want, ie cross-compile the stdlib for a single target without having to build the host compiler or stdlib. I used this build-script command to cross-compile the stdlib for Android with the prebuilt official compiler for linux. You may want to try that pull, and help get it merged upstream if it helps you.

2 Likes
6

@Finagolfin what we want to do is a bit more interesting than what you did here.

But before that we /already/ support building using a just build swift with build-script. We just need to get it on a bot. It is almost there. There is even a preset I created for it. I think it has atrophied slightly though b/c we don't have that bot. Looks at @mishal_shah = p.

7

what we want to do is a bit more interesting than what you did here.

Great, I'm looking forward to it.

But before that we /already/ support building using a just build swift with build-script.

Yes, I know, I saw that preset and used parts of it in my own build-script invocation, though updated to use the new build-script flags instead.

We just need to get it on a bot. It is almost there. There is even a preset I created for it. I think it has atrophied slightly though b/c we don't have that bot.

Good to hear it will be put on some CI.

All my small linked pull does is make sure that standalone and cross-compiled stdlib support works by moving around some stdlib flags that have been added in the CMake config over the years, which break this use case. I use that pull in its entirety to cross-compile the stdlib alone for Android.

I ask that you review and merge that pull before the Swift 5.3 branch, so that I won't have to patch that too locally, and then you can do the larger rework you want this summer.