How to build swift-corelibs-foundation?

DarkDust · April 22, 2025, 8:56pm

I wanted to work on swift-corelibs-foundation but I found it to be a pretty frustrating experience. Building it on macOS doesn't work, so I tried to set up a Linux VM instead.

The actual question I'm having is: is there some recommended setup, some tips or hints on how to build swift-corelibs-foundation?

People are making contributions so I feel I'm missing something and have taken the wrong road, again. Any guidance is appreciated.

After other attempts, here's my current setup:

A VM of Ubuntu 24.04.2 LTS on ARM64 (for performance on my MacBook Pro).
I have Swift 6.0.3 installed via swiftly.
Current main branch of swift-corelibs-foundation (6a0dd7335e77dcf33e6ab4d73d8e8ab3d1d347d0 at the time of this writing).
Buildung using DISPATCH_INCLUDE_PATH=/home/user/.local/share/swiftly/toolchains/6.0.3/usr/lib/swift swift build

I get several errors, for example:

Sources/Foundation/NSCalendar.swift:45:13: error: switch must be exhaustive (lots of missing calendar enum cases)
Sources/Foundation/NSString.swift:1695:2: error: replaced function '_cfStringEncodingConvert(string:using:allowLossyConversion:)' could not be found
Sources/Foundation/NSString.swift:1700:2: error: replaced function '_cfMakeStringFromBytes(_:encoding:)' could not be found

(It's easy to work around the NSCalendar issue, but I'm lost about the "replaced function could not be found" errors.)

Finagolfin · April 24, 2025, 7:02pm

Not that I know of, but now that it switched to being a Swift package with Swift 6, it shouldn't be that hard.

I don't recommend using the Swiftly toolchain to build Swift toolchain repos, as others have reported issues with it. Instead, download the latest trunk snapshot toolchain for your distro yourself, unpack it, and add it to your PATH.

Check out the trunk source snapshot tag that matches your trunk toolchain, eg git checkout swift-DEVELOPMENT-SNAPSHOT-2025-04-12-a, and see if that works.

If not, try checking out the dependencies swift-foundation, swift-foundation-icu, and swift-syntax to the same trunk tag in the same directory, plus swift-collections to the 1.1.3 tag, then build with SWIFTCI_USE_LOCAL_DEPS=1 swift build, to replicate how the CI builds it.

jmschonfeld · April 25, 2025, 3:40am

To me all of these errors sound like you might be building a commit of swift-corelibs-foundation that is newer than the commit of the swift-foundation dependency. This can happen when doing a git pull of the SCL-F repo without asking SwiftPM to pull dependencies as well. Could you try running swift package update in your SCL-F checkout to pull the latest commits for the swift-foundation dependency and try building again? Using a Linux VM on your Mac with a swift toolchain installed using the command you provided should be sufficient to work. For reference:

Note: Subsequent SwiftPM builds will not automatically update to newer commits of downstream Foundation projects like swift-foundation-icu. If you encounter build failures after checking out a new branch or pulling from the remote repo, you can run swift package update to update your local dependency checkouts to match top-of-tree to ensure you have the required versions of dependencies.

github.com/swiftlang/swift-foundation

Foundation_Build_Process.md

main

# Foundation Build Process

This document outlines how Foundation (swift-corelibs-foundation, swift-foundation, and swift-foundation-icu) is built in various scenarios. Foundation is built in a variety of different ways between at desk, in testing CI, and in full toolchain builds. Each build variant serves a purpose outlined below but may have subtle behavioral differences compared to other variants.

For the purpose of this document, an "Individual Build" indicates a build of just a particular repo (and any repo within the realm of Foundation/our packages that it depends on). This is different from a toolchain build which must always build the entire toolchain (the compilers, the standard library, all of Foundation, and other projects like dispatch/swiftpm/etc.).

## Getting Started

If you're looking to quickly get a build of Foundation locally to test a change, the fastest way is to build via SwiftPM. To do so, follow these steps:

**Via the Command Line (all platforms):** run `swift test` to build the project and run all unit tests, or `swift build` to just build the project without running tests

**Via Xcode (Apple platforms only):** Open the `Package.swift` file in Xcode and click *Product > Test* to build the project and run all unit tests, or *Product > Build* to just build the project without running tests

This is the quickest way to iterate on Foundation to add new APIs or fix bugs. SwiftPM's command line interface and IDE integrations provide a variety of options to run specific tests or run tests in parallel to achieve results faster. Check out the detailed sections below for more information about the SwiftPM build or the other build variants for Foundation.

## Individual Build (via SwiftPM)

Each individual project can be built via SwiftPM. This is useful when quickly iterating at desk and when developing within IDEs with integrated SwiftPM support such as Xcode. The SwiftPM build is configured via the `Package.swift` manifest at the root directory of each project. This build format supports building only Foundation components and their immediate dependencies without the need to build other toolchain libraries such as the standard library or the compiler itself. This is also the only build format that supports building and running unit tests.

This file has been truncated. show original

DarkDust · April 27, 2025, 6:04pm

Thanks for the feedback. After more experiments I now almost succeed in build swift-corelibs-foundation. It builds but can't link because it exhausts the number of open files. I suspect it's SwiftPM bug 4584:

[1/1] Planning build
Building for debugging...
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: fatal error: out of file descriptors and couldn't close any
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: fatal error: out of file descriptors and couldn't close any
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[1/3] Linking plutil

I even raised the number of open files a process can have to 100 million (!) via sysctl and limits.conf but that's not enough.

So here's my current setup:

Ubuntu 24.02.
Downloaded and installed Swift 6.1-RELEASE from the "Install Swift" page.
Then did the equivalent of the following script:

TAG=swift-6.1-RELEASE
git clone https://github.com/swiftlang/swift-collections
for repo in swift-corelibs-foundation swift-foundation swift-foundation-icu swift-syntax ; do
  git clone https://github.com/swiftlang/$repo
  cd $repo
  git checkout $TAG
  cd ..
done
cd swift-corelibs-foundation
DISPATCH_INCLUDE_PATH=/path/to/swift-6.1-RELEASE-ubuntu24.04-aarch64/usr/lib/swift/ SWIFTCI_USE_LOCAL_DEPS=1 swift build

That seems to work well, no more compile errors. But unfortunately, the aforementioned linker error.

$ cat /etc/sysctl.conf
fs.nr_open=102457600

$ cat /etc/security/limits.conf
* soft nofile 104857600
* hard nofile 104857600

$ ulimit -n
104857600

Finagolfin · April 27, 2025, 6:41pm

Does using lld instead work? Try <env vars> swift build -Xswiftc -use-ld=lld.

DarkDust · April 27, 2025, 7:37pm

Not much better, unfortunately:

error: link command failed with exit code 1 (use -v to see invocation)
ld.lld: error: cannot open /home/marc/utm/swift-corelibs-foundation/.build/aarch64-unknown-linux-gnu/debug/CoreFoundation.build/CFStream.c.o: Too many open files
[about 20 similar lines omitted]
ld.lld: error: cannot open /home/marc/utm/swift-corelibs-foundation/.build/aarch64-unknown-linux-gnu/debug/CoreFoundation.build/CFUnicodePrecomposition.c.o: Too many open files
ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[104/106] Linking plutil

Finagolfin · April 27, 2025, 7:56pm

Unless you have many other builds going on simultaneously, there's no way you've got millions of file descriptors open when linking Foundation alone. Something else is going on: I suggest you use tools like lsof to investigate further, or search online for known bugs of whatever VM tool you are using.

DarkDust · May 4, 2025, 11:20am

I tried to find out what's going on but ultimately gave up and tried with a different VM: instead of Ubuntu 24.04 on my MacBook Pro using UTM (ARM64), I'm now using Ubuntu 22.04 on a professionally hosted VM (x86_64, Hetzner). And that finally works!

No idea what makes the difference, whether it's the different Ubuntu version or CPU architecture. Anyway, I can finally start working on my PR so thank you for your help!