Configure utils/build-script for building header files only in CI

Development Compiler
1

I have a mixed C++/Swift package that I want to build in CI.

The C++ portion needs to include headers from the swiftlang/swift repo, specifically the C++ target has the similar configuration as in SwiftCompilerSources:

private extension Target {
  static func compilerModuleTarget(
    name: String,
    dependencies: [Dependency],
    path: String? = nil,
    sources: [String]? = nil,
    swiftSettings: [SwiftSetting] = []) -> Target {
      .target(
        name: name,
        dependencies: dependencies,
        path: path ?? "Sources/\(name)",
        exclude: ["CMakeLists.txt"],
        sources: sources,
        swiftSettings: [
          .interoperabilityMode(.Cxx),
          .unsafeFlags([
            "-static",
            "-Xcc", "-DCOMPILED_WITH_SWIFT", "-Xcc", "-DPURE_BRIDGING_MODE",
            "-Xcc", "-UIBOutlet", "-Xcc", "-UIBAction", "-Xcc", "-UIBInspectable",
            "-Xcc", "-I../include",
            "-Xcc", "-I../../llvm-project/llvm/include",
            "-Xcc", "-I../../llvm-project/clang/include",
            "-Xcc", "-I../../build/Default/swift/include",
            "-Xcc", "-I../../build/Default/llvm/include",
            "-Xcc", "-I../../build/Default/llvm/tools/clang/include",
            "-cross-module-optimization",
          ]),
        ] + swiftSettings)
    }
}

This works perfectly well locally. I simply follow the Getting Started guide in the swiftlang/swift repo, calling utils/build-script with the same configuration as in the guide:

utils/build-script --skip-build-benchmarks \
  --swift-darwin-supported-archs "$(uname -m)" \
  --release-debuginfo --swift-disable-dead-stripping \
  --bootstrapping=hosttools

The problem is this takes over 3 hours, and I would rather not build the whole thing from scratch every CI trigger when I just need some header files.

I’ve searched these forums and found two possible approaches:

  1. Use some build preset instead of the Getting Started configuration.
  2. Pass –-skip-build to utils/build-script and then call Ninja directly.

So my questions are:

Is there a suggested build preset that would just build the header files. Or at least a preset that would skip as much as possible? Maybe [preset: buildbot,tools=RA,stdlib=RD,test=no]?

What Ninja targets would be appropriate? The Ninja swift target suggested elsewhere on these forums doesn’t seem to exist anymore.

Many thanks!

2

+1 for this. Still have not found a satisfactory solution yet.

3

The most slimmed down set of commands I could figure out was:

utils/build-script –-release –-skip-build ...

ninja -C ../build/Ninja-ReleaseAssert/llvm-macosx-arm64
ninja -C ../build/Ninja-ReleaseAssert/cmark-macosx-arm64
ninja -C ../build/Ninja-ReleaseAssert/swift-macosx-arm64 swift-frontend

Note the Ninja target in the last line is swift-frontend not bin/swift-frontend and reported elsewhere.

But this still takes over 3 hours.

I think the path forward is to enable sccache and configure the GitHub - Mozilla-Actions/sccache-action: sccache github action action, seeing as I am using GitHub Workflows. Will report back.

4

I got the build down to 56 minutes in GitHub, with sccache reporting a 99% cache hit rate.

About 10 minutes is calling utils/update-checkout, so I might be able to optimise that further.

1 Like
5

Another approach I'd like to take is use a prebuilt header only git package (Failed due to the bad support of swift repo header location + SPM integrate issue) or include a zip library or artifactbundle. (including swift_6.2_macosx-arm64 / swift_6.2_macosx-x86_64 / swift_6.2_linux_aarch64 etc.)

So we will only need to build it once and use the downloaded content for later use.

1 Like
6

Update on this:

I went down the route of creating a prebuilt repo and in the process discovered exactly what build outputs were actually needed, which were … not a lot.

All the llvm/ADT and llvm/Support headers used by the runtime / standard library are already vendored inside swift/stdlib/include, so we don’t need llvm-project.

The only build output from swift was include/swift/Runtime/CMakeConfig.h, which is just:

// This file is processed by CMake.
// See https://cmake.org/cmake/help/v3.0/command/configure_file.html.

#ifndef SWIFT_RUNTIME_CMAKECONFIG_H
#define SWIFT_RUNTIME_CMAKECONFIG_H

#define SWIFT_VERSION_MAJOR "@SWIFT_VERSION_MAJOR@"
#define SWIFT_VERSION_MINOR "@SWIFT_VERSION_MINOR@"

#endif

I've updated by Github action to perform a sparse checkout and write CMakeConfig.h manually.

The only code update was to remove the dependency on SwiftEquatableSupport.h and reimplement EquatableProtocolDescriptor:

#include <swift/Demangling/ManglingMacros.h>

extern "C" const void *PROTOCOL_DESCR_SYM(SQ);
static constexpr auto &EquatableProtocolDescriptor = PROTOCOL_DESCR_SYM(SQ);

Now the whole build takes less than a minute.

The GitHub action:

name: Checkout Swift Headers
description: |
  This action checks-out certain header files from the swiftlang/swift
  repository.

inputs:
  path:
    description: Relative path under $GITHUB_WORKSPACE to place the repository
    required: true
  swift-version-major:
    description: The Swift toolchain major version.
    required: true
  swift-version-minor:
    description: The Swift toolchain minor version.
    required: true

runs:
  using: "composite"
  steps:
    - uses: actions/checkout@v4
      with:
        repository: swiftlang/swift
        path: ${{ inputs.path }}
        ref: release/${{ inputs.swift-version-major }}.${{ inputs.swift-version-minor }}
        sparse-checkout: |
          include/swift
          stdlib/include
          stdlib/public/SwiftShims
          stdlib/public/runtime/MetadataAllocatorTags.def
    - name: Write CMakeConfig.h file
      run: |
        cat > include/swift/Runtime/CMakeConfig.h << EOF
        #ifndef SWIFT_RUNTIME_CMAKECONFIG_H
        #define SWIFT_RUNTIME_CMAKECONFIG_H
        #define SWIFT_VERSION_MAJOR "${SWIFT_VERSION_MAJOR}"
        #define SWIFT_VERSION_MINOR "${SWIFT_VERSION_MINOR}"
        #endif
        EOF
      shell: bash
      working-directory: ${{ inputs.path }}
      env:
        SWIFT_VERSION_MAJOR: ${{ inputs.swift-version-major }}
        SWIFT_VERSION_MINOR: ${{ inputs.swift-version-minor }}

Updated Package.swift:

extension Target {
    fileprivate static func swiftRuntimeTarget(
        name: String,
        dependencies: [Dependency] = [],
        path: String? = nil,
        sources: [String]? = nil,
        cxxSettings: [CXXSetting] = []
    ) -> Target {
        .target(
            name: name,
            dependencies: dependencies,
            path: path ?? "Sources/\(name)",
            sources: sources,
            cxxSettings: [
                .unsafeFlags([
                    "-static",
                    "-DCOMPILED_WITH_SWIFT",
                    "-DPURE_BRIDGING_MODE",
                    "-isystem", "\(swiftCheckoutPath)/include",
                    "-isystem", "\(swiftCheckoutPath)/stdlib/include",
                    "-isystem", "\(swiftCheckoutPath)/stdlib/public/SwiftShims"
                ])
            ] + cxxSettings
        )
    }
}
1 Like