Reproducing the toolchain used by Xcode

kbongort · October 29, 2021, 7:13pm

Hi, I'm trying to make tweaks to the toolchain for use in Xcode, but I'm having trouble because it seems that Xcode's toolchain may include changes that aren't published to the release branches of swift, llvm-project, etc. For example, this change to support MacCatalyst availability checks doesn't exist in the 5.5 release branch, but must exist in the clang that ships with Xcode.

Am I missing something, or is Apple keeping secrets that make it harder for external devs to work with Xcode + the open source toolchain?

jonprescott · October 29, 2021, 8:23pm

Apple starts with the open-source compiler code(clang, swift) and adds it's own modifications for whatever reasons it wants, many having to do with deeper integration with Xcode, and many are new features that are destined for open-source once they have been fleshed out in the proprietary environment.

These modifications are proprietary to Apple. It's one of the reasons that Apple ditched gcc and went to clang when gcc was re-licensed under GPL version 3. Modifications may transition to the open-source distribution later.

kbongort · October 29, 2021, 8:35pm

Right, that's the sense I've got. But withholding changes is actively making it harder for people interested in contributing to the project (e.g. me) to do so, because first I need to try to reproduce Apple's secret changes to get a toolchain on par with the one that ships with Xcode, and only then can I do meaningful work on it.

I can understand that Apple may have some things they'd like to keep secret, but to the maximum extent possible, I'd like to see the entire toolchain open-sourced alongside Xcode releases. Swift's README has instructions for installing the toolchain into Xcode. It'd be great if that actually worked for complex projects that rely on Apple's as-yet proprietary modifications.

As with the example mentioned above, some of the modifications seem to exist in open source already, just not in the release branch.

jonprescott · October 29, 2021, 9:24pm

It sounds like you want to tinker with the integration of Xcode and the swift/clang infrastructures. You have now entered the proprietary Apple environment, for which Apple has long declared by actions and words that they are not interested in external developers directly interacting with the Xcode environment and application. They will take Feedback reports for suggestions that may or may not be realized, but, direct interaction is something that they have not allowed for decades. The only supported way to extend Xcode by external developers is via the Xcode extension process, which currently only provides for extensions to the Xcode editor.

kbongort · October 29, 2021, 9:30pm

No, I am not interested in tinkering with the integration of Xcode. My interest is strictly contained to the Swift frontend at the moment, as used by any clients including Xcode, SwiftPM, or the driver. However, I want to test my changes against an extant codebase that uses Xcode to build. This is made extremely difficult by the fact that I can't land my changes atop a drop-in replacement for the toolchain that ships with Xcode.

I think it makes sense to allow developers to be able to test toolchain changes against existing code that happens to use Xcode to build.

cltnschlosser · October 30, 2021, 3:53am

If you’re only changing the frontend you can set a build variable in Xcode to override the frontend. SWIFT_DRIVER_SWIFT_FRONTEND_EXEC

Where this gets used:

github.com

apple/swift-driver/blob/51103743d41546b6ed5191d419c4d53bebf69a4c/Sources/SwiftDriver/Toolchains/Toolchain.swift#L162


      
          /// Returns true if `SWIFT_DRIVER_TESTS_ENABLE_EXEC_PATH_FALLBACK` is set to `1`.
          private var fallbackToExecutableDefaultPath: Bool {
            env["SWIFT_DRIVER_TESTS_ENABLE_EXEC_PATH_FALLBACK"] == "1"
          }
          
          
/// Looks for the executable in the `SWIFT_DRIVER_TOOLNAME_EXEC` environment variable, if found nothing,
          /// looks in the `executableDir`, `xcrunFind` or in the `searchPaths`.
          /// - Parameter executable: executable to look for [i.e. `swift`].
          func lookup(executable: String) throws -> AbsolutePath {
            if let overrideString = envVar(forExecutable: executable) {
              return try AbsolutePath(validating: overrideString)
            } else if let toolDir = toolDirectory,
                      let path = lookupExecutablePath(filename: executable, searchPaths: [toolDir]) {
              // Looking for tools from the tools directory.
              return path
            } else if let path = lookupExecutablePath(filename: executable, searchPaths: [executableDir]) {
              return path
            } else if let path = try? xcrunFind(executable: executable) {
              return path
            } else if ![executableName("swift-frontend"), executableName("swift")].contains(executable),
                      let parentDirectory = try? getToolPath(.swiftCompiler).parentDirectory,

kbongort · October 30, 2021, 7:32am

Thanks, I'm aware of that variable but not sure where to set it in Xcode. Can you point that out to me please?

Getting the clang binary out of the equation would be helpful, though I think I've managed to find the necessary cherry-picks and fixes for that.

Even in the swift frontend, there are commits that should probably be in the release branch but aren't, like 04135a3.

cltnschlosser · October 30, 2021, 2:04pm

It’s a user defined setting in the project build settings: https://stackoverflow.com/questions/3968919/how-can-i-access-a-user-defined-xcode-build-setting

cltnschlosser · October 30, 2021, 2:06pm

Assertions will be disabled in releases, so that might be why that commit didn’t get cherry-picked.

kbongort · October 31, 2021, 5:41am

Are you sure this actually works? I tried setting it like this, but Xcode still used its own swift-frontend.