We are starting to code more of the build system interaction with llbuild in Swift, which uses bindings in the llbuild repository. As part of that, we would like to use the infrastructure we have built up for SwiftPM. SwiftPM depends on llbuild, so this would introduce a cycle to the existing process.
To solve this, we are creating a new repository "GitHub - apple/swift-tools-support-core: Contains common infrastructural code for both SwiftPM and llbuild." which will be a new Swift package which contains the infrastructural code that both SwiftPM and llbuild need. We are still working out the details around the build process and CI integration, and we plan to use the JIRA component "llbuild" to track work related to this repository.
NOTE: This code is meant to support building Swift open source projects, and not intended as general purpose, Apple-supported APIs.
Would this include the code within the Utility library of SPM, (things like the ArgumentParser) or would it be just llbuild integration? In any case, it would great to have the Utility code in a smaller repo than SPM with proper semver releases
And indeed, this would be a great chance for us to start using semver so we can also live on the same experience we are trying to encourage other packages to use!
We want tools-support-core to be a Swift package but we also want to leverage Swift CI infrastructure to add CI support. We need a way to bootstrap this library on the CI without using SwiftPM as we need a cross platform solution to easily bootstrap for all platforms that are currently supported and the new platforms that might be added in the future. To solve this problem, we will add a new backend in SwiftPM to emit a Makefile. The Makefile support will be only accessible using an hidden flag and will be intended to be used only for Swift related projects. We will check-in the Makefile to tools-support-code and execute it on the CI using Swift's build-script to build and test the project. Most of the time, a contributor is expected to only require a recent trunk toolchain to build the repository using the SwiftPM that is present in the toolchain.
Makefile location and layout:
TBD.
Module prefix:
Currently, Swift requires module names to be unique across the package graph. To avoid conflict with other modules in the package graph, we will use the prefix "TSC" in all of the modules in this repository.
Git history:
Git history is usually a very useful feature but re-writing it could be annoying. Given that we don't really have huge amount of code that we want to factor out, we will do a "best effort" of preserving the git history.
ToolsSupportCore product:
We will create a "ToolsSupportCore" library product in the Swift package. This product will re-export all library targets that the package wants to make "public".
I like the idea of moving core modules from SwiftPM (Basic, Utilities, etc) to this new ToolsSupportCore repository. It would make it easier to do proper semver releases that way.
I'll be happy to help with this migration, if there is any way possible.
The major issue here is figuring out the bootstrapping process on the CI as we would need to build this new package before building SwiftPM itself. I am now less convinced of the Makefile approach described above and I think we should look at different options like CMake or using an older version of SwiftPM to bootstrap.
CMake is powerful but last I checked (and just know with a quick Google) it didn't support Swift projects so it will add some pain there but it should be very doable.
If we can just use SwiftPM it has the advantage of being just 1 technology people need to know work on TSC components. Would it be sufficient to use the most recent official release of SwiftPM for this so it is stable but hopefully new enough to meet the needs of the project?
@Aciid have agreed on the following plan to unblock the efforts here, so that Ben's llbuild work can avoid introducing unnecessary duplication.
Here is the concrete proposal for the next steps we plan to follow:
(In swift-package-manager) We will factor out the TSC libraries (Basic & Utility) into TSC prefixed module names. We will also keep the SPMUtility module for things we believe are SwiftPM specific.
(in swift-tools-support-core) We will make an initial commit of all of the SwiftPM TSC code. The TSC repository will now be the authoritative source for this code, even though it is duplicated in SwiftPM.
(in swift-package-manager) We will make a script to import the code from TSC, to be used in the short term until we have updated the build scripts to allow the decomposition.
At this point, llbuild's SwiftPM package can adopt TSC, even though its CMake build won't.
We will add Swift CI support for TSC. (cc @mishal_shah)
In the immediate future, llbuild won't adopt TSC in places that are used by SwiftPM, but at some point we will also need to switch llbuild to use CMake 3.15 features for building Swift, so that we can also adopt TSC in more places (like the Swift bindings that are used by SwiftPM).
I'm assuming TSC will be a branch dependency for swiftpm to match how it would build with cmake? In that case CI support needs to come before we can move swiftpm's package over, as sourcekit-lsp imports libSwiftpM via the package. CI will need to test TSC and everything downstream, including SwiftPM and SourceKit-LSP.
Thank you for resurrecting this! I've been meaning to raise the issue again because swift-format also dips into SwiftPM for the argument parsing utilities, so I'm glad that we have a path forward to clean that up soon.
@blangmuir is right. We can't do "TSC repository will now be the authoritative source for this code, even though it is duplicated in SwiftPM." without being able to test downstream dependencies. Maybe development should happen in the SwiftPM repository until a proper CI is in place and we sync back changes to TSC.
Yep, that sounds right. I'll prepare a PR for swift-tools-support-core with the initial commit to copy code from SwiftPM. I'll rename SPMUtility to Utility and SPMLibc to Libc - should I do the same in SwiftPM or is there a specific reason for those prefixes?
Thanks for bringing it up again @ddunbar - I do much rather use the parsing lib from SwiftPM instead of the hacked together version I built for this purpose.
These should use more specific prefixed names, like TSCUtility or ToolsSupportUtility. With very general names, you run the risk of colliding with other modules that someone might have elsewhere in their dependency graph and there’s no way to resolve those collisions.
Will Basic also be included? swift-format is using that as well.
True, then I'll add prefixes there. Currently I'm splitting the following modules into their SPM specific code and core code: SourceControl, Basic, Utility and TestSupport. The parts that are not directly related to SwiftPM will land in those targets.
I'll add the prefix TSC to the targets in SwiftPM to better highlight that they need to be copied to that repository for now and will be removed eventually.
