# Factoring out a new repo for shared code between llbuild & SwiftPM **URL:** \ **Category:** Package Manager\ **Tags:** llbuild\ **Created:** [March 27, 2018, 3:33pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429 "2018-03-27T15:33:18Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\ **Post date:** [March 27, 2018, 3:33pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/1 "2018-03-27T15:33:18Z")
We are starting to code more of the build system interaction with llbuild in Swift, which uses bindings in the llbuild repository. As part of that, we would like to use the infrastructure we have built up for SwiftPM. SwiftPM depends on llbuild, so this would introduce a cycle to the existing process. To solve this, we are creating a new repository "[GitHub - apple/swift-tools-support-core: Contains common infrastructural code for both SwiftPM and llbuild.](http://github.com/apple/swift-tools-support-core)" which will be a new Swift package which contains the infrastructural code that both SwiftPM and llbuild need. We are still working out the details around the build process and CI integration, and we plan to use the JIRA component "llbuild" to track work related to this repository. **NOTE** : This code is meant to support building Swift open source projects, and not intended as general purpose, Apple-supported APIs. ---
**Author:** ![tkremenek](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tkremenek/32/9115_2.png) [@tkremenek](https://forums.swift.org/u/tkremenek)\ **Post date:** [March 27, 2018, 5:06pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/2 "2018-03-27T17:06:22Z")
> [@ddunbar](#): > > To solve this, we are creating a new repository “[GitHub - apple/swift-tools-support-core: Contains common infrastructural code for both SwiftPM and llbuild.](http://github.com/apple/swift-tools-support-core) ” which will be a new Swift package which contains the infrastructural code that both SwiftPM and llbuild need The repository has been created. ---
**Author:** ![Yonas\_Kolb](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/yonas_kolb/32/10222_2.png) [@Yonas\_Kolb](https://forums.swift.org/u/Yonas_Kolb)\ **Post date:** [March 31, 2018, 4:39am UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/3 "2018-03-31T04:39:28Z")
Would this include the code within the `Utility` library of SPM, (things like the `ArgumentParser`) or would it be just llbuild integration? In any case, it would great to have the Utility code in a smaller repo than SPM with proper semver releases 👍 ---
**Author:** ![gmosx](https://avatars.discourse-cdn.com/v4/letter/g/fbc32d/32.png) [@gmosx](https://forums.swift.org/u/gmosx)\ **Post date:** [March 31, 2018, 8:49am UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/4 "2018-03-31T08:49:26Z")
Indeed, it would be a great idea to extract Utility (and Basic) as separate packages. ---
**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\ **Post date:** [April 1, 2018, 12:12am UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/5 "2018-04-01T00:12:48Z")
It's basically what is in the Utility product. And indeed, this would be a great chance for us to start using semver so we can also live on the same experience we are trying to encourage other packages to use! ---
**Author:** ![Aciid](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aciid/32/429_2.png) [@Aciid](https://forums.swift.org/u/Aciid)\ **Post date:** [April 9, 2018, 3:59pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/6 "2018-04-09T15:59:27Z")
#### Here is the initial implementation plan: ##### Bootstrapping and CI infrastructure: We want tools-support-core to be a Swift package but we also want to leverage Swift CI infrastructure to add CI support. We need a way to bootstrap this library on the CI without using SwiftPM as we need a cross platform solution to easily bootstrap for all platforms that are currently supported and the new platforms that might be added in the future. To solve this problem, we will add a new backend in SwiftPM to emit a Makefile. The Makefile support will be only accessible using an hidden flag and will be intended to be used only for Swift related projects. We will check-in the Makefile to tools-support-code and execute it on the CI using Swift's build-script to build and test the project. Most of the time, a contributor is expected to only require a recent trunk toolchain to build the repository using the SwiftPM that is present in the toolchain. ##### Makefile location and layout: TBD. ##### Module prefix: Currently, Swift requires module names to be unique across the package graph. To avoid conflict with other modules in the package graph, we will use the prefix "TSC" in all of the modules in this repository. ##### Git history: Git history is usually a very useful feature but re-writing it could be annoying. Given that we don't really have huge amount of code that we want to factor out, we will do a "best effort" of preserving the git history. ##### ToolsSupportCore product: We will create a "ToolsSupportCore" library product in the Swift package. This product will re-export all library targets that the package wants to make "public". ---
**Author:** ![eneko](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/eneko/32/12117_2.png) [@eneko](https://forums.swift.org/u/eneko)\ **Post date:** [August 23, 2018, 5:20pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/7 "2018-08-23T17:20:50Z")
I like the idea of moving core modules from SwiftPM (Basic, Utilities, etc) to this new ToolsSupportCore repository. It would make it easier to do proper semver releases that way. I'll be happy to help with this migration, if there is any way possible. Thank you, Eneko ---
**Author:** ![beefon](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/beefon/32/13391_2.png) [@beefon](https://forums.swift.org/u/beefon)\ **Post date:** [October 18, 2018, 3:28pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/8 "2018-10-18T15:28:27Z")
Any plans to actually commit the sources? 🙂 ---
**Author:** ![Aciid](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aciid/32/429_2.png) [@Aciid](https://forums.swift.org/u/Aciid)\ **Post date:** [October 18, 2018, 3:33pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/9 "2018-10-18T15:33:33Z")
This is something that we really want to do but are lacking time and bandwidth to make progress. ---
**Author:** ![Braden\_Scothern](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/braden_scothern/32/1580_2.png) [@Braden\_Scothern](https://forums.swift.org/u/Braden_Scothern)\ **Post date:** [October 19, 2018, 4:54am UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/10 "2018-10-19T04:54:23Z")
If you have a general idea of which components should get pulled out I'm happy to start working on this. I can preserve the git history as well, I've broken apart and combined different repos several times. ---
**Author:** ![Aciid](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aciid/32/429_2.png) [@Aciid](https://forums.swift.org/u/Aciid)\ **Post date:** [October 19, 2018, 3:35pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/11 "2018-10-19T15:35:06Z")
The major issue here is figuring out the bootstrapping process on the CI as we would need to build this new package before building SwiftPM itself. I am now less convinced of the Makefile approach described above and I think we should look at different options like CMake or using an older version of SwiftPM to bootstrap. ---
**Author:** ![Braden\_Scothern](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/braden_scothern/32/1580_2.png) [@Braden\_Scothern](https://forums.swift.org/u/Braden_Scothern)\ **Post date:** [October 19, 2018, 3:45pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/12 "2018-10-19T15:45:36Z")
CMake is powerful but last I checked (and just know with a quick Google) it didn't support Swift projects so it will add some pain there but it should be very doable. If we can just use SwiftPM it has the advantage of being just 1 technology people need to know work on TSC components. Would it be sufficient to use the most recent official release of SwiftPM for this so it is stable but hopefully new enough to meet the needs of the project? ---
**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\ **Post date:** [August 27, 2019, 6:07pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/13 "2019-08-27T18:07:34Z")
This effort got sidelined for a long while, but @benjamin_herzog's recent work in llbuild ([[WIP] Create a basic tooling infrastructure for build analysis tools. by BenchR267 · Pull Request #542 · apple/swift-llbuild · GitHub](https://github.com/apple/swift-llbuild/pull/542)) is causing it to be revived. @Aciid have agreed on the following plan to unblock the efforts here, so that Ben's llbuild work can avoid introducing unnecessary duplication. Here is the concrete proposal for the next steps we plan to follow: 1. (In swift-package-manager) We will factor out the TSC libraries (Basic & Utility) into TSC prefixed module names. We will also keep the SPMUtility module for things we believe are SwiftPM specific. 2. (in swift-tools-support-core) We will make an initial commit of all of the SwiftPM TSC code. The TSC repository will now be the _authoritative_ source for this code, even though it is duplicated in SwiftPM. 3. (in swift-package-manager) We will make a script to import the code from TSC, to be used in the short term until we have updated the build scripts to allow the decomposition. 4. At this point, llbuild's SwiftPM package can adopt TSC, even though its CMake build won't. 5. We will add Swift CI support for TSC. (cc @mishal_shah) 6. In line with @compnerd's plans, the Swift project will adopt CMake 3.15 or later (which has Swift build support). See [Bumping CMake minimum to 3.15](https://forums.swift.org/t/bumping-cmake-minimum-to-3-15/28179) 7. We will make TSC buildable with CMake. 8. We will make SwiftPM buildable with CMake. 9. We will delete SwiftPM's duplicate copy of the TSC code, and instead use the TSC CMake build (including integration into Swift's build-script). 10. Rejoicing in the streets. ---
**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\ **Post date:** [August 27, 2019, 6:22pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/14 "2019-08-27T18:22:09Z")
@NeoNacho pointed out a missing item. In the immediate future, llbuild won't adopt TSC in places that are used by SwiftPM, but at some point we will also need to switch llbuild to use CMake 3.15 features for building Swift, so that we can also adopt TSC in more places (like the Swift bindings that are used by SwiftPM). ---
**Author:** ![blangmuir](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/blangmuir/32/277_2.png) [@blangmuir](https://forums.swift.org/u/blangmuir)\ **Post date:** [August 27, 2019, 9:00pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/15 "2019-08-27T21:00:05Z")
> [@ddunbar](#): > > - At this point, llbuild's SwiftPM package can adopt TSC, even though its CMake build won't. > - We will add Swift CI support for TSC. (cc @mishal_shah) I'm assuming TSC will be a branch dependency for swiftpm to match how it would build with cmake? In that case CI support needs to come _before_ we can move swiftpm's package over, as sourcekit-lsp imports libSwiftpM via the package. CI will need to test TSC and everything downstream, including SwiftPM and SourceKit-LSP. ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [August 28, 2019, 6:31am UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/16 "2019-08-28T06:31:08Z")
Thank you for resurrecting this! I've been meaning to raise the issue again because swift-format also dips into SwiftPM for the argument parsing utilities, so I'm glad that we have a path forward to clean that up soon. ---
**Author:** ![Aciid](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aciid/32/429_2.png) [@Aciid](https://forums.swift.org/u/Aciid)\ **Post date:** [August 28, 2019, 3:32pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/17 "2019-08-28T15:32:55Z")
@blangmuir is right. We can't do "TSC repository will now be the _authoritative_ source for this code, even though it is duplicated in SwiftPM." without being able to test downstream dependencies. Maybe development should happen in the SwiftPM repository until a proper CI is in place and we sync back changes to TSC. ---
**Author:** ![benjamin\_herzog](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/benjamin_herzog/32/22799_2.png) [@benjamin\_herzog](https://forums.swift.org/u/benjamin_herzog)\ **Post date:** [August 29, 2019, 12:10pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/18 "2019-08-29T12:10:46Z")
Yep, that sounds right. I'll prepare a PR for swift-tools-support-core with the initial commit to copy code from SwiftPM. I'll rename SPMUtility to Utility and SPMLibc to Libc - should I do the same in SwiftPM or is there a specific reason for those prefixes? Thanks for bringing it up again @ddunbar - I do much rather use the parsing lib from SwiftPM instead of the hacked together version I built for this purpose. ---
**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\ **Post date:** [August 29, 2019, 1:04pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/19 "2019-08-29T13:04:10Z")
> [@benjamin\_herzog](#): > > I'll rename SPMUtility to Utility and SPMLibc to Libc These should use more specific prefixed names, like TSCUtility or ToolsSupportUtility. With very general names, you run the risk of colliding with other modules that someone might have elsewhere in their dependency graph and there’s no way to resolve those collisions. Will Basic also be included? swift-format is using that as well. ---
**Author:** ![benjamin\_herzog](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/benjamin_herzog/32/22799_2.png) [@benjamin\_herzog](https://forums.swift.org/u/benjamin_herzog)\ **Post date:** [August 29, 2019, 1:20pm UTC](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429/20 "2019-08-29T13:20:22Z")
True, then I'll add prefixes there. Currently I'm splitting the following modules into their SPM specific code and core code: SourceControl, Basic, Utility and TestSupport. The parts that are not directly related to SwiftPM will land in those targets. I'll add the prefix TSC to the targets in SwiftPM to better highlight that they need to be copied to that repository for now and will be removed eventually. [Next page](https://forums.swift.org/t/factoring-out-a-new-repo-for-shared-code-between-llbuild-swiftpm/11429.md?page=2)