LLVM monorepo transition happening on Thursday, the 17th of October

Hi,

I would like to provide an update to the monorepo transition posts (https://forums.swift.org/t/llvm-monorepo-transition/25689 and https://forums.swift.org/t/llvm-monorepo-transition-update/27079) where I announced that Apple is working on the LLVM project monorepo transition for Swift, and posted links to the prototype monorepo that can be used with Swift.

Duncan (@dexonsmith), Mishal (@mishal_shah) and I are working on executing the transition before the LLVM Dev Meeting that is happening in two weeks. The final llvm-project-v3 monorepo prototype, which is currently being finalized, will replace the existing https://github.com/apple/llvm-project-v2 prototype. We are aiming to publish the llvm-project-v3 prototype monorepo by Friday, 11th of October.

The transition itself will happen next week, between the 14th and the 18th of October. It will involve a couple of steps spread across a couple of days, however, we will have one day (still TBD) when the main transition will happen. The transition day will look something like this:

We are not expecting any downtime for Swift PR testing. We will ask the individual contributors to rerun the update checkout script again after the transition occurs to ensure that their Swift checkout updates itself to use the monorepo.

I will update this thread when we will publish the finalized llvm-project-v3 prototype monorepo, and when we will determine what the actual transition day is. I’ll keep updating the forums throughout the transition day as well to keep everyone up-to-date on what the status is.

7 Likes

I would like to provide an additional update to this thread. Today, the following two new repositories have been created on github.com/apple:

The new llvm-project-v3 repository replaces the previous WIP downstream monorepo (https://github.com/apple/llvm-project-v2). The convention that's used for branches in the new repository is the same as in llvm-project-v2. If you've been using llvm-project-v2 for your work, please migrate to v3 before the end of next week. At the end of next week we'll stop updating llvm-project-v2, and will delete it the following week.

This is super exciting :) Thanks for all the work and the updates!

A couple of questions:

  • The apple-llvm-mt user has a space in its email address (mt @ apple-llvm). Is this intentional?
  • Will llvm-project-v3 be renamed to llvm-project, or will llvm-project be regenerated without the merges from the split repositories?
  • Once we stop merging from the split repositories, we'll just be merging from llvm.org wholesale, correct? E.g. when we merge from apple/master into swift/master-next, it'll be merging changes across the entire monorepo, and not e.g. dropping any LLDB changes and having those be integrated separately.

llvm.org/master -> apple/master -> swift/master-next

This week will provide a tool which will show a status of the automerger to help users see the topology and the backlog. In the future we are planning to create PRs for merge conflicts on github.com/apple/llvm-project, so that they can be resolved even by contributors outside of Apple. Unfortunately I don't have a time frame when that will be enabled yet.

  • There's a few scripts that were assuming email addresses would be free of spaces, but it's easy enough to fix those up, and changing them to be resilient is a good thing regardless. (Things breaking on spaces is one of my least favorite aspects of shell scripts.)
  • Cool, sounds good.
  • Creating PRs for merge conflicts and allowing outside contributors to resolve them would be awesome! Looking forward to more updates on that whenever it's ready :)

I hate to say it, but the spaces thing definitely has the potential to be one of those things that breaks random tooling for years, so if regeneration is at all in the cards, it would be really nice.

7 Likes

Due to a bug in the generator we had to re-generate v3 to produce llvm-project-v4. The issue with mt @ apple-llvm should now be fixed in v4. I'll update this thread when llvm-project-v4 is public.

4 Likes

That sounds great, I'm glad we had an opportunity to fix this.

I would like to provide two additional updates to this thread:

  1. We decided to perform the transition this Thursday, the 17th of October. We expect that there will be a couple of hours where it won't be possible to commit downstream changes to swift- llvm, clang, lldb, compiler-rt, and clang-tools-extra, until GitHub - apple/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. This fork is used to manage Apple’s stable releases of Clang as well as support the Swift project. is ready to receive commits. We are not expecting any downtime for Swift PR testing.

  2. The following two new repositories have been created on github.com/apple:

The new llvm-project-v4 repository replaces the previous WIP downstream monorepo (https://github.com/apple/llvm-project-v3 ). The convention that's used for branches in the new repository is the same as in llvm-project-v3. If you've been using llvm-project-v3 for your work, please migrate to v4 before the end of next week. At the end of next week we'll stop updating llvm-project-v3, and will delete it the following week.

The transition process is starting now. @mishal_shah has disabled commit access to the legacy split repositories (The GitHub swift-specific split llvm-project repositories are now read-only).

We are in the process of finalizing GitHub - apple/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. This fork is used to manage Apple’s stable releases of Clang as well as support the Swift project.. We have published the v5 monorepo, which contains the final commit hashes that will be identical to the ones used for the final GitHub - apple/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. This fork is used to manage Apple’s stable releases of Clang as well as support the Swift project. monorepo:

1 Like

I would like to provide an update to the monorepo transition work that happened today:

We have decided to delay giving commit access to GitHub - apple/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. This fork is used to manage Apple’s stable releases of Clang as well as support the Swift project., as @mishal_shah and I are still verifying that PR testing infrastructure and CI is working as expected with the monorepo. The commit access will be restored tomorrow after the final CI and PR testing checks are performed.

6 Likes

I would like to provide an update to the monorepo transition work that happened today:

2 Likes

Congrats on all of the hard work!

One question: what are the refs/am/changes/ and refs/am/merge-candidate/ namespaces in the GitHub repository? I've never seen these before and I've had no luck finding docs online about these namespaces. (Maybe they're new GitHub namespaces? Like refs/pull/?)

Is it intentional? If so, then what is it? Or is it an accident of force pushing a clone of a local repository? (This can easily happen.)

The refs/am namespace is used by our automerger, which propagates changes from llvm.org. I haven't had the time to document it properly yet, but we're aiming to make it open and usable by the community, so that merge conflicts can be resolved even outside of Apple.

We have restored commit access to the other branches in GitHub - apple/llvm-project: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. This fork is used to manage Apple’s stable releases of Clang as well as support the Swift project..

Please see this post for more details: The GitHub swift-specific split llvm-project repositories are now read-only - #4 by Alex_L.

@Alex_L following up on this:

This week will provide a tool which will show a status of the automerger to help users see the topology and the backlog.

Was this tool released? Apologizes if I just missed it.