LLVM monorepo transition happening on Thursday, the 17th of October

Hi,

I would like to provide an update to the monorepo transition posts (https://forums.swift.org/t/llvm-monorepo-transition/25689 and https://forums.swift.org/t/llvm-monorepo-transition-update/27079) where I announced that Apple is working on the LLVM project monorepo transition for Swift, and posted links to the prototype monorepo that can be used with Swift.

Duncan (@dexonsmith), Mishal (@mishal_shah) and I are working on executing the transition before the LLVM Dev Meeting that is happening in two weeks. The final llvm-project-v3 monorepo prototype, which is currently being finalized, will replace the existing https://github.com/apple/llvm-project-v2 prototype. We are aiming to publish the llvm-project-v3 prototype monorepo by Friday, 11th of October.

The transition itself will happen next week, between the 14th and the 18th of October. It will involve a couple of steps spread across a couple of days, however, we will have one day (still TBD) when the main transition will happen. The transition day will look something like this:

We are not expecting any downtime for Swift PR testing. We will ask the individual contributors to rerun the update checkout script again after the transition occurs to ensure that their Swift checkout updates itself to use the monorepo.

I will update this thread when we will publish the finalized llvm-project-v3 prototype monorepo, and when we will determine what the actual transition day is. I’ll keep updating the forums throughout the transition day as well to keep everyone up-to-date on what the status is.

7 Likes

I would like to provide an additional update to this thread. Today, the following two new repositories have been created on github.com/apple:

The new llvm-project-v3 repository replaces the previous WIP downstream monorepo (https://github.com/apple/llvm-project-v2). The convention that's used for branches in the new repository is the same as in llvm-project-v2. If you've been using llvm-project-v2 for your work, please migrate to v3 before the end of next week. At the end of next week we'll stop updating llvm-project-v2, and will delete it the following week.

This is super exciting :) Thanks for all the work and the updates!

A couple of questions:

  • The apple-llvm-mt user has a space in its email address (mt @ apple-llvm). Is this intentional?
  • Will llvm-project-v3 be renamed to llvm-project, or will llvm-project be regenerated without the merges from the split repositories?
  • Once we stop merging from the split repositories, we'll just be merging from llvm.org wholesale, correct? E.g. when we merge from apple/master into swift/master-next, it'll be merging changes across the entire monorepo, and not e.g. dropping any LLDB changes and having those be integrated separately.
  • I don't think the mt @ apple-llvm addresses were intentional, but I don't think we'll be able to change them now. Are they causing problems on your end?

  • It won't be regenerated, but not quite renamed either. An empty github.com/apple/llvm-project repository will be created by cloning github.com/llvm/llvm-project. Then the github.com/apple/llvm-project-v3 branches will be pushed to github.com/apple/llvm-project, so it's kind of a rename. Right now we are 99% sure that the commit hashes in llvm-project-v3 are the ones that will be used in the final llvm-project.

  • Correct. The merges will bring in the entire monorepo contents. The ToT auto merger edges will look like this:

llvm.org/master -> apple/master -> swift/master-next

This week will provide a tool which will show a status of the automerger to help users see the topology and the backlog. In the future we are planning to create PRs for merge conflicts on github.com/apple/llvm-project, so that they can be resolved even by contributors outside of Apple. Unfortunately I don't have a time frame when that will be enabled yet.

  • There's a few scripts that were assuming email addresses would be free of spaces, but it's easy enough to fix those up, and changing them to be resilient is a good thing regardless. (Things breaking on spaces is one of my least favorite aspects of shell scripts.)
  • Cool, sounds good.
  • Creating PRs for merge conflicts and allowing outside contributors to resolve them would be awesome! Looking forward to more updates on that whenever it's ready :)

I hate to say it, but the spaces thing definitely has the potential to be one of those things that breaks random tooling for years, so if regeneration is at all in the cards, it would be really nice.

7 Likes

Due to a bug in the generator we had to re-generate v3 to produce llvm-project-v4. The issue with mt @ apple-llvm should now be fixed in v4. I'll update this thread when llvm-project-v4 is public.

4 Likes

That sounds great, I'm glad we had an opportunity to fix this.

I would like to provide two additional updates to this thread:

  1. We decided to perform the transition this Thursday, the 17th of October. We expect that there will be a couple of hours where it won't be possible to commit downstream changes to swift- llvm, clang, lldb, compiler-rt, and clang-tools-extra, until github.com/apple/llvm-project is ready to receive commits. We are not expecting any downtime for Swift PR testing.

  2. The following two new repositories have been created on github.com/apple:

The new llvm-project-v4 repository replaces the previous WIP downstream monorepo (https://github.com/apple/llvm-project-v3 ). The convention that's used for branches in the new repository is the same as in llvm-project-v3. If you've been using llvm-project-v3 for your work, please migrate to v4 before the end of next week. At the end of next week we'll stop updating llvm-project-v3, and will delete it the following week.

The transition process is starting now. @mishal_shah has disabled commit access to the legacy split repositories (The GitHub swift-specific split llvm-project repositories are now read-only).

We are in the process of finalizing github.com/apple/llvm-project. We have published the v5 monorepo, which contains the final commit hashes that will be identical to the ones used for the final github.com/apple/llvm-project monorepo:

1 Like

I would like to provide an update to the monorepo transition work that happened today:

We have decided to delay giving commit access to https://github.com/apple/llvm-project, as @mishal_shah and I are still verifying that PR testing infrastructure and CI is working as expected with the monorepo. The commit access will be restored tomorrow after the final CI and PR testing checks are performed.

6 Likes

I would like to provide an update to the monorepo transition work that happened today:

  • The swift/master branch in https://github.com/apple/llvm-project is now open for commits. Changes to that branch require a passing pull request before they can be merged. The swift/master branch allows LLDB changes only , and does not accept changes in other split directories in llvm-project. This branch is downstream of apple/stable/20190619 , and all the llvm/clang/compiler-rt/lld changes that are needed for swift/master should be merged into apple/stable/20190619 . As of today you are still not allowed to merge PRs into apple/stable/20190619 (the only exception are PRs that unblock CI or PR testing issues). We’re aiming to allow commit access to that branch by Monday.

  • The apple/master and the swift/master-next branches are not yet open for merges (the only exception are PRs that unblock CI or PR testing issues). We are aiming to allow commit access to these branches by Monday as well.

  • swift-ci now uses the new https://github.com/apple/llvm-project monorepo for its pull request testing!

2 Likes

Congrats on all of the hard work!

One question: what are the refs/am/changes/ and refs/am/merge-candidate/ namespaces in the GitHub repository? I've never seen these before and I've had no luck finding docs online about these namespaces. (Maybe they're new GitHub namespaces? Like refs/pull/?)

Is it intentional? If so, then what is it? Or is it an accident of force pushing a clone of a local repository? (This can easily happen.)

The refs/am namespace is used by our automerger, which propagates changes from llvm.org. I haven't had the time to document it properly yet, but we're aiming to make it open and usable by the community, so that merge conflicts can be resolved even outside of Apple.

We have restored commit access to the other branches in https://github.com/apple/llvm-project.

Please see this post for more details: The GitHub swift-specific split llvm-project repositories are now read-only.

@Alex_L following up on this:

This week will provide a tool which will show a status of the automerger to help users see the topology and the backlog.

Was this tool released? Apologizes if I just missed it.

Terms of Service

Privacy Policy

Cookie Policy