proposed change for master-next merges


(Bob Wilson) #1

I would like to make a change in the way we handle the master-next branch.

Summary: I’d like to switch to a model where we continuously test against the latest upstream LLVM changes. The goal is to simplify the process and make it easier to collaborate on maintaining master-next.

Background: We develop Swift against “stable” branches of LLVM (which I am using here to refer to the llvm, clang, and compiler-rt repositories) that are typically rebranched from trunk once for each release, with other commits individually cherry-picked for specific bug fixes and other changes. This insulates Swift development from the churn of changes in LLVM. At the same time, we maintain the “master-next” branches of Swift repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from the “upstream-with-swift” branches in our GitHub LLVM repos. We have existing automation to continuously merge changes from llvm.org <http://llvm.org/> into those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the Swift team is designated as the "merge czar" and is responsible for this. This merge typically happens once every few weeks. Michael Gottesman developed some internal tools to help automate the process, but someone still needs to drive those tools manually. The process involves merging “master” to “master-next” for all the Swift repos and updating the “stable-next” branches of the GitHub LLVM repos for Swift. The “stable-next” branches are basically snapshots of the LLVM upstream-with-swift branches at the point where master-next was most recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with the stable-next branches of LLVM (https://ci.swift.org/view/swift-master-next). The merge czar can use these bots to confirm that everything is working after a merge.

Reasons to change: The current process has the advantage that the merge czar can choose when to do a merge and can schedule that around other work, but it has some significant problems.

- It is difficult for multiple people to collaborate on updating master-next. The changes involved are often rev-locked between Swift and the LLVM repos, so there is no good way for someone to fix a problem without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge czar for the last few months, and it took me a while to figure out how to do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos, further adding to the complexity.

- The tools we have to help automate the process are currently internal to Apple and require ongoing maintenance. They could be cleaned up to release publicly but that would take more work.

Proposal: We already have Jenkins bots testing master-next. I would like to add a job to continuously merge master to master-next and change the existing bots to build against the “upstream-with-swift” branches in our GitHub LLVM repos. The bots would then detect any new problems soon after they are introduced. Anyone could fix those problems, whether they are merge conflicts, build failures, or test issues. A partial fix could be applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We already have automatic merging bots, so adding another one would not be difficult.

The biggest advantage is that it provides a straightforward model that anyone can understand: master-next becomes just another branch that anyone can modify, build and test in the usual way. Collaboration is no more difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to let the master-next branch break occasionally. An LLVM change might break things in a way that takes some time to fix, and the master-next bots would continue to fail during that time. Someone might want to apply a partial fix that does not resolve all the issues, and we would want to allow that even if the bots still fail. That would mean we would have to relax (or override) the requirement for PR testing for commits in that kind of situation. In the worst case, if new problems are introduced more quickly than we can fix them, this approach could fall apart. My experience as merge czar over the last few months suggests that is unlikely. Usually there are no more than a few problems per week and most of them are easy to fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot that tests master-next building against upstream-with-swift. If that goes well, and if there are no objections to this proposal, we can add the automerger to merge master into master-next. At the same time, we would update the other master-next bots to use upstream-with-swift instead of stable-next for the LLVM repos.

Alternative: We can achieve some of the same goals at a considerable increase in complexity by introducing an automatic gated merge solution. We would have automation perform the merge and commit it as long as everything works. If there were any problems, the automation would create a pull request that would need to be manually updated to resolve the problems. People could still collaborate by working together on the pull request branch. Until the problems were resolved, no further merging would take place. I would like to try the simple approach before considering this more complex solution, since I don’t think it will be necessary, at least in the near future.

Any objections to this? Comments or suggestions?


(Ted Kremenek) #2

This approach seems good to me. It strikes a good pragmatic balance.

···

On Dec 7, 2016, at 7:30 PM, Bob Wilson via swift-dev <swift-dev@swift.org> wrote:

I would like to make a change in the way we handle the master-next branch.

Summary: I’d like to switch to a model where we continuously test against the latest upstream LLVM changes. The goal is to simplify the process and make it easier to collaborate on maintaining master-next.

Background: We develop Swift against “stable” branches of LLVM (which I am using here to refer to the llvm, clang, and compiler-rt repositories) that are typically rebranched from trunk once for each release, with other commits individually cherry-picked for specific bug fixes and other changes. This insulates Swift development from the churn of changes in LLVM. At the same time, we maintain the “master-next” branches of Swift repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from the “upstream-with-swift” branches in our GitHub LLVM repos. We have existing automation to continuously merge changes from llvm.org <http://llvm.org/> into those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the Swift team is designated as the "merge czar" and is responsible for this. This merge typically happens once every few weeks. Michael Gottesman developed some internal tools to help automate the process, but someone still needs to drive those tools manually. The process involves merging “master” to “master-next” for all the Swift repos and updating the “stable-next” branches of the GitHub LLVM repos for Swift. The “stable-next” branches are basically snapshots of the LLVM upstream-with-swift branches at the point where master-next was most recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with the stable-next branches of LLVM (https://ci.swift.org/view/swift-master-next). The merge czar can use these bots to confirm that everything is working after a merge.

Reasons to change: The current process has the advantage that the merge czar can choose when to do a merge and can schedule that around other work, but it has some significant problems.

- It is difficult for multiple people to collaborate on updating master-next. The changes involved are often rev-locked between Swift and the LLVM repos, so there is no good way for someone to fix a problem without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge czar for the last few months, and it took me a while to figure out how to do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos, further adding to the complexity.

- The tools we have to help automate the process are currently internal to Apple and require ongoing maintenance. They could be cleaned up to release publicly but that would take more work.

Proposal: We already have Jenkins bots testing master-next. I would like to add a job to continuously merge master to master-next and change the existing bots to build against the “upstream-with-swift” branches in our GitHub LLVM repos. The bots would then detect any new problems soon after they are introduced. Anyone could fix those problems, whether they are merge conflicts, build failures, or test issues. A partial fix could be applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We already have automatic merging bots, so adding another one would not be difficult.

The biggest advantage is that it provides a straightforward model that anyone can understand: master-next becomes just another branch that anyone can modify, build and test in the usual way. Collaboration is no more difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to let the master-next branch break occasionally. An LLVM change might break things in a way that takes some time to fix, and the master-next bots would continue to fail during that time. Someone might want to apply a partial fix that does not resolve all the issues, and we would want to allow that even if the bots still fail. That would mean we would have to relax (or override) the requirement for PR testing for commits in that kind of situation. In the worst case, if new problems are introduced more quickly than we can fix them, this approach could fall apart. My experience as merge czar over the last few months suggests that is unlikely. Usually there are no more than a few problems per week and most of them are easy to fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot that tests master-next building against upstream-with-swift. If that goes well, and if there are no objections to this proposal, we can add the automerger to merge master into master-next. At the same time, we would update the other master-next bots to use upstream-with-swift instead of stable-next for the LLVM repos.

Alternative: We can achieve some of the same goals at a considerable increase in complexity by introducing an automatic gated merge solution. We would have automation perform the merge and commit it as long as everything works. If there were any problems, the automation would create a pull request that would need to be manually updated to resolve the problems. People could still collaborate by working together on the pull request branch. Until the problems were resolved, no further merging would take place. I would like to try the simple approach before considering this more complex solution, since I don’t think it will be necessary, at least in the near future.

Any objections to this? Comments or suggestions?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Saleem Abdulrasool) #3

Having been involved in the update process for the next branches, I'm
really excited to see this type of change.

I think that the "simple" approach is both better to work and collaborate
in as well as closer to the llvm development model which makes it easier to
cross pollinate.

The one thing that I think could be more strongly called out is that normal
PR CI shouldn't be gated on master next passing.

Beyond that, I think that this proposal should ease collaboration and
maintenance pains in the current process.

···

On Wed, Dec 7, 2016 at 7:30 PM Bob Wilson via swift-dev <swift-dev@swift.org> wrote:

I would like to make a change in the way we handle the master-next branch.

*Summary:* I’d like to switch to a model where we continuously test
against the latest upstream LLVM changes. The goal is to simplify the
process and make it easier to collaborate on maintaining master-next.

*Background:* We develop Swift against “stable” branches of LLVM (which I
am using here to refer to the llvm, clang, and compiler-rt repositories)
that are typically rebranched from trunk once for each release, with other
commits individually cherry-picked for specific bug fixes and other
changes. This insulates Swift development from the churn of changes in
LLVM. At the same time, we maintain the “master-next” branches of Swift
repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from
the “upstream-with-swift” branches in our GitHub LLVM repos. We have
existing automation to continuously merge changes from llvm.org into
those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the
Swift team is designated as the "merge czar" and is responsible for this.
This merge typically happens once every few weeks. Michael Gottesman
developed some internal tools to help automate the process, but someone
still needs to drive those tools manually. The process involves merging
“master” to “master-next” for all the Swift repos and updating the
“stable-next” branches of the GitHub LLVM repos for Swift. The
“stable-next” branches are basically snapshots of the LLVM
upstream-with-swift branches at the point where master-next was most
recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with
the stable-next branches of LLVM (
https://ci.swift.org/view/swift-master-next). The merge czar can use
these bots to confirm that everything is working after a merge.

*Reasons to change: *The current process has the advantage that the merge
czar can choose when to do a merge and can schedule that around other work,
but it has some significant problems.

- It is difficult for multiple people to collaborate on updating
master-next. The changes involved are often rev-locked between Swift and
the LLVM repos, so there is no good way for someone to fix a problem
without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge
czar for the last few months, and it took me a while to figure out how to
do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos,
further adding to the complexity.

- The tools we have to help automate the process are currently internal to
Apple and require ongoing maintenance. They could be cleaned up to release
publicly but that would take more work.

*Proposal: *We already have Jenkins bots testing master-next. I would
like to add a job to continuously merge master to master-next and change
the existing bots to build against the “upstream-with-swift” branches in
our GitHub LLVM repos. The bots would then detect any new problems soon
after they are introduced. Anyone could fix those problems, whether they
are merge conflicts, build failures, or test issues. A partial fix could be
applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We
already have automatic merging bots, so adding another one would not be
difficult.

The biggest advantage is that it provides a straightforward model that
anyone can understand: master-next becomes just another branch that anyone
can modify, build and test in the usual way. Collaboration is no more
difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to
let the master-next branch break occasionally. An LLVM change might break
things in a way that takes some time to fix, and the master-next bots would
continue to fail during that time. Someone might want to apply a partial
fix that does not resolve all the issues, and we would want to allow that
even if the bots still fail. That would mean we would have to relax (or
override) the requirement for PR testing for commits in that kind of
situation. In the worst case, if new problems are introduced more quickly
than we can fix them, this approach could fall apart. My experience as
merge czar over the last few months suggests that is unlikely. Usually
there are no more than a few problems per week and most of them are easy to
fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot
that tests master-next building against upstream-with-swift. If that goes
well, and if there are no objections to this proposal, we can add the
automerger to merge master into master-next. At the same time, we would
update the other master-next bots to use upstream-with-swift instead of
stable-next for the LLVM repos.

*Alternative: *We can achieve some of the same goals at a considerable
increase in complexity by introducing an automatic gated merge solution. We
would have automation perform the merge and commit it as long as everything
works. If there were any problems, the automation would create a pull
request that would need to be manually updated to resolve the problems.
People could still collaborate by working together on the pull request
branch. Until the problems were resolved, no further merging would take
place. I would like to try the simple approach before considering this more
complex solution, since I don’t think it will be necessary, at least in the
near future.

Any objections to this? Comments or suggestions?
_______________________________________________

swift-dev mailing list

swift-dev@swift.org

https://lists.swift.org/mailman/listinfo/swift-dev


(Michael Gottesman) #4

Having been involved in the update process for the next branches, I'm really excited to see this type of change.

I think that the "simple" approach is both better to work and collaborate in as well as closer to the llvm development model which makes it easier to cross pollinate.

The one thing that I think could be more strongly called out is that normal PR CI shouldn't be gated on master next passing.

Beyond that, I think that this proposal should ease collaboration and maintenance pains in the current process.

I agree with everything said here. The collaboration issue is the high order bit that needs to be fixed. Any other issues that come up can be fixed incrementally if necessary based on experience (for instance using the "back up" alternative). This is even more true since it is clear that that infrastructure would take time to develop and an incremental solution now that improves the collaboration will not make it more difficult to develop such a solution.

My only strong feeling here is that I think we need documentation of this process in ./docs in addition to this proposal.

Michael

···

On Dec 8, 2016, at 7:43 AM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org> wrote:

On Wed, Dec 7, 2016 at 7:30 PM Bob Wilson via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
I would like to make a change in the way we handle the master-next branch.

Summary: I’d like to switch to a model where we continuously test against the latest upstream LLVM changes. The goal is to simplify the process and make it easier to collaborate on maintaining master-next.

Background: We develop Swift against “stable” branches of LLVM (which I am using here to refer to the llvm, clang, and compiler-rt repositories) that are typically rebranched from trunk once for each release, with other commits individually cherry-picked for specific bug fixes and other changes. This insulates Swift development from the churn of changes in LLVM. At the same time, we maintain the “master-next” branches of Swift repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from the “upstream-with-swift” branches in our GitHub LLVM repos. We have existing automation to continuously merge changes from llvm.org <http://llvm.org/> into those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the Swift team is designated as the "merge czar" and is responsible for this. This merge typically happens once every few weeks. Michael Gottesman developed some internal tools to help automate the process, but someone still needs to drive those tools manually. The process involves merging “master” to “master-next” for all the Swift repos and updating the “stable-next” branches of the GitHub LLVM repos for Swift. The “stable-next” branches are basically snapshots of the LLVM upstream-with-swift branches at the point where master-next was most recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with the stable-next branches of LLVM (https://ci.swift.org/view/swift-master-next). The merge czar can use these bots to confirm that everything is working after a merge.

Reasons to change: The current process has the advantage that the merge czar can choose when to do a merge and can schedule that around other work, but it has some significant problems.

- It is difficult for multiple people to collaborate on updating master-next. The changes involved are often rev-locked between Swift and the LLVM repos, so there is no good way for someone to fix a problem without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge czar for the last few months, and it took me a while to figure out how to do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos, further adding to the complexity.

- The tools we have to help automate the process are currently internal to Apple and require ongoing maintenance. They could be cleaned up to release publicly but that would take more work.

Proposal: We already have Jenkins bots testing master-next. I would like to add a job to continuously merge master to master-next and change the existing bots to build against the “upstream-with-swift” branches in our GitHub LLVM repos. The bots would then detect any new problems soon after they are introduced. Anyone could fix those problems, whether they are merge conflicts, build failures, or test issues. A partial fix could be applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We already have automatic merging bots, so adding another one would not be difficult.

The biggest advantage is that it provides a straightforward model that anyone can understand: master-next becomes just another branch that anyone can modify, build and test in the usual way. Collaboration is no more difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to let the master-next branch break occasionally. An LLVM change might break things in a way that takes some time to fix, and the master-next bots would continue to fail during that time. Someone might want to apply a partial fix that does not resolve all the issues, and we would want to allow that even if the bots still fail. That would mean we would have to relax (or override) the requirement for PR testing for commits in that kind of situation. In the worst case, if new problems are introduced more quickly than we can fix them, this approach could fall apart. My experience as merge czar over the last few months suggests that is unlikely. Usually there are no more than a few problems per week and most of them are easy to fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot that tests master-next building against upstream-with-swift. If that goes well, and if there are no objections to this proposal, we can add the automerger to merge master into master-next. At the same time, we would update the other master-next bots to use upstream-with-swift instead of stable-next for the LLVM repos.

Alternative: We can achieve some of the same goals at a considerable increase in complexity by introducing an automatic gated merge solution. We would have automation perform the merge and commit it as long as everything works. If there were any problems, the automation would create a pull request that would need to be manually updated to resolve the problems. People could still collaborate by working together on the pull request branch. Until the problems were resolved, no further merging would take place. I would like to try the simple approach before considering this more complex solution, since I don’t think it will be necessary, at least in the near future.

Any objections to this? Comments or suggestions?
_______________________________________________

swift-dev mailing list

swift-dev@swift.org <mailto:swift-dev@swift.org>

https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Bob Wilson) #5

Since the feedback on this was all positive, we’re moving ahead with this change. It seems like a good time for a status update.

Some changes are already done:

* We test up a test bot to build against the upstream-with-swift branches of LLVM/Clang (https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-osx-master-next-upstream-with-swift/). This has been working well.

* Mishal set up an automerger to continuously update the Swift master-next branch with changes from master.

* I audited the master-next branches in the other repos (e.g., swiftpm, etc.) and found that none of them have any changes from master. For now, we will simply stop using those branches, and once we’re done with this transition and things are settled, we can remove them.

* We’ve had some discussions about how LLDB will fit into this plan. See the recent thread on “Changes to LLDB Branch Management” for details on that.

Here is what is left to do:

* Update all of our master-next bots to build against the “new” branches (e.g., upstream-with-swift instead of stable-next).

* Update documentation.

* Remove unused branches.

···

On Dec 8, 2016, at 10:54 AM, Michael Gottesman <mgottesman@apple.com> wrote:

On Dec 8, 2016, at 7:43 AM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Having been involved in the update process for the next branches, I'm really excited to see this type of change.

I think that the "simple" approach is both better to work and collaborate in as well as closer to the llvm development model which makes it easier to cross pollinate.

The one thing that I think could be more strongly called out is that normal PR CI shouldn't be gated on master next passing.

Beyond that, I think that this proposal should ease collaboration and maintenance pains in the current process.

I agree with everything said here. The collaboration issue is the high order bit that needs to be fixed. Any other issues that come up can be fixed incrementally if necessary based on experience (for instance using the "back up" alternative). This is even more true since it is clear that that infrastructure would take time to develop and an incremental solution now that improves the collaboration will not make it more difficult to develop such a solution.

My only strong feeling here is that I think we need documentation of this process in ./docs in addition to this proposal.

Michael

On Wed, Dec 7, 2016 at 7:30 PM Bob Wilson via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
I would like to make a change in the way we handle the master-next branch.

Summary: I’d like to switch to a model where we continuously test against the latest upstream LLVM changes. The goal is to simplify the process and make it easier to collaborate on maintaining master-next.

Background: We develop Swift against “stable” branches of LLVM (which I am using here to refer to the llvm, clang, and compiler-rt repositories) that are typically rebranched from trunk once for each release, with other commits individually cherry-picked for specific bug fixes and other changes. This insulates Swift development from the churn of changes in LLVM. At the same time, we maintain the “master-next” branches of Swift repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from the “upstream-with-swift” branches in our GitHub LLVM repos. We have existing automation to continuously merge changes from llvm.org <http://llvm.org/> into those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the Swift team is designated as the "merge czar" and is responsible for this. This merge typically happens once every few weeks. Michael Gottesman developed some internal tools to help automate the process, but someone still needs to drive those tools manually. The process involves merging “master” to “master-next” for all the Swift repos and updating the “stable-next” branches of the GitHub LLVM repos for Swift. The “stable-next” branches are basically snapshots of the LLVM upstream-with-swift branches at the point where master-next was most recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with the stable-next branches of LLVM (https://ci.swift.org/view/swift-master-next). The merge czar can use these bots to confirm that everything is working after a merge.

Reasons to change: The current process has the advantage that the merge czar can choose when to do a merge and can schedule that around other work, but it has some significant problems.

- It is difficult for multiple people to collaborate on updating master-next. The changes involved are often rev-locked between Swift and the LLVM repos, so there is no good way for someone to fix a problem without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge czar for the last few months, and it took me a while to figure out how to do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos, further adding to the complexity.

- The tools we have to help automate the process are currently internal to Apple and require ongoing maintenance. They could be cleaned up to release publicly but that would take more work.

Proposal: We already have Jenkins bots testing master-next. I would like to add a job to continuously merge master to master-next and change the existing bots to build against the “upstream-with-swift” branches in our GitHub LLVM repos. The bots would then detect any new problems soon after they are introduced. Anyone could fix those problems, whether they are merge conflicts, build failures, or test issues. A partial fix could be applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We already have automatic merging bots, so adding another one would not be difficult.

The biggest advantage is that it provides a straightforward model that anyone can understand: master-next becomes just another branch that anyone can modify, build and test in the usual way. Collaboration is no more difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to let the master-next branch break occasionally. An LLVM change might break things in a way that takes some time to fix, and the master-next bots would continue to fail during that time. Someone might want to apply a partial fix that does not resolve all the issues, and we would want to allow that even if the bots still fail. That would mean we would have to relax (or override) the requirement for PR testing for commits in that kind of situation. In the worst case, if new problems are introduced more quickly than we can fix them, this approach could fall apart. My experience as merge czar over the last few months suggests that is unlikely. Usually there are no more than a few problems per week and most of them are easy to fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot that tests master-next building against upstream-with-swift. If that goes well, and if there are no objections to this proposal, we can add the automerger to merge master into master-next. At the same time, we would update the other master-next bots to use upstream-with-swift instead of stable-next for the LLVM repos.

Alternative: We can achieve some of the same goals at a considerable increase in complexity by introducing an automatic gated merge solution. We would have automation perform the merge and commit it as long as everything works. If there were any problems, the automation would create a pull request that would need to be manually updated to resolve the problems. People could still collaborate by working together on the pull request branch. Until the problems were resolved, no further merging would take place. I would like to try the simple approach before considering this more complex solution, since I don’t think it will be necessary, at least in the near future.

Any objections to this? Comments or suggestions?
_______________________________________________

swift-dev mailing list

swift-dev@swift.org <mailto:swift-dev@swift.org>

https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


(Bob Wilson) #6

Since the feedback on this was all positive, we’re moving ahead with this change. It seems like a good time for a status update.

Some changes are already done:

* We test up a test bot to build against the upstream-with-swift branches of LLVM/Clang (https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-osx-master-next-upstream-with-swift/). This has been working well.

I think we can remove this test bot now, so don’t be surprised if it goes away soon.

* Mishal set up an automerger to continuously update the Swift master-next branch with changes from master.

* I audited the master-next branches in the other repos (e.g., swiftpm, etc.) and found that none of them have any changes from master. For now, we will simply stop using those branches, and once we’re done with this transition and things are settled, we can remove them.

* We’ve had some discussions about how LLDB will fit into this plan. See the recent thread on “Changes to LLDB Branch Management” for details on that.

Here is what is left to do:

* Update all of our master-next bots to build against the “new” branches (e.g., upstream-with-swift instead of stable-next).

New update: Mishal switched all the master-next bots to build against Clang/LLVM upstream-with-swift, so this step is done now.

The bad news is that things are in kind of bad shape. Beginning on Wednesday, the build stopped working — it is getting killed by timeouts, so maybe something is hanging.

···

On Feb 9, 2017, at 5:49 PM, Bob Wilson via swift-dev <swift-dev@swift.org> wrote:

* Update documentation.

* Remove unused branches.

On Dec 8, 2016, at 10:54 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

On Dec 8, 2016, at 7:43 AM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Having been involved in the update process for the next branches, I'm really excited to see this type of change.

I think that the "simple" approach is both better to work and collaborate in as well as closer to the llvm development model which makes it easier to cross pollinate.

The one thing that I think could be more strongly called out is that normal PR CI shouldn't be gated on master next passing.

Beyond that, I think that this proposal should ease collaboration and maintenance pains in the current process.

I agree with everything said here. The collaboration issue is the high order bit that needs to be fixed. Any other issues that come up can be fixed incrementally if necessary based on experience (for instance using the "back up" alternative). This is even more true since it is clear that that infrastructure would take time to develop and an incremental solution now that improves the collaboration will not make it more difficult to develop such a solution.

My only strong feeling here is that I think we need documentation of this process in ./docs in addition to this proposal.

Michael

On Wed, Dec 7, 2016 at 7:30 PM Bob Wilson via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
I would like to make a change in the way we handle the master-next branch.

Summary: I’d like to switch to a model where we continuously test against the latest upstream LLVM changes. The goal is to simplify the process and make it easier to collaborate on maintaining master-next.

Background: We develop Swift against “stable” branches of LLVM (which I am using here to refer to the llvm, clang, and compiler-rt repositories) that are typically rebranched from trunk once for each release, with other commits individually cherry-picked for specific bug fixes and other changes. This insulates Swift development from the churn of changes in LLVM. At the same time, we maintain the “master-next” branches of Swift repos to keep up to date with trunk LLVM. For Swift, our “trunk” comes from the “upstream-with-swift” branches in our GitHub LLVM repos. We have existing automation to continuously merge changes from llvm.org <http://llvm.org/> into those upstream-with-swift branches.

We currently use a manual process to update master-next. Someone on the Swift team is designated as the "merge czar" and is responsible for this. This merge typically happens once every few weeks. Michael Gottesman developed some internal tools to help automate the process, but someone still needs to drive those tools manually. The process involves merging “master” to “master-next” for all the Swift repos and updating the “stable-next” branches of the GitHub LLVM repos for Swift. The “stable-next” branches are basically snapshots of the LLVM upstream-with-swift branches at the point where master-next was most recently merged.

Swift CI includes a set of Jenkins bots to test master-next building with the stable-next branches of LLVM (https://ci.swift.org/view/swift-master-next). The merge czar can use these bots to confirm that everything is working after a merge.

Reasons to change: The current process has the advantage that the merge czar can choose when to do a merge and can schedule that around other work, but it has some significant problems.

- It is difficult for multiple people to collaborate on updating master-next. The changes involved are often rev-locked between Swift and the LLVM repos, so there is no good way for someone to fix a problem without doing the whole merge process.

- The current system is hard to understand. I’ve been serving as the merge czar for the last few months, and it took me a while to figure out how to do it well.

- It requires extra “stable-next” branches in our GitHub LLVM repos, further adding to the complexity.

- The tools we have to help automate the process are currently internal to Apple and require ongoing maintenance. They could be cleaned up to release publicly but that would take more work.

Proposal: We already have Jenkins bots testing master-next. I would like to add a job to continuously merge master to master-next and change the existing bots to build against the “upstream-with-swift” branches in our GitHub LLVM repos. The bots would then detect any new problems soon after they are introduced. Anyone could fix those problems, whether they are merge conflicts, build failures, or test issues. A partial fix could be applied directly without needing to resolve all of the outstanding issues.

This would avoid the need for our current internal merging tools. We already have automatic merging bots, so adding another one would not be difficult.

The biggest advantage is that it provides a straightforward model that anyone can understand: master-next becomes just another branch that anyone can modify, build and test in the usual way. Collaboration is no more difficult than for other branches.

The cost of this simpler approach is that we would need to be willing to let the master-next branch break occasionally. An LLVM change might break things in a way that takes some time to fix, and the master-next bots would continue to fail during that time. Someone might want to apply a partial fix that does not resolve all the issues, and we would want to allow that even if the bots still fail. That would mean we would have to relax (or override) the requirement for PR testing for commits in that kind of situation. In the worst case, if new problems are introduced more quickly than we can fix them, this approach could fall apart. My experience as merge czar over the last few months suggests that is unlikely. Usually there are no more than a few problems per week and most of them are easy to fix.

I propose to roll this out in steps. First, we can add a new Jenkins bot that tests master-next building against upstream-with-swift. If that goes well, and if there are no objections to this proposal, we can add the automerger to merge master into master-next. At the same time, we would update the other master-next bots to use upstream-with-swift instead of stable-next for the LLVM repos.

Alternative: We can achieve some of the same goals at a considerable increase in complexity by introducing an automatic gated merge solution. We would have automation perform the merge and commit it as long as everything works. If there were any problems, the automation would create a pull request that would need to be manually updated to resolve the problems. People could still collaborate by working together on the pull request branch. Until the problems were resolved, no further merging would take place. I would like to try the simple approach before considering this more complex solution, since I don’t think it will be necessary, at least in the near future.

Any objections to this? Comments or suggestions?
_______________________________________________

swift-dev mailing list

swift-dev@swift.org <mailto:swift-dev@swift.org>

https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev