[Review] SE-0145: Package Manager Version Pinning

Evolution Discussion
1

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  swift-evolution/0145-package-manager-version-pinning.md at master · apple/swift-evolution · GitHub

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  swift-evolution/process.md at master · apple/swift-evolution · GitHub

Thank you,

Anders Bertelrud
Review Manager

2

Hello swift-build-dev,

I just wanted to make sure that everyone on this list saw the Swift Evolution proposal for Package Manager Version Pinning that was sent to the swift-evolution list. The proposal announcement is included below.

Kind regards,

Anders

···

Begin forwarded message:

From: Anders Bertelrud via swift-evolution <swift-evolution@swift.org>
Subject: [swift-evolution] [Review] SE-0145: Package Manager Version Pinning
Date: October 31, 2016 at 14.23.35 PDT
To: swift-evolution <swift-evolution@swift.org>
Reply-To: Anders Bertelrud <anders@apple.com>

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

3

Hi Swift community

* What is your evaluation of the proposal?

The proposal looks good and it solves the addressed problems. Resolving dependency graph is not a trivial problem and I like how this proposal tackles 1 problem at the time.

* Is the problem being addressed significant enough to warrant a change to Swift?

Yes it does. With the "Version Pinning" it would be very convenient to share that information with others

Does this proposal fit well with the feel and direction of Swift?

Yes it does.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Some other dependency managers choose to generate pin files by default. I feel it's a correct choice to make "Pin file" optional since SwiftPM requires a versioning tag for a dependency (which is option in others dependency managers I've used)

* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I tried to study more about dependency pinning and locking, how they are implemented tin other dependency manager and what impact they have.

Additional notes:
When I run swift package update on unpinned package, would it be possible to roll back to previously used dependencies?

Use case: Let's say I have a package with a dependency A
.Package(url: "https://A.git", versions: Version(1,0,0)..<Version(2,0,0)),

At first fetch I get A dependency with v1.0. Then I run `swift package update` and A gets updated to 1.1
The package is not pinned. If that dependency update break my package is there an easy way for me to roll back?

Best Regards
Kostiantyn

···

On 31 Oct 2016, at 22:23, Anders Bertelrud via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

4

  * What is your evaluation of the proposal?

+1

  * Is the problem being addressed significant enough to warrant a change to Swift?

Yes, version pinning is very important to dependency management in production environments.

  * Does this proposal fit well with the feel and direction of Swift?

Yes.

  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I have used various dependency management systems and had seen some evolving in particular domains (pip’s eventual emergence over the last decade in Python land comes to mind. Linux package managers as found in Debian/CentOS/Arch/Gentoo, albeit less analogous here). Ruby gems, Cocoapods and Carthage are the ones I’m using in production today.

Excluding a few cosmetic differences (.pins extensions, whether to automatically pin, pinning commands etc), the proposed mechanisms are actually very similar to the ones we know and love in iOS/macOS development today: there’s some remote sources where the dependencies reside, most likely version controlled repositories with explicit ways to identify a certain state in their history. Using a text file to record the state identifiers, the manager can ensure shared, reproducible building environment. No surprise here.

As for the difference: I can’t recall a time when I really thought about the file extension of Podfile.lock or Carthage.resolved or Gems.WhatEverTheHeck. The proposal had dedicated a good potion to explain the choice of .pins, that deserves some extra credit IMHO. Explicit pinning commands gives us more control. In a team environment, if you are using a pinning feature, it’s *really* annoying to have accidentally triggers a change in the .lock/.resolved files. This is happens when your team opt-out a latest version of the dependency. Automatic pinning, therefore, is not a feature, it’s the lack of one.

  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read initial draft of the proposal and its discussions. Read the actual proposal.

5

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
I think we should try to build a system which is robust and works all of the time.

Being able to reproduce any build is a great value, no matter which package we are working on.
We should build our package management in a way that reproducible builds are the default.
Therefor I strongly propose to always store the exact version of all dependencies under
version control.

Just adding opt-ins (pinning of individual packages) creates new problems:
How should the user/maintainer select which packages to pin? How do we maintain this list?
E.g. even after the proposed `swift package pin —all`, you can end up with unpinned dependencies
after a `swift package --repin` when one dependency introduces some new sub-dependency.
Or should we also pin these sub-dependencies when the parent was already pinned?
But then what to do when the newly introduces sub-dependency was already part of the
closed set of all dependencies before, but was explicitly not pinned?

We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.
Just store the version of all dependencies but make it easy to change/update them.
Of course, the used versions must be under control of the top-level package which is being built.
All the version information stored within dependencies may be a nice hint for the top-level
maintainer, but should not be used by the build system.
This way, we don’t have to differentiate between top-level and library packages.

I don’t think we have to be worried about weakening semver compatibility.
Library packages will be included by many top-level packages, each with their own
update cycle. Each update will be performed according to the semver specification.
CI systems also do not have to just build the current version, but could also use
an extra builder which first updates all dependencies and then does its tests.
Not versioning some dependencies does not provide any benefit here.
I also agree that it’s good to encouraging frequent updates, but we should find a different means.
We should make it easy to update dependencies, but leave it to each package maintainer to control
when and what to update.

If we don’t ‚pin‘ individual packages but simply store all versions, then I also propose
to use the name `Package.versions` for our new file.

Is the problem being addressed significant enough to warrant a change to Swift?

yes.
Reproducible builds are important and should be supported by swift-pm.

Does this proposal fit well with the feel and direction of Swift?

Reproducible builds fit well into the safe-by-default approach of Swift.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I always hated `npm` and all those node packages which had overly restrictive dependency requirements.
I guess these were used in order to have more control about which version to use, but this resulted
in both packages which I could not build out of the box (because some dependencies changed),
and packages I could not update because of conflicting requirements.
Using loose version requirements + semver in `Package.swift` files together with exact versions
of all dependencies stored in the top-level package solves everything: you get reproducible builds
(based on `Package.versions`) together with easy updates (based on all packages’ `Package.versions`).

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the proposal and the comments.

-- Martin

6

What is your evaluation of the proposal?

General +1, with reservations. Novel elements of the proposed behavior will need careful evaluation and refinement as we see how they plays out in practice, with open-mindedness from both users and the core team.

Breaking it down:

+1 on having this feature in some form. It’s essential.

+1 on making user choice about .gitignore the thing that controls whether and how pinning is shared within a team. That’s simple, clear, accommodates a wide range of needs, and is consistent with other package managers.

+1 that pinfiles have no effect whatsoever on dependent projects. That’s the only sensible way for it to work, but since there was some debate about that, I’ll just reiterate support.

-1 on making dependencies unpinned by default. Trying to induce unexpected behavior to encourage testing can be a good technique — in contexts where testing is the goal. My gut tells me that doing this when building is the goal will cause a lot of confusion and kvetching. I follow the proposal’s argument that unexpected breakages are a nice way to make strict semver a community norm … and I just do not buy it.

However, given that we hashed this out at great length and the core team is still enamored of the idea, I’m willing to give it a try! I’d love to be proved wrong.

+1 on the proposed command structure given that I’ve lost the aforementioned “always pin” argument. Living in a sometimes-pinned-sometimes-not world is going to be confusing, but the proposed commands help as best they can.

¿-1? This is a big one. If I do:

    swift package pin --all

…and then add a new dependency, is the new dependency also pinned? It should be. To pin or not to pin is typically a project- and team-wide policy decision. I do see the use case for pinning just one ill-behaved dependency, but more typically pinning is something that is built in to a team’s testing process and their assumptions about a whole build’s behavior.

The proposal is vague on this point, but could be interpreted to mean that --all does not pin new dependencies: “Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.” (Aside: there should be a semicolon instead of a comma in that sentence.) I assume — hope — that this is not the case! If --all does not affect new dependencies, then I’m -1 on the proposal.

¿-1? The proposal mentions that SwiftPM already effectively performs local pinning, but is ambiguous about whether this behavior remains separate from the new pinfile. I’m dubious about having two separate pinning mechanisms, one visible and one invisible.

+1 to the proposal’s repeated mentions of clear output and helpful diagnostics. Since this proposal introduces behavior that’s somewhat off the beaten path for package managers, this will be essential.

On the pin/lock controversy

I don’t care. Computer science is full of heavily overloaded terms where a loose underlying concept takes on radically different meanings (bridge, channel, dispatch, edge, graph, header, key, model, module, node, open, parameter, port, process, protocol, query, return, row, source, union). We do just fine disambiguating all these in context, thank you very much. Renaming “lock” to “pin” solves a problem that doesn’t exist.

However, we programmers are _also_ used to dealing with synonyms or partially overlapping near-synonyms (nil / null; closure / lambda / block; field / instance variable; tagged union / associated type enum; etc) and we also do just fine with those too. I’m sure we’ll learn to deal with lock / pin, and nobody will care after 6 months.

In short, “pin” is an unobjectionable solution to a non-problem. Core team is excited about “pin?” Grand. It’s a fine term. Do it and move on.

Is the problem being addressed significant enough to warrant a change to Swift?

It’s essential. SwiftPM will be impractical in many real-world situations until this is sorted out.

Does this proposal fit well with the feel and direction of Swift?

Yes, it’s consistent with the general approach of SwiftPM.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I’ve used bundler, Carthage, and CocoaPods extensively. All of them always generate a lockfile (Gemfile.lock, Cartfile.resolved, and Podfile.lock). All of them use these files as the unique mechanism for version locking, and all use version control of that file as the unique mechanism for controlling whether to locked versions are shared across teams.

We have many years of evidence that this model works well.

Note that all of these package managers also work in in environments that do not support using multiple versions of a dependency in the same artifact at the same time. Therefore this statement from the proposal:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

···

—

I’ve also used npm and bower, which either do not have version locking or only provide it via add-ons. It’s a nightmare. Lack of locking has caused headaches and lost hours — not hypothetic headaches, but real ones on actual projects — in two scenarios:

1. onboarding new developers who get fresh, incompatible dependency versions on initial checkout; and
2. picking projects back up for a new round of development.

Are these two situations really the right time for people to accidentally test whether their dependencies have properly followed semantic versioning? No. There are better ways, and better times. I am troubled by the insistence on ignoring experience here. However, as I said above, I’m willing to give it a try. I will keep an open mind in the name of bold experimentation, and would be happy to have my concerns proven wrong.

Please do keep in mind, however, that this is an experiment. Be ready for all that careful theorizing to be falsified by experience. You may have to murder this darling: “Kill your darlings” writing advice: What writer really said to murder your babies?

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

In depth, though I only read some of the discussion thread.

Cheers,

Paul

On Oct 31, 2016, at 4:23 PM, Anders Bertelrud via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

7

Hi all,

  * What is your evaluation of the proposal?

-1

  * Is the problem being addressed significant enough to warrant a change to Swift?

Yes, this is significant problem that basically prevents SwiftPM from being used in production environment due to making builds not consistently reproducible without committing source code of the compiled dependencies.

  * Does this proposal fit well with the feel and direction of Swift?

No, it doesn't feel right, as it breaks existing conventions and disregards the experience with other package managers that provide .lock files by default and those that don't (npm) now have replacements that do lock by default (yarn, https://code.facebook.com/posts/1840075619545360\).

I also disagree with the naming of the feature (pinning) as it breaks existing conventions and makes it confusing to people coming from other environments and ecosystems. I use lockfiles (as also most of the developers I know) much more frequently (almost every day) than POSIX locks (almost never, many thanks to GCD and other high-level concurrency features in other languages for that). I'm afraid the argument about overloading doesn't convince me at all, as many terms are overloaded, but that never was a problem as an established context and conventions matter more.

  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Yes, I use yarn, CocoaPods and Carthage on daily basis, and locking dependencies by default was never a problem with those. On the contrary, I had a lot of bad experience with npm, which doesn't lock by default.

  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I tracked this proposal from the draft version and did and studied how package managers for other ecosystems has evolved.

···

On 31 Oct 2016, at 21:23, Anders Bertelrud via swift-evolution <swift-evolution@swift.org> wrote:

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

8

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

···

On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

9

No contradiction:
Storing versions of all dependencies and having reproducible builds is independent from updating dependencies.
What you need is some automatic (e.g. driven by CI) system to update your dependencies.

When performing an update is the right time to select what you want to update.
Then you can also test the new set and document that you chose to update them.

Updates should always be explicit actions, not happening randomly at checkout time.

— Martin

···

Am 03.11.2016 um 03:22 schrieb Daniel Duan <daniel@duan.org>:
On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

10

Could you specify that in the tag of the dependency you need? >= 3.0.0 vs = 2.0.0 etc... I am not sure why in some cases we need to be unique over state of the art like Cocoapods in some areas. What is the added value (not saying that there is none)?

···

Sent from my iPhone

On 3 Nov 2016, at 02:22, Daniel Duan via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

11

Also, we should be really careful with our naming.
For me, pin or lock means: don’t touch it, keep this version.

If your use-case really is to keep a specific version, then well you should specify that as a requirement in your dependencies.
Then add this information (together with some comment why you absolutely need that version) to your `Package.swift`.

I want reproducible checkouts and builds, but I don’t want to make people feel that versions are set in stone.
We should encourage frequent updates. So we should not lock versions, we should just track and manage them.

If a package is actively maintained, then the maintainer will care about updating and adapting to changed dependencies.
If the package is not maintained any more, then it will not be adapted to changed dependencies anyway.
In this case it does not help to randomly break the build of dependent packages.

The best way out is to give dependent package maintainers the power to:
* get enough information to see that there is a problem with a dependency
* get enough time to be able to act accordingly (without having to rush because builds are already breaking everywhere)

With proper management of dependency versions, we can provide that.
Think of it as the git equivalent in package management :-)

— Martin

···

Am 03.11.2016 um 03:22 schrieb Daniel Duan <daniel@duan.org>:

On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

12

Note that this only partially true. It is strongly recommended to not check in your Gemfile.lock when developing a gem (see http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/\), but only when you’re developing an app. This means that pinning by default is effectively not performed when doing library development in the Ruby ecosystem.

Cheers,
Boris

···

On 4 Nov 2016, at 17:06, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

13

Thank you for articulating this, Paul.

Personally, I think we have all the evidence we need--including languages that have tried both and settled on pin-by-default--to know how this will turn out. But if the default is to not pin, I can simply write a `fish` function to shorthand the pinning commands and forget about this issue. No big deal.

I'm also less agnostic about the "pin" vs. "lock" question: I think "lock" is nearly universal prior art and we ought to stick to it. However, it's ultimately just a name. People will figure it out.

···

On Nov 4, 2016, at 9:06 AM, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

I’ve used bundler, Carthage, and CocoaPods extensively. All of them always generate a lockfile (Gemfile.lock, Cartfile.resolved, and Podfile.lock). All of them use these files as the unique mechanism for version locking, and all use version control of that file as the unique mechanism for controlling whether to locked versions are shared across teams.

We have many years of evidence that this model works well.

Note that all of these package managers also work in in environments that do not support using multiple versions of a dependency in the same artifact at the same time. Therefore this statement from the proposal:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

—

I’ve also used npm and bower, which either do not have version locking or only provide it via add-ons. It’s a nightmare. Lack of locking has caused headaches and lost hours — not hypothetic headaches, but real ones on actual projects — in two scenarios:

1. onboarding new developers who get fresh, incompatible dependency versions on initial checkout; and
2. picking projects back up for a new round of development.

Are these two situations really the right time for people to accidentally test whether their dependencies have properly followed semantic versioning? No. There are better ways, and better times. I am troubled by the insistence on ignoring experience here. However, as I said above, I’m willing to give it a try. I will keep an open mind in the name of bold experimentation, and would be happy to have my concerns proven wrong.

Please do keep in mind, however, that this is an experiment. Be ready for all that careful theorizing to be falsified by experience. You may have to murder this darling: “Kill your darlings” writing advice: What writer really said to murder your babies?

--
Brent Royal-Gordon
Architechies

14

Hi all,

  * What is your evaluation of the proposal?

-1

  * Is the problem being addressed significant enough to warrant a change to Swift?

Yes, this is significant problem that basically prevents SwiftPM from being used in production environment due to making builds not consistently reproducible without committing source code of the compiled dependencies.

  * Does this proposal fit well with the feel and direction of Swift?

No, it doesn't feel right, as it breaks existing conventions and disregards the experience with other package managers that provide .lock files by default and those that don't (npm) now have replacements that do lock by default (yarn, https://code.facebook.com/posts/1840075619545360\).

Yarn, however, can include multiple versions of a package. We cannot, and your response doesn't acknowledge the impact of that.

Note I'm not saying I disagree with you, but this argument isn't compelling unless you acknowledge the problems particular to Swift.

- Daniel

···

On Nov 4, 2016, at 5:28 AM, Max Desiatov via swift-evolution <swift-evolution@swift.org> wrote:

On 31 Oct 2016, at 21:23, Anders Bertelrud via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I also disagree with the naming of the feature (pinning) as it breaks existing conventions and makes it confusing to people coming from other environments and ecosystems. I use lockfiles (as also most of the developers I know) much more frequently (almost every day) than POSIX locks (almost never, many thanks to GCD and other high-level concurrency features in other languages for that). I'm afraid the argument about overloading doesn't convince me at all, as many terms are overloaded, but that never was a problem as an established context and conventions matter more.

  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

Yes, I use yarn, CocoaPods and Carthage on daily basis, and locking dependencies by default was never a problem with those. On the contrary, I had a lot of bad experience with npm, which doesn't lock by default.

  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I tracked this proposal from the draft version and did and studied how package managers for other ecosystems has evolved.

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

15

Thanks to everyone who participated in this review!

Based on the pretty universal negative feedback, we are going to reject this proposal as is, and take it back for another round of revisions.

Our revised plan is:
1. To introduce an "autopin" behavior to cover the problem Paul outlined where `pin --all` effectively needs to be "sticky" for any new dependencies which come into play.
2. To make auto pinning on by default, with an explicit mechanism for projects to opt out.

I hope to have this written up for review next week.

Thanks!
- Daniel

···

On Nov 4, 2016, at 9:06 AM, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

General +1, with reservations. Novel elements of the proposed behavior will need careful evaluation and refinement as we see how they plays out in practice, with open-mindedness from both users and the core team.

Breaking it down:

+1 on having this feature in some form. It’s essential.

+1 on making user choice about .gitignore the thing that controls whether and how pinning is shared within a team. That’s simple, clear, accommodates a wide range of needs, and is consistent with other package managers.

+1 that pinfiles have no effect whatsoever on dependent projects. That’s the only sensible way for it to work, but since there was some debate about that, I’ll just reiterate support.

-1 on making dependencies unpinned by default. Trying to induce unexpected behavior to encourage testing can be a good technique — in contexts where testing is the goal. My gut tells me that doing this when building is the goal will cause a lot of confusion and kvetching. I follow the proposal’s argument that unexpected breakages are a nice way to make strict semver a community norm … and I just do not buy it.

However, given that we hashed this out at great length and the core team is still enamored of the idea, I’m willing to give it a try! I’d love to be proved wrong.

+1 on the proposed command structure given that I’ve lost the aforementioned “always pin” argument. Living in a sometimes-pinned-sometimes-not world is going to be confusing, but the proposed commands help as best they can.

¿-1? This is a big one. If I do:

    swift package pin --all

…and then add a new dependency, is the new dependency also pinned? It should be. To pin or not to pin is typically a project- and team-wide policy decision. I do see the use case for pinning just one ill-behaved dependency, but more typically pinning is something that is built in to a team’s testing process and their assumptions about a whole build’s behavior.

The proposal is vague on this point, but could be interpreted to mean that --all does not pin new dependencies: “Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.” (Aside: there should be a semicolon instead of a comma in that sentence.) I assume — hope — that this is not the case! If --all does not affect new dependencies, then I’m -1 on the proposal.

¿-1? The proposal mentions that SwiftPM already effectively performs local pinning, but is ambiguous about whether this behavior remains separate from the new pinfile. I’m dubious about having two separate pinning mechanisms, one visible and one invisible.

+1 to the proposal’s repeated mentions of clear output and helpful diagnostics. Since this proposal introduces behavior that’s somewhat off the beaten path for package managers, this will be essential.

On the pin/lock controversy

I don’t care. Computer science is full of heavily overloaded terms where a loose underlying concept takes on radically different meanings (bridge, channel, dispatch, edge, graph, header, key, model, module, node, open, parameter, port, process, protocol, query, return, row, source, union). We do just fine disambiguating all these in context, thank you very much. Renaming “lock” to “pin” solves a problem that doesn’t exist.

However, we programmers are _also_ used to dealing with synonyms or partially overlapping near-synonyms (nil / null; closure / lambda / block; field / instance variable; tagged union / associated type enum; etc) and we also do just fine with those too. I’m sure we’ll learn to deal with lock / pin, and nobody will care after 6 months.

In short, “pin” is an unobjectionable solution to a non-problem. Core team is excited about “pin?” Grand. It’s a fine term. Do it and move on.

Is the problem being addressed significant enough to warrant a change to Swift?

It’s essential. SwiftPM will be impractical in many real-world situations until this is sorted out.

Does this proposal fit well with the feel and direction of Swift?

Yes, it’s consistent with the general approach of SwiftPM.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I’ve used bundler, Carthage, and CocoaPods extensively. All of them always generate a lockfile (Gemfile.lock, Cartfile.resolved, and Podfile.lock). All of them use these files as the unique mechanism for version locking, and all use version control of that file as the unique mechanism for controlling whether to locked versions are shared across teams.

We have many years of evidence that this model works well.

Note that all of these package managers also work in in environments that do not support using multiple versions of a dependency in the same artifact at the same time. Therefore this statement from the proposal:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

—

I’ve also used npm and bower, which either do not have version locking or only provide it via add-ons. It’s a nightmare. Lack of locking has caused headaches and lost hours — not hypothetic headaches, but real ones on actual projects — in two scenarios:

1. onboarding new developers who get fresh, incompatible dependency versions on initial checkout; and
2. picking projects back up for a new round of development.

Are these two situations really the right time for people to accidentally test whether their dependencies have properly followed semantic versioning? No. There are better ways, and better times. I am troubled by the insistence on ignoring experience here. However, as I said above, I’m willing to give it a try. I will keep an open mind in the name of bold experimentation, and would be happy to have my concerns proven wrong.

Please do keep in mind, however, that this is an experiment. Be ready for all that careful theorizing to be falsified by experience. You may have to murder this darling: “Kill your darlings” writing advice: What writer really said to murder your babies?

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

In depth, though I only read some of the discussion thread.

Cheers,

Paul

On Oct 31, 2016, at 4:23 PM, Anders Bertelrud via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

16

Daniel Duan

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

No contradiction:
Storing versions of all dependencies and having reproducible builds is independent from updating dependencies.

Ok, agreed.

What you need is some automatic (e.g. driven by CI) system to update your dependencies.

When you say "update", I assume you mean the action of increasing version numbers of certain dependency? I don't see how CI is going to help with that.

When performing an update is the right time to select what you want to update.
Then you can also test the new set and document that you chose to update them.

What kind of "documenting" are we talking about? I thought the .pins is a place record precise version of the dependency we need.

Updates should always be explicit actions, not happening randomly at checkout time.

I agree strongly that update should be explicit. Which is why running a pin command for a specific dependency *after* it's been tested is better. I'm guessing you are saying this because I mentioned dependencies that aren't pinned. If you have worked with internal frameworks that gets changed frequently with a CI running unit tests and integration tests, you'd realize that pinning it is a waste of your teams time. But there's nothing to prevent you from doing it with this proposal.

— Martin

It comes down to this: if you want automatic pinning, your build script should run the pin commands whenever you deem necessary.

The converse isn't true: if you force everyone to automatically pin, then there's no way to opt-out. Whether that's desirable is not up to anyone to decide.

···

Sent from my iPhone
On Nov 2, 2016, at 11:17 PM, Martin Waitz <tali@admingilde.org> wrote:

Am 03.11.2016 um 03:22 schrieb Daniel Duan <daniel@duan.org>:
On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

17

Could you specify that in the tag of the dependency you need? >= 3.0.0 vs = 2.0.0 etc…

That’s missing the point of pinning though. You can already do this in Package.swift. Pinning is, in a way, guarding against violation of semvar (in case some packages introduce breaking changes without bumping version).

I am not sure why in some cases we need to be unique over state of the art like Cocoapods in some areas. What is the added value (not saying that there is none)?

You can combine mechanisms to get your desired policy: get “pin by default” by always running “pin —all” along side other build steps. The converse isn’t true: you cannot get automatically updated dependencies (often your internal packages) at install without changing and checking in Podfile.lock. This issue manifests even more if you try to not distribute Podfile.lock: you get one anyways after your first install, so you’d have to remember to remove it before the next install).

(I’m not criticizing the practice of pinning everything by default, just want to point out it’s not flexible for cases where it’s not flexible. No one should decide those cases for anyone!)

The value added is well explained in the proposal. Simply saying “Cocoapods is good enough, why don’t we copy it” isn’t very constructive and interesting to me.

···

On Nov 3, 2016, at 1:02 AM, Goffredo Marocchi <panajev@gmail.com> wrote:

Sent from my iPhone

On 3 Nov 2016, at 02:22, Daniel Duan via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 2, 2016, at 3:46 PM, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:

What is your evaluation of the proposal?

+1 for using reproducible versions of dependencies
-1 for the actual proposal

My problem with this proposal is that it tries to please everybody by introducing options everywhere.
…
We should just drop all these problems and design a system which works for all use-cases
without having to manually pin dependencies.

The second half of the sentence contradicts with the first half. What if I need to only pin some of the dependencies, have have the rest update automatically?

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

18

-1, though I do strongly think we should have version pinning.

I agree with the others who believe that dependencies should be pinned by
default. This doesn't mean that you have to check the pin/lock file into
source control. I have a good amount of experience developing apps and
libraries with Cocoapods, Carthage, and Bundler, and I've always found it
to be crucial to be able to easily reproduce builds.

Regarding the naming, I don't have a strong opinion, but I find the
argument that the term "lock" is overloaded and therefore confusing to be
weak, given that it is a term used by many package managers.

···

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

19

If SwiftPM used that bundler-like behavior, then presumably the same recommendations about what to check in would apply. By “ubiquitous version pinning,” I mean that the package management tool _always_ generates a lock file, and developers decide for themselves how to manage it.

That section in the proposal made the argument that Swift is fundamentally different from other languages in ways that make always generating a pins file uniquely dangerous. That is false.

Cheers, P

···

On Nov 4, 2016, at 11:20 AM, Boris Buegling <bbuegling@apple.com> wrote:

On 4 Nov 2016, at 17:06, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

Note that this only partially true. It is strongly recommended to not check in your Gemfile.lock when developing a gem (see http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/\), but only when you’re developing an app. This means that pinning by default is effectively not performed when doing library development in the Ruby ecosystem.

20

Thank you, Daniel. I have marked the SE-0145 as such in the swift-evolution repository.

Anders

···

On 2016-11-10, at 09.41, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Thanks to everyone who participated in this review!

Based on the pretty universal negative feedback, we are going to reject this proposal as is, and take it back for another round of revisions.

Our revised plan is:
1. To introduce an "autopin" behavior to cover the problem Paul outlined where `pin --all` effectively needs to be "sticky" for any new dependencies which come into play.
2. To make auto pinning on by default, with an explicit mechanism for projects to opt out.

I hope to have this written up for review next week.

Thanks!
- Daniel

On Nov 4, 2016, at 9:06 AM, Paul Cantrell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

What is your evaluation of the proposal?

General +1, with reservations. Novel elements of the proposed behavior will need careful evaluation and refinement as we see how they plays out in practice, with open-mindedness from both users and the core team.

Breaking it down:

+1 on having this feature in some form. It’s essential.

+1 on making user choice about .gitignore the thing that controls whether and how pinning is shared within a team. That’s simple, clear, accommodates a wide range of needs, and is consistent with other package managers.

+1 that pinfiles have no effect whatsoever on dependent projects. That’s the only sensible way for it to work, but since there was some debate about that, I’ll just reiterate support.

-1 on making dependencies unpinned by default. Trying to induce unexpected behavior to encourage testing can be a good technique — in contexts where testing is the goal. My gut tells me that doing this when building is the goal will cause a lot of confusion and kvetching. I follow the proposal’s argument that unexpected breakages are a nice way to make strict semver a community norm … and I just do not buy it.

However, given that we hashed this out at great length and the core team is still enamored of the idea, I’m willing to give it a try! I’d love to be proved wrong.

+1 on the proposed command structure given that I’ve lost the aforementioned “always pin” argument. Living in a sometimes-pinned-sometimes-not world is going to be confusing, but the proposed commands help as best they can.

¿-1? This is a big one. If I do:

    swift package pin --all

…and then add a new dependency, is the new dependency also pinned? It should be. To pin or not to pin is typically a project- and team-wide policy decision. I do see the use case for pinning just one ill-behaved dependency, but more typically pinning is something that is built in to a team’s testing process and their assumptions about a whole build’s behavior.

The proposal is vague on this point, but could be interpreted to mean that --all does not pin new dependencies: “Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.” (Aside: there should be a semicolon instead of a comma in that sentence.) I assume — hope — that this is not the case! If --all does not affect new dependencies, then I’m -1 on the proposal.

¿-1? The proposal mentions that SwiftPM already effectively performs local pinning, but is ambiguous about whether this behavior remains separate from the new pinfile. I’m dubious about having two separate pinning mechanisms, one visible and one invisible.

+1 to the proposal’s repeated mentions of clear output and helpful diagnostics. Since this proposal introduces behavior that’s somewhat off the beaten path for package managers, this will be essential.

On the pin/lock controversy

I don’t care. Computer science is full of heavily overloaded terms where a loose underlying concept takes on radically different meanings (bridge, channel, dispatch, edge, graph, header, key, model, module, node, open, parameter, port, process, protocol, query, return, row, source, union). We do just fine disambiguating all these in context, thank you very much. Renaming “lock” to “pin” solves a problem that doesn’t exist.

However, we programmers are _also_ used to dealing with synonyms or partially overlapping near-synonyms (nil / null; closure / lambda / block; field / instance variable; tagged union / associated type enum; etc) and we also do just fine with those too. I’m sure we’ll learn to deal with lock / pin, and nobody will care after 6 months.

In short, “pin” is an unobjectionable solution to a non-problem. Core team is excited about “pin?” Grand. It’s a fine term. Do it and move on.

Is the problem being addressed significant enough to warrant a change to Swift?

It’s essential. SwiftPM will be impractical in many real-world situations until this is sorted out.

Does this proposal fit well with the feel and direction of Swift?

Yes, it’s consistent with the general approach of SwiftPM.

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I’ve used bundler, Carthage, and CocoaPods extensively. All of them always generate a lockfile (Gemfile.lock, Cartfile.resolved, and Podfile.lock). All of them use these files as the unique mechanism for version locking, and all use version control of that file as the unique mechanism for controlling whether to locked versions are shared across teams.

We have many years of evidence that this model works well.

Note that all of these package managers also work in in environments that do not support using multiple versions of a dependency in the same artifact at the same time. Therefore this statement from the proposal:

Overconstraint is much more of a risk in Swift than in other languages using this style of package management.

…is incorrect.

In particular, note that Ruby does not support using multiple versions of a lib simultaneously, and that fact alone — even in the presence of _ubiquitous_ version pinning — has been sufficient to encourage widespread mindfulness about semver compliance. All of the concerns expressed in the “Pin by default” section of the proposal also apply to Ruby, and have failed to materialize there.

—

I’ve also used npm and bower, which either do not have version locking or only provide it via add-ons. It’s a nightmare. Lack of locking has caused headaches and lost hours — not hypothetic headaches, but real ones on actual projects — in two scenarios:

1. onboarding new developers who get fresh, incompatible dependency versions on initial checkout; and
2. picking projects back up for a new round of development.

Are these two situations really the right time for people to accidentally test whether their dependencies have properly followed semantic versioning? No. There are better ways, and better times. I am troubled by the insistence on ignoring experience here. However, as I said above, I’m willing to give it a try. I will keep an open mind in the name of bold experimentation, and would be happy to have my concerns proven wrong.

Please do keep in mind, however, that this is an experiment. Be ready for all that careful theorizing to be falsified by experience. You may have to murder this darling: “Kill your darlings” writing advice: What writer really said to murder your babies?

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

In depth, though I only read some of the discussion thread.

Cheers,

Paul

On Oct 31, 2016, at 4:23 PM, Anders Bertelrud via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hello Swift community,

The review of SE-0145 "Package Manager Version Pinning" begins now and runs through November 4. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at

  https://lists.swift.org/mailman/listinfo/swift-evolution

or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

Anders Bertelrud
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution