Proposal: Package Manager Version Pinning


(Ankit Aggarwal) #1

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

···

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.


(Daniel Duan) #2

This is a familiar feature among package managers and has been proven useful in practice. Two points regarding the proposed text:

1. “honor” is mis-spelled in “weird queen’s dialect”.
2. SHA/man-in-the-middle attack section needs either more detailed explanation/reference materials, or shouldn’t be included at all (assuming when the hash feature gets introduced, there’d be another proposal for it).

Overall strong +1.

···

On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(orta) #3

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

···

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #4

This is a familiar feature among package managers and has been proven useful in practice. Two points regarding the proposed text:

1. “honor” is mis-spelled in “weird queen’s dialect”.
2. SHA/man-in-the-middle attack section needs either more detailed explanation/reference materials, or shouldn’t be included at all (assuming when the hash feature gets introduced, there’d be another proposal for it).

I agree this needs more discussion, but given that we aren't proposing anything here, I feel like it is just a shoutout to an area for future development. What is the harm of having it there?

One of the possible values of having it in is jumpstarting the actual conversation on the topic!

- Daniel

···

On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev <swift-build-dev@swift.org> wrote:

Overall strong +1.

On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #5

Hey Orta,

Thanks for this feedback, this was one of the hotly debated points as you might imagine.

Can you elaborate on exactly how you would prefer this to work?

The way I see it, the only thing that this changes is that the initial package **author** who wants to be in this situation has to, one time, run "package pin --all", and then "git add Package.pins". The latter is always going to be necessary, so are you arguing the first should just be done by default, or is there something deeper here?

- Daniel

···

On Oct 14, 2016, at 12:29 AM, orta therox via swift-evolution <swift-evolution@swift.org> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Daniel Duan) #6

The spelling nitpick should have been on GitHub in retrospect. I wrongly assumed everyone know of our preference for dialect :slight_smile:

Point 1-7 are all concrete descriptions of features. That's probably why the SHA point feels out-of-place. Perhaps it deserves its own section.

···

On Oct 14, 2016, at 9:26 AM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev <swift-build-dev@swift.org> wrote:

This is a familiar feature among package managers and has been proven useful in practice. Two points regarding the proposed text:

1. “honor” is mis-spelled in “weird queen’s dialect”.
2. SHA/man-in-the-middle attack section needs either more detailed explanation/reference materials, or shouldn’t be included at all (assuming when the hash feature gets introduced, there’d be another proposal for it).

I agree this needs more discussion, but given that we aren't proposing anything here, I feel like it is just a shoutout to an area for future development. What is the harm of having it there?

One of the possible values of having it in is jumpstarting the actual conversation on the topic!

- Daniel

Overall strong +1.

On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar, Ankit Aggarwal
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #7

The spelling nitpick should have been on GitHub in retrospect. I wrongly assumed everyone know of our preference for dialect :slight_smile:

Point 1-7 are all concrete descriptions of features. That's probably why the SHA point feels out-of-place. Perhaps it deserves its own section.

Good point, let's move it to a future directions section or something.

- Daniel

···

On Oct 14, 2016, at 9:35 AM, Daniel Duan <daniel@duan.org> wrote:

On Oct 14, 2016, at 9:26 AM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

On Oct 13, 2016, at 11:16 PM, Daniel Duan via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

This is a familiar feature among package managers and has been proven useful in practice. Two points regarding the proposed text:

1. “honor” is mis-spelled in “weird queen’s dialect”.
2. SHA/man-in-the-middle attack section needs either more detailed explanation/reference materials, or shouldn’t be included at all (assuming when the hash feature gets introduced, there’d be another proposal for it).

I agree this needs more discussion, but given that we aren't proposing anything here, I feel like it is just a shoutout to an area for future development. What is the harm of having it there?

One of the possible values of having it in is jumpstarting the actual conversation on the topic!

- Daniel

Overall strong +1.

On Oct 13, 2016, at 11:01 PM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Haravikk) #8

I'm sorry, but I throw an exception at this; when you say "weird queen's dialect" do you mean… English? I don't go around referring to the other variation as "weird can't spell dialect" (but maybe I should) :stuck_out_tongue:
The u in words like honour is what makes it pronounced like "onner" rather than "hone-ore", same with colour being "culler" rather than "coal-ore". Sorry but it really annoys me when people consider the English version of English to be the weird one, as the clue is literally in the name as to which version is correct. This is why it still galls me that the language in macOS is "British English" yet the other one is just "English" rather than "American English" or whatever, it's a hideous double standard.

That aside, I'm a strong +1 for the feature :wink:

···

On 14 Oct 2016, at 07:16, Daniel Duan via swift-evolution <swift-evolution@swift.org> wrote:

1. “honor” is mis-spelled in “weird queen’s dialect”.


(Alexis) #9

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

I’ve never seen this cause any actual confusion, nor has anyone I know who teaches/develops these sorts of tools. As far as I can tell, the broader programming community is rapidly converging on this as standard terminology:

* Gemfile.lock (Ruby)
* Cargo.lock (Rust)
* Composer.lock (PHP)
* yarn.lock (JS)
* pubspec.lock (Dart)
* Podfile.lock (Swift/Objc!)

Diverging from this seems counter-productive.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

I agree with the others that this is the better solution.

With regards to the constraining problem, the key insight adopted by Cargo/Yarn/Bundler is to distinguish libraries from applications. A library shouldn’t pin its dependencies, while an application should. This ensures that the ecosystem itself is maximally unconstrained, while ensuring actual applications continue to reliably build, regardless of ecosystem changes and the computer that it was built on. If a version of a library has trouble building with different versions, it should ideally specify that with its dependency constraints, not a lockfile.

This also ensures that there’s diverse testing of versions: CI for applications will verify particular configurations, while CI for libraries will verify the latest-and-greatest works.

···

On Oct 14, 2016, at 2:01 AM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org> wrote:

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Eloy Duran) #10

I cannot agree more with Orta.

It drives the user away from taking full advantage of semantic versioning.

While I ideally subscribe to this thought, the truth of the matter is that this has proven unreliable on multiple occasions. Promoting the idea that it works is going to inevitably lead users into problems when they did not take explicit user action to update dependencies.

Semantic versioning is a great tool when you decide to sit down and explicitly update dependencies.

We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

Again, ideally I would subscribe to this, but the truth is that when people are working on their projects they *really really really* do *not* like their builds breaking, especially not when it’s seemingly caused outside of their own fault.

In the end, it comes down to whether you prioritise surfacing bugs (long term happiness) over user-experience (short term happiness). As a dependency manager author, I can tell you that I agree with your ideals and would want to choose the former, but as a user of dependency managers I would choose UX and Getting (The) Things (I Really Wanted To Do) Done over surfacing bugs *any* day.

[A.] Eloy Durán
    Artsy <http://artsy.net/>
    
    eloy@artsy.net <mailto:eloy@artsy.net> Twitter <https://twitter.com/alloy>
    GitHub <https://github.com/alloy>

···

On 14 Oct 2016, at 09:29, orta therox via swift-build-dev <swift-build-dev@swift.org> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Max Desiatov) #11

I also strongly agree with this, I'd prefer version pinning to happen by default, rather than with explicit command as it will make builds reproducible by default.

I totally agree that we can rely on past experience with other package managers (npm being the case), where pinning with a separate command caused more harm than good.

Overall, I think that it would be great if Package.lock was created by default when it's not present and be updated only with an explicit command for updating.

With best regards, Max.

···

On 14 Oct 2016, at 08:29, orta therox via swift-build-dev <swift-build-dev@swift.org> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Rob Allen) #12

Hi,

As noted by others, the standard extension for this file .lock, so I would prefer to keep that convention.

As I would never want my CI or any random team member to build a different package to what I've developed and tested already, I will always want Package.lock to exist. It's much easier if the system always creates Package.lock on swift package update and always uses it (if it exists) with swift build. This way, I get to choose when to go from 1.3.1 to 1.3.2 just in case I happen to depend on that bug that's now been fixed…

I can't see any downside to not creating it automatically and it results in so much more consistency across a team.

Regards,

Rob...

···

On 14 Oct 2016, at 17:29, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

Hey Orta,

Thanks for this feedback, this was one of the hotly debated points as you might imagine.

Can you elaborate on exactly how you would prefer this to work?

The way I see it, the only thing that this changes is that the initial package **author** who wants to be in this situation has to, one time, run "package pin --all", and then "git add Package.pins". The latter is always going to be necessary, so are you arguing the first should just be done by default, or is there something deeper here?

- Daniel

On Oct 14, 2016, at 12:29 AM, orta therox via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.


(Daniel Dunbar) #13

Hey Eloy,

Same question as I sent to Orta, can you detail exactly what you would prefer to change?

I cannot agree more with Orta.

> It drives the user away from taking full advantage of semantic versioning.

While I ideally subscribe to this thought, the truth of the matter is that this has proven unreliable on multiple occasions. Promoting the idea that it works is going to inevitably lead users into problems when they did not take explicit user action to update dependencies.

This makes a lot of sense to me. The open question in my mind is, given our other goals, are we in a position to make this model work, or to make it "the right model for us". Some of the other things I am considering (and maybe should go into the proposal):

1. Swift is still evolving in a much greater way than other languages. That means we continue to need to have a very strong push towards forward versions. This is also true for the package manager features, we need the ecosystem to move forward aggressively so that we can move our own features forward aggressively.

2. We have some lofty goals around semantic versioning, like trying to have a much deeper integration with the compiler, API, and ABI, and testing in order to help manage this more effectively.

Semantic versioning is a great tool when you decide to sit down and explicitly update dependencies.

The problem is they go hand in hand. The more people pin versus following the semantic versioning, the more and more likely it is that those specifications are wrong. That leads to more pinning, which leads to more wrong specifications.

My preference is that we very aggressively commit to using semantic versioning, and then follow up with the entire cross functional design (an index, if we ever do one, the compiler features, any IDE integration) to make this work. I think we are in a better position than other tools which didn't have the ability to make as many cross functional changes (e.g., integration with the compiler to assist in semantic versioning).

On the other hand, I am also very pragmatic, and I respect your experience here... if this model simply isn't going to work, then we shouldn't try to go that way.

The big struggle I have is that if we go the other direction, and as a result people's semantic versions become poorly specified, we will never be able to recover. The converse is not true, if we start with this direction and realize it doesn't work, we can relax our behavior.

> We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

Again, ideally I would subscribe to this, but the truth is that when people are working on their projects they *really really really* do *not* like their builds breaking, especially not when it’s seemingly caused outside of their own fault.

One discussion we had a lot was that there are very different workflows between when you are largely the consumer of packages, versus when you are working on a package that is shared.

We generally agree that when you are simply the consumer of packages, pinning makes sense.

However, when you are primarily a distributor of packages (and this is expected to *most* of the momentum behind SwiftPM in the near term), I believe that it is very important to the ecosystem that the semver specs be correct, and so even if the team *wants* to pin, doing so would be actively harmful.

In the end, it comes down to whether you prioritise surfacing bugs (long term happiness) over user-experience (short term happiness). As a dependency manager author, I can tell you that I agree with your ideals and would want to choose the former, but as a user of dependency managers I would choose UX and Getting (The) Things (I Really Wanted To Do) Done over surfacing bugs *any* day.

I really appreciate the feedback. Do my arguments that we might be in a position to do better here have any weight with you?

- Daniel

···

On Oct 14, 2016, at 4:06 AM, Eloy Durán via swift-evolution <swift-evolution@swift.org> wrote:

[A.] Eloy Durán
    Artsy <http://artsy.net/>
    
    eloy@artsy.net <mailto:eloy@artsy.net> Twitter <https://twitter.com/alloy>
    GitHub <https://github.com/alloy>

On 14 Oct 2016, at 09:29, orta therox via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Daniel Dunbar) #14

Can you check my reply to Eloy and see how it weighs with you?

- Daniel

···

On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev <swift-build-dev@swift.org> wrote:

I also strongly agree with this, I'd prefer version pinning to happen by default, rather than with explicit command as it will make builds reproducible by default.

I totally agree that we can rely on past experience with other package managers (npm being the case), where pinning with a separate command caused more harm than good.

Overall, I think that it would be great if Package.lock was created by default when it's not present and be updated only with an explicit command for updating.

With best regards, Max.

On 14 Oct 2016, at 08:29, orta therox via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Huon Wilson) #15

Not only “shouldn’t” but can’t: Rust’s cargo (I don’t know for sure about Bundler and Yarn, but they’re all in the same vein, and pretty much from the same person) will just ignore any non-top-level lock files, i.e. the package that user is working on has a record of the exact versions of everything in their dependency graph, and the user can update all or just parts of that graph fairly precisely, as desired.

My impression is that other ecosystems without pinning-by-default have seriously suffered from the lack of reliable reproducible builds. For instance, a major complaint I have seen about "go get” is that one can come back to some code after a few months to fix a minor bug and have no chance of getting the code to compile, basically just requiring a full rewrite, because dependencies' code has changed (or disappeared entirely) and there’s no record of what the original was, despite your code being completely untouched in all that time. Of course, a hard push towards API-level semantic versioning may allay the worst of the code-changing problems, but unless there’s going to be some seriously fancy static analysis (something something halting problem), there will be a lot of breaking changes that can’t be picked up.

···

On Oct 14, 2016, at 09:43, Alexis via swift-evolution <swift-evolution@swift.org> wrote:

A library shouldn’t pin its dependencies, while an application should.

---

To reframe the discussion slightly, not pinning dependencies effectively means that the tools are making “random" changes to your code outside your control. Sure, those changes were transitively made by possibly-computer-assisted humans and may fix bugs, but those pesky meat-bags can still make mistakes, and, they don’t see the interactions between all the different packages in your specific application.

IME having the predictability of “never changing code under my feet” combined with pervasive semantic versioning hits an empowering middle ground of being able to depend on many & varied packages and update them to the latest versions without fear (for one, it’s easy to revert the lock file if something breaks during an update). Feedback from others about cargo is the same, often along the lines of “it’s the best package manager I’ve ever used”.

Also, more on the side of implementation details, having a lock file by default probably mean one can (get closer to) never hitting the network unless the user asks for it, making working offline or on a low-speed/high-latency connection easier, i.e. no need to check for new versions of the whole dependency graph on O(every) build, which aggressively keeping everything up to date would seemingly have to do? (I don’t know swift-pm’s current behavior here, so maybe there’s some tricks that avoid it.)

Huon


(Daniel Dunbar) #16

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

I’ve never seen this cause any actual confusion, nor has anyone I know who teaches/develops these sorts of tools. As far as I can tell, the broader programming community is rapidly converging on this as standard terminology:

* Gemfile.lock (Ruby)
* Cargo.lock (Rust)
* Composer.lock (PHP)
* yarn.lock (JS)
* pubspec.lock (Dart)
* Podfile.lock (Swift/Objc!)

Diverging from this seems counter-productive.

I replied to Max on this point, can you check that?

I will add that I am completely open to being compelled by "this is becoming standard, we should follow".

Also, let's break this into its own fork of the thread rather than mix with the behavior.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

I agree with the others that this is the better solution.

With regards to the constraining problem, the key insight adopted by Cargo/Yarn/Bundler is to distinguish libraries from applications. A library shouldn’t pin its dependencies, while an application should. This ensures that the ecosystem itself is maximally unconstrained, while ensuring actual applications continue to reliably build, regardless of ecosystem changes and the computer that it was built on. If a version of a library has trouble building with different versions, it should ideally specify that with its dependency constraints, not a lock file.

This I think is one big crux of the current discussion, and one of the things not in the proposal was my expectation that most packages right now are going to be shared (I personally wouldn't say libraries vs applications, since applications can also be shared as build tools, I think it comes down to whether it is shared or not).

If we agree that a library shouldn't pin its dependencies, how would you surface this feature? What exact behavior would you prefer?

- Daniel

···

On Oct 14, 2016, at 9:43 AM, Alexis via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 14, 2016, at 2:01 AM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This also ensures that there’s diverse testing of versions: CI for applications will verify particular configurations, while CI for libraries will verify the latest-and-greatest works.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Daniel Dunbar) #17

A library shouldn’t pin its dependencies, while an application should.

Not only “shouldn’t” but can’t: Rust’s cargo (I don’t know for sure about Bundler and Yarn, but they’re all in the same vein, and pretty much from the same person) will just ignore any non-top-level lock files, i.e. the package that user is working on has a record of the exact versions of everything in their dependency graph, and the user can update all or just parts of that graph fairly precisely, as desired.

My impression is that other ecosystems without pinning-by-default have seriously suffered from the lack of reliable reproducible builds. For instance, a major complaint I have seen about "go get” is that one can come back to some code after a few months to fix a minor bug and have no chance of getting the code to compile, basically just requiring a full rewrite, because dependencies' code has changed (or disappeared entirely) and there’s no record of what the original was, despite your code being completely untouched in all that time.

This wouldn't have been our behavior under this proposal, unless you recloned the package. Maybe that is what you meant though?

Of course, a hard push towards API-level semantic versioning may allay the worst of the code-changing problems, but unless there’s going to be some seriously fancy static analysis (something something halting problem), there will be a lot of breaking changes that can’t be picked up.

---

To reframe the discussion slightly, not pinning dependencies effectively means that the tools are making “random" changes to your code outside your control. Sure, those changes were transitively made by possibly-computer-assisted humans and may fix bugs, but those pesky meat-bags can still make mistakes, and, they don’t see the interactions between all the different packages in your specific application.

IME having the predictability of “never changing code under my feet” combined with pervasive semantic versioning hits an empowering middle ground of being able to depend on many & varied packages and update them to the latest versions without fear (for one, it’s easy to revert the lock file if something breaks during an update). Feedback from others about cargo is the same, often along the lines of “it’s the best package manager I’ve ever used”.

To be clear, exactly when are you expecting the tools to "change code under my feet".

In our design, the only time we ever change a *local build* is in response to an explicit user action, so we aren't changing code under your feet necessarily.

If you throw away a checkout, and fetch it again, then yes we get a different version. Is this the use case you are talking about (and by implication talking about committing the file)?

If you reset your local workspace completely, then yes we will get a different version (since we had to reclone).

I just want to make sure it is clear to everyone that we are not proposing we automatically change your build version underneath you any time we happen to fetch something new from the package. We only update in response to user action.

Also, more on the side of implementation details, having a lock file by default probably mean one can (get closer to) never hitting the network unless the user asks for it, making working offline or on a low-speed/high-latency connection easier, i.e. no need to check for new versions of the whole dependency graph on O(every) build, which aggressively keeping everything up to date would seemingly have to do? (I don’t know swift-pm’s current behavior here, so maybe there’s some tricks that avoid it.)

This is part of the proposed design (this is the current implementation, really), FWIW, although that may not have been clear at all.

- Daniel

···

On Oct 14, 2016, at 10:37 AM, Huon Wilson via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 14, 2016, at 09:43, Alexis via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Huon
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Goffredo Marocchi) #18

Let's not promote this idea English has comprehensible and easy to refer to pronunciation rules ;). It needs reforming :P, but let's keep it for English-Evolution ;).

···

Sent from my iPhone

On 14 Oct 2016, at 13:42, Haravikk via swift-evolution <swift-evolution@swift.org> wrote:

On 14 Oct 2016, at 07:16, Daniel Duan via swift-evolution <swift-evolution@swift.org> wrote:

1. “honor” is mis-spelled in “weird queen’s dialect”.

I'm sorry, but I throw an exception at this; when you say "weird queen's dialect" do you mean… English? I don't go around referring to the other variation as "weird can't spell dialect" (but maybe I should) :stuck_out_tongue:
The u in words like honour is what makes it pronounced like "onner" rather than "hone-ore", same with colour being "culler" rather than "coal-ore". Sorry but it really annoys me when people consider the English version of English to be the weird one, as the clue is literally in the name as to which version is correct. This is why it still galls me that the language in macOS is "British English" yet the other one is just "English" rather than "American English" or whatever, it's a hideous double standard.

That aside, I'm a strong +1 for the feature :wink:
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Max Desiatov) #19

I also agree with the point that .lock extension better suits here and adheres to a convention already established in other languages. I personally would prefer the file to be named Package.lock, not Package.pins and also think that it would help newcomers who already used other package managers, especially CocoaPods.

With best regards, Max.

···

On 14 Oct 2016, at 17:43, Alexis via swift-evolution <swift-evolution@swift.org> wrote:

On Oct 14, 2016, at 2:01 AM, Ankit Aggarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

I’ve never seen this cause any actual confusion, nor has anyone I know who teaches/develops these sorts of tools. As far as I can tell, the broader programming community is rapidly converging on this as standard terminology:

* Gemfile.lock (Ruby)
* Cargo.lock (Rust)
* Composer.lock (PHP)
* yarn.lock (JS)
* pubspec.lock (Dart)
* Podfile.lock (Swift/Objc!)

Diverging from this seems counter-productive.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

I agree with the others that this is the better solution.

With regards to the constraining problem, the key insight adopted by Cargo/Yarn/Bundler is to distinguish libraries from applications. A library shouldn’t pin its dependencies, while an application should. This ensures that the ecosystem itself is maximally unconstrained, while ensuring actual applications continue to reliably build, regardless of ecosystem changes and the computer that it was built on. If a version of a library has trouble building with different versions, it should ideally specify that with its dependency constraints, not a lockfile.

This also ensures that there’s diverse testing of versions: CI for applications will verify particular configurations, while CI for libraries will verify the latest-and-greatest works.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(orta) #20

Afraid it doesn’t convince me. Even if you have an index that has strict semver adherence, the idea that you can trust people / machines to actually understand whether something will break other people's build seems unreasonable. Updates to code ships bugs. Updates you don’t expect gives you bugs you didn’t expect.

Alexis hits it on the head with the differentiation between a library and an app - libraries should define their own supported ranges, and they require far less locking than applications. An application should always have a lock so you have determinate builds and can update dependencies on purpose.

We generally agree that when you are simply the consumer of packages, pinning makes sense.

Isn’t this going to be the majority of the behaviour for the tool on the long run? So I would expect that to be where locking happens.

Sidenote: FWIW not a fan of the name change from lockfile - it feels arbitrary, but I figured the change to a separate command was a bigger elephant in the room.

···

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
On 14 Oct 2016, at 17:43, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

Can you check my reply to Eloy and see how it weighs with you?

- Daniel

On Oct 14, 2016, at 9:33 AM, Max Desiatov via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I also strongly agree with this, I'd prefer version pinning to happen by default, rather than with explicit command as it will make builds reproducible by default.

I totally agree that we can rely on past experience with other package managers (npm being the case), where pinning with a separate command caused more harm than good.

Overall, I think that it would be great if Package.lock was created by default when it's not present and be updated only with an explicit command for updating.

With best regards, Max.

On 14 Oct 2016, at 08:29, orta therox via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Please don’t make this a separate command, it should ideally be created at the end of an build (when there isn’t one already) or an update of your dependencies - most people will be expecting to get the same set of dependencies as the rest of their team. This pattern makes that harder.

NPM shrinkwrap is an example of this, and it’s a bad one - I’ve wasted a lot of time trying to keep that up to date for our npm projects. Facebook made a replacement for NPM with mainly the feature of “always locking” in yarn <https://yarnpkg.com/> and I’d expect that to take a lot of the JS mindshare on this one feature alone.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/>CocoaPods <http://cocoapods.org/> / CocoaDocs <http://cocoadocs.org/> / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12>
@orta <http://twitter.com/orta> / orta.github.com <http://orta.github.com/>
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer> ATM

On 14 Oct 2016, at 07:01, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi,

We're proposing version pinning feature in Swift Package Manager. The proposal is available here <https://github.com/aciidb0mb3r/swift-evolution/blob/version-pinning/proposals/NNNN-Version-Pinning.md> and also in this email:

Feedback welcomed!

Thanks,
Ankit

--------

Package Manager Version Pinning
Proposal: SE-XXXX
Author: Daniel Dunbar <https://github.com/ddunbar>, Ankit Aggarwal <https://github.com/aciidb0mb3r>
Review Manager: TBD
Status: Discussion
Introduction
This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions.

Motivation
As used in this proposal, version pinning refers to the practice of controlling exactly which specific version of a dependency is selected by the dependency resolution algorithm, independent from the semantic versioning specification. Thus, it is a way of instructing the package manager to select a particular version from among all of the versions of a package which could be chosen while honoring the dependency constraints.

Terminology

We have chosen to use "pinning" to refer to this feature, over "lockfiles", since the term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.

Philosophy

Our philosophy with regard to pinning is that we actively want to encourage packages to develop against the latest semantically appropriate versions of their dependencies, in order to foster rapid development amongst the ecosystem and strong reliance on the semantic versioning concept. Our design for version pinning is thus intended to be a feature for package authors and users to use in crafting specific workflows, not be a mechanism by which most of the packages in the ecosystem pin themselves to specific versions of each other.

Use Cases

Our proposal is designed to satisfy several different use cases for such a behavior:

Standardizing team workflows

When collaborating on a package, it can be valuable for team members (and continuous integration) to all know they are using the same exact version of dependencies, to avoid "works for me" situations.

This can be particularly important for certain kinds of open source projects which are actively being cloned by new users, and which want to have some measure of control around exactly which available version of a dependency is selected.

Difficult to test packages or dependencies

Complex packages which have dependencies which may be hard to test, or hard to analyze when they break, may choose to maintain careful control over what versions of their upstream dependencies they recommend -- even if conceptually they regularly update those recommendations following the true semantic version specification of the dependency.

Dependency locking w.r.t. deployment

When stabilizing a release for deployment, or building a version of a package for deployment, it is important to be able to lock down the exact versions of dependencies in use, so that the resulting product can be exactly recreated later if necessary.

Proposed solution
We will introduce support for an optional new file Package.pins adjacent to the Package.swift manifest, called the "pins file". We will also introduce a number of new commands (see below) for maintaining the pins file.

This file will record the active version pin information for the package, including data such as the package identifier, the pinned version, and explicit information on the pinned version (e.g., the commit hash/SHA for the resolved tag).

The exact file format is unspecified/implementation defined, however, in practice it will be a JSON data file.

This file may be checked into SCM by the user, so that its effects apply to all users of the package. However, it may also be maintained only locally (e.g., placed in the .gitignore file). We intend to leave it to package authors to decide which use case is best for their project.

In the presence of a Package.pins file, the package manager will respect the pinned dependencies recorded in the file whenever it needs to do dependency resolution (e.g., on the initial checkout or when updating).

The pins file will not override Manifest specified version requirements and it will be an error (with proper diagnostics) if there is a conflict between the pins and the manifest specification.

Detailed Design
We will add a new command pin to swift package tool with following semantics:

$ swift package pin ( [--all] | [<package-name>] [<version>] ) [--message <message>]
The package-name refers to the name of the package as specified in its manifest.

This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with --all) the behaviour is to pin to the current package version in use. Examples:

$ swift package pin --all - pins all the dependencies.
$ swift package pin Foo - pins Foo at current resolved version.
$ swift package pin Foo 1.2.3 - pins Foo at 1.2.3. The specified version should be valid and resolvable.
The --reason option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example:

$ swift package pin Foo --reason "The patch updates for Foo are really unstable and need screening."
Dependencies are never automatically pinned, pinning is only ever taken as a result of an explicit user action.

We will add a new command unpin:

$ swift package unpin ( [--all] | [<package-name>] )
This is the counterpart to the pin command, and unpins one or all packages.

We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet.

We will extend the workflow for update to honour version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument --repin:

$ swift package update [--repin]
Update command errors if there are no unpinned packages which can be updated.

Otherwise, the behaviour is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins.

The [--repin] argument can be used to lift the version pinning restrictions. In this case, the behaviour is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions.

The update and checkout will both emit logs, notifying the user that pinning is in effect.

The swift package show-dependencies subcommand will be updated to indicate if a dependency is pinned.

As a future extension, we anticipate using the SHA information recorded in a pins file as a security feature, to prevent man-in-the-middle attacks on parts of the package graph.

Impact on existing code
There will be change in the behaviours of swift build and swift package update in presence of the pins file, as noted in the proposal however the existing package will continue to build without any modifications.

Alternative considered
We considered making the pinning behavior default on running swift build, however we think that pinning by default is likely to make the package graph more constrained than it should be. It drives the user away from taking full advantage of semantic versioning. We think it will be good for the package ecosystem if such a restriction is not the default behavior and that this design will lead to faster discovery of bugs and fixes in the upstream.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev