[Draft] Package Manager Revised Dependency Resolution


(Rick Ballard) #1

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.


(Paul Cantrell) #2

Sounds familiar…

    https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029034.html

…so obviously I’m all for it!

P

···

On Apr 26, 2017, at 7:25 PM, Rick Ballard via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(David Hart) #3

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

···

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution
Proposal: SE-NNNN
Author: Rick Ballard
Review Manager: TBD
Status: Draft in progress
Bug: TBD
Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

Motivation

When SE-0145 Package Manager Version Pinning was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Martin Waitz) #4

Hello Rick,

thanks for the great proposal!
Strong +1 from me :slight_smile:

— Martin

···

Am 27.04.2017 um 02:25 schrieb Rick Ballard via swift-evolution <swift-evolution@swift.org>:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Felipe Cypriano) #5

This new proposal is great, I'm all in.

On Wed, Apr 26, 2017, at 17:25, Rick Ballard via swift-evolution wrote:> Hi all,

We have a proposal we'd like feedback on to revise how Swift Package
Manager dependency resolution, updating, and pinning works. These
changes weren't planned in the roadmap we published a few months ago,
but it's become clear since then that we have some holes in our
dependency resolution behavior that need fixing. We've also come to
the conclusion that our original design for pinning was
overcomplicated and should be revised.>
Please give us your feedback; we're hoping to submit this proposal for
official review next week.>
The current draft of the proposal can be found at
https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6.
I'm also including it below.>
Thanks,

- Rick

Package Manager Revised Dependency Resolution

* Proposal: SE-NNNN[1]
* Author: Rick Ballard[2]
* Review Manager: TBD
* Status: Draft in progress
* Bug: TBD> Introduction

This proposal makes the package manager's dependency resolution
behavior clearer and more intuitive. It removes the pinning commands
(swift package pin & swift package unpin), replaces the swift package
fetch command with a new swift package resolve command with improved
behavior, and replaces the optional Package.pins file with a
Package.resolved file which is always created during dependency
resolution.> Motivation

When SE-0145 Package Manager Version Pinning[3] was proposed, it was
observed that the proposal was overly complex. In particular, it
introduced a configuration option allowing some packages to have
autopinning on (the default), while others turned it off; this option
affected the behavior of other commands (like swift package update,
which has a --repinflag that does nothing for packages that use
autopinning). This configuration option has proved to be unnecessarily
confusing.> In the existing design, when autopinning is on (which is true by
default) the swift package pin command can't be used to pin packages
at specific revisions while allowing other packages to be updated. In
particular, if you edit your package's version requirements in the
Package.swift manifest, there is no way to resolve your package graph
to conform to those new requirements without automatically repinning
all packages to the latest allowable versions. Thus, specific,
intentional pins can not be preserved without turning off autopinning.> The problems here stem from trying to use one mechanism (pinning) to
solve two different use cases: wanting to record and share resolved
dependency versions, vs wanting to keep a badly-behaved package at a
specific version. We think the package manager could be simplified by
splitting these two use cases out into different mechanisms ("resolved
versions" vs "pinning"), instead of using an "autopinning" option
which makes these two features mutually-exclusive and confusing.> Additionally, some dependency resolution behaviors were not well-
specified and do not behave well. The package manager is lax about
detecting changes to the versions specified in the Package.swift
manifest or Package.pinspinfile, and fails to automatically update
packages when needed, or to issue errors if the version requirements
are unsatisfiable, until the user explicitly runs swift package
update, or until a new user without an existing checkout attempts to
build. We'd like to clarify and revise the rules around when and how
the package manager performs dependency resolution.> Proposed solution

The pinning feature will be removed. This removes the swift package
pin and swift package unpin commands, the --repin flag to swift
package update, and use of the Package.pins file.> In a future version of the package manager we may re-introduce
pinning. If we do, pins will only be recorded in the Package.pins file
when explicitly set with swift package pin, and any pinned
dependencies will *not* be updated by the swift package update
command; instead, they would need to be unpinned to be updated. This
would be a purely additive feature which packages could use in
addition to the resolved versions feature when desired.> A new "resolved versions" feature will be added, which behaves very
similarly to how pinning previously behaved when autopinning was on.
The version of every resolved dependency will be recorded in a
Package.resolved file in the top-level package, and when this file is
present in the top-level package it will be used when performing
dependency resolution, rather than the package manager finding the
latest eligible version of each package. swift package updatewill
update all dependencies to the latest eligible versions and update the
Package.resolved file accordingly.> Resolved versions will always be recorded by the package manager. Some
users may chose to add the Package.resolved file to their package's
.gitignore file. When this file is checked in, it allows a team to
coordinate on what versions of the dependencies they should use. If
this file is gitignored, each user will seperately choose when to get
new versions based on when they run the swift package update command,
and new users will start with the latest eligible version of each
dependency. Either way, for a package which is a dependency of other
packages (e.g. a library package), that package's Package.resolved
file will not have any effect on its client packages.> The existing swift package fetch command will be deprecated, removed
from the help message, and removed completely in a future release of
the Package Manager. In its place, a new swift package resolve command
will be added. The behavior of resolve will be to resolve
dependencies, taking into account the current version restrictions in
the Package.swift manifest and Package.resolved resolved versions
file, and issuing an error if the graph cannot be resolved. For
packages which have previously resolved versions recorded in the
Package.resolved file, the resolvecommand will resolve to those
versions as long as they are still eligible. If the resolved versions
file changes (e.g. because a teammate pushed a new version of the
file) the next resolve command will update packages to match that
file. After a successful resolve command, the checked out versions of
all dependencies and the versions recorded in the resolved versions
file will match. In most cases the resolve command will perform no
changes unless the Package.swiftmanifest or Package.resolved file have
changed.> The following commands will implicitly invoke the swift package
resolve functionality before running, and will cancel with an error if
dependencies cannot be resolved:

* swift build
* swift test
* swift package generate-xcodeproj> The swift package show-dependencies command will also implicitly
invoke swift package resolve, but it will show whatever information
about the dependency graph is available even if the resolve fails.> The swift package edit command will implicitly invoke swift package
resolve, but if the resolve fails yet did identify and fetch a package
with the package name the command supplied, the command will allow
that package to be edited anyway. This is useful if you wish to use
the edit command to edit version requirements and fix an unresolvable
dependency graph. swift package unedit will unedit the package and
*then* perform a resolve.> Detailed design

The resolve command is allowed to automatically add new dependencies
to the resolved versions file, and to remove dependencies which are no
longer in the dependency graph. It can also automatically update the
recorded versions of any package whose previously-resolved version is
no longer allowed by the version requirements from the
Package.swiftmanifests. When changed version requirements force a
dependency to be automatically re-resolved, the latest eligible
version will be chosen; any other dependencies affected by that change
will prefer to remain at their previously-resolved versions as long as
those versions are eligible, and will otherwise update likewise.> The Package.resolved resolved versions file will record the git
revision used for each resolved dependency in addition to its version.
In future versions of the package manager we may use this information
to detect when a previously-resolved version of a package resolves to
a new revision, and warn the user if this happens.> The swift package resolve command will not actually perform a git
fetch on any dependencies unless it needs to in order to correctly
resolve dependencies. As such, if all dependencies are already
resolved correctly and allowed by the version constraints in the
Package.swift manifest and Package.resolved resolved versions file,
the resolvecommand will not need to do anything (e.g. a normal swift
build won't hit the network or make unnecessary changes during its
implicit resolve).> If a dependency is in edit mode, it is allowed to have a different
version checked out than that recorded in the resolved versions file.
The version recorded for an edited package will not change
automatically. If a swift package updateoperation is performed while
any packages are in edit mode, the versions of those edited packages
will be removed from the resolved versions file, so that when those
packages leave edit mode the next resolution will record a new version
for them. Any packages in the dependency tree underneath an edited
package will also have their resolved version removed by swift package
update, as otherwise the resolved versions file might record versions
that wouldn't have been chosen without whatever edited package
modifications have been made.> Alternatives considered

We considered repurposing the existing fetch command for this new
behavior, instead of renaming the command to resolve. However, the
name fetch is defined by git to mean getting the latest content for a
repository over the network. Since this package manager command does
not always actually fetch new content from the network, it is
confusing to use the name fetch. In the future, we may offer
additional control over when dependency resolution is allowed to
perform network access, and we will likely use the word fetch in flag
names that control that behavior.> We considered continuing to write out the Package.pins file for
packages whose Swift tools version[4] was less than 4.0, for maximal
compatibility with the Swift 3.1 tools. However, as the old pinning
behavior was a workflow feature and not a fundamental piece of
package compatibility, we do not consider it necessary to support in
the 4.0 tools.> We considered keeping the pin and unpin commands, with the new
behavior as discussed briefly in this proposal. While we think we may
wish to bring this feature back in the future, we do not consider it
critical for this release; the workflow it supports (updating all
packages except a handful which have been pinned) is not something
most users will need, and there are workarounds (e.g. specify an
explicit dependency in the Package.swift manifest).> Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved
versions file, to be consistent with many other package managers. We
expect that the decision not to use this extension will be
controversial, as following established precedent is valuable.
However, we think that a "lockfile" is a very poor name for this
concept, and that using that name would cause confusion when we re-
introduce pins. Specifically:

* Calling this a "lock" implies a stronger lockdown of dependencies
   than is supported by the actual behavior. As a simple update
   command will reset the locks, and a change to the specified
   versions in Package.swift will override them, they're not really
   "locked" at all. This is misleading.
* When we re-introduce pinning, it would be very confusing to have
   both "locks" and "pins". Having "resolved versions" and "pins" is
   not so confusing.
* The term "lock" is already overloaded between POSIX file locks and
   locks in concurrent programming.> For comparison, here is a list of other package managers which
implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name Yarn JS yarn.lock
Composer PHP composer.lock Cargo Rust Cargo.lock Bundler Ruby
Gemfile.lock CocoaPods ObjC/Swift Podfile.lock Glide Go glide.lock Pub
Dart pubspec.lock Mix Elixir mix.lock rebar3 Erlang rebar.lock Carton
Perl carton.lock Carthage ObjC/Swift Cartfile.resolved Pip Python
requirements.txt NPM JS npm-shrinkwrap.json Meteor JS versions> Some arguments for using ".lock" instead of ".resolved" are:

* Users of other package managers will already be familiar with the
   terminology and behavior.
* For packages which support multiple package managers, it will be
   possible to put "*.lock" into the gitignore file instead of needing
   a seperate entry for "*.resolved".> However, we do not feel that these arguments outweigh the problems
with the term "lock". If providing feedback asking that we reconsider
this decision, please be clear about why the above decision is
incorrect, with new information not already considered.>
_________________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Links:

  1. https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md
  2. https://github.com/rballard
  3. https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md
  4. https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md


(Rick Ballard) #6

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

···

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jon Shier) #7

`install` only sounds like it should install things in the system if that’s the only type of manager you’ve ever used. If I’ve only ever used brew, of course I’ll assume that every other thing that calls itself a package manager will operate similarly. Thankfully, people learn quickly and it’s pretty easy to tell the difference between project package managers and system package mangers. Pretty much anyone developing on Apple platforms will be familiar with brew and CocoaPods / bundler, all of which use the `install` verb, so following those would be an easy way to gain immediate familiarity. SPM really shouldn't be too different given that it will have to interoperate on systems with these tools, so choosing your own verbs would be far more confusing than using `install`. Carthage’s use of `bootstrap` is weird outlier that is always confusing to me. Plus, using `install`, allows you to use `.installed` for the file, which I think makes more sense than `.lock` or `.resolved`.
  A second issue is that `resolve` doesn’t sound like it installs anything at all. It sounds like a command that would just print the resolved versions of all my dependencies but do nothing else.

Jon

···

On Apr 30, 2017, at 12:06 AM, Rick Ballard via swift-evolution <swift-evolution@swift.org> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(David Hart) #8

Hi Rick,

Thanks for taking the time to reply. I discussed the same argument with Ankit Aggarwal, and here are my counter-arguments:

It seems to me like the survey should take into account the developer population using each tool. If we are doing such a survey to see what tool a programmer coming to Swift is coming from, there is much less chance he has previous knowledge of rebar3 than of npm. The majority of the ones which are widely used seem to use install: I’ve updated the survey bellow by ordering them by stars on GitHub. It’s a crude approximation of developer population, but I think it’s good enough for us.
If we do end up implementing install to mean install to the system, how would we update the system packages?
Solution 1 would be to add a —system/—global flag on update. In that case, that would bring inconsistency by using two different commands to “install/resolve” and one command but different flags for updating.
Solution 2 would be to introduce a different command to update, for example upgrade. In that case, we get into the scenario of Homebrew, where two synonymous commands (update and upgrade) are used to do two different things (I have a hard time remembering which is which).
As can be seen in the table bellow, all package managers which offer both an installation for the local project and for the system always use the same commands names for both. I would vote for a system where:

install [package]: resolves and fetches one or all packages locally in the project
install —system package: fetches and installs a specific package in the system
update [package]: resolves and updates one or all packages locally in the project
update —system package: updates a specific package in the system
Here is the same survey you did, but with more information:

Package Manager GitHub Stars Language Resolved File Name Local Install Command Local Update Command System Install Command System Update Command
Yarn 24711 JS yarn.lock install upgrade global install global upgrade
NPM 12214 JS npm-shrinkwrap.json install update install -g update -g
CocoaPods 9328 ObjC/Swift Podfile.lock install update N/A N/A
Carthage 9320 ObjC/Swift Cartfile.resolved bootstrap update N/A N/A
Composer 9313 PHP composer.lock install update N/A N/A
Glide 4349 Go glide.lock install update N/A N/A
Bundler 3771 Ruby Gemfile.lock install update install(with gem) update(with gem)
Pip 3555 Python requirements.txt N/A N/A install install --upgrade
Cargo 2206 Rust Cargo.lock N/A update N/A N/A
rebar3 622 Erlang rebar.lock get-deps upgrade N/A N/A
Carton 393 Perl carton.lock install update N/A N/A
Pub 40 Dart pubspec.lock get upgrade N/A N/A
Meteor N/A (1) JS versions add update N/A N/A
Mix N/A (2) Elixir mix.lock deps.get deps.update N/A N/A
We only have the number of GitHub stars for the the whole Meteor project (37255) - so are not easily comparable with the stars for other package manager projects in the list.
We only have the number of GitHub stars for the the whole Elixir project (9928) - so are not easily comparable with the stars for other package manager projects in the list.

···

On 30 Apr 2017, at 06:06, Rick Ballard <rballard@apple.com> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(David Hart) #9

  `install` only sounds like it should install things in the system if that’s the only type of manager you’ve ever used. If I’ve only ever used brew, of course I’ll assume that every other thing that calls itself a package manager will operate similarly. Thankfully, people learn quickly and it’s pretty easy to tell the difference between project package managers and system package mangers. Pretty much anyone developing on Apple platforms will be familiar with brew and CocoaPods / bundler, all of which use the `install` verb, so following those would be an easy way to gain immediate familiarity. SPM really shouldn't be too different given that it will have to interoperate on systems with these tools, so choosing your own verbs would be far more confusing than using `install`. Carthage’s use of `bootstrap` is weird outlier that is always confusing to me. Plus, using `install`, allows you to use `.installed` for the file, which I think makes more sense than `.lock` or `.resolved`.

Yeah, I’ve always had a hard time remembering the bootstrap command specifically because it chose to be different than the norm.

···

On 1 May 2017, at 00:46, Jon Shier <jon@jonshier.com> wrote:

  A second issue is that `resolve` doesn’t sound like it installs anything at all. It sounds like a command that would just print the resolved versions of all my dependencies but do nothing else.

Jon

On Apr 30, 2017, at 12:06 AM, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rick Ballard) #10

Thank you David and Jon for your feedback, time, and attention on this; I very much appreciate it.

If/when we do implement an `install` verb, it's actually likely _not_ going to do the same thing as `resolve` but in a system location. Instead, any `install` we implement will likely install the built products of a package. That's a pretty substantial difference vs the other package managers which use `install`. It also means that we likely won't need an update/upgrade command for installed products at all.

With the exception of CocoaPods, it appears that the other package managers that use `install` for what I'm calling "resolve" are package managers for non-compiled languages. There's no real difference between a source package and a library product for them; sources can be pulled down and then used by other clients directly. However, in Swift, you need to actually build your dependencies before they become available. [Hat tip to Martin Waitz, who just brought this up as well].

Additionally, I don't think we have any aspirations of making it easy to install even built dependencies into some global location that would get automatically searched by every other package. Instead, we want to have "hermetic builds", where the entire dependency graph is explicit and self-contained (with the exception of dependencies provided by your OS, like system frameworks). So our `install` would really serve quite a different purpose than e.g. yarn's `global install`.

FWIW, I don't think most beginning users will ever even have to know about or use the `resolve` verb; you can do everything you need to with `swift build`, `swift update`, etc which implicitly run `resolve` for you. The more important parts of this proposal, in my opinion, are the behavior of `resolve` (which will usually be implicit), and the "Package.resolved" file. I agree with Jon that the name `resolve` isn't super clear re: the fact that it actually fetches the dependencies that it resolves when needed, but since invoking this verb directly is already an advanced-user feature I'm not too concerned about that – and I'd like it to match the name of the "Package.resolved" file, which is the most accurate and descriptive name we've come up with for that file.

Thanks,

  - Rick

···

On Apr 30, 2017, at 10:50 AM, David Hart <david@hartbit.com> wrote:

Hi Rick,

Thanks for taking the time to reply. I discussed the same argument with Ankit Aggarwal, and here are my counter-arguments:

It seems to me like the survey should take into account the developer population using each tool. If we are doing such a survey to see what tool a programmer coming to Swift is coming from, there is much less chance he has previous knowledge of rebar3 than of npm. The majority of the ones which are widely used seem to use install: I’ve updated the survey bellow by ordering them by stars on GitHub. It’s a crude approximation of developer population, but I think it’s good enough for us.
If we do end up implementing install to mean install to the system, how would we update the system packages?
Solution 1 would be to add a —system/—global flag on update. In that case, that would bring inconsistency by using two different commands to “install/resolve” and one command but different flags for updating.
Solution 2 would be to introduce a different command to update, for example upgrade. In that case, we get into the scenario of Homebrew, where two synonymous commands (update and upgrade) are used to do two different things (I have a hard time remembering which is which).
As can be seen in the table bellow, all package managers which offer both an installation for the local project and for the system always use the same commands names for both. I would vote for a system where:

install [package]: resolves and fetches one or all packages locally in the project
install —system package: fetches and installs a specific package in the system
update [package]: resolves and updates one or all packages locally in the project
update —system package: updates a specific package in the system
Here is the same survey you did, but with more information:

Package Manager GitHub Stars Language Resolved File Name Local Install Command Local Update Command System Install Command System Update Command
Yarn 24711 JS yarn.lock install upgrade global install global upgrade
NPM 12214 JS npm-shrinkwrap.json install update install -g update -g
CocoaPods 9328 ObjC/Swift Podfile.lock install update N/A N/A
Carthage 9320 ObjC/Swift Cartfile.resolved bootstrap update N/A N/A
Composer 9313 PHP composer.lock install update N/A N/A
Glide 4349 Go glide.lock install update N/A N/A
Bundler 3771 Ruby Gemfile.lock install update install(with gem) update(with gem)
Pip 3555 Python requirements.txt N/A N/A install install --upgrade
Cargo 2206 Rust Cargo.lock N/A update N/A N/A
rebar3 622 Erlang rebar.lock get-deps upgrade N/A N/A
Carton 393 Perl carton.lock install update N/A N/A
Pub 40 Dart pubspec.lock get upgrade N/A N/A
Meteor N/A (1) JS versions add update N/A N/A
Mix N/A (2) Elixir mix.lock deps.get deps.update N/A N/A
We only have the number of GitHub stars for the the whole Meteor project (37255) - so are not easily comparable with the stars for other package manager projects in the list.
We only have the number of GitHub stars for the the whole Elixir project (9928) - so are not easily comparable with the stars for other package manager projects in the list.

On 30 Apr 2017, at 06:06, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Martin Waitz) #11

Hello,

Many of the listed package managers are for interpreted languages.
So after fetching all dependencies, your package is completely usable. It is „installed locally“.
But Swift packages have to be compiled. You have to build them to be able to use them.
For me, ‚install' comes after compilation, not before.

Using ‚resolve‘ is thus a much better name for the proposed command.
The verb ‚install‘ should only be used when it really installs the package in some way.

···


Martin

Am 01.05.2017 um 00:46 schrieb Jon Shier via swift-evolution <swift-evolution@swift.org>:

  `install` only sounds like it should install things in the system if that’s the only type of manager you’ve ever used. If I’ve only ever used brew, of course I’ll assume that every other thing that calls itself a package manager will operate similarly. Thankfully, people learn quickly and it’s pretty easy to tell the difference between project package managers and system package mangers. Pretty much anyone developing on Apple platforms will be familiar with brew and CocoaPods / bundler, all of which use the `install` verb, so following those would be an easy way to gain immediate familiarity. SPM really shouldn't be too different given that it will have to interoperate on systems with these tools, so choosing your own verbs would be far more confusing than using `install`. Carthage’s use of `bootstrap` is weird outlier that is always confusing to me. Plus, using `install`, allows you to use `.installed` for the file, which I think makes more sense than `.lock` or `.resolved`.
  A second issue is that `resolve` doesn’t sound like it installs anything at all. It sounds like a command that would just print the resolved versions of all my dependencies but do nothing else.

Jon

On Apr 30, 2017, at 12:06 AM, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick


(Rien) #12

+ 1

Hello,

Many of the listed package managers are for interpreted languages.
So after fetching all dependencies, your package is completely usable. It is „installed locally“.
But Swift packages have to be compiled. You have to build them to be able to use them.
For me, ‚install' comes after compilation, not before.

Using ‚resolve‘ is thus a much better name for the proposed command.
The verb ‚install‘ should only be used when it really installs the package in some way.

Absolutely. I struggle to see any value in “install” at all. “Resolve” feels much more natural.

Regards,
Rien

Site: http://balancingrock.nl
Blog: http://swiftrien.blogspot.com
Github: http://github.com/Balancingrock
Project: http://swiftfire.nl - A server for websites build in Swift

···

On 02 May 2017, at 00:00, Martin Waitz via swift-evolution <swift-evolution@swift.org> wrote:


Martin

Am 01.05.2017 um 00:46 schrieb Jon Shier via swift-evolution <swift-evolution@swift.org>:

  `install` only sounds like it should install things in the system if that’s the only type of manager you’ve ever used. If I’ve only ever used brew, of course I’ll assume that every other thing that calls itself a package manager will operate similarly. Thankfully, people learn quickly and it’s pretty easy to tell the difference between project package managers and system package mangers. Pretty much anyone developing on Apple platforms will be familiar with brew and CocoaPods / bundler, all of which use the `install` verb, so following those would be an easy way to gain immediate familiarity. SPM really shouldn't be too different given that it will have to interoperate on systems with these tools, so choosing your own verbs would be far more confusing than using `install`. Carthage’s use of `bootstrap` is weird outlier that is always confusing to me. Plus, using `install`, allows you to use `.installed` for the file, which I think makes more sense than `.lock` or `.resolved`.
  A second issue is that `resolve` doesn’t sound like it installs anything at all. It sounds like a command that would just print the resolved versions of all my dependencies but do nothing else.

Jon

On Apr 30, 2017, at 12:06 AM, Rick Ballard via swift-evolution <swift-evolution@swift.org> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jon Shier) #13

I suppose much of my confusion in naming comes from SPM’s (apparent) desire to be both a package manager (like CocoaPods) and builder (like make, locally). I also have to admit I’m not too familiar with SPM, aside from supporting Alamofire’s integration with it. I’m also an iOS app developer, not Linux command line tool developer, so my expectations for SPM are somewhat different. Additionally, SPM’s integration with Xcode is unknown at this time, so I have little context in with to place my requirements and how they could be satisfied with such a solution. My use cases usually align with bundler / CocoaPods’: how do I install the locked versions of my dependencies and how do I update all or individual dependencies, locking their version in the .lock file. Generally, I suppose that means that I’d expect dependency tools to be able integrate with other build systems, so SPM is a bit strange in that regard. Also, the asymmetry of swift update to update my dependencies and swift package resolve to get the currently locked versions are a bit strange, since those commands are essentially equally important if using a separate build system (install / resolve is also run in CI for every build), so that threw me for a bit of a loop.
  
Jon Shier

···

On May 1, 2017, at 6:18 PM, Rick Ballard <rballard@apple.com> wrote:

Thank you David and Jon for your feedback, time, and attention on this; I very much appreciate it.

If/when we do implement an `install` verb, it's actually likely _not_ going to do the same thing as `resolve` but in a system location. Instead, any `install` we implement will likely install the built products of a package. That's a pretty substantial difference vs the other package managers which use `install`. It also means that we likely won't need an update/upgrade command for installed products at all.

With the exception of CocoaPods, it appears that the other package managers that use `install` for what I'm calling "resolve" are package managers for non-compiled languages. There's no real difference between a source package and a library product for them; sources can be pulled down and then used by other clients directly. However, in Swift, you need to actually build your dependencies before they become available. [Hat tip to Martin Waitz, who just brought this up as well].

Additionally, I don't think we have any aspirations of making it easy to install even built dependencies into some global location that would get automatically searched by every other package. Instead, we want to have "hermetic builds", where the entire dependency graph is explicit and self-contained (with the exception of dependencies provided by your OS, like system frameworks). So our `install` would really serve quite a different purpose than e.g. yarn's `global install`.

FWIW, I don't think most beginning users will ever even have to know about or use the `resolve` verb; you can do everything you need to with `swift build`, `swift update`, etc which implicitly run `resolve` for you. The more important parts of this proposal, in my opinion, are the behavior of `resolve` (which will usually be implicit), and the "Package.resolved" file. I agree with Jon that the name `resolve` isn't super clear re: the fact that it actually fetches the dependencies that it resolves when needed, but since invoking this verb directly is already an advanced-user feature I'm not too concerned about that – and I'd like it to match the name of the "Package.resolved" file, which is the most accurate and descriptive name we've come up with for that file.

Thanks,

  - Rick

On Apr 30, 2017, at 10:50 AM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Hi Rick,

Thanks for taking the time to reply. I discussed the same argument with Ankit Aggarwal, and here are my counter-arguments:

It seems to me like the survey should take into account the developer population using each tool. If we are doing such a survey to see what tool a programmer coming to Swift is coming from, there is much less chance he has previous knowledge of rebar3 than of npm. The majority of the ones which are widely used seem to use install: I’ve updated the survey bellow by ordering them by stars on GitHub. It’s a crude approximation of developer population, but I think it’s good enough for us.
If we do end up implementing install to mean install to the system, how would we update the system packages?
Solution 1 would be to add a —system/—global flag on update. In that case, that would bring inconsistency by using two different commands to “install/resolve” and one command but different flags for updating.
Solution 2 would be to introduce a different command to update, for example upgrade. In that case, we get into the scenario of Homebrew, where two synonymous commands (update and upgrade) are used to do two different things (I have a hard time remembering which is which).
As can be seen in the table bellow, all package managers which offer both an installation for the local project and for the system always use the same commands names for both. I would vote for a system where:

install [package]: resolves and fetches one or all packages locally in the project
install —system package: fetches and installs a specific package in the system
update [package]: resolves and updates one or all packages locally in the project
update —system package: updates a specific package in the system
Here is the same survey you did, but with more information:

Package Manager GitHub Stars Language Resolved File Name Local Install Command Local Update Command System Install Command System Update Command
Yarn 24711 JS yarn.lock install upgrade global install global upgrade
NPM 12214 JS npm-shrinkwrap.json install update install -g update -g
CocoaPods 9328 ObjC/Swift Podfile.lock install update N/A N/A
Carthage 9320 ObjC/Swift Cartfile.resolved bootstrap update N/A N/A
Composer 9313 PHP composer.lock install update N/A N/A
Glide 4349 Go glide.lock install update N/A N/A
Bundler 3771 Ruby Gemfile.lock install update install(with gem) update(with gem)
Pip 3555 Python requirements.txt N/A N/A install install --upgrade
Cargo 2206 Rust Cargo.lock N/A update N/A N/A
rebar3 622 Erlang rebar.lock get-deps upgrade N/A N/A
Carton 393 Perl carton.lock install update N/A N/A
Pub 40 Dart pubspec.lock get upgrade N/A N/A
Meteor N/A (1) JS versions add update N/A N/A
Mix N/A (2) Elixir mix.lock deps.get deps.update N/A N/A
We only have the number of GitHub stars for the the whole Meteor project (37255) - so are not easily comparable with the stars for other package manager projects in the list.
We only have the number of GitHub stars for the the whole Elixir project (9928) - so are not easily comparable with the stars for other package manager projects in the list.

On 30 Apr 2017, at 06:06, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:

Thanks for the feedback, David, and apologies for the slow reply. My biggest reservation with the word "install" is that it really sounds like it should install things into the system, or another shareable location, instead of fetching dependencies into the dependency location for a single top-level package. We might actually want to add a real "install" command that does some type of installation some day in the future, though we might also choose to forever leave that sort of thing to your system package manager. And while there is some precedence for other package managers using "install" to fetch dependencies, I think there's broader precedence for it meaning to either install things into the system, or at least create an installable "build root", with e.g. `make install`, `xcodebuild install`, `brew install`, etc.

I just did a quick survey of the same package managers I surveyed previously to see if they all used this term, and here's what I found:

yarn: `install`
composer: `install`
cargo: No true equivalent, just supprts `update` or `build`. Also, uses `install` to mean something different – `cargo install` installs binary packages only, to a installation root.
bundler: `install`
cocoapods: `install`
glide: `install`
pub: `get`
mix: `deps.get`
rebar3: `get-deps`
carton: `install`
carthage: `bootstrap`
pip: Uses `install`, but since pip doesn't enforce the top-level-package seperation of SwiftPM, Pip's use of `install` is actually more accurate.
npm: `install`
meteor: no true equivalent, just supports `update` or `build`

Given that this isn't a universal term, I think I'm comfortable going against the flow a little bit if it means that we get to have clear and accurate command-line verbs. But I think this is worth adding to the "Alternatives Considered" section before we put this up for review!

  - Rick

On Apr 26, 2017, at 10:42 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

Very happy about this proposal as the pinning feature was un-necessarily complicated and because SwiftPM will now work like many other package managers out there: users won't be surprised.

By the way, why wasn't resolve called install instead, mirroring the terminology used everywhere else? It feels like the name was chosen specifically to denote that it's doing something different than other dependency managers out there. But that doesn't seem to be the case.

On 27 Apr 2017, at 02:25, Rick Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi all,

We have a proposal we'd like feedback on to revise how Swift Package Manager dependency resolution, updating, and pinning works. These changes weren't planned in the roadmap we published a few months ago, but it's become clear since then that we have some holes in our dependency resolution behavior that need fixing. We've also come to the conclusion that our original design for pinning was overcomplicated and should be revised.

Please give us your feedback; we're hoping to submit this proposal for official review next week.

The current draft of the proposal can be found at https://github.com/rballard/swift-evolution/commit/e5e7ce76f29c855aa7162ed3733b09e701d662d6. I'm also including it below.

Thanks,

  - Rick

Package Manager Revised Dependency Resolution

Proposal: SE-NNNN <https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md>
Author: Rick Ballard <https://github.com/rballard>
Review Manager: TBD
Status: Draft in progress
Bug: TBD
<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#introduction>Introduction

This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (swift package pin & swift package unpin), replaces the swift package fetch command with a new swift package resolve command with improved behavior, and replaces the optional Package.pins file with a Package.resolved file which is always created during dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#motivation>Motivation

When SE-0145 Package Manager Version Pinning <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md> was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like swift package update, which has a --repinflag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing.

In the existing design, when autopinning is on (which is true by default) the swift package pin command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the Package.swift manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning.

The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing.

Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the Package.swift manifest or Package.pinspinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs swift package update, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#proposed-solution>Proposed solution

The pinning feature will be removed. This removes the swift package pin and swift package unpin commands, the --repin flag to swift package update, and use of the Package.pins file.

In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the Package.pins file when explicitly set with swift package pin, and any pinned dependencies will not be updated by the swift package update command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired.

A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a Package.resolved file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. swift package updatewill update all dependencies to the latest eligible versions and update the Package.resolved file accordingly.

Resolved versions will always be recorded by the package manager. Some users may chose to add the Package.resolved file to their package's .gitignore file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will seperately choose when to get new versions based on when they run the swift package update command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's Package.resolved file will not have any effect on its client packages.

The existing swift package fetch command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new swift package resolve command will be added. The behavior of resolve will be to resolve dependencies, taking into account the current version restrictions in the Package.swift manifest and Package.resolved resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the Package.resolved file, the resolvecommand will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next resolve command will update packages to match that file. After a successful resolve command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the resolve command will perform no changes unless the Package.swiftmanifest or Package.resolved file have changed.

The following commands will implicitly invoke the swift package resolve functionality before running, and will cancel with an error if dependencies cannot be resolved:

swift build
swift test
swift package generate-xcodeproj
The swift package show-dependencies command will also implicitly invoke swift package resolve, but it will show whatever information about the dependency graph is available even if the resolve fails.

The swift package edit command will implicitly invoke swift package resolve, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the edit command to edit version requirements and fix an unresolvable dependency graph. swift package unedit will unedit the package and then perform a resolve.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#detailed-design>Detailed design

The resolve command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the Package.swiftmanifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise.

The Package.resolved resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens.

The swift package resolve command will not actually perform a git fetch on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the Package.swift manifest and Package.resolved resolved versions file, the resolvecommand will not need to do anything (e.g. a normal swift build won't hit the network or make unnecessary changes during its implicit resolve).

If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a swift package updateoperation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by swift package update, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made.

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#alternatives-considered>Alternatives considered

We considered repurposing the existing fetch command for this new behavior, instead of renaming the command to resolve. However, the name fetch is defined by git to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name fetch. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word fetch in flag names that control that behavior.

We considered continuing to write out the Package.pins file for packages whose Swift tools version <https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md> was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools.

We considered keeping the pin and unpin commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the Package.swift manifest).

<https://github.com/rballard/swift-evolution/blob/e5e7ce76f29c855aa7162ed3733b09e701d662d6/proposals/NNNN-package-manager-revised-dependency-resolution.md#why-we-didnt-use-packagelock>Why we didn't use "Package.lock"

We considered using the .lock file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically:

Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple update command will reset the locks, and a change to the specified versions in Package.swift will override them, they're not really "locked" at all. This is misleading.
When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing.
The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming.
For comparison, here is a list of other package managers which implement similar behavior and their name for this file:

Package Manager Language Resolved versions file name
Yarn JS yarn.lock
Composer PHP composer.lock
Cargo Rust Cargo.lock
Bundler Ruby Gemfile.lock
CocoaPods ObjC/Swift Podfile.lock
Glide Go glide.lock
Pub Dart pubspec.lock
Mix Elixir mix.lock
rebar3 Erlang rebar.lock
Carton Perl carton.lock
Carthage ObjC/Swift Cartfile.resolved
Pip Python requirements.txt
NPM JS npm-shrinkwrap.json
Meteor JS versions
Some arguments for using ".lock" instead of ".resolved" are:

Users of other package managers will already be familiar with the terminology and behavior.
For packages which support multiple package managers, it will be possible to put "*.lock" into the gitignore file instead of needing a seperate entry for "*.resolved".
However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution