Proposal Discussion Thread: SwiftPM: Locking and Overriding Dependencies

[Max_Howell]

The following is a draft proposal, feedback welcome.

···

____________
SwiftPM Dependency Version Locking
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell <https://github.com/mxcl&gt;
Status: Discussion
Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.

Swift-evolution thread <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology
A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.
A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build
Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.

Motivation
In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:

To ensure they are all using the same versions of their dependencies for any given version-control commit.
Ensure they are all using the same versions of the underlying Swift toolchain
Be able to override or modify dependency specifications for the whole team for specific commits.
Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.

Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.

Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:

The URL and versions of cloned dependencies
An inline diff of any local modifications made to those packages relative to their pristine cloned states
The exact version of the Swift toolchain used as part of the last successful build of the package
This file is generated by SwiftPM.

This file should be checked-in with projects.

This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.

This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.

Detailed Design
In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.

Packages/VersionLocks.json

The exact design of the contents of this file will be explored during iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot; "ref": "1.2.3" } ] }

Workflow — Regular Build

User runs swift build
If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.
If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *
If Packages/VersionLocks.json is present its dependency graph is used
If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.

Workflow — Making Modifications

User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.

Workflow — Overriding Packages

User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their own fork
swift build errors out.
User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for dependencies, only for projects.

If a package author requires an override they have a few options:

Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.
Advise end-users in a package README that they should override the dependency themselves.
2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Impact on existing code
This proposal will have no impact on existing code.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread it might not be the best idea.

Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.

[benrimmington]

<https://github.com/apple/swift-package-manager/blob/master/Documentation/Internals/SwiftBasedManifestFormat.md#discussion&gt;

We decided to use a Swift-based format for the manifest because we believe it gives developers the best experience for working with and describing their project. The primary alternative we considered was to use a declarative format encoded in a common data format like JSON. Although that would simplify implementation of the tooling around the manifest, it has the downside that users must then learn this additional language, and the development of high quality tools for that (documentation, syntax coloring, parsing diagnostics) isn't aligned with our goal of building great tools for Swift. In contrast, using the Swift language means that we can leverage all of the work on Swift to make those tools great.

Could you generate a similar Swift-based format for the lockfile?

-- Ben

···

On 17 Mar 2016, at 18:23, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

The following is a draft proposal, feedback welcome.

____________
SwiftPM Dependency Version Locking
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell <https://github.com/mxcl&gt;
Status: Discussion
Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.

Swift-evolution thread <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology
A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.
A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build
Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.

Motivation
In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:

To ensure they are all using the same versions of their dependencies for any given version-control commit.
Ensure they are all using the same versions of the underlying Swift toolchain
Be able to override or modify dependency specifications for the whole team for specific commits.
Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.

Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.

Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:

The URL and versions of cloned dependencies
An inline diff of any local modifications made to those packages relative to their pristine cloned states
The exact version of the Swift toolchain used as part of the last successful build of the package
This file is generated by SwiftPM.

This file should be checked-in with projects.

This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.

This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.

Detailed Design
In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.

Packages/VersionLocks.json

The exact design of the contents of this file will be explored during iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot; "ref": "1.2.3" } ] }

Workflow — Regular Build

User runs swift build
If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.
If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *
If Packages/VersionLocks.json is present its dependency graph is used
If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.

Workflow — Making Modifications

User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.

Workflow — Overriding Packages

User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their own fork
swift build errors out.
User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for dependencies, only for projects.

If a package author requires an override they have a few options:

Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.
Advise end-users in a package README that they should override the dependency themselves.
2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Impact on existing code
This proposal will have no impact on existing code.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread it might not be the best idea.

Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.

[ddunbar (Daniel Dunbar)]

My thoughts on this proposal:

1. I agree with some of the other comments that I would prefer the version file be adjacent to "Package.swift". When the Packages directory isn't being checked in, I really would like to think of it as an "implementation detail" and not embed functionality inside it that would make it hard to remove or change in the future.

2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.

3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).

4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.

5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?

6. I wonder if we should be defining, as Eloy alludes to, two different things:
- The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.
- The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.
Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.

- Daniel

···

On Mar 17, 2016, at 11:23 AM, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

The following is a draft proposal, feedback welcome.

____________
SwiftPM Dependency Version Locking
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell <https://github.com/mxcl&gt;
Status: Discussion
Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.

Swift-evolution thread <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology
A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.
A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build
Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.

Motivation
In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:

To ensure they are all using the same versions of their dependencies for any given version-control commit.
Ensure they are all using the same versions of the underlying Swift toolchain
Be able to override or modify dependency specifications for the whole team for specific commits.
Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.

Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.

Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:

The URL and versions of cloned dependencies
An inline diff of any local modifications made to those packages relative to their pristine cloned states
The exact version of the Swift toolchain used as part of the last successful build of the package
This file is generated by SwiftPM.

This file should be checked-in with projects.

This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.

This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.

Detailed Design
In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.

Packages/VersionLocks.json

The exact design of the contents of this file will be explored during iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot; "ref": "1.2.3" } ] }

Workflow — Regular Build

User runs swift build
If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.
If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *
If Packages/VersionLocks.json is present its dependency graph is used
If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.

Workflow — Making Modifications

User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.

Workflow — Overriding Packages

User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their own fork
swift build errors out.
User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for dependencies, only for projects.

If a package author requires an override they have a few options:

Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.
Advise end-users in a package README that they should override the dependency themselves.
2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Impact on existing code
This proposal will have no impact on existing code.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread it might not be the best idea.

Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Max_Howell]

I have revised the proposal based on feedback:

https://github.com/mxcl/swift-evolution/blob/lockfiles/proposals/NNNN-swiftpm-dependency-lockfiles.md

Changed location and name, included SHAs, revised build-workflow steps.

[Lukas_Stabe]

This is a very welcome addition. Thanks!

A nitpicky detail: I’d prefer the format for the lock file to be yaml or toml over json, since those read a little easier in git diffs (which is, in my experience with cocoapods lockfiles, the only place I ever interact with the lockfiles contents).

— Lukas

···

On 17 Mar 2016, at 19:23, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

The following is a draft proposal, feedback welcome.

____________
SwiftPM Dependency Version Locking
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell <https://github.com/mxcl&gt;
Status: Discussion
Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.

Swift-evolution thread <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology
A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.
A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build
Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.

Motivation
In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:

To ensure they are all using the same versions of their dependencies for any given version-control commit.
Ensure they are all using the same versions of the underlying Swift toolchain
Be able to override or modify dependency specifications for the whole team for specific commits.
Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.

Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.

Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:

The URL and versions of cloned dependencies
An inline diff of any local modifications made to those packages relative to their pristine cloned states
The exact version of the Swift toolchain used as part of the last successful build of the package
This file is generated by SwiftPM.

This file should be checked-in with projects.

This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.

This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.

Detailed Design
In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.

Packages/VersionLocks.json

The exact design of the contents of this file will be explored during iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot; "ref": "1.2.3" } ] }

Workflow — Regular Build

User runs swift build
If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.
If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *
If Packages/VersionLocks.json is present its dependency graph is used
If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.

Workflow — Making Modifications

User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.

Workflow — Overriding Packages

User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their own fork
swift build errors out.
User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for dependencies, only for projects.

If a package author requires an override they have a few options:

Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.
Advise end-users in a package README that they should override the dependency themselves.
2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Impact on existing code
This proposal will have no impact on existing code.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread it might not be the best idea.

Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Honza_Dvorsky]

Very happy this is coming! I really like the details of this proposal as
well.

Just one thing - having the `VersionLocks` in `Packages/VersionLocks` seems
like a bad idea to me. Couple of hurdles right away
- for people who don't want to check in their dependencies, we can't just
add `Packages` to our `.gitignore`, because that would make it impossible
to check in the `VersionLocks` file (users would need to add exceptions to
ignore Packages but keep the lockfile, which might result on users
accidentally not committing their lockfile at all, which would defeat its
whole purpose).
- `swift build --clean=dist` deletes the whole `Packages` directory,
removing the lockfile, while I can imagine just wanting to delete and
re-pull my dependencies with `swift build --clean=dist; swift build`. This
will potentially generate a new lockfile, even though there was a
completely valid one which I didn't explicitly say I want to overwrite.
Again, we can add code to work around it, but needing to add so much
exception code is a code smell to me.

The two points above are just the low-hanging fruit that came into my mind.
I prefer the approach CocoaPods takes with `Podfile.lock` being next to the
repo manifest, instead of in the `Pods` folder (which instead contains a
`Manifest.lock` AFAIK). So does Rubygems and others.

For me, a regular troubleshooting step is removing the `Packages` and
`.build` folders and rebuilding. In my opinion, this step should **not**
include the risk of altering the locked dependency graph, because both
folders are just "Derived Data", however the VersionLocks file is a source
of truth.

And mixing source of truth with generated files feels very wrong to me.

I'm happy to be proven wrong, but I'd suggest to move the location of the
lockfile next to `Package.swift`. Having it in `Packages` causes a couple
of issues right away without bringing any tangible benefits as far as I can
see (not to mention diverging from other package managers like CocoaPods
and Rubygems, which all keep the lockfile next to their manifest files -
and seem to be happy with it for years).

Honza

···

On Fri, Mar 18, 2016 at 9:45 AM Ben Rimmington via swift-evolution < swift-evolution@swift.org> wrote:

<
https://github.com/apple/swift-package-manager/blob/master/Documentation/Internals/SwiftBasedManifestFormat.md#discussion
>

We decided to use a Swift-based format for the manifest because we believe
it gives developers the best experience for working with and describing
their project. The primary alternative we considered was to use a
declarative format encoded in a common data format like JSON. Although that
would simplify implementation of the tooling around the manifest, it has
the downside that users must then learn this additional language, and the
development of high quality tools for that (documentation, syntax coloring,
parsing diagnostics) isn't aligned with our goal of building great tools
for Swift. In contrast, using the Swift language means that we can leverage
all of the work on Swift to make those tools great.

Could you generate a similar Swift-based format for the lockfile?

-- Ben

On 17 Mar 2016, at 18:23, Max Howell via swift-evolution < > swift-evolution@swift.org> wrote:

The following is a draft proposal, feedback welcome.

____________
SwiftPM Dependency Version Locking

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
   - Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell
   <https://github.com/mxcl&gt;
   - Status: *Discussion*
   - Review manager: Rick Ballard

Introduction

This proposal seeks to declare a new, generated file
Packages/VersionLocks.json that describes the exact state of a package’s
dependency graph and then by default will be respected when executing most
package manager commands. Thus it is considered a “version lock” for a
package’s dependency sources.

Swift-evolution thread
<https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology

   - A *package* refers to a published, versioned git repository designed
   to be consumed as a dependency by SwiftPM.
   - A *project* refers to an end-user workspace that uses SwiftPM (via a
   Package.swift and swift build) fetching and building packages as part
   of its build

Describing this distinction is required because both the above have the
same form, but are used differently by an end-user. An end-user may publish
packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that
projects can easily and trivially become packages when using SwiftPM.
Encouraging a vibrant packaging ecosystem is one of our goals.
Motivation

In a vibrant packaging ecosystem, dependencies update continuously with
bug-fixes and new features. A development team needs:

   1. To ensure they are all using the same versions of their
   dependencies for any given version-control commit.
   2. Ensure they are all using the same versions of the underlying Swift
   toolchain
   3. Be able to override or modify dependency specifications for the
   whole team for specific commits.

Currently with SwiftPM it is possible to fulfill *1.* by committing the
sources of a package’s dependencies with the package itself, but this is
not always desirable. There is no way to achieve 2. and 3. with SwiftPM
alone.

Additionally, there is not currently a way to know which version of Swift
a package requires to build. At this time this situation is particularly
precarious because Swift itself is not backwards compatible. As a Swift
developer at the very least recording which Swift version a package was
built with by the package developer is essential information in order to
assess a package's suitability. Practically the package manager could in
the future use this information to aid an end-user or even fix the problem
when packages fail to compile.
Proposed Solution

A file: Packages/VersionLocks.json will be created alongside the
Package.swift file. Its contents will describe:

   - The URL and versions of cloned dependencies
   - An inline diff of any *local* modifications made to those packages
   relative to their pristine cloned states
   - The exact version of the Swift toolchain used as part of the last
   successful build of the package

This file is generated by SwiftPM.

This file *should* be checked-in with projects.

This file is *generated* and should not be edited by users. If the file
is edited by users the behavior is undefined.

This file *should* be checked-in with packages designed for consumption
in projects, *however* SwiftPM will not use the checkout files of
dependencies when determining a project’s dependency graph (this would make
dependency graphs much less likely to resolve due to overly strict
versioning requirements). In the future we may choose to make it possible
for end-users to attempt to build a package using all checkout files since
in certain deployment scenarios where an exact graph has already been
tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in
Packages/VersionLocks.json as part of the flow described in the next
section. Modifications here means: changes to git remotes and the git-ref
of the checked-out HEAD.
Detailed Design

In a fresh clone that does not contain a Packages directory swift build will
determine the dependency graph, clone the packages into Packages and
generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package
sources. If the user then runs swift build again the package manager will
error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`

It is an error to build against an unlocked dependency graph, but to
facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the
lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if
necessary) recording the current version of the Swift toolchain that
achieved the build.
Packages/VersionLocks.json

The exact design of the contents of this file will be explored during
iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "
https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone":
"Packages/Alamofire-1.2.3", "origin": "
https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, {
"clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot;
"ref": "1.2.3" } ] }
Workflow — Regular Build

   1. User runs swift build
   2. If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips
   to 7.
   3. If Packages/ contains clones and no VersionLocks.json the lockfile
   is generated from the clones
   4. If Packages/ contains checked out sources without git information
   and no VersionLocks.json SwiftPM fetches the git information and
   provided there is no diff, generates the Lockfile, if there is variation it
   is an error *
   5. If Packages/VersionLocks.json is present its dependency graph is
   used
   6. If Packages doesn't exist or is empty the dependency graph is
   resolved, packages are cloned and the Lockfile is generated
   7.

   Build, if Packages are missing because we skipped from 2. the build
   will error, it is the user's responsibility to instruct SwiftPM to
   --update or to fix their dependency graph some other way.
   -

      This scenario is so users can check in their complete dependency
      sources to their tree instead of / as well as the VersionLocks.json file:
      a situation which sometimes is necessary if your dependencies are removed
      from their third party online location, etc.

Workflow — Making Modifications

   1. User makes local modification to a dependency’s sources
   2. User runs swift build
   3. swift build errors out.
   4. User must either lock the graph or run with --ignore-lock

The error-out is likely to be considered tedious by users, however we
consider it important that users are made aware and forced to act when they
modify their dependencies and thus are exposing their team/users to
so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are
not uploaded to a location accessible to the rest of the team they will
fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified
origin has the current locked ref and error out as appropriate.
Workflow — Overriding Packages

   1. User steps into a Package directory eg. Packages/Foo-1.2.3
   2. User changes the origin of Foo to their own fork
   3. User alters HEAD to point to a fix in their own fork
   4. swift build errors out.
   5. User must either lock the graph or run with --ignore-lock

Running swift build --lock regenerates the lockfile, the new origin and
tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for
dependencies, only for projects.

If a package author requires an override they have a few options:

   1. Change the Package.swift dependency specification. This should only
   be done as a last resort, for example, a critical bug must be fixed in a
   dependency and that dependency author is not being responsive. It is up to
   the Package author to ensure this scenario goes well. SwiftPM itself wants
   to guard against these conditions with our proposed “publish & lint” step
   that validates such decisions before signing a published package tag. But
   we are not there yet and thus package authors should be responsible.
   2. Advise end-users in a package README that they should override the
   dependency themselves.

*2* is preferred, but *1* will happen. We consider it our responsibility
to develop tooling that makes *1.* safe or unnecessary, but we are not
there yet.
Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build
--update will fetch the latest versions of all dependencies and update
the lockfile.
Impact on existing code

This proposal will have no impact on existing code.
Alternatives Considered

One alternative is to allow mentioning refs in manifest file while
declaring a dependency but as discussed in this
<http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread
it might not be the best idea.

Using Git submodules for this feature was considered. However something
additionally would be required to specify swift version and record local
diffs. Also this would lock us into git, and despite the fact that
currently we only use git, we have not yet ruled out supporting other
version control systems.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Max_Howell]

<https://github.com/apple/swift-package-manager/blob/master/Documentation/Internals/SwiftBasedManifestFormat.md#discussion&gt;

We decided to use a Swift-based format for the manifest because we believe it gives developers the best experience for working with and describing their project. The primary alternative we considered was to use a declarative format encoded in a common data format like JSON. Although that would simplify implementation of the tooling around the manifest, it has the downside that users must then learn this additional language, and the development of high quality tools for that (documentation, syntax coloring, parsing diagnostics) isn't aligned with our goal of building great tools for Swift. In contrast, using the Swift language means that we can leverage all of the work on Swift to make those tools great.

Could you generate a similar Swift-based format for the lockfile?

The file is not meant to be user edited and JSON would make the data more accessible to other tooling.

We’d go with Swift again if there were concrete reasons for it.

[Max_Howell]

2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.

I like PackageVersions.json

3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).

We don’t, but I agree, we should aim to pick some names and use them consistently.

4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.

Good point, this should be there.

5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?

The latter, if we are to support checking in the `Packages` directory, we should handle it when it is so. Is there a simpler way you can see?

6. I wonder if we should be defining, as Eloy alludes to, two different things:
- The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.
- The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.
Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.

I’m not sure we need a second file, since the versions of the “installed dependencies” are recorded in the directory names as well as that we also do full clones, so that information is part of the clone & checkout.

[Kostiantyn_Koval]

I like the proposal.

I have 1 concern.
As a package author, like PromiseKit <https://github.com/mxcl/PromiseKit&gt; I often have a need to fix some of my dependencies and publish it to my package users.

With the current proposal I can do it:
  - Advise it in README, which is not really a solution done by swift package manager.
  - update Manifest.swift file. There is no convent way to do it now.

I would love to have some functionality to make it simpler for me to specify (modify Manifest.swift or smt else) that Dependency X should:
  - use specific tag X
  - use commit X

But maybe that should be part of another proposal. The lock file would perfectly solve the problems for developing projects that uses packages.

- Kostiantyn

···

On 20 Mar 2016, at 06:07, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

My thoughts on this proposal:

1. I agree with some of the other comments that I would prefer the version file be adjacent to "Package.swift". When the Packages directory isn't being checked in, I really would like to think of it as an "implementation detail" and not embed functionality inside it that would make it hard to remove or change in the future.

2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.

3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).

4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.

5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?

6. I wonder if we should be defining, as Eloy alludes to, two different things:
- The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.
- The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.
Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.

- Daniel

On Mar 17, 2016, at 11:23 AM, Max Howell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

The following is a draft proposal, feedback welcome.

____________
SwiftPM Dependency Version Locking
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md&gt;
Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r&gt;, Max Howell <https://github.com/mxcl&gt;
Status: Discussion
Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file Packages/VersionLocks.json that describes the exact state of a package’s dependency graph and then by default will be respected when executing most package manager commands. Thus it is considered a “version lock” for a package’s dependency sources.

Swift-evolution thread <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html&gt;
Terminology
A package refers to a published, versioned git repository designed to be consumed as a dependency by SwiftPM.
A project refers to an end-user workspace that uses SwiftPM (via a Package.swift and swift build) fetching and building packages as part of its build
Describing this distinction is required because both the above have the same form, but are used differently by an end-user. An end-user may publish packages, but will eventually consume those packages in a project.

As justification for this confusion, it is considered a feature that projects can easily and trivially become packages when using SwiftPM. Encouraging a vibrant packaging ecosystem is one of our goals.

Motivation
In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and new features. A development team needs:

To ensure they are all using the same versions of their dependencies for any given version-control commit.
Ensure they are all using the same versions of the underlying Swift toolchain
Be able to override or modify dependency specifications for the whole team for specific commits.
Currently with SwiftPM it is possible to fulfill 1. by committing the sources of a package’s dependencies with the package itself, but this is not always desirable. There is no way to achieve 2. and 3. with SwiftPM alone.

Additionally, there is not currently a way to know which version of Swift a package requires to build. At this time this situation is particularly precarious because Swift itself is not backwards compatible. As a Swift developer at the very least recording which Swift version a package was built with by the package developer is essential information in order to assess a package's suitability. Practically the package manager could in the future use this information to aid an end-user or even fix the problem when packages fail to compile.

Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the Package.swift file. Its contents will describe:

The URL and versions of cloned dependencies
An inline diff of any local modifications made to those packages relative to their pristine cloned states
The exact version of the Swift toolchain used as part of the last successful build of the package
This file is generated by SwiftPM.

This file should be checked-in with projects.

This file is generated and should not be edited by users. If the file is edited by users the behavior is undefined.

This file should be checked-in with packages designed for consumption in projects, however SwiftPM will not use the checkout files of dependencies when determining a project’s dependency graph (this would make dependency graphs much less likely to resolve due to overly strict versioning requirements). In the future we may choose to make it possible for end-users to attempt to build a package using all checkout files since in certain deployment scenarios where an exact graph has already been tested, this is a solid reliabiity feature.

Any local, modifications made to the clones in Packages are recorded in Packages/VersionLocks.json as part of the flow described in the next section. Modifications here means: changes to git remotes and the git-ref of the checked-out HEAD.

Detailed Design
In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate a Packages/VersionLocks.json file.

The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:

error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.

When swift build --lock is specified the package manager regenerates the lockfile detailing the active git remote and the SHA that is checked-out.

Every time swift build completes a build the lockfile is updated (if necessary) recording the current version of the Swift toolchain that achieved the build.

Packages/VersionLocks.json

The exact design of the contents of this file will be explored during iterative development, but here is a possible example:

json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "https://github.com/mxcl/PromiseKit&quot; "ref": "3.0.3" }, { "clone": "Packages/Alamofire-1.2.3", "origin": "https://github.com/a-fork-somewhere/Alamofire&quot; "ref": "crucial-fix" }, { "clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick&quot; "ref": "1.2.3" } ] }

Workflow — Regular Build

User runs swift build
If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips to 7.
If Packages/ contains clones and no VersionLocks.json the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no VersionLocks.json SwiftPM fetches the git information and provided there is no diff, generates the Lockfile, if there is variation it is an error *
If Packages/VersionLocks.json is present its dependency graph is used
If Packages doesn't exist or is empty the dependency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.

This scenario is so users can check in their complete dependency sources to their tree instead of / as well as the VersionLocks.json file: a situation which sometimes is necessary if your dependencies are removed from their third party online location, etc.

Workflow — Making Modifications

User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we consider it important that users are made aware and forced to act when they modify their dependencies and thus are exposing their team/users to so-called “dependency hell”.

Runing swift build --lock regenerates the lockfile, but does not build.

Modifications must be committed. This means that if the modifications are not uploaded to a location accessible to the rest of the team they will fail to build when they update their checkouts.

The package manager could check for this by asking git if the specified origin has the current locked ref and error out as appropriate.

Workflow — Overriding Packages

User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their own fork
swift build errors out.
User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and tag is stored. Thus a fresh clone of this project would use these overrides.

It is important to note that this workflow will not be respected for dependencies, only for projects.

If a package author requires an override they have a few options:

Change the Package.swift dependency specification. This should only be done as a last resort, for example, a critical bug must be fixed in a dependency and that dependency author is not being responsive. It is up to the Package author to ensure this scenario goes well. SwiftPM itself wants to guard against these conditions with our proposed “publish & lint” step that validates such decisions before signing a published package tag. But we are not there yet and thus package authors should be responsible.
Advise end-users in a package README that they should override the dependency themselves.
2 is preferred, but 1 will happen. We consider it our responsibility to develop tooling that makes 1. safe or unnecessary, but we are not there yet.

Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.

Impact on existing code
This proposal will have no impact on existing code.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/&gt; thread it might not be the best idea.

Using Git submodules for this feature was considered. However something additionally would be required to specify swift version and record local diffs. Also this would lock us into git, and despite the fact that currently we only use git, we have not yet ruled out supporting other version control systems.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Max_Howell]

This is a very welcome addition. Thanks!

A nitpicky detail: I’d prefer the format for the lock file to be yaml or toml over json, since those read a little easier in git diffs (which is, in my experience with cocoapods lockfiles, the only place I ever interact with the lockfiles contents).

Probably we could do TOML.

I picked JSON because it is a more familiar and more widely supported (by tools) format.

YAML would be more work since we have no available YAML parser.

[hartbit (David Hart)]

I think TOML is no-go as its readme says:

Be warned, this spec is still changing a lot. Until it's marked as 1.0, you should assume that it is unstable and act accordingly.

And I agree about YAML: no parser, more work, and more complicated.

JSON sounds like a sane format.

···

On 04 Apr 2016, at 19:41, Max Howell via swift-evolution <swift-evolution@swift.org> wrote:

This is a very welcome addition. Thanks!

A nitpicky detail: I’d prefer the format for the lock file to be yaml or toml over json, since those read a little easier in git diffs (which is, in my experience with cocoapods lockfiles, the only place I ever interact with the lockfiles contents).

Probably we could do TOML.

I picked JSON because it is a more familiar and more widely supported (by tools) format.

YAML would be more work since we have no available YAML parser.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Max_Howell]

Very happy this is coming! I really like the details of this proposal as well.

Just one thing - having the `VersionLocks` in `Packages/VersionLocks` seems like a bad idea to me. Couple of hurdles right away
- for people who don't want to check in their dependencies, we can't just add `Packages` to our `.gitignore`, because that would make it impossible to check in the `VersionLocks` file (users would need to add exceptions to ignore Packages but keep the lockfile, which might result on users accidentally not committing their lockfile at all, which would defeat its whole purpose).

Yes, you have a point. I think it’s not a major issue, since `swift build —init` can be adapted to set up the gitignore properly, but we shouldn’t do it “just because”. I really dislike build systems cluttering the root directory of a project with more and more generated files. And for me the lockfile and the Packages directory are (more) on the same level of derivedness. And I think it makes it clearer that the lock applies only to your dependencies and not your own package.

`Packages` is generated, and the lockfile is generated into it. We hide .build for this reason.

- `swift build --clean=dist` deletes the whole `Packages` directory, removing the lockfile, while I can imagine just wanting to delete and re-pull my dependencies with `swift build --clean=dist; swift build`. This will potentially generate a new lockfile, even though there was a completely valid one which I didn't explicitly say I want to overwrite. Again, we can add code to work around it, but needing to add so much exception code is a code smell to me.

Should a dist clean delete the lockfile? Probably IMO. If not, we can change the behavior.

The two points above are just the low-hanging fruit that came into my mind. I prefer the approach CocoaPods takes with `Podfile.lock` being next to the repo manifest, instead of in the `Pods` folder (which instead contains a `Manifest.lock` AFAIK). So does Rubygems and others.

Indeed, it is not conventional. I’m not committed to the location in the proposal, but I do still prefer it.

For me, a regular troubleshooting step is removing the `Packages` and `.build` folders and rebuilding. In my opinion, this step should **not** include the risk of altering the locked dependency graph, because both folders are just "Derived Data", however the VersionLocks file is a source of truth.

True. Though we can make a clean step that leaves it, or adjust dist. But I agree that it is nice that you can just rm -rf. However there are trade offs either way, having it be easy to rm -rf isn’t exactly a design goal, it is just some nice behavior that fell out of having a nice initial design.

And mixing source of truth with generated files feels very wrong to me.

I'm happy to be proven wrong, but I'd suggest to move the location of the lockfile next to `Package.swift`. Having it in `Packages` causes a couple of issues right away without bringing any tangible benefits as far as I can see (not to mention diverging from other package managers like CocoaPods and Rubygems, which all keep the lockfile next to their manifest files - and seem to be happy with it for years).

If it is to be at the root, then probably it should have a `Package.something` name to make its association clear. Putting it in `Packages/` makes it possible to give it a better more descriptive name.

[Max_Howell]

  - update Manifest.swift file. There is no convent way to do it now.

It is not convenient enough to edit it with a text editor and commit? Seems pretty convenient to me.

[ddunbar (Daniel Dunbar)]

2. I like VersionLocks.json well enough, but would like to see a discussion about possible alternatives. My personal proposal (in line with #1) is to use "PackageVersions.json" which has a nice agreement with Package.swift and would mean two common metadata files show up adjacent. I don't really want to bike shed on the name, but I suspect whatever we pick first will last for a while so I would at least like to review the various alternatives. I also will throw out that my personal opinion is we don't need to pick a name that bears much resemblance with existing terminology, whatever we pick will eventually become "the standard" for the SwiftPM ecosystem so I would prefer to pick the most-descriptive-possible name up front, not one that alludes to the same concept in other systems.

I like PackageVersions.json

Cool. There are still some instances of `VersionLocks.json` in the doc, btw.

3. I like the terminology section here, I almost feel like we should adopt that as official terminology in our documentation (which I don't think we have yet, correct me if I am wrong).

We don’t, but I agree, we should aim to pick some names and use them consistently.

4. I would like it if the lock file recorded the exact SHA it received, and validate that when retrieving. This helps protect users against MITM attacks or unexpected changes if an upstream modifies a tag. It also can be used as part of safety checks when migrating to an alternate repository host which is expected to have the same content.

Good point, this should be there.

5. The "workflow - build" sections #2,3,4 are rather complicated. Is this because the proposal is trying to work with existing Packages layouts, or because the proposal is trying to handle the various variations of what the user may have checked in inside the Packages subdirectory?

The latter, if we are to support checking in the `Packages` directory, we should handle it when it is so. Is there a simpler way you can see?

I think the minimal feature here is that swiftpm grows the ability to read a specification file which declares how the package dependency tree is resolved. These seems like it should be primitive, and straightforward, so it is worrisome that the proposal is so complex.

What is your mental model for what checked in Packages should be? I can see several ways to interpret them:
A. The checked in copy is just a replica of the dependency repository. It exists to ensure the product can be built in a self contained manner.
B. The checked in copy *is* the dependency, it is basically a "convenient fork at a pinned version". SwiftPM recognizes the use case so that it can provide convenient features for the use case (like easily updating the fork).
C. Something in between. The checked in copy exists to ensure self containment, but it is also not treated like a fork.

I feel that too many problems are being combined into one proposal, and I think it makes it hard to work through and discuss the ramifications of all the changes here. There are at least five problems discussed in the current proposal:
1. The feature for pinning of package versions.
2. The workflow for interacting with the pinning feature (i.e., --lock, etc.).
3. Interactions between checked in Package trees and package versions.
4. Swift toolchain versioning issues.
5. Local diffs.
#1 and #2 obviously make sense together, and I can see how #1 and #3 might have to be in the same proposal (since implementing #1 may require solving #3 to not break things). I would like to see the rest be teased apart just so it is easier to understand all the implications.

A priori, I don't see how that minimal feature should require significant discussion w.r.t. checked in Packages. For example, consider the three models I gave above:
- If the expected model is (A), then the behavior I get with checked in Packages should always be exactly the same as if I blew it away (assuming the remote hasn't had content deleted). Therefore, the only new behaviors that need come with version pinning are probably a few cases of error detection (when updating versus a remote which has deleted content).
- If the expected model is (B), then I would expect the behavior to be that there *must* be a PackageVersions.json, and the Packages *must* match those in that file. This may be what the existing rules are trying to codify, if so I think it would be most clear to simply specify the intent.
  o I can see how there are workflow issues around the user editing their "local fork" and needing to update both the PackageVersions.json, but it seems like they follow from the basic behavior of "the PackageVersions *must* match the local fork".
- If the expected model is (C), then I think it is important to clarify exactly what a checked in Packages subdirectory is for first. I have trouble seeing what that is when the implementation uses it *and* the user may modify it.

6. I wonder if we should be defining, as Eloy alludes to, two different things:
- The version lock file, which defines the expected versions for the package manager to use when it is doing package resolution.
- The package state file (in Packages.swift), which is used by the package manager to track information on the Packages/ subdir state in order to provide useful features primarily focused at the scenarios when the user is modifying those files.
Currently it seems like a lot of the behaviors in the proposal are focused at the latter case, but they feel like they should be decoupled problems to me.

I’m not sure we need a second file, since the versions of the “installed dependencies” are recorded in the directory names as well as that we also do full clones, so that information is part of the clone & checkout.

Good point. I thought it might help to have all the data combined in a file we could easily read, but I guess the only thing we can't infer is if the user makes a modification to a tag or something where we wouldn't have the previous hash.

- Daniel

···

On Mar 21, 2016, at 4:05 PM, Max Howell <max.howell@apple.com> wrote:

[Eloy_Duran]

Drive-by chiming in on this.

The reason CocoaPods has two copies of the lockfile is so that the versions of the installed dependencies (Manifest.lock) can be compared to the required dependency versions (Podfile.lock). This is needed for when the dependency dir is ignored from SCM and this is why that lockfile is located inside the dependency dir.

– Eloy

···

On 18 Mar 2016, at 23:07, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

The two points above are just the low-hanging fruit that came into my mind. I prefer the approach CocoaPods takes with `Podfile.lock` being next to the repo manifest, instead of in the `Pods` folder (which instead contains a `Manifest.lock` AFAIK). So does Rubygems and others.

Indeed, it is not conventional. I’m not committed to the location in the proposal, but I do still prefer it.