[Proposal] Lock file for Swift Package Manager


(Ankit Agarwal) #1

Lock File for Swift Package ManagerIntroduction

A Package.lock file containing list of resolved dependencies generated by
swiftpm.
Motivation

Package.lock file can be helpful in situations like :
Reproduce exact versions of dependencies on different machine
* Multiple developers working on a package would want to use the exact
versions (including minor and patch) of the dependencies declared in the
manifest file
* Also helpful when a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit
Sometimes it might be helpful to lock a dependency to a particular commit
ref for which a tagged version is unavailable in cases such as :

* Forking a 3rd party library and making it swiftpm compatible for
temporary use until officially supported by the author
* Package is in active development and not ready for a release tag
Proposed Solution

swiftpm generates a simple Package.lock file after resolving the dependency
graph for that package in some simple format.
Detailed Design
1. Initial$ swift build resolves the dependency graph and generates a
Package.lock file similar to :

ssh://github.com/foo/bar "v1.2.3"http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"

lock file will only be re-modified by $ swift build if Package.swift is
modified by the user.
$ swift build always ignores the lock file and uses local state of Packages
dir / Package.swift

2. User modifies the cloned packages in Packages dir and when satisfied
with the current code of the dependency, commits and pushes it.
To lock the current state of Packages user can run $ swift build --lock which
might result something similar to

ssh://github.com/foo/bar
"248441ff375a19c4365d00d6b0706e11173074f6"http://github.com/foo/baz
"v1.0.0"
../local/git/repo "v3.4.4"

the lock file is committed into the package repo for others to use.

3. A command like $ swift build --bootstrap will always use the lock file
to fetch and checkout the dependencies.
This is useful because running $ swift build might give a higher patch or
minor version of the dependency.

4. If some dependency depends on commit hash (ie non-tagged commit) the
author mentions that in their readme and the end user and maybe other
parallel dependencies will have to use only that commit hash in order to
avoid dependency hell.

5. Allow declaring a dependency without versions in the manifest file for
user wanting to use a untagged dependency. This should probably only be
allowed in debug configuration.
Impact on existing code
None as this would be additional functionality to swift package manager
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while
declaring a dependency but as discussed in this
<https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html>
thread it might not be the best idea.

···

--
Ankit


(Thomas Guthrie) #2

Personally, I’d be more in favour of having something similar to Cargo (Rust’s package/crate manager):

1. `swift build`

Almost the same as it is now, expect if there’s no Package.lock it creates one and subsequent builds use the same dependencies.

2. `swift build --update` or maybe eventually `swift update`

Updates the dependencies in Package.lock to their newest versions and writes them to disk. It should probably build the project as well but possibly makes less sense if its `swift update`.

Similar to Bundler and Cargo you’d check in your Package.lock for app projects and ignore it for library projects.

I’m not really sure what their motivation was for having a lock file always created, it definitely favours “app” projects heavily, but I’ve been messing around with Rust recently and it works pretty well honestly. Maybe there’s a way of making the experience better when the package is solely a library? Personally, if you’re developing a library and `swift build` updates a dependency that breaks everything it’s probably better to know then, whereas with an app you probably want to be working to a lock file and checking what happens when you update dependencies individually.

As for the format of Package.lock, I think it might have to be more complicated than shown to be able to handle the possibility of multiple versions of a dependency etc? Haven’t had a chance to mess around with swiftpm enough yet to say though.

(/end ramble of first thoughts)

— Thomas

···

On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution <swift-evolution@swift.org> wrote:

Lock File for Swift Package Manager
Introduction
A Package.lock file containing list of resolved dependencies generated by swiftpm.

Motivation
Package.lock file can be helpful in situations like :

Reproduce exact versions of dependencies on different machine

* Multiple developers working on a package would want to use the exact versions (including minor and patch) of the dependencies declared in the manifest file
* Also helpful when a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :

* Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
* Package is in active development and not ready for a release tag
Proposed Solution
swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.

Detailed Design
1. Initial$ swift build resolves the dependency graph and generates a Package.lock file similar to :
ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

2. User modifies the cloned packages in Packages dir and when satisfied with the current code of the dependency, commits and pushes it.
To lock the current state of Packages user can run $ swift build --lock which might result something similar to
ssh://github.com/foo/bar <http://github.com/foo/bar> "248441ff375a19c4365d00d6b0706e11173074f6"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
the lock file is committed into the package repo for others to use.

3. A command like $ swift build --bootstrap will always use the lock file to fetch and checkout the dependencies.
This is useful because running $ swift build might give a higher patch or minor version of the dependency.

4. If some dependency depends on commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that commit hash in order to avoid dependency hell.

5. Allow declaring a dependency without versions in the manifest file for user wanting to use a untagged dependency. This should probably only be allowed in debug configuration.
Impact on existing code
None as this would be additional functionality to swift package manager
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> thread it might not be the best idea.

--
Ankit
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Paul Cantrell) #3

+1 for Ankit’s general idea. Details of the proposal aside, I’ll say from experience with bundler that it’s immensely useful — a lifesaver! — to know the exact version of the dependencies another author was using. This has saved my neck more than once.

IMO it’s useful to have a lock file checked in even for libraries — just not pushed forward to client projects. You still want to know what versions the library’s tests last passed against, both for CI and for diagnosing downstream breakage.

-1 to this:

[The] lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

To lock the current state of Packages user can run $ swift build --lock

A couple of problems with that:

(1) Package.swift can specify a version range. You may want to update to the latest patch release without actually modifying Package.swift. I agree with Thomas: there should be a command to update dependencies to the latest matching version. This command should also be able update a single dependency:

  swift build --update SomePackage

(2) I don’t like the idea of the build system running in two separate modes, where sometimes the lock file is ignored and sometimes takes precedence. (If there’s a desire to run in an “unlocked” mode, how about it just doesn't generate the .lock if not already present, and always uses it if it is present?) In practice, though, I’ve found the bundler model works quite well: always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version.

Cheers,

Paul

···

On Dec 20, 2015, at 9:51 AM, Thomas Guthrie via swift-evolution <swift-evolution@swift.org> wrote:

Personally, I’d be more in favour of having something similar to Cargo (Rust’s package/crate manager):

1. `swift build`

Almost the same as it is now, expect if there’s no Package.lock it creates one and subsequent builds use the same dependencies.

2. `swift build --update` or maybe eventually `swift update`

Updates the dependencies in Package.lock to their newest versions and writes them to disk. It should probably build the project as well but possibly makes less sense if its `swift update`.

Similar to Bundler and Cargo you’d check in your Package.lock for app projects and ignore it for library projects.

I’m not really sure what their motivation was for having a lock file always created, it definitely favours “app” projects heavily, but I’ve been messing around with Rust recently and it works pretty well honestly. Maybe there’s a way of making the experience better when the package is solely a library? Personally, if you’re developing a library and `swift build` updates a dependency that breaks everything it’s probably better to know then, whereas with an app you probably want to be working to a lock file and checking what happens when you update dependencies individually.

As for the format of Package.lock, I think it might have to be more complicated than shown to be able to handle the possibility of multiple versions of a dependency etc? Haven’t had a chance to mess around with swiftpm enough yet to say though.

(/end ramble of first thoughts)

— Thomas

On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Lock File for Swift Package Manager
Introduction
A Package.lock file containing list of resolved dependencies generated by swiftpm.

Motivation
Package.lock file can be helpful in situations like :

Reproduce exact versions of dependencies on different machine

* Multiple developers working on a package would want to use the exact versions (including minor and patch) of the dependencies declared in the manifest file
* Also helpful when a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :

* Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
* Package is in active development and not ready for a release tag
Proposed Solution
swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.

Detailed Design
1. Initial$ swift build resolves the dependency graph and generates a Package.lock file similar to :
ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

2. User modifies the cloned packages in Packages dir and when satisfied with the current code of the dependency, commits and pushes it.
To lock the current state of Packages user can run $ swift build --lock which might result something similar to
ssh://github.com/foo/bar <http://github.com/foo/bar> "248441ff375a19c4365d00d6b0706e11173074f6"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
the lock file is committed into the package repo for others to use.

3. A command like $ swift build --bootstrap will always use the lock file to fetch and checkout the dependencies.
This is useful because running $ swift build might give a higher patch or minor version of the dependency.

4. If some dependency depends on commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that commit hash in order to avoid dependency hell.

5. Allow declaring a dependency without versions in the manifest file for user wanting to use a untagged dependency. This should probably only be allowed in debug configuration.
Impact on existing code
None as this would be additional functionality to swift package manager
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> thread it might not be the best idea.

--
Ankit
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rick Ballard) #4

Thanks for pushing on this, Ankit, Thomas, and Paul. As Doug mentioned in another thread, Apple is heading into our holiday shutdown until the new year, so I think we should schedule the actual evolution review for the first week of January. Let's get this hashed out in the meantime and ready for that review now, though.

Here are my comments:

– I know Cargo uses these terms already, but --lock and "lockfile" are very generic terms. A "package manager lockfile" could easily refer to a file multiple package manager processes use to avoid corrupting a shared database. I think calling this a "deplock" file (for "dependency lock") is much more specific without being much more verbose. And I'd suggest calling the option something like --lock-deps.

– Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps instead. (If we have this flag at all... see below).

– I'm concerned about the lockfile being a 2nd source of truth (vs Package.swift) that could easily get out of sync. For example, if you update Package.swift to require a new minimum version, but forget to update the lockfile (or forget to commit the updated lockfile), users will wind up silently using a different version than that allowed by the Package.swift's specified version.

– I'm also uneasy with the lockfile being toml while Package.swift is swift. That seems inconsistent and requires users to work in two different configuration file syntaxes (even if toml is very simple).

– To address both of the above points, maybe the dep-lock info should be stored in Package.swift itself. In this case, any tool which automatically modifies your Package.swift for you would autoclear the dependency lock when it updates a package version, and since that data is in the same file, those two changes would get checked in together. When you're hand-editing the file, it'd be a lot easier to remember to clear (or update) the locked dependency. And you'd ensure that the Package.swift data is always in sync with your dependency lock across revisions and branches, since both data is in the same file (at least for direct dependencies).

Since dependency locks apply to non-direct dependencies as well, we would need to add a new package property for modeling the dependency lock for an otherwise-unspecified dependency. And we might require that the locks fully specify the properties of the dependency they apply to, so if the required version of an indirect dependency changes, we can tell that the dependency lock is out of date. For example, say your "FooApp" package depends on "Lib1" and "Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)), and that's what we'd record for the dependencyLock:

let package =
Package(
    name: "FooApp",
    dependencies: [
        .Package(url: "ssh://git@example.com/Lib1.git"),
  .Package(url: "ssh://git@example.com/Lib2.git"),
    ],
    dependencyLocks: [
  .DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485, .Package(url: "ssh://git@example.com/Lib1.git"),
  .DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2, .Package(url: "ssh://git@example.com/Lib2.git"),
        .DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc, .Package(url: "ssh://git@example.com/LibBar.git", versions: Version(1,0,1)..<Version(1,5,0)),
    ],
)

If Lib2 then updated its version specification to Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out of date and prompt you to create a new lock.

Note that we wouldn't expect you to have to hand-author the .DependencyLock (and manually repeat the version range for packages you depend on); normally this would be autogenerated by the package manager.

– It would be nice if we could warn, when building with dependency locks, if your dependencyLock revision does not match a dependency's version specifier. For example, if your dependency specifies (2,0,1)..<Version(2,1,0), and your dependencyLock is db2e873d530c72023cf00ce7fe9a37211b8d2fbc, we would check and make sure that some revision 2.0.1 or greater contains db2e873d530c72023cf00ce7fe9a37211b8d2fbc, but that it's not reachable from 2.1.0 or greater. That said, `git tag --contains` is probably not fast enough for nontrivial repositories to run on each build for the purposes of issuing this warning, so this might be a non-starter for performance reasons.

– Should we always automatically use the deplock info? Ankit's proposal said no, while Thomas and Paul said yes. I think that it makes sense for most use cases to use a stable version of your dependencies and only update to a newer version explicitly, instead of having that happen implicitly when you build if there happens to be a new version. That favors always using the deplock info. Thus I think I like Paul's proposal to "always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version." So I'd suggest:

  – `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
  – `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.
  – `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
  – `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.
  – `swift build` clones any dependencies that aren't already cloned, checks out the locked commit for all dependencies (whether they were already cloned before or not), and adds dependency locks to Package.swift for any packages that don't have a lock already. This also warns if the dependency lock's package specifier doesn't match the actual package specifier where that dependency is defined, which indicates that your locks are out of date with respect to your dependency specifications.

One downside to this behavior is it makes it easy to mess up when modifying your dependencies locally. If you make an edit to a cloned dependency and commit it, and then `swift build` the top-level package, swiftpm will automatically revert HEAD of that dependency to the locked commit, so it won't actually build your change. In order to avoid this, you need to run `swift build --lock-deps` after committing a local change to a cloned dependency, and it's easy to forget to do so. That said, the alternative behavior – where `swift build` preserves the state of your dependencies by default – means that if you've built a package in the past, and you pull and get a new Package.swift with new dependency locks, you won't automatically get those dependencies updated when you build, since you already have cloned dependencies whose HEADs are stale and don't match the new dependency locks. The latter problem seems worse than the former. I'm open to ideas to how to solve both problems nicely; the ways I've thought of so far make this proposal even more complex.

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away. To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

– Likewise, this mechanism could be used to allow you to override the source of a dependency for your indirect dependencies. For example, if you depend on "Lib1", which depends on git@github.com:Somewhere/LibFoo.git, but you actually want to use your own fork of LibFoo – git@github.com:YourName/LibFoo.git – the dependency lock would allow that override. This would be done with an "overridingURL" property on the dependency lock.

– I am concerned about the complexity and additional learning curve this behavior brings to the package manager. That said, this seems like important functionality.

Thoughts?

  - Rick

···

On Dec 20, 2015, at 1:22 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org> wrote:

+1 for Ankit’s general idea. Details of the proposal aside, I’ll say from experience with bundler that it’s immensely useful — a lifesaver! — to know the exact version of the dependencies another author was using. This has saved my neck more than once.

IMO it’s useful to have a lock file checked in even for libraries — just not pushed forward to client projects. You still want to know what versions the library’s tests last passed against, both for CI and for diagnosing downstream breakage.

-1 to this:

[The] lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

To lock the current state of Packages user can run $ swift build --lock

A couple of problems with that:

(1) Package.swift can specify a version range. You may want to update to the latest patch release without actually modifying Package.swift. I agree with Thomas: there should be a command to update dependencies to the latest matching version. This command should also be able update a single dependency:

  swift build --update SomePackage

(2) I don’t like the idea of the build system running in two separate modes, where sometimes the lock file is ignored and sometimes takes precedence. (If there’s a desire to run in an “unlocked” mode, how about it just doesn't generate the .lock if not already present, and always uses it if it is present?) In practice, though, I’ve found the bundler model works quite well: always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version.

Cheers,

Paul

On Dec 20, 2015, at 9:51 AM, Thomas Guthrie via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Personally, I’d be more in favour of having something similar to Cargo (Rust’s package/crate manager):

1. `swift build`

Almost the same as it is now, expect if there’s no Package.lock it creates one and subsequent builds use the same dependencies.

2. `swift build --update` or maybe eventually `swift update`

Updates the dependencies in Package.lock to their newest versions and writes them to disk. It should probably build the project as well but possibly makes less sense if its `swift update`.

Similar to Bundler and Cargo you’d check in your Package.lock for app projects and ignore it for library projects.

I’m not really sure what their motivation was for having a lock file always created, it definitely favours “app” projects heavily, but I’ve been messing around with Rust recently and it works pretty well honestly. Maybe there’s a way of making the experience better when the package is solely a library? Personally, if you’re developing a library and `swift build` updates a dependency that breaks everything it’s probably better to know then, whereas with an app you probably want to be working to a lock file and checking what happens when you update dependencies individually.

As for the format of Package.lock, I think it might have to be more complicated than shown to be able to handle the possibility of multiple versions of a dependency etc? Haven’t had a chance to mess around with swiftpm enough yet to say though.

(/end ramble of first thoughts)

— Thomas

On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Lock File for Swift Package Manager
Introduction
A Package.lock file containing list of resolved dependencies generated by swiftpm.

Motivation
Package.lock file can be helpful in situations like :

Reproduce exact versions of dependencies on different machine

* Multiple developers working on a package would want to use the exact versions (including minor and patch) of the dependencies declared in the manifest file
* Also helpful when a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :

* Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
* Package is in active development and not ready for a release tag
Proposed Solution
swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.

Detailed Design
1. Initial$ swift build resolves the dependency graph and generates a Package.lock file similar to :
ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

2. User modifies the cloned packages in Packages dir and when satisfied with the current code of the dependency, commits and pushes it.
To lock the current state of Packages user can run $ swift build --lock which might result something similar to
ssh://github.com/foo/bar <http://github.com/foo/bar> "248441ff375a19c4365d00d6b0706e11173074f6"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
the lock file is committed into the package repo for others to use.

3. A command like $ swift build --bootstrap will always use the lock file to fetch and checkout the dependencies.
This is useful because running $ swift build might give a higher patch or minor version of the dependency.

4. If some dependency depends on commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that commit hash in order to avoid dependency hell.

5. Allow declaring a dependency without versions in the manifest file for user wanting to use a untagged dependency. This should probably only be allowed in debug configuration.
Impact on existing code
None as this would be additional functionality to swift package manager
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> thread it might not be the best idea.

--
Ankit
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Max Howell) #5

– Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps instead. (If we have this flag at all... see below).

I’d prefer a workflow that omits —bootstrap, to repeat my previous email:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

– I'm also uneasy with the lockfile being toml while Package.swift is swift. That seems inconsistent and requires users to work in two different configuration file syntaxes (even if toml is very simple).

The proposal is not TOML, just some trivial unspecified markup. If we are to have two files then I think it should either be Swift or just a basic list like the proposal says.

– To address both of the above points, maybe the dep-lock info should be stored in Package.swift itself. In this case, any tool which automatically modifies your Package.swift for you would autoclear the dependency lock when it updates a package version, and since that data is in the same file, those two changes would get checked in together. When you're hand-editing the file, it'd be a lot easier to remember to clear (or update) the locked dependency. And you'd ensure that the Package.swift data is always in sync with your dependency lock across revisions and branches, since both data is in the same file (at least for direct dependencies).

This certainly needs more discussion. I’ll wait for Daniel to chime in.

Since dependency locks apply to non-direct dependencies as well, we would need to add a new package property for modeling the dependency lock for an otherwise-unspecified dependency. And we might require that the locks fully specify the properties of the dependency they apply to, so if the required version of an indirect dependency changes, we can tell that the dependency lock is out of date. For example, say your "FooApp" package depends on "Lib1" and "Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)), and that's what we'd record for the dependencyLock:

let package =
Package(
    name: "FooApp",
    dependencies: [
        .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
  .Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
    ],
    dependencyLocks: [
  .DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485, .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
  .DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2, .Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
        .DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc, .Package(url: "ssh://git@example.com/LibBar.git <ssh://git@example.com/LibBar.git>", versions: Version(1,0,1)..<Version(1,5,0)),
    ],
)

This makes sense, if we go one-file.

If Lib2 then updated its version specification to Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out of date and prompt you to create a new lock.

Indeed, this is what I envisage: respect the lock file as the primary truth and if the manifest and lock file conflict, error (for major version changes) or warn.

Note that we wouldn't expect you to have to hand-author the .DependencyLock (and manually repeat the version range for packages you depend on); normally this would be autogenerated by the package manager.

Indeed.

– Should we always automatically use the deplock info? Ankit's proposal said no, while Thomas and Paul said yes. I think that it makes sense for most use cases to use a stable version of your dependencies and only update to a newer version explicitly, instead of having that happen implicitly when you build if there happens to be a new version. That favors always using the deplock info. Thus I think I like Paul's proposal to "always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version." So I'd suggest:

  – `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
  – `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.
  – `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
  – `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.
  – `swift build` clones any dependencies that aren't already cloned, checks out the locked commit for all dependencies (whether they were already cloned before or not), and adds dependency locks to Package.swift for any packages that don't have a lock already. This also warns if the dependency lock's package specifier doesn't match the actual package specifier where that dependency is defined, which indicates that your locks are out of date with respect to your dependency specifications.

Sounds good, though I’d drop -deps from both.

One downside to this behavior is it makes it easy to mess up when modifying your dependencies locally. If you make an edit to a cloned dependency and commit it, and then `swift build` the top-level package, swiftpm will automatically revert HEAD of that dependency to the locked commit, so it won't actually build your change. In order to avoid this, you need to run `swift build --lock-deps` after committing a local change to a cloned dependency, and it's easy to forget to do so. That said, the alternative behavior – where `swift build` preserves the state of your dependencies by default – means that if you've built a package in the past, and you pull and get a new Package.swift with new dependency locks, you won't automatically get those dependencies updated when you build, since you already have cloned dependencies whose HEADs are stale and don't match the new dependency locks. The latter problem seems worse than the former. I'm open to ideas to how to solve both problems nicely; the ways I've thought of so far make this proposal even more complex.

IMO if the Packages/ directory has unstaged or uncommitted/unpushed changes should be a build error.

However it is important to allow editing of Packages/. In other language packaging systems, fixing or modifying your dependencies is tedious, and this discourages people from improving the ecosystem.

So in my opinion it is important that this is possible and thus I’d like to propose a flag, —ignore-lock perhaps that allows the above error to be ignored, albeit with a very visible warning with a link to documentation about this.

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away. To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

The proposal should mention this.

– Likewise, this mechanism could be used to allow you to override the source of a dependency for your indirect dependencies. For example, if you depend on "Lib1", which depends on git@github.com:Somewhere/LibFoo.git, but you actually want to use your own fork of LibFoo – git@github.com:YourName/LibFoo.git – the dependency lock would allow that override. This would be done with an "overridingURL" property on the dependency lock.

– I am concerned about the complexity and additional learning curve this behavior brings to the package manager. That said, this seems like important functionality.

I think we can minimize the complexity further.


(Paul Cantrell) #6

I share Rick’s concerns about any proposal that pushes locked versions downstream to dependent projects. Two thoughts on that:

First, the specific: a lockfile should lock the versions for all dependencies, direct and indirect. Library lockfiles should have no effect on downstream projects. (Bundler does it this way, and it plays out well in practice.)

Second, some general thoughts on the principles underlying that, which also address the concern about multiple sources of truth:

The dependency specification (Package.swift) and the lockfile serve distinct roles. The dependency spec says “here’s information about what I need, and which versions I think I’m compatible with.” The lockfile says “here are the specific versions I’m actually using to build and test with.”

In short, Package.swift is a declaration of intent, whereas the lockfile is a statement of fact. Package.swift is a set of constraints; the lockfile is a snapshot. Package.swift describes things what may happen; the lockfile describes what did happen.

The lockfile’s primary purposes are (1) to prevent external variables from causing surprising breakages, and (2) to diagnose the “it works here but not there” problem. This is why lockfiles should be checked in, but shouldn’t affect downstream projects.

  – `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
  – `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.

  – `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
  – `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.

This set of capabilities seems right, though I’m not entirely sure about the separate “lock” operation. Are you thinking of the case when somebody manually checks out a different version in a submodule, totally bypassing the package manager?

That particular command line syntax is a bit Byzantine. I suspect I’d be Googling it a lot.

I’m not so sure “build” makes sense as the top-level command. What about (just brainstorming):

  swift package update
  swift package update <Lib>
  swift package lock
  swift package lock <Lib>

…or maybe “deps” instead of “package”?

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away.

I would imagine that…

• Package.swift can specify any tag or branch, not just a version number.
• The update and lock commands write a specific commit to the lockfile.
• Perhaps, for readability, if there is a tag for the commit, the tag name goes in the lockfile instead of a raw commit hash, and that tag name might be a version number — but it’s always specific commits in the lockfile.

To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

That seems overly complex at first glance. If Package.swift does specify a branch, I interpret that as saying, “I depend on prerelease features of this package. Resolve dependencies at your own risk!” At that point, having a version number in there as well is just confusion. When the code on that branch gets release, you go back to the version specifier.

Cheers, P

···

On Dec 22, 2015, at 11:57 AM, Rick Ballard <rballard@apple.com> wrote:

Thanks for pushing on this, Ankit, Thomas, and Paul. As Doug mentioned in another thread, Apple is heading into our holiday shutdown until the new year, so I think we should schedule the actual evolution review for the first week of January. Let's get this hashed out in the meantime and ready for that review now, though.

Here are my comments:

– I know Cargo uses these terms already, but --lock and "lockfile" are very generic terms. A "package manager lockfile" could easily refer to a file multiple package manager processes use to avoid corrupting a shared database. I think calling this a "deplock" file (for "dependency lock") is much more specific without being much more verbose. And I'd suggest calling the option something like --lock-deps.

– Likewise, --bootstrap isn't very clear. I'd suggest --use-locked-deps instead. (If we have this flag at all... see below).

– I'm concerned about the lockfile being a 2nd source of truth (vs Package.swift) that could easily get out of sync. For example, if you update Package.swift to require a new minimum version, but forget to update the lockfile (or forget to commit the updated lockfile), users will wind up silently using a different version than that allowed by the Package.swift's specified version.

– I'm also uneasy with the lockfile being toml while Package.swift is swift. That seems inconsistent and requires users to work in two different configuration file syntaxes (even if toml is very simple).

– To address both of the above points, maybe the dep-lock info should be stored in Package.swift itself. In this case, any tool which automatically modifies your Package.swift for you would autoclear the dependency lock when it updates a package version, and since that data is in the same file, those two changes would get checked in together. When you're hand-editing the file, it'd be a lot easier to remember to clear (or update) the locked dependency. And you'd ensure that the Package.swift data is always in sync with your dependency lock across revisions and branches, since both data is in the same file (at least for direct dependencies).

Since dependency locks apply to non-direct dependencies as well, we would need to add a new package property for modeling the dependency lock for an otherwise-unspecified dependency. And we might require that the locks fully specify the properties of the dependency they apply to, so if the required version of an indirect dependency changes, we can tell that the dependency lock is out of date. For example, say your "FooApp" package depends on "Lib1" and "Lib2", which both depend on "LibBar". "Lib1" might specify that it depends on LibBar versions: Version(1,0,1)..<Version(2,0,0)), while "Lib2" might specify LibBar versions: Version(1,0,0)..<Version(1,5,0)). FooApp's indirect dependency on LibBar is thus constrained to versions: Version(1,0,1)..<Version(1,5,0)), and that's what we'd record for the dependencyLock:

let package =
Package(
    name: "FooApp",
    dependencies: [
        .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
  .Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
    ],
    dependencyLocks: [
  .DependencyLock(lockRevision: c611ad62500182cae041abe83db908c2ea8e4485, .Package(url: "ssh://git@example.com/Lib1.git <ssh://git@example.com/Lib1.git>"),
  .DependencyLock(lockRevision: 1fb095a46ff55161876380067344ff641b8e95e2, .Package(url: "ssh://git@example.com/Lib2.git <ssh://git@example.com/Lib2.git>"),
        .DependencyLock(lockRevision: db2e873d530c72023af00ce7fe9a37211b8d2fbc, .Package(url: "ssh://git@example.com/LibBar.git <ssh://git@example.com/LibBar.git>", versions: Version(1,0,1)..<Version(1,5,0)),
    ],
)

If Lib2 then updated its version specification to Version(1,0,0)..<Version(1,6,0). we could tell that our dependency lock was out of date and prompt you to create a new lock.

Note that we wouldn't expect you to have to hand-author the .DependencyLock (and manually repeat the version range for packages you depend on); normally this would be autogenerated by the package manager.

– It would be nice if we could warn, when building with dependency locks, if your dependencyLock revision does not match a dependency's version specifier. For example, if your dependency specifies (2,0,1)..<Version(2,1,0), and your dependencyLock is db2e873d530c72023cf00ce7fe9a37211b8d2fbc, we would check and make sure that some revision 2.0.1 or greater contains db2e873d530c72023cf00ce7fe9a37211b8d2fbc, but that it's not reachable from 2.1.0 or greater. That said, `git tag --contains` is probably not fast enough for nontrivial repositories to run on each build for the purposes of issuing this warning, so this might be a non-starter for performance reasons.

– Should we always automatically use the deplock info? Ankit's proposal said no, while Thomas and Paul said yes. I think that it makes sense for most use cases to use a stable version of your dependencies and only update to a newer version explicitly, instead of having that happen implicitly when you build if there happens to be a new version. That favors always using the deplock info. Thus I think I like Paul's proposal to "always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version." So I'd suggest:

  – `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
  – `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.
  – `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
  – `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.
  – `swift build` clones any dependencies that aren't already cloned, checks out the locked commit for all dependencies (whether they were already cloned before or not), and adds dependency locks to Package.swift for any packages that don't have a lock already. This also warns if the dependency lock's package specifier doesn't match the actual package specifier where that dependency is defined, which indicates that your locks are out of date with respect to your dependency specifications.

One downside to this behavior is it makes it easy to mess up when modifying your dependencies locally. If you make an edit to a cloned dependency and commit it, and then `swift build` the top-level package, swiftpm will automatically revert HEAD of that dependency to the locked commit, so it won't actually build your change. In order to avoid this, you need to run `swift build --lock-deps` after committing a local change to a cloned dependency, and it's easy to forget to do so. That said, the alternative behavior – where `swift build` preserves the state of your dependencies by default – means that if you've built a package in the past, and you pull and get a new Package.swift with new dependency locks, you won't automatically get those dependencies updated when you build, since you already have cloned dependencies whose HEADs are stale and don't match the new dependency locks. The latter problem seems worse than the former. I'm open to ideas to how to solve both problems nicely; the ways I've thought of so far make this proposal even more complex.

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away. To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

– Likewise, this mechanism could be used to allow you to override the source of a dependency for your indirect dependencies. For example, if you depend on "Lib1", which depends on git@github.com:Somewhere/LibFoo.git, but you actually want to use your own fork of LibFoo – git@github.com:YourName/LibFoo.git – the dependency lock would allow that override. This would be done with an "overridingURL" property on the dependency lock.

– I am concerned about the complexity and additional learning curve this behavior brings to the package manager. That said, this seems like important functionality.

Thoughts?

  - Rick

On Dec 20, 2015, at 1:22 PM, Paul Cantrell via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

+1 for Ankit’s general idea. Details of the proposal aside, I’ll say from experience with bundler that it’s immensely useful — a lifesaver! — to know the exact version of the dependencies another author was using. This has saved my neck more than once.

IMO it’s useful to have a lock file checked in even for libraries — just not pushed forward to client projects. You still want to know what versions the library’s tests last passed against, both for CI and for diagnosing downstream breakage.

-1 to this:

[The] lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

To lock the current state of Packages user can run $ swift build --lock

A couple of problems with that:

(1) Package.swift can specify a version range. You may want to update to the latest patch release without actually modifying Package.swift. I agree with Thomas: there should be a command to update dependencies to the latest matching version. This command should also be able update a single dependency:

  swift build --update SomePackage

(2) I don’t like the idea of the build system running in two separate modes, where sometimes the lock file is ignored and sometimes takes precedence. (If there’s a desire to run in an “unlocked” mode, how about it just doesn't generate the .lock if not already present, and always uses it if it is present?) In practice, though, I’ve found the bundler model works quite well: always generate .lock if absent, always use the locked version if present, and use a separate command to update the locked version.

Cheers,

Paul

On Dec 20, 2015, at 9:51 AM, Thomas Guthrie via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Personally, I’d be more in favour of having something similar to Cargo (Rust’s package/crate manager):

1. `swift build`

Almost the same as it is now, expect if there’s no Package.lock it creates one and subsequent builds use the same dependencies.

2. `swift build --update` or maybe eventually `swift update`

Updates the dependencies in Package.lock to their newest versions and writes them to disk. It should probably build the project as well but possibly makes less sense if its `swift update`.

Similar to Bundler and Cargo you’d check in your Package.lock for app projects and ignore it for library projects.

I’m not really sure what their motivation was for having a lock file always created, it definitely favours “app” projects heavily, but I’ve been messing around with Rust recently and it works pretty well honestly. Maybe there’s a way of making the experience better when the package is solely a library? Personally, if you’re developing a library and `swift build` updates a dependency that breaks everything it’s probably better to know then, whereas with an app you probably want to be working to a lock file and checking what happens when you update dependencies individually.

As for the format of Package.lock, I think it might have to be more complicated than shown to be able to handle the possibility of multiple versions of a dependency etc? Haven’t had a chance to mess around with swiftpm enough yet to say though.

(/end ramble of first thoughts)

— Thomas

On 20 Dec 2015, at 09:01, Ankit Agarwal via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Lock File for Swift Package Manager
Introduction
A Package.lock file containing list of resolved dependencies generated by swiftpm.

Motivation
Package.lock file can be helpful in situations like :

Reproduce exact versions of dependencies on different machine

* Multiple developers working on a package would want to use the exact versions (including minor and patch) of the dependencies declared in the manifest file
* Also helpful when a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :

* Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
* Package is in active development and not ready for a release tag
Proposed Solution
swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.

Detailed Design
1. Initial$ swift build resolves the dependency graph and generates a Package.lock file similar to :
ssh://github.com/foo/bar <http://github.com/foo/bar> "v1.2.3"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
lock file will only be re-modified by $ swift build if Package.swift is modified by the user.
$ swift build always ignores the lock file and uses local state of Packages dir / Package.swift

2. User modifies the cloned packages in Packages dir and when satisfied with the current code of the dependency, commits and pushes it.
To lock the current state of Packages user can run $ swift build --lock which might result something similar to
ssh://github.com/foo/bar <http://github.com/foo/bar> "248441ff375a19c4365d00d6b0706e11173074f6"
http://github.com/foo/baz "v1.0.0"
../local/git/repo "v3.4.4"
the lock file is committed into the package repo for others to use.

3. A command like $ swift build --bootstrap will always use the lock file to fetch and checkout the dependencies.
This is useful because running $ swift build might give a higher patch or minor version of the dependency.

4. If some dependency depends on commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that commit hash in order to avoid dependency hell.

5. Allow declaring a dependency without versions in the manifest file for user wanting to use a untagged dependency. This should probably only be allowed in debug configuration.
Impact on existing code
None as this would be additional functionality to swift package manager
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html> thread it might not be the best idea.

--
Ankit
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Joshua Scott Emmons) #7

I’d prefer a workflow that omits —bootstrap, to repeat my previous email:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

Maybe it's me, but I've been burned by "Why isn't this updating — oh, there's a lock file!" more than once. Having "`swift build` always use a lock file if present" isn't simpler. It requires I reason about the state of the lock file every time I build.

If we want to be consistent, we should make `swift build` always build according to Package.swift — always. Then, if we want to build from locked versions, `swift build --use-locked-deps` would always build from the lock file (helpfully alerting us if it doesn't exist). This way we don't have to remember which does what when something exists or doesn't.

To look at it another way, I'm always aware when I want to build my packages to exactly match locked versions. I'm usually installing or deploying and am focused on the task of building, itself. Asking me to to be explicit with `--use-locked-deps` at this step isn't a huge leap.

I'm not always aware of when I *don't* want to build with locks, though. In that case I'm usually focused on development or testing or debugging. Building is just a side-effect. Asking me to be explicit with my intentions at this point (or, worse, asking me to reason about how build will behave based on the presence of other factors) feels like a nuisance.

Just my 2¢,

···

--
-jemmons


(Max Howell) #8

I share Rick’s concerns about any proposal that pushes locked versions downstream to dependent projects. Two thoughts on that:

First, the specific: a lockfile should lock the versions for all dependencies, direct and indirect. Library lockfiles should have no effect on downstream projects. (Bundler does it this way, and it plays out well in practice.)

Indeed, if the proposal is not clear here, it should be.

Second, some general thoughts on the principles underlying that, which also address the concern about multiple sources of truth:

The dependency specification (Package.swift) and the lockfile serve distinct roles. The dependency spec says “here’s information about what I need, and which versions I think I’m compatible with.” The lockfile says “here are the specific versions I’m actually using to build and test with.”

In short, Package.swift is a declaration of intent, whereas the lockfile is a statement of fact. Package.swift is a set of constraints; the lockfile is a snapshot. Package.swift describes things what may happen; the lockfile describes what did happen.

The lockfile’s primary purposes are (1) to prevent external variables from causing surprising breakages, and (2) to diagnose the “it works here but not there” problem. This is why lockfiles should be checked in, but shouldn’t affect downstream projects.

  – `swift build --update-deps` updates all dependencies to the latest allowed version and sets/updates dependency locks for each dependency.
  – `swift build --update-dep=<package name>` updates the named dependency to the latest allowed version and sets/updates dependency lock for that dependency.

  – `swift build --lock-deps` looks at the HEAD of all cloned dependencies and sets the dependency lock in the top-level Package.swift to that HEAD commit. This is useful when you want to explicitly lock to something other than the latest allowed version of a package.
  – `swift build --lock-dep=<package name> looks at HEAD of the cloned named dependency and sets the dependency lock in the the top-level Package.swift for that one dependency.

This set of capabilities seems right, though I’m not entirely sure about the separate “lock” operation. Are you thinking of the case when somebody manually checks out a different version in a submodule, totally bypassing the package manager?

Yes. The user has the capability of eg. specifying a branch. Package.swift cannot do this, but the lock file can override it.

That particular command line syntax is a bit Byzantine. I suspect I’d be Googling it a lot.

I think it can be simplified. Dropping the -dep suffix alone makes it more conventional.

I’m not so sure “build” makes sense as the top-level command. What about (just brainstorming):

  swift package update
  swift package update <Lib>
  swift package lock
  swift package lock <Lib>

…or maybe “deps” instead of “package”?

`swift build` is already the command, we could at a later time split the switches out into new sub commands of the swift multitool, but that would be another later proposal after more functionality is available and we understand the overall UX for the CLI command better.

– This proposal still doesn't fully address how you can use a branch for your dependency instead of a version tag, which was one of the reasons this topic came up in the first place. You could do so by checking out the branch in your cloned dependency and then using `swift build --lock-deps` to lock to that commit, but if you do a `swift build --update-deps` we'll blow that away.

I would imagine that…

• Package.swift can specify any tag or branch, not just a version number.

This is not intended by this proposal. Just versions.

• The update and lock commands write a specific commit to the lock file.

Yes.

• Perhaps, for readability, if there is a tag for the commit, the tag name goes in the lockfile instead of a raw commit hash, and that tag name might be a version number — but it’s always specific commits in the lock file.

Has value.

To solve this, perhaps a dependency lock could have an additional optional "overridingRef" property which, if specified, overrides the version specifier for the package. That means that `swift build --update-deps` will now update the package to what that ref points to. The --lock-dep option could allow a follow-on option --lock-overriding-ref which takes the overriding ref to set.

That seems overly complex at first glance. If Package.swift does specify a branch, I interpret that as saying, “I depend on prerelease features of this package. Resolve dependencies at your own risk!” At that point, having a version number in there as well is just confusion. When the code on that branch gets release, you go back to the version specifier.

Package.swift cannot specify a branch in this proposal.

···

On Dec 22, 2015, at 3:24 PM, Paul Cantrell <cantrell@pobox.com> wrote:


(Max Howell) #9

I’d prefer a workflow that omits —bootstrap, to repeat my previous email:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

Maybe it's me, but I've been burned by "Why isn't this updating — oh, there's a lock file!" more than once. Having "`swift build` always use a lock file if present" isn't simpler. It requires I reason about the state of the lock file every time I build.

There will be a dedicated `—update` command that updates and it will update the lock per the current proposal. Expecting the build command to update the checked out dependencies every time seems unintuitive. It is after all a build command.

If we want to be consistent, we should make `swift build` always build according to Package.swift — always. Then, if we want to build from locked versions, `swift build --use-locked-deps` would always build from the lock file (helpfully alerting us if it doesn't exist). This way we don't have to remember which does what when something exists or doesn’t.

If we didn’t have lock files I still would not want `swift build` to update my dependencies before building, I would still want a separate --update switch, and in this case it would thus behave the same as if there is a lock file.

Updating your dependencies every build is asking for trouble, you wouldn’t want to be debugging and then also possibly have your underlying library dependencies change in behavior.

To look at it another way, I'm always aware when I want to build my packages to exactly match locked versions. I'm usually installing or deploying and am focused on the task of building, itself. Asking me to to be explicit with `--use-locked-deps` at this step isn't a huge leap.

Maybe I don’t understand the problem here. Do you usually use tools that update dependencies before every build?

I'm not always aware of when I *don't* want to build with locks, though. In that case I'm usually focused on development or testing or debugging. Building is just a side-effect. Asking me to be explicit with my intentions at this point (or, worse, asking me to reason about how build will behave based on the presence of other factors) feels like a nuisance.

I think I don't understand your concern. The lockfile enables fresh checkouts can build deterministically with a known dependency graph. You don’t have to commit it, but you should. If you checkout and then want newer dependencies you build with --update.

Maybe if you provide your example from a time a lockfile hindered your workflow we can take it into account.


(Joshua Scott Emmons) #10

I’d prefer a workflow that omits —bootstrap, to repeat my previous email:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

Maybe it's me, but I've been burned by "Why isn't this updating — oh, there's a lock file!" more than once. Having "`swift build` always use a lock file if present" isn't simpler. It requires I reason about the state of the lock file every time I build.

There will be a dedicated `—update` command that updates and it will update the lock per the current proposal. Expecting the build command to update the checked out dependencies every time seems unintuitive. It is after all a build command.

...

If we didn’t have lock files I still would not want `swift build` to update my dependencies before building, I would still want a separate --update switch, and in this case it would thus behave the same as if there is a lock file.

Updating your dependencies every build is asking for trouble, you wouldn’t want to be debugging and then also possibly have your underlying library dependencies change in behavior.

Right. I didn't mean to insinuate we should be pulling fresh dependencies before each build. Here's the scenario

* I'm working on a package that depends on FooPackage Version(1,0,0)..<Version(2,0,0)
* I `swift --update`. This pulls FooPackage v1.1 and sets the lockfile. I commit the lockfile to git.
* FooPackage v1.2 is released
* A co-worker `swift --update`s to FooPackage v1.2, her lockfile is updated and she pushes it to git.
* Version 1.3 is released.
* I pull my co-worker's commits.

Here's the state of things at this point:
* The FooPackage currently checked out in my Packages directory is v1.1
* My lockfile points at FooPackage v1.2
* The latest FooPackage I could be using is v1.3

There are three things that should be possible at this point:
1. Build with the existing v1.1 dep.
2. Update deps to the lockfile version (v1.2).
3. Update deps and lockfile to the latest conforming version (1.3).

I'm simply saying that I'd expect vanilla `swift build` to perform option #1 (perhaps with a warning message that we're out of sync with the lockfile). Like you say, "expecting the build command to update the checked out dependencies every time seems unintuitive. It is after all a build command." So I'd expect it to build with what's already there. If I wanted to update deps to the lockfile version, I could be explicit: `swift build --use-locked-deps`.

My understanding, though, is that if a lockfile is present, `swift build` would operate like option #2 and I'd need to be explicit about my desire to ignore the lockfile (maybe with `swift build --ignore-locked-deps` or the like). This feels inconsistent in that sometimes `swift build` just builds what's there, and sometimes updates and builds depending on the state of the lockfile. It also feels a little wrong that vanilla `build` could checkout other branches without me passing `--something`.

But it certainly sounds like I'm in the minority here, so no big deal.

···

On Dec 22, 2015, at 2:50 PM, Max Howell <max.howell@apple.com> wrote:

--
-jemmons


(Adam Sharp) #11

It looks like the difference here between spm and the other dependency managers I'm familiar with (CocoaPods and Bundler) is that here we've got a single action ("build") that both installs dependencies and builds them, whereas in the others there is an "install" step that is separate.

Because those two things are combined into the one command, it's impossible for spm to correctly guess the user's intent in all situations. There may be a sensible default, however.

–Adam

···

On 23 Dec 2015, at 9:28 AM, Joshua Scott Emmons via swift-build-dev <swift-build-dev@swift.org> wrote:

There are three things that should be possible at this point:
1. Build with the existing v1.1 dep.
2. Update deps to the lockfile version (v1.2).
3. Update deps and lockfile to the latest conforming version (1.3).


(Max Howell) #12

Right. I didn't mean to insinuate we should be pulling fresh dependencies before each build. Here's the scenario

* I'm working on a package that depends on FooPackage Version(1,0,0)..<Version(2,0,0)
* I `swift --update`. This pulls FooPackage v1.1 and sets the lockfile. I commit the lockfile to git.
* FooPackage v1.2 is released
* A co-worker `swift --update`s to FooPackage v1.2, her lockfile is updated and she pushes it to git.
* Version 1.3 is released.
* I pull my co-worker's commits.

Here's the state of things at this point:
* The FooPackage currently checked out in my Packages directory is v1.1
* My lockfile points at FooPackage v1.2
* The latest FooPackage I could be using is v1.3

There are three things that should be possible at this point:
1. Build with the existing v1.1 dep.
2. Update deps to the lockfile version (v1.2).
3. Update deps and lockfile to the latest conforming version (1.3).

I'm simply saying that I'd expect vanilla `swift build` to perform option #1 (perhaps with a warning message that we're out of sync with the lockfile). Like you say, "expecting the build command to update the checked out dependencies every time seems unintuitive. It is after all a build command." So I'd expect it to build with what's already there. If I wanted to update deps to the lockfile version, I could be explicit: `swift build --use-locked-deps`.

Right, I understand now.

My understanding, though, is that if a lockfile is present, `swift build` would operate like option #2 and I'd need to be explicit about my desire to ignore the lockfile (maybe with `swift build --ignore-locked-deps` or the like). This feels inconsistent in that sometimes `swift build` just builds what's there, and sometimes updates and builds depending on the state of the lockfile. It also feels a little wrong that vanilla `build` could checkout other branches without me passing `--something`.

I agree, that the functionality is less simple and in fact it is ingenious of me to claim it just “builds”.

I think #2 is the correct choice because if the lockfile is present then the team that works here is stating that they *want* their team to use those dependencies. And they want everyone on their team to be using the exact same sources so any bugs reported do not have to take into account any variation in dependency sources.

However I would be happy to see a proposal for a per-project, per-user configuration to override this, or perhaps just a command line flag.

The tool should cater to all needs (if possible and practical), but with sensible defaults.

But it certainly sounds like I'm in the minority here, so no big deal.

Perhaps, but we’re all a minority in some workflow: development is a personal process. You should submit a proposal.


(Joshua Scott Emmons) #13

I think #2 is the correct choice because if the lockfile is present then the team that works here is stating that they *want* their team to use those dependencies. And they want everyone on their team to be using the exact same sources so any bugs reported do not have to take into account any variation in dependency sources.

Completely agree — except that according to the proposal, the lockfile is generated *every* time `swift build` is run (assuming a lockfile doesn't already exist). So we can't really treat its presence as indicating some sort of intention. It's going to be there all the time for everyone whether they understand it, want to use it, or not. Unless I'm missing something?

Actually, if we changed this part of the proposal such that the lockfile would *only* be generated explicitly (in response to `swift build --lock`, for example?), I'd be in agreement with you that vanilla `swift build` should always use the lockfile by default. For, in this case, there would be only two ways a lockfile could exist:

1) you explicitly create it
2) the team/maintainers of the code think it's important

In either case, building with the lockfile is clearly the Right Thing to do.

This feels like a much more fruitful compromise than per-project, per-user overrides… assuming anyone else is on board with forcing lockfile creation to be explicit. Thoughts?

···

--
-jemmons


(Ankit Agarwal) #14

Thank you all for the valuable feedback I have updated the proposal to
reflect the changes which are agreed upon by majority. Please go through
the updated proposal below and provide more feedback.

Lock File for Swift Package ManagerIntroduction

A Package.lock file containing list of resolved dependencies generated by
swiftpm.
Motivation

Package.lock file can be helpful in situations like :
Reproduce exact versions of dependencies on different machine

   - Multiple developers working on a package would want to use the exact
   versions(including minor and patch) of the dependencies declared in the
   manifest file
   - When a build is being performed on a remote machine eg CI

Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit
ref for which a tagged version is unavailable in cases such as :

   - Forking a 3rd party library and making it swiftpm compatible for
   temporary use until officially supported by the author
   - Package is in active development and not ready for a release tag

Proposed Solution

swiftpm generates a simple Package.lock file after resolving the dependency
graph for that package in some simple format.
Detailed DesignPackage.lock aka the lock file

Package.lock is a plain text auto-generated file containing list of
resolved dependencies: one line per dependency in format

<gitURL> <resolvedVersion>

"github.com/foo/bar" "v1.1"
"../mygitdir" "v2.3"
"github.com/my/forked/dep" "8b3989be184375ae6e84e8a0254e5258789b23e5"

   - Package.lock will always contain the exact version of the dependency
   resolved by SPM
   - User is expected to commit the lock file into the git repo for others
   to reproduce the exact versions of dependencies on their system
   - lock file of dependencies are ignored and only their Package.swift is
   taken into consideration for resolving all the dependencies

User is not expected to interact with this file as it'll always be
generated by SPM so it might make sense to:

   - not to use swift syntax in this file.
   - keep it seperate from Package.swift to avoid confusion for new users.

Another option could be using very minimal valid swift syntax for eg :

[
("github.com/foo/bar", "v1.1"),
("../mygitdir","v2.3"),
("github.com/my/forked/dep","8b3989be184375ae6e84e8a0254e5258789b23e5")
]

Workflow$ swift build :

···

-

   Dependencies declared in Package.swift and Package.lock is not present :

   Resolves Package.swift and generates Package.lock; uses the generated
   Package.lock to fetch and build the dependencies
   -

   Package.lock already present :

   Uses the generated Package.lock to fetch and build the dependencies.

   if Package.swift has been modified SPM generates a warning for user to
   consider updating the lock file using $ swift build --update (explained
   below)
   -

   Packages/ is out of sync with lock file :

   *Uncommited changes in Packages/* : User is probably trying to modify
   the dep so suggest him to use the $ swift build --ignore-lock command
   (explained below)

   *Commited changes in Packages/* : User is done making changes in the
   dependency but forgot to lock them so HEADs don't match, suggest
him to use $
   swift build --lock command (explained below)

$ swift build --update :

   - Remove current Package.lock if present and re-resolve Package.swift to
   generate a new Package.lock file for all the deps.
   - To update only one dependency use $ swift build --update=<dep_name>

$ swift build --lock :

   - Locks the current HEAD packages in Packages/ and writes them into the
   lock file uses tags where available otherwise refs
   - To lock only one dependency use $ swift build --lock=<dep_name>

$ swift build --ignore-lock :

   - Ignores the lock file and use HEAD of Package/ to build the deps
   without writing to lock file
   - Useful for development

Workflow for Pointing a dependency to an untagged commit

   - Create Package.swift and declare dependencies
   - Run $ swift build to resolve and build the Package
   - Modify and play with the checkedout dependencies in Packages/
   - Commit and push the changes to the git remote mentioned in
   Package.swift
   - Once satisfied with the current state of a package lock it using $
   swift build --lock=<dep_name>
   - Commit the lock file and push for others to use

If a dependency(foo) is using a dependency(bar) with an untagged commit

   - Author of foo mentions in their readme that they're using bar on an
   untagged commit
   - Package(baz) wanting to use foo and bar both will need to lock his bar
   package to same untagged commit

Impact on existing code

None on the code but old users will not be able to run $ swift build with
changes in their Packages/ which is possible right now, they'll need to use $
swift build --ignore-lock instead which will be communicated using warnings
as stated above.
Alternatives Considered

One alternative is to allow mentioning refs in manifest file while
declaring a dependency but as discussed in this
<file:///Users/aciid/Dropbox/notes/%22https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/>
thread
it might not be the best idea.


(Pierre Monod-Broca) #15

Personally, I often work offline, and currently I'm often working on a project and its dependencies at the same time.
For those two reasons, *in the current proposal state* #1 would be my preferred choice (with proper warnings).

I would totally support for #2 to be the default, if it doesn't touch dependencies with local modifications, committed or uncommitted, but rather simply warn about them. Detection of committed changes could be achieved with spm keeping a copy of the lock it used last, for exemple.

Also that makes me think there could be a warning if #3 hasn't been used in a while.

···

Le 23 déc. 2015 à 18:17, Max Howell via swift-evolution <swift-evolution@swift.org> a écrit :

Right. I didn't mean to insinuate we should be pulling fresh dependencies before each build. Here's the scenario

* I'm working on a package that depends on FooPackage Version(1,0,0)..<Version(2,0,0)
* I `swift --update`. This pulls FooPackage v1.1 and sets the lockfile. I commit the lockfile to git.
* FooPackage v1.2 is released
* A co-worker `swift --update`s to FooPackage v1.2, her lockfile is updated and she pushes it to git.
* Version 1.3 is released.
* I pull my co-worker's commits.

Here's the state of things at this point:
* The FooPackage currently checked out in my Packages directory is v1.1
* My lockfile points at FooPackage v1.2
* The latest FooPackage I could be using is v1.3

There are three things that should be possible at this point:
1. Build with the existing v1.1 dep.
2. Update deps to the lockfile version (v1.2).
3. Update deps and lockfile to the latest conforming version (1.3).

I'm simply saying that I'd expect vanilla `swift build` to perform option #1 (perhaps with a warning message that we're out of sync with the lockfile). Like you say, "expecting the build command to update the checked out dependencies every time seems unintuitive. It is after all a build command." So I'd expect it to build with what's already there. If I wanted to update deps to the lockfile version, I could be explicit: `swift build --use-locked-deps`.

--
Pierre


(Max Howell) #16

I think #2 is the correct choice because if the lockfile is present then the team that works here is stating that they *want* their team to use those dependencies. And they want everyone on their team to be using the exact same sources so any bugs reported do not have to take into account any variation in dependency sources.

Completely agree — except that according to the proposal, the lockfile is generated *every* time `swift build` is run (assuming a lockfile doesn't already exist). So we can't really treat its presence as indicating some sort of intention. It's going to be there all the time for everyone whether they understand it, want to use it, or not. Unless I'm missing something?

Actually, if we changed this part of the proposal such that the lockfile would *only* be generated explicitly (in response to `swift build --lock`, for example?), I'd be in agreement with you that vanilla `swift build` should always use the lockfile by default. For, in this case, there would be only two ways a lockfile could exist:

1) you explicitly create it
2) the team/maintainers of the code think it's important

In either case, building with the lockfile is clearly the Right Thing to do.

This feels like a much more fruitful compromise than per-project, per-user overrides… assuming anyone else is on board with forcing lockfile creation to be explicit. Thoughts?

I think it is reasonable to always create it, you don’t *have* to commit it and check-it-in.

By always creating it we are suggesting the file is important, and that you *should* check it in. Which is right, it is important, and you *should* check it in. However you don’t *have* to, and when it comes to development, it is important to offer choices.


(Max Howell) #17

Personally, I often work offline, and currently I'm often working on a project and its dependencies at the same time.
For those two reasons, *in the current proposal state* #1 would be my preferred choice (with proper warnings).

I think it is reasonable for the build to proceed if there is no network connectivity. Just with warnings, since it is possible it will fail/be broken/etc.


(Max Howell) #18

To some length I have discussed the filename issue with Rick and I think I agree with him that `.lock` was a bad choice from the packaging community in general.

This is a command line tool and there is a long history of .lock files meaning “the file is locked because it is in use, do not edit the file without the .lock extension” on UNIX.

Carthage agreed, they chose ‘.resolved’.

NPM creates a “npm-shrinkwrap.json” file (although it will only do this on demand). I like the name “shrink-wrap” since it conveys the ‘sealed’ nature of the process.

We should consider using a proper format like JSON and using its extension too.

···

On Jan 4, 2016, at 1:17 PM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi
any updates on this?

On Wednesday 23 December 2015, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:
Thank you all for the valuable feedback I have updated the proposal to reflect the changes which are agreed upon by majority. Please go through the updated proposal below and provide more feedback.

Lock File for Swift Package Manager
Introduction
A Package.lock file containing list of resolved dependencies generated by swiftpm.

Motivation
Package.lock file can be helpful in situations like :

Reproduce exact versions of dependencies on different machine

Multiple developers working on a package would want to use the exact versions(including minor and patch) of the dependencies declared in the manifest file
When a build is being performed on a remote machine eg CI
Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit ref for which a tagged version is unavailable in cases such as :

Forking a 3rd party library and making it swiftpm compatible for temporary use until officially supported by the author
Package is in active development and not ready for a release tag
Proposed Solution
swiftpm generates a simple Package.lock file after resolving the dependency graph for that package in some simple format.

Detailed Design
Package.lock aka the lock file
Package.lock is a plain text auto-generated file containing list of resolved dependencies: one line per dependency in format

<gitURL> <resolvedVersion>

"github.com/foo/bar <http://github.com/foo/bar>" "v1.1"
"../mygitdir" "v2.3"
"github.com/my/forked/dep <http://github.com/my/forked/dep>" "8b3989be184375ae6e84e8a0254e5258789b23e5"
Package.lock will always contain the exact version of the dependency resolved by SPM
User is expected to commit the lock file into the git repo for others to reproduce the exact versions of dependencies on their system
lock file of dependencies are ignored and only their Package.swift is taken into consideration for resolving all the dependencies
User is not expected to interact with this file as it'll always be generated by SPM so it might make sense to:

not to use swift syntax in this file.
keep it seperate from Package.swift to avoid confusion for new users.
Another option could be using very minimal valid swift syntax for eg :

[
("github.com/foo/bar <http://github.com/foo/bar>", "v1.1"),
("../mygitdir","v2.3"),
("github.com/my/forked/dep <http://github.com/my/forked/dep>","8b3989be184375ae6e84e8a0254e5258789b23e5")
]
Workflow
$ swift build :

Dependencies declared in Package.swift and Package.lock is not present :

Resolves Package.swift and generates Package.lock; uses the generated Package.lock to fetch and build the dependencies

Package.lock already present :

Uses the generated Package.lock to fetch and build the dependencies.

if Package.swift has been modified SPM generates a warning for user to consider updating the lock file using $ swift build --update (explained below)

Packages/ is out of sync with lock file :

Uncommited changes in Packages/ : User is probably trying to modify the dep so suggest him to use the $ swift build --ignore-lock command (explained below)

Commited changes in Packages/ : User is done making changes in the dependency but forgot to lock them so HEADs don't match, suggest him to use $ swift build --lock command (explained below)

$ swift build --update :

Remove current Package.lock if present and re-resolve Package.swift to generate a new Package.lock file for all the deps.
To update only one dependency use $ swift build --update=<dep_name>
$ swift build --lock :

Locks the current HEAD packages in Packages/ and writes them into the lock file uses tags where available otherwise refs
To lock only one dependency use $ swift build --lock=<dep_name>
$ swift build --ignore-lock :

Ignores the lock file and use HEAD of Package/ to build the deps without writing to lock file
Useful for development
Workflow for Pointing a dependency to an untagged commit

Create Package.swift and declare dependencies
Run $ swift build to resolve and build the Package
Modify and play with the checkedout dependencies in Packages/
Commit and push the changes to the git remote mentioned in Package.swift
Once satisfied with the current state of a package lock it using $ swift build --lock=<dep_name>
Commit the lock file and push for others to use
If a dependency(foo) is using a dependency(bar) with an untagged commit

Author of foo mentions in their readme that they're using bar on an untagged commit
Package(baz) wanting to use foo and bar both will need to lock his bar package to same untagged commit
Impact on existing code
None on the code but old users will not be able to run $ swift build with changes in their Packages/ which is possible right now, they'll need to use $ swift build --ignore-lock instead which will be communicated using warnings as stated above.

Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this <> thread it might not be the best idea.

--
Ankit


(Ankit Agarwal) #19

agreed that .lock extension might be misleading and should be renamed to
Package-resolved.extension

Do you have specific reason for considering json/json like format or
just want to stay scaleable in case of future requirements in that file?
As of now I am not sure what could be added more to the file except URL and
resolved version...perhaps resolved version range? but idk if that's
important to be written to file.

···

On Tuesday 5 January 2016, Max Howell <max.howell@apple.com> wrote:

To some length I have discussed the filename issue with Rick and I think I
agree with him that `.lock` was a bad choice from the packaging community
in general.

This is a command line tool and there is a long history of .lock files
meaning “the file is locked because it is in use, do not edit the file
without the .lock extension” on UNIX.

Carthage agreed, they chose ‘.resolved’.

NPM creates a “npm-shrinkwrap.json” file (although it will only do this on
demand). I like the name “shrink-wrap” since it conveys the ‘sealed’ nature
of the process.

We should consider using a proper format like JSON and using its extension
too.

On Jan 4, 2016, at 1:17 PM, Ankit Agarwal <ankit@ankit.im > <javascript:_e(%7B%7D,'cvml','ankit@ankit.im');>> wrote:

Hi
any updates on this?

On Wednesday 23 December 2015, Ankit Agarwal <ankit@ankit.im > <javascript:_e(%7B%7D,'cvml','ankit@ankit.im');>> wrote:

Thank you all for the valuable feedback I have updated the proposal to
reflect the changes which are agreed upon by majority. Please go through
the updated proposal below and provide more feedback.

Lock File for Swift Package ManagerIntroduction

A Package.lock file containing list of resolved dependencies generated by
swiftpm.
Motivation

Package.lock file can be helpful in situations like :
Reproduce exact versions of dependencies on different machine

   - Multiple developers working on a package would want to use the
   exact versions(including minor and patch) of the dependencies declared in
   the manifest file
   - When a build is being performed on a remote machine eg CI

Pointing a dependency to an untagged commit

Sometimes it might be helpful to lock a dependency to a particular commit
ref for which a tagged version is unavailable in cases such as :

   - Forking a 3rd party library and making it swiftpm compatible for
   temporary use until officially supported by the author
   - Package is in active development and not ready for a release tag

Proposed Solution

swiftpm generates a simple Package.lock file after resolving the
dependency graph for that package in some simple format.
Detailed DesignPackage.lock aka the lock file

Package.lock is a plain text auto-generated file containing list of
resolved dependencies: one line per dependency in format

<gitURL> <resolvedVersion>

"github.com/foo/bar" "v1.1"
"../mygitdir" "v2.3"
"github.com/my/forked/dep" "8b3989be184375ae6e84e8a0254e5258789b23e5"

   - Package.lock will always contain the exact version of the
   dependency resolved by SPM
   - User is expected to commit the lock file into the git repo for
   others to reproduce the exact versions of dependencies on their system
   - lock file of dependencies are ignored and only their Package.swift
   is taken into consideration for resolving all the dependencies

User is not expected to interact with this file as it'll always be
generated by SPM so it might make sense to:

   - not to use swift syntax in this file.
   - keep it seperate from Package.swift to avoid confusion for new
   users.

Another option could be using very minimal valid swift syntax for eg :

[
("github.com/foo/bar", "v1.1"),
("../mygitdir","v2.3"),
("github.com/my/forked/dep","8b3989be184375ae6e84e8a0254e5258789b23e5")
]

Workflow$ swift build :

   -

   Dependencies declared in Package.swift and Package.lock is not
   present :

   Resolves Package.swift and generates Package.lock; uses the generated
   Package.lock to fetch and build the dependencies
   -

   Package.lock already present :

   Uses the generated Package.lock to fetch and build the dependencies.

   if Package.swift has been modified SPM generates a warning for user
   to consider updating the lock file using $ swift build --update (explained
   below)
   -

   Packages/ is out of sync with lock file :

   *Uncommited changes in Packages/* : User is probably trying to modify
   the dep so suggest him to use the $ swift build --ignore-lock command
   (explained below)

   *Commited changes in Packages/* : User is done making changes in the
   dependency but forgot to lock them so HEADs don't match, suggest him to use $
   swift build --lock command (explained below)

$ swift build --update :

   - Remove current Package.lock if present and re-resolve Package.swift
   to generate a new Package.lock file for all the deps.
   - To update only one dependency use $ swift build --update=<dep_name>

$ swift build --lock :

   - Locks the current HEAD packages in Packages/ and writes them into
   the lock file uses tags where available otherwise refs
   - To lock only one dependency use $ swift build --lock=<dep_name>

$ swift build --ignore-lock :

   - Ignores the lock file and use HEAD of Package/ to build the deps
   without writing to lock file
   - Useful for development

Workflow for Pointing a dependency to an untagged commit

   - Create Package.swift and declare dependencies
   - Run $ swift build to resolve and build the Package
   - Modify and play with the checkedout dependencies in Packages/
   - Commit and push the changes to the git remote mentioned in
   Package.swift
   - Once satisfied with the current state of a package lock it using $
   swift build --lock=<dep_name>
   - Commit the lock file and push for others to use

If a dependency(foo) is using a dependency(bar) with an untagged commit

   - Author of foo mentions in their readme that they're using bar on an
   untagged commit
   - Package(baz) wanting to use foo and bar both will need to lock his
   bar package to same untagged commit

Impact on existing code

None on the code but old users will not be able to run $ swift build with
changes in their Packages/ which is possible right now, they'll need to use $
swift build --ignore-lock instead which will be communicated using
warnings as stated above.
Alternatives Considered

One alternative is to allow mentioning refs in manifest file while
declaring a dependency but as discussed in this thread it might not be
the best idea.

--
Ankit

--
Ankit


(Max Howell) #20

Another option that has occurred to me after talking with Tim Kientzle is to use git submodules

submodules lock other clones to a specific revision. Our Packages/ directory is already a directory of clones.

I have not given it a lot of thought yet, but there is possibly power in reusing this existing part of git.

Mailed here for input.