On Fri, Mar 18, 2016 at 9:45 AM Ben Rimmington via swift-evolution < swift-evolution@swift.org> wrote:
<
https://github.com/apple/swift-package-manager/blob/master/Documentation/Internals/SwiftBasedManifestFormat.md#discussion
>
We decided to use a Swift-based format for the manifest because we believe
it gives developers the best experience for working with and describing
their project. The primary alternative we considered was to use a
declarative format encoded in a common data format like JSON. Although that
would simplify implementation of the tooling around the manifest, it has
the downside that users must then learn this additional language, and the
development of high quality tools for that (documentation, syntax coloring,
parsing diagnostics) isn't aligned with our goal of building great tools
for Swift. In contrast, using the Swift language means that we can leverage
all of the work on Swift to make those tools great.
Could you generate a similar Swift-based format for the lockfile?
-- Ben
On 17 Mar 2016, at 18:23, Max Howell via swift-evolution < > swift-evolution@swift.org> wrote:
The following is a draft proposal, feedback welcome.
____________
SwiftPM Dependency Version Locking
- Proposal: SE-NNNN
<https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-dependency-lockfiles.md>
- Author(s): Ankit Agarwal <https://github.com/aciidb0mb3r>, Max Howell
<https://github.com/mxcl>
- Status: *Discussion*
- Review manager: Rick Ballard
Introduction
This proposal seeks to declare a new, generated file
Packages/VersionLocks.json that describes the exact state of a package’s
dependency graph and then by default will be respected when executing most
package manager commands. Thus it is considered a “version lock” for a
package’s dependency sources.
Swift-evolution thread
<https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/000067.html>
Terminology
- A *package* refers to a published, versioned git repository designed
to be consumed as a dependency by SwiftPM.
- A *project* refers to an end-user workspace that uses SwiftPM (via a
Package.swift and swift build) fetching and building packages as part
of its build
Describing this distinction is required because both the above have the
same form, but are used differently by an end-user. An end-user may publish
packages, but will eventually consume those packages in a project.
As justification for this confusion, it is considered a feature that
projects can easily and trivially become packages when using SwiftPM.
Encouraging a vibrant packaging ecosystem is one of our goals.
Motivation
In a vibrant packaging ecosystem, dependencies update continuously with
bug-fixes and new features. A development team needs:
1. To ensure they are all using the same versions of their
dependencies for any given version-control commit.
2. Ensure they are all using the same versions of the underlying Swift
toolchain
3. Be able to override or modify dependency specifications for the
whole team for specific commits.
Currently with SwiftPM it is possible to fulfill *1.* by committing the
sources of a package’s dependencies with the package itself, but this is
not always desirable. There is no way to achieve 2. and 3. with SwiftPM
alone.
Additionally, there is not currently a way to know which version of Swift
a package requires to build. At this time this situation is particularly
precarious because Swift itself is not backwards compatible. As a Swift
developer at the very least recording which Swift version a package was
built with by the package developer is essential information in order to
assess a package's suitability. Practically the package manager could in
the future use this information to aid an end-user or even fix the problem
when packages fail to compile.
Proposed Solution
A file: Packages/VersionLocks.json will be created alongside the
Package.swift file. Its contents will describe:
- The URL and versions of cloned dependencies
- An inline diff of any *local* modifications made to those packages
relative to their pristine cloned states
- The exact version of the Swift toolchain used as part of the last
successful build of the package
This file is generated by SwiftPM.
This file *should* be checked-in with projects.
This file is *generated* and should not be edited by users. If the file
is edited by users the behavior is undefined.
This file *should* be checked-in with packages designed for consumption
in projects, *however* SwiftPM will not use the checkout files of
dependencies when determining a project’s dependency graph (this would make
dependency graphs much less likely to resolve due to overly strict
versioning requirements). In the future we may choose to make it possible
for end-users to attempt to build a package using all checkout files since
in certain deployment scenarios where an exact graph has already been
tested, this is a solid reliabiity feature.
Any local, modifications made to the clones in Packages are recorded in
Packages/VersionLocks.json as part of the flow described in the next
section. Modifications here means: changes to git remotes and the git-ref
of the checked-out HEAD.
Detailed Design
In a fresh clone that does not contain a Packages directory swift build will
determine the dependency graph, clone the packages into Packages and
generate a Packages/VersionLocks.json file.
The user can now step into the Packages directory and modify package
sources. If the user then runs swift build again the package manager will
error out:
error: dependency sources have been modified
execute `swift build --lock` or `swift build --ignore-lock`
It is an error to build against an unlocked dependency graph, but to
facilitate fixing bugs etc. an ignore flag can be specified.
When swift build --lock is specified the package manager regenerates the
lockfile detailing the active git remote and the SHA that is checked-out.
Every time swift build completes a build the lockfile is updated (if
necessary) recording the current version of the Swift toolchain that
achieved the build.
Packages/VersionLocks.json
The exact design of the contents of this file will be explored during
iterative development, but here is a possible example:
json { "packages": [ { "clone": "Packages/PromiseKit-3.0.3", "origin": "
https://github.com/mxcl/PromiseKit" "ref": "3.0.3" }, { "clone":
"Packages/Alamofire-1.2.3", "origin": "
https://github.com/a-fork-somewhere/Alamofire" "ref": "crucial-fix" }, {
"clone": "Packages/Quick-1.2.3", "origin": "https://github.com/Quick/Quick"
"ref": "1.2.3" } ] }
Workflow — Regular Build
1. User runs swift build
2. If Packages/ contains clones and a VersionLocks.jsonSwiftPM skips
to 7.
3. If Packages/ contains clones and no VersionLocks.json the lockfile
is generated from the clones
4. If Packages/ contains checked out sources without git information
and no VersionLocks.json SwiftPM fetches the git information and
provided there is no diff, generates the Lockfile, if there is variation it
is an error *
5. If Packages/VersionLocks.json is present its dependency graph is
used
6. If Packages doesn't exist or is empty the dependency graph is
resolved, packages are cloned and the Lockfile is generated
7.
Build, if Packages are missing because we skipped from 2. the build
will error, it is the user's responsibility to instruct SwiftPM to
--update or to fix their dependency graph some other way.
-
This scenario is so users can check in their complete dependency
sources to their tree instead of / as well as the VersionLocks.json file:
a situation which sometimes is necessary if your dependencies are removed
from their third party online location, etc.
Workflow — Making Modifications
1. User makes local modification to a dependency’s sources
2. User runs swift build
3. swift build errors out.
4. User must either lock the graph or run with --ignore-lock
The error-out is likely to be considered tedious by users, however we
consider it important that users are made aware and forced to act when they
modify their dependencies and thus are exposing their team/users to
so-called “dependency hell”.
Runing swift build --lock regenerates the lockfile, but does not build.
Modifications must be committed. This means that if the modifications are
not uploaded to a location accessible to the rest of the team they will
fail to build when they update their checkouts.
The package manager could check for this by asking git if the specified
origin has the current locked ref and error out as appropriate.
Workflow — Overriding Packages
1. User steps into a Package directory eg. Packages/Foo-1.2.3
2. User changes the origin of Foo to their own fork
3. User alters HEAD to point to a fix in their own fork
4. swift build errors out.
5. User must either lock the graph or run with --ignore-lock
Running swift build --lock regenerates the lockfile, the new origin and
tag is stored. Thus a fresh clone of this project would use these overrides.
It is important to note that this workflow will not be respected for
dependencies, only for projects.
If a package author requires an override they have a few options:
1. Change the Package.swift dependency specification. This should only
be done as a last resort, for example, a critical bug must be fixed in a
dependency and that dependency author is not being responsive. It is up to
the Package author to ensure this scenario goes well. SwiftPM itself wants
to guard against these conditions with our proposed “publish & lint” step
that validates such decisions before signing a published package tag. But
we are not there yet and thus package authors should be responsible.
2. Advise end-users in a package README that they should override the
dependency themselves.
*2* is preferred, but *1* will happen. We consider it our responsibility
to develop tooling that makes *1.* safe or unnecessary, but we are not
there yet.
Workflow — Updating Packages
SwiftPM has no update mechanism yet, but once it does running swift build
--update will fetch the latest versions of all dependencies and update
the lockfile.
Impact on existing code
This proposal will have no impact on existing code.
Alternatives Considered
One alternative is to allow mentioning refs in manifest file while
declaring a dependency but as discussed in this
<http://markdownlivepreview.com/"https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20151214/> thread
it might not be the best idea.
Using Git submodules for this feature was considered. However something
additionally would be required to specify swift version and record local
diffs. Also this would lock us into git, and despite the fact that
currently we only use git, we have not yet ruled out supporting other
version control systems.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution
