Proposal: Git Branch and ref support for dependencies in Swift Package Manager

[Ankit_Agarwal1 (Ankit Agarwal)]

Hi,

Here is a proposal of the adding git branch support feature in SPM

*Introduction*

Pointing to branch or a commit ref for dependencies in Package.swift as
opposed to only a tagged release.

*Motivation*

* Try a package which is almost stable or useable but not yet ready for a
release/pre-release so not tagged (eg: new feature being introduced by a
library)
* While developing packages, one would want to point a package that uses
the package to a develop branch (eg: Developing Foo package, Bar uses Foo
and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while
developing/testing (eg: Fork a library not compatible with SPM to make it
compatible)
* One wants to point to some commit but doesn't have a branch/tag created
for that

*Proposed solution*

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git", branch:
"develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git", commit:
"d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

*Detailed design*

* Only the root Package.swift would be able to use branch/ref feature to
avoid dependency hell, any other dependency fetched in current Package
should not compile if that dependency contains another dependency pointing
to a branch/ref
* This feature should strictly be used for testing/developing purpose and
should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to
active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref, disregard
any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless
any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those
changes

*Impact on existing code*

None as this will be a new functionality

*Alternatives considered*

One option is to only allow a commit ref and not a branch so SPM will not
have to worry about fast forwarding but this is a desired feature.

···

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com> wrote:

> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an
implementation, we should agree on a design for how you'd do this. While
this isn't at the top of our priority list at the moment, we'd welcome both
design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift
evolution process at
https://github.com/apple/swift-evolution/blob/master/process.md\. We'd be
happy to discuss this here as part of your process for putting a proposal
together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers be
specified?

– Right now we require you to tag something as a versioned "release".
Should we require that you tag a branch before someone can make a package
depend on it? It could be convenient to be able to just depend on a branch,
but the meaning of depending on a branch changes over time as more commits
come in. Is it harmful to allow packages to depend on something that's not
an identified commit?

        – Note that we have yet to design our security story (
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\);
what we settle on there might require dependencies to be specified as a
specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a
different branch, without having to modify and commit a change to that
package's Package.swift?

– We may want to design a way for packages to support different versions
of the Swift language, as the language continues to change – e.g. a branch
of the package for the last released swift vs the current under development
swift snapshot. Is supporting dependencies on package branches a part of
how we'll do that?

Thanks,

        - Rick

--
Ankit

[Ankit_Agarwal1 (Ankit Agarwal)]

Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and
wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it out
in the current package

···

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

*Introduction*

Pointing to branch or a commit ref for dependencies in Package.swift as
opposed to only a tagged release.

*Motivation*

* Try a package which is almost stable or useable but not yet ready for a
release/pre-release so not tagged (eg: new feature being introduced by a
library)
* While developing packages, one would want to point a package that uses
the package to a develop branch (eg: Developing Foo package, Bar uses Foo
and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while
developing/testing (eg: Fork a library not compatible with SPM to make it
compatible)
* One wants to point to some commit but doesn't have a branch/tag created
for that

*Proposed solution*

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git", branch:
"develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git", commit:
"d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

*Detailed design*

* Only the root Package.swift would be able to use branch/ref feature to
avoid dependency hell, any other dependency fetched in current Package
should not compile if that dependency contains another dependency pointing
to a branch/ref
* This feature should strictly be used for testing/developing purpose and
should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to
active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref,
disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless
any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those
changes

*Impact on existing code*

None as this will be a new functionality

*Alternatives considered*

One option is to only allow a commit ref and not a branch so SPM will not
have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com> wrote:

> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of
SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an
implementation, we should agree on a design for how you'd do this. While
this isn't at the top of our priority list at the moment, we'd welcome both
design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift
evolution process at
https://github.com/apple/swift-evolution/blob/master/process.md\. We'd be
happy to discuss this here as part of your process for putting a proposal
together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers
be specified?

– Right now we require you to tag something as a versioned "release".
Should we require that you tag a branch before someone can make a package
depend on it? It could be convenient to be able to just depend on a branch,
but the meaning of depending on a branch changes over time as more commits
come in. Is it harmful to allow packages to depend on something that's not
an identified commit?

        – Note that we have yet to design our security story (
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\);
what we settle on there might require dependencies to be specified as a
specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a
different branch, without having to modify and commit a change to that
package's Package.swift?

– We may want to design a way for packages to support different versions
of the Swift language, as the language continues to change – e.g. a branch
of the package for the last released swift vs the current under development
swift snapshot. Is supporting dependencies on package branches a part of
how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

[Max_Howell]

Hi everyone,

I’d like to move this forward, I have the following points for discussion:

···

-----------

I would like to suspend this proposal’s submission pending update support which is due to be implemented after xcodeproj lands (which is done).

My rationale is: it impacts this proposal

-----------

I would like to suggest that the lockfile be placed as so:

    Packages/Lockfile

For me this fulfills the best of both worlds: it no longer seems to be a UNIX-style lockfile for the Package.swift file *and* it more clearly shows what precisely is being locked while continuing the perhaps unwise tradition of naming the file a “lockfile”

-----------

I would like to propose that initially the method that is used for altering branches, origins and pinning versions is not given an explicit CLI UI and instead the user must use explicit git commands in their Packages directory to set the “locks”.

The benefit here is that the process we are using is transparent, which allows developers more power over this feature and enables greater understanding over what happens.

Once the modifications are made the lock file can be regenerated by executing a command (TBD). Trying to run swift-build with modifications without first generating the lock results in a prominent warning, or perhaps even an error (TBD).

-----------

I would like the propose that any local changes to cloned packages that are NOT pinned to a commit are diff’d and the diff be embedded in the lockfile. This ensures that we:

1) Encourage users to edit, fix and improve their packages
2) Their changes get checked in and thus other people using the Package absolutely will get these changes without them having to explicitly publish their changes somewhere accessible to their whole team.

On Dec 14, 2015, at 10:43 AM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

Introduction

Pointing to branch or a commit ref for dependencies in Package.swift as opposed to only a tagged release.

Motivation

* Try a package which is almost stable or useable but not yet ready for a release/pre-release so not tagged (eg: new feature being introduced by a library)
* While developing packages, one would want to point a package that uses the package to a develop branch (eg: Developing Foo package, Bar uses Foo and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while developing/testing (eg: Fork a library not compatible with SPM to make it compatible)
* One wants to point to some commit but doesn't have a branch/tag created for that

Proposed solution

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git <http://git@example.com/Greeter.git&gt;&quot;, branch: "develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git <http://git@example.com/FooBar.git&gt;&quot;, commit: "d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

Detailed design

* Only the root Package.swift would be able to use branch/ref feature to avoid dependency hell, any other dependency fetched in current Package should not compile if that dependency contains another dependency pointing to a branch/ref
* This feature should strictly be used for testing/developing purpose and should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref, disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those changes

Impact on existing code

None as this will be a new functionality

Alternatives considered

One option is to only allow a commit ref and not a branch so SPM will not have to worry about fast forwarding but this is a desired feature.

[fluidsonic (Marc Knaup)]

+1 since I had to do exactly that a couple of times with CocoaPods.
There were issue in third party Pods which were fixed in a specific commit
or branch but the official release would still take a while. Delaying our
app release because of that was not an option and referring to a fixed yet
unreleased version the best solution.

···

On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" < swift-build-dev@swift.org> wrote:

Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and
wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it
out in the current package

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

*Introduction*

Pointing to branch or a commit ref for dependencies in Package.swift as
opposed to only a tagged release.

*Motivation*

* Try a package which is almost stable or useable but not yet ready for a
release/pre-release so not tagged (eg: new feature being introduced by a
library)
* While developing packages, one would want to point a package that uses
the package to a develop branch (eg: Developing Foo package, Bar uses Foo
and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while
developing/testing (eg: Fork a library not compatible with SPM to make it
compatible)
* One wants to point to some commit but doesn't have a branch/tag created
for that

*Proposed solution*

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git", branch:
"develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git", commit:
"d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

*Detailed design*

* Only the root Package.swift would be able to use branch/ref feature to
avoid dependency hell, any other dependency fetched in current Package
should not compile if that dependency contains another dependency pointing
to a branch/ref
* This feature should strictly be used for testing/developing purpose and
should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due
to active development of that dep and wants latest ref available in that
branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref,
disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref
unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those
changes

*Impact on existing code*

None as this will be a new functionality

*Alternatives considered*

One option is to only allow a commit ref and not a branch so SPM will not
have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com> wrote:

> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of
SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an
implementation, we should agree on a design for how you'd do this. While
this isn't at the top of our priority list at the moment, we'd welcome both
design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift
evolution process at
https://github.com/apple/swift-evolution/blob/master/process.md\. We'd
be happy to discuss this here as part of your process for putting a
proposal together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers
be specified?

– Right now we require you to tag something as a versioned "release".
Should we require that you tag a branch before someone can make a package
depend on it? It could be convenient to be able to just depend on a branch,
but the meaning of depending on a branch changes over time as more commits
come in. Is it harmful to allow packages to depend on something that's not
an identified commit?

        – Note that we have yet to design our security story (
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\);
what we settle on there might require dependencies to be specified as a
specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a
different branch, without having to modify and commit a change to that
package's Package.swift?

– We may want to design a way for packages to support different versions
of the Swift language, as the language continues to change – e.g. a branch
of the package for the last released swift vs the current under development
swift snapshot. Is supporting dependencies on package branches a part of
how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Kostiantyn_Koval]

+1 for adding this.

Ability to point to commit instead of a tag is crucial.
Often happens that a bug was fixed in the depended library but it wasn't released/tagged.
As well it helps during development.

Kostiantyn

···

On 14 Dec 2015, at 20:14, Marc Knaup via swift-build-dev <swift-build-dev@swift.org> wrote:

+1 since I had to do exactly that a couple of times with CocoaPods.
There were issue in third party Pods which were fixed in a specific commit or branch but the official release would still take a while. Delaying our app release because of that was not an option and referring to a fixed yet unreleased version the best solution.

On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it out in the current package

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

Introduction

Pointing to branch or a commit ref for dependencies in Package.swift as opposed to only a tagged release.

Motivation

* Try a package which is almost stable or useable but not yet ready for a release/pre-release so not tagged (eg: new feature being introduced by a library)
* While developing packages, one would want to point a package that uses the package to a develop branch (eg: Developing Foo package, Bar uses Foo and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while developing/testing (eg: Fork a library not compatible with SPM to make it compatible)
* One wants to point to some commit but doesn't have a branch/tag created for that

Proposed solution

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git <http://git@example.com/Greeter.git&gt;&quot;, branch: "develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git <http://git@example.com/FooBar.git&gt;&quot;, commit: "d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

Detailed design

* Only the root Package.swift would be able to use branch/ref feature to avoid dependency hell, any other dependency fetched in current Package should not compile if that dependency contains another dependency pointing to a branch/ref
* This feature should strictly be used for testing/developing purpose and should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref, disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those changes

Impact on existing code

None as this will be a new functionality

Alternatives considered

One option is to only allow a commit ref and not a branch so SPM will not have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:
> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an implementation, we should agree on a design for how you'd do this. While this isn't at the top of our priority list at the moment, we'd welcome both design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift evolution process at https://github.com/apple/swift-evolution/blob/master/process.md\. We'd be happy to discuss this here as part of your process for putting a proposal together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers be specified?

– Right now we require you to tag something as a versioned "release". Should we require that you tag a branch before someone can make a package depend on it? It could be convenient to be able to just depend on a branch, but the meaning of depending on a branch changes over time as more commits come in. Is it harmful to allow packages to depend on something that's not an identified commit?

        – Note that we have yet to design our security story (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\); what we settle on there might require dependencies to be specified as a specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a different branch, without having to modify and commit a change to that package's Package.swift?

– We may want to design a way for packages to support different versions of the Swift language, as the language continues to change – e.g. a branch of the package for the last released swift vs the current under development swift snapshot. Is supporting dependencies on package branches a part of how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Kostiantyn_Koval]

There is a plan to add ability to support fetching packages by url for development purpose described here
Importing Dependencies by Source URL form "PackageManagerCommunityProposal"
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#importing-dependencies-by-source-url

Kostiantyn

···

On 14 Dec 2015, at 22:36, Kostiantyn Koval <konstantin.koval1@gmail.com> wrote:

+1 for adding this.

Ability to point to commit instead of a tag is crucial.
Often happens that a bug was fixed in the depended library but it wasn't released/tagged.
As well it helps during development.

Kostiantyn

On 14 Dec 2015, at 20:14, Marc Knaup via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

+1 since I had to do exactly that a couple of times with CocoaPods.
There were issue in third party Pods which were fixed in a specific commit or branch but the official release would still take a while. Delaying our app release because of that was not an option and referring to a fixed yet unreleased version the best solution.

On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it out in the current package

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

Introduction

Pointing to branch or a commit ref for dependencies in Package.swift as opposed to only a tagged release.

Motivation

* Try a package which is almost stable or useable but not yet ready for a release/pre-release so not tagged (eg: new feature being introduced by a library)
* While developing packages, one would want to point a package that uses the package to a develop branch (eg: Developing Foo package, Bar uses Foo and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while developing/testing (eg: Fork a library not compatible with SPM to make it compatible)
* One wants to point to some commit but doesn't have a branch/tag created for that

Proposed solution

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git <http://git@example.com/Greeter.git&gt;&quot;, branch: "develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git <http://git@example.com/FooBar.git&gt;&quot;, commit: "d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

Detailed design

* Only the root Package.swift would be able to use branch/ref feature to avoid dependency hell, any other dependency fetched in current Package should not compile if that dependency contains another dependency pointing to a branch/ref
* This feature should strictly be used for testing/developing purpose and should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref, disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those changes

Impact on existing code

None as this will be a new functionality

Alternatives considered

One option is to only allow a commit ref and not a branch so SPM will not have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:
> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an implementation, we should agree on a design for how you'd do this. While this isn't at the top of our priority list at the moment, we'd welcome both design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift evolution process at https://github.com/apple/swift-evolution/blob/master/process.md\. We'd be happy to discuss this here as part of your process for putting a proposal together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers be specified?

– Right now we require you to tag something as a versioned "release". Should we require that you tag a branch before someone can make a package depend on it? It could be convenient to be able to just depend on a branch, but the meaning of depending on a branch changes over time as more commits come in. Is it harmful to allow packages to depend on something that's not an identified commit?

        – Note that we have yet to design our security story (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\); what we settle on there might require dependencies to be specified as a specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a different branch, without having to modify and commit a change to that package's Package.swift?

– We may want to design a way for packages to support different versions of the Swift language, as the language continues to change – e.g. a branch of the package for the last released swift vs the current under development swift snapshot. Is supporting dependencies on package branches a part of how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Ankit_Agarwal1 (Ankit Agarwal)]

Looks good.

What about getting packages that don't have any semver tags initially?
For eg I create a package and want to use it in a different package while
also developing the original package and I did a `git init` but not `git
tag v0.0.1`?

···

On Fri, Mar 4, 2016 at 1:37 AM, Max Howell <max.howell@apple.com> wrote:

Hi everyone,

I’d like to move this forward, I have the following points for discussion:

-----------

I would like to suspend this proposal’s submission pending update support
which is due to be implemented after xcodeproj lands (which is done).

My rationale is: it impacts this proposal

-----------

I would like to suggest that the lockfile be placed as so:

    Packages/Lockfile

For me this fulfills the best of both worlds: it no longer seems to be a
UNIX-style lockfile for the Package.swift file *and* it more clearly shows
what precisely is being locked while continuing the perhaps unwise
tradition of naming the file a “lockfile”

-----------

I would like to propose that initially the method that is used for
altering branches, origins and pinning versions is not given an explicit
CLI UI and instead the user must use explicit git commands in their
Packages directory to set the “locks”.

The benefit here is that the process we are using is transparent, which
allows developers more power over this feature and enables greater
understanding over what happens.

Once the modifications are made the lock file can be regenerated by
executing a command (TBD). Trying to run swift-build with modifications
without first generating the lock results in a prominent warning, or
perhaps even an error (TBD).

-----------

I would like the propose that any local changes to cloned packages that
are NOT pinned to a commit are diff’d and the diff be embedded in the
lockfile. This ensures that we:

1) Encourage users to edit, fix and improve their packages
2) Their changes get checked in and thus other people using the Package
absolutely will get these changes without them having to explicitly publish
their changes somewhere accessible to their whole team.

On Dec 14, 2015, at 10:43 AM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

*Introduction*

Pointing to branch or a commit ref for dependencies in Package.swift as
opposed to only a tagged release.

*Motivation*

* Try a package which is almost stable or useable but not yet ready for a
release/pre-release so not tagged (eg: new feature being introduced by a
library)
* While developing packages, one would want to point a package that uses
the package to a develop branch (eg: Developing Foo package, Bar uses Foo
and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while
developing/testing (eg: Fork a library not compatible with SPM to make it
compatible)
* One wants to point to some commit but doesn't have a branch/tag created
for that

*Proposed solution*

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git", branch:
"develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git", commit:
"d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

*Detailed design*

* Only the root Package.swift would be able to use branch/ref feature to
avoid dependency hell, any other dependency fetched in current Package
should not compile if that dependency contains another dependency pointing
to a branch/ref
* This feature should strictly be used for testing/developing purpose and
should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to
active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref,
disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless
any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those
changes

*Impact on existing code*

None as this will be a new functionality

*Alternatives considered*

One option is to only allow a commit ref and not a branch so SPM will not
have to worry about fast forwarding but this is a desired feature.

--
Ankit

[Ankit_Agarwal1 (Ankit Agarwal)]

I think even the source URL will also work on tags

···

On Tue, Dec 15, 2015 at 1:33 PM, Kostiantyn Koval < konstantin.koval1@gmail.com> wrote:

There is a plan to add ability to support fetching packages by url for
development purpose described here
Importing Dependencies by Source URL form
"PackageManagerCommunityProposal"

https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#importing-dependencies-by-source-url

Kostiantyn

On 14 Dec 2015, at 22:36, Kostiantyn Koval <konstantin.koval1@gmail.com> > wrote:

+1 for adding this.

Ability to point to commit instead of a tag is crucial.
Often happens that a bug was fixed in the depended library but it wasn't
released/tagged.
As well it helps during development.

Kostiantyn

On 14 Dec 2015, at 20:14, Marc Knaup via swift-build-dev < > swift-build-dev@swift.org> wrote:

+1 since I had to do exactly that a couple of times with CocoaPods.
There were issue in third party Pods which were fixed in a specific commit
or branch but the official release would still take a while. Delaying our
app release because of that was not an option and referring to a fixed yet
unreleased version the best solution.
On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" < > swift-build-dev@swift.org> wrote:

Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and
wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it
out in the current package

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

*Introduction*

Pointing to branch or a commit ref for dependencies in Package.swift as
opposed to only a tagged release.

*Motivation*

* Try a package which is almost stable or useable but not yet ready for
a release/pre-release so not tagged (eg: new feature being introduced by a
library)
* While developing packages, one would want to point a package that uses
the package to a develop branch (eg: Developing Foo package, Bar uses Foo
and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while
developing/testing (eg: Fork a library not compatible with SPM to make it
compatible)
* One wants to point to some commit but doesn't have a branch/tag
created for that

*Proposed solution*

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git", branch:
"develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git", commit:
"d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

*Detailed design*

* Only the root Package.swift would be able to use branch/ref feature to
avoid dependency hell, any other dependency fetched in current Package
should not compile if that dependency contains another dependency pointing
to a branch/ref
* This feature should strictly be used for testing/developing purpose
and should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due
to active development of that dep and wants latest ref available in that
branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref,
disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref
unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those
changes

*Impact on existing code*

None as this will be a new functionality

*Alternatives considered*

One option is to only allow a commit ref and not a branch so SPM will
not have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com> wrote:

> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of
SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on
an implementation, we should agree on a design for how you'd do this. While
this isn't at the top of our priority list at the moment, we'd welcome both
design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift
evolution process at
https://github.com/apple/swift-evolution/blob/master/process.md\. We'd
be happy to discuss this here as part of your process for putting a
proposal together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers
be specified?

– Right now we require you to tag something as a versioned "release".
Should we require that you tag a branch before someone can make a package
depend on it? It could be convenient to be able to just depend on a branch,
but the meaning of depending on a branch changes over time as more commits
come in. Is it harmful to allow packages to depend on something that's not
an identified commit?

        – Note that we have yet to design our security story (
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\);
what we settle on there might require dependencies to be specified as a
specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a
different branch, without having to modify and commit a change to that
package's Package.swift?

– We may want to design a way for packages to support different
versions of the Swift language, as the language continues to change – e.g.
a branch of the package for the last released swift vs the current under
development swift snapshot. Is supporting dependencies on package branches
a part of how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________

swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

--
Ankit

[Max_Howell]

Looks good.

What about getting packages that don't have any semver tags initially?
For eg I create a package and want to use it in a different package while also developing the original package and I did a `git init` but not `git tag v0.0.1`?

Personally my thinking for this use case has been a separate feature in Package.swift, eg (not concrete proposal):

Package(
    dependencies: [DevPackage(localPath: “../Foo”)])

[Ankit_Agarwal1 (Ankit Agarwal)]

I like this idea, Can I spawn this feature discussion to a new thread and
maybe try to write a proposal for it?

···

On Fri, Mar 4, 2016 at 1:53 AM, Max Howell <max.howell@apple.com> wrote:

Personally my thinking for this use case has been a separate feature in
Package.swift, eg (not concrete proposal):

Package(
    dependencies: [DevPackage(localPath: “../Foo”)])

--
Ankit

[ddunbar (Daniel Dunbar)]

Hi Ankit,

Thanks for starting work on this proposal, I agree this is a feature we sorely need. I think there are two problems this proposal is tackling, and it may be worth to call them out separately:

1. It may be necessary to point at an alternate "namespace" for pulling a dependency from (branch, etc.), because swiftpm support is not in the mainline.

2. It is useful when developing packages and hierarchies of packages to be able to point to non-tagged commits.

The reason I think it makes sense to separate these is that #1 is a problem which applies to all of the packages in a dependency tree. For example, some popular Swift projects may want to add support for swiftpm + Swift 2.2, but they can't put that into their mainline, or their mainline tags. One natural way to support that use case would be to have "namespace tags", like "swiftpm-1.2". That is something which deserves its own proposal, if anyone wants to tackle it.

For #2, I think we should encourage a usage model where the "Package.swift is the thing you should check in". We want to always encourage people to write manifests appropriate for sharing. The need to *override* the behavior strikes me as a mostly local issue, which should be addressed by a mechanism outside of the Package.swift.

Max and I just discussed this briefly, and came up with what I think is a pretty good strategy here:

1. Instead of tackling this problem in the manifest, start by coming up with a proposal for supporting "lock files". Something like:
- Support a Packages.lock file in the root folder.
- Define a simple syntax for associating package URLs with locked versions.
- Implement package manager support for using that file.

Once that is in place, you will already be able to use that file to solve this particular problem (you can just edit the lock file to point at a branch or at a commit). We already know we need "lock file" support, so this is also handy in that it is solving this problem with a generally useful feature.

2. Max had a great suggestion for how "locking" should work:
- "swift build --lock" (or however it is spelled) would simply create the .lock file based on the current state of your Packages tree.

This simple idea gives a great workflow for developing packages: you start by declaring your dependencies in your Package.swift, you use `swift build` to pull them down for you, then you start hacking on them (in your local checkout) to get the changes you need, iterating using `swift build --lock && swift build`, and then once you have something that works you can move to pushing those changes back and you already have the right .lock file

That is another feature to be proposed and implemented, but I think can be decoupled from the basic initial Packages.lock support.

What do you think?

- Daniel

···

On Dec 15, 2015, at 6:09 AM, Ankit Agarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

I think even the source URL will also work on tags

On Tue, Dec 15, 2015 at 1:33 PM, Kostiantyn Koval <konstantin.koval1@gmail.com <mailto:konstantin.koval1@gmail.com>> wrote:
There is a plan to add ability to support fetching packages by url for development purpose described here
Importing Dependencies by Source URL form "PackageManagerCommunityProposal"
https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#importing-dependencies-by-source-url

Kostiantyn

On 14 Dec 2015, at 22:36, Kostiantyn Koval <konstantin.koval1@gmail.com <mailto:konstantin.koval1@gmail.com>> wrote:

+1 for adding this.

Ability to point to commit instead of a tag is crucial.
Often happens that a bug was fixed in the depended library but it wasn't released/tagged.
As well it helps during development.

Kostiantyn

On 14 Dec 2015, at 20:14, Marc Knaup via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

+1 since I had to do exactly that a couple of times with CocoaPods.
There were issue in third party Pods which were fixed in a specific commit or branch but the official release would still take a while. Delaying our app release because of that was not an option and referring to a fixed yet unreleased version the best solution.

On Dec 14, 2015 8:04 PM, "Ankit Agarwal via swift-build-dev" <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Correcting one of the sentences in the Detailed design section

* If pointed to a branch, there might be two broad use cases
1. User wants to point a branch due to active development of that dep and wants latest ref available in that branch
2. User is actively developing a dep in that branch and want to test it out in the current package

On Tue, Dec 15, 2015 at 12:13 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:

Hi,

Here is a proposal of the adding git branch support feature in SPM

Introduction

Pointing to branch or a commit ref for dependencies in Package.swift as opposed to only a tagged release.

Motivation

* Try a package which is almost stable or useable but not yet ready for a release/pre-release so not tagged (eg: new feature being introduced by a library)
* While developing packages, one would want to point a package that uses the package to a develop branch (eg: Developing Foo package, Bar uses Foo and wants to point Foo dep to develop branch)
* One would want to point to his own fork but not create a release while developing/testing (eg: Fork a library not compatible with SPM to make it compatible)
* One wants to point to some commit but doesn't have a branch/tag created for that

Proposed solution

Allow refs and branch in Package.swift

let package = Package(
    name: "Hello",
    dependencies: [
        .Package(url: "ssh://git@example.com/Greeter.git <http://git@example.com/Greeter.git&gt;&quot;, branch: "develop", shouldFastForward: true),
        .Package(url: "ssh://git@example.com/FooBar.git <http://git@example.com/FooBar.git&gt;&quot;, commit: "d8ec7ca398a3ac3990477028117384d05ca7734e"),
    ]
)

Detailed design

* Only the root Package.swift would be able to use branch/ref feature to avoid dependency hell, any other dependency fetched in current Package should not compile if that dependency contains another dependency pointing to a branch/ref
* This feature should strictly be used for testing/developing purpose and should not be deployed to production environments

SPM could have the following behavior when running `swift build` :

* If pointed to a branch, there might be two use cases
Since there is a high probability that user wants to point a branch due to active development of that dep and wants latest ref available in that branch
If a dependency is not cloned, clone it and checkout that branch
If shouldFastForward is on -> Always try to be on the latest ref, disregard any local changes made to the checked out package
If shouldFastForward is false -> Always try to be on the latest ref unless any local changes made to the checked out package

* If pointed to a ref :
If that dependency is not cloned, clone it and checkout that ref.
Consecutive `swift build` will not affect the cloned package
If changes are made in the cloned repo, rebuild that package with those changes

Impact on existing code

None as this will be a new functionality

Alternatives considered

One option is to only allow a commit ref and not a branch so SPM will not have to worry about fast forwarding but this is a desired feature.

On Tue, Dec 8, 2015 at 4:24 AM, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:
> On Dec 5, 2015, at 5:59 AM, Ankit Agarwal <ankit@ankit.im <mailto:ankit@ankit.im>> wrote:
>
> Hi,
>
> Is pointing to a branch instead of version for a package in scope of SPM?
> if it is, I'd love to try to implement it

Hi Ankit,

This is in scope, though not yet designed. Prior to anyone working on an implementation, we should agree on a design for how you'd do this. While this isn't at the top of our priority list at the moment, we'd welcome both design contributions and eventual implementation.

If you'd like to put a proposal together for this, please see the Swift evolution process at https://github.com/apple/swift-evolution/blob/master/process.md\. We'd be happy to discuss this here as part of your process for putting a proposal together. Some things to think about in this area are:

– How should refs (branches or tags) that aren't simple version numbers be specified?

– Right now we require you to tag something as a versioned "release". Should we require that you tag a branch before someone can make a package depend on it? It could be convenient to be able to just depend on a branch, but the meaning of depending on a branch changes over time as more commits come in. Is it harmful to allow packages to depend on something that's not an identified commit?

        – Note that we have yet to design our security story (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#security-and-signing\); what we settle on there might require dependencies to be specified as a specific tagged commit, so that it can be signed.

– Should it be possible to override a package's dependency to use a different branch, without having to modify and commit a change to that package's Package.swift?

– We may want to design a way for packages to support different versions of the Swift language, as the language continues to change – e.g. a branch of the package for the last released swift vs the current under development swift snapshot. Is supporting dependencies on package branches a part of how we'll do that?

Thanks,

        - Rick

--
Ankit

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

--
Ankit

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Max_Howell]

2. Max had a great suggestion for how "locking" should work:
- "swift build --lock" (or however it is spelled) would simply create the .lock file based on the current state of your Packages tree.

This simple idea gives a great workflow for developing packages: you start by declaring your dependencies in your Package.swift, you use `swift build` to pull them down for you, then you start hacking on them (in your local checkout) to get the changes you need, iterating using `swift build --lock && swift build`, and then once you have something that works you can move to pushing those changes back and you already have the right .lock file

To add more detail here, I propose that, initially you `swift build`:

    $ swift build
    $ cd Packages
    $ ls
    FooBar-1.2.3

Then you can step into Packages/FooBar-1.2.3 and change the branch:

    $ git checkout mybranch

Then build against that branch

    $ swift build

Now, to ensure consumers of your terminal-package or other members of your team are using the same packages as you, you need to commit the lock file:

    $ git add ../../Packages.lock
    $ git commit -m Lock against devbranch for FooBar
    $ git push

Any .lock files that are cloned into dependencies are *ignored*. If a dependency depends on a branch of a package they should specify to the end-user that they will need to lock against that branch in their README.

The justification here is: dependency graphs are fragile, and introducing additional fragility should be avoided directly as part of our design going forward.

Now, we can add a convenience command line to do the above steps all in one, but exposing and documenting the mechanism transparently in the above manner offers consumers of swiftpm power and flexibility.

In addition the above workflow exposes a simple method for fixing issues in your dependencies directly without the need for convoluted tooling. Simply step into your Packages directory, make your changes and change the remote to your own fork, then lock.

Part of the proposal should thus make swiftpm emit warnings for modified packages without changes that are committed or pushed remotely. We don’t want people to accidentally push non-atomic changes to their dependency graph resulting in their teams not being able to build their projects.

Max

[Drew_Crawford1 (Drew Crawford)]

+1 for lockfiles, although I would implement them slightly differently.

Again, this is a place where looking to Cargo's history is useful. Cargo.toml is roughly analagous to Packages.swift:

[dependencies]
openssl-sys = { version = "0.5.0", git = "https://github.com/wycats/openssl-sys&quot; }

Then when you run `cargo update` it (in addition to building) generates a lock file:

[[package]]
name = "openssl-sys"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index&quot;

This lock file contains the exact software versions used to build. Subsequent invocation of `cargo build` use the version specified in the lockfile.

So to summarize:

1. `build` uses exact version specified in lockfile
2. `update` updates the lockfile to point to the newest versions compatible with the requirements in Packages.swift
3. lockfile always points to exact versions used

···

On Dec 15, 2015, at 2:30 PM, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

2. Max had a great suggestion for how "locking" should work:
- "swift build --lock" (or however it is spelled) would simply create the .lock file based on the current state of your Packages tree.

This simple idea gives a great workflow for developing packages: you start by declaring your dependencies in your Package.swift, you use `swift build` to pull them down for you, then you start hacking on them (in your local checkout) to get the changes you need, iterating using `swift build --lock && swift build`, and then once you have something that works you can move to pushing those changes back and you already have the right .lock file

To add more detail here, I propose that, initially you `swift build`:

    $ swift build
    $ cd Packages
    $ ls
    FooBar-1.2.3

Then you can step into Packages/FooBar-1.2.3 and change the branch:

    $ git checkout mybranch

Then build against that branch

    $ swift build

Now, to ensure consumers of your terminal-package or other members of your team are using the same packages as you, you need to commit the lock file:

    $ git add ../../Packages.lock
    $ git commit -m Lock against devbranch for FooBar
    $ git push

Any .lock files that are cloned into dependencies are *ignored*. If a dependency depends on a branch of a package they should specify to the end-user that they will need to lock against that branch in their README.

The justification here is: dependency graphs are fragile, and introducing additional fragility should be avoided directly as part of our design going forward.

Now, we can add a convenience command line to do the above steps all in one, but exposing and documenting the mechanism transparently in the above manner offers consumers of swiftpm power and flexibility.

In addition the above workflow exposes a simple method for fixing issues in your dependencies directly without the need for convoluted tooling. Simply step into your Packages directory, make your changes and change the remote to your own fork, then lock.

Part of the proposal should thus make swiftpm emit warnings for modified packages without changes that are committed or pushed remotely. We don’t want people to accidentally push non-atomic changes to their dependency graph resulting in their teams not being able to build their projects.

Max

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

[Ankit_Agarwal1 (Ankit Agarwal)]

Considering all the suggestions so far I am thinking of the following
workflow :

.lock file will contain all the resolved dependencies in some simple format
containing resolved version number or commit hash in case of untagged
package.

1. Running initial `$ swift build` fetches all the dependencies specified
in manifest file and generates a .lock file if not present and if present,
updates .lock only for a new dependency introduced or for some dependency
removed (or maybe not generate .lock at all?) but `$ swift build` always
uses local state of Packages dir

2. To update the .lock file run something like `$ swift build --lock` which
will always update the .lock file with local state of Packages dir.

3. Commit and push the .lock file so others can reproduce the same
environment using some command like `$ swift build --bootstrap` (like
carthage)

4. If some dependency depends on branch/commit hash (ie non-tagged commit)
the author mentions that in their readme and the end user and maybe other
parallel dependencies will have to use only that tag in order to avoid
dependency hell, this should probably be tackled with the #1 problem Daniel
mentioned (alternate namespaces)

5. This proposal will address only .lock file problem and pointing to a
different namespace will be taken care in a separate proposal.

6. One issue that'll arrive with solving #2 outside manifest file is user
wanting to use untagged dependency will have to enter some "fake" version
in the manifest file.

Does this sound consistent with what you guys have in mind?

···

On Wed, Dec 16, 2015 at 3:11 AM, Drew Crawford via swift-build-dev < swift-build-dev@swift.org> wrote:

+1 for lockfiles, although I would implement them slightly differently.

Again, this is a place where looking to Cargo's history is useful.
Cargo.toml is roughly analagous to Packages.swift:

[dependencies]
openssl-sys = { version = "0.5.0", git = "
https://github.com/wycats/openssl-sys&quot; }

Then when you run `cargo update` it (in addition to building) generates a
lock file:

[[package]]
name = "openssl-sys"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index&quot;

This lock file contains the exact software versions used to build.
Subsequent invocation of `cargo build` use the version specified in the
lockfile.

So to summarize:

1. `build` uses exact version specified in lockfile
2. `update` updates the lockfile to point to the newest versions
compatible with the requirements in Packages.swift
3. lockfile always points to exact versions used

On Dec 15, 2015, at 2:30 PM, Max Howell via swift-build-dev < > swift-build-dev@swift.org> wrote:

2. Max had a great suggestion for how "locking" should work:
- "swift build --lock" (or however it is spelled) would simply create the
.lock file based on the current state of your Packages tree.

This simple idea gives a great workflow for developing packages: you start
by declaring your dependencies in your Package.swift, you use `swift build`
to pull them down for you, then you start hacking on them (in your local
checkout) to get the changes you need, iterating using `swift build --lock
&& swift build`, and then once you have something that works you can move
to pushing those changes back and you already have the right .lock file

To add more detail here, I propose that, initially you `swift build`:

    $ swift build
    $ cd Packages
    $ ls
    FooBar-1.2.3

Then you can step into Packages/FooBar-1.2.3 and change the branch:

    $ git checkout mybranch

Then build against that branch

    $ swift build

Now, to ensure consumers of your terminal-package or other members of your
team are using the same packages as you, you need to commit the lock file:

    $ git add ../../Packages.lock
    $ git commit -m Lock against devbranch for FooBar
    $ git push

Any .lock files that are cloned into dependencies are *ignored*. If a
dependency depends on a branch of a package they should specify to the
end-user that they will need to lock against that branch in their README.

The justification here is: dependency graphs are fragile, and introducing
additional fragility should be avoided directly as part of our design going
forward.

Now, we can add a convenience command line to do the above steps all in
one, but exposing and documenting the mechanism transparently in the above
manner offers consumers of swiftpm power and flexibility.

In addition the above workflow exposes a simple method for fixing issues
in your dependencies directly without the need for convoluted tooling.
Simply step into your Packages directory, make your changes and change the
remote to your own fork, then lock.

Part of the proposal should thus make swiftpm emit warnings for modified
packages without changes that are committed or pushed remotely. We don’t
want people to accidentally push non-atomic changes to their dependency
graph resulting in their teams not being able to build their projects.

Max
_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

--
Ankit

[Max_Howell]

Personally my thinking for this use case has been a separate feature in Package.swift, eg (not concrete proposal):

Package(
    dependencies: [DevPackage(localPath: “../Foo”)])

I like this idea, Can I spawn this feature discussion to a new thread and maybe try to write a proposal for it?

If you have the time, then please feel free!

Otherwise we will get around to it eventually.

[Max_Howell]

Considering all the suggestions so far I am thinking of the following workflow :

.lock file will contain all the resolved dependencies in some simple format containing resolved version number or commit hash in case of untagged package.

1. Running initial `$ swift build` fetches all the dependencies specified in manifest file and generates a .lock file if not present and if present, updates .lock only for a new dependency introduced or for some dependency removed (or maybe not generate .lock at all?) but `$ swift build` always uses local state of Packages dir

2. To update the .lock file run something like `$ swift build --lock` which will always update the .lock file with local state of Packages dir.

3. Commit and push the .lock file so others can reproduce the same environment using some command like `$ swift build --bootstrap` (like carthage)

Per 1. `swift build` alone will do this. So unless `--bootstrap` has some other property, we probably don’t need it.

4. If some dependency depends on branch/commit hash (ie non-tagged commit) the author mentions that in their readme and the end user and maybe other parallel dependencies will have to use only that tag in order to avoid dependency hell, this should probably be tackled with the #1 problem Daniel mentioned (alternate namespaces)

Yes, this seems like a good compromise to me. Projects that depend on non-standard branches have a solution they can use, but it is in their interest to speed up resolution of whatever situation is causing them to require the branch because it is slightly inconvenient to them and their users.

I’m a big believer in making things possible, but creating social pressures that directs things for the overall good of the ecosystem.

5. This proposal will address only .lock file problem and pointing to a different namespace will be taken care in a separate proposal.

As I understood Daniel’s 1), it is a tagging namespace for projects that cannot or do not want higher order tags like 1.2.3 to be consumed as part of SwiftPM’s dependency resolution. I was thinking of tags like:

    swiftpm-1.2.3

Would be treated as version 1.2.3, if there is also a 1.2.3 tag, it will be ignored in preference to the swiftpm-1.2.3 tag if the swiftpm-1.2.3 tag is present.

This seems pretty simple, so maybe we can put it in with this proposal, otherwise

6. One issue that'll arrive with solving #2 outside manifest file is user wanting to use untagged dependency will have to enter some "fake" version in the manifest file.

Current discussion implies some `DevTarget` or similar concept in the manifest. This would override the need for git tags. However we need to flesh this out more as I have a number of concerns. After this perhaps! :)

Does this sound consistent with what you guys have in mind?

Yes, sounds just right. Thanks for writing this up. If you want to write up the proposal we can go from there. If you don’t have the time, I should have time early next week. Would be good to wait on Daniel for his opinion too, but he’s on break right now. It would be good to try and get this implemented sooner rather than later.

···

On Dec 19, 2015, at 8:41 AM, Ankit Agarwal <ankit@ankit.im> wrote:

[Ankit_Agarwal1 (Ankit Agarwal)]

Considering all the suggestions so far I am thinking of the following
workflow :

.lock file will contain all the resolved dependencies in some simple
format containing resolved version number or commit hash in case of
untagged package.

1. Running initial `$ swift build` fetches all the dependencies specified
in manifest file and generates a .lock file if not present and if present,
updates .lock only for a new dependency introduced or for some dependency
removed (or maybe not generate .lock at all?) but `$ swift build` always
uses local state of Packages dir

2. To update the .lock file run something like `$ swift build --lock`
which will always update the .lock file with local state of Packages dir.

3. Commit and push the .lock file so others can reproduce the same
environment using some command like `$ swift build --bootstrap` (like
carthage)

Per *1.* `swift build` alone will do this. So unless `--bootstrap` has
some other property, we probably don’t need it.

I think we need the bootstrap property because `swift build` will read the
local state of packages and builds the package where as bootstrap will be
always reading from lock file and reproducing *exact* versions of the
already present lock file. for eg :

Person A:
Runs `swift build`
fetches dependencies Foo at v1.2.*3* and Bar at v2.3.*4* and locks it.

Person B:
Runs `swift build`
fetches dependencies Foo at v1.2.*4* and Bar at v2.3.*6* (this can differ
in patch versions which might not be the intention as both want to
replicate exact versions, as running swift build will fetch the latest
patch and minor version if only major is specified)
but if he runs `swift build --bootstrap` he'll get Foo at v1.2.*3* and Bar
at v2.3.*4*

4. If some dependency depends on branch/commit hash (ie non-tagged commit)

the author mentions that in their readme and the end user and maybe other
parallel dependencies will have to use only that tag in order to avoid
dependency hell, this should probably be tackled with the #1 problem Daniel
mentioned (alternate namespaces)

Yes, this seems like a good compromise to me. Projects that depend on
non-standard branches have a solution they can use, but it is in their
interest to speed up resolution of whatever situation is causing them to
require the branch because it is slightly inconvenient to them and their
users.

I’m a big believer in making things possible, but creating social
pressures that directs things for the overall good of the ecosystem.

5. This proposal will address only .lock file problem and pointing to a
different namespace will be taken care in a separate proposal.

As I understood Daniel’s 1), it is a tagging namespace for projects that
cannot or do not want higher order tags like 1.2.3 to be consumed as part
of SwiftPM’s dependency resolution. I was thinking of tags like:

    swiftpm-1.2.3

Would be treated as version 1.2.3, if there is also a 1.2.3 tag, it will
be ignored in preference to the swiftpm-1.2.3 tag if the swiftpm-1.2.3 tag
is present.

This seems pretty simple, so maybe we can put it in with this proposal,
otherwise

Agreed

6. One issue that'll arrive with solving #2 outside manifest file is user
wanting to use untagged dependency will have to enter some "fake" version
in the manifest file.

Current discussion implies some `DevTarget` or similar concept in the
manifest. This would override the need for git tags. However we need to
flesh this out more as I have a number of concerns. After this perhaps! :)

Does this sound consistent with what you guys have in mind?

Yes, sounds just right. Thanks for writing this up. If you want to write
up the proposal we can go from there. If you don’t have the time, I should
have time early next week. Would be good to wait on Daniel for his opinion
too, but he’s on break right now. It would be good to try and get this
implemented sooner rather than later.

I was on break which ended now so I have plenty of time to do the proposal
as well as try to do the implementation. Should I draft the proposal and
raise a PR inside swift-package-manager/Documentation/Internals/ folder?

···

On Sat, Dec 19, 2015 at 11:27 PM, Max Howell <max.howell@apple.com> wrote:

On Dec 19, 2015, at 8:41 AM, Ankit Agarwal <ankit@ankit.im> wrote:

--
Ankit

[Max_Howell]

I think we need the bootstrap property because `swift build` will read the local state of packages and builds the package where as bootstrap will be always reading from lock file and reproducing exact versions of the already present lock file. for eg :

Person A:
Runs `swift build`
fetches dependencies Foo at v1.2.3 and Bar at v2.3.4 and locks it.

Person B:
Runs `swift build`
fetches dependencies Foo at v1.2.4 and Bar at v2.3.6 (this can differ in patch versions which might not be the intention as both want to replicate exact versions, as running swift build will fetch the latest patch and minor version if only major is specified)
but if he runs `swift build --bootstrap` he'll get Foo at v1.2.3 and Bar at v2.3.4

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

This makes understanding what happens simpler: `swift build` always uses the lock file if it is present.

This makes reliably building apps possible since you will always be building what everyone else built when the sources where committed.

[orta]

This behaviour is consistent with every dependency manager I’ve used, and really should be the way SPM works. If you build, it bases it off the lockfile, if you specifically decide to update, then you should say so.

···

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/&gt;CocoaPods <http://cocoapods.org/&gt; / CocoaDocs <gt3captcha-ios-xcframework - CocoaDocs.org; / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12&gt;
@orta <http://twitter.com/orta&gt; / orta.github.com <http://orta.github.com/&gt;
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer&gt; ATM

On 22 Dec 2015, at 18:47, Max Howell via swift-build-dev <swift-build-dev@swift.org> wrote:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

[Eloy_Duran]

In addition, people like to unlock specific dependencies and their implicit dependencies. E.g.:

1. Install dependencies as per the lockfile: $ swift build
2. Unlock all dependencies: $ swift build —update
3. Unlock specific dependency and its implicit dependencies: $ swift build —update [Dependency Name]

···

On 22 Dec 2015, at 20:53, orta therox via swift-build-dev <swift-build-dev@swift.org> wrote:

This behaviour is consistent with every dependency manager I’ve used, and really should be the way SPM works. If you build, it bases it off the lockfile, if you specifically decide to update, then you should say so.

--

[A.] Orta Therox

w/ Artsy <http://artsy.net/&gt;CocoaPods <http://cocoapods.org/&gt; / CocoaDocs <gt3captcha-ios-xcframework - CocoaDocs.org; / GIFs.app <https://itunes.apple.com/us/app/gifs/id961850017?l=en&mt=12&gt;
@orta <http://twitter.com/orta&gt; / orta.github.com <http://orta.github.com/&gt;
Artsy is totally hiring iOS Devs <https://artsy.net/job/mobile-engineer&gt; ATM

On 22 Dec 2015, at 18:47, Max Howell via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

If the lock file is committed I think swift-build should always use it. If the user wants newer updates they can execute `swift build —update`.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev