I have recently been building up some relatively large package trees (for example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to the source package. This lets me iterate very quickly without needing to push to an origin. Of course, it doesn’t work when pushing them to an actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to the origin from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo <http://github.com/ddunbar/foo>
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be github.com/dunbar/bar, which happens to be exactly what I want.
This even works for absolute path URLs, the other URL components would come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
One really nice thing about this scheme is that it simplifies mirroring of trees of packages. For example, if I wanted to clone the above examples onto, say, an enterprise SCM server, then as long as I cloned all of them into the same relative organization, I wouldn’t need to rewrite any of the URL references in the packages themselves during the clone.
I have recently been building up some relatively large package trees (for
example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to
the source package. This lets me iterate very quickly without needing to
push to an origin. Of course, it doesn’t work when pushing them to an
actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to
the *origin* from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be
github.com/dunbar/bar, which happens to be exactly what I want.
On top of my head, this could break in these cases:
1. I fork foo but not bar and now I can't build foo.
2. I "download" foo and have no origin/SCM.
Maybe we should require a fallback base URL?
This even works for absolute path URLs, the other URL components would
come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
I don't think I understand how would this work. We have no way of computing
the canonical path using the origin in this case.
···
On Fri, Jan 5, 2018 at 12:23 PM, Daniel Dunbar via swift-build-dev < swift-build-dev@swift.org> wrote:
One really nice thing about this scheme is that it simplifies mirroring of
trees of packages. For example, if I wanted to clone the above examples
onto, say, an enterprise SCM server, then as long as I cloned all of them
into the same relative organization, I wouldn’t need to rewrite any of the
URL references in the packages themselves during the clone.
Noting the concerns mentioned in the rest of the thread, I want to suggest that if you were going down this route the most sensible thing to do is probably to build URL’s using the WHATWG URL spec § 4.4 (URL Standard), which specifies an algorithm for combining a URL (whether relative-path or absolute) with a base URL. That gets you the maximum generality for this approach.
*That said*, I do wonder whether it would be better to support this approach differently, rather than allow the subtle and easy-to-break approach of relative URLs. For example, we could emulate Go’s GOPATH notion by providing a local mirror of the dependency tree as an environment variable, where SwiftPM would attempt to find the dependencies there before looking to the network.
Cory
···
On 5 Jan 2018, at 20:23, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:
I have recently been building up some relatively large package trees (for example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to the source package. This lets me iterate very quickly without needing to push to an origin. Of course, it doesn’t work when pushing them to an actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to the origin from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo <http://github.com/ddunbar/foo>
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be github.com/dunbar/bar <http://github.com/dunbar/bar>, which happens to be exactly what I want.
This even works for absolute path URLs, the other URL components would come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
One really nice thing about this scheme is that it simplifies mirroring of trees of packages. For example, if I wanted to clone the above examples onto, say, an enterprise SCM server, then as long as I cloned all of them into the same relative organization, I wouldn’t need to rewrite any of the URL references in the packages themselves during the clone.
I have recently been building up some relatively large package trees (for example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to the source package. This lets me iterate very quickly without needing to push to an origin. Of course, it doesn’t work when pushing them to an actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to the origin from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo <http://github.com/ddunbar/foo>
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be github.com/dunbar/bar <http://github.com/dunbar/bar>, which happens to be exactly what I want.
On top of my head, this could break in these cases:
1. I fork foo but not bar and now I can't build foo.
Yup. This is the flip side of the situation where you want to mirror a tree, when you actively want to only pull one part of a tree its more work. I don’t know which is more common and worth optimizing for.
2. I "download" foo and have no origin/SCM.
I’m not sure how often this happens. That feels like it might be specific to particular projects (ones that distribute source tarballs, e.g.), and they could just choose not to use the feature.
Maybe we should require a fallback base URL?
This even works for absolute path URLs, the other URL components would come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
I don't think I understand how would this work. We have no way of computing the canonical path using the origin in this case.
I don’t follow. This is the same as the other case, it only works if we do have an origin URL.
- Daniel
···
On Jan 5, 2018, at 1:00 PM, Ankit Agarwal <ankit@ankit.im> wrote:
On Fri, Jan 5, 2018 at 12:23 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
One really nice thing about this scheme is that it simplifies mirroring of trees of packages. For example, if I wanted to clone the above examples onto, say, an enterprise SCM server, then as long as I cloned all of them into the same relative organization, I wouldn’t need to rewrite any of the URL references in the packages themselves during the clone.
I have recently been building up some relatively large package trees (for example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to the source package. This lets me iterate very quickly without needing to push to an origin. Of course, it doesn’t work when pushing them to an actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to the origin from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo <http://github.com/ddunbar/foo>
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be github.com/dunbar/bar <http://github.com/dunbar/bar>, which happens to be exactly what I want.
This even works for absolute path URLs, the other URL components would come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
One really nice thing about this scheme is that it simplifies mirroring of trees of packages. For example, if I wanted to clone the above examples onto, say, an enterprise SCM server, then as long as I cloned all of them into the same relative organization, I wouldn’t need to rewrite any of the URL references in the packages themselves during the clone.
Thoughts?
Noting the concerns mentioned in the rest of the thread, I want to suggest that if you were going down this route the most sensible thing to do is probably to build URL’s using the WHATWG URL spec § 4.4 (URL Standard), which specifies an algorithm for combining a URL (whether relative-path or absolute) with a base URL. That gets you the maximum generality for this approach.
Thanks for the reference!
*That said*, I do wonder whether it would be better to support this approach differently, rather than allow the subtle and easy-to-break approach of relative URLs. For example, we could emulate Go’s GOPATH notion by providing a local mirror of the dependency tree as an environment variable, where SwiftPM would attempt to find the dependencies there before looking to the network.
Yeah, my takeaway from the feedback so far is “this isn’t a good idea”.
I do think the GOPATH convention is really convenient in many situations, I’d be curious to explore how this might look. It has never occurred to me that GOPATH can be used to prototype potential layouts for source repositories, even before they have been created/pushed.
- Daniel
···
On Jan 8, 2018, at 2:16 AM, Cory Benfield <cbenfield@apple.com <mailto:cbenfield@apple.com>> wrote:
On 5 Jan 2018, at 20:23, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
I have recently been building up some relatively large package trees (for example, porting Boost to build with SwiftPM).
The way I do this is that locally I use remote URLs which are relative to the source package. This lets me iterate very quickly without needing to push to an origin. Of course, it doesn’t work when pushing them to an actual SCM server.
However, what if we allowed relative-path URLs to be resolved relative to the origin from which SwiftPM found them? Concretely:
If I have:
github.com/ddunbar/foo
and it’s Package.swift has:
dependencies: [.package(url: “../bar”, …)]
then the idea is to make SwiftPM resolve ../bar to be github.com/dunbar/bar, which happens to be exactly what I want.
On top of my head, this could break in these cases:
1. I fork foo but not bar and now I can't build foo.
Yup. This is the flip side of the situation where you want to mirror a tree, when you actively want to only pull one part of a tree its more work. I don’t know which is more common and worth optimizing for.
This seems like a very common situation. I fork many libraries to make modifications and open pull requests. I would be very surprised if the very act of forking broke my ability to build with SwiftPM.
2. I "download" foo and have no origin/SCM.
I’m not sure how often this happens. That feels like it might be specific to particular projects (ones that distribute source tarballs, e.g.), and they could just choose not to use the feature.
This is also very common IMHO. If I want to quickly test a new library, I don’t go through the trouble of cloning the repo: I simple use the button on GitHub to download the latest version of master and test the library from my downloaded folder. I do this because many libraries (on iOS) come with example executable that demo the library functionality.
···
On 6 Jan 2018, at 02:10, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:
On Jan 5, 2018, at 1:00 PM, Ankit Agarwal <ankit@ankit.im> wrote:
On Fri, Jan 5, 2018 at 12:23 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:
Maybe we should require a fallback base URL?
This even works for absolute path URLs, the other URL components would come from the origin, so I could also reference:
dependencies: [.package(url: “/apple/swift-package-manager”)]
if I wanted to depend on SwiftPM.
I don't think I understand how would this work. We have no way of computing the canonical path using the origin in this case.
I don’t follow. This is the same as the other case, it only works if we do have an origin URL.
- Daniel
One really nice thing about this scheme is that it simplifies mirroring of trees of packages. For example, if I wanted to clone the above examples onto, say, an enterprise SCM server, then as long as I cloned all of them into the same relative organization, I wouldn’t need to rewrite any of the URL references in the packages themselves during the clone.