Advice on using SwiftPM for developing large projects


(Geordie J) #1

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

Cheers and thank you in advance,
Geordie


(Ankit Aggarwal) #2

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo
enhancement to Swift Package Manager and was referred to the mailing list
(thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate
modules. For iOS we had been managing this with various git submodules
containing Xcode projects. This worked great, but doesn’t gel with SwiftPM:
making a commit (until relatively recently a tag as well), pushing to
origin, waiting for SPM to download everything that is already on the local
machine, seeing that the entire thing doesn’t even compile, and repeating,
doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in
editable mode. But this has proven to be very frustrating too: I have spent
literally weeks fighting against changes made as new SPM versions are
released with different quirks that each time break how or whether that
workaround works. In that time I’m sure we could have gone a long way to
developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is
basically edit mode but it uses the existing sources (if present). Have you
tried it? If yes, I would love to hear about the issues you're having in
using it.

Multi-package repos certainly sound like the solution we’re looking for.
It was proposed over a year ago here: https://github.com/
ddunbar/swift-evolution/blob/multi-package-repos/proposals/
NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider
working on this proposal six months ago was I'd understood that a) the
proposal had been accepted in some way, and b) that active work was being
done on it already.

I’m not sure if either of those are true today. So as an
aside/meta-question, does anybody know of a resource that tracks progress
of evolution proposals?

We're actively working on a proposal for multi-package repositories and we
will email it to this list once a draft is ready!

I did find this commit by Ankit Aggarwal: https://github.com/
apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e96060
2c84eede5e which seems to tantalisingly point in the right direction, via
the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be
modified by the user.

So I’m looking for input as to whether we as individuals or as a community
can do anything to push progress on this proposal: whether there is another
way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for
review on swift-evolution. In general, we would love to see more proposals
from the community but in this case, we already have something that we want
to propose.

At this point I’d even be overjoyed with a solution that just spits out
the clang / swiftc / linker calls made by `swift build` so we could write
our own script and avoid re-working-around changes every release or so. We
really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository
and then declaring multiple products in it. It would be similar to mono
repository but then you can't use individual packages as a dependency.
Another option could be creating a package structure and adding symlinks to
the different repositories (which could be checked out relative to the
package).

We also have a slack channel
<https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html>
if you want to chat about some more workarounds.

···

On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev < swift-build-dev@swift.org> wrote:

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Joe DeCapo) #3

I posted earlier this week about tool I wrote for managing multiple repositories that could possibly help, at least for the short-term: https://github.com/JrGoodle/clowder

Kind of like Ankit suggested, you could create a root repo containing the Package config for the multiple products, and specify relative locations for all the various repos to be cloned in under the root repo. The example LLVM projects clowder.yaml config file has repo's nested in other ones (you'd just need to be careful with the clean commands because some options would delete the nested repo if you tell it to delete files ignored by git). https://github.com/JrGoodle/llvm-projects/blob/master/clowder.yaml

I didn't know about the top of tree development mode, but that probably seems like the way to go, and I think clowder would combine nicely with it as well. It would probably be useful for daily development, since you can have the repos all track branches and pull all the changes in one command. Or you can specify tags/commits to pin to as well.

I just got started with SwiftPM recently, so I haven't tried this out yet, but I'm gonna give it a go with some personal projects I've been working on.

P.S. Sorry for the blatant "marketing" of a personal project, but I think it's pretty useful and am dying for some outside feedback :slight_smile:

···

On Nov 30, 2017, at 11:10 AM, Ankit Aggarwal via swift-build-dev <swift-build-dev@swift.org> wrote:

On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Geordie J) #4

Hi Ankit, thanks for your quick reply.

I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :slight_smile: The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:

1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:

2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 and a seemingly endless cycle of frustration, particularly when combined with the next point:

3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).

Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)

4. The more recent bug (https://bugs.swift.org/browse/SR-6493) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.

6. Why are we dealing with all this complexity we don’t need?

The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.

Kind Regards,
Geordie

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

···

Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal@apple.com>:
On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #5

Hi Geordie,

Hi Ankit, thanks for your quick reply.

I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :slight_smile: The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!

I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

This is certainly our goal, getting it right requires tuning and time.

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:

1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:

2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 and a seemingly endless cycle of frustration, particularly when combined with the next point:

3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).

Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)

4. The more recent bug (https://bugs.swift.org/browse/SR-6493) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

I agree, this one is really frustrating me too and I think we need to get it resolved.

5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.

6. Why are we dealing with all this complexity we don’t need?

The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.

Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.
2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.

As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.

Kind Regards,
Geordie

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.

- Daniel

···

On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org> wrote:

Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal@apple.com <mailto:ankit_aggarwal@apple.com>>:
On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Joe DeCapo) #6

I played around with this and was able to get it to work for some personal projects I've been working on. I have a Vapor app that depends on 2 libraries I made. I'm using the clowder tool I wrote to manage cloning the repositories into the correct location, but you should be able to easily get it to work via symlinks.

Here's the global Package.swift: https://github.com/polka-dot-cat/BugHub-Global/blob/master/Package.swift
Here's the clowder.yaml config file: https://github.com/polka-dot-cat/BugHub-clowder/blob/master/versions/global/clowder.yaml

Here are the commands to clone everything and build:

mkdir BugHub
cd BugHub
clowder init git@github.com:polka-dot-cat/BugHub-clowder.git
clowder link -v global  # This links the custom clowder.yaml version containing this specific setup
clowder herd
cd BugHub-Global
swift build

It successfully built for me and I was still able to maintain the Package.swift files in the individual repositories for when I want to build them there. The Vapor app depends on some resources, so I'd still have to tweak things to get everything working correctly, but I mainly just wanted to see if I could get it to build. It's definitely a bit of a hack, but hopefully this helps.

···

On Dec 1, 2017, at 7:16 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org> wrote:

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.


(Geordie J) #7

Hi Daniel,

Thanks for your reply!

Hi Geordie,

Hi Ankit, thanks for your quick reply.

I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :slight_smile: The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!

I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

This is certainly our goal, getting it right requires tuning and time.

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:

1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:

2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 and a seemingly endless cycle of frustration, particularly when combined with the next point:

3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).

Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)

4. The more recent bug (https://bugs.swift.org/browse/SR-6493) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

I agree, this one is really frustrating me too and I think we need to get it resolved.

5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.

6. Why are we dealing with all this complexity we don’t need?

The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.

I agree with this point wholeheartedly. I just have the impression that, if SwiftPM were structured around the case of just “building” rather than on managing dependencies, our case would be trivial and every other case would be no more complex. I have been very surprised from the outset that even the most basic of SPM demos require initting a git repo. I have the impression I’m not alone here.

I don’t know enough about the behind the scenes nor the current SPM codebase to comment on how realistic this wish is, but as I alluded to in previous emails, I kind of wish there were two projects, “swift build” (which builds) and “swift package” (which manages deps). It seems the two are mixed together in a way that is currently very difficult (at least as a user) to disentangle. I’m not suggesting they’d have to be two codebases either. Those two commands already exist under one project as is, just their functionality doesn’t always match what is on the tin, in my opinion.

Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.

The docker case building for two platforms is definitely unorthodox, and I realise that optimising towards that is (very) unrealistic. But I don’t see anything unorthodox at all about the vision of just being able to:

- create a directory on an offline machine without git installed
- make a Package.swift and a Sources directory with some code in it
- type "swift build” to build it
- add this newly created package as a dependency of another one (its parent, say) via the filesystem, without any behind the scenes copying or magic required.

The git requirement (etc.) seems antithetical to Swift’s progressive disclosure principle. This is what I’m referring to filesystem-based packages being the simple “base case”. Structuring in this way would allow for the docker case and many others (running "swift build" from a more minimal ubuntu image, etc).

2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.

Can you think of a case where filesystem-based packages would break existing features or make the SPM codebase more complex? Because the more I write about this, the more I think they’d be the basis of any proposal I would like to make.

As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?

Hmm this may work, but it again seems to be straying into the land of an unpredictable level of complexity in practice. It’s unlikely to work with the docker environment, for example. And a many of the packages this would apply to would only used in a single project, so it’d be centralising things unnecessarily (== complexity). I could see great use for in this concept for smaller “toolset” library packages though, say with some helpers for scripting or async.

All this aside, it’d be hard to argue that this would be simpler than filesystem-based packages for the SPM codebase or for the user’s mental model.

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.

Great, thanks.

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.

Kind Regards,
Geordie

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.

Ok thanks for the info!

- Geordie

···

Am 01.12.2017 um 16:48 schrieb Daniel Dunbar <daniel_dunbar@apple.com>:

On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal@apple.com <mailto:ankit_aggarwal@apple.com>>:
On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

- Daniel

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #8

Hi Daniel,

Thanks for your reply!

Hi Geordie,

Hi Ankit, thanks for your quick reply.

I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :slight_smile: The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!

I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

This is certainly our goal, getting it right requires tuning and time.

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:

1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:

2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 and a seemingly endless cycle of frustration, particularly when combined with the next point:

3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).

Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)

4. The more recent bug (https://bugs.swift.org/browse/SR-6493) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

I agree, this one is really frustrating me too and I think we need to get it resolved.

5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.

6. Why are we dealing with all this complexity we don’t need?

The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.

I agree with this point wholeheartedly. I just have the impression that, if SwiftPM were structured around the case of just “building” rather than on managing dependencies, our case would be trivial and every other case would be no more complex. I have been very surprised from the outset that even the most basic of SPM demos require initting a git repo. I have the impression I’m not alone here.

I don’t know enough about the behind the scenes nor the current SPM codebase to comment on how realistic this wish is, but as I alluded to in previous emails, I kind of wish there were two projects, “swift build” (which builds) and “swift package” (which manages deps). It seems the two are mixed together in a way that is currently very difficult (at least as a user) to disentangle. I’m not suggesting they’d have to be two codebases either. Those two commands already exist under one project as is, just their functionality doesn’t always match what is on the tin, in my opinion.

Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.

The docker case building for two platforms is definitely unorthodox, and I realise that optimising towards that is (very) unrealistic. But I don’t see anything unorthodox at all about the vision of just being able to:

- create a directory on an offline machine without git installed
- make a Package.swift and a Sources directory with some code in it
- type "swift build” to build it
- add this newly created package as a dependency of another one (its parent, say) via the filesystem, without any behind the scenes copying or magic required.

The git requirement (etc.) seems antithetical to Swift’s progressive disclosure principle. This is what I’m referring to filesystem-based packages being the simple “base case”. Structuring in this way would allow for the docker case and many others (running "swift build" from a more minimal ubuntu image, etc).

Just to make sure I am on the same page — the first three of these already work, it is the last bullet point that you find objectionable? I think we are all in agreement that this is a pain point we would like to solve (the question is how).

2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.

Can you think of a case where filesystem-based packages would break existing features or make the SPM codebase more complex? Because the more I write about this, the more I think they’d be the basis of any proposal I would like to make.

The big concern is not wanting users to unintentionally end up in a situation where their package cannot be effectively shared or built reproducibly by other developers.

As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?

Hmm this may work, but it again seems to be straying into the land of an unpredictable level of complexity in practice. It’s unlikely to work with the docker environment, for example. And a many of the packages this would apply to would only used in a single project, so it’d be centralising things unnecessarily (== complexity). I could see great use for in this concept for smaller “toolset” library packages though, say with some helpers for scripting or async.

All this aside, it’d be hard to argue that this would be simpler than filesystem-based packages for the SPM codebase or for the user’s mental model.

How would you imagine filesystem based packages working — that you would just give SwiftPM the path to the package, and it would accept it if it finds a Package.swift there but no git repository?

- Daniel

···

On Dec 1, 2017, at 8:24 AM, Geordie J <geojay@gmail.com> wrote:

Am 01.12.2017 um 16:48 schrieb Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>>:

On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal@apple.com <mailto:ankit_aggarwal@apple.com>>:
On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.

Great, thanks.

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.

Kind Regards,
Geordie

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.

Ok thanks for the info!

- Geordie

- Daniel

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Geordie J) #9

Hi Daniel,

Thanks for your reply!

Hi Geordie,

Hi Ankit, thanks for your quick reply.

I am very sorry in advance for my frustrated tone in what follows. I realise I am an a-hole :slight_smile: The SPM team is doing really great things, it’s a useful product and once it’s stable it’s going to be an incredible tool for putting complex software together quickly.

We honestly appreciate the feedback about pain points, as long as the tone is respectful (which I think yours is) we are more than happy to hear constructive criticism!

I just wish there was a little more focus on simplicity and getting what I see as the "base case” right: that “swift build” just builds with no fuss, no mess, no maintenance, (no internet connection!) – everything else is a bonus. So I’d be really stoked to see that being prioritised.

This is certainly our goal, getting it right requires tuning and time.

Hello Swift Build Devs!

I filed a bug to track progress of the proposed Multi-Package Repo enhancement to Swift Package Manager and was referred to the mailing list (thanks to Jordan Rose for the tip!)

We have a reasonably complex project consisting of five or so separate modules. For iOS we had been managing this with various git submodules containing Xcode projects. This worked great, but doesn’t gel with SwiftPM: making a commit (until relatively recently a tag as well), pushing to origin, waiting for SPM to download everything that is already on the local machine, seeing that the entire thing doesn’t even compile, and repeating, doesn’t work for our daily active development.

The only viable workaround for this has been to put all of the packages in editable mode. But this has proven to be very frustrating too: I have spent literally weeks fighting against changes made as new SPM versions are released with different quirks that each time break how or whether that workaround works. In that time I’m sure we could have gone a long way to developing and releasing a fix instead.

In Swift 4, we introduced a top of the tree development mode, which is basically edit mode but it uses the existing sources (if present). Have you tried it? If yes, I would love to hear about the issues you're having in using it.

Yes, this is what we’ve been using for the said 6 months (since swift 4 pre-release). I think we’ve discussed some of the issues elsewhere but here there are again, maybe in more detail:

1. The package editable modes add the dependencies at an absolute path. This doesn’t gel with our workflow because we build for two different platforms from the same project using docker (which necessarily sees the same files at different absolute paths compared to the host machine). The suggested workaround for this was to use different build directories per platform, which would be ok, but has the issues of:

2. We are unable to put the packages in editable mode via the docker’s version of SPM because our git credentials for our private repos are not stored there (and they are public docker images we’d rather not have to “fork” per dev to add their individual credentials). For whatever reason git never asks for the credentials when called from SPM in that environment, it either fails silently or with an unhelpful error message. SPM then in many cases makes any previously editable packages non-editable again, leading to https://bugs.swift.org/browse/SR-6492 and a seemingly endless cycle of frustration, particularly when combined with the next point:

3. It also adds another unnecessary copy of the files onto the file system: one more source of truth, one more thing to set up in our editors (dirs to ignore etc), more mess with git management and more wasted disk space. But maybe the biggest killer here is the wasted time in downloading each repo – often for the 10th time because the above has failed somewhere and the only way of debugging is to delete the build directory and start again (some of our repos are hundreds of megabytes each and our internet connections are modest).

Once this whole process finally completes successfully (I’m not exaggerating that these first steps have consumed entire weeks of my year trying to get a repeatable build across our dev team), SPM proceeds to ignore the downloaded repo entirely and instead use the copy that had been on the filesystem all along(!!)

4. The more recent bug (https://bugs.swift.org/browse/SR-6493) where we can’t even put our packages in editable mode because the “identifier” doesn’t equal the directory name. Yes there is a workaround, but a workaround for a bug that only exists because of complexity unnecessary for the job at hand has a particularly sour taste to it. More mess, more maintenance.

I agree, this one is really frustrating me too and I think we need to get it resolved.

5. The fact that we cannot store this edited state in git (particularly due to point 1), requiring extra scripts that put the packages, which are already on the local filesystem, into editable mode. This step repeatedly causes dev downtime for us because of slight variations in git setup or a myriad of other unknowns. That and the script requires maintenance itself, which again feels like a band-aid on a band-aid.

6. Why are we dealing with all this complexity we don’t need?

The whole thing has a fragile feel to it, to say the least. And given that we’re not using any of the dependency-management of SPM whatsoever, it’s a particularly bitter pill to swallow.

One thing to be clear, your way of using SwiftPM is not the “golden path” that we believe most users are using. Deviating that is probably causing you to bump into more pain points and workflows that need tuning.

I agree with this point wholeheartedly. I just have the impression that, if SwiftPM were structured around the case of just “building” rather than on managing dependencies, our case would be trivial and every other case would be no more complex. I have been very surprised from the outset that even the most basic of SPM demos require initting a git repo. I have the impression I’m not alone here.

I don’t know enough about the behind the scenes nor the current SPM codebase to comment on how realistic this wish is, but as I alluded to in previous emails, I kind of wish there were two projects, “swift build” (which builds) and “swift package” (which manages deps). It seems the two are mixed together in a way that is currently very difficult (at least as a user) to disentangle. I’m not suggesting they’d have to be two codebases either. Those two commands already exist under one project as is, just their functionality doesn’t always match what is on the tin, in my opinion.

Thank you for writing up the concrete list of issues, that is very helpful. Here is my perspective on this:
1. As I said, I think your way of using SwiftPM is slightly unorthodox. One thing that means is we have to carefully balance the features you want versus their impact on all the other workflows.

The docker case building for two platforms is definitely unorthodox, and I realise that optimising towards that is (very) unrealistic. But I don’t see anything unorthodox at all about the vision of just being able to:

- create a directory on an offline machine without git installed
- make a Package.swift and a Sources directory with some code in it
- type "swift build” to build it
- add this newly created package as a dependency of another one (its parent, say) via the filesystem, without any behind the scenes copying or magic required.

The git requirement (etc.) seems antithetical to Swift’s progressive disclosure principle. This is what I’m referring to filesystem-based packages being the simple “base case”. Structuring in this way would allow for the docker case and many others (running "swift build" from a more minimal ubuntu image, etc).

Just to make sure I am on the same page — the first three of these already work, it is the last bullet point that you find objectionable? I think we are all in agreement that this is a pain point we would like to solve (the question is how).

Ok interesting, I just tried it and you’re right. This wasn’t the case with SPM in Swift 3 as far as I recall.

So yes, it’d be great if you could add a package dependency like this:

        .target(
            name: "spmtest",
            dependencies: [
                .package(relativePath: "anotherPackage")
            ]),

and have SwiftPM not blink an eye, not copy any files etc., just use those existing files as is. In our case “anotherPackage” would be a git submodule, but it shouldn’t matter either way (re: distributing packages, SPM seems to clone recursively as is anyway).

2. I agree there is a problem here. I myself encounter a similar situation relatively frequently with the work I am doing, and don’t yet know how to solve it.
3. Ultimately, the most constructive form of conversation is to try and identify individual features which (a) help and (b) integrate well with all the other features, and then write (or coauthor, if you want help) a proposal for it.

Can you think of a case where filesystem-based packages would break existing features or make the SPM codebase more complex? Because the more I write about this, the more I think they’d be the basis of any proposal I would like to make.

The big concern is not wanting users to unintentionally end up in a situation where their package cannot be effectively shared or built reproducibly by other developers.

I see the concern, which is why my contrived example above specifies “relativePath” explicitly. I do wonder how this is any different to a local filesystem URL though – do we not currently have the same issue there?

As one example, a feature that I have long believed makes sense for SwiftPM is a way for developers to have a global “set of packages I contribute to”, and then SwiftPM features to always use those versions when resolving dependencies, instead of manually managing them. I think such a feature might fit with your use case, and solve many of your problems. What do you think?

Hmm this may work, but it again seems to be straying into the land of an unpredictable level of complexity in practice. It’s unlikely to work with the docker environment, for example. And a many of the packages this would apply to would only used in a single project, so it’d be centralising things unnecessarily (== complexity). I could see great use for in this concept for smaller “toolset” library packages though, say with some helpers for scripting or async.

All this aside, it’d be hard to argue that this would be simpler than filesystem-based packages for the SPM codebase or for the user’s mental model.

How would you imagine filesystem based packages working — that you would just give SwiftPM the path to the package, and it would accept it if it finds a Package.swift there but no git repository?

Ideally yes, as above.

- Geordie

···

Am 01.12.2017 um 17:32 schrieb Daniel Dunbar <daniel_dunbar@apple.com>:

On Dec 1, 2017, at 8:24 AM, Geordie J <geojay@gmail.com <mailto:geojay@gmail.com>> wrote:

Am 01.12.2017 um 16:48 schrieb Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>>:

On Dec 1, 2017, at 5:16 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Am 30.11.2017 um 18:09 schrieb Ankit Aggarwal <ankit_aggarwal@apple.com <mailto:ankit_aggarwal@apple.com>>:
On Thu, Nov 30, 2017 at 8:37 AM, Geordie J via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

- Daniel

Multi-package repos certainly sound like the solution we’re looking for. It was proposed over a year ago here: https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md. I guess the reason I didn’t consider working on this proposal six months ago was I'd understood that a) the proposal had been accepted in some way, and b) that active work was being done on it already.

I’m not sure if either of those are true today. So as an aside/meta-question, does anybody know of a resource that tracks progress of evolution proposals?

We're actively working on a proposal for multi-package repositories and we will email it to this list once a draft is ready!

Is the one I linked to no longer valid? It’s been online for over a year now. In that kind of timeframe “once a draft is ready” quickly starts to sound purely placative. Is this a priority or a “nice to have”? How do we get involved with pushing this forward?

Rick (CC’d) is working on revising the next version of this proposal and will post it once ready.

Great, thanks.

I did find this commit by Ankit Aggarwal: https://github.com/apple/swift-package-manager/commit/1ff987c64c00e9dc60e040f6e960602c84eede5e which seems to tantalisingly point in the right direction, via the isLocal property. I couldn’t see any way of using it though.

The isLocal property is an implementation detail and is not expected to be modified by the user.

I figured as much, but is it possible to use it in any way? It appears to be exactly what we need, but I couldn’t see any public API exposing its use.

So I’m looking for input as to whether we as individuals or as a community can do anything to push progress on this proposal: whether there is another way around it, or what else we could do?

Once a draft is out, we can discuss it on this list and then put it up for review on swift-evolution. In general, we would love to see more proposals from the community but in this case, we already have something that we want to propose.

At this point I’d even be overjoyed with a solution that just spits out the clang / swiftc / linker calls made by `swift build` so we could write our own script and avoid re-working-around changes every release or so. We really don’t need any management of dependencies at all, just building.

One option could be consolidating all the submodules into one repository and then declaring multiple products in it. It would be similar to mono repository but then you can't use individual packages as a dependency. Another option could be creating a package structure and adding symlinks to the different repositories (which could be checked out relative to the package).

For the sake of the proposal / this discussion, I should mention that we’re working on a proprietary project with a major subcomponent that we intend to open-source as an SPM dependency. That as well as an internal module that structurally makes no sense to “live” in the root project (this is not the only root project that uses it). So we can’t really consolidate the submodules.

I do appreciate the suggestion though: I will play around with symlinking the various dependencies into the root project’s Sources directory and see if I can get SPM to build them all at that root level without destroying each individual repo’s Package.swift and structure. Although this, again, is a hack, it sounds like the most promising solution for now.

Kind Regards,
Geordie

We also have a slack channel <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160530/000497.html> if you want to chat about some more workarounds.

Is it the mailing list or the slack channel that is considered “canonical” in terms of evolution discussions?

The mailing list and the swift-evolution repository are the “authoritative” sources for discussion, Slack is great for working out details but ultimately any decisions should be communicated through one of the those.

Ok thanks for the info!

- Geordie

- Daniel

Cheers and thank you in advance,
Geordie

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev