SwiftPM manual dependency management

Geordie_J · July 20, 2017, 5:04pm

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule http://blah.com/mysubmodule.git\` <http://blah.com/mysubmodule.git`> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM still unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.json`, meaning this setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.

tldr; *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of .Package.local(named: "XYZ") that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.

Aciid · July 21, 2017, 5:33am

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex
app with multiple dependencies, all of which are being developed locally
and in parallel. The reason for this is compatibility with an existing
module/import structure used by our iOS app. Maybe I’m doing something very
wrong but my experience so far (2 months in) is that this is extremely
difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule
http://blah.com/mysubmodule.git\` in the Packages subdirectory and SwiftPM
would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option
for this purpose. So far my experience with this has not been good. Firstly
because SwiftPM *still* unnecessarily tries to clone my repos itself
(some of which are huge), and secondly because this creates an absolute
path dependency in `.build/dependencies-state.json`, meaning this setup
isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of
complexity, having to tag each commit for SwiftPM to build. The fact that
we'd need to make a commit to test whether the project even builds is
insane enough as is, let alone the tagging and trying to tell the base
project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies
(as dynamic libraries) really are shared between multiple
targets/sub-dependencies, which SwiftPM seems to deal with quite well.

*tldr;* *Please* let us manage dependencies ourselves. It’d be so easy if
Package.swift had an option along the lines of *.Package.local(named:
"XYZ")* that it then looked for in ./Packages/XYZ. Again, maybe I’m
overlooking something but this seems like an obvious and vital option to
have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple
and painless as it sounds? I would be prepared to make a pull request along
these lines.

I think you're not really using the Top of Tree feature. You need to add
each dependency using its canonical URL, hosted at some server like github.
After adding the dependencies, you can use edit feature to put a dependency
in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path
../path/to/self/managed/checkout/of/the/package

The package manager will then stop using the cloned repository and use the
checkout present at that path (regardless of the state it is in). Sharing
this setup is not automatic, but simple. Each user just needs to run the
above command once per dependency. Also, you only need to do this if you're
actively working on a dependency. The new manifest also supports using
branch instead of version range, which is very helpful during the
development period. Let me know if something is unclear or if you have more
questions!

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in
the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just
left us alone.. We were able to commit `workspace-state.json` into our base
project’s git repo and the rest Just Worked™. Now with the absolute paths
being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not
stable and will change without notice.

···

On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users < swift-users@swift.org> wrote:

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Geordie_J · July 21, 2017, 12:39pm

Hi Ankit, thanks for your reply.

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule http://blah.com/mysubmodule.git\` <http://blah.com/mysubmodule.git`> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM still unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.json`, meaning this setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.

tldr; *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of .Package.local(named: "XYZ") that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.

I think you're not really using the Top of Tree feature. You need to add each dependency using its canonical URL, hosted at some server like github. After adding the dependencies, you can use edit feature to put a dependency in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path ../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case of misunderstanding the feature or the docs.

The package manager will then stop using the cloned repository and use the checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially huge repository. Why force everyone on the dev team to clone a huge repo twice, only to *never* use one of the clones. Also, SwiftPM breaks when —path points at Packages/PackageName, which is exactly where I’d expect the package to be, not in some arbitrary external path (+ some kind of internal checkout cache that will never be used) as well.

I haven’t tried to test this recently because it’s a slow process but I have the impression the deps could be even be cloned more than twice, depending on how cleverly SwiftPM realises that multiple Packages have the same dependency.

Also, this makes managing interdependent state of development amongst dependencies more difficult than needed. How do we guarantee that devs are on the same commit when using top of tree development? Tagging and managing version numbers etc for day-to-day development is emphatically not an option for us. Since SwiftPM packages only work from a git context anyway, why not allow use of git’s established pattern of dealing with this, namely submodules?

Sharing this setup is not automatic, but simple. Each user just needs to run the above command once per dependency.

We have about 10 dependencies, all of which will always be in this state. This seems like a lot of overhead and room for user error, plus it’s a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will always be working on the dependencies. This is the core of what we’re doing, not a short aside. This is what makes me think we are either doing something wrong, or there is a big feature gap (as it appears from here).

The new manifest also supports using branch instead of version range, which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how we were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case clearer:

Main Project (git repo, not a Swift Package, contains no swift code directly)
–– Dependencies (external)
–– Subproject (internal git submodule, is a Swift Package, has multiple Swift Targets)
–––– Dependency A (internal, git submodule)
–––––––– Huge external C-language dependencies (managed via git submodules)
–––– Dependency B (internal, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (internal, git submodule)
–––––––– Depends on internal dependency A
–––––––– Depends on internal dependency B
–––––––– etc.
–––– Dependency D (internal, git submodule)

I think the friction is coming from the fact that we’d like to use SwiftPM just to build, rather than to manage our dependencies.

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the submodules at Project/Submodule instead of Project/Package/Submodule and then running swift package edit Submodule —path ./Submodule, just that this process would have to be manual for each new dev cloning the repo. And then we’d still have two checkouts of the same thing. Yes, this works, it just seems very inefficient and still hacky. And it’s very possible it'll break again with future SwiftPM versions.

I’m just surprised the idea of a "local dependency" is not seen as a first class citizen in SwiftPM, still trying to understand the logic behind that. Maybe you can give me an idea of the reasoning behind this?

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not stable and will change without notice.

This was not our preferred way of going about it of course. But it was (unfortunately) the best solution to the problem.

···

Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <swift-users@swift.org>:
On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Aciid · July 21, 2017, 1:24pm

+swift-build-dev

Hi Ankit, thanks for your reply.

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule http://blah.com/mysubmodule.git\` <http://blah.com/mysubmodule.git`> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM still unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.json`, meaning this setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.

tldr; *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of .Package.local(named: "XYZ") that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.

I think you're not really using the Top of Tree feature. You need to add each dependency using its canonical URL, hosted at some server like github. After adding the dependencies, you can use edit feature to put a dependency in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path ../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case of misunderstanding the feature or the docs.

Ah, cool!

The package manager will then stop using the cloned repository and use the checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially huge repository. Why force everyone on the dev team to clone a huge repo twice, only to *never* use one of the clones.

Sure, you could end up with clones but that is only an implementation detail. SwiftPM can become smart enough and remove a clone that is not being used.

Also, SwiftPM breaks when —path points at Packages/PackageName, which is exactly where I’d expect the package to be, not in some arbitrary external path (+ some kind of internal checkout cache that will never be used) as well.

If --path Packages/PackageName doesn't work, that is a bug! Do you mind filing a JIRA at bugs.swift.org <Issues · apple/swift · GitHub?

I understand internal caches could be an inconvenience if repositories are large but we can definitely improve how caching works.

I haven’t tried to test this recently because it’s a slow process but I have the impression the deps could be even be cloned more than twice, depending on how cleverly SwiftPM realises that multiple Packages have the same dependency.

Right now, SwiftPM will never clone a dependency twice.

Also, this makes managing interdependent state of development amongst dependencies more difficult than needed. How do we guarantee that devs are on the same commit when using top of tree development? Tagging and managing version numbers etc for day-to-day development is emphatically not an option for us.

When you use top of the tree mode, the checkout is expected to be managed by the user. That is exactly what top of the tree mode means. If we get local package support in the manifest, you will still need to manage the checkout state of those dependencies.

Since SwiftPM packages only work from a git context anyway, why not allow use of git’s established pattern of dealing with this, namely submodules?

I am not what do you mean here exactly.

Sharing this setup is not automatic, but simple. Each user just needs to run the above command once per dependency.

We have about 10 dependencies, all of which will always be in this state. This seems like a lot of overhead and room for user error, plus it’s a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will always be working on the dependencies. This is the core of what we’re doing, not a short aside. This is what makes me think we are either doing something wrong, or there is a big feature gap (as it appears from here).

I agree that a local dependency feature will be very helpful here but also note that you only need to run the edit command once (unless you delete the build folder). You can write a simple script that just put every dependency you have in edit mode.

The new manifest also supports using branch instead of version range, which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how we were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case clearer:

Main Project (git repo, not a Swift Package, contains no swift code directly)
–– Dependencies (external)
–– Subproject (internal git submodule, is a Swift Package, has multiple Swift Targets)
–––– Dependency A (internal, git submodule)
–––––––– Huge external C-language dependencies (managed via git submodules)
–––– Dependency B (internal, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (internal, git submodule)
–––––––– Depends on internal dependency A
–––––––– Depends on internal dependency B
–––––––– etc.
–––– Dependency D (internal, git submodule)

I think the friction is coming from the fact that we’d like to use SwiftPM just to build, rather than to manage our dependencies.

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the submodules at Project/Submodule instead of Project/Package/Submodule and then running swift package edit Submodule —path ./Submodule, just that this process would have to be manual for each new dev cloning the repo. And then we’d still have two checkouts of the same thing. Yes, this works, it just seems very inefficient and still hacky. And it’s very possible it'll break again with future SwiftPM versions.

I think we do allow editing a package that is inside another package as a submodule. If that doesn't work, do you mind filing a JIRA with a sample project setup?

I’m just surprised the idea of a "local dependency" is not seen as a first class citizen in SwiftPM, still trying to understand the logic behind that. Maybe you can give me an idea of the reasoning behind this?

I understand your usecase will benefit a lot from the local dependency feature. We did consider adding this feature, I think the main concern was package authors ending up publishing package which only works for them incase they forget updating their manifest before a release. Also, note that SwiftPM is still under heavy development and we can definitely re-consider adding this!

Also, I think this is a good usecase of a multipackage repository feature. We don't have that yet but you can read some of the discussion here <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20161107/000722.html>\.

PS: We also have a slack channel if you want to hang out - [swift-build-dev] Swift Package Manager Slack Team

···

On 21-Jul-2017, at 6:09 PM, Geordie J <geojay@gmail.com> wrote:

Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>>:
On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not stable and will change without notice.

This was not our preferred way of going about it of course. But it was (unfortunately) the best solution to the problem.

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

--
Ankit

Geordie_J · July 22, 2017, 10:07am

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex
app with multiple dependencies, all of which are being developed locally
and in parallel. The reason for this is compatibility with an existing
module/import structure used by our iOS app. Maybe I’m doing something very
wrong but my experience so far (2 months in) is that this is extremely
difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule
http://blah.com/mysubmodule.git\` in the Packages subdirectory and
SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development
option for this purpose. So far my experience with this has not been good.
Firstly because SwiftPM *still* unnecessarily tries to clone my repos
itself (some of which are huge), and secondly because this creates an
absolute path dependency in `.build/dependencies-state.json`, meaning this
setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of
complexity, having to tag each commit for SwiftPM to build. The fact that
we'd need to make a commit to test whether the project even builds is
insane enough as is, let alone the tagging and trying to tell the base
project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies
(as dynamic libraries) really are shared between multiple
targets/sub-dependencies, which SwiftPM seems to deal with quite well.

*tldr;* *Please* let us manage dependencies ourselves. It’d be so easy
if Package.swift had an option along the lines of *.Package.local(named:
"XYZ")* that it then looked for in ./Packages/XYZ. Again, maybe I’m
overlooking something but this seems like an obvious and vital option to
have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple
and painless as it sounds? I would be prepared to make a pull request along
these lines.

I think you're not really using the Top of Tree feature. You need to add
each dependency using its canonical URL, hosted at some server like github.
After adding the dependencies, you can use edit feature to put a dependency
in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path
../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case of
misunderstanding the feature or the docs.

The package manager will then stop using the cloned repository and use the
checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially
huge repository. Why force everyone on the dev team to clone a huge repo
twice, only to *never* use one of the clones. Also, SwiftPM breaks when
—path points at Packages/PackageName, which is exactly where I’d expect the
package to be, not in some arbitrary external path (+ some kind of internal
checkout cache that will never be used) as well.

I haven’t tried to test this recently because it’s a slow process but I
have the impression the deps could be even be cloned more than twice,
depending on how cleverly SwiftPM realises that multiple Packages have the
same dependency.

Also, this makes managing interdependent state of development amongst
dependencies more difficult than needed. How do we guarantee that devs are
on the same commit when using top of tree development? Tagging and managing
version numbers etc for day-to-day development is emphatically not an
option for us. Since SwiftPM packages only work from a git context anyway,
why not allow use of git’s established pattern of dealing with this, namely
submodules?

Sharing this setup is not automatic, but simple. Each user just needs to
run the above command once per dependency.

We have about 10 dependencies, *all *of which will* always* be in this
state. This seems like a lot of overhead and room for user error, plus it’s
a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will *always* be working on the dependencies. This
is the core of what we’re doing, not a short aside. This is what makes me
think we are either doing something wrong, or there is a big feature gap
(as it appears from here).

The new manifest also supports using branch instead of version range,
which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how we
were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case
clearer:

Main Project (git repo, not a Swift Package, contains no swift code
directly)
–– Dependencies (external)
–– Subproject (*internal* git submodule, is a Swift Package, has multiple
Swift Targets)
–––– Dependency A (*internal*, git submodule)
–––––––– Huge external C-language dependencies (managed via git submodules)
–––– Dependency B (*internal*, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (*internal*, git submodule)
–––––––– Depends on *internal* dependency A
–––––––– Depends on *internal* dependency B
–––––––– etc.
–––– Dependency D (*internal*, git submodule)

Reading over this entire thread, I've come across this again, which I think
sums up the pain point better than anything else:

*I think the friction is coming from the fact that we’d like to use
SwiftPM just to build, rather than to manage our dependencies.*

When I say "swift build", I expect swift to build! Not to check for commits
and tags and dependencies, not to clone anything, just build what is there.
For all the other stuff we have the "swift package ..." commands.

Imagine if "swift package update" did the step that currently happens
before "swift build" builds the project (dep management, cloning etc). And
to retain old behaviour we could have "swift build --update-deps" with a
note on failing "swift build" builds suggesting users try the update flag
to get the old behaviour.

I think this would be an equally welcome addition for users not currently
on a strong internet connection.

The question would be how Swift knows which dependency is which. The answer
should be pretty simple: look in each Packages/package/Package.swift for
the available package names and build them if there's a dependency on them
somewhere in the graph.

There's something about that idea that seems at odds with the current
git-centric model, though I'm still skeptical that the git-centric model is
a reasonable base case. For example, it's confusing because
Sources/TargetName uses a filesystem convention while package dependencies
currently do not, but kind of actually do after "swift package edit" has
been run, and would have to with any of the current proposals. Is the
complexity of using git as the base case becoming clear? No matter what
we're doing, the reality is that we end up with files in our filesystem. To
me having the dep as a remote git repo is actually the edge case, which is
the opposite of the current model.

To do anything with a moderate level of complexity took me weeks to get
figure out the ins and outs of and get to a point where it was repeatable
within the team. Again, I think this can be done better. And I think
assuming that dependencies are local unless specified otherwise would make
this a lot easier. Why not for example just have "swift package update"
check out the git repos into its internal build cache and symlink them into
Packages/PackageName, always building packages from ./Packages? I'd find
that a lot more consistent and transparent.

In the meantime the multi-package repo proposal seems like a step in the
right direction. In a way though it seems to me like a crutch for an
underlying inconsistency in both tooling namespaces ("swift build" vs
"swift package") and in convention vs configuration (specific directory
structure vs automated dep management via git).

I'm not sure what the implications of this are, but I'm starting to wonder
whether "swift build" and "swift package" are conceptually two different
but related projects, and whether it'd be a good idea in the medium-long
term to more clearly separate them.

Cheers,
Geordie

PS. The top-of-tree workaround does work after all, but is complicated
because we're also running "swift build" from within a docker image to
build for other platforms, so the absolute paths are (inescapably)
different between the environments.

Again, the idea of having absolute paths there at all seems unnecessary,
but until multi-package repos are available it seems the best option will
be scripting a find-and-replace in ".build/dependencies-state.json" before
running "swift build"...

···

Geordie J <geojay@gmail.com> schrieb am Fr. 21. Juli 2017 um 14:39: > Hi Ankit, thanks for your reply. > > Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users < > swift-users@swift.org>:

On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users < > swift-users@swift.org> wrote:

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the
submodules at *Project/Submodule* instead of *Project/Package/Submodule*
and then running *swift package edit Submodule —path ./Submodule*, just
that this process would have to be manual for each new dev cloning the
repo. And then we’d still have two checkouts of the same thing. Yes, this
works, it just seems very inefficient and still hacky. And it’s very
possible it'll break again with future SwiftPM versions.

I’m just surprised the idea of a "local dependency" is not seen as a first
class citizen in SwiftPM, still trying to understand the logic behind that.
Maybe you can give me an idea of the reasoning behind this?

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling
in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just
left us alone.. We were able to commit `workspace-state.json` into our base
project’s git repo and the rest Just Worked™. Now with the absolute paths
being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not
stable and will change without notice.

This was not our preferred way of going about it of course. But it was
(unfortunately) the best solution to the problem.

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Geordie_J · July 21, 2017, 2:39pm

Hi Ankit,

+swift-build-dev

Hi Ankit, thanks for your reply.

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule http://blah.com/mysubmodule.git\` <http://blah.com/mysubmodule.git`> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM still unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.json`, meaning this setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.

tldr; *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of .Package.local(named: "XYZ") that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.

I think you're not really using the Top of Tree feature. You need to add each dependency using its canonical URL, hosted at some server like github. After adding the dependencies, you can use edit feature to put a dependency in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path ../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case of misunderstanding the feature or the docs.

Ah, cool!

The package manager will then stop using the cloned repository and use the checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially huge repository. Why force everyone on the dev team to clone a huge repo twice, only to *never* use one of the clones.

Sure, you could end up with clones but that is only an implementation detail. SwiftPM can become smart enough and remove a clone that is not being used.

Also, SwiftPM breaks when —path points at Packages/PackageName, which is exactly where I’d expect the package to be, not in some arbitrary external path (+ some kind of internal checkout cache that will never be used) as well.

If --path Packages/PackageName doesn't work, that is a bug! Do you mind filing a JIRA at bugs.swift.org <Issues · apple/swift · GitHub?

SwiftPM in this case tries to make a symlink from Packages/PackageName -> Packages/PackageName and doesn’t allow continuing from there.

I understand internal caches could be an inconvenience if repositories are large but we can definitely improve how caching works.

It just seems unnecessary. Imagine (as an extreme example) you pull in the apple/swift repo as a dependency, with all its tens of thousands of commits, then you have a git submodule also in the repo with the same dependency. I’d just expect SwiftPM not to clone the package at all if I tell it I want it locally.

I haven’t tried to test this recently because it’s a slow process but I have the impression the deps could be even be cloned more than twice, depending on how cleverly SwiftPM realises that multiple Packages have the same dependency.

Right now, SwiftPM will never clone a dependency twice.

Ok good to know. (Edit: this appears to be untrue, see below)

Also, this makes managing interdependent state of development amongst dependencies more difficult than needed. How do we guarantee that devs are on the same commit when using top of tree development? Tagging and managing version numbers etc for day-to-day development is emphatically not an option for us.

When you use top of the tree mode, the checkout is expected to be managed by the user. That is exactly what top of the tree mode means. If we get local package support in the manifest, you will still need to manage the checkout state of those dependencies.

Yes, that is what I’m after: managing checkout state of dependencies ourselves.

Since SwiftPM packages only work from a git context anyway, why not allow use of git’s established pattern of dealing with this, namely submodules?

I am not what do you mean here exactly.

I think my sentiments here are covered by the multi package repo proposal, thanks!

Sharing this setup is not automatic, but simple. Each user just needs to run the above command once per dependency.

We have about 10 dependencies, all of which will always be in this state. This seems like a lot of overhead and room for user error, plus it’s a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will always be working on the dependencies. This is the core of what we’re doing, not a short aside. This is what makes me think we are either doing something wrong, or there is a big feature gap (as it appears from here).

I agree that a local dependency feature will be very helpful here but also note that you only need to run the edit command once (unless you delete the build folder). You can write a simple script that just put every dependency you have in edit mode.

Yes, this is manageable, if it works. It still feels like a 2nd or 3rd class citizen though, and a workaround / hack for something that seems like what I would have expected to be base case.

It seems like SwiftPM is based on a git workflow as its core element which appears to complicate things. This process of running an arbitrary script after running "git pull" is also just one more thing that needs doing (and is another source of human error). It also requires a working tag for each of those repos to clone from to "get started", which to me is an unnecessary requirement that again comes with lots of unnecessary cognitive load.

This also appears to just not work with our nested dependency structure. See below.

The new manifest also supports using branch instead of version range, which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how we were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case clearer:

Main Project (git repo, not a Swift Package, contains no swift code directly)
–– Dependencies (external)
–– Subproject (internal git submodule, is a Swift Package, has multiple Swift Targets)
–––– Dependency A (internal, git submodule)
–––––––– Huge external C-language dependencies (managed via git submodules)
–––– Dependency B (internal, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (internal, git submodule)
–––––––– Depends on internal dependency A
–––––––– Depends on internal dependency B
–––––––– etc.
–––– Dependency D (internal, git submodule)

I think the friction is coming from the fact that we’d like to use SwiftPM just to build, rather than to manage our dependencies.

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the submodules at Project/Submodule instead of Project/Package/Submodule and then running swift package edit Submodule —path ./Submodule, just that this process would have to be manual for each new dev cloning the repo. And then we’d still have two checkouts of the same thing. Yes, this works, it just seems very inefficient and still hacky. And it’s very possible it'll break again with future SwiftPM versions.

I think we do allow editing a package that is inside another package as a submodule. If that doesn't work, do you mind filing a JIRA with a sample project setup?

I have just spent the last couple of hours trying to implement this "top of tree" workaround. The 1st level dependencies appear to work as expected (top of tree) but it appears the subdependencies (2nd / 3rd level nesting) are being lifted from the git checkout in .build and not the expected local one, leading to all sorts of weird build errors that I’m finding difficult to fit into my mental model of what is supposed to be happening.

Also, if I try to package edit DepWithHugeSubDep —path XYZ from within one of the dependencies then SwiftPM will clone it a third time, because DepWithHugeSubDep then tries to create its own .build context. I’m trying this now to see if it fixes the other issues we were having but I’m not sure it’ll work either.

I’m just surprised the idea of a "local dependency" is not seen as a first class citizen in SwiftPM, still trying to understand the logic behind that. Maybe you can give me an idea of the reasoning behind this?

I understand your usecase will benefit a lot from the local dependency feature. We did consider adding this feature, I think the main concern was package authors ending up publishing package which only works for them incase they forget updating their manifest before a release. Also, note that SwiftPM is still under heavy development and we can definitely re-consider adding this!

Whether a subdependency exists seems like a pretty simple thing for SwiftPM to check for after loading a Package from the network. Surely a warning here would suffice - the package maintainer gets bombarded by emails and pays more attention next time. This doesn’t seem any more or less likely than package maintainers introducing any other kind of bug in packages they release.

Also, I think this is a good usecase of a multipackage repository feature. We don't have that yet but you can read some of the discussion here <https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20161107/000722.html>\.

This is excellent, in particular this:

let package = Package(
    dependencies: [
        .Package(subpackage: "Foo")
    ])
which is exactly what we need.

PS: We also have a slack channel if you want to hang out - [swift-build-dev] Swift Package Manager Slack Team

Thanks, I’ll check it out!

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not stable and will change without notice.

This was not our preferred way of going about it of course. But it was (unfortunately) the best solution to the problem.

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

--
Ankit

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

Cheers,
Geordie

···

Am 21.07.2017 um 15:32 schrieb Ankit Aggarwal via swift-users <swift-users@swift.org>:

On 21-Jul-2017, at 6:09 PM, Geordie J <geojay@gmail.com <mailto:geojay@gmail.com>> wrote:

Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>>:
On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

taylorswift · July 22, 2017, 3:50am

I just want to add that I have been immensely frustrated and inconvenienced
by the lack of this, what I believe is, basic and essential feature in the
SwiftPM.

···

On Fri, Jul 21, 2017 at 9:32 AM, Ankit Aggarwal via swift-users < swift-users@swift.org> wrote:

I’m just surprised the idea of a "local dependency" is not seen as a first
class citizen in SwiftPM, still trying to understand the logic behind that.
Maybe you can give me an idea of the reasoning behind this?

I understand your usecase will benefit a lot from the local dependency
feature. We did consider adding this feature, I think the main concern was
package authors ending up publishing package which only works for them
incase they forget updating their manifest before a release. Also, note
that SwiftPM is still under heavy development and we can definitely
re-consider adding this!

Also, I think this is a good usecase of a multipackage repository feature.
We don't have that yet but you can read some of the discussion here
<https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20161107/000722.html>
.

PS: We also have a slack channel if you want to hang out -
The swift-build-dev Archives
Week-of-Mon-20160530/000497.html

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Aciid · July 22, 2017, 11:44am

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex app with multiple dependencies, all of which are being developed locally and in parallel. The reason for this is compatibility with an existing module/import structure used by our iOS app. Maybe I’m doing something very wrong but my experience so far (2 months in) is that this is extremely difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule http://blah.com/mysubmodule.git\` <http://blah.com/mysubmodule.git`> in the Packages subdirectory and SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development option for this purpose. So far my experience with this has not been good. Firstly because SwiftPM still unnecessarily tries to clone my repos itself (some of which are huge), and secondly because this creates an absolute path dependency in `.build/dependencies-state.json`, meaning this setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of complexity, having to tag each commit for SwiftPM to build. The fact that we'd need to make a commit to test whether the project even builds is insane enough as is, let alone the tagging and trying to tell the base project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the dependencies (as dynamic libraries) really are shared between multiple targets/sub-dependencies, which SwiftPM seems to deal with quite well.

tldr; *Please* let us manage dependencies ourselves. It’d be so easy if Package.swift had an option along the lines of .Package.local(named: "XYZ") that it then looked for in ./Packages/XYZ. Again, maybe I’m overlooking something but this seems like an obvious and vital option to have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple and painless as it sounds? I would be prepared to make a pull request along these lines.

I think you're not really using the Top of Tree feature. You need to add each dependency using its canonical URL, hosted at some server like github. After adding the dependencies, you can use edit feature to put a dependency in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path ../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case of misunderstanding the feature or the docs.

The package manager will then stop using the cloned repository and use the checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially huge repository. Why force everyone on the dev team to clone a huge repo twice, only to *never* use one of the clones. Also, SwiftPM breaks when —path points at Packages/PackageName, which is exactly where I’d expect the package to be, not in some arbitrary external path (+ some kind of internal checkout cache that will never be used) as well.

I haven’t tried to test this recently because it’s a slow process but I have the impression the deps could be even be cloned more than twice, depending on how cleverly SwiftPM realises that multiple Packages have the same dependency.

Also, this makes managing interdependent state of development amongst dependencies more difficult than needed. How do we guarantee that devs are on the same commit when using top of tree development? Tagging and managing version numbers etc for day-to-day development is emphatically not an option for us. Since SwiftPM packages only work from a git context anyway, why not allow use of git’s established pattern of dealing with this, namely submodules?

Sharing this setup is not automatic, but simple. Each user just needs to run the above command once per dependency.

We have about 10 dependencies, all of which will always be in this state. This seems like a lot of overhead and room for user error, plus it’s a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will always be working on the dependencies. This is the core of what we’re doing, not a short aside. This is what makes me think we are either doing something wrong, or there is a big feature gap (as it appears from here).

The new manifest also supports using branch instead of version range, which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how we were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case clearer:

Main Project (git repo, not a Swift Package, contains no swift code directly)
–– Dependencies (external)
–– Subproject (internal git submodule, is a Swift Package, has multiple Swift Targets)
–––– Dependency A (internal, git submodule)
–––––––– Huge external C-language dependencies (managed via git submodules)
–––– Dependency B (internal, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (internal, git submodule)
–––––––– Depends on internal dependency A
–––––––– Depends on internal dependency B
–––––––– etc.
–––– Dependency D (internal, git submodule)

Reading over this entire thread, I've come across this again, which I think sums up the pain point better than anything else:

I think the friction is coming from the fact that we’d like to use SwiftPM just to build, rather than to manage our dependencies.

When I say "swift build", I expect swift to build! Not to check for commits and tags and dependencies, not to clone anything, just build what is there. For all the other stuff we have the "swift package ..." commands.

Imagine if "swift package update" did the step that currently happens before "swift build" builds the project (dep management, cloning etc). And to retain old behaviour we could have "swift build --update-deps" with a note on failing "swift build" builds suggesting users try the update flag to get the old behaviour.

One of the core functionality of the package manager is managing dependencies. The "swift build" command doesn't do any dependency related operation unless it needs to. If there are no dependencies cloned yet, it doesn't make sense for "swift build" to error out and say run "swift package resolve" because it can already run that command if needed.

I think this would be an equally welcome addition for users not currently on a strong internet connection.

If you already have all the dependencies cloned, the "swift build" command will never try to connect to internet. We're also considering an offline mode for the "package resolve" command but that is for different reasons.

The question would be how Swift knows which dependency is which. The answer should be pretty simple: look in each Packages/package/Package.swift for the available package names and build them if there's a dependency on them somewhere in the graph.

That sounds like a very fragile implementation. The package authors shouldn't need to care how their dependencies are being handled by the package manager, unless of course it is being edited.

There's something about that idea that seems at odds with the current git-centric model, though I'm still skeptical that the git-centric model is a reasonable base case. For example, it's confusing because Sources/TargetName uses a filesystem convention while package dependencies currently do not, but kind of actually do after "swift package edit" has been run, and would have to with any of the current proposals. Is the complexity of using git as the base case becoming clear? No matter what we're doing, the reality is that we end up with files in our filesystem. To me having the dep as a remote git repo is actually the edge case, which is the opposite of the current model.

Note that the convention system for targets is greatly simplified in Swift 4 and fully customizable. I don't think target and package dependencies are similar enough to relate them to the convention system. The package manager is fully responsible for managing the dependencies. The dependencies need to come from somewhere, which is the git url currently. The multipackage repository will allow having the dependencies in the same repository, I think this is what really fits your usecase. Unfortunately, we couldn't finish design for this feature in enough timeframe for Swift 4.

To do anything with a moderate level of complexity took me weeks to get figure out the ins and outs of and get to a point where it was repeatable within the team. Again, I think this can be done better. And I think assuming that dependencies are local unless specified otherwise would make this a lot easier. Why not for example just have "swift package update" check out the git repos into its internal build cache and symlink them into Packages/PackageName, always building packages from ./Packages? I'd find that a lot more consistent and transparent.

Again, the package authors shouldn't need to worry about how the dependencies are being managed. The editable packages proposal <https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md> briefly explains the motivation behind making the dependencies an implementation detail.

We have discussed whether or not hiding the sources for non-editable packages is the right default. The motivation for hiding the sources is that in a large, mature, stable ecosystem there are likely to be a large number of packages involved in any particular project build, and many of those are likely to be uninteresting to the package developer. In particular, while a project developer might be interested in the source of their direct dependencies, the sources of that packages own dependencies is an "implementation detail" from the perspective of the project developer.

The package update command does actually need to fetch the latest tags from internet to do the dependency resolution. It may be possible to resolve the dependencies with the tags we have in cache but that may not be ideal. However, this is still useful in case there is no internet connection, so we plan to introduce an offline mode to handle those cases.

In the meantime the multi-package repo proposal seems like a step in the right direction. In a way though it seems to me like a crutch for an underlying inconsistency in both tooling namespaces ("swift build" vs "swift package") and in convention vs configuration (specific directory structure vs automated dep management via git).

I'm not sure what the implications of this are, but I'm starting to wonder whether "swift build" and "swift package" are conceptually two different but related projects, and whether it'd be a good idea in the medium-long term to more clearly separate them.

I think I tried to explain the reason why "swift build" can run git commands above.

Cheers,
Geordie

PS. The top-of-tree workaround does work after all, but is complicated because we're also running "swift build" from within a docker image to build for other platforms, so the absolute paths are (inescapably) different between the environments.

Again, the idea of having absolute paths there at all seems unnecessary, but until multi-package repos are available it seems the best option will be scripting a find-and-replace in ".build/dependencies-state.json" before running "swift build"...

One solution could be using a different build folder when using docker.

···

On 22-Jul-2017, at 3:37 PM, Geordie Jay <geojay@gmail.com> wrote:
Geordie J <geojay@gmail.com <mailto:geojay@gmail.com>> schrieb am Fr. 21. Juli 2017 um 14:39: > Hi Ankit, thanks for your reply. > >> Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>>:

On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

---

Lets see what others think about reconsidering local dependencies feature as that can probably help many such usecases until multi-package repository feature arrives.

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the submodules at Project/Submoduleinstead of Project/Package/Submodule and then running swift package edit Submodule —path ./Submodule, just that this process would have to be manual for each new dev cloning the repo. And then we’d still have two checkouts of the same thing. Yes, this works, it just seems very inefficient and still hacky. And it’s very possible it'll break again with future SwiftPM versions.

I’m just surprised the idea of a "local dependency" is not seen as a first class citizen in SwiftPM, still trying to understand the logic behind that. Maybe you can give me an idea of the reasoning behind this?

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just left us alone.. We were able to commit `workspace-state.json` into our base project’s git repo and the rest Just Worked™. Now with the absolute paths being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not stable and will change without notice.

This was not our preferred way of going about it of course. But it was (unfortunately) the best solution to the problem.

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users

--
Ankit

David_Sweeris · July 22, 2017, 5:46pm

How can I get involved in the evolution of this? The evolutions are always uneditable uncommentable markdown files on a repo somewhere and the mailing lists are in my practical experience inpentrible especially for entries you weren't subscribed to the list for, or were "only" subscribed to the digest for. I wish they were on google docs or a hackpad equivalent. Is there "officially recognised" discussion on the SwiftPM dev slack channel regarding evolution topics?

Yeah, I'm not a mailing list fan, either (at least for long or active threads), and you have nicely summarized the reasons we decided to move to a discourse server. It's just that it hasn't happened yet.

To answer your question more directly, you can of course discuss things on slack, twitter, in-person, off-list email, or in any other way, but AFAIK of you want your conversation to be "on-record", you need to have it on-list. Until we move to discourse, of course.

I realize that doesn't actually solve the problem, I hope it at least gives you some moral support.

- Dave Sweeris (who speaks only for himself and shouldn't be taken as official source of information on this matter)

···

Sent from my iPhone

On Jul 22, 2017, at 05:16, Geordie Jay via swift-build-dev <swift-build-dev@swift.org> wrote:

Geordie_J · July 22, 2017, 12:15pm

Hi Ankit, thanks for your explanations.

Hi all,

My team and I are trying to use SwiftPM to develop a relatively complex
app with multiple dependencies, all of which are being developed locally
and in parallel. The reason for this is compatibility with an existing
module/import structure used by our iOS app. Maybe I’m doing something very
wrong but my experience so far (2 months in) is that this is extremely
difficult with SwiftPM.

What I’d love to be able to do is to just run `git add submodule
http://blah.com/mysubmodule.git\` in the Packages subdirectory and
SwiftPM would just let me manage dependencies from there myself.

I was excited to see that SwiftPM 4 has a "Top of Tree" development
option for this purpose. So far my experience with this has not been good.
Firstly because SwiftPM *still* unnecessarily tries to clone my repos
itself (some of which are huge), and secondly because this creates an
absolute path dependency in `.build/dependencies-state.json`, meaning this
setup isn’t sharable within our dev team.

Attempting this with "local" git urls adds an almost absurd level of
complexity, having to tag each commit for SwiftPM to build. The fact that
we'd need to make a commit to test whether the project even builds is
insane enough as is, let alone the tagging and trying to tell the base
project to use a newer minor version etc etc.

Adding multiple subtargets is also not an option because the
dependencies (as dynamic libraries) really are shared between multiple
targets/sub-dependencies, which SwiftPM seems to deal with quite well.

*tldr;* *Please* let us manage dependencies ourselves. It’d be so easy
if Package.swift had an option along the lines of *.Package.local(named:
"XYZ")* that it then looked for in ./Packages/XYZ. Again, maybe I’m
overlooking something but this seems like an obvious and vital option to
have. It’d also simplify the introductory SwiftPM docs significantly.

Is anyone else having this issue? Would this change really be as simple
and painless as it sounds? I would be prepared to make a pull request along
these lines.

I think you're not really using the Top of Tree feature. You need to add
each dependency using its canonical URL, hosted at some server like github.
After adding the dependencies, you can use edit feature to put a dependency
in Top of Tree mode. To do so, run:

$ swift package edit <PackageName> --path
../path/to/self/managed/checkout/of/the/package

Yes, this is what I tried this week. I’m pretty sure this is not a case
of misunderstanding the feature or the docs.

The package manager will then stop using the cloned repository and use
the checkout present at that path (regardless of the state it is in).

Yes, but then I have – per dependency – two checkouts of a potentially
huge repository. Why force everyone on the dev team to clone a huge repo
twice, only to *never* use one of the clones. Also, SwiftPM breaks when
—path points at Packages/PackageName, which is exactly where I’d expect the
package to be, not in some arbitrary external path (+ some kind of internal
checkout cache that will never be used) as well.

I haven’t tried to test this recently because it’s a slow process but I
have the impression the deps could be even be cloned more than twice,
depending on how cleverly SwiftPM realises that multiple Packages have the
same dependency.

Also, this makes managing interdependent state of development amongst
dependencies more difficult than needed. How do we guarantee that devs are
on the same commit when using top of tree development? Tagging and managing
version numbers etc for day-to-day development is emphatically not an
option for us. Since SwiftPM packages only work from a git context anyway,
why not allow use of git’s established pattern of dealing with this, namely
submodules?

Sharing this setup is not automatic, but simple. Each user just needs to
run the above command once per dependency.

We have about 10 dependencies, *all *of which will* always* be in this
state. This seems like a lot of overhead and room for user error, plus it’s
a huge workaround for something that could be very simple.

Also, you only need to do this if you're actively working on a dependency.

The point is that we will *always* be working on the dependencies. This
is the core of what we’re doing, not a short aside. This is what makes me
think we are either doing something wrong, or there is a big feature gap
(as it appears from here).

The new manifest also supports using branch instead of version range,
which is very helpful during the development period.

This has much the same result as top-of-tree development, but it is how
we were able to "hack" SwiftPM 3 into leaving us alone.

Let me know if something is unclear or if you have more questions!

Maybe an overview of our structure would be helpful to make our use case
clearer:

Main Project (git repo, not a Swift Package, contains no swift code
directly)
–– Dependencies (external)
–– Subproject (*internal* git submodule, is a Swift Package, has
multiple Swift Targets)
–––– Dependency A (*internal*, git submodule)
–––––––– Huge external C-language dependencies (managed via git
submodules)
–––– Dependency B (*internal*, git submodule)
–––––––– Depends on internal dependency D
–––– Dependency C (*internal*, git submodule)
–––––––– Depends on *internal* dependency A
–––––––– Depends on *internal* dependency B
–––––––– etc.
–––– Dependency D (*internal*, git submodule)

Reading over this entire thread, I've come across this again, which I
think sums up the pain point better than anything else:

*I think the friction is coming from the fact that we’d like to use
SwiftPM just to build, rather than to manage our dependencies.*

When I say "swift build", I expect swift to build! Not to check for
commits and tags and dependencies, not to clone anything, just build what
is there. For all the other stuff we have the "swift package ..." commands.

Imagine if "swift package update" did the step that currently happens
before "swift build" builds the project (dep management, cloning etc). And
to retain old behaviour we could have "swift build --update-deps" with a
note on failing "swift build" builds suggesting users try the update flag
to get the old behaviour.

One of the core functionality of the package manager is managing
dependencies. The "swift build" command doesn't do any dependency related
operation unless it needs to. If there are no dependencies cloned yet, it
doesn't make sense for "swift build" to error out and say run "swift
package resolve" because it can already run that command if needed.

I think this would be an equally welcome addition for users not currently
on a strong internet connection.

If you already have all the dependencies cloned, the "swift build" command
will never try to connect to internet.

Assuming multipackage repos are accepted this will be true for all
important cases, I agree.

We're also considering an offline mode for the "package resolve" command

but that is for different reasons.

The question would be how Swift knows which dependency is which. The
answer should be pretty simple: look in each Packages/package/Package.swift
for the available package names and build them if there's a dependency on
them somewhere in the graph.

That sounds like a very fragile implementation. The package authors
shouldn't need to care how their dependencies are being handled by the
package manager, unless of course it is being edited.

There's something about that idea that seems at odds with the current
git-centric model, though I'm still skeptical that the git-centric model is
a reasonable base case. For example, it's confusing because
Sources/TargetName uses a filesystem convention while package dependencies
currently do not, but kind of actually do after "swift package edit" has
been run, and would have to with any of the current proposals. Is the
complexity of using git as the base case becoming clear? No matter what
we're doing, the reality is that we end up with files in our filesystem. To
me having the dep as a remote git repo is actually the edge case, which is
the opposite of the current model.

Note that the convention system for targets is greatly simplified in Swift
4 and fully customizable.

I did notice yesterday that tools version 4.0 allows many different target
structures. I guess this is nice but having a "convention" that could be
one of many things (different locations for target sources) also makes
SwiftPM more complex than simple in my opinion.

I don't think target and package dependencies are similar enough to relate

them to the convention system. The package manager is fully responsible for
managing the dependencies. The dependencies need to come from somewhere,
which is the git url currently. The multipackage repository will allow
having the dependencies in the same repository, I think this is what really
fits your usecase. Unfortunately, we couldn't finish design for this
feature in enough timeframe for Swift 4.

How can I get involved in the evolution of this? The evolutions are always
uneditable uncommentable markdown files on a repo somewhere and the mailing
lists are in my practical experience inpentrible especially for entries you
weren't subscribed to the list for, or were "only" subscribed to the digest
for. I wish they were on google docs or a hackpad equivalent. Is there
"officially recognised" discussion on the SwiftPM dev slack channel
regarding evolution topics?

To do anything with a moderate level of complexity took me weeks to get
figure out the ins and outs of and get to a point where it was repeatable
within the team. Again, I think this can be done better. And I think
assuming that dependencies are local unless specified otherwise would make
this a lot easier. Why not for example just have "swift package update"
check out the git repos into its internal build cache and symlink them into
Packages/PackageName, always building packages from ./Packages? I'd find
that a lot more consistent and transparent.

Again, the package authors shouldn't need to worry about how the
dependencies are being managed. The editable packages proposal
<https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md> briefly
explains the motivation behind making the dependencies an implementation
detail.

We have discussed whether or not hiding the sources for non-editable
packages is the right default. The motivation for hiding the sources is
that in a large, mature, stable ecosystem there are likely to be a large
number of packages involved in any particular project build, and many of
those are likely to be uninteresting to the package developer. In
particular, while a project developer might be interested in the source of
their direct dependencies, the sources of that packages own dependencies is
an "implementation detail" from the perspective of the project developer.

Being interested in the source of my direct dependencies is exactly the
current case. I can see the point about having many indirect package
dependencies, but this is what I'm used to from other ecosystems and to be
honest it has never bothered me there. More often I have found it
convenient to track down bugs in 3rd party libraries etc. but I agree this
would still be possible with the checkouts in .build

Have a great weekend,
Geordie

The package update command does actually need to fetch the latest tags
from internet to do the dependency resolution. It may be possible to
resolve the dependencies with the tags we have in cache but that may not be
ideal. However, this is still useful in case there is no internet
connection, so we plan to introduce an offline mode to handle those cases.

In the meantime the multi-package repo proposal seems like a step in the
right direction. In a way though it seems to me like a crutch for an
underlying inconsistency in both tooling namespaces ("swift build" vs
"swift package") and in convention vs configuration (specific directory
structure vs automated dep management via git).

I'm not sure what the implications of this are, but I'm starting to wonder
whether "swift build" and "swift package" are conceptually two different
but related projects, and whether it'd be a good idea in the medium-long
term to more clearly separate them.

I think I tried to explain the reason why "swift build" can run git
commands above.

Cheers,
Geordie

PS. The top-of-tree workaround does work after all, but is complicated
because we're also running "swift build" from within a docker image to
build for other platforms, so the absolute paths are (inescapably)
different between the environments.

Again, the idea of having absolute paths there at all seems unnecessary,
but until multi-package repos are available it seems the best option will
be scripting a find-and-replace in ".build/dependencies-state.json" before
running "swift build"...

One solution could be using a different build folder when using docker.

Possible, true, I didn't think of that. It'd mean even more git checkout
deadweight of huge dependencies though. And the script to put the Packages
in editable mode would then have to be maintained for multiple
environments. The easier option is just altering the absolute paths for now.

···

Ankit Aggarwal <ankit_aggarwal@apple.com> schrieb am Sa. 22. Juli 2017 um 13:44:

On 22-Jul-2017, at 3:37 PM, Geordie Jay <geojay@gmail.com> wrote:
Geordie J <geojay@gmail.com> schrieb am Fr. 21. Juli 2017 um 14:39: > >> Hi Ankit, thanks for your reply. >> >> Am 21.07.2017 um 07:33 schrieb Ankit Aggarwal via swift-users < >> swift-users@swift.org>:

On Thu, Jul 20, 2017 at 10:34 PM, Geordie J via swift-users < >> swift-users@swift.org> wrote:

---

Lets see what others think about reconsidering local dependencies feature
as that can probably help many such usecases until multi-package repository
feature arrives.

Again, this could be solved with a simple API addition in the manifest:

Package(
  …
  dependencies: [
    .package.local(named: "Dependency A")
    .package.local(named: "Dependency B")
    ...
  ]
)

At the end of the day it seems we can work around this by cloning the
submodules at *Project/Submodule*instead of *Project/Package/Submodule* and
then running *swift package edit Submodule —path ./Submodule*, just that
this process would have to be manual for each new dev cloning the repo. And
then we’d still have two checkouts of the same thing. Yes, this works, it
just seems very inefficient and still hacky. And it’s very possible it'll
break again with future SwiftPM versions.

I’m just surprised the idea of a "local dependency" is not seen as a
first class citizen in SwiftPM, still trying to understand the logic behind
that. Maybe you can give me an idea of the reasoning behind this?

Best Regards,
Geordie

Best Regards,
Geordie

PS. In SwiftPM 3 we had been using a hack that worked great: by filling
in the dependencies' "basedOn" key in `workspace-state.json`, SwiftPM just
left us alone.. We were able to commit `workspace-state.json` into our base
project’s git repo and the rest Just Worked™. Now with the absolute paths
being checked for this doesn’t seem to be an option.

Please do not rely on internals of the package manager as they're not
stable and will change without notice.

This was not our preferred way of going about it of course. But it was
(unfortunately) the best solution to the problem.

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

--
Ankit

Geordie_J · July 22, 2017, 7:07pm

This is morale-boosting news! Was this discussed somewhere public? I can hardly wait, and would be willing to put some of my spare time into facilitating the move if that’s at all interesting to those making the decisions on this one.

Cheers

···

Am 22.07.2017 um 19:46 schrieb David Sweeris <davesweeris@mac.com>:

Sent from my iPhone

On Jul 22, 2017, at 05:16, Geordie Jay via swift-build-dev <swift-build-dev@swift.org> wrote:

How can I get involved in the evolution of this? The evolutions are always uneditable uncommentable markdown files on a repo somewhere and the mailing lists are in my practical experience inpentrible especially for entries you weren't subscribed to the list for, or were "only" subscribed to the digest for. I wish they were on google docs or a hackpad equivalent. Is there "officially recognised" discussion on the SwiftPM dev slack channel regarding evolution topics?

Yeah, I'm not a mailing list fan, either (at least for long or active threads), and you have nicely summarized the reasons we decided to move to a discourse server. It's just that it hasn't happened yet.

To answer your question more directly, you can of course discuss things on slack, twitter, in-person, off-list email, or in any other way, but AFAIK of you want your conversation to be "on-record", you need to have it on-list. Until we move to discourse, of course.

I realize that doesn't actually solve the problem, I hope it at least gives you some moral support.

- Dave Sweeris (who speaks only for himself and shouldn't be taken as official source of information on this matter)

David_Sweeris · July 22, 2017, 11:09pm

Yay! morale += 1

Yeah, I can't remember if it was a formal proposal, but it was discussed extensively on the swift-evolution list a while back (maybe late last year or early this year, I think?), and the conclusion was that we'd move to Discourse (it has a mailing list mode, for the people who prefer that). In any case, I'd imagine it hasn't happened yet simply because the people who'd need to be involved are slammed with trying to meet the 4.0 release deadline.

Within the past day or so, someone on evolution rezzed Ted's announcement thread on the matter. I'd guess that'd be as good a place as any to start, if you want to try to get "the ones making the decisions" to let you help.

- Dave Sweeris

···

On Jul 22, 2017, at 12:07, Geordie J <geojay@gmail.com> wrote:

Am 22.07.2017 um 19:46 schrieb David Sweeris <davesweeris@mac.com>:

Sent from my iPhone

On Jul 22, 2017, at 05:16, Geordie Jay via swift-build-dev <swift-build-dev@swift.org> wrote:

How can I get involved in the evolution of this? The evolutions are always uneditable uncommentable markdown files on a repo somewhere and the mailing lists are in my practical experience inpentrible especially for entries you weren't subscribed to the list for, or were "only" subscribed to the digest for. I wish they were on google docs or a hackpad equivalent. Is there "officially recognised" discussion on the SwiftPM dev slack channel regarding evolution topics?

Yeah, I'm not a mailing list fan, either (at least for long or active threads), and you have nicely summarized the reasons we decided to move to a discourse server. It's just that it hasn't happened yet.

To answer your question more directly, you can of course discuss things on slack, twitter, in-person, off-list email, or in any other way, but AFAIK of you want your conversation to be "on-record", you need to have it on-list. Until we move to discourse, of course.

I realize that doesn't actually solve the problem, I hope it at least gives you some moral support.

- Dave Sweeris (who speaks only for himself and shouldn't be taken as official source of information on this matter)

This is morale-boosting news! Was this discussed somewhere public? I can hardly wait, and would be willing to put some of my spare time into facilitating the move if that’s at all interesting to those making the decisions on this one.

David_Sweeris · July 22, 2017, 11:11pm

Whoops, I was wrong, it's a new thread called "Setting expectations on when we move to Discourse"

···

On Jul 22, 2017, at 16:09, David Sweeris via swift-build-dev <swift-build-dev@swift.org> wrote:

Within the past day or so, someone on evolution rezzed Ted's announcement thread on the matter.

Geordie_J · July 23, 2017, 12:18am

Thanks for the update on this one. Judging by the thread, it looks like it’ll need to be handled internally by Apple. Also, I have no idea how to reply to the existing post without breaking the internet so I’m going to just wait patiently instead..

···

Am 23.07.2017 um 01:11 schrieb David Sweeris <davesweeris@mac.com <mailto:davesweeris@mac.com>>:

On Jul 22, 2017, at 16:09, David Sweeris via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Within the past day or so, someone on evolution rezzed Ted's announcement thread on the matter.

Whoops, I was wrong, it's a new thread called "Setting expectations on when we move to Discourse"