Draft SwiftPM proposal: Multi-package repositories


(Daniel Dunbar) #1

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
  https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel


(Russ Bishop) #2

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ

···

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel


(Daniel Dunbar) #3

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

···

On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com> wrote:

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ


(David Hart) #4

If we're going down the road of KISS, why not require all packages to be in direct sub-directories? Is that too constraining?

I've read the proposal and I'm also concerned about the potential complexity. But I also like, as Daniel has said, that it allows the naive solution - simple subdirectories for each package to work without learning a new file syntax.

On a side note, do we really need a "subpackage" argument for the Package initializer and not roll everything under "package"?

David.

···

On 13 Nov 2016, at 06:54, Daniel Dunbar via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com> wrote:

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Russ Bishop) #5

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

If you’re big enough that you’re creating sub packages frequently then you’re big enough to use automated tooling and it won’t matter so much what we do as long as the file format doesn’t promote impossible merges. Ideally the sub package specifiers can be one-liners so most 3-way merge tools won’t have trouble.

Let’s be absurd and imagine a monorepo with 100,000 packages created over a 5-year period. That’s one new package every ~6 minutes only during 8-hr workdays in a single timezone. It would not be possible to manually commit anything to the monorepo in that scenario let alone a change to a package file, you would have to use tooling that rolled changes to various levels of upstream. It would be trivial for a package-specific script to simply form the union of all unique packages across the downstream sources it is looking at to create the proper top-level manifest.

More realistically if you took 10,000 as the upper-bound that’s one commit per hour to the top-level manifest which is easily doable with manual merging.

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

I can see pros and cons to a new file format or using the Package manifest so I would go along with either.

Russ

···

On Nov 12, 2016, at 9:54 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:


(Chéyo Jiménez) #6

Hi Daniel,

I think this is an excellent idea! This would also solve the “local only” packages problem.
http://stackoverflow.com/questions/40775726/can-i-make-a-local-module-with-the-swift-package-manager

By treating the git repo still as a single package, we can then just allow local dependencies that live somewhere in the repo.

let package = Package(
    name: “myMainPackage",
    dependencies: [
    .Package(url: “./allMyLocalPackages/packageOne/“), // don’t have to specify version because it is inherited from main package.
    .Package(url: “./allMyLocalPackages/packageTwo/“),
    .Package(url: “./allMyLocalPackages/packageThree/“),
   ]
    )

I think this would lower the scope of the proposal and it would address the issue of being able to split up a mono repo.

Should I propose this as an alternative or collaborate on the draft that you have?

I have a very specific example where I want to be able to split up a repo so I can test them together on CI.
https://github.com/exercism/xswift/commit/4935b94c78a69f88b42c7a518c16e0c8b4f6fe8d#diff-37ca2dd15ca0f6b1b49e78db084ef5b9R21

Thank you.

···

On Nov 12, 2016, at 9:54 PM, Daniel Dunbar via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com <mailto:xenadu@gmail.com>> wrote:

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Daniel Dunbar) #7

If we're going down the road of KISS, why not require all packages to be
in direct sub-directories? Is that too constraining?

Yes, I think so. Projects should be able to be in control of their
non-package directory heirarchy I think.

I've read the proposal and I'm also concerned about the potential
complexity. But I also like, as Daniel has said, that it allows the naive
solution - simple subdirectories for each package to work without learning
a new file syntax.

On a side note, do we really need a "subpackage" argument for the Package
initializer and not roll everything under "package"?

How would this look?

- Daniel

···

On Sunday, November 13, 2016, David Hart via swift-build-dev < swift-build-dev@swift.org> wrote:

David.

> On 13 Nov 2016, at 06:54, Daniel Dunbar via swift-evolution < > swift-evolution@swift.org <javascript:;>> wrote:
>
>
>> On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com > <javascript:;>> wrote:
>>
>>
>>> On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev < > swift-build-dev@swift.org <javascript:;>> wrote:
>>>
>>> Hi all,
>>>
>>> I'm reposting a request for feedback on my proposal for extending
SwiftPM to support multiple packages inside one repository (i.e. "monorepo"
support, although it is a misnomer in this use case).
>>> https://github.com/ddunbar/swift-evolution/blob/multi-
package-repos/proposals/NNNN-swiftpm-multi-package-repos.md
>>>
>>> I would like to move this proposal forward so we can start on an
implementation, even if we need to refine it over time, but I was hoping to
get at least some concrete feedback first.
>>>
>>> Thanks,
>>> - Daniel
>>
>>
>> It seems like you’re going through contortions to deal with arbitrary
directory layouts and some odd consequences fall out of that decision. Not
being able to deterministically detect non-unique sub-packages is one.
>>
>> Why not just require a top-level Package.swift that explicitly
specifies the sub-packages? The name for the sub-package should be in the
main package manifest. You’d gain the ability to import all the
sub-packages in one go; importing the root package without any sub-packages
specified automatically imports all sub-packages. This also allows library
authors to organize a library into sub-packages later without breakage.
Come up with a convention, e.g. a sub-package is in “/subpackageName” but
allow overriding that default. That allows reorganization if needed but the
convention should work for most libraries.
>
> Mostly because I am concerned this doesn't scale well to *very* large
repositories, in which commits to that file would be "contentious" (in the
lock contention sense, not subject to debate sense). Of course, this
argument is a little bogus as the current proposal doesn't scale that great
either since we have to discover the packages (although I believe we can
probably do a good job of caching this information).
>
> It certainly would simplify the implementation & proposal to have this.
>
> The other reason is it is yet another thing for people to maintain (and
remember the syntax for). Most repos are small enough that I think the
current proposal would perform fine and have a tendency to do what people
might naively expect (even if they didn't really think about why). On the
other hand, this file is likely to be quite static, so I'm not sure that is
a very important issue.
>
> I was already on the fence on this, but I hadn't considered the benefits
you mention of allowing import of the package w/ no sub package specifier
to mean import of all sub-packages. That tips me a little more towards
thinking maybe a better proposal is to KISS and require this in some root
file (whether or not that root file is itself a package manifest or a
different kind of file is another question, you assume it would be the
regular package manifest but I don't think it *need* be, and there is some
value in not having any nesting relationship amongst packages).
>
> - Daniel
>
>> A top-level Package.swift would also allow immediate detection of
non-unique sub-packages, etc. Also if you are using things like git
submodules, subtree, or some other mechanism that ends up putting package
files in your source tree you don’t automatically re-export that package
unless you take explicit action.
>>
>>
>> I like the idea in general.
>>
>>
>> Russ
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <javascript:;>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <javascript:;>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #8

Hi Daniel,

I think this is an excellent idea! This would also solve the “local only” packages problem.
http://stackoverflow.com/questions/40775726/can-i-make-a-local-module-with-the-swift-package-manager

By treating the git repo still as a single package, we can then just allow local dependencies that live somewhere in the repo.

let package = Package(
    name: “myMainPackage",
    dependencies: [
    .Package(url: “./allMyLocalPackages/packageOne/“), // don’t have to specify version because it is inherited from main package.
    .Package(url: “./allMyLocalPackages/packageTwo/“),
    .Package(url: “./allMyLocalPackages/packageThree/“),
   ]
    )

I think this would lower the scope of the proposal and it would address the issue of being able to split up a mono repo.

Should I propose this as an alternative or collaborate on the draft that you have?

I'm not exactly sure what change you are proposing, can you elaborate? What is "allMyLocalPackages" in your email?

- Daniel

···

On Jan 17, 2017, at 2:04 PM, Jose Cheyo Jimenez <cheyo@masters3d.com> wrote:

I have a very specific example where I want to be able to split up a repo so I can test them together on CI.
https://github.com/exercism/xswift/commit/4935b94c78a69f88b42c7a518c16e0c8b4f6fe8d#diff-37ca2dd15ca0f6b1b49e78db084ef5b9R21

Thank you.

On Nov 12, 2016, at 9:54 PM, Daniel Dunbar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com <mailto:xenadu@gmail.com>> wrote:

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Chéyo Jiménez) #9

Hi Daniel,

I think this is an excellent idea! This would also solve the “local only” packages problem.
http://stackoverflow.com/questions/40775726/can-i-make-a-local-module-with-the-swift-package-manager

By treating the git repo still as a single package, we can then just allow local dependencies that live somewhere in the repo.

let package = Package(
    name: “myMainPackage",
    dependencies: [
    .Package(url: “./allMyLocalPackages/packageOne/“), // don’t have to specify version because it is inherited from main package.
    .Package(url: “./allMyLocalPackages/packageTwo/“),
    .Package(url: “./allMyLocalPackages/packageThree/“),
   ]
    )

I think this would lower the scope of the proposal and it would address the issue of being able to split up a mono repo.

Should I propose this as an alternative or collaborate on the draft that you have?

I'm not exactly sure what change you are proposing, can you elaborate? What is "allMyLocalPackages" in your email?

- Daniel

That is just a directory that it is not named sources.

After reading the road map. I think what I am referring to here is something along the line of
– Overriding Package Conventions
– Support for Top-of-Tree

But not quite Multi-package repository because I just want to be to able to have multiple packages in one repo but not necessary have them exposed as individual sub packages.

This is the same use case as the above stackoverflow multiple local packages question.

···

On Jan 24, 2017, at 8:31 AM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

On Jan 17, 2017, at 2:04 PM, Jose Cheyo Jimenez <cheyo@masters3d.com <mailto:cheyo@masters3d.com>> wrote:

I have a very specific example where I want to be able to split up a repo so I can test them together on CI.
https://github.com/exercism/xswift/commit/4935b94c78a69f88b42c7a518c16e0c8b4f6fe8d#diff-37ca2dd15ca0f6b1b49e78db084ef5b9R21

Thank you.

On Nov 12, 2016, at 9:54 PM, Daniel Dunbar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Nov 12, 2016, at 9:43 PM, Russ Bishop <xenadu@gmail.com <mailto:xenadu@gmail.com>> wrote:

On Nov 12, 2016, at 1:02 PM, Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi all,

I'm reposting a request for feedback on my proposal for extending SwiftPM to support multiple packages inside one repository (i.e. "monorepo" support, although it is a misnomer in this use case).
https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md

I would like to move this proposal forward so we can start on an implementation, even if we need to refine it over time, but I was hoping to get at least some concrete feedback first.

Thanks,
- Daniel

It seems like you’re going through contortions to deal with arbitrary directory layouts and some odd consequences fall out of that decision. Not being able to deterministically detect non-unique sub-packages is one.

Why not just require a top-level Package.swift that explicitly specifies the sub-packages? The name for the sub-package should be in the main package manifest. You’d gain the ability to import all the sub-packages in one go; importing the root package without any sub-packages specified automatically imports all sub-packages. This also allows library authors to organize a library into sub-packages later without breakage. Come up with a convention, e.g. a sub-package is in “/subpackageName” but allow overriding that default. That allows reorganization if needed but the convention should work for most libraries.

Mostly because I am concerned this doesn't scale well to *very* large repositories, in which commits to that file would be "contentious" (in the lock contention sense, not subject to debate sense). Of course, this argument is a little bogus as the current proposal doesn't scale that great either since we have to discover the packages (although I believe we can probably do a good job of caching this information).

It certainly would simplify the implementation & proposal to have this.

The other reason is it is yet another thing for people to maintain (and remember the syntax for). Most repos are small enough that I think the current proposal would perform fine and have a tendency to do what people might naively expect (even if they didn't really think about why). On the other hand, this file is likely to be quite static, so I'm not sure that is a very important issue.

I was already on the fence on this, but I hadn't considered the benefits you mention of allowing import of the package w/ no sub package specifier to mean import of all sub-packages. That tips me a little more towards thinking maybe a better proposal is to KISS and require this in some root file (whether or not that root file is itself a package manifest or a different kind of file is another question, you assume it would be the regular package manifest but I don't think it *need* be, and there is some value in not having any nesting relationship amongst packages).

- Daniel

A top-level Package.swift would also allow immediate detection of non-unique sub-packages, etc. Also if you are using things like git submodules, subtree, or some other mechanism that ends up putting package files in your source tree you don’t automatically re-export that package unless you take explicit action.

I like the idea in general.

Russ

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution