importing a *local* native dependency

Development Package Manager
1

I'm trying to package a Swift library. This Swift library is tightly coupled to a native library. By "tightly coupled" here I really mean that only half the library is written in Swift, and the other half, as an implementation detail, is in some other language.

You *probably* do not have a build environment for this other language just lying around, so my plan (actually, my current practice) is to distribute binaries of the non-Swift part, just checked into source control, and then link against them with a module map. Yes, I realize this isn't portable, but neither is anything else in this project, so I'll save that particular surprise for another day.

The trouble is, swiftpm wants there to be some "other" repo called `CFoo` containing only a modulemap with paths that point to... /usr/local I guess? <https://github.com/apple/swift-package-manager/blob/master/Documentation/SystemModules.md&gt; But

1. There is no other repo
2. You have not installed the non-Swift half of my library to /usr/local
3. You do not want to link against "some" version of the binary blob you have lying around from times past. You want to link against exactly the one I give you, because the work is versioned as a whole.

I see two solutions to this problem.

# Non-URL dependencies

Behind this door, I can write a Package.swift like this:

let package = Package(
    name: "Foo",
    dependencies: [
        .Package(path: "CFoo"),
    ],
)

The semantic of this is that Foo depends on a package CFoo that is merely a folder in the current repository (e.g., we do not need to clone it from somewhere else).

In my case CFoo would contain a module.map and a binary blob, but in the general case it is any arbitrary kind of package.

This solves my three problems since 1) There's only one repo, 2) The binary is located with repo-based paths, 3) You're linking against the particular CFoo distributed in the repository.

# Custom include paths

Another possibility goes like this:

let package = Package(
    name: "Foo",
    extraIncludePaths: ["CFoo"]
)

The semantic of this is that we pass `-I CFoo` to swiftc (e.g. that is emitted into `other-args` of llbuild.yaml)

In my case CFoo contains a module.map and a binary blob, but in the general case it contains any kind of header path that a package may want to import.

There are certainly other possibilities besides these two, I'm just suggesting them because they seem straightforward based on my poking around in the sourcecode today.

I am happy to contrib either of these (or something else proposed) but would like to achieve consensus on a design instead of PRing by surprise.

2

Just to be clear...

If I PRed one or both of these, it would be rejected?

I'm going to cook up some solution in any case, it's a question of whether I should be gearing up to distribute and maintain a long term fork. It's undesirable since it complicates upstreaming further work I do in my tree.

But it will be unavoidable if I need a solution before you want to design it, which sounds like it may be the present situation.

···

Sent from my iPhone

On Dec 25, 2015, at 9:36 AM, Max Howell <max.howell@apple.com> wrote:

This fulls under our "mixed targets" future plan.

For now I think you'll not be able to use swiftpm at all. Sorry :(

On Dec 25, 2015, at 3:19 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org> wrote:

I'm trying to package a Swift library. This Swift library is tightly coupled to a native library. By "tightly coupled" here I really mean that only half the library is written in Swift, and the other half, as an implementation detail, is in some other language.

You *probably* do not have a build environment for this other language just lying around, so my plan (actually, my current practice) is to distribute binaries of the non-Swift part, just checked into source control, and then link against them with a module map. Yes, I realize this isn't portable, but neither is anything else in this project, so I'll save that particular surprise for another day.

The trouble is, swiftpm wants there to be some "other" repo called `CFoo` containing only a modulemap with paths that point to... /usr/local I guess? But

1. There is no other repo
2. You have not installed the non-Swift half of my library to /usr/local
3. You do not want to link against "some" version of the binary blob you have lying around from times past. You want to link against exactly the one I give you, because the work is versioned as a whole.

I see two solutions to this problem.

# Non-URL dependencies

Behind this door, I can write a Package.swift like this:

let package = Package(
    name: "Foo",
    dependencies: [
        .Package(path: "CFoo"),
    ],
)

The semantic of this is that Foo depends on a package CFoo that is merely a folder in the current repository (e.g., we do not need to clone it from somewhere else).

In my case CFoo would contain a module.map and a binary blob, but in the general case it is any arbitrary kind of package.

This solves my three problems since 1) There's only one repo, 2) The binary is located with repo-based paths, 3) You're linking against the particular CFoo distributed in the repository.

# Custom include paths

Another possibility goes like this:

let package = Package(
    name: "Foo",
    extraIncludePaths: ["CFoo"]
)

The semantic of this is that we pass `-I CFoo` to swiftc (e.g. that is emitted into `other-args` of llbuild.yaml)

In my case CFoo contains a module.map and a binary blob, but in the general case it contains any kind of header path that a package may want to import.

There are certainly other possibilities besides these two, I'm just suggesting them because they seem straightforward based on my poking around in the sourcecode today.

I am happy to contrib either of these (or something else proposed) but would like to achieve consensus on a design instead of PRing by surprise.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev

3

Hi Drew,

This use case – facilitating packages that include pre-built binaries – is not one that we want to support right now. For the foreseeable future we'd really like to stick to source-based distribution, as discussed in https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#source-based-distribution\.

Your proposal for dependencies on subpaths within a repo is one that we might consider at some point independent of this problem, but we'd need a better driving reason to do so. (We've had some discussions internally about whether to enforce that each package lives at the top level of a repo, vs supporting repos that contains multiple packages, and, at least for now, I think we'd like to have a repo per package). Your second proposal, for custom include paths, seems really specific to supporting pre-built binaries, so I don't think that aligns with our goals.

Our "Future Features" list does include "Support for Other Build Systems" (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#support-for-other-build-systems\), so long-term I think I'd like to see us develop a proposal for that as the solution to your problem here.

So at the moment, I don't think I would want to accept a PR for either of these proposals. But I really appreciate your bringing this up.

Thank you,

  - Rick

···

On Dec 25, 2015, at 8:22 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org> wrote:

Just to be clear...

If I PRed one or both of these, it would be rejected?

I'm going to cook up some solution in any case, it's a question of whether I should be gearing up to distribute and maintain a long term fork. It's undesirable since it complicates upstreaming further work I do in my tree.

But it will be unavoidable if I need a solution before you want to design it, which sounds like it may be the present situation.

Sent from my iPhone

On Dec 25, 2015, at 9:36 AM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com>> wrote:

This fulls under our "mixed targets" future plan.

For now I think you'll not be able to use swiftpm at all. Sorry :(

On Dec 25, 2015, at 3:19 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I'm trying to package a Swift library. This Swift library is tightly coupled to a native library. By "tightly coupled" here I really mean that only half the library is written in Swift, and the other half, as an implementation detail, is in some other language.

You *probably* do not have a build environment for this other language just lying around, so my plan (actually, my current practice) is to distribute binaries of the non-Swift part, just checked into source control, and then link against them with a module map. Yes, I realize this isn't portable, but neither is anything else in this project, so I'll save that particular surprise for another day.

The trouble is, swiftpm wants there to be some "other" repo called `CFoo` containing only a modulemap with paths that point to... /usr/local I guess? <https://github.com/apple/swift-package-manager/blob/master/Documentation/SystemModules.md&gt; But

1. There is no other repo
2. You have not installed the non-Swift half of my library to /usr/local
3. You do not want to link against "some" version of the binary blob you have lying around from times past. You want to link against exactly the one I give you, because the work is versioned as a whole.

I see two solutions to this problem.

# Non-URL dependencies

Behind this door, I can write a Package.swift like this:

let package = Package(
    name: "Foo",
    dependencies: [
        .Package(path: "CFoo"),
    ],
)

The semantic of this is that Foo depends on a package CFoo that is merely a folder in the current repository (e.g., we do not need to clone it from somewhere else).

In my case CFoo would contain a module.map and a binary blob, but in the general case it is any arbitrary kind of package.

This solves my three problems since 1) There's only one repo, 2) The binary is located with repo-based paths, 3) You're linking against the particular CFoo distributed in the repository.

# Custom include paths

Another possibility goes like this:

let package = Package(
    name: "Foo",
    extraIncludePaths: ["CFoo"]
)

The semantic of this is that we pass `-I CFoo` to swiftc (e.g. that is emitted into `other-args` of llbuild.yaml)

In my case CFoo contains a module.map and a binary blob, but in the general case it contains any kind of header path that a package may want to import.

There are certainly other possibilities besides these two, I'm just suggesting them because they seem straightforward based on my poking around in the sourcecode today.

I am happy to contrib either of these (or something else proposed) but would like to achieve consensus on a design instead of PRing by surprise.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

4

Our "Future Features" list does include "Support for Other Build Systems" (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#support-for-other-build-systems\), so long-term I think I'd like to see us develop a proposal for that as the solution to your problem here.

If I'm not mistaken, one solution to this problem would be

* Add support for custom include paths
* Add a "preflight step" where a package can run a bash script as part of its build process
* Make the bash script run the external build system and point the include paths to a modulemap pointing to built products

I have an implementation + tests for all of these, plus I've used them to package libdispatch, in a private fork. That solution exists today.

I'm not saying it's the best solution. Would it make sense to sit down and spec out what is the "right way" to support an automake project like libdispatch? I would be more than happy to reimplement this some other way, if someone can explain what way that is.

I'm trying to minimize the scope of what I'm going to maintain myself in my fork, but being able to link C and Rust dependencies is non-optional for me, so I just went ahead and did it. I think it would be unfortunate if this feature is only available from me for awhile.

Your second proposal, for custom include paths, seems really specific to supporting pre-built binaries

Based on the above, this seems plainly false. I am using it to package libdispatch, which is a source package. There may be good reasons not to do this, but I don't think this argument is one.

···

On Jan 2, 2016, at 2:01 PM, Rick Ballard <rballard@apple.com> wrote:

Hi Drew,

This use case – facilitating packages that include pre-built binaries – is not one that we want to support right now. For the foreseeable future we'd really like to stick to source-based distribution, as discussed in https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#source-based-distribution\.

Your proposal for dependencies on subpaths within a repo is one that we might consider at some point independent of this problem, but we'd need a better driving reason to do so. (We've had some discussions internally about whether to enforce that each package lives at the top level of a repo, vs supporting repos that contains multiple packages, and, at least for now, I think we'd like to have a repo per package). Your second proposal, for custom include paths, seems really specific to supporting pre-built binaries, so I don't think that aligns with our goals.

Our "Future Features" list does include "Support for Other Build Systems" (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#support-for-other-build-systems\), so long-term I think I'd like to see us develop a proposal for that as the solution to your problem here.

So at the moment, I don't think I would want to accept a PR for either of these proposals. But I really appreciate your bringing this up.

Thank you,

  - Rick

On Dec 25, 2015, at 8:22 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Just to be clear...

If I PRed one or both of these, it would be rejected?

I'm going to cook up some solution in any case, it's a question of whether I should be gearing up to distribute and maintain a long term fork. It's undesirable since it complicates upstreaming further work I do in my tree.

But it will be unavoidable if I need a solution before you want to design it, which sounds like it may be the present situation.

Sent from my iPhone

On Dec 25, 2015, at 9:36 AM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com>> wrote:

This fulls under our "mixed targets" future plan.

For now I think you'll not be able to use swiftpm at all. Sorry :(

On Dec 25, 2015, at 3:19 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I'm trying to package a Swift library. This Swift library is tightly coupled to a native library. By "tightly coupled" here I really mean that only half the library is written in Swift, and the other half, as an implementation detail, is in some other language.

You *probably* do not have a build environment for this other language just lying around, so my plan (actually, my current practice) is to distribute binaries of the non-Swift part, just checked into source control, and then link against them with a module map. Yes, I realize this isn't portable, but neither is anything else in this project, so I'll save that particular surprise for another day.

The trouble is, swiftpm wants there to be some "other" repo called `CFoo` containing only a modulemap with paths that point to... /usr/local I guess? <https://github.com/apple/swift-package-manager/blob/master/Documentation/SystemModules.md&gt; But

1. There is no other repo
2. You have not installed the non-Swift half of my library to /usr/local
3. You do not want to link against "some" version of the binary blob you have lying around from times past. You want to link against exactly the one I give you, because the work is versioned as a whole.

I see two solutions to this problem.

# Non-URL dependencies

Behind this door, I can write a Package.swift like this:

let package = Package(
    name: "Foo",
    dependencies: [
        .Package(path: "CFoo"),
    ],
)

The semantic of this is that Foo depends on a package CFoo that is merely a folder in the current repository (e.g., we do not need to clone it from somewhere else).

In my case CFoo would contain a module.map and a binary blob, but in the general case it is any arbitrary kind of package.

This solves my three problems since 1) There's only one repo, 2) The binary is located with repo-based paths, 3) You're linking against the particular CFoo distributed in the repository.

# Custom include paths

Another possibility goes like this:

let package = Package(
    name: "Foo",
    extraIncludePaths: ["CFoo"]
)

The semantic of this is that we pass `-I CFoo` to swiftc (e.g. that is emitted into `other-args` of llbuild.yaml)

In my case CFoo contains a module.map and a binary blob, but in the general case it contains any kind of header path that a package may want to import.

There are certainly other possibilities besides these two, I'm just suggesting them because they seem straightforward based on my poking around in the sourcecode today.

I am happy to contrib either of these (or something else proposed) but would like to achieve consensus on a design instead of PRing by surprise.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

5

Our "Future Features" list does include "Support for Other Build Systems" (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#support-for-other-build-systems\), so long-term I think I'd like to see us develop a proposal for that as the solution to your problem here.

If I'm not mistaken, one solution to this problem would be

* Add support for custom include paths
* Add a "preflight step" where a package can run a bash script as part of its build process
* Make the bash script run the external build system and point the include paths to a modulemap pointing to built products

I have an implementation + tests for all of these, plus I've used them to package libdispatch, in a private fork. That solution exists today.

I'm not saying it's the best solution. Would it make sense to sit down and spec out what is the "right way" to support an automake project like libdispatch? I would be more than happy to reimplement this some other way, if someone can explain what way that is.

I think this would be a reasonable approach, yes. I do think that long term we should have some support for projects that build using an "external" system. We will want to design that carefully though, so this might be more lead time than you want.

One thing I would rather not do now is support arbitrary things that introduce external dependencies (like custom include paths). If we are going to support external things, I would still like it to be done in a way where everything is done internal to the package's sandbox.

For your problem specifically, would it be addressed if:
- libdispatch was shipped as part of the Swift distribution? This should happen eventually once the Linux version is stable enough. This isn't a generally solution, of course.
- the other C parts were made to fit into my "C language targets" proposal? Does it cover what you need to do?

- Daniel

···

On Jan 2, 2016, at 1:08 PM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org> wrote:

I'm trying to minimize the scope of what I'm going to maintain myself in my fork, but being able to link C and Rust dependencies is non-optional for me, so I just went ahead and did it. I think it would be unfortunate if this feature is only available from me for awhile.

Your second proposal, for custom include paths, seems really specific to supporting pre-built binaries

Based on the above, this seems plainly false. I am using it to package libdispatch, which is a source package. There may be good reasons not to do this, but I don't think this argument is one.

On Jan 2, 2016, at 2:01 PM, Rick Ballard <rballard@apple.com <mailto:rballard@apple.com>> wrote:

Hi Drew,

This use case – facilitating packages that include pre-built binaries – is not one that we want to support right now. For the foreseeable future we'd really like to stick to source-based distribution, as discussed in https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#source-based-distribution\.

Your proposal for dependencies on subpaths within a repo is one that we might consider at some point independent of this problem, but we'd need a better driving reason to do so. (We've had some discussions internally about whether to enforce that each package lives at the top level of a repo, vs supporting repos that contains multiple packages, and, at least for now, I think we'd like to have a repo per package). Your second proposal, for custom include paths, seems really specific to supporting pre-built binaries, so I don't think that aligns with our goals.

Our "Future Features" list does include "Support for Other Build Systems" (https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageManagerCommunityProposal.md#support-for-other-build-systems\), so long-term I think I'd like to see us develop a proposal for that as the solution to your problem here.

So at the moment, I don't think I would want to accept a PR for either of these proposals. But I really appreciate your bringing this up.

Thank you,

  - Rick

On Dec 25, 2015, at 8:22 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Just to be clear...

If I PRed one or both of these, it would be rejected?

I'm going to cook up some solution in any case, it's a question of whether I should be gearing up to distribute and maintain a long term fork. It's undesirable since it complicates upstreaming further work I do in my tree.

But it will be unavoidable if I need a solution before you want to design it, which sounds like it may be the present situation.

Sent from my iPhone

On Dec 25, 2015, at 9:36 AM, Max Howell <max.howell@apple.com <mailto:max.howell@apple.com>> wrote:

This fulls under our "mixed targets" future plan.

For now I think you'll not be able to use swiftpm at all. Sorry :(

On Dec 25, 2015, at 3:19 AM, Drew Crawford via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

I'm trying to package a Swift library. This Swift library is tightly coupled to a native library. By "tightly coupled" here I really mean that only half the library is written in Swift, and the other half, as an implementation detail, is in some other language.

You *probably* do not have a build environment for this other language just lying around, so my plan (actually, my current practice) is to distribute binaries of the non-Swift part, just checked into source control, and then link against them with a module map. Yes, I realize this isn't portable, but neither is anything else in this project, so I'll save that particular surprise for another day.

The trouble is, swiftpm wants there to be some "other" repo called `CFoo` containing only a modulemap with paths that point to... /usr/local I guess? <https://github.com/apple/swift-package-manager/blob/master/Documentation/SystemModules.md&gt; But

1. There is no other repo
2. You have not installed the non-Swift half of my library to /usr/local
3. You do not want to link against "some" version of the binary blob you have lying around from times past. You want to link against exactly the one I give you, because the work is versioned as a whole.

I see two solutions to this problem.

# Non-URL dependencies

Behind this door, I can write a Package.swift like this:

let package = Package(
    name: "Foo",
    dependencies: [
        .Package(path: "CFoo"),
    ],
)

The semantic of this is that Foo depends on a package CFoo that is merely a folder in the current repository (e.g., we do not need to clone it from somewhere else).

In my case CFoo would contain a module.map and a binary blob, but in the general case it is any arbitrary kind of package.

This solves my three problems since 1) There's only one repo, 2) The binary is located with repo-based paths, 3) You're linking against the particular CFoo distributed in the repository.

# Custom include paths

Another possibility goes like this:

let package = Package(
    name: "Foo",
    extraIncludePaths: ["CFoo"]
)

The semantic of this is that we pass `-I CFoo` to swiftc (e.g. that is emitted into `other-args` of llbuild.yaml)

In my case CFoo contains a module.map and a binary blob, but in the general case it contains any kind of header path that a package may want to import.

There are certainly other possibilities besides these two, I'm just suggesting them because they seem straightforward based on my poking around in the sourcecode today.

I am happy to contrib either of these (or something else proposed) but would like to achieve consensus on a design instead of PRing by surprise.

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

6

I think this is a very reasonable limitation.

What if we did something like this?

* Run a "./preflight" script/executable if it exists. This could call an external build system. Alternatively the path to the preflight script could be specified in Package.swift
* Define a particular folder like include/ that we pass as `-I include/` if it exists.

Then "preflight" can create that folder in some way–whether that is copying header files around or what–and have it be included in the include paths.

That way the include path is internal to the package sandbox.

···

On Jan 2, 2016, at 3:13 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

One thing I would rather not do now is support arbitrary things that introduce external dependencies (like custom include paths). If we are going to support external things, I would still like it to be done in a way where everything is done internal to the package's sandbox.