Swift Package Manager - Support for downloading/installing external dependencies that a Swift Package may have

Development Package Manager
1

Hello,

We are currently developing a Swift Package that has a dependency on a C
open source library. This C open source library, on Linux, can be
installed by executing: ‘apt-get install -y <library-name>’.

Though a developer who wanted to use our Swift Package in their
application could specify in their Package.swift file the dependency on
our Swift Package, that would not be enough. The developer would also need
to execute manually the ‘apt-get install -y <library-name>’. We are
thinking this extra step will probably dissuade developers from using our
Swift Package.

We took a quick look around to see what other package management tools are
doing such as NPM (for NodeJS). While doing this, we ran into the
following NodeJS module: node-sass (
https://www.npmjs.com/package/node-sass\). According to their
documentation, it is a library that provides binding to libsass, which is
a library written in C. A NodeJS developer who wants to leverage node-sass
in their application only has to execute: 'npm install node-sass'. Behind
the scenes, NPM seems to be executing all the required commands (such as
apt-get installs) on behalf of the developer for installing any
dependencies (see
https://gist.github.com/edouard-lopez/503d40a5c1a49cf8ae87\). Hence, a
NodeJS developer is completely abstracted from any low level dependencies
(e.g. code written in C/C++) that are required and he/she only has to
execute one command to leverage the NodeJS module (i.e. npm install
<library-name>).

We are wondering if there are any plans to provide similar functionality
in the Swift Package Manager. Maybe something along the lines of these two
points below?

1) Swift Packages can specify dependencies on other libraries that are not
Swift Packages (such as C libraries). The Swift Package Manager could then
"parse" this metadata, download and install the corresponding dependencies
on the system (e.g. by executing apt-get or similar commands). This could
be part of the Packages.swift file or a new file for specifying
dependencies that are not Swift Packages.

2) Define a naming convention for a script that, if found inside a Swift
Package, is executed by the Swift Package Manager. This script could
contain any set of instructions that should be executed as part of the
installation of the Swift Package, such as downloading and installing any
dependencies that the Swift Package needs. For instance, such a file could
include instructions similar to those found here:

Any thoughts on this? Thanks.

Regards,
   Ricardo Olivieri
   Software Engineer

2

Yes, we plan to provide something here.

1) Swift Packages can specify dependencies on other libraries that are not Swift Packages (such as C libraries). The Swift Package Manager could then "parse" this metadata, download and install the corresponding dependencies on the system (e.g. by executing apt-get or similar commands). This could be part of the Packages.swift file or a new file for specifying dependencies that are not Swift Packages.

It couldn’t really be this simple, but we can do something like this.

2) Define a naming convention for a script that, if found inside a Swift Package, is executed by the Swift Package Manager. This script could contain any set of instructions that should be executed as part of the installation of the Swift Package, such as downloading and installing any dependencies that the Swift Package needs. For instance, such a file could include instructions similar to those found here: Installing/Compiling libsass and sassc on Ubuntu 14.04+/Linux Mint 17+ (needed by node-sass) · GitHub

We don’t intend to execute arbitrary scripts.

3

Thanks Max for your answer. To your point, thinking about it more, it's
probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be
implemented and that there are plans to provide something as part of SPM
that can take of installing/downloading external [non-Swift package]
dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other
details about the plans for this feature that could be shared at this
point?

Regards,
   Ricardo Olivieri
   Software Engineer

···

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 01/25/2016 06:53 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Yes, we plan to provide something here.

1) Swift Packages can specify dependencies on other libraries that are not
Swift Packages (such as C libraries). The Swift Package Manager could then
"parse" this metadata, download and install the corresponding dependencies
on the system (e.g. by executing apt-get or similar commands). This could
be part of the Packages.swift file or a new file for specifying
dependencies that are not Swift Packages.

It couldn’t really be this simple, but we can do something like this.

2) Define a naming convention for a script that, if found inside a Swift
Package, is executed by the Swift Package Manager. This script could
contain any set of instructions that should be executed as part of the
installation of the Swift Package, such as downloading and installing any
dependencies that the Swift Package needs. For instance, such a file could
include instructions similar to those found here:

We don’t intend to execute arbitrary scripts.

4

Thanks Max for your answer. To your point, thinking about it more, it's probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be implemented and that there are plans to provide something as part of SPM that can take of installing/downloading external [non-Swift package] dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other details about the plans for this feature that could be shared at this point?

High-ish. There are gaping holes in SwiftPM that are currently higher priority.

This ticket should wait for my up-coming proposal that will improve module-map-packages IMO. Then we can put this in there and implement it at the same time.

IMO it should work like Homebrew’s caveats. If the build fails it outputs suggested commands to install the missing system-packages, then the user can type those commands in if they so choose.

5

Thanks Max again for your response.

I see what you are saying about having SPM fail the build and output
suggested commands to install the missing system-packages (as opposed to
having SPM perform system-package installations).

We are still wondering if we could have a solution built into SPM that
would not required any manual intervention from a human. For instance,
would implementing something like the following be in the plans for SPM
(this is somewhat similar to the original option #1 we shared previously)?

Swift Packages can specify dependencies on other libraries that are not
Swift Packages (such as C libraries). The Swift Package Manager could then
"parse" this metadata, download the C code for that external dependency
(say, from the its GitHub repo), compile it, and make it available to
other Swift Packages through the module.modulemap file. The metadata could
be part of the Packages.swift file or a new file for specifying
dependencies that are not Swift Packages.

Your thoughts?

Regards,
   Ricardo Olivieri
   Software Engineer

···

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org, Samuel Kallner <KALLNER@il.ibm.com>,
Daniel Firsht/Austin/IBM@IBMUS, Andrew M Trice/Washington/IBM@IBMUS,
Robert F Dickerson/Austin/IBM@IBMUS
Date: 01/28/2016 01:08 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Thanks Max for your answer. To your point, thinking about it more, it's
probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be
implemented and that there are plans to provide something as part of SPM
that can take of installing/downloading external [non-Swift package]
dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other
details about the plans for this feature that could be shared at this
point?

High-ish. There are gaping holes in SwiftPM that are currently higher
priority.

This ticket should wait for my up-coming proposal that will improve
module-map-packages IMO. Then we can put this in there and implement it at
the same time.

IMO it should work like Homebrew’s caveats. If the build fails it outputs
suggested commands to install the missing system-packages, then the user
can type those commands in if they so choose.

6

Hello Max, just following up on the previous email. Does the approach
described in the previous message fit with the plans and vision for SPM?
If it does and our team were to take a first shot at implementing
something like it, would our pull request have good chances of getting
merged with the SPM repo?

Regards,
   Ricardo Olivieri
   Software Engineer

···

From: Ricardo N Olivieri/Austin/IBM
To: Max Howell <max.howell@apple.com>
Cc: swift-build-dev@swift.org
Date: 02/01/2016 09:12 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have

Thanks Max again for your response.

I see what you are saying about having SPM fail the build and output
suggested commands to install the missing system-packages (as opposed to
having SPM perform system-package installations).

We are still wondering if we could have a solution built into SPM that
would not required any manual intervention from a human. For instance,
would implementing something like the following be in the plans for SPM
(this is somewhat similar to the original option #1 we shared previously)?

Swift Packages can specify dependencies on other libraries that are not
Swift Packages (such as C libraries). The Swift Package Manager could then
"parse" this metadata, download the C code for that external dependency
(say, from the its GitHub repo), compile it, and make it available to
other Swift Packages through the module.modulemap file. The metadata could
be part of the Packages.swift file or a new file for specifying
dependencies that are not Swift Packages.

Your thoughts?

Regards,
   Ricardo Olivieri
   Software Engineer

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org, Samuel Kallner <KALLNER@il.ibm.com>,
Daniel Firsht/Austin/IBM@IBMUS, Andrew M Trice/Washington/IBM@IBMUS,
Robert F Dickerson/Austin/IBM@IBMUS
Date: 01/28/2016 01:08 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Thanks Max for your answer. To your point, thinking about it more, it's
probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be
implemented and that there are plans to provide something as part of SPM
that can take of installing/downloading external [non-Swift package]
dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other
details about the plans for this feature that could be shared at this
point?

High-ish. There are gaping holes in SwiftPM that are currently higher
priority.

This ticket should wait for my up-coming proposal that will improve
module-map-packages IMO. Then we can put this in there and implement it at
the same time.

IMO it should work like Homebrew’s caveats. If the build fails it outputs
suggested commands to install the missing system-packages, then the user
can type those commands in if they so choose.

7

Hi Ricardo,

Respectfully I’d rather you waited, I have a proposal in the works for major changes to the content of Package.swift and this would have quite an impact on this feature.

I will try to get the proposal to the list soon.

If you want to do the PR anyway we will accept it provided it is not executed as a default (requiring a flag marked explicitly as temporary) and that we will remove the flag when the feature is properly implemented and integrated.

···

Hello Max, just following up on the previous email. Does the approach described in the previous message fit with the plans and vision for SPM? If it does and our team were to take a first shot at implementing something like it, would our pull request have good chances of getting merged with the SPM repo?

Regards,
  Ricardo Olivieri
  Software Engineer

From: Ricardo N Olivieri/Austin/IBM
To: Max Howell <max.howell@apple.com>
Cc: swift-build-dev@swift.org
Date: 02/01/2016 09:12 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for downloading/installing external dependencies that a Swift Package may have

Thanks Max again for your response.

I see what you are saying about having SPM fail the build and output suggested commands to install the missing system-packages (as opposed to having SPM perform system-package installations).

We are still wondering if we could have a solution built into SPM that would not required any manual intervention from a human. For instance, would implementing something like the following be in the plans for SPM (this is somewhat similar to the original option #1 we shared previously)?
Swift Packages can specify dependencies on other libraries that are not Swift Packages (such as C libraries). The Swift Package Manager could then "parse" this metadata, download the C code for that external dependency (say, from the its GitHub repo), compile it, and make it available to other Swift Packages through the module.modulemap file. The metadata could be part of the Packages.swift file or a new file for specifying dependencies that are not Swift Packages.

Your thoughts?

Regards,
  Ricardo Olivieri
  Software Engineer

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org, Samuel Kallner <KALLNER@il.ibm.com>, Daniel Firsht/Austin/IBM@IBMUS, Andrew M Trice/Washington/IBM@IBMUS, Robert F Dickerson/Austin/IBM@IBMUS
Date: 01/28/2016 01:08 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Thanks Max for your answer. To your point, thinking about it more, it's probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be implemented and that there are plans to provide something as part of SPM that can take of installing/downloading external [non-Swift package] dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other details about the plans for this feature that could be shared at this point?

High-ish. There are gaping holes in SwiftPM that are currently higher priority.

This ticket should wait for my up-coming proposal that will improve module-map-packages IMO. Then we can put this in there and implement it at the same time.

IMO it should work like Homebrew’s caveats. If the build fails it outputs suggested commands to install the missing system-packages, then the user can type those commands in if they so choose.

8

Thanks Max for your response! I see what you are saying about your
proposal having an impact on the feature we are thinking of. It would
probably work best if we can align our feature with the changes you plan
to include in your proposal. Do you think you will be documenting/adding
your proposal to the list sometime this week or the following?

Regards,
   Ricardo Olivieri
   Software Engineer

···

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org
Date: 02/17/2016 01:30 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Hi Ricardo,

Respectfully I’d rather you waited, I have a proposal in the works for
major changes to the content of Package.swift and this would have quite an
impact on this feature.

I will try to get the proposal to the list soon.

If you want to do the PR anyway we will accept it provided it is not
executed as a default (requiring a flag marked explicitly as temporary)
and that we will remove the flag when the feature is properly implemented
and integrated.

Hello Max, just following up on the previous email. Does the approach
described in the previous message fit with the plans and vision for SPM?
If it does and our team were to take a first shot at implementing
something like it, would our pull request have good chances of getting
merged with the SPM repo?

Regards,
  Ricardo Olivieri
  Software Engineer

From: Ricardo N Olivieri/Austin/IBM
To: Max Howell <max.howell@apple.com>
Cc: swift-build-dev@swift.org
Date: 02/01/2016 09:12 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have

Thanks Max again for your response.

I see what you are saying about having SPM fail the build and output
suggested commands to install the missing system-packages (as opposed to
having SPM perform system-package installations).

We are still wondering if we could have a solution built into SPM that
would not required any manual intervention from a human. For instance,
would implementing something like the following be in the plans for SPM
(this is somewhat similar to the original option #1 we shared previously)?
Swift Packages can specify dependencies on other libraries that are not
Swift Packages (such as C libraries). The Swift Package Manager could then
"parse" this metadata, download the C code for that external dependency
(say, from the its GitHub repo), compile it, and make it available to
other Swift Packages through the module.modulemap file. The metadata could
be part of the Packages.swift file or a new file for specifying
dependencies that are not Swift Packages.

Your thoughts?

Regards,
  Ricardo Olivieri
  Software Engineer

From: Max Howell <max.howell@apple.com>
To: Ricardo N Olivieri/Austin/IBM@IBMUS
Cc: swift-build-dev@swift.org, Samuel Kallner <KALLNER@il.ibm.com>,
Daniel Firsht/Austin/IBM@IBMUS, Andrew M Trice/Washington/IBM@IBMUS,
Robert F Dickerson/Austin/IBM@IBMUS
Date: 01/28/2016 01:08 PM
Subject: Re: [swift-build-dev] Swift Package Manager - Support for
downloading/installing external dependencies that a Swift Package may have
Sent by: max.howell@apple.com

Thanks Max for your answer. To your point, thinking about it more, it's
probably not a great idea to have SPM execute arbitrary scripts.

It is good news to hear that something similar to option #1 could be
implemented and that there are plans to provide something as part of SPM
that can take of installing/downloading external [non-Swift package]
dependencies that a Swift Package may depend on.

Does providing this functionality in SPM have a high priority? Any other
details about the plans for this feature that could be shared at this
point?

High-ish. There are gaping holes in SwiftPM that are currently higher
priority.

This ticket should wait for my up-coming proposal that will improve
module-map-packages IMO. Then we can put this in there and implement it at
the same time.

IMO it should work like Homebrew’s caveats. If the build fails it outputs
suggested commands to install the missing system-packages, then the user
can type those commands in if they so choose.