[Discussion Starter] Package post-clone script


(Honza Dvorsky) #1

Hi All,

today I was trying to investigate how to build a proper cross-platform
server-side Swift project and only use SwiftPM to do it. In particular, I
was interested in getting OpenSSL/libcurl working as a dependency (built
from source). The main reason is that currently the big Swift server
frameworks all require manual steps during deployment (running apt-get/brew
to install libcurl/openssl/others) and I am trying to get rid of these
manual steps, eventually only requiring a `swift build` and run.

With the great initial work
<https://github.com/apple/swift-package-manager/pull/183> to support
building C-family packages in SwiftPM done by Ankit, I was able to have a
simple C package as a dependency, as advertised. However, the reality of
big cross-platform open source projects like OpenSSL and libcurl is that in
order to actually build it from source, a configure (or similar) script has
to be run to get the source code ready for compilation. And this is where I
got stuck.

Thus I wanted to kick off a discussion of what approach of solving this
problem with SwiftPM should be. The simple solution, which I'm tentatively
proposing, is to have an optional "post-clone" script in the package's
repository. Then, the Package.swift would optionally contain a field for
the path to this script - and if present, it'd get run after SwiftPM clones
this package. Note that this would be a "post-clone" script, not a
"prebuild" script - I imagine it'd only run once when cloning and then only
after each clean.

In order to get users to stop having to run manual script to install all
dependencies, I believe that we need to allow packages to declare what work
needs to be done on its source before compilation can begin.

What do you think?

Honza Dvorsky


(Max Howell) #2

I have a proposal I’d like to push today that adds knowledge to SwiftPM about how to install system dependencies for System Module Packages.

We are not keen to add arbitrary script execution to SwiftPM, because: 1) Arbitrary scripts cannot be controlled and this leads to dependency hell and 2) arbitrary scripts mean your package graph may do anything, which nobody wants.

Now I’m fine with the “root package” ie. the package the user has control over doing more, but this would be a different discussion.

So the question becomes, is this sufficient? To have `brew install openssl` run, or do you really need to install from source?

Max

···

On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org> wrote:

Hi All,

today I was trying to investigate how to build a proper cross-platform server-side Swift project and only use SwiftPM to do it. In particular, I was interested in getting OpenSSL/libcurl working as a dependency (built from source). The main reason is that currently the big Swift server frameworks all require manual steps during deployment (running apt-get/brew to install libcurl/openssl/others) and I am trying to get rid of these manual steps, eventually only requiring a `swift build` and run.

With the great initial work <https://github.com/apple/swift-package-manager/pull/183> to support building C-family packages in SwiftPM done by Ankit, I was able to have a simple C package as a dependency, as advertised. However, the reality of big cross-platform open source projects like OpenSSL and libcurl is that in order to actually build it from source, a configure (or similar) script has to be run to get the source code ready for compilation. And this is where I got stuck.

Thus I wanted to kick off a discussion of what approach of solving this problem with SwiftPM should be. The simple solution, which I'm tentatively proposing, is to have an optional "post-clone" script in the package's repository. Then, the Package.swift would optionally contain a field for the path to this script - and if present, it'd get run after SwiftPM clones this package. Note that this would be a "post-clone" script, not a "prebuild" script - I imagine it'd only run once when cloning and then only after each clean.

In order to get users to stop having to run manual script to install all dependencies, I believe that we need to allow packages to declare what work needs to be done on its source before compilation can begin.


(Honza Dvorsky) #3

I personally don't like the idea of an arbitrary script either, it was just
the first obvious solution to a problem I believe needs solving. The
advantage of building from source even of these base dependencies is that
the exact version of system dependencies
- can be ensured to be the same version across platforms
- their version is part of the explicit dependency graph

Currently, AFAIR, installing Redis with homebrew and apt-get each produce a
different version, which IMO defeats the point of having a dependency
manager in the first place. But I guess if your proposal could ensure that
the same version is installed on all platforms, this particular problem
would be solved.

So the question becomes, is this sufficient? To have `brew install

openssl` run, or do you really need to install from source?

I'm still not sure. Now I think your proposal will probably help with most
cases (which is a very good way to approach things), but definitely not
all. Imagine you yourself having a cross-platform C project which needs
platform configuring before compilation. Even when you're completely in
control, the fact that you still can't get this working is troubling. But
maybe I'm just inflating extremely uncommon usecases.

One more thing - consider two Swift packages each needing a different
version of OpenSSL. Does your proposal handle that case?

Anyway, I'd love to read your proposal and then re-evaluate whether I still
feel like there's a large enough benefit in introducing such customization
to SwiftPM :slight_smile:

Honza

···

On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com> wrote:

On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev < > swift-build-dev@swift.org> wrote:

Hi All,

today I was trying to investigate how to build a proper cross-platform
server-side Swift project and only use SwiftPM to do it. In particular, I
was interested in getting OpenSSL/libcurl working as a dependency (built
from source). The main reason is that currently the big Swift server
frameworks all require manual steps during deployment (running apt-get/brew
to install libcurl/openssl/others) and I am trying to get rid of these
manual steps, eventually only requiring a `swift build` and run.

With the great initial work
<https://github.com/apple/swift-package-manager/pull/183> to support
building C-family packages in SwiftPM done by Ankit, I was able to have a
simple C package as a dependency, as advertised. However, the reality of
big cross-platform open source projects like OpenSSL and libcurl is that in
order to actually build it from source, a configure (or similar) script has
to be run to get the source code ready for compilation. And this is where I
got stuck.

Thus I wanted to kick off a discussion of what approach of solving this
problem with SwiftPM should be. The simple solution, which I'm tentatively
proposing, is to have an optional "post-clone" script in the package's
repository. Then, the Package.swift would optionally contain a field for
the path to this script - and if present, it'd get run after SwiftPM clones
this package. Note that this would be a "post-clone" script, not a
"prebuild" script - I imagine it'd only run once when cloning and then only
after each clean.

In order to get users to stop having to run manual script to install all
dependencies, I believe that we need to allow packages to declare what work
needs to be done on its source before compilation can begin.

I have a proposal I’d like to push today that adds knowledge to SwiftPM
about how to install system dependencies for System Module Packages.

We are not keen to add arbitrary script execution to SwiftPM, because: 1)
Arbitrary scripts cannot be controlled and this leads to dependency hell
and 2) arbitrary scripts mean your package graph may do anything, which
nobody wants.

Now I’m fine with the “root package” ie. the package the user has control
over doing more, but this would be a different discussion.

So the question becomes, is this sufficient? To have `brew install
openssl` run, or do you really need to install from source?

Max


(Chris Bailey) #4

To add one more question on the installation of system dependencies - does
that then drive a need to install as root/sudo? If so, I can see that
being a challenge for any cloud based application deployments, where your
both unlikely to be able to run as root/sudo, or indeed be able to install
into /usr/lib

Chris

···

From: Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org>
To: Max Howell <max.howell@apple.com>
Cc: "swift-build-dev@swift.org" <swift-build-dev@swift.org>
Date: 21/03/2016 18:38
Subject: Re: [swift-build-dev] [Discussion Starter] Package
post-clone script
Sent by: swift-build-dev-bounces@swift.org

I personally don't like the idea of an arbitrary script either, it was
just the first obvious solution to a problem I believe needs solving. The
advantage of building from source even of these base dependencies is that
the exact version of system dependencies
- can be ensured to be the same version across platforms
- their version is part of the explicit dependency graph

Currently, AFAIR, installing Redis with homebrew and apt-get each produce
a different version, which IMO defeats the point of having a dependency
manager in the first place. But I guess if your proposal could ensure that
the same version is installed on all platforms, this particular problem
would be solved.

So the question becomes, is this sufficient? To have `brew install

openssl` run, or do you really need to install from source?

I'm still not sure. Now I think your proposal will probably help with most
cases (which is a very good way to approach things), but definitely not
all. Imagine you yourself having a cross-platform C project which needs
platform configuring before compilation. Even when you're completely in
control, the fact that you still can't get this working is troubling. But
maybe I'm just inflating extremely uncommon usecases.

One more thing - consider two Swift packages each needing a different
version of OpenSSL. Does your proposal handle that case?

Anyway, I'd love to read your proposal and then re-evaluate whether I
still feel like there's a large enough benefit in introducing such
customization to SwiftPM :slight_smile:

Honza

On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com> wrote:

On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev < swift-build-dev@swift.org> wrote:

Hi All,

today I was trying to investigate how to build a proper cross-platform
server-side Swift project and only use SwiftPM to do it. In particular, I
was interested in getting OpenSSL/libcurl working as a dependency (built
from source). The main reason is that currently the big Swift server
frameworks all require manual steps during deployment (running
apt-get/brew to install libcurl/openssl/others) and I am trying to get rid
of these manual steps, eventually only requiring a `swift build` and run.

With the great initial work to support building C-family packages in
SwiftPM done by Ankit, I was able to have a simple C package as a
dependency, as advertised. However, the reality of big cross-platform open
source projects like OpenSSL and libcurl is that in order to actually
build it from source, a configure (or similar) script has to be run to get
the source code ready for compilation. And this is where I got stuck.

Thus I wanted to kick off a discussion of what approach of solving this
problem with SwiftPM should be. The simple solution, which I'm tentatively
proposing, is to have an optional "post-clone" script in the package's
repository. Then, the Package.swift would optionally contain a field for
the path to this script - and if present, it'd get run after SwiftPM
clones this package. Note that this would be a "post-clone" script, not a
"prebuild" script - I imagine it'd only run once when cloning and then
only after each clean.

In order to get users to stop having to run manual script to install all
dependencies, I believe that we need to allow packages to declare what
work needs to be done on its source before compilation can begin.

I have a proposal I’d like to push today that adds knowledge to SwiftPM
about how to install system dependencies for System Module Packages.

We are not keen to add arbitrary script execution to SwiftPM, because: 1)
Arbitrary scripts cannot be controlled and this leads to dependency hell
and 2) arbitrary scripts mean your package graph may do anything, which
nobody wants.

Now I’m fine with the “root package” ie. the package the user has control
over doing more, but this would be a different discussion.

So the question becomes, is this sufficient? To have `brew install
openssl` run, or do you really need to install from source?

Max
_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #5

My $0.02 here is that I do want to see us have some support for plugging in arbitrary build processes. However, I want it to be done in a very controlled manner, so that we are guaranteed we can evolve the package manager in sensible ways. I definitely would not want this to take the form of an arbitrary "post clone" script that we just run, with no other imposed structure.

I have a concept in mind, but have not had time to write it up.

I think Max and I agree that we don't want to support "scripts" just as a way to build things which we should have well-defined, proper ways to build in the first place. It would be nice to see how ffar we can get with our own features or adaption of existing technologies before adding this capability...

- Daniel

···

On Mar 21, 2016, at 11:38 AM, Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org> wrote:

I personally don't like the idea of an arbitrary script either, it was just the first obvious solution to a problem I believe needs solving. The advantage of building from source even of these base dependencies is that the exact version of system dependencies
- can be ensured to be the same version across platforms
- their version is part of the explicit dependency graph

Currently, AFAIR, installing Redis with homebrew and apt-get each produce a different version, which IMO defeats the point of having a dependency manager in the first place. But I guess if your proposal could ensure that the same version is installed on all platforms, this particular problem would be solved.

> So the question becomes, is this sufficient? To have `brew install openssl` run, or do you really need to install from source?

I'm still not sure. Now I think your proposal will probably help with most cases (which is a very good way to approach things), but definitely not all. Imagine you yourself having a cross-platform C project which needs platform configuring before compilation. Even when you're completely in control, the fact that you still can't get this working is troubling. But maybe I'm just inflating extremely uncommon usecases.

One more thing - consider two Swift packages each needing a different version of OpenSSL. Does your proposal handle that case?

Anyway, I'd love to read your proposal and then re-evaluate whether I still feel like there's a large enough benefit in introducing such customization to SwiftPM :slight_smile:

Honza

On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com <mailto:max.howell@apple.com>> wrote:

On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

Hi All,

today I was trying to investigate how to build a proper cross-platform server-side Swift project and only use SwiftPM to do it. In particular, I was interested in getting OpenSSL/libcurl working as a dependency (built from source). The main reason is that currently the big Swift server frameworks all require manual steps during deployment (running apt-get/brew to install libcurl/openssl/others) and I am trying to get rid of these manual steps, eventually only requiring a `swift build` and run.

With the great initial work <https://github.com/apple/swift-package-manager/pull/183> to support building C-family packages in SwiftPM done by Ankit, I was able to have a simple C package as a dependency, as advertised. However, the reality of big cross-platform open source projects like OpenSSL and libcurl is that in order to actually build it from source, a configure (or similar) script has to be run to get the source code ready for compilation. And this is where I got stuck.

Thus I wanted to kick off a discussion of what approach of solving this problem with SwiftPM should be. The simple solution, which I'm tentatively proposing, is to have an optional "post-clone" script in the package's repository. Then, the Package.swift would optionally contain a field for the path to this script - and if present, it'd get run after SwiftPM clones this package. Note that this would be a "post-clone" script, not a "prebuild" script - I imagine it'd only run once when cloning and then only after each clean.

In order to get users to stop having to run manual script to install all dependencies, I believe that we need to allow packages to declare what work needs to be done on its source before compilation can begin.

I have a proposal I’d like to push today that adds knowledge to SwiftPM about how to install system dependencies for System Module Packages.

We are not keen to add arbitrary script execution to SwiftPM, because: 1) Arbitrary scripts cannot be controlled and this leads to dependency hell and 2) arbitrary scripts mean your package graph may do anything, which nobody wants.

Now I’m fine with the “root package” ie. the package the user has control over doing more, but this would be a different discussion.

So the question becomes, is this sufficient? To have `brew install openssl` run, or do you really need to install from source?

Max

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Honza Dvorsky) #6

Thanks everyone, I agree with most things said here. Daniel, please let us
know when you have time to write up your proposed solution. As I'm trying
to get more serious with server-side Swift applications, the inability to
use external libraries like libCurl and OpenSSL without manual installation
are holding me back most at the moment.

I'll welcome any solution that gets us closer to solving that problem and
honestly, I'm not a fan of the arbitrary script solution either, so I'm
happy to see any realistic alternatives.

Honza

···

On Wed, Mar 23, 2016 at 12:05 AM Daniel Dunbar <daniel_dunbar@apple.com> wrote:

My $0.02 here is that I do want to see us have some support for plugging
in arbitrary build processes. However, I want it to be done in a very
controlled manner, so that we are guaranteed we can evolve the package
manager in sensible ways. I definitely would not want this to take the form
of an arbitrary "post clone" script that we just run, with no other imposed
structure.

I have a concept in mind, but have not had time to write it up.

I think Max and I agree that we don't want to support "scripts" just as a
way to build things which we should have well-defined, proper ways to build
in the first place. It would be nice to see how ffar we can get with our
own features or adaption of existing technologies before adding this
capability...

- Daniel

On Mar 21, 2016, at 11:38 AM, Honza Dvorsky via swift-build-dev < > swift-build-dev@swift.org> wrote:

I personally don't like the idea of an arbitrary script either, it was
just the first obvious solution to a problem I believe needs solving. The
advantage of building from source even of these base dependencies is that
the exact version of system dependencies
- can be ensured to be the same version across platforms
- their version is part of the explicit dependency graph

Currently, AFAIR, installing Redis with homebrew and apt-get each produce
a different version, which IMO defeats the point of having a dependency
manager in the first place. But I guess if your proposal could ensure that
the same version is installed on all platforms, this particular problem
would be solved.

> So the question becomes, is this sufficient? To have `brew install
openssl` run, or do you really need to install from source?

I'm still not sure. Now I think your proposal will probably help with most
cases (which is a very good way to approach things), but definitely not
all. Imagine you yourself having a cross-platform C project which needs
platform configuring before compilation. Even when you're completely in
control, the fact that you still can't get this working is troubling. But
maybe I'm just inflating extremely uncommon usecases.

One more thing - consider two Swift packages each needing a different
version of OpenSSL. Does your proposal handle that case?

Anyway, I'd love to read your proposal and then re-evaluate whether I
still feel like there's a large enough benefit in introducing such
customization to SwiftPM :slight_smile:

Honza

On Mon, Mar 21, 2016 at 6:19 PM Max Howell <max.howell@apple.com> wrote:

On Mar 19, 2016, at 10:55 AM, Honza Dvorsky via swift-build-dev < >> swift-build-dev@swift.org> wrote:

Hi All,

today I was trying to investigate how to build a proper cross-platform
server-side Swift project and only use SwiftPM to do it. In particular, I
was interested in getting OpenSSL/libcurl working as a dependency (built
from source). The main reason is that currently the big Swift server
frameworks all require manual steps during deployment (running apt-get/brew
to install libcurl/openssl/others) and I am trying to get rid of these
manual steps, eventually only requiring a `swift build` and run.

With the great initial work
<https://github.com/apple/swift-package-manager/pull/183> to support
building C-family packages in SwiftPM done by Ankit, I was able to have a
simple C package as a dependency, as advertised. However, the reality of
big cross-platform open source projects like OpenSSL and libcurl is that in
order to actually build it from source, a configure (or similar) script has
to be run to get the source code ready for compilation. And this is where I
got stuck.

Thus I wanted to kick off a discussion of what approach of solving this
problem with SwiftPM should be. The simple solution, which I'm tentatively
proposing, is to have an optional "post-clone" script in the package's
repository. Then, the Package.swift would optionally contain a field for
the path to this script - and if present, it'd get run after SwiftPM clones
this package. Note that this would be a "post-clone" script, not a
"prebuild" script - I imagine it'd only run once when cloning and then only
after each clean.

In order to get users to stop having to run manual script to install all
dependencies, I believe that we need to allow packages to declare what work
needs to be done on its source before compilation can begin.

I have a proposal I’d like to push today that adds knowledge to SwiftPM
about how to install system dependencies for System Module Packages.

We are not keen to add arbitrary script execution to SwiftPM, because: 1)
Arbitrary scripts cannot be controlled and this leads to dependency hell
and 2) arbitrary scripts mean your package graph may do anything, which
nobody wants.

Now I’m fine with the “root package” ie. the package the user has control
over doing more, but this would be a different discussion.

So the question becomes, is this sufficient? To have `brew install
openssl` run, or do you really need to install from source?

Max

_______________________________________________

swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev