Hard-Coded Paths in glibc.modulemap


(Shao Miller) #1

Good day, Swift package manager development folks.

(There are at least two separate issues being inquired about, but with the same introductory context.)

"Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

Thank you for your time and attention.

···

--

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com


(Daniel Dunbar) #2

Why do you want the headers inside the app sandbox? Usually they would remain outside.

Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

- Daniel

···

On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev <swift-build-dev@swift.org> wrote:

Good day, Swift package manager development folks.

(There are at least two separate issues being inquired about, but with the same introductory context.)

"Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

Thank you for your time and attention.
--

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Shao Miller) #3

Thank you for your kind response.

As mentioned, there is no choice: If the headers aren't present in the base image that a particular Cloud provider provides, they can only be present in the application sand-box by one's own hand.

All Swift build-packs to date and to my knowledge use Swift 2.2 and do not use the Swift 3 'swift build' process. I'm trying to develop the next generation.

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

···

On 6/8/2016 23:08, Daniel Dunbar wrote:

Why do you want the headers inside the app sandbox? Usually they would remain outside.

Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

  - Daniel

On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev <swift-build-dev@swift.org> wrote:

Good day, Swift package manager development folks.

(There are at least two separate issues being inquired about, but with the same introductory context.)

"Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

Thank you for your time and attention.
--

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Brian Croom) #4

IBM's buildpack as well as the ones from which it descends (
https://github.com/cloudfoundry-community/swift-buildpack and
https://github.com/kylef/heroku-buildpack-swift) are all based around
SwiftPM and the `swift build` command. I have not personally experienced
the problem you are describing either, although I have not tried pushing
any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have you
tried using any of these other buildpacks with your app code?

Brian

···

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev < swift-build-dev@swift.org>:

Thank you for your kind response.

As mentioned, there is no choice: If the headers aren't present in the
base image that a particular Cloud provider provides, they can only be
present in the application sand-box by one's own hand.

All Swift build-packs to date and to my knowledge use Swift 2.2 and do not
use the Swift 3 'swift build' process. I'm trying to develop the next
generation.

Shao Miller
Synthetel Corporation
T: +1.9053927729
E: swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
W: https://www.synthetel.com

On 6/8/2016 23:08, Daniel Dunbar wrote:

Why do you want the headers inside the app sandbox? Usually they would remain outside.

Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

- Daniel

On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev <swift-build-dev@swift.org> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> wrote:

Good day, Swift package manager development folks.

(There are at least two separate issues being inquired about, but with the same introductory context.)

"Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

Thank you for your time and attention.

--

Shao Miller

Synthetel Corporation

T: +1.9053927729 <tel:+1.9053927729>

E: swift-build-dev@synthetel.com <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

W: https://www.synthetel.com

_______________________________________________

swift-build-dev mailing list
swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Shao Miller) #5

Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc -I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have been dumped. The file /app/.delta/usr/include/complex.h exists. The error is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error: header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I /app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm -lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift build'. The two build-packs I'd looked at earlier today did not and I thought I'd recalled them having derived from the others you've mentioned. I was mistaken. My command somewhat resembles the IBM Bluemix build-pack command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc -I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its subprocesses during the compilation yields only these two instances of "complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT (No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43) = 43

Am I making an obvious mistake?

[1] https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2] https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

···

On 6/9/2016 01:15, Brian Croom wrote:

IBM's buildpack as well as the ones from which it descends (https://github.com/cloudfoundry-community/swift-buildpack and https://github.com/kylef/heroku-buildpack-swift) are all based around SwiftPM and the `swift build` command. I have not personally experienced the problem you are describing either, although I have not tried pushing any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
    W: https://www.synthetel.com

    On 6/8/2016 23:08, Daniel Dunbar wrote:

    Why do you want the headers inside the app sandbox? Usually they would remain outside.

    Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev<swift-build-dev@swift.org> >>> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:+1.9053927729>

    E:swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

    W:https://www.synthetel.com

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev


(Brian Croom) #6

What underlying OS is your build process running on? And where are you
getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded
absolute paths to certain system headers. The failure you are seeing is
presumably due to the fact that system headers are not present at those
paths. You may have to manually modify the modulemap to point to the actual
location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand what's
going on here: https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

Brian

···

torsdag 9 juni 2016 skrev Shao Miller <swift-build-dev@synthetel.com>:

Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc
-I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have been
dumped. The file /app/.delta/usr/include/complex.h exists. The error is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error:

header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I
/app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm
-lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift
build'. The two build-packs I'd looked at earlier today did not and I
thought I'd recalled them having derived from the others you've mentioned.
I was mistaken. My command somewhat resembles the IBM Bluemix build-pack
command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc
-I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its
subprocesses during the compilation yields only these two instances of
"complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT

(No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43) =
43

Am I making an obvious mistake?

[1]
https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2]
https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

On 6/9/2016 01:15, Brian Croom wrote:

IBM's buildpack as well as the ones from which it descends (
https://github.com/cloudfoundry-community/swift-buildpack and
https://github.com/kylef/heroku-buildpack-swift) are all based around
SwiftPM and the `swift build` command. I have not personally experienced
the problem you are describing either, although I have not tried pushing
any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have
you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
    W: https://www.synthetel.com

    On 6/8/2016 23:08, Daniel Dunbar wrote:

    Why do you want the headers inside the app sandbox? Usually they
would remain outside.

    Have you looked at IBM's CloudFoundry build pack (
https://github.com/IBM-Swift/swift-buildpack)? How does it handle the
problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev< >>>> swift-build-dev@swift.org> >>>> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but
with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry
are agonizingly locked-down environments. Essentially Swift and all of its
dependencies and one's project's dependencies must be stuffed into an
arbitrary directory (henceforth referred to as "the hole," but usually
/app/ ) and build processes performed without any root-user privileges.
One consequence is that one cannot use the OS' package-management system to
install dependencies, but one must obtain them and wrestle them into "the
hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to be
deployed via CloudFoundryish options and utilizing the 'swift build'
command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the
/usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a
hard-coded path to a ///usr/include/complex.h header-file. As is usually
the case, this hard-coded path will only work in a narrow set of
environments, which excludes "the hole" that CloudFoundry provides. I have
attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line
arguments, but I do not observe these paths (nor sub-paths) being tried for
the complex.h header-file during complication. I used 'strace' to trace
the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package
manager that its [unfortunately] hard-coded paths are relative to some
particular path? If not, would it be sensible to introduce some logic to
specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:+1.9053927729>

    E:swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

    W:https://www.synthetel.com

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #7

The IBM build pack installs a number of system dependencies which should
include those headers, though. You should have the headers rooted at /usr,
not try and have them in the app sandbox.

- Daniel

···

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev < swift-build-dev@swift.org> wrote:

What underlying OS is your build process running on? And where are you
getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded
absolute paths to certain system headers. The failure you are seeing is
presumably due to the fact that system headers are not present at those
paths. You may have to manually modify the modulemap to point to the actual
location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand
what's going on here:
https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

Brian

torsdag 9 juni 2016 skrev Shao Miller <swift-build-dev@synthetel.com>:

Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc
-I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have been
dumped. The file /app/.delta/usr/include/complex.h exists. The error is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error:

header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I
/app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm
-lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift
build'. The two build-packs I'd looked at earlier today did not and I
thought I'd recalled them having derived from the others you've mentioned.
I was mistaken. My command somewhat resembles the IBM Bluemix build-pack
command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc
-I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its
subprocesses during the compilation yields only these two instances of
"complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT

(No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43)
= 43

Am I making an obvious mistake?

[1]
https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2]
https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

On 6/9/2016 01:15, Brian Croom wrote:

IBM's buildpack as well as the ones from which it descends (
https://github.com/cloudfoundry-community/swift-buildpack and
https://github.com/kylef/heroku-buildpack-swift) are all based around
SwiftPM and the `swift build` command. I have not personally experienced
the problem you are describing either, although I have not tried pushing
any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have
you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
    W: https://www.synthetel.com

    On 6/8/2016 23:08, Daniel Dunbar wrote:

    Why do you want the headers inside the app sandbox? Usually they
would remain outside.

    Have you looked at IBM's CloudFoundry build pack (
https://github.com/IBM-Swift/swift-buildpack)? How does it handle the
problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev< >>>>> swift-build-dev@swift.org> >>>>> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> >>>>> wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but
with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry
are agonizingly locked-down environments. Essentially Swift and all of its
dependencies and one's project's dependencies must be stuffed into an
arbitrary directory (henceforth referred to as "the hole," but usually
/app/ ) and build processes performed without any root-user privileges.
One consequence is that one cannot use the OS' package-management system to
install dependencies, but one must obtain them and wrestle them into "the
hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to
be deployed via CloudFoundryish options and utilizing the 'swift build'
command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the
/usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a
hard-coded path to a ///usr/include/complex.h header-file. As is usually
the case, this hard-coded path will only work in a narrow set of
environments, which excludes "the hole" that CloudFoundry provides. I have
attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line
arguments, but I do not observe these paths (nor sub-paths) being tried for
the complex.h header-file during complication. I used 'strace' to trace
the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package
manager that its [unfortunately] hard-coded paths are relative to some
particular path? If not, would it be sensible to introduce some logic to
specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:+1.9053927729>

    E:swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

    W:https://www.synthetel.com

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Shao Miller) #8

Thanks, but let me try to explain for a third time: The build environment provided by CloudFoundry-style providers is locked-down. You have no choice where things go. They must all go inside a sandbox. That means you cannot put headers underneath the /usr/ directory. There is no way out of the sandbox, as there are no root-user privileges.

This is why I'm asking about whether or not it's reasonable for Swift to avoid hard-coded paths, or to at least allow a given prefix to be specified.

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

···

On 6/9/2016 11:31, Daniel Dunbar wrote:

The IBM build pack installs a number of system dependencies which should include those headers, though. You should have the headers rooted at /usr, not try and have them in the app sandbox.

- Daniel

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev > <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:

    What underlying OS is your build process running on? And where are
    you getting your Swift toolchain?

    Within your toolchain is a `glibc.modulemap` which contains
    hard-coded absolute paths to certain system headers. The failure
    you are seeing is presumably due to the fact that system headers
    are not present at those paths. You may have to manually modify
    the modulemap to point to the actual location of the headers on
    the system that is performing the build.

    This old bug contains some tidbits that may also help you
    understand what's going on here:
    https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

    Brian

    torsdag 9 juni 2016 skrev Shao Miller
    <swift-build-dev@synthetel.com
    <mailto:swift-build-dev@synthetel.com>>:

        Thank you for your kind response.

        The command I execute is: swift build -Xcc -I/app/.delta/
        -Xswiftc -I/app/.delta/ -v

        The /app/.delta/ directory is where Swift and most
        dependencies have been dumped. The file
        /app/.delta/usr/include/complex.h exists. The error is:

            /app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14:
            error: header '///usr/include/complex.h' not found
                  header "///usr/include/complex.h"
                         ^
            <unknown>:0: error: could not build Objective-C module
            'SwiftGlibc'
            error: exit(1): /app/.delta/usr/bin/swiftc
            --driver-mode=swift -I /app/.delta/usr/lib/swift/pm -L
            /app/.delta/usr/lib/swift/pm -lPackageDescription
            /app/PerfectTemplate/Package.swift -fileno 3

        Thank you for pointing out that these build-packs are using
        'swift build'. The two build-packs I'd looked at earlier
        today did not and I thought I'd recalled them having derived
        from the others you've mentioned. I was mistaken. My command
        somewhat resembles the IBM Bluemix build-pack command[1],
        which is:
        swift build --configuration release -Xcc -fblocks -Xcc
        -I$BUILD_DIR/.apt/usr/include ...and so on...

        I am using this[2] Swift. Running 'strace' on the BASh and
        all of its subprocesses during the compilation yields only
        these two instances of "complex":

            [pid 28678] stat("///usr/include/complex.h",
            0x7ffe84061ef0) = -1 ENOENT (No such file or directory)
            [pid 28679] write(2, "header '///usr/include/complex.h'
            not found", 43) = 43

        Am I making an obvious mistake?

        [1]
        https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
        [2]
        https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

        Shao Miller
        Synthetel Corporation
        T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729
        <tel:%2B1.9053927729>>
        E: swift-build-dev@synthetel.com
        W: https://www.synthetel.com

        On 6/9/2016 01:15, Brian Croom wrote:

            IBM's buildpack as well as the ones from which it descends
            (https://github.com/cloudfoundry-community/swift-buildpack
            and https://github.com/kylef/heroku-buildpack-swift) are
            all based around SwiftPM and the `swift build` command. I
            have not personally experienced the problem you are
            describing either, although I have not tried pushing any
            apps using more recent Swift toolchain snapshots.

            Can you provide more details about how the error presents
            itself? Have you tried using any of these other buildpacks
            with your app code?

            Brian

            onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev
            <swift-build-dev@swift.org
            <mailto:swift-build-dev@swift.org>>:

                Thank you for your kind response.

                As mentioned, there is no choice: If the headers
            aren't present in
                the base image that a particular Cloud provider
            provides, they can
                only be present in the application sand-box by one's
            own hand.

                All Swift build-packs to date and to my knowledge use
            Swift 2.2
                and do not use the Swift 3 'swift build' process. I'm
            trying to
                develop the next generation.

                Shao Miller
                Synthetel Corporation
                T: +1.9053927729 <tel:%2B1.9053927729>
            <tel:+1.9053927729 <tel:%2B1.9053927729>>
                E: swift-build-dev@synthetel.com
                           <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
                W: https://www.synthetel.com

                On 6/8/2016 23:08, Daniel Dunbar wrote:

                    Why do you want the headers inside the app
                sandbox? Usually they would remain outside.

                    Have you looked at IBM's CloudFoundry build pack
                (https://github.com/IBM-Swift/swift-buildpack)? How
                does it handle the problem you are trying to solve?

                      - Daniel

                        On Jun 8, 2016, at 8:03 PM, Shao Miller via > swift-build-dev<swift-build-dev@swift.org> > > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> > wrote:

                        Good day, Swift package manager development folks.

                        (There are at least two separate issues being
                    inquired about, but with the same introductory
                    context.)

                        "Cloudy" deployment options derived from or
                    akin to CloudFoundry are agonizingly locked-down
                    environments. Essentially Swift and all of its
                    dependencies and one's project's dependencies must
                    be stuffed into an arbitrary directory (henceforth
                    referred to as "the hole," but usually /app/ ) and
                    build processes performed without any root-user
                    privileges. One consequence is that one cannot
                    use the OS' package-management system to install
                    dependencies, but one must obtain them and wrestle
                    them into "the hole," instead. The strategy seems
                    rather silly.

                        While developing a so-called "buildpack" for
                    Swift 3 projects to be deployed via
                    CloudFoundryish options and utilizing the 'swift
                    build' command, I have come across a few issues.

                        One issue is that 'swift build' wants to do
                    something with the
                    /usr/lib/swift/linux/x86_64/glibc.modulemap file,
                    but that file contains a hard-coded path to a
                    ///usr/include/complex.h header-file. As is
                    usually the case, this hard-coded path will only
                    work in a narrow set of environments, which
                    excludes "the hole" that CloudFoundry provides. I
                    have attempted to use '-Xcc -I/the/hole' and
                    '-Xswiftc -I/the/hole' command-line arguments, but
                    I do not observe these paths (nor sub-paths) being
                    tried for the complex.h header-file during
                    complication. I used 'strace' to trace the
                    compilation process, including all subprocesses.

                        Is there some other mechanism to instruct the
                    Swift 3 package manager that its [unfortunately]
                    hard-coded paths are relative to some particular
                    path? If not, would it be sensible to introduce
                    some logic to specify such a prefix path?

                        Thank you for your time and attention.

                        --

                        Shao Miller

                        Synthetel Corporation

                        T: +1.9053927729 <tel:%2B1.9053927729>
                    <tel:+1.9053927729 <tel:%2B1.9053927729>>

                    E:swift-build-dev@synthetel.com
                                           <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

                        W:https://www.synthetel.com

                    _______________________________________________

                        swift-build-dev mailing list

                    swift-build-dev@swift.org
                                           <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>

                    https://lists.swift.org/mailman/listinfo/swift-build-dev

    _______________________________________________
    swift-build-dev mailing list
    swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
    https://lists.swift.org/mailman/listinfo/swift-build-dev


(Shao Miller) #9

Thank you again.

My answers two your two questions are: Ubuntu 14.04. From here[3]. (Despite having shared this[2] link below, by mistake.)

The CloudFoundry build process is an automated process which involves downloading Swift. If that automated process then further has to modify portions of Swift source-code to "play nice" with the Cloud Foundry build environment's [silly] limitations, that is very tedious and means that a "delta" for a specific version of Swift has to be integrated into the automated build process. I am suggesting that hard-coded paths in Swift source-code is not a good idea and I am hopeful to meet with agreement or with a discovery that I'm missing a key idea for why these paths ought to remain hard-coded.

My two questions are:

Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

[3] https://swift.org/builds/development/ubuntu1404/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu14.04.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

···

On 6/8/2016 23:03, Shao Miller via swift-build-dev wrote:

On 6/9/2016 08:32, Brian Croom wrote:

What underlying OS is your build process running on? And where are you getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded absolute paths to certain system headers. The failure you are seeing is presumably due to the fact that system headers are not present at those paths. You may have to manually modify the modulemap to point to the actual location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand what's going on here: https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

Brian

torsdag 9 juni 2016 skrev Shao Miller <swift-build-dev@synthetel.com <mailto:swift-build-dev@synthetel.com>>:

    Thank you for your kind response.

    The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc
    -I/app/.delta/ -v

    The /app/.delta/ directory is where Swift and most dependencies
    have been dumped. The file /app/.delta/usr/include/complex.h
    exists. The error is:

        /app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14:
        error: header '///usr/include/complex.h' not found
              header "///usr/include/complex.h"
                     ^
        <unknown>:0: error: could not build Objective-C module
        'SwiftGlibc'
        error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift
        -I /app/.delta/usr/lib/swift/pm -L
        /app/.delta/usr/lib/swift/pm -lPackageDescription
        /app/PerfectTemplate/Package.swift -fileno 3

    Thank you for pointing out that these build-packs are using 'swift
    build'. The two build-packs I'd looked at earlier today did not
    and I thought I'd recalled them having derived from the others
    you've mentioned. I was mistaken. My command somewhat resembles
    the IBM Bluemix build-pack command[1], which is:
    swift build --configuration release -Xcc -fblocks -Xcc
    -I$BUILD_DIR/.apt/usr/include ...and so on...

    I am using this[2] Swift. Running 'strace' on the BASh and all of
    its subprocesses during the compilation yields only these two
    instances of "complex":

        [pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) =
        -1 ENOENT (No such file or directory)
        [pid 28679] write(2, "header '///usr/include/complex.h' not
        found", 43) = 43

    Am I making an obvious mistake?

    [1]
    https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
    [2]
    https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    W: https://www.synthetel.com

    On 6/9/2016 01:15, Brian Croom wrote:

        IBM's buildpack as well as the ones from which it descends
        (https://github.com/cloudfoundry-community/swift-buildpack and
        https://github.com/kylef/heroku-buildpack-swift) are all based
        around SwiftPM and the `swift build` command. I have not
        personally experienced the problem you are describing either,
        although I have not tried pushing any apps using more recent
        Swift toolchain snapshots.

        Can you provide more details about how the error presents
        itself? Have you tried using any of these other buildpacks
        with your app code?

        Brian

        onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev
        <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

            Thank you for your kind response.

            As mentioned, there is no choice: If the headers aren't
        present in
            the base image that a particular Cloud provider provides,
        they can
            only be present in the application sand-box by one's own hand.

            All Swift build-packs to date and to my knowledge use
        Swift 2.2
            and do not use the Swift 3 'swift build' process. I'm
        trying to
            develop the next generation.

            Shao Miller
            Synthetel Corporation
            T: +1.9053927729 <tel:+1.9053927729>
            E: swift-build-dev@synthetel.com
                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
            W: https://www.synthetel.com

            On 6/8/2016 23:08, Daniel Dunbar wrote:

                Why do you want the headers inside the app sandbox?
            Usually they would remain outside.

                Have you looked at IBM's CloudFoundry build pack
            (https://github.com/IBM-Swift/swift-buildpack)? How does
            it handle the problem you are trying to solve?

                  - Daniel

                    On Jun 8, 2016, at 8:03 PM, Shao Miller via > swift-build-dev<swift-build-dev@swift.org> > > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> > wrote:

                    Good day, Swift package manager development folks.

                    (There are at least two separate issues being
                inquired about, but with the same introductory context.)

                    "Cloudy" deployment options derived from or akin
                to CloudFoundry are agonizingly locked-down
                environments. Essentially Swift and all of its
                dependencies and one's project's dependencies must be
                stuffed into an arbitrary directory (henceforth
                referred to as "the hole," but usually /app/ ) and
                build processes performed without any root-user
                privileges. One consequence is that one cannot use
                the OS' package-management system to install
                dependencies, but one must obtain them and wrestle
                them into "the hole," instead. The strategy seems
                rather silly.

                    While developing a so-called "buildpack" for Swift
                3 projects to be deployed via CloudFoundryish options
                and utilizing the 'swift build' command, I have come
                across a few issues.

                    One issue is that 'swift build' wants to do
                something with the
                /usr/lib/swift/linux/x86_64/glibc.modulemap file, but
                that file contains a hard-coded path to a
                ///usr/include/complex.h header-file. As is usually
                the case, this hard-coded path will only work in a
                narrow set of environments, which excludes "the hole"
                that CloudFoundry provides. I have attempted to use
                '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole'
                command-line arguments, but I do not observe these
                paths (nor sub-paths) being tried for the complex.h
                header-file during complication. I used 'strace' to
                trace the compilation process, including all subprocesses.

                    Is there some other mechanism to instruct the
                Swift 3 package manager that its [unfortunately]
                hard-coded paths are relative to some particular
                path? If not, would it be sensible to introduce some
                logic to specify such a prefix path?

                    Thank you for your time and attention.

                    --

                    Shao Miller

                    Synthetel Corporation

                    T: +1.9053927729 <tel:+1.9053927729>

                E:swift-build-dev@synthetel.com
                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>

                    W:https://www.synthetel.com

                    _______________________________________________

                    swift-build-dev mailing list

                swift-build-dev@swift.org
                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>

                https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #10

Thanks, but let me try to explain for a third time: The build environment provided by CloudFoundry-style providers is locked-down. You have no choice where things go. They must all go inside a sandbox. That means you cannot put headers underneath the /usr/ directory. There is no way out of the sandbox, as there are no root-user privileges.

You have stated that, but the IBM build pack works, uses CloudFoundry, and it clearly uses apt-get to install things in the system. You haven't explained how your environment is different yet.

- Daniel

···

On Jun 9, 2016, at 8:35 AM, Shao Miller via swift-build-dev <swift-build-dev@swift.org> wrote:

This is why I'm asking about whether or not it's reasonable for Swift to avoid hard-coded paths, or to at least allow a given prefix to be specified.

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com <mailto:swift-build-dev@synthetel.com>
W: https://www.synthetel.com <https://www.synthetel.com/>

On 6/9/2016 11:31, Daniel Dunbar wrote:

The IBM build pack installs a number of system dependencies which should include those headers, though. You should have the headers rooted at /usr, not try and have them in the app sandbox.

- Daniel

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev < <mailto:swift-build-dev@swift.org>swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> wrote:
What underlying OS is your build process running on? And where are you getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded absolute paths to certain system headers. The failure you are seeing is presumably due to the fact that system headers are not present at those paths. You may have to manually modify the modulemap to point to the actual location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand what's going on here: <https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15 <https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15>

Brian

torsdag 9 juni 2016 skrev Shao Miller < <mailto:swift-build-dev@synthetel.com>swift-build-dev@synthetel.com <mailto:swift-build-dev@synthetel.com>>:
Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc -I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have been dumped. The file /app/.delta/usr/include/complex.h exists. The error is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error: header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I /app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm -lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift build'. The two build-packs I'd looked at earlier today did not and I thought I'd recalled them having derived from the others you've mentioned. I was mistaken. My command somewhat resembles the IBM Bluemix build-pack command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc -I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its subprocesses during the compilation yields only these two instances of "complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT (No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43) = 43

Am I making an obvious mistake?

[1] https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2] https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>
E: swift-build-dev@synthetel.com <>
W: https://www.synthetel.com <https://www.synthetel.com/>

On 6/9/2016 01:15, Brian Croom wrote:
IBM's buildpack as well as the ones from which it descends (https://github.com/cloudfoundry-community/swift-buildpack and https://github.com/kylef/heroku-buildpack-swift) are all based around SwiftPM and the `swift build` command. I have not personally experienced the problem you are describing either, although I have not tried pushing any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <swift-build-dev@swift.org <> <mailto:swift-build-dev@swift.org <>>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>
    E: swift-build-dev@synthetel.com <>
    <javascript:_e(%7B%7D,'cvml <javascript:_e(%7B%7D,'cvml>',' <>swift-build-dev@synthetel.com <mailto:swift-build-dev@synthetel.com>');>
    W: https://www.synthetel.com <https://www.synthetel.com/>

    On 6/8/2016 23:08, Daniel Dunbar wrote:
    Why do you want the headers inside the app sandbox? Usually they would remain outside.

    Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev< <>swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>> >> <javascript:_e(%7B%7D,'cvml <javascript:_e(%7B%7D,'cvml>',' <>swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>');> wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>

    E:swift-build-dev@synthetel.com <>
    <javascript:_e(%7B%7D,'cvml <javascript:_e(%7B%7D,'cvml>',' <>swift-build-dev@synthetel.com <mailto:swift-build-dev@synthetel.com>');>

    W:https://www.synthetel.com <https://www.synthetel.com/>

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org <>
    <javascript:_e(%7B%7D,'cvml <javascript:_e(%7B%7D,'cvml>',' <>swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Brian Croom) #11

Clang modulemap files currently have a limitation in that they can only use
absolute paths for headers. Because the buildpack environment does not
allow installing files under /usr, you should look into editing the
modulemap to point to the correct location.

I am surprised, though, that a standard libc header like complex.h wouldn't
be available under /usr in your environment already. For example the Cloud
Foundry cflinuxfs2 environment provides an Ubuntu image with libc
development files already installed, which I would expect to work.

···

torsdag 9 juni 2016 skrev Shao Miller via swift-build-dev < swift-build-dev@swift.org>:

Thanks, but let me try to explain for a third time: The build environment
provided by CloudFoundry-style providers is locked-down. You have no
choice where things go. They must all go inside a sandbox. That means you
cannot put headers underneath the /usr/ directory. There is no way out of
the sandbox, as there are no root-user privileges.

This is why I'm asking about whether or not it's reasonable for Swift to
avoid hard-coded paths, or to at least allow a given prefix to be specified.

Shao Miller
Synthetel Corporation
T: +1.9053927729
E: swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
W: https://www.synthetel.com

On 6/9/2016 11:31, Daniel Dunbar wrote:

The IBM build pack installs a number of system dependencies which should
include those headers, though. You should have the headers rooted at /usr,
not try and have them in the app sandbox.

- Daniel

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev < > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> > swift-build-dev@swift.org > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

What underlying OS is your build process running on? And where are you
getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded
absolute paths to certain system headers. The failure you are seeing is
presumably due to the fact that system headers are not present at those
paths. You may have to manually modify the modulemap to point to the actual
location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand
what's going on here:
<https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15>
https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

Brian

torsdag 9 juni 2016 skrev Shao Miller <
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>>:

Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc
-I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have
been dumped. The file /app/.delta/usr/include/complex.h exists. The error
is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error:

header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I
/app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm
-lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift
build'. The two build-packs I'd looked at earlier today did not and I
thought I'd recalled them having derived from the others you've mentioned.
I was mistaken. My command somewhat resembles the IBM Bluemix build-pack
command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc
-I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its
subprocesses during the compilation yields only these two instances of
"complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT

(No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43)
= 43

Am I making an obvious mistake?

[1]
https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2]
https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

On 6/9/2016 01:15, Brian Croom wrote:

IBM's buildpack as well as the ones from which it descends (
https://github.com/cloudfoundry-community/swift-buildpack and
https://github.com/kylef/heroku-buildpack-swift) are all based around
SwiftPM and the `swift build` command. I have not personally experienced
the problem you are describing either, although I have not tried pushing
any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have
you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>
    W: https://www.synthetel.com

    On 6/8/2016 23:08, Daniel Dunbar wrote:

    Why do you want the headers inside the app sandbox? Usually they
would remain outside.

    Have you looked at IBM's CloudFoundry build pack (
https://github.com/IBM-Swift/swift-buildpack)? How does it handle the
problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev<

swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>>
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>
wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but
with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry
are agonizingly locked-down environments. Essentially Swift and all of its
dependencies and one's project's dependencies must be stuffed into an
arbitrary directory (henceforth referred to as "the hole," but usually
/app/ ) and build processes performed without any root-user privileges.
One consequence is that one cannot use the OS' package-management system to
install dependencies, but one must obtain them and wrestle them into "the
hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to
be deployed via CloudFoundryish options and utilizing the 'swift build'
command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the
/usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a
hard-coded path to a ///usr/include/complex.h header-file. As is usually
the case, this hard-coded path will only work in a narrow set of
environments, which excludes "the hole" that CloudFoundry provides. I have
attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line
arguments, but I do not observe these paths (nor sub-paths) being tried for
the complex.h header-file during complication. I used 'strace' to trace
the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package
manager that its [unfortunately] hard-coded paths are relative to some
particular path? If not, would it be sensible to introduce some logic to
specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:+1.9053927729>

    E:swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>

    W:https://www.synthetel.com

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Brian Croom) #12

Note that the IBM buildpack works around the environmental restrictions by
configuring apt-get to install packages in subdirectories of the buildpack
sandbox, rather than in the standard system directories. This works great,
but can still cause issues with components that search for things with
absolute paths to the standard directories.

···

torsdag 9 juni 2016 skrev Daniel Dunbar via swift-build-dev < swift-build-dev@swift.org>:

On Jun 9, 2016, at 8:35 AM, Shao Miller via swift-build-dev < > swift-build-dev@swift.org > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

Thanks, but let me try to explain for a third time: The build environment
provided by CloudFoundry-style providers is locked-down. You have no
choice where things go. They must all go inside a sandbox. That means you
cannot put headers underneath the /usr/ directory. There is no way out of
the sandbox, as there are no root-user privileges.

You have stated that, but the IBM build pack works, uses CloudFoundry, and
it clearly uses apt-get to install things in the system. You haven't
explained how your environment is different yet.

- Daniel

This is why I'm asking about whether or not it's reasonable for Swift to
avoid hard-coded paths, or to at least allow a given prefix to be specified.

Shao Miller
Synthetel Corporation
T: +1.9053927729
E: swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
W: https://www.synthetel.com

On 6/9/2016 11:31, Daniel Dunbar wrote:

The IBM build pack installs a number of system dependencies which should
include those headers, though. You should have the headers rooted at /usr,
not try and have them in the app sandbox.

- Daniel

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev < > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');> > swift-build-dev@swift.org > <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

What underlying OS is your build process running on? And where are you
getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded
absolute paths to certain system headers. The failure you are seeing is
presumably due to the fact that system headers are not present at those
paths. You may have to manually modify the modulemap to point to the actual
location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand
what's going on here:
<https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15>
https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

Brian

torsdag 9 juni 2016 skrev Shao Miller <
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>>:

Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc
-I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have
been dumped. The file /app/.delta/usr/include/complex.h exists. The error
is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error:

header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I
/app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm
-lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift
build'. The two build-packs I'd looked at earlier today did not and I
thought I'd recalled them having derived from the others you've mentioned.
I was mistaken. My command somewhat resembles the IBM Bluemix build-pack
command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc
-I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its
subprocesses during the compilation yields only these two instances of
"complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT

(No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43)
= 43

Am I making an obvious mistake?

[1]
https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2]
https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

On 6/9/2016 01:15, Brian Croom wrote:

IBM's buildpack as well as the ones from which it descends (
https://github.com/cloudfoundry-community/swift-buildpack and
https://github.com/kylef/heroku-buildpack-swift) are all based around
SwiftPM and the `swift build` command. I have not personally experienced
the problem you are describing either, although I have not tried pushing
any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have
you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <
swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>
    W: https://www.synthetel.com

    On 6/8/2016 23:08, Daniel Dunbar wrote:

    Why do you want the headers inside the app sandbox? Usually they
would remain outside.

    Have you looked at IBM's CloudFoundry build pack (
https://github.com/IBM-Swift/swift-buildpack)? How does it handle the
problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev<

swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>>
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>
wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but
with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry
are agonizingly locked-down environments. Essentially Swift and all of its
dependencies and one's project's dependencies must be stuffed into an
arbitrary directory (henceforth referred to as "the hole," but usually
/app/ ) and build processes performed without any root-user privileges.
One consequence is that one cannot use the OS' package-management system to
install dependencies, but one must obtain them and wrestle them into "the
hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to
be deployed via CloudFoundryish options and utilizing the 'swift build'
command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the
/usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a
hard-coded path to a ///usr/include/complex.h header-file. As is usually
the case, this hard-coded path will only work in a narrow set of
environments, which excludes "the hole" that CloudFoundry provides. I have
attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line
arguments, but I do not observe these paths (nor sub-paths) being tried for
the complex.h header-file during complication. I used 'strace' to trace
the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package
manager that its [unfortunately] hard-coded paths are relative to some
particular path? If not, would it be sensible to introduce some logic to
specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:+1.9053927729>

    E:swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
<javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>

    W:https://www.synthetel.com

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
<javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Daniel Dunbar) #13

Ah, thanks! That was the missing piece of info and I missed that in the script.

If you truly need to install a full set of headers in a sandboxed location, then the compiler arguments to use to point at that location are the `--sysroot` ones for Clang. I haven't tried this myself, so YMMV, but something like `-Xcc --sysroot -Xcc /app/.delta` *might* work.

- Daniel

···

On Jun 9, 2016, at 9:09 AM, Brian Croom <brian.s.croom@gmail.com> wrote:

Note that the IBM buildpack works around the environmental restrictions by configuring apt-get to install packages in subdirectories of the buildpack sandbox, rather than in the standard system directories. This works great, but can still cause issues with components that search for things with absolute paths to the standard directories.

torsdag 9 juni 2016 skrev Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

On Jun 9, 2016, at 8:35 AM, Shao Miller via swift-build-dev <swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

Thanks, but let me try to explain for a third time: The build environment provided by CloudFoundry-style providers is locked-down. You have no choice where things go. They must all go inside a sandbox. That means you cannot put headers underneath the /usr/ directory. There is no way out of the sandbox, as there are no root-user privileges.

You have stated that, but the IBM build pack works, uses CloudFoundry, and it clearly uses apt-get to install things in the system. You haven't explained how your environment is different yet.

- Daniel

This is why I'm asking about whether or not it's reasonable for Swift to avoid hard-coded paths, or to at least allow a given prefix to be specified.

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
W: https://www.synthetel.com <https://www.synthetel.com/>

On 6/9/2016 11:31, Daniel Dunbar wrote:

The IBM build pack installs a number of system dependencies which should include those headers, though. You should have the headers rooted at /usr, not try and have them in the app sandbox.

- Daniel

On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev < <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:
What underlying OS is your build process running on? And where are you getting your Swift toolchain?

Within your toolchain is a `glibc.modulemap` which contains hard-coded absolute paths to certain system headers. The failure you are seeing is presumably due to the fact that system headers are not present at those paths. You may have to manually modify the modulemap to point to the actual location of the headers on the system that is performing the build.

This old bug contains some tidbits that may also help you understand what's going on here: <https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15 <https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15>

Brian

torsdag 9 juni 2016 skrev Shao Miller < <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>swift-build-dev@synthetel.com <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>>:
Thank you for your kind response.

The command I execute is: swift build -Xcc -I/app/.delta/ -Xswiftc -I/app/.delta/ -v

The /app/.delta/ directory is where Swift and most dependencies have been dumped. The file /app/.delta/usr/include/complex.h exists. The error is:

/app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14: error: header '///usr/include/complex.h' not found
      header "///usr/include/complex.h"
             ^
<unknown>:0: error: could not build Objective-C module 'SwiftGlibc'
error: exit(1): /app/.delta/usr/bin/swiftc --driver-mode=swift -I /app/.delta/usr/lib/swift/pm -L /app/.delta/usr/lib/swift/pm -lPackageDescription /app/PerfectTemplate/Package.swift -fileno 3

Thank you for pointing out that these build-packs are using 'swift build'. The two build-packs I'd looked at earlier today did not and I thought I'd recalled them having derived from the others you've mentioned. I was mistaken. My command somewhat resembles the IBM Bluemix build-pack command[1], which is:
swift build --configuration release -Xcc -fblocks -Xcc -I$BUILD_DIR/.apt/usr/include ...and so on...

I am using this[2] Swift. Running 'strace' on the BASh and all of its subprocesses during the compilation yields only these two instances of "complex":

[pid 28678] stat("///usr/include/complex.h", 0x7ffe84061ef0) = -1 ENOENT (No such file or directory)
[pid 28679] write(2, "header '///usr/include/complex.h' not found", 43) = 43

Am I making an obvious mistake?

[1] https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
[2] https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>
E: swift-build-dev@synthetel.com <>
W: https://www.synthetel.com <https://www.synthetel.com/>

On 6/9/2016 01:15, Brian Croom wrote:
IBM's buildpack as well as the ones from which it descends (https://github.com/cloudfoundry-community/swift-buildpack and https://github.com/kylef/heroku-buildpack-swift) are all based around SwiftPM and the `swift build` command. I have not personally experienced the problem you are describing either, although I have not tried pushing any apps using more recent Swift toolchain snapshots.

Can you provide more details about how the error presents itself? Have you tried using any of these other buildpacks with your app code?

Brian

onsdag 8 juni 2016 skrev Shao Miller via swift-build-dev <swift-build-dev@swift.org <> <mailto:swift-build-dev@swift.org <>>>:

    Thank you for your kind response.

    As mentioned, there is no choice: If the headers aren't present in
    the base image that a particular Cloud provider provides, they can
    only be present in the application sand-box by one's own hand.

    All Swift build-packs to date and to my knowledge use Swift 2.2
    and do not use the Swift 3 'swift build' process. I'm trying to
    develop the next generation.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>
    E: swift-build-dev@synthetel.com <>
    <javascript:_e(%7B%7D,'cvml <>',' <>swift-build-dev@synthetel.com <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>
    W: https://www.synthetel.com <https://www.synthetel.com/>

    On 6/8/2016 23:08, Daniel Dunbar wrote:
    Why do you want the headers inside the app sandbox? Usually they would remain outside.

    Have you looked at IBM's CloudFoundry build pack (https://github.com/IBM-Swift/swift-buildpack)? How does it handle the problem you are trying to solve?

      - Daniel

    On Jun 8, 2016, at 8:03 PM, Shao Miller via swift-build-dev< <>swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> >>> <javascript:_e(%7B%7D,'cvml <>',' <>swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');> wrote:

    Good day, Swift package manager development folks.

    (There are at least two separate issues being inquired about, but with the same introductory context.)

    "Cloudy" deployment options derived from or akin to CloudFoundry are agonizingly locked-down environments. Essentially Swift and all of its dependencies and one's project's dependencies must be stuffed into an arbitrary directory (henceforth referred to as "the hole," but usually /app/ ) and build processes performed without any root-user privileges. One consequence is that one cannot use the OS' package-management system to install dependencies, but one must obtain them and wrestle them into "the hole," instead. The strategy seems rather silly.

    While developing a so-called "buildpack" for Swift 3 projects to be deployed via CloudFoundryish options and utilizing the 'swift build' command, I have come across a few issues.

    One issue is that 'swift build' wants to do something with the /usr/lib/swift/linux/x86_64/glibc.modulemap file, but that file contains a hard-coded path to a ///usr/include/complex.h header-file. As is usually the case, this hard-coded path will only work in a narrow set of environments, which excludes "the hole" that CloudFoundry provides. I have attempted to use '-Xcc -I/the/hole' and '-Xswiftc -I/the/hole' command-line arguments, but I do not observe these paths (nor sub-paths) being tried for the complex.h header-file during complication. I used 'strace' to trace the compilation process, including all subprocesses.

    Is there some other mechanism to instruct the Swift 3 package manager that its [unfortunately] hard-coded paths are relative to some particular path? If not, would it be sensible to introduce some logic to specify such a prefix path?

    Thank you for your time and attention.

    --

    Shao Miller

    Synthetel Corporation

    T: +1.9053927729 <tel:%2B1.9053927729> <tel:+1.9053927729 <tel:%2B1.9053927729>>

    E:swift-build-dev@synthetel.com <>
    <javascript:_e(%7B%7D,'cvml <>',' <>swift-build-dev@synthetel.com <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>

    W:https://www.synthetel.com <https://www.synthetel.com/>

    _______________________________________________

    swift-build-dev mailing list

    swift-build-dev@swift.org <>
    <javascript:_e(%7B%7D,'cvml <>',' <>swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>

    https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
https://lists.swift.org/mailman/listinfo/swift-build-dev


(Shao Miller) #14

Thank you. The 'swiftc' command doesn't appear to care about the '--sysroot' parameter with respect to header-paths. 'strace' yields only examination of the absolute paths for the headers.

Shao Miller
Synthetel Corporation
T: +1.9053927729 <tel:+1.9053927729>
E: swift-build-dev@synthetel.com
W: https://www.synthetel.com

···

On 6/9/2016 12:13, Daniel Dunbar wrote:

Ah, thanks! That was the missing piece of info and I missed that in the script.

If you truly need to install a full set of headers in a sandboxed location, then the compiler arguments to use to point at that location are the `--sysroot` ones for Clang. I haven't tried this myself, so YMMV, but something like `-Xcc --sysroot -Xcc /app/.delta` *might* work.

- Daniel

On Jun 9, 2016, at 9:09 AM, Brian Croom <brian.s.croom@gmail.com >> <mailto:brian.s.croom@gmail.com>> wrote:

Note that the IBM buildpack works around the environmental restrictions by configuring apt-get to install packages in subdirectories of the buildpack sandbox, rather than in the standard system directories. This works great, but can still cause issues with components that search for things with absolute paths to the standard directories.

torsdag 9 juni 2016 skrev Daniel Dunbar via swift-build-dev <swift-build-dev@swift.org <mailto:swift-build-dev@swift.org>>:

    On Jun 9, 2016, at 8:35 AM, Shao Miller via swift-build-dev >>> <swift-build-dev@swift.org >>> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

    Thanks, but let me try to explain for a third time: The build
    environment provided by CloudFoundry-style providers is
    locked-down. You have no choice where things go. They must all
    go inside a sandbox. That means you cannot put headers
    underneath the /usr/ directory. There is no way out of the
    sandbox, as there are no root-user privileges.

    You have stated that, but the IBM build pack works, uses
    CloudFoundry, and it clearly uses apt-get to install things in
    the system. You haven't explained how your environment is
    different yet.

     - Daniel

    This is why I'm asking about whether or not it's reasonable for
    Swift to avoid hard-coded paths, or to at least allow a given
    prefix to be specified.

    Shao Miller
    Synthetel Corporation
    T: +1.9053927729 <tel:+1.9053927729>
    E: swift-build-dev@synthetel.com
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>
    W: https://www.synthetel.com <https://www.synthetel.com/>

    On 6/9/2016 11:31, Daniel Dunbar wrote:

    The IBM build pack installs a number of system dependencies
    which should include those headers, though. You should have the
    headers rooted at /usr, not try and have them in the app sandbox.

     - Daniel

    On Thu, Jun 9, 2016 at 5:32 AM, Brian Croom via swift-build-dev >>>> <swift-build-dev@swift.org >>>> <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>> wrote:

        What underlying OS is your build process running on? And
        where are you getting your Swift toolchain?

        Within your toolchain is a `glibc.modulemap` which contains
        hard-coded absolute paths to certain system headers. The
        failure you are seeing is presumably due to the fact that
        system headers are not present at those paths. You may have
        to manually modify the modulemap to point to the actual
        location of the headers on the system that is performing
        the build.

        This old bug contains some tidbits that may also help you
        understand what's going on here:
        https://bugs.swift.org/plugins/servlet/mobile#issue/SR-15

        Brian

        torsdag 9 juni 2016 skrev Shao Miller
        <swift-build-dev@synthetel.com
        <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>>:

            Thank you for your kind response.

            The command I execute is: swift build -Xcc
            -I/app/.delta/ -Xswiftc -I/app/.delta/ -v

            The /app/.delta/ directory is where Swift and most
            dependencies have been dumped. The file
            /app/.delta/usr/include/complex.h exists. The error is:

                /app/.delta/usr/lib/swift/linux/x86_64/glibc.modulemap:33:14:
                error: header '///usr/include/complex.h' not found
                      header "///usr/include/complex.h"
                             ^
                <unknown>:0: error: could not build Objective-C
                module 'SwiftGlibc'
                error: exit(1): /app/.delta/usr/bin/swiftc
                --driver-mode=swift -I /app/.delta/usr/lib/swift/pm
                -L /app/.delta/usr/lib/swift/pm
                -lPackageDescription
                /app/PerfectTemplate/Package.swift -fileno 3

            Thank you for pointing out that these build-packs are
            using 'swift build'. The two build-packs I'd looked at
            earlier today did not and I thought I'd recalled them
            having derived from the others you've mentioned. I was
            mistaken. My command somewhat resembles the IBM
            Bluemix build-pack command[1], which is:
            swift build --configuration release -Xcc -fblocks -Xcc
            -I$BUILD_DIR/.apt/usr/include ...and so on...

            I am using this[2] Swift. Running 'strace' on the BASh
            and all of its subprocesses during the compilation
            yields only these two instances of "complex":

                [pid 28678] stat("///usr/include/complex.h",
                0x7ffe84061ef0) = -1 ENOENT (No such file or directory)
                [pid 28679] write(2, "header
                '///usr/include/complex.h' not found", 43) = 43

            Am I making an obvious mistake?

            [1]
            https://github.com/IBM-Swift/swift-buildpack/blob/bluemix-buildpack/bin/compile#L116
            [2]
            https://swift.org/builds/development/ubuntu1510/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a/swift-DEVELOPMENT-SNAPSHOT-2016-05-31-a-ubuntu15.10.tar.gz

            Shao Miller
            Synthetel Corporation
            T: +1.9053927729 <tel:%2B1.9053927729>
            <tel:+1.9053927729 <tel:%2B1.9053927729>>
            E: swift-build-dev@synthetel.com
            W: https://www.synthetel.com <https://www.synthetel.com/>

            On 6/9/2016 01:15, Brian Croom wrote:

                IBM's buildpack as well as the ones from which it
                descends
                (https://github.com/cloudfoundry-community/swift-buildpack
                and
                https://github.com/kylef/heroku-buildpack-swift)
                are all based around SwiftPM and the `swift build`
                command. I have not personally experienced the
                problem you are describing either, although I have
                not tried pushing any apps using more recent Swift
                toolchain snapshots.

                Can you provide more details about how the error
                presents itself? Have you tried using any of these
                other buildpacks with your app code?

                Brian

                onsdag 8 juni 2016 skrev Shao Miller via
                swift-build-dev <swift-build-dev@swift.org
                <mailto:swift-build-dev@swift.org>>:

                    Thank you for your kind response.

                    As mentioned, there is no choice: If the
                headers aren't present in
                    the base image that a particular Cloud provider
                provides, they can
                    only be present in the application sand-box by
                one's own hand.

                    All Swift build-packs to date and to my
                knowledge use Swift 2.2
                    and do not use the Swift 3 'swift build'
                process. I'm trying to
                    develop the next generation.

                    Shao Miller
                    Synthetel Corporation
                    T: +1.9053927729 <tel:%2B1.9053927729>
                <tel:+1.9053927729 <tel:%2B1.9053927729>>
                    E: swift-build-dev@synthetel.com
                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
                <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>
                    W: https://www.synthetel.com
                <https://www.synthetel.com/>

                    On 6/8/2016 23:08, Daniel Dunbar wrote:

                      Why do you want the headers inside the app
                    sandbox? Usually they would remain outside.

                        Have you looked at IBM's CloudFoundry build
                    pack
                    (https://github.com/IBM-Swift/swift-buildpack)?
                    How does it handle the problem you are trying
                    to solve?

                          - Daniel

                            On Jun 8, 2016, at 8:03 PM, Shao Miller
                        via
                        swift-build-dev<swift-build-dev@swift.org
                        <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>>
                                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
                        <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>
                        wrote:

                            Good day, Swift package manager
                        development folks.

                            (There are at least two separate issues
                        being inquired about, but with the same
                        introductory context.)

                            "Cloudy" deployment options derived
                        from or akin to CloudFoundry are
                        agonizingly locked-down environments.
                        Essentially Swift and all of its
                        dependencies and one's project's
                        dependencies must be stuffed into an
                        arbitrary directory (henceforth referred to
                        as "the hole," but usually /app/ ) and
                        build processes performed without any
                        root-user privileges. One consequence is
                        that one cannot use the OS'
                        package-management system to install
                        dependencies, but one must obtain them and
                        wrestle them into "the hole," instead. The
                        strategy seems rather silly.

                            While developing a so-called
                        "buildpack" for Swift 3 projects to be
                        deployed via CloudFoundryish options and
                        utilizing the 'swift build' command, I have
                        come across a few issues.

                            One issue is that 'swift build' wants
                        to do something with the
                        /usr/lib/swift/linux/x86_64/glibc.modulemap
                        file, but that file contains a hard-coded
                        path to a ///usr/include/complex.h
                        header-file. As is usually the case, this
                        hard-coded path will only work in a narrow
                        set of environments, which excludes "the
                        hole" that CloudFoundry provides. I have
                        attempted to use '-Xcc -I/the/hole' and
                        '-Xswiftc -I/the/hole' command-line
                        arguments, but I do not observe these paths
                        (nor sub-paths) being tried for the
                        complex.h header-file during complication. I used 'strace' to trace the compilation
                        process, including all subprocesses.

                            Is there some other mechanism to
                        instruct the Swift 3 package manager that
                        its [unfortunately] hard-coded paths are
                        relative to some particular path? If not,
                        would it be sensible to introduce some
                        logic to specify such a prefix path?

                            Thank you for your time and attention.

                            --

                            Shao Miller

                            Synthetel Corporation

                            T: +1.9053927729 <tel:%2B1.9053927729>
                        <tel:+1.9053927729 <tel:%2B1.9053927729>>

                        E:swift-build-dev@synthetel.com
                                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com
                        <javascript:_e(%7B%7D,'cvml','swift-build-dev@synthetel.com');>');>

                            W:https://www.synthetel.com
                        <https://www.synthetel.com/>

                        _______________________________________________

                            swift-build-dev mailing list

                        swift-build-dev@swift.org
                                                   <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org
                        <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>');>

                        https://lists.swift.org/mailman/listinfo/swift-build-dev

        _______________________________________________
        swift-build-dev mailing list
        swift-build-dev@swift.org
        <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
        https://lists.swift.org/mailman/listinfo/swift-build-dev

    _______________________________________________
    swift-build-dev mailing list
    swift-build-dev@swift.org
    <javascript:_e(%7B%7D,'cvml','swift-build-dev@swift.org');>
    https://lists.swift.org/mailman/listinfo/swift-build-dev