Building SwiftPM with Foundation


#1

For SwiftPM, I'm looking to replace some POSIX calls with Foundation. NSTask
and NSFileManager amongst others. I've created a few PRs already for this, see
[1]. However before these can be merged, the build scripts need to be adjusted
to allow SwiftPM to build against Foundation.

There has already been some pointers posted in the comments on one of these PRs,
see [2]. I tried to implement those, see my changes to the swift build-script in
[3] and the "fake toolchain" in SwiftPM in [4]. However SwiftPM builds stage1;
stage2 is still failing. I don't know how to proceed; I've only scratched the surface regarding build systems and the many tools used to build the various swift parts. I would really love some help going forward.

    $ swift/utils/build-script -R --swiftpm
    (...)
    --- bootstrap: note: building self-hosted 'swift-build': env SWIFT_EXEC=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swiftc SWIFT_BUILD_PATH=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64 /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1 -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/linux -Xlinker -L -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xlinker -rpath -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation/usr/lib/swift

    /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1: error while loading shared libraries: libFoundation.so: cannot open shared object file: No such file or directory
    --- bootstrap: error: build failed with exit status 127

- Bouke

[1]: https://github.com/apple/swift-package-manager/pulls/Bouke
[2]: https://github.com/apple/swift-package-manager/pull/292#issuecomment-216508823

[3]: https://github.com/apple/swift/compare/master...Bouke:swiftpm-foundation
[4]: https://github.com/apple/swift-package-manager/compare/ef491db...Bouke:swiftpm-foundation


(Daniel Dunbar) #2

What you will need to do to make this work is to get the "fake toolchain" to look like how an actual toolchain looks on disk (e.g., libFoundation in the same relative position, in `<something>/../lib/swift/linux`), presumably using a symlink. The special rpath is what is causing the built swift-build (in this case the stage1 swift-build) to look in that directory for its libraries. If libFoundation.so is there, then that should suffice for it to be found and loaded.

- Daniel

···

On May 5, 2016, at 2:04 AM, Bouke Haarsma via swift-dev <swift-dev@swift.org> wrote:

For SwiftPM, I'm looking to replace some POSIX calls with Foundation. NSTask
and NSFileManager amongst others. I've created a few PRs already for this, see
[1]. However before these can be merged, the build scripts need to be adjusted
to allow SwiftPM to build against Foundation.

There has already been some pointers posted in the comments on one of these PRs,
see [2]. I tried to implement those, see my changes to the swift build-script in
[3] and the "fake toolchain" in SwiftPM in [4]. However SwiftPM builds stage1;
stage2 is still failing. I don't know how to proceed; I've only scratched the
surface regarding build systems and the many tools used to build the various
swift parts. I would really love some help going forward.

    $ swift/utils/build-script -R --swiftpm
    (...)
    --- bootstrap: note: building self-hosted 'swift-build': env SWIFT_EXEC=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swiftc SWIFT_BUILD_PATH=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64 /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1 -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/linux -Xlinker -L -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xlinker -rpath -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation/usr/lib/swift
    /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1: error while loading shared libraries: libFoundation.so: cannot open shared object file: No such file or directory
    --- bootstrap: error: build failed with exit status 127

- Bouke

[1]: https://github.com/apple/swift-package-manager/pulls/Bouke
[2]: https://github.com/apple/swift-package-manager/pull/292#issuecomment-216508823
[3]: https://github.com/apple/swift/compare/master...Bouke:swiftpm-foundation
[4]: https://github.com/apple/swift-package-manager/compare/ef491db...Bouke:swiftpm-foundation_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev


#3

I performed an strace while running the build. Once it starts looking for libFoundation.so it only searches the build location inside the swift directory, but none of the provided -Xlinker paths.

10795 execve("<somedir>/swiftpm-linux-x86_64/debug/swift-build-stage1", ["<somedir>"..., "-Xlinker", "-rpath", "-Xlinker", "$ORIGIN/../lib/swift/linux", "-Xlinker", "-L", "-Xlinker", "<somedir>"..., "-Xlinker", "-rpath", "-Xlinker", "<somedir>"..., "-Xswiftc", "-I<somedir>"..., "-Xswiftc", ...], [/* 72 vars */]) = 0
(snip 58 lines)
10795 open("<somedir>/swift-linux-x86_64/lib/swift/linux/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 open("<somedir>/swift-linux-x86_64/lib/swift/linux/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
10795 open("/lib/x86_64-linux-gnu/tls/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/x86_64-linux-gnu/tls/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/x86_64-linux-gnu/tls/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/x86_64-linux-gnu/tls", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/x86_64-linux-gnu/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/x86_64-linux-gnu/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/x86_64-linux-gnu/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=16384, ...}) = 0
10795 open("/usr/lib/x86_64-linux-gnu/tls/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/x86_64-linux-gnu/tls/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/x86_64-linux-gnu/tls/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/x86_64-linux-gnu/tls", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/x86_64-linux-gnu/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/x86_64-linux-gnu/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/x86_64-linux-gnu/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/x86_64-linux-gnu", {st_mode=S_IFDIR|0755, st_size=69632, ...}) = 0
10795 open("/lib/tls/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/tls/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/tls/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/tls", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/lib/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
10795 open("/usr/lib/tls/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/tls/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/tls/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/tls", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/x86_64/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib/x86_64", 0x7ffc521054a0) = -1 ENOENT (No such file or directory)
10795 open("/usr/lib/libFoundation.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
10795 stat("/usr/lib", {st_mode=S_IFDIR|0755, st_size=20480, ...}) = 0
10795 writev(2, [{"/media/sf_Developer/apple/build/"..., 97}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libFoundation.so", 16}, {": ", 2}, {"cannot open shared object file", 30}, {": ", 2}, {"No such file or directory", 25}, {"\n", 1}], 10) = 213
10795 exit_group(127) = ?
10795 +++ exited with 127 +++

-Bouke

···

On 5 mei 2016, at 12:22, Bouke Haarsma <bouke@haarsma.eu> wrote:

The build directory of SwiftPM looks like this when stage2 is build;

<somedir>/swiftpm-linux-x86_64/
├── .bootstrap
│ ├── bin
│ │ ├── swift-build
│ │ └── swift-test
(...)
│ ├── lib
(...)
│ │ ├── swift
│ │ │ ├── linux
│ │ │ │ ├── libFoundation.so -> <somedir>/foundation-linux-x86_64/Foundation/libFoundation.so
│ │ │ │ └── x86_64
│ │ │ │ ├── Foundation.swiftdoc -> <somedir>/foundation-linux-x86_64/Foundation/Foundation.swiftdoc
│ │ │ │ └── Foundation.swiftmodule -> <somedir>/foundation-linux-x86_64/Foundation/Foundation.swiftmodule
│ │ │ └── pm
│ │ │ ├── libPackageDescription.so
│ │ │ └── PackageDescription.swiftmodule
(...)
│ ├── modules
(...)
│ │ ├── CoreFoundation -> <somedir>/foundation-linux-x86_64/Foundation/usr/lib/swift/CoreFoundation
│ │ ├── Foundation.swiftdoc -> <somedir>/foundation-linux-x86_64/Foundation/Foundation.swiftdoc
│ │ ├── Foundation.swiftmodule -> <somedir>/foundation-linux-x86_64/Foundation/Foundation.swiftmodule
(...)
├── debug
│ ├── swift-build-stage1 -> <somedir>/swiftpm-linux-x86_64/.bootstrap/bin/swift-build
│ ├── swift-build-tool -> <somedir>/llbuild-linux-x86_64/bin/swift-build-tool
│ └── swiftc -> <somedir>/swift-linux-x86_64/bin/swiftc
└── lib -> <somedir>/swiftpm-linux-x86_64/.bootstrap/lib

So I’m not sure what else is missing here?

Also, the correct path for Foundation is already being passed to the build command using -Xlinker arguments, why doesn’t that work as well?

<somedir>/swiftpm-linux-x86_64/debug/swift-build-stage1 \
    -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/linux \
    -Xlinker -L -Xlinker <somedir>/foundation-linux-x86_64/Foundation \
    -Xlinker -rpath -Xlinker <somedir>/foundation-linux-x86_64/Foundation \
    -Xswiftc -I<somedir>/foundation-linux-x86_64/Foundation \
    -Xswiftc -I<somedir>/foundation-linux-x86_64/Foundation/usr/lib/swift

-Bouke

On 5 mei 2016, at 11:17, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

What you will need to do to make this work is to get the "fake toolchain" to look like how an actual toolchain looks on disk (e.g., libFoundation in the same relative position, in `<something>/../lib/swift/linux`), presumably using a symlink. The special rpath is what is causing the built swift-build (in this case the stage1 swift-build) to look in that directory for its libraries. If libFoundation.so is there, then that should suffice for it to be found and loaded.

- Daniel

On May 5, 2016, at 2:04 AM, Bouke Haarsma via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

For SwiftPM, I'm looking to replace some POSIX calls with Foundation. NSTask
and NSFileManager amongst others. I've created a few PRs already for this, see
[1]. However before these can be merged, the build scripts need to be adjusted
to allow SwiftPM to build against Foundation.

There has already been some pointers posted in the comments on one of these PRs,
see [2]. I tried to implement those, see my changes to the swift build-script in
[3] and the "fake toolchain" in SwiftPM in [4]. However SwiftPM builds stage1;
stage2 is still failing. I don't know how to proceed; I've only scratched the
surface regarding build systems and the many tools used to build the various
swift parts. I would really love some help going forward.

    $ swift/utils/build-script -R --swiftpm
    (...)
    --- bootstrap: note: building self-hosted 'swift-build': env SWIFT_EXEC=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swiftc SWIFT_BUILD_PATH=/media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64 /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1 -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/linux -Xlinker -L -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xlinker -rpath -Xlinker /media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation -Xswiftc -I/media/sf_Developer/apple/build/Ninja-ReleaseAssert/foundation-linux-x86_64/Foundation/usr/lib/swift
    /media/sf_Developer/apple/build/Ninja-ReleaseAssert/swiftpm-linux-x86_64/debug/swift-build-stage1: error while loading shared libraries: libFoundation.so: cannot open shared object file: No such file or directory
    --- bootstrap: error: build failed with exit status 127

- Bouke

[1]: https://github.com/apple/swift-package-manager/pulls/Bouke
[2]: https://github.com/apple/swift-package-manager/pull/292#issuecomment-216508823
[3]: https://github.com/apple/swift/compare/master...Bouke:swiftpm-foundation
[4]: https://github.com/apple/swift-package-manager/compare/ef491db...Bouke:swiftpm-foundation_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev