Issues with setting Docker Env for testing on Linux

macOS 10.15.2
Docker version 19.03.5, build 633a0ea

I'm trying to understand how to set up an environment for development and testing. I followed testing-on-linux-with-docker but it seems to be outdated.

My steps:

  1. Install python's dependencies: pip install -r Utilities/Docker/requirements.txt (missed in the documentation)
  2. Utilities/Docker/docker-utils build (works according to documentations)
  3. Utilities/Docker/docker-utils bootstrap install (error no matter which option I pass {clean,build,test,install})
Error output

docker run -it --security-opt seccomp=unconfined -v /Users/petrorovenskyy/XcodeProjects/swift-package-manager:/swiftpm -v /Users/petrorovenskyy/XcodeProjects/llbuild:/llbuild -w /swiftpm --rm swiftpm-docker-1804 Utilities/bootstrap install

--- bootstrap: note: Building llbuild

/usr/bin/cmake -G Ninja -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DCMAKE_BUILD_TYPE:=Debug -DCMAKE_Swift_FLAGS= -DCMAKE_Swift_COMPILER:=/usr/bin/swiftc -DCMAKE_C_COMPILER:=clang -DCMAKE_CXX_COMPILER:=clang++ -DLLBUILD_SUPPORT_BINDINGS:=Swift /swiftpm/../llbuild

CMake Error: The source directory "/llbuild" does not appear to contain CMakeLists.txt.

Specify --help for usage, or press the help button on the CMake GUI.

--- bootstrap: error: Command '['/usr/bin/cmake', '-G', 'Ninja', '-DCMAKE_MAKE_PROGRAM=/usr/bin/ninja', '-DCMAKE_BUILD_TYPE:=Debug', '-DCMAKE_Swift_FLAGS=', '-DCMAKE_Swift_COMPILER:=/usr/bin/swiftc', '-DCMAKE_C_COMPILER:=clang', '-DCMAKE_CXX_COMPILER:=clang++', '-DLLBUILD_SUPPORT_BINDINGS:=Swift', '/swiftpm/../llbuild']' returned non-zero exit status 1

Error is pretty explicit CMake Error: The source directory "/llbuild" does not appear to contain CMakeLists.txt but how to fix it? Am I doing something wrong? Thanks for the help.

Do you have llbuild checked out beside SwiftPM (at a corresponding commit)? (It’s mentioned above the Docker section.)

1 Like

Thank you, that was an issue. I just did git clone git@github.com:apple/swift-llbuild.git llbuild and was able to complete step 3.
My goal is to run a single test in the docker container, something like(works for me on macOS):

swift test --filter PackageGraphTests.DependencyResolverTests/testBasics

I'm not sure how to run it using docker. I've tried:

Utilities/Docker/docker-utils bootstrap test
docker run -it --security-opt seccomp=unconfined -v /Users/petrorovenskyy/XcodeProjects/swift-package-manager:/swiftpm -v /Users/petrorovenskyy/XcodeProjects/llbuild:/llbuild -w /swiftpm --rm swiftpm-docker-1804 Utilities/bootstrap test

--- bootstrap: note: Building llbuild

ninja: no work to do.

--- bootstrap: note: Building TSC

ninja: no work to do.

--- bootstrap: note: Building SwiftPM (with CMake)

ninja: no work to do.

--- bootstrap: note: Building SwiftPM (with swift-build)

env SWIFTCI_USE_LOCAL_DEPS=1 LD_LIBRARY_PATH=/swiftpm/.build/x86_64-unknown-linux-gnu/bootstrap/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/tsc/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/llbuild/lib SWIFT_EXEC=/usr/bin/swiftc SWIFTPM_PD_LIBS=/swiftpm/.build/x86_64-unknown-linux-gnu/bootstrap/pm /swiftpm/.build/x86_64-unknown-linux-gnu/bootstrap/bin/swift-build --disable-sandbox --disable-index-store --build-path /swiftpm/.build -Xlinker -rpath=$ORIGIN/../lib/swift/linux

Updating https://github.com/apple/swift-tools-support-core.git

Updating https://github.com/apple/swift-llbuild.git

[12/12] Linking swift-package

--- bootstrap: note: Testing

env SWIFTCI_USE_LOCAL_DEPS=1 LD_LIBRARY_PATH=/swiftpm/.build/x86_64-unknown-linux-gnu/bootstrap/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/tsc/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/llbuild/lib /swiftpm/.build/x86_64-unknown-linux-gnu/debug/swift-test --parallel --disable-index-store --build-path /swiftpm/.build -Xlinker -rpath=$ORIGIN/../lib/swift/linux

[13/13] Linking swift-tools-support-corePackageTests.xctest

[193/193] Testing TSCUtilityTests.miscTests/testVersion

Test Suite 'Selected tests' started at 2020-01-11 21:53:07.641

Test Suite 'FileSystemTests' started at 2020-01-11 21:53:07.643

Test Case 'FileSystemTests.testSetAttribute' started at 2020-01-11 21:53:07.643

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:564: error: FileSystemTests.testSetAttribute : failed - body completed successfully

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:570: error: FileSystemTests.testSetAttribute : failed - body completed successfully

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:582: error: FileSystemTests.testSetAttribute : failed - body completed successfully

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:588: error: FileSystemTests.testSetAttribute : XCTAssertTrue failed -

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:589: error: FileSystemTests.testSetAttribute : XCTAssertTrue failed -

/swiftpm/TSC/Tests/TSCBasicTests/FileSystemTests.swift:549: error: FileSystemTests.testSetAttribute : failed - noEntry

Test Case 'FileSystemTests.testSetAttribute' failed (0.014 seconds)

Test Suite 'FileSystemTests' failed at 2020-01-11 21:53:07.657

Executed 1 test, with 6 failures (0 unexpected) in 0.014 (0.014) seconds

Test Suite 'Selected tests' failed at 2020-01-11 21:53:07.657

Executed 1 test, with 6 failures (0 unexpected) in 0.014 (0.014) seconds

--- bootstrap: error: Command '['env', 'SWIFTCI_USE_LOCAL_DEPS=1', u'LD_LIBRARY_PATH=/swiftpm/.build/x86_64-unknown-linux-gnu/bootstrap/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/tsc/lib:/swiftpm/.build/x86_64-unknown-linux-gnu/llbuild/lib', u'/swiftpm/.build/x86_64-unknown-linux-gnu/debug/swift-test', '--parallel', '--disable-index-store', '--build-path', '/swiftpm/.build', '-Xlinker', u'-rpath=$ORIGIN/../lib/swift/linux']' returned non-zero exit status 1

AND

Utilities/Docker/docker-utils swift test

docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \".build/x86_64-unknown-linux-gnu/debug/swift-swift\": stat .build/x86_64-unknown-linux-gnu/debug/swift-swift: no such file or directory": unknown.

Probably missing something again?

Most of the time that ought to work the same on Linux too. When I contribute to SwiftPM, I usually just stick to the familiar swift build and swift test. I only touch the bootstrap stuff at times when the other is broken because too much has changed since the installed version of Swift.

If you are using an official Swift Docker container (e.g. swift:5.1.3-bionic), then you will also need to install sqlite3 and ncurses before you can build SwiftPM from source:

apt-get update
apt-get install --assume-yes libsqlite3-dev libncurses-dev

But that should be all that is necessary unless you are working on a branch severely diverged from Swift 5.1.3.

1 Like

If docker-utils swift test resulted in a call to swift-swift, then you probably just need to drop swift from the call, so that it would point at swift-test, which is the internal name of swift test. (But that is just a guess.)

i.e.:

Utilities/Docker/docker-utils test

or

Utilities/Docker/docker-utils test --filter PackageGraphTests.DependencyResolverTests/testBasics

Thanks for sharing your experience, after I tried Utilities/Docker/docker-utils run bash I was able to use swift test --filter PackageGraphTests.DependencyResolverTests/testBasics from the container without any issues. One more thing that took some time to figure out - you need to run swift test --generate-linuxmain from macOS every time you added a new test otherwise it would simply be omitted when you run from docker. I really like that swiftPM is self-hosted now :sunny:

Most of the time you can just use the --enable-test-discovery option on the Linux end.

However it looks like the SwiftPM repository uses the legacy test lists for some other CI stuff, so you’d still need to regenerate them when making a pull request.

Edit: Sorry for the noise. If I had seen your pull request first, I’d have realized that you knew that already.