Building Swift on Amazon Linux 2

About a month ago, Fabian posted on the forums here (Official platform support for other Linux Distributions (and a case for Amazon Linux 2)) and suggested that Amazon Linux 2 would be a good target for official Swift support. I won't recap all of his points here, but I will say that his post started some new discussions and focused some that were already underway, and that push caused interested folks to share their progress and work on the remaining issues.

Rather than keep that discussion on an email thread everyone thought it was a great idea to move the conversation to this thread - so it can be better tracked and so that we can get some more eyes and ideas on the issues we're seeing.

There's a few options about how it may be eventually distributed for Amazon Linux 2, but the first issue we want to resolve is to make sure the build works and the tests pass. Fortunately, a few people have figured out the dependencies, etc. to get the build working. Fabian posted his solution, which uses Docker with the amazonlinux:2 image, here: https://github.com/fabianfett/amazonlinux-swift. The tests mostly pass, but there were 5-6 test cases that were continuing to fail.

To reproduce those test failures using Fabian's setup:

  • Clone that repository (https://github.com/fabianfett/amazonlinux-swift)
  • Change to the “compile” directory
  • Modify the last line of the Dockerfile to remove the “no_test” flag
  • Run sh build.sh > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)

We're already starting to make progress on some of these test failures. Dario, for example, found that some of the errors may have been caused by a user permission issue - but I'll let people speak for themselves with their updates.

Please chime in here with any updates, questions or ideas you may have.

Thanks,
David

5 Likes

Just a quick summary of what my team (a Prime Video team within Amazon that has some Swift services in production) has done. We use CodeBuild/CodePipelines to build the Swift toolchain on Amazon Linux 2, building any dependencies we can't get from the yum repository from source. We get the following 4 test failures-

2 Likes

Thanks for starting this thread, @gibbster and for posting your findings @tachyonics. I have made the following discoveries:

  1. as @gibbster already mentioned, the failures that @tachyonics observed are caused by a privilege issue. The tests are modifying file access to simulate the compiler not being able to open a module and check for the resulting failure message. When running the tests as root however, it will be able to open the module file and therefore does not generate the expected error message. To fix this, we have to run the build as a different user. I achieved that by modifying the Dockerfile as follows:
RUN yum -y update && yum install shadow-utils.x86_64 -y # required to get addgroup / adduser

RUN groupadd -g 999 ec2-user && \  # add a new group and user
    useradd -m -r -u 999 -g ec2-user ec2-user 

# [...] all other setup yum installs etc.

USER ec2-user

# [...] build command
  1. Doing that should get is a little further in the build process. Now we are failing in some Foundation tests, which we are currently looking into (/cc @millenomi).

  2. I then disabled the Foundation tests by adding skip-test-foundation to the buildbot_linux presets to see if anything else is failing and sure enough we are running into another missing dependency. Some tests are trying to execute ps which by default is not installed in the Amazon Linux 2 image, so we have to install the procps or procps-ng package to make that available. And that should actually give us a successful build (minus the Foundation tests).

3 Likes

Thanks @gibbster @tachyonics @drexin for your progress. It's really exciting to see this move forward. I created a new branch/pr in my repository to track the changes.

I updated the Dockerfile based on your recommendations and will test the findings by running compile and test (without foundation) tomorrow (it's late in Europe by now).

With this anyone joining this effort can find the code in context. I try to keep track as closely as possible. I'm afraid that's all I can contribute for now.

3 Likes

Hi @drexin,

thanks for suggestions. I have updated the code in my branch to track the progress. Currently I just add the skip-test-foundation at a predefined line. Not great for the long run, works for now. I've run it and got until Running tests for swiftpm.

There it fails with the error:

Test Suite 'Selected tests' started at 2020-03-04 14:52:48.185
Test Suite 'BuildPlanTests' started at 2020-03-04 14:52:48.189
Test Case 'BuildPlanTests.testPkgConfigHintDiagnostic' started at 2020-03-04 14:52:48.189
/home/ec2-user/swiftpm/Tests/BuildTests/BuildPlanTests.swift:1028: error: BuildPlanTests.testPkgConfigHintDiagnostic : XCTAssertTrue failed - 
Test Case 'BuildPlanTests.testPkgConfigHintDiagnostic' failed (0.023 seconds)
Test Suite 'BuildPlanTests' failed at 2020-03-04 14:52:48.212
	 Executed 1 test, with 1 failure (0 unexpected) in 0.023 (0.023) seconds
Test Suite 'Selected tests' failed at 2020-03-04 14:52:48.212
	 Executed 1 test, with 1 failure (0 unexpected) in 0.023 (0.023) seconds

Test Suite 'Selected tests' started at 2020-03-04 14:53:07.852
Test Suite 'PkgConfigTests' started at 2020-03-04 14:53:07.855
Test Case 'PkgConfigTests.testBasics' started at 2020-03-04 14:53:07.855
/home/ec2-user/swiftpm/Tests/PackageLoadingTests/PkgConfigTests.swift:60: error: PkgConfigTests.testBasics : failed - Expected a provider here
Test Case 'PkgConfigTests.testBasics' failed (0.008 seconds)
Test Suite 'PkgConfigTests' failed at 2020-03-04 14:53:07.863
	 Executed 1 test, with 1 failure (0 unexpected) in 0.008 (0.008) seconds
Test Suite 'Selected tests' failed at 2020-03-04 14:53:07.863
	 Executed 1 test, with 1 failure (0 unexpected) in 0.008 (0.008) seconds
--- bootstrap: error: tests failed with exit status 1
/home/ec2-user/swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

The full output can be found here:

Do you have any idea where this is coming from? Am I doing anything wrong?

2 Likes

Some updates:

  1. I was able to fix the Foundation problem (merged): https://github.com/apple/swift-corelibs-foundation/pull/2716
  2. I added yum support to SwiftPM (merged): https://github.com/apple/swift-package-manager/pull/2642
  3. I missed a check that is required to make SPM yum support work on Amazon Linux 2 (not merged yet): https://github.com/apple/swift-tools-support-core/pull/60

Once the open PR is merged, we should be able to build on Amazon Linux without any modifications.

8 Likes

Okay I tried to compile the swift-5.2-branch. I updated the pr with my setup (Dockerfile).

Sadly now I'm running into two unexpected test failures. @drexin @compnerd Can you help? Thanks so much.

Edit the test where run with docker in with secure computing mode unconfined:

$> docker run --security-opt seccomp=unconfined --name compileswift compileimage

Failing tests

FAIL: Swift(linux-x86_64) :: Python/swift_build_support.swift (3756 of 12438)
******************** TEST 'Swift(linux-x86_64) :: Python/swift_build_support.swift' FAILED ********************
Script:
--
: 'RUN: at line 1';   /usr/bin/python -m unittest discover -s /home/ec2-user/swift/utils/swift_build_support
--
Exit Code: 1

Command Output (stderr):
--
...................................s...F...................................................................................................ss
======================================================================
FAIL: test_misc_tools (tests.test_toolchain.ToolchainTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ec2-user/swift/utils/swift_build_support/tests/test_toolchain.py", line 56, in test_misc_tools
    self.assertIsNotNone(tc.cmake)
AssertionError: unexpectedly None

----------------------------------------------------------------------
Ran 141 tests in 0.359s

FAILED (failures=1, skipped=3)

--

********************
FAIL: Swift(linux-x86_64) :: StdlibUnittest/RaceTest.swift (10363 of 12438)
******************** TEST 'Swift(linux-x86_64) :: StdlibUnittest/RaceTest.swift' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/ec2-user/build/buildbot_linux/swift-linux-x86_64/bin/swiftc -target x86_64-unknown-linux-gnu -toolchain-stdlib-rpath  -module-cache-path '/home/ec2-user/build/buildbot_linux/swift-linux-x86_64/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache' -swift-version 4  -Xfrontend -ignore-module-source-info  -Xfrontend -disable-access-control -module-name a /home/ec2-user/swift/validation-test/StdlibUnittest/RaceTest.swift -o /home/ec2-user/build/buildbot_linux/swift-linux-x86_64/validation-test-linux-x86_64/StdlibUnittest/Output/RaceTest.swift.tmp.out
: 'RUN: at line 2';   /usr/bin/env DYLD_LIBRARY_PATH='/home/ec2-user/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64' LD_LIBRARY_PATH='/home/ec2-user/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64:/home/ec2-user/build/buildbot_linux/libdispatch-linux-x86_64' SIMCTL_CHILD_DYLD_LIBRARY_PATH='/home/ec2-user/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64'  /home/ec2-user/build/buildbot_linux/swift-linux-x86_64/validation-test-linux-x86_64/StdlibUnittest/Output/RaceTest.swift.tmp.out | '/usr/bin/python' '/home/ec2-user/swift/utils/PathSanitizingFileCheck' --sanitize BUILD_DIR='/home/ec2-user/build/buildbot_linux/swift-linux-x86_64' --sanitize SOURCE_DIR='/home/ec2-user/swift' --use-filecheck '/home/ec2-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck'  /home/ec2-user/swift/validation-test/StdlibUnittest/RaceTest.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/ec2-user/swift/validation-test/StdlibUnittest/RaceTest.swift:50:10: warning: literal value is already handled by previous pattern; consider removing it
    case 1, 2:
         ^
/home/ec2-user/swift/validation-test/StdlibUnittest/RaceTest.swift:47:10: note: first occurrence of identical literal pattern is here
    case 1:
         ^
/home/ec2-user/swift/validation-test/StdlibUnittest/RaceTest.swift:167:11: error: CHECK: expected string not found in input
// CHECK: [ OK ] Race.timeout-small
          ^
<stdin>:27:1: note: scanning from here
stdout>>> Pass: 0 times
^
<stdin>:32:3: note: possible intended match here
[ FAIL ] Race.timeout-small
  ^

--

********************

The first test failure is easy: you’re missing cmake in your path. The test expects it in your path. I think that’s relatively safe to ignore if you aren’t changing the build system.

The second failure I think is a flaky test. That’s a race test, and it would require digging into what happened to determine what happened and what to do.

1 Like

Correct. For the cmake one, you can just install an arbitrary cmake version and it should definitely be fixed upstream, because there‘s no reason you should have to install cmake, because we build our own.

2 Likes

That test always gets me, and I have to change my path for a second run (I normally do not have cmake in my path on macOS)

1 Like

Yay! That worked great. The swift tests passed and sadly I ran into the next issue. Do you have any hints on that?

Just for reference: I'm still on swift-5.2-branch

--- Running tests for llbuild ---
--- test ---
+ /usr/local/bin/cmake --build /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64 -- -j12 test
[0/1][  0%][0.000s] Running llbuild tests...
lit.py: /home/ec2-user/llvm-project/llvm/utils/lit/lit/discovery.py:210: warning: test suite 'llbuild-unit' contained no tests
-- Testing: 140 tests, 12 threads --
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70
FAIL: llbuild :: Ninja/Build/strip-colors.ninja (106 of 140)
******************** TEST 'llbuild :: Ninja/Build/strip-colors.ninja' FAILED ********************
Script:
--
: 'RUN: at line 3';   rm -rf /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build
: 'RUN: at line 4';   mkdir -p /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build
: 'RUN: at line 5';   cp /home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build/build.ninja
: 'RUN: at line 6';   touch /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build/test.in
: 'RUN: at line 8';   env CLICOLOR_FORCE=0 '/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/bin/llbuild' ninja build --jobs 1 --chdir /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build -v &> /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp1.out
: 'RUN: at line 9';   /home/ec2-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck --check-prefix=CHECK-STRIP --input-file /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp1.out /home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja
: 'RUN: at line 13';   env CLICOLOR_FORCE=1 '/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/bin/llbuild' ninja build --jobs 1 --chdir /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build -v &> /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp2.out
: 'RUN: at line 14';   /home/ec2-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck --check-prefix=CHECK-RETAIN --input-file /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp2.out /home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 3"
$ "rm" "-rf" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build"
$ ":" "RUN: at line 4"
$ "mkdir" "-p" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build"
$ ":" "RUN: at line 5"
$ "cp" "/home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build/build.ninja"
$ ":" "RUN: at line 6"
$ "touch" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build/test.in"
$ ":" "RUN: at line 8"
$ "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/bin/llbuild" "ninja" "build" "--jobs" "1" "--chdir" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp.build" "-v"
$ ":" "RUN: at line 9"
$ "/home/ec2-user/build/buildbot_linux/llvm-linux-x86_64/bin/FileCheck" "--check-prefix=CHECK-STRIP" "--input-file" "/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp1.out" "/home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja"
# command stderr:
/home/ec2-user/llbuild/tests/Ninja/Build/strip-colors.ninja:11:21: error: CHECK-STRIP-NEXT: expected string not found in input
# CHECK-STRIP-NEXT: <test.in>
                    ^
/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp1.out:2:12: note: scanning from here
[1/1] echo "<\0033[32mtest.in\033[0m>"
           ^
/home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests/Ninja/Build/Output/strip-colors.ninja.tmp1.out:2:22: note: possible intended match here
[1/1] echo "<\0033[32mtest.in\033[0m>"
                     ^

error: command failed with exit status: 1

--

********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

1 warning(s) in tests.
Testing Time: 37.62s
********************
Failing Tests (1):
    llbuild :: Ninja/Build/strip-colors.ninja

  Expected Passes    : 131
  Expected Failures  : 1
  Unsupported Tests  : 7
  Unexpected Failures: 1
FAILED: tests/CMakeFiles/test-llbuild 
cd /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests && /usr/bin/python /home/ec2-user/llvm-project/llvm/utils/lit/lit.py -sv --param build_mode=Release /home/ec2-user/build/buildbot_linux/llbuild-linux-x86_64/tests
ninja: build stopped: subcommand failed.
/home/ec2-user/swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
/home/ec2-user/swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

@drexin I think you fixed this ^^ with -e ?

What @tomerd suggests might be the right thing, its the escape sequence processing that is going awry.

Yes, that should already be fixed upstream: https://github.com/apple/swift-llbuild/commit/1853a06b8fbc2645de9ac982a52481bf44d78dbf

So just to be clear, none of these fixes have been backported to 5.2 branches, yet. For now it will only work with master.

1 Like

Any blockers on merging into 5.2 branches?

5.2 will be released momentarily, so it's locked.

1 Like
Terms of Service

Privacy Policy

Cookie Policy