Swift-clang testing on linux?

Development Compiler
1

Do the build bots test swift-llvm or swift-clang on Linux? I've seen the following failures for months with swift-clang:

llvm-lit: /home/dave/s/u/llvm/utils/lit/lit/llvm/config.py:337: note: using clang: /home/dave/s/u/t/bin/clang
FAIL: Clang :: Parser/pragma-attribute-context.cpp (7643 of 14002)
******************** TEST 'Clang :: Parser/pragma-attribute-context.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/dave/s/u/t/bin/clang -cc1 -internal-isystem /home/dave/s/u/t/lib/clang/8.0.0/include -nostdsysteminc -verify -std=c++11 /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp
: 'RUN: at line 2';   /home/dave/s/u/t/bin/clang -cc1 -internal-isystem /home/dave/s/u/t/lib/clang/8.0.0/include -nostdsysteminc -xobjective-c++ -verify -std=c++11 /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp
--
Exit Code: 1

Command Output (stderr):
--
error: 'warning' diagnostics expected but not seen: 
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 28: 'f' is only available on macOS 1000 or newer
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 29: 'g' is only available on macOS 1000 or newer
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 30: 'h' is only available on macOS 1000 or newer
error: 'note' diagnostics expected but not seen: 
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 13: marked
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 19: marked
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 23: marked
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 28: annotate 'a'
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 29: annotate 'b'
  File /home/dave/s/u/clang/test/Parser/pragma-attribute-context.cpp Line 30: annotate 'c'
9 errors generated.

--

********************
FAIL: Clang :: Refactor/Extract/extract-capture-self.m (7927 of 14002)
******************** TEST 'Clang :: Refactor/Extract/extract-capture-self.m' FAILED ********************
Script:
--
: 'RUN: at line 36';   clang-refactor-test perform -action extract -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:18:3-18:12 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:20:3-20:18 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:21:3-21:16 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:23:3-23:20 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:24:3-24:24 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:18:3-25:23 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:25:3-25:23 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m
: 'RUN: at line 46';   clang-refactor-test perform -action extract -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:39:3-39:27 -selected=/home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m:40:3-40:21 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK2 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m
: 'RUN: at line 54';   clang-refactor-test perform -action extract -selected=rhs-prop /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m
--
Exit Code: 2

Command Output (stderr):
--
error: inconsistent results in a single action range!
FileCheck error: '-' is empty.
FileCheck command line:  /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Extract/extract-capture-self.m

--

********************
FAIL: Clang :: Refactor/Rename/ObjCClass.m (7997 of 14002)
******************** TEST 'Clang :: Refactor/Rename/ObjCClass.m' FAILED ********************
Script:
--
: 'RUN: at line 7';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:1:8 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 8';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:4:12 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 9';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:2:8 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK2 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 17';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:11:17 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 18';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:12:3 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 19';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:21:20 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 30';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:14:7 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 31';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:23:3 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 32';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:24:9 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 33';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:25:14 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 34';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:26:18 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK3 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 36';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:12:7 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK4 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 37';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:26:3 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK4 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 47';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:41:12 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 48';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:44:17 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 50';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:41:16 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK5 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 51';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:44:21 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK5 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 61';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:55:12 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK6 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 62';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:58:17 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK6 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 64';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:58:21 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK7 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 74';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:68:12 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK6 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 75';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:71:17 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK6 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 94';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:81:7 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK8 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
: 'RUN: at line 95';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:89:3 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK8 /home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m
--
Exit Code: 1

Command Output (stderr):
--
/home/dave/s/u/clang/test/Refactor/Rename/ObjCClass.m:12:32: error: CHECK1: expected string not found in input
 I1 *interfaceIVar; // CHECK1: rename [[@LINE]]:3 -> [[@LINE]]:5
                               ^
<stdin>:4:1: note: scanning from here
rename 21:20 -> 21:22
^
<stdin>:4:1: note: with expression "@LINE" equal to "12"
rename 21:20 -> 21:22
^
<stdin>:4:1: note: with expression "@LINE" equal to "12"
rename 21:20 -> 21:22
^

--

********************
FAIL: Clang :: Refactor/Rename/ObjCImplementationTURequests.m (8037 of 14002)
******************** TEST 'Clang :: Refactor/Rename/ObjCImplementationTURequests.m' FAILED ********************
Script:
--
: 'RUN: at line 9';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m:2:7 -new-name=foo -implementation-tu="/home/dave/s/u/clang/test/Refactor/Rename/Inputs/ObjCImplementationTURequestsImplementation.m" -dump-symbols /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m
: 'RUN: at line 12';   not clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m:2:7 -new-name=foo -implementation-tu="/home/dave/s/u/clang/test/Refactor/Rename/MissingFile.m" -dump-symbols /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m 2>&1 | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK-ERR1 /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m
: 'RUN: at line 30';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m:16:7 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK-NO /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m
: 'RUN: at line 31';   clang-refactor-test rename-initiate -at=/home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m:25:7 -new-name=foo /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m | /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK-NO /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m
--
Exit Code: 2

Command Output (stderr):
--
error: could not rename symbol at the given location
FileCheck error: '-' is empty.
FileCheck command line:  /home/dave/s/u/t/bin/FileCheck --check-prefix=CHECK-NO /home/dave/s/u/clang/test/Refactor/Rename/ObjCImplementationTURequests.m

--

********************
FAIL: Clang :: SemaObjC/class-stub-attr.m (9956 of 14002)
******************** TEST 'Clang :: SemaObjC/class-stub-attr.m' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/dave/s/u/t/bin/clang -target x86_64-apple-darwin -fsyntax-only -Xclang -verify /home/dave/s/u/clang/test/SemaObjC/class-stub-attr.m
: 'RUN: at line 2';   /home/dave/s/u/t/bin/clang -target x86_64-apple-darwin -x objective-c++ -fsyntax-only -Xclang -verify /home/dave/s/u/clang/test/SemaObjC/class-stub-attr.m
--
Exit Code: 1

Command Output (stderr):
--
error: 'warning' diagnostics seen but not expected: 
  (frontend): include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead
1 error generated.

--

********************
FAIL: Clang :: SemaObjC/class-stub-attr-unsupported.m (9961 of 14002)
******************** TEST 'Clang :: SemaObjC/class-stub-attr-unsupported.m' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/dave/s/u/t/bin/clang -target i386-apple-darwin -fsyntax-only -Xclang -verify /home/dave/s/u/clang/test/SemaObjC/class-stub-attr-unsupported.m
: 'RUN: at line 2';   /home/dave/s/u/t/bin/clang -target i386-apple-darwin -x objective-c++ -fsyntax-only -Xclang -verify /home/dave/s/u/clang/test/SemaObjC/class-stub-attr-unsupported.m
--
Exit Code: 1

Command Output (stderr):
--
error: 'warning' diagnostics seen but not expected: 
  (frontend): include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead
1 error generated.

--

********************
Testing Time: 23.49s
********************
Failing Tests (6):
    Clang :: Parser/pragma-attribute-context.cpp
    Clang :: Refactor/Extract/extract-capture-self.m
    Clang :: Refactor/Rename/ObjCClass.m
    Clang :: Refactor/Rename/ObjCImplementationTURequests.m
    Clang :: SemaObjC/class-stub-attr-unsupported.m
    Clang :: SemaObjC/class-stub-attr.m

  Expected Passes    : 13747
  Expected Failures  : 26
  Unsupported Tests  : 223
  Unexpected Failures: 6
FAILED: tools/clang/test/CMakeFiles/check-clang
3

There is an internal bot that runs this. I have been pushing to get a public bot for this purpose. There is a bandwidth issue (in terms of getting up bots).

@mishal_shah

4

Interesting. I'm not surprised that the "stable" swift-llvm and swift-clang branches might get light testing. Does the internal bot run the swift-llvm and swift-clang tests on just Mac OS? Or other platforms? Separably, how open are people to "alt CI" bots? In particular, it would be nice to get more regular unified build testing. I recently updated my unified build script to let @compnerd watch it since it tends to have similar problems to Windows.

5

The tests were actually broken on Linux for a long time. They have been fixed recentlyish (but only on upstream-with-swift). If there is interest, porting the changes to stable should be easy. The changes were largely trivial and had to do with assumptions of how the tests were being run.

6

Ya, back porting the fixes would be nice. It’d fix a hole in my testing. Is there a GitHub PR I can cherry-pick from?

7

As you wish - backport changes to enable the test suite to pass on Linux by compnerd · Pull Request #308 · apple/swift-clang · GitHub

One minor request: please let me know how it turns out and if you are still seeing any failures as I am not after that set of patches (granted that I am working with upstream-with-swift).