Requesting help with stdlib testing

[Morten_Bek_Ditlevsen]

Hi all,
I am trying to make a change to the stdlib and test it as well.
But I am stuck with the whole lit.py system and getting output from my test.

I am following the guide at:

Swift.org

Using the latest toolchain to build the stdlib and start the tests.
I am running the suggested command:

./swift/utils/build-script --preset=$PRESET_NAME toolchain_path=$TOOLCHAIN_BIN_DIR

Where my preset is a debug build that includes tests.

It works perfectly, but I can't find the output of the tests - other than the 'FAIL: ...' message. I would like to know what actually failed...
Also I would like to only run one single test and have tried setting the environment variable LIT_FILTER=stdlib/CodableTests.swift, but the build script command above still runs all stdlib tests.

I have also tried running lit.py manually as such:

llvm-project/llvm/utils/lit/lit.py -sv build/stdlib_DA_standalone/swift-macosx-x86_64/test-macosx-x86_64/stdlib

But then all the tests fail to find the StdlibUnittest library.

I am hoping that someone could point me in the correct direction for where to look for the test output. :-)

[nnnnnnnn (Nate Cook)]

Have you tried using ./swift/utils/run-test? You can pass the path to one or more tests to run only those tests.

[benrimmington]

I've never seen that QuickStart contribution page before.

Do you know where it's linked from, and who's responsible for updating it?

It contains some outdated information, such as:

• the macOS 10.14.4 and Xcode 11 beta 1 requirements,
• the 2019-06-06 snapshot,
• the CMake and Sccache versions.

Should the page link to the new How-To Guides instead?

[Morten_Bek_Ditlevsen]

Hi @nnnnnnnn and @benrimmington,

Thanks for your replies!
The ./swift/utils/run-test gives me the same issue as running lit.py manually. Namely that the imported library StdlibUnitTest can't be found.
I am running it as follows:

./swift/utils/run-test swift/test/stdlib/ --build-dir=./build/stdlib_DA_standalone/swift-macosx-x86_64/ --build=skip

I'm sure it can be fixed by adding environmen or parameters for pointing to the library:

find . -name "StdlibUnittest.*"
./swift/stdlib/private/StdlibUnittest/StdlibUnittest.swift
./build/stdlib_DA_standalone/swift-macosx-x86_64/lib/swift/macosx/StdlibUnittest.swiftmodule
./build/stdlib_DA_standalone/swift-macosx-x86_64/lib/swift/macosx/StdlibUnittest.swiftmodule/StdlibUnittest.opt.bitstream
./build/stdlib_DA_standalone/swift-macosx-x86_64/stdlib/private/StdlibUnittest/OSX/x86_64/StdlibUnittest.o
./build/stdlib_DA_standalone/swift-macosx-x86_64/stdlib/private/StdlibUnittest/OSX/x86_64/StdlibUnittest.opt.bitstream

I'm just not certain how to specify the path.

@benrimmington: No, I don't know where it is linked from - I found it by googling the specific thing that I was trying to achieve: Building (and testing) stdlib using the latest development toolchain so that I don't have to compile the entire toolchain again. I can't find a similar guide in the documentation in the swift repo.
I am of course using the latest development toolchain and not the one referred to in the guide. :-) And besides from me not being able to find the test output - the guide did work out well for me.

Sincerely,
/morten

[Max_Desiatov]

@benrimmington

I've never seen that QuickStart contribution page before.

Do you know where it's linked from, and who's responsible for updating it?

It contains some outdated information, such as:

• the macOS 10.14.4 and Xcode 11 beta 1 requirements,
• the 2019-06-06 snapshot,
• the CMake and Sccache versions.

Should the page link to the new How-To Guides instead?

@Varun_Gandhi you previously worked on How-To Guides, do you know if it would be possible to update the page on swift.org as well?

[Buttaface]

The full test output is dumped to an XML file in the build directory, something like build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/lit-tests.xml, which is what I get when running the compiler testsuite natively on Android. As for running a single stdlib test, you could try modifying the filter line in the preset to be more specific.

[brentdax (Becca Royal-Gordon)]

@benrimmington

I've never seen that QuickStart contribution page before.

I believe it was prepared for last year’s Try Swift San Jose event. @Michael_Gottesman prepared a new standard library development workflow for that event; I’m not sure if things have changed since. Michael?

[Varun_Gandhi]

Yes, we have an Apple-internal process for making changes to the website. Fixing this particular issue has been on my Todo list but I've been occupied with some other work so I haven't gotten around to it yet...