Running specific stdlib test?

Development Standard Library
1

Updated version of this old thread: <run a specific test file>

I'm playing with some changes to swift/stdlib/public/core/Codable.swift, and adding tests for the changes in swift/test/stdlib/CodableTests.swift. I've managed to successfully run the tests with:

./swift/utils/build-script --preset=stdlib_DA_standalone,build,test toolchain_path=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-17-a.xctoolchain/usr/bin

However, this takes a long time and there is a ton of output I don't care about. Is there a more concise way to run just the test file I'm working on? I tried a few things with ./swift/utils/run-test, such as:

./swift/utils/run-test swift/test/stdlib/CodableTests.swift toolchain_path=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-17-a.xctoolchain/usr/bin --build-dir=./build/stdlib_DA_standalone/swift-macosx-arm64/

But then I got errors like:

/Users/zev/Projects/workspace/swift/test/stdlib/CodableTests.swift:24:8: error: no such module 'StdlibUnittest'
  22 | class TestCodableSuper : XCTestCase { }
  23 | #else
  24 | import StdlibUnittest
     |        `- error: no such module 'StdlibUnittest'
  25 | class TestCodableSuper { }
  26 | #endif

--
2

Have you taken a look at running tests? --filter="test-i-want-to-run" may be what you are looking for.

3

Thanks. I had not seen that. I'm still a little confused, but I feel like I'm getting closer.

Confusion #1: is there a workflow for doing an incremental build and then running just the test I want? or do I have to keep bounding back and forth between a run-build command and a run-test command that uses --filter?

(More confusions to follow, but I want to do this step by step to make sure I'm on the right track.)

4

Confusion #1: is there a workflow for doing an incremental build and then running just the test I want? or do I have to keep bounding back and forth between a run-build command and a run-test command that uses --filter ?

Just to give an example, not saying this is the best way to do things, when I am working on something I always:

  1. Create a small targeted test case of my own mytest.swift.
  2. Make some changes to the codebase.
  3. Build incrementally with ninja.
  4. Run mytest.swift (more detail on this to follow).

To be clear I only do step 1 once but repeat 2-4 many times, maybe adding additional cases to mytest.swift as I go. To actually run the test, if I am working on diagnostics I can just do:

../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/bin/swift-frontend \
-typecheck -verify mytest.swift

If you just want to run CodableTests.swift I would do it this way (assuming you are starting from your /swift directory):

../llvm-project/llvm/utils/lit/lit.py -s -vv \
../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/test-macosx-arm64/CodableTests.swift

If you wanted to do 3 and 4 in one step, you could make a script:

#!/bin/bash

# Exit on error
set -e

# Run ninja to build stdlib
ninja -C ../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64 stdlib

# Run CodableTests.swift
../llvm-project/llvm/utils/lit/lit.py -s -vv ../build/Ninja-RelWithDebInfoAssert/swift-macosx-arm64/test-macosx-arm64/stdlib/CodableTests.swift

And if you were working on something else you could change the script to build a different module/run a different test.

5

Thanks! I’ll try this next time I get a moment to work on it.

Also, is it just me, or does it seem like that workflow is common enough that it should be build into the scripts provided by the repository, in the interest of making it easier for outside/casual contributors?

1 Like
6

I’ve found the Getting Started guide to be quite good, but there’s always room for improvement. That said, a script that fits every contributor’s needs might be tricky since workflows can vary quite a bit.

7

OK! I'm back with a more focused question. I'm trying to spend as little time on the build script stuff as possible so I can actually work on the changes I want to make, at least to get a proof-of-concept put together. To that end, I haven't gone deep on the various guides, but I've skimmed the relevant sections as best I can, and I think I'm at least vaguely moving in the right direction, but please let me know if something I'm doing doesn't make sense!

Here's the script I've put together:

#!/bin/bash

set -e

# build
./swift/utils/build-script --preset=stdlib_DA_standalone,build toolchain_path=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-17-a.xctoolchain/usr/bin

# test
llvm-project/llvm/utils/lit/lit.py -a build/stdlib_DA_standalone/swift-macosx-arm64/test-macosx-arm64/ --filter="CodableTests.swift"

Once I've run it once to get a build in place, here's the output of a subsequent run: Output of zevtest.sh.txt · GitHub

The relevant part seems to be the error I mentioned before:

/Users/zev/Projects/workspace/swift/test/stdlib/CodableTests.swift:24:8: error: no such module 'StdlibUnittest'
  22 | class TestCodableSuper : XCTestCase { }
  23 | #else
  24 | import StdlibUnittest
     |        `- error: no such module 'StdlibUnittest'
  25 | class TestCodableSuper { }
  26 | #endif

I am able to run the tests if I skip the second line in my script, and just add ,test in my `build-script invocation:

./swift/utils/build-script --preset=stdlib_DA_standalone,build,test toolchain_path=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-17-a.xctoolchain/usr/bin

However, that performs a full test, rather than just running CodableTests.swift. I do just want to run CodableTests.swift, rather than writing my own new test file, I think. What am I missing?

8

if you try running the tests via run-test does that help? e.g. something like:

# use `--build=skip` if you don't want to build first
utils/run-test --lit ../llvm-project/llvm/utils/lit/lit.py \
    <build-dir> \
    --build=true \
    --filter="<tests>"
1 Like