GSoC 2025: Progress & Feedback Request for an Improved Swift Testing Console Output

[tienquocbui (Kelvin Bui)]

Hello everyone,

My name is Kelvin Bui, and I'm excited to be participating in Google Summer of Code 2025 with Swift.org! My project, mentored by Stuart (@smontgomery), is the Improved Console Output for Swift Testing.

The main goal is to make the testing experience in Swift clearer, more useful for diagnosing failures, and more productive for all developers. After about a month of exploring different designs and gathering feedback, I'd like to share our current progress and ask for your valuable input on a few key design questions.

Motivation: Problems We're Trying to Solve

Based on community feedback and initial research, we've identified several key areas for improvement in the current console output:

• Difficulty understanding test relationships: Without a clear visual structure, it can be hard to see how tests are organized into suites, especially in projects with many tests.
• Difficulty locating failures: In large test runs, failures can be lost in a long sea of success messages, making it hard to quickly assess the health of a run, especially in CI logs.
• Lack of "liveness" and progress indication: The current output is a static log, which doesn't provide a real-time sense of progress or how much work is remaining.

The Core Challenge: "Liveness" vs. a structured hierarchy

A central challenge we identified is the trade-off between providing a "live," responsive output and a perfectly structured hierarchical view. A live, indented hierarchy requires significant buffering of test results, which can cause a noticeable delay before anything is printed. This can make the test run feel slow or "stuck," especially when tests are fast and run in parallel.

Proposed Solution: A Two-Phase Architecture

To solve this, we are proposing a new two-phase architecture:

1. Live Phase (while tests are running): This phase focuses on immediate, real-time feedback. The plan is to show a live-updating progress bar and print out notifications for issues (especially failures) as soon as they occur.
2. Summary Phase (after tests have finished): Once the run is complete, we will print a comprehensive, detailed hierarchical summary. This summary will contain the full, structured tree of all suites and tests, along with their results and detailed failure information.

Demo & Mockups

We have focused on two distinct usage scenarios: users running from an interactive terminal, called the "Live Mode", and tests running in an automated system, the "CI Mode".

I've put together a short video showing an early implementation of the hierarchical output. It demonstrates how tests, suites, and issues can be rendered in a nested structure.

Click the image below to watch the demo video on Google Drive

To show the full vision, here are the latest mockups for both the Live Mode and CI Mode .

Request for Feedback

This is a work in progress, and we'd love to get the community's thoughts on a few key design decisions. Please note that these ideas have been presented to the Swift Testing workgroup but have not been formally endorsed yet.

1. Presentation of failures: integrated in the hierarchy vs separate section? In our new model, the hierarchy is printed at the end. This allows us to merge the detailed failure information (like the evaluated #expect expression and a trimmed stack trace) directly into the tree as sub-nodes. The alternative is to keep the hierarchy succinct and have a separate FAILED TEST DETAILS section at the very bottom.

• Question: Which approach do you prefer? A single, all-in-one hierarchical summary, or a cleaner hierarchy followed by a separate, detailed failure report section?

2. Use of symbols vs text for status indicators
We are considering two main styles for indicators like "pass" or "fail":

• Unicode/SF Symbols (e.g., ✓, ✗): SF Symbols look great and provide a rich experience, but they are specific to Apple platforms, leading to an inconsistent look on Linux and Windows.
• Text Labels (e.g., [PASS], [FAIL]): These are 100% cross-platform compatible and are easily searchable in CI logs.
• Question: What should the default experience be? Should we prioritize a consistent cross-platform look with text labels, and make symbols an opt-in feature?

Next Steps

Our immediate priority is to finalize the implementation of the hierarchical summary. After that, we will focus on building out the live progress bar and the final summary section with performance insights.

Thank you for taking the time to read this update. I'm excited to hear your thoughts and feedback!

Best,
Kelvin

[grynspan (Jonathan Grynspan)]

@tienquocbui

Live Phase (while tests are running): This phase focuses on immediate, real-time feedback. The plan is to show a live-updating progress bar and print out notifications for issues (especially failures) as soon as they occur.

As I think we've discussed previously, a live progress bar is not possible to implement reliably and portably using the set of standard ANSI escape codes that are universally supported across terminal applications. So while a progress bar is certainly a nice bonus, we will also need to support a fallback mechanism for progress reporting that doesn't rely on rewriting over past output.

@tienquocbui

Unicode/SF Symbols (e.g., ✓, ✗): SF Symbols look great and provide a rich experience, but they are specific to Apple platforms, leading to an inconsistent look on Linux and Windows.

The set of available Unicode characters is highly dependent on the font your terminal application uses and how it handles missing characters. If you take a look at Swift Testing's current implementation, we emit a completely different set of characters on Windows (where the ✓ and ✗ characters don't render correctly by default.)

So the look of these symbols is going to be inconsistent across platforms anyway. If you want to focus on a consistent cross-platform appearance, you should assume only ASCII characters are available and extrapolate your design from there.

Edit: To clarify this point:

@tienquocbui

A central challenge we identified is the trade-off between providing a "live," responsive output and a perfectly structured hierarchical view.

stdout and stderr, being process-wide streams, are incompatible with structured output. So to implement live output we would need to also implement a harness process that handles events from the test process and writes them to the console, but which does not forward output that a test writes to those streams.

[hassila (Joakim Hassila)]

Looks promising!

Just to shortly reply to your concrete questions:

• Question: Which approach do you prefer? A single, all-in-one hierarchical summary, or a cleaner hierarchy followed by a separate, detailed failure report section?

I’d prefer a detailed failure report section.

Should we prioritize a consistent cross-platform look with text labels, and make symbols an opt-in feature?

Using symbols are nice, but then I would spend the time to also make sure it uses the facilities on windows and Linux (e.g. sf symbols on macOS but Unicode on the other platforms, only falling back on text if not possible). Trying to keep all platforms as nice as possible.

[hassila (Joakim Hassila)]

So one more thought, there are 6 kinds of symbols here - must admit I can’t be certain what all of them means even if I can guess. Also not clear what is summarized on each of the two rows?

Perhaps a short table summary with both symbol and text would be clearer?

[maartene (Maarten Engels)]

Hi Kelvin,

Thank you for your awesome work on this project

@tienquocbui

Question: Which approach do you prefer? A single, all-in-one hierarchical summary, or a cleaner hierarchy followed by a separate, detailed failure report section?

My personal preference is to have a specific failed test details section. Because the failed tests are for me the most interesting. Especially when running in CI.

@tienquocbui

Question: What should the default experience be? Should we prioritize a consistent cross-platform look with text labels, and make symbols an opt-in feature?

I would prefer the usage of symbols where possible/available. And the [PASS]/[FAIL] for where they are unavailable. And a command line argument to 'force' the [PASS]/[FAIL] style. I.e. my CI might support symbols, but when scanning/searching logs, searching for a string is much easier.

With regards to the mockup of the hierarchy, I have a question:
How will the hierarchy look when the suite/test names span more than the width of your terminal window? I.e. how will you wrap around?

[bkhouri (Bassam (Sam) Khouri)]

Hello,

I am really looking forward to seeing the outcome of this work. Thank you.

Here are some of my thoughts:

• I would like the flexibility of specify the level of detail to provide, both during live execution and summary report. e.g.:

   -r chars              Show extra test summary info as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except
                         passed (p/P), or (A)ll. (w)arnings are enabled by default (see --disable-warnings), 'N' can be used to reset the list. (default: 'fE').
  

• Use of fully qualified test ID and absolute path when referencing a test (`swift test` output does not differentiate tests with the same name from different suites · Issue #841 · swiftlang/swift-testing · GitHub)
• During Summary report:
  • I care more about test failures and less so about passed tests.
  • List passed tests before failed tests
  • There should be a section where all failed tests, in each suite, is collected together
  • Failed tests should contain test output and expectation failure messages
  • Collect failed tests to a log file, but this may be outside the scope of the current project.
  • Usage of symbols should be clear and not ambiguous. If symbols can be ambiguous and there is no way to clarify them, use text instead. e.g.: PASS, FAIL, XFAIL, SKIPPED, etc..

Cheers,

Sam

[tienquocbui (Kelvin Bui)]

Hi Jonathan, thank you so much for the detailed and insightful feedback! This is incredibly helpful.

Thank you for the clarification on Unicode character portability, especially the example of how swift-testing currently handles Windows. That's great context. This strengthens the case for making simple ASCII labels (like [PASS], [FAIL]) the default experience to ensure true cross-platform consistency. The Unicode/SF Symbols can then remain an explicit opt-in for users who know their environment supports them.

I also really appreciate the clarification about the necessity of a harness process for reliable live output. It makes perfect sense that to truly protect the structured output from being corrupted by interleaved stdout/stderr from parallel tests, the rendering needs to happen in a separate, supervising process. This is a key architectural constraint to keep in mind. For this GSoC project, we will focus on building the reporter's logic and UI in a way that it can be easily integrated into such a harness in the future.
Thanks again for taking the time to share your expertise!

[tienquocbui (Kelvin Bui)]

That's a very interesting perspective on the symbol strategy. I like the idea of striving for the "best possible native experience" on each platform (SF Symbols on macOS, Unicode on Linux, etc.) rather than defaulting to the most basic option for the sake of identical consistency. This is really valuable input as we weigh the trade-offs between a completely unified look and a richer, platform-adapted one.

We'll definitely take these points into account as we refine the design. Thanks again for your thoughts!

[tienquocbui (Kelvin Bui)]

To answer your immediate question about what the symbols mean, I've been working from this "Symbol Showcase" table which acts as a legend. I'm attaching it here for context:

However, you are absolutely right. Your suggestion to combine the symbol with text is the best path forward. I think a great solution would be to create a list where each line explicitly pairs the icon with its label, this way, the summary is still compact but completely unambiguous. It helps users learn the meaning of the icons over time while always being clear. I will discuss this further with my mentor. Thanks a lot!

[tienquocbui (Kelvin Bui)]

Hi Maarten, thank you for the feedback!

@maartene

With regards to the mockup of the hierarchy, I have a question:
How will the hierarchy look when the suite/test names span more than the width of your terminal window? I.e. how will you wrap around?

It's a critical edge case to get right. My plan is to implement intelligent, indented wrapping. When a name is too long for the terminal width, the first line will be printed normally. Any subsequent wrapped lines will be prefixed with the correct indentation and the │ (vertical line) character to maintain the tree's visual structure. The duration would remain on the final wrapped line.

My mentor, Stuart, had also pointed out that we can query the terminal's width using tools like stty size , which is the foundation for this kind of responsive layout.

It would look something like this:

 │
 ├─ ✓ ThisIsAVeryVeryLongAndDescriptiveTestNameThatNeeds
 │    ToWrapAroundToTheNextLine                             (0.12s)
 │

This is really helpful for thinking through these important details. Thanks again for your input!

[tienquocbui (Kelvin Bui)]

Hi Sam, I really like your suggestion for providing fine-grained control over the summary's level of detail, like the -r chars example you provided. This is a more powerful version of the "hide successes" option we've been considering, and it fits perfectly with the modular, configurable architecture we're aiming for. And yeah we will definitely explore this.

@bkhouri

Use of fully qualified test ID and absolute path when referencing a test (swift test output does not differentiate tests with the same name from different suites · Issue #841 · swiftlang/swift-testing · GitHub)

I also strongly agree with your points about the summary report - using fully qualified test IDs to solve the ambiguity from Issue #841 is a primary goal of the new hierarchical design. Your thoughts on collecting all failures together and the option to log them to a file are also great points that we'll keep in mind as we design the final summary section. This is fantastic input and will be very helpful as we continue to refine the design. Thanks again!

[bkhouri (Bassam (Sam) Khouri)]

@tienquocbui

I also really appreciate the clarification about the necessity of a harness process for reliable live output. It makes perfect sense that to truly protect the structured output from being corrupted by interleaved stdout/stderr from parallel tests, the rendering needs to happen in a separate, supervising process. This is a key architectural constraint to keep in mind. For this GSoC project, we will focus on building the reporter's logic and UI in a way that it can be easily integrated into such a harness in the future.

From a design perspective, I think SwiftPM should handle the logic of emitting the text to the console by inspecting Swift Testing's the event stream, assuming the user provided, but hidden, --event-stream-output-path command line argument to swift test is respected.

To aid with the real-time processing, SwiftPM has a progress animators, though I haven't used them and not sure if the current ones will be helpful.

@rauhul also posted a draft PR (WIP: New and user selectable progress bars by rauhul · Pull Request #7897 · swiftlang/swift-package-manager · GitHub) that changes swift build progress indicator, which can be used as reference/inspiration.

[grynspan (Jonathan Grynspan)]

@tienquocbui

I like the idea of striving for the "best possible native experience" on each platform (SF Symbols on macOS, Unicode on Linux, etc.) rather than defaulting to the most basic option for the sake of identical consistency.

We can also find a middle ground, I think, where we use mostly ASCII but construct symbols from it. For example:

[!]: issue recorded
[?]: warning
[√]: pass
[x]: fail

etc.

Some artistry would be needed to pick good characters, of course.

[bkhouri (Bassam (Sam) Khouri)]

Out of curiosity, are we also considering addressing `swift test` output is buffered (rather than streamed) for swift-testing tests on Windows · Issue #8928 · swiftlang/swift-package-manager · GitHub as port of this GSoC project?

[smontgomery (Stuart Montgomery)]

I think that would be handled separately, as it's more about the layering between Swift package manager and Swift Testing, and not about the content of the output generated.

[Hacksaw]

I'd love it if the runtimes were visually tied to the entry they are from. The giant gulf makes it hard to line up what belongs to what. A faint leader line could be one solution, or putting the time on one side or the other of the check mark.

[bkhouri (Bassam (Sam) Khouri)]

I understand this is a work in progress, but it there a way we can preview the current status of the console output?

[smontgomery (Stuart Montgomery)]

Hey @bkhouri, I've been advising w/Kelvin on this project and wanted to comment on its anticipated rollout in experimental form. He is just now beginning to post PRs with these changes, and they will need to be reviewed by code owners of the repo which will take a bit of time. Initially, all newly-added code for this feature will be experimental, not enabled by default. Once this series of PRs land, the feature will be in main development snapshots on Swift.org and at that point it should become possible for anyone interested to try out the feature by running swift test with a specific opt-in environment variable set.

This process may take several weeks at least, but we're making progress. I expect Kelvin will post again on the forums when it's ready to try out.