Hello everyone! My name is Kelvin Bui, and I'm excited to share my GSoC 2025 project, where I worked on improving the console output for the Swift Testing framework with my mentor, @smontgomery.
1. Overview
This summer, as part of Google Summer of Code 2025, I had the incredible opportunity to work on improving the console output for the Swift Testing framework. The primary goal was to transform the existing static log into a modern, clear, and highly useful tool for developers. The main achievement of this project is the design and implementation of a new, two-phase console reporter, with a primary focus on delivering a comprehensive Hierarchical Summary and Detailed Failure Report. It also leverages serialization to allow emitting output from a separate supervisor process, in alignment with future project directions.
2. Problem & Motivation
The default console output for swift test was functional but presented several challenges for developers working on large and complex projects:
- Difficulty Locating Failures: In large test suites, critical failure messages were often lost in a long stream of success outputs, making it hard to quickly assess the health of a test run.
- Lack of Structure: The flat log format made it difficult to understand the relationship between tests and their parent suites, especially in projects with many nested modules.
Our vision was to solve these problems by creating an advanced console reporter that is both informative and intuitive.
3. The Technical Journey & Implementation
The project evolved through several distinct phases, adapting to technical challenges and expert feedback.
flowchart LR
%% Phase 1: Research, Design & Prototyping
subgraph Phase1["Phase 1: Research"]
A["Kickoff"] --> B{"Research<br/>Brainstorm"}
B --> C["Synthesize<br/>Ideas"]
C --> D["Mockups"]
D --> E{"Initial<br/>Direction"}
end
%% Phase 2: Architecture & Community Feedback
subgraph Phase2["Phase 2: Architecture"]
E --> F["Two-Phase<br/>Plan"]
F --> G["Skeleton<br/>framework"]
G --> H["Forums<br/>Post"]
H --> I{"Community<br/>Feedback"}
end
%% Phase 3: Core Implementation & Refactoring
subgraph Phase3["Phase 3: Implementation"]
I --> J["Serialization<br/>Pivot"]
J --> K["Hierarchical<br/>Summary"]
K --> L["Cross-Platform<br/>Support"]
L --> M1["Graph<br/>Refactor"]
end
%% Phase 4: GSoC Final Submission
subgraph Phase4["Phase 4: Submission"]
M1 --> M["Final<br/>Testing"]
M --> N["GSoC<br/>Report"]
N --> O["Submit<br/>Work"]
O --> P["Complete"]
end
%% Phase 5: Future Work (Post-GSoC Commitment)
subgraph Phase5["Phase 5: Future Work"]
P --> Q["Continue<br/>Contrib."]
Q --> R["Live Phase"]
R --> R1["Progress Bar"]
R --> R2["Live Reporting"]
R --> R3["Test Heartbeat"]
Q --> S["More Config<br/>Options"]
S --> T["Harness<br/>Integration"]
T --> U["Default in<br/>Future Release"]
end
%% Key Achievements Sidebar
subgraph Achievements["Achievements"]
Achievement1["ABI<br/>Arch"]
Achievement2["Tree<br/>UI"]
Achievement3["Cross<br/>Platform"]
Achievement4["Graph<br/>Integration"]
Achievement5["Production<br/>Ready"]
end
%% Connect achievements to relevant phases
J -.-> Achievement1
K -.-> Achievement2
L -.-> Achievement3
M1 -.-> Achievement4
M -.-> Achievement5
%% Styling
classDef phaseBox fill:#f7fafc,stroke:#2b6cb0,stroke-width:2px,color:#1a365d
classDef futureBox fill:#fffaf0,stroke:#d69e2e,stroke-width:2px,color:#744210
classDef achievementBox fill:#f0fff4,stroke:#38a169,stroke-width:2px,color:#22543d
classDef startNode fill:#e6fffa,stroke:#319795,stroke-width:2px,color:#234e52
classDef decisionNode fill:#fef5e7,stroke:#d69e2e,stroke-width:2px,color:#744210
classDef processNode fill:#edf2f7,stroke:#4299e1,stroke-width:2px,color:#2a69ac
classDef milestoneNode fill:#fed7d7,stroke:#e53e3e,stroke-width:2px,color:#742a2a
classDef achievementNode fill:#c6f6d5,stroke:#48bb78,stroke-width:2px,color:#22543d
classDef futureNode fill:#fef5e7,stroke:#ed8936,stroke-width:2px,color:#7b341e
%% Apply node styles
class Phase1,Phase2,Phase3,Phase4 phaseBox
class Phase5 futureBox
class Achievements achievementBox
class A,P,U startNode
class B,E,I,T decisionNode
class C,D,F,G,H,J,K,L,M1,M,N,O milestoneNode
class Q,S processNode
class R,R1,R2,R3 futureNode
class Achievement1,Achievement2,Achievement3,Achievement4,Achievement5 achievementNode
Phase 1: Design, Research, and Prototyping
The project began not with code, but with design. This initial phase focused on researching existing pain points by reviewing community feedback on the Swift Forums and open GitHub issues. Through a brainstorming process, several versions of mockups were created to visualize a better console experience. This led to the core architectural proposal of a Two-Phase Reporter ("Live Phase" for real-time updates and "Summary Phase" for a final, detailed report), which was refined through early discussions with my mentor.
Phase 2: Foundational Implementation & Community Feedback
With a clear design vision, the next step was to build a foundational "skeleton" for the new AdvancedConsoleOutputRecorder. This minimal implementation was created to establish the entry points and the environment variable for enabling the experimental feature. Crucially, at this stage, we shared the mockups and the architectural plan with the broader community via a detailed Swift Forums post. The feedback received was invaluable and directly shaped the project's direction.
I also presented these mockups and the skeleton implementation to the Swift Testing workgroup; the live feedback received there directly influenced several subsequent design decisions.
Phase 3: Major Architectural Refactor & Core Feature Implementation
Responding to expert feedback from core team members, the project underwent a significant architectural pivot. The reporter was refactored to use a serialization-based event model (EncodedEvent) instead of directly handling in-process objects. This change prepares the reporter for future out-of-process execution, where events can be serialized, transmitted, and processed by a separate harness process. With this robust foundation in place, I then implemented the main user-facing features: the Hierarchical Summary and the Detailed Failure Report.
Phase 4: Cross-Platform Polish & Finalization
To ensure the reporter works reliably for all users, the final phase focused on cross-platform compatibility. This included implementing an ASCII fallback for the tree-drawing characters to ensure the UI never "breaks" on terminals with limited Unicode font support (especially on Windows and minimal Linux environments). This phase also involved intensive testing and bug fixing to stabilize the features for the final submission.
4. Key Technical Achievements
Hierarchical Test Display:
- A clear, indented tree structure that visualizes the relationship between suites and tests.
- An "issues as sub-nodes" model for displaying rich failure context directly within the hierarchy.
- An "out-dented" suite summary line that cleanly concludes each suite's output.
Serialization-Based Architecture:
- A future-proof design that processes
ABI.EncodedEventobjects, preparing the reporter for an out-of-process harness architecture. - Logic to build an in-memory representation of the test plan by consuming initial test discovery events and then looking up test details by ID for subsequent events.
Rich Failure Reporting:
- A dedicated
FAILED TEST DETAILSsection that provides comprehensive information for each failure, including the fully qualified test name, a detailed error message, and the exact source location (file:line).
Performance & Concurrency:
- A thread-safe data model that uses a single lock per event to efficiently and safely collect results from parallel test runs, following direct guidance from my mentor.
Cross-Platform Compatibility:
- An intelligent ASCII fallback mechanism for tree-drawing characters, ensuring the UI is robust across all terminal environments.
flowchart LR
%% Define styles
classDef framework fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#0d47a1
classDef recorder fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px,color:#4a148c
classDef data fill:#e8f5e8,stroke:#388e3c,stroke-width:2px,color:#1b5e20
classDef output fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#e65100
subgraph SwiftTesting["Swift Testing Framework"]
TestRunner["Test Runner"]
EventStream["Event Stream<br/>(ABI.EncodedEvent)"]
HumanReadable["HumanReadableOutputRecorder<br/>(Message Generation)"]
end
subgraph AdvancedRecorder["AdvancedConsoleOutputRecorder"]
EventProcessor["Event Processor<br/>(_processABIEvent)"]
HierarchyBuilder["Hierarchy Builder<br/>(_buildTestHierarchy)"]
MessageCapture["Message Capture<br/>(HumanReadable.record)"]
TreeRenderer["Tree Renderer<br/>(_renderHierarchyNode)"]
FailureFormatter["Failure Details Formatter<br/>(_formatDetailedIssueDescription)"]
SourceReader["Source Code Reader<br/>(_getSourceCodeContext)"]
end
subgraph DataStructures["Data Structures"]
GraphTree["Graph<String, _HierarchyNode?><br/>(testTree)"]
TestDataDict["[String: _TestData]<br/>(testData + issueMessages)"]
LockedContext["Locked<_Context><br/>(Thread-safe storage)"]
end
subgraph OutputSections["Output Sections"]
HierarchySection["HIERARCHICAL<br/>TEST RESULTS<br/>(Quick Overview)"]
DetailsSection["FAILED TEST DETAILS<br/>(Comprehensive Analysis)"]
end
subgraph OutputTargets["Output Targets"]
UnicodeTerminal["Unicode Terminal"]
ASCIITerminal["ASCII Terminal"]
end
%% Main flow
TestRunner -->|generates| EventStream
EventStream -->|processes| EventProcessor
HumanReadable -->|generates detailed<br/>messages| MessageCapture
EventProcessor -->|builds hierarchy| HierarchyBuilder
EventProcessor -->|captures messages| MessageCapture
HierarchyBuilder -->|stores in| GraphTree
MessageCapture -->|stores in| TestDataDict
GraphTree -->|stored in| LockedContext
TestDataDict -->|stored in| LockedContext
TreeRenderer -->|reads from| LockedContext
FailureFormatter -->|reads from| LockedContext
SourceReader -->|reads files for| FailureFormatter
TreeRenderer -->|renders| HierarchySection
FailureFormatter -->|renders| DetailsSection
HierarchySection -->|outputs to| UnicodeTerminal
HierarchySection -->|outputs to| ASCIITerminal
DetailsSection -->|outputs to| UnicodeTerminal
DetailsSection -->|outputs to| ASCIITerminal
%% Apply styles
class TestRunner,EventStream,HumanReadable framework
class EventProcessor,HierarchyBuilder,MessageCapture,TreeRenderer,FailureFormatter,SourceReader recorder
class GraphTree,TestDataDict,LockedContext data
class HierarchySection,DetailsSection,UnicodeTerminal,ASCIITerminal output
This diagram shows the complete architecture of the
AdvancedConsoleOutputRecorderwith all the sophisticated features we've implemented, including the dual-section output approach, source code context reading, and platform-specific rendering.
5. Visuals: Before & After
The Advanced Console Output Recorder fundamentally transforms how developers interact with test results. By introducing a visual hierarchy and detailed contextual information, it elevates the diagnostic experience from a flat, hard-to-parse log into a structured, scannable report.
Current Console Output:
Prior to this project, the default
swift testoutput presented a linear, undifferentiated stream of events. While functional, this format lacked visual cues, making it challenging to quickly identify test relationships, pinpoint failures within large suites, or understand the overall structure of a test run. The absence of clear demarcation between test suites and individual tests often led to slower debugging cycles and increased cognitive load for developers.
New Hierarchical Summary:
The new recorder introduces a rich, hierarchical display that immediately brings order and clarity to test output. Using Unicode box-drawing characters (with ASCII fallback), it clearly visualizes the nested structure of test modules, suites, and individual tests. This not only makes the test results significantly easier to read and navigate but also provides crucial context around failed tests, displaying detailed issue descriptions directly within their respective hierarchical nodes. This transformation dramatically improves the developer experience by offering instant insights into test organization and failure points.
6. Challenges & Lessons Learned
Technical lessons (depth):
- Designing for serialization & ABI: Refactoring to consume ABI.EncodedEvent taught me the value of decoupling - moving from in-process object graphs to a serialized event stream increases robustness and enables harness-style execution. It also exposed careful trade-offs: serialization simplifies process boundaries but requires stable encoding contracts and more explicit discovery/lookup logic for test metadata.
- Building a thread-safe collector for results required balancing lock granularity and throughput. The adopted strategy (a single lock per event bucket, plus careful mutation patterns) simplifies correctness while keeping contention low for typical test workloads.
- Cross-platform fallbacks: Supporting ASCII fallback for line-drawing characters taught me to design for the weakest terminal capability first, then progressively enhance for modern terminals. Testing across Windows/Linux/macOS terminals and non-interactive CI runners uncovered many small but critical edge-cases (line-wrapping, width detection fallbacks).
- Deciding what to show inline vs. in a details section is a UX trade-off, too much inline detail makes the tree noisy; too little hides useful context. The two-tier approach (short inline message + full details section) balances these needs.
Collaboration & process lessons:
- Presenting to the workgroup matters: Early live presentation to the Swift Testing workgroup exposed design assumptions quickly and gave direction that influenced major architectural choices.
- Iterative, small PRs win: Breaking work into digestible PRs (skeleton → ABI pivot → UI features) made reviews actionable and reduced reviewer fatigue.
- Incorporating targeted reviewer comments (e.g., ABI pivot) showed the importance of documenting design decisions (why we choose serialization, how the tree is constructed), which speeds later onboarding and review cycles.
Personal growth:
I learned how to accept and act on high-quality feedback from senior engineers, and how to reframe technical trade-offs as testable hypotheses. The project sharpened my ability to move from mockups to production code while keeping maintainability and cross-platform compatibility as first-class concerns.
7. Future Work & My Commitment
[!IMPORTANT]
While GSoC is concluding, the work on this new reporter is not yet complete.
- Future Work:
- The Live Phase: The next major step is to implement the "Live Phase," including the live-updating progress bar and the "heartbeat" mechanism for long-running tests.
- More Configurability: Adding more user-facing options, like the
-r charssuggestion from the community, to allow fine-grained control over the output. - Harness Integration: The ultimate goal is to move this reporter into a separate harness process invoked by SwiftPM.
- My Commitment: This project has been technically demanding and incredibly rewarding. I am highly motivated by this work and I am committed to continuing my contribution voluntarily after the GSoC program concludes. My goal is to help complete the Live Phase and see this new reporter through to its official launch.
8. List of Contributions
9. Acknowledgements
I'd like to express my deepest gratitude to my mentor, Stuart Montgomery, for his exceptional guidance, patience, and technical insights throughout this entire project. His mentorship has been invaluable.
I'd also like to thank Jonathan Grynspan for his crucial, in-depth architectural feedback, which significantly improved the project's long-term viability. Thank you as well to Swift Testing Workgroup, and all the other members of the Swift community who provided thoughtful feedback on the forums. This project would not have been possible without your collective expertise and support.
10. Contact
I am passionate about open-source development and systems programming in Swift. I welcome any connections, discussions, or opportunities for collaboration.
- GitHub: @tienquocbui
- LinkedIn: buitienquoc
- Email: tien-quoc.bui@epita.fr