[Pitch] A Structured Approach to Build and Test Results Data

Evolution Pitches
, , , ,
1

Hello Swift community!

I would like to pitch the creation of a package containing APIs that provide a unified approach to publishing and consuming results data generated by developer activities, including (but not limited to) building and testing, both at desk and in continuous integration services. A few examples of this data are build logs, test outcomes, issues (build, test, runtime, etc.), retry details, CI job orchestration, package resolution, and VM provisioning steps.

This proposal represents the work of my team of Xcode engineers at Apple who are responsible for test and build results. There is functioning prototype code for many aspects of the proposal below, but those implementations are in no way assumed to be foregone conclusions about how the pitched project will actually unfold. Rather, we hope to use them to guide discussions and welcome alternative approaches.

We hope to break ground on this project soon and look forward to your feedback.

Problem

Developer activities such as building and testing produce considerable amounts of data designed to answer two key questions:

  • Did it work?
  • If not, why?

This data takes many forms, ranging from raw text logging to domain-specific customized structures to arbitrary assets such as screen recordings. Processing the data to answer the key questions - especially the "why" - is challenging for both humans and machines. The biggest cause of this difficulty is the lack of consistent structuring. As a result, tools cannot efficiently and reliably direct a user to the relevant bits that can help them triage and diagnose problems in their work.

Goals

  • Organize data from build and test actions using a consistent structure.
  • Facilitate viewing of results information by humans.
  • Allow efficient machine-parsing of results by tools and IDEs.
  • Collect results data in a single place which can be inspected or transferred as one artifact.
  • Permit multiple entities (e.g. processes, hosts/devices, services) to contribute to a single stream of events.
  • Support extensibility by allowing publishers to represent custom data when necessary.
  • Enable sophisticated UI tools to “stream” events and display live updates efficiently.
  • Simplify analysis by modern AI/LLM tools with limited context windows.
  • Align the experience of working with results data from local and CI operations.
  • Provide the data foundations for historical metrics tracking (build performance, test results/durations)

An approach to structured results

This project proposes several API modules to provide a structured results foundation for test frameworks, build systems, CI infrastructure, IDEs, and other tooling.

A new Package

These APIs would be implemented in a new package, housed in a new repo within the GitHub repo under /swiftlang . Initially we would expose the libraries in this package for other toolchain projects to import/link, but not for regular users to directly import in their own projects. The CLI tool would eventually be exposed in OSS toolchains and made available to end users.

Publishing APIs

The publisher modules are meant to be adopted by entities that produce results data, such as a test framework or build system. These modules will have an extremely minimal set of dependencies so that adopters such as a test framework can do so without introducing a cycle or other conflict with projects that use the test framework. The publisher approach is also designed as a distributed system, where results data can be emitted from multiple processes on multiple hosts using a "channel" abstraction.

Consumption APIs

To process the structured results data, the Consumer modules will provide APIs that can be used to read, filter, and transform the data, both "live" - while it is actively being produced - and with completed results. While the publisher is meant to be minimal in its dependencies, the consumer does not have the same constraints.

CLI

A command line tool that wraps the Consumer module is included for utility. It will support actions such as:

  • merging results from different jobs
  • exporting results data to other standard formats
  • exporting attached assets to files
  • getting aggregate information from a result, including such things as the number of test failures, a list of build issues, duration of the job or portions of the job
  • etc.

Data format/specification

The data will be emitted as JSON Lines ("JSONL", https://jsonlines.org), where each line represents a distinct telemetry event - a span start, a "record" (leaf node), or a span finish. JSONL makes it easy to manage multiple discrete streams from different (distributed) producers, flexible batch management, and easy (cheap/high performance) manipulation in terms of combining data without having to process it.

Each event contains a structured "payload" with a core schema expressed in the project for things such as test runs, compilation tasks, build/test diagnostics, user-defined activities, and more. This schema also supports extension in clients and composition of other, arbitrary data. Payload extensibility will make this generalized system for structured results function as a compliment (rather than a competitor) to the adoption of domain-specific schemas, such as proposed for SARIF in https://forums.swift.org/t/pitch-sarif-support-for-swift-diagnostics/85513.

The structured results schema (including its core types) will be published using OpenAPI or similar approaches to enable generation of interfaces in other languages, as needed by third party tooling.

Example usage scenario

A user initiates a Test action from an IDE on a connected mobile device

In this scenario there are at least two processes involved, executing on different hosts - the IDE (e.g. Xcode) and the test runner (e.g. Swift Testing). Each generates data relevant to the results, so each is considered a producer with its own channel for publishing. Channels - while distinct streams from each other - are also spans that encode parent references like every other event, providing context within the larger “session” of results. The IDE channel begins emitting events as soon as the user action commences, providing a high level organization of the result data, perhaps including details about the code being tested, versions of languages/compilers/other tools and any other context that is pertinent. The test runner on the device creates its own channel when it launches, with some results metadata forwarded from the IDE, and begins to emit its own events including any initialization/preparation data and then the tree of its test executions and their individual results. The output of this channel may be written to a file on the device and retrieved on completion or streamed back over a socket to the IDE so that it can show live results as tests are executed. The consumer module would be used for capturing, aggregating, and extracting data for display to the user.

Candidate Clients

The success of this effort is predicated on buy-in and adoption by projects which generate and consume the kind of data defined here as “developer results”. We’ve identified the following as ideal adopters and are either actively engaged with them now or planning to reach out shortly:

  • Swift Testing
  • Swift Build
  • Swift Package Manager
  • Corelibs XCTest

Alternatives considered

Using an “off the shelf” telemetry solution

The proposed direction for this project is strongly influenced by telemetry solutions, so it's a reasonable question whether those systems would be sufficient in themselves. There are two considerations that led to the decision to develop a solution that borrows from telemetry concepts but is not itself a pure telemetry solution. The first of these is the value of arbitrary (and potentially large) attachments as diagnostic tools. The second is the importance of a strong schema of types for the various actions (build, test, etc.) that goes beyond the attribute dictionary level of user data provided by telemetry systems both in terms of its structure and its extensibility.

Use Protobuf or another serialization format

The advantages of JSONL are covered in the data format section above, but there are alternative ways data might be serialized including formats such as Protobuf, which offer their own advantages. This is an area where the technology choice will continue to be evaluated, with tradeoffs such as dependency requirements, storage characteristics, parsing efficiency, and others will be considered. It’s also possible that the serialization approach will be “pluggable”, with clients being able to select from a set of options or provide their own solution.

Open source the .xcresult bundle format

A predecessor technology in this problem space is Apple's Result Kit/result bundles (".xcresult") produced by Xcode during building and testing actions. Result bundles generated by Xcode may only be inspected using its included xcresulttool CLI tool. The Result Kit approach differs considerably from the one proposed in this project because much of the data is post-processed before being written, rather than preserving the original "raw" sequence of events. In addition, for CI systems and other services, it's preferable to decouple the storage of large attachment files from the event data. Another consideration is that formulating the result as a stream of events, rather than a finalized artifact, makes it possible to incrementally deliver results back to users. Finally, the bundle structure itself is an Apple-platform concept, which makes it less appealing for a solution aimed at a cross-platform audience. That said, it is still possible that we would define a "bundle-like" directory structure for managing combined event and attachment data in local and peer-sharing scenarios.

Implementing a storage service

Storage of results data in cloud services is a closely related problem but one we think should remain decoupled, at least in the initial phases of the project. The focus here is to design APIs and protocols that efficiently support cloud storage services, without implementing those services themselves. Storage requirements vary considerably between organizations and use cases, in terms of:

  • scale of the data produced
  • analysis goals for the data
  • duration of storage
16 Likes
Testing Workgroup meeting April 20, 2026
2

Hi Wil!

This was brought up in the Testing Workgroup meeting today, and I have a couple questions on it.

This is very light on actual details. Can you expand on that? What exactly are you proposing to build here? What problem(s) does this solve?

Thanks! Appreciate the effort you've put into this so far!

2 Likes
3

Hi Rachel!

Thanks for the response, questions are good :)

This is very light on actual details.

That's fair - this is more of a "vision" kind of document than a technical pitch, but we were encouraged to engage with the community here first, with a focus that's more around the problem space than the details of our prototyped solutions to date.

What exactly are you proposing to build here?

We're proposing to introduce a new package with several API modules. Some of these modules are aimed at the publishing side of results data, the rest are on the consuming side. Publisher modules are intended to be lightweight with minimal dependencies so that libraries like Swift Testing can easily adopt. Some aspects of consumers may pull in more dependencies to provide network functionality for receiving data or processing capabilities for analyzing data.

What problem(s) does this solve?

  • Issues from disparate systems typically lack context that goes beyond the borders of the system that produced them. For example, a test framework can provide rich details about problems that originate within the user's code under test or the test framework itself, but it's difficult for them to provide data about the environment in which they are executing, the process that launched or coordinated them, other processes that impact execution, and the broader range of activities within the device(s) involved in the testing. This has particular impact in CI environments, where testing may be distributed across many hosts and involve libraries and processes that the testing system itself can't reason about. When a failure crosses the boundary of just the test code, it can be very challenging for humans or machines to track down the root cause, because the lack of context between the diagnostics from each participating entity makes it difficult to match up the relevant data.
  • Lack of a standard and consistent approach to reporting results introduces duplication in tooling. For example, many teams use multiple testing systems and it's difficult to collate the results from these different systems in a single interface. Tool maintainers end up implementing multiple solutions for different results formats that solve very similar (but not identical) problems repeatedly. Aligning the structure and approach for this data enables reuse in tooling instead of reimplementation.
  • Scaling of results data management is a significant problem for very large projects - long term storage, rapid analysis, live updates are all areas where individual systems implement approaches that work "well enough" until deployed in an environment where they need to scale.
  • Consumability of results data by AI models is greatly improved when well defined, standard structures are in place.
4

Great to see this proposal! Overall I think this is the right direction. A few questions/comments below:

Q1: Is the described format exclusively streaming or aggregated? The proposal mostly speaks to the streaming part ("a span start, a "record" (leaf node), or a span finish"), but less to the aggregated data. Would large aggregated JSON payloads arrive at the end of a stream, or would querying aggregated data be handled through a separate API? I ask to better understand if folks who want to read aggregated data would be expected to put a "consumer" client that vends an aggregate API (as a separate target/project), or if vending aggregated data is expected to be done as part of this initial project as well.

Initially we would expose the libraries in this package for other toolchain projects to import/link, but not for regular users to directly import in their own projects. The CLI tool would eventually be exposed in OSS toolchains and made available to end users.

I assume the initially here means that the package would become API stable eventually to allow other packages (especially libraries) to depend on it, but wanted to get an explicit confirmation - will it? I think it's fine for you to release 0.x versions initially as the format is being stabilized.

The data will be emitted as JSON Lines

I agree that this is the right baseline, but as you touch on, I think the concept of a "content type" should be built-in from the start to allow requesting a different format, such as CBOR. CBOR represents basically the same data as JSON, just is more efficiently encoded. For example, the AT protocol does something similar - using JSON and CBOR together, allowing consumers to request the format that works best for them (JSON for wide support, often without external dependencies, and CBOR for most CPU/memory efficiency in constrained environments).

The structured results schema (including its core types) will be published using OpenAPI or similar approaches to enable generation of interfaces in other languages, as needed by third party tooling.

:clap:

2 Likes
5

Candidate Clients

The success of this effort is predicated on buy-in and adoption by projects which generate and consume the kind of data defined here as “developer results”. We’ve identified the following as ideal adopters and are either actively engaged with them now or planning to reach out shortly:

  • Swift Testing
  • Swift Build
  • Swift Package Manager
  • Corelibs XCTest

The pitch is a bit abstract, but perhaps benchmark results would also fit in here.

4 Likes
6

Would love to see this, could be a significant competitive advantage for Swift compared to other programming languages' tooling. The key is to focus on helping the dev solve real problems, which we are all doing ad-hoc with our own scripted or manual solutions now. For example, some concurrency regression caused significant trunk CI test failures in the last couple weeks: ideally, this structured approach would allow easily writing automated tooling that would notify us and help pin down the cause much quicker.

My only caution is against using a text format like JSON for the event stream. I would much prefer a binary format, as these can be large data sets and since it is highly repetitive data, the compression and speed from a binary format cannot be matched. Provide all the text formats you want for the user to view and export, but please use a binary format for the actual data manipulation and storage. May seem obvious to the Swift devs who chose a binary swiftmodule format- which unfortunately doesn't seem to have a good way to view as text, beyond dumping the bitcode as I've found- but worth ensuring.

2 Likes
7

FWIW, JSON Lines content type over HTTP is often used with gzip, making it quite efficient, plus JSON and gzip are often supported out of the box by a platform, allowing even Swift scripts to consume the stream, without any dependencies. So I think JSON is a great (even default) option.

But I agree another (efficient) option for a binary format is also desirable (thus my CBOR suggestion).

2 Likes
8

I disagree, been arguing this off and on with other devs for decades now, didn't think I'd have to with the guy who also asked for a binary format. :wink: No general-purpose compression algorithm like gzip can ever match a well-designed binary format that is based on your knowledge of the regularities in your data. It is why HTTP itself went binary from HTTP/2 onwards and we don't use gzip on raw image data.

Sure, JSON and gzip often don't require added dependencies, but pretty sure we could have a small decoder for any new format available as a source package and tool in the Swift toolchain. Not saying a new binary format is the obvious choice here, as each format has tradeoffs, but it might end up working out best.

9

I guess both can be served - Wil's proposal calls out JSON as the starting point, with potential binary formats as additional options. I understand both to be "APIs", the internal representation could be different still and could even change without breaking API.

But being able to write a simple web dashboard using JavaScript only, without any large framework dependencies, that consumes the data streamed from the server, is an important use case. That's why I'd like us to preserve the maximal compatibility option of JSON.

1 Like
10

You're not wrong, but do keep in mind that Wil's proposal is for an extensible data format and as such he/his code does not necessarily know what those regularities are. When dealing with an open format, the trade-offs are different, and ease-of-extension is in this case going to be more important than raw compressibility.

2 Likes
11

That can be preserved by requiring that the external JSON API always exists, without requiring that JSON be used internally for search and storage.

We know a lot about the regularities, as these are machine-produced compiler error messages and so on. Binary formats are extensible too, and the format's layout can be versioned for maximal flexibility.

12

Hi Honza,

Thank you for your questions and ideas! I'm one of the engineers at Apple working on this project.

Q1: Is the described format exclusively streaming or aggregated? The proposal mostly speaks to the streaming part ("a span start, a "record" (leaf node), or a span finish"), but less to the aggregated data. Would large aggregated JSON payloads arrive at the end of a stream, or would querying aggregated data be handled through a separate API? I ask to better understand if folks who want to read aggregated data would be expected to put a "consumer" client that vends an aggregate API (as a separate target/project), or if vending aggregated data is expected to be done as part of this initial project as well.

  • Data is streamed through a pluggable fixture called a "Transport" which, by default, emits JSON Lines formatted data. The default Transport implementation writes to a local file, but alternative implementations can batch the stream via the network (to an HTTP endpoint, for example) or use IPC to send it to another process on the same host, or any other mechanism that's needed for custom configurations. Fault tolerance is a "P0" requirement - we expect that producers may be interrupted at any time for any reason and their result data must be captured without loss. With that context, any kind of aggregation work should not be part of the results streams but be the product of post-processing. So, no large payloads at the end of a stream except to the extent that they reflect some activity that is a specific characteristic of a client.
  • Users will of course need to read aggregated data! This document does not go into depth about consumers, but the prototyping work so far includes consumers that can ingest stream data, both progressively (i.e. as it is being generated) and from completed streams (finished jobs). These consumers can process the data to answer questions such as
    • How many tests executed/passed/failed?
    • What are all the failure diagnostics?
    • What are the N (e.g. 20) events preceding this failure?
    • How long did X (event or span or groups thereof) take to complete?
    • Etc.
  • Modules for handling aggregated data are expected to be part of this project.

I assume the initially here means that the package would become API stable eventually to allow other packages (especially libraries) to depend on it, but wanted to get an explicit confirmation - will it? I think it's fine for you to release 0.x versions initially as the format is being stabilized.

API stability is expected, hopefully within the first year.

the concept of a "content type" should be built-in from the start to allow requesting a different format

We completely agree. The Transport interface is how we plan to provide this kind of flexibility. While this document frames the Transport as a component of the Publisher module, it can also be considered as the glue between publishers and consumers. The default Transport emits a JSONL file on the publishing side and the corresponding consumer component ingests that content, but an alternative Transport pair could handle emitting CBOR for a publisher and reading CBOR for a consumer which then performs the kind of aggregate processing I described in my answer to the previous question.

1 Like