If it's only available from the open-source toolchains, then adding it as a dependency would pose problems for developers using the toolchain provided with Xcode. 
1 Like
I did not find any mentions of standardisation for reading test vectors, files with expected results often JSON files, but might also be other file formats, e.g. binary.
This is a big pain point today, especially since Xcode and SPM behaves differently regarding definition of Bundle which almost all implementations rely on. Apple's swift-crypto, of which I'm an active contributor, in fact has two:
- Callers of wycheproof does not define the JSON test vectors in Package.swift, and uses this quite fragile solution:
let testsDirectory: String = URL(fileURLWithPath: "\(#file)").pathComponents.dropLast(3).joined(separator: "/")
let fileURL: URL? = URL(fileURLWithPath: "\(testsDirectory)/Test Vectors/\(jsonName).json")
- Tests of recently added HPKE does define vectors in
Package.swiftusing.resources([.copy...]), and then usesBundle.modulewhich only gets defined by having declaredresourceIIUC.
And the these different solution both use #if IN_SWIFTPM conditionals controlling if/how Bundle is used, due to difference in behaviour between Xcode and SPM. See StackOverflow thread about this for for info.
I would LOVE if we could find a better solution here... a solution for swift-testing sharing the same API for Xcode and SPM which is easy to use and intuitive. Has this been given any thoughts?
I would love to get something like:
@Test("HPKE vectors test", vector: "hpke-test-vectors.json")
func auth(_ vector: HKPETestVector) throws {
#expect(try calculate(vector.input) == vector.output)
}
Probably where ".json" can be omitted, and where one could pass in decoder: JSONDecoder() as argument in the @Test macro as well if needed. Where HKPETestVector conforms to Decodable.... Fine if this requires us to specify .resources in Package.swift.
What do you all think?
1 Like
Yay! @smontgomery fixed this in #expect(throws:) expression should accept non-Void return type and describe returned values by stmontgomery · Pull Request #21 · apple/swift-testing · GitHub 
2 Likes
I'd also call out the the coverage based part of what libFuzzer does; just using a corpus helps, but the fact that libFuzzer can look at the coverage to confirm how well it is exercising the code likely is a big part of what helps it be so complete.
Another advantage to reusing things (libFuzzer) is it might open doors do enable things like oss-fuzz does, where there are multiple different fuzzing engines that can be linked in because they all use common interfaces (it's been a while since I looked, but I believe that's part of how that is done).
2 Likes
One thing I don't see in the vision document is support for multiple output formats. One of XCTest's biggest issues is that it can't output machine-readable results, especially when streaming. Result bundles are more machine readable but their format is undocumented and subject to change over time. So while Sajjon has shown how the human-readable output can be improved, I'd like to see a standardized machine format as well so we can properly integrate Swift test output into CI (any CI!).
6 Likes
In the Flexibility section of the vision document, there's a brief mention of this:
- Allow observing test events: Some use cases require an ability to observe test events—for example, to perform custom reporting or analysis of results. A testing library should offer API hooks for event handling.
I agree though that I'd like to get more insight into what the thinking is here for a Swift native solution. XCTest is somewhat unique in how it takes advantage of macOS bundles and the Objective-C runtime to do things like allow a user to inject an observer by populating the NSPrincipalClass field of the Info.plist.
Since swift-testing already appears to be doing some Swift runtime metadata discovery for tests and suites, perhaps the same model could work for test observation? Maybe the runner could scan the binary for any types that conform to a test observation protocol and then instantiate those.
In the vision document we mentioned a core goal being integration with CI systems, IDEs, etc. Many of those require machine-readable results formats, so we know that supporting this sort of functionality will be important.
The package currently avoids specifying any particular output format intentionally, and instead provides an interface that can be used by CI systems, IDEs, etc. to build their integrations. The output provided via Event.Recorder and XCTestScaffold is meant to be human-readable and is implemented atop those same interfaces.
When it comes to machine-readable output, we anticipate working with other common tools in the Swift ecosystem as this project matures, to explore how this package can best integrate with them using these same interfaces.
3 Likes
Good question, needs to be investigated and answered for a real implementation, but for now I need to know if maintainers of swift-testing are interested in having the feature at all - given it would be feasible to overcome all challenges implementing this.
But anyway it would be a requirement to be able to opt out of it if default, for performance sensitive/focused testing. But I think (maybe naively) that most unit tests are not performance focused, and for most devs it would be a nice addition seeing some kind of progress (for long running tests).
Here are some SF Symbol variants
4 Likes
Radiance looks great, especially if the color is based on the previous result of the test (if that's at all trackable).
1 Like
Will this work on platforms other than MacOS, such as Linux?
The takeaway I have from this mini-discussion is that I think it's important that users of this framework can provide their own drivers for different styles of testing. Bonus points if all the Xcode infrastructure like test chevrons can plug in, and not just the assertions. Hooking libFuzzer up to XCTest is a touch painful, to the point that certain modes like crash testing are basically off the table.
I had a pipe dream of being able to do Eiffel-style automatic test case generation and contract-oriented programming on top of whatever we provide here...
5 Likes
It’s great to see Swift getting improved test support, thanks for all your work on it! 
Very exciting to hear this! I have some thoughts on this and will chime in when the thread is posted.
Calling other tests
One thing I didn’t see mentioned is whether a test can call another test:
@Test exampleTest() {
otherTest(42)
}
@Test(arguments: 0, 1, 2)
otherTest(_ int: Int) { … }
This may not be useful that often, but I don’t see any reason not to allow it.
Suite initializers
If a type contains test functions declared as instance methods, it must be possible to initialize an instance of the type with a zero-argument initializer
It might be useful if collection arguments could be specified for suite initializers. This would combing combinatorially with collection arguments for tests so users would need to be careful, but it could be a nice way to factor shared arguments out of individual tests by lifting them into properties.
Test functions are still Swift functions and can be called under the usual circumstances. So yes, your example will work as you'd expect, and any test failures while calling otherTest() will be attributed to exampleTest().
During expansion of the @Test macro, the only part of the syntax tree visible to the macro plugin is the test function and its attributes. The available initializers on the type containing the test function are not visible or enumerable, nor is a containing type's @Suite attribute if present, so it is currently necessary to constrain how an instance of a suite type is initialized to something that is consistent and, ideally, automatically available. We're interested in extending the capabilities of test functions as members of types, however it may require significant enhancements to Swift macros first.
There may be other ways to accomplish what you're looking for. If you like, we can continue this discussion in a separate thread.
These progress indicators are very cool! Two questions however:
- How does it play along with any log output the tested code might print?
- [Way too early a question but asking anyway…] Can you make it produce sensible output when run in a non-interactive terminal such as CI?
Yes sure, just need to fine a suitable good looking sequence of unicode chars as fallback on Linux and Windows, see here where swift-testing fallbacks to unicode for symbols for fail/pass
E.g. braile pattern dots, ⠓
⠖ etc could probably form a nice pattern, especially following a Karnaugh-ian order, i.e. minimum changes to the dots between the symbols for smoothest possible animation.
I can update my POC later today!
1 Like
Ideally it would be a sequence of characters showing a bike shed progressing from unpainted to fully painted. 
9 Likes
I'm really excited to see some thought put into what the future of the testing framework looks like for Swift and there are some really good ideas in here.
One thing that I was surprised by is that tests are defined as functions that are annotated as @Test. Here is what I have had in my brain as a future direction for testing (I was even going to try building this on top of XCTest): Expanded.swift · GitHub. Something I thought was really cool in Kotlin is that because it supports any string as a function name, even if it has spaces, you can right tests with a more readable name instead of being limited by function names. Also I think being able to define test variations using things like normal forin loops could make it a lot more readable.
Excited to see this evolve!
3 Likes
Always very happy to see improvements to Swift’s testing! Thanks for driving this forward!
We maintain a number of Swift packages and many of them share a common theme: improving Swift’s testing story. Hopefully some of our learnings can be shared. I'll try to itemize each issue and how our libraries have been solving for them.
Assertion readability
XCTAssertEqual’s output is not very readable for larger data structures. Even smaller ones are difficult to scrutinize:
var other = user
other.name += "!"
XCTAssertEqual(user, other)
XCTAssertEqual failed: ("User(favoriteNumbers: [42, 1729], id: 2, name: "Blob")") is not equal to ("User(favoriteNumbers: [42, 1729], id: 2, name: "Blob!")")
We try to offer an improvement in our swift-custom-dump library, which provides a custom “dump” function that is a customizable, pretty-printed version of Swift's dump, and it uses this function in custom XCTAssertNoDifference and XCTAssertDifference functions:
XCTAssertNoDifference(user, other)
User( favoriteNumbers: […], id: 2, - name: "Blob" + name: "Blob!" )(First: −, Second: +)
(XCTAssertDifference also provides a nice way of writing assertions for how larger structures are modified, either exhaustively over the course of an operation, or non-exhaustively asserting against just a few given fields.)
We're not the only library to address this problem. Difference is another popular solution. And the recent swift-power-assert also tries to provide better failure messages.
Ideally, Swift would provide a good story here by default. SwiftSyntax's macro testing helpers are already providing a nicer experience in assertMacroExpansion, which will print a difference when the expected expansion doesn't match. I hope Apple will follow these examples more generally!
I took #expect for a spin and found that it does not currently address this problem.
Testing concurrency
Writing reliable async tests in Swift is notoriously difficult.
Our swift-concurrency-extras surfaces a global override to allow concurrent code to run serially, making it possible to write reliable async tests:
await withMainSerialExecutor {
// Code in here runs serially
}
It's a hack, but it demonstrates a real deficiency in Swift testing that ought to be resolved.
Testing time-based asynchrony
Swift doesn't provide any out-of-the-box helpers for testing APIs that can sleep for a duration. So we provide swift-clocks (and combine-schedulers) with a number of helpers for testing time-based asynchrony, particularly a Test{Clock,Scheduler} that can manipulate the flow of time, an Immediate{Clock,Scheduler} that can sub into tests that don’t care about waiting for time to pass, and an Unimplemented{Clock,Scheduler} that is a stub that will emit test failures when code interacts with it (a good default value to help notify you if a test interacts with time-based asynchrony in a non-testable way).
While Combine is a private framework, Clock is a standard library protocol that is not super testable out of the box. While I’m not sure if swift-testing should be responsible for closing the gap, I do think Swift should ideally provide first-class helpers for writing tests that interact with the Clock protocol.
Snapshot testing
Snapshot testing has become more and more popular, but Swift doesn't offer any solutions out of the box. It's been mentioned a few times elsewhere in this thread, but we provide some libraries for it:
- swift-snapshot-testing provides helpers for recording references and testing against them, either on disk or inline in the test case.
- swift-macro-testing extends the inline snapshotting capability of swift-snapshot-testing for testing macros.
Writing assertions in application/library code
It can be helpful to write test helpers for your application/library code, but one must tediously introduce a brand new MyLibraryTestSupport module to do so. This is because the act of importing XCTest into an app or library target leads to linker issues.
But it is also handy to write test assertions directly in application/library code: they can act as softer preconditions than fatal errors and immediately integrate with your test suite.
To work around these limitations we built xctest-dynamic-overlay, which allows you to call XCTFail from application and library code by dynamically loading XCTest. And so it's possible to instrument application and library code with test failures that will be caught by your suite. We also log purple runtime warnings when XCTFail is invoked outside a test suite, making it easier for developers to catch issues while they run their code in a simulator or on device. While it may seem strange to want to emit test failures from application and library code, we’ve found many useful applications, and XCTest Dynamic Overlay is a dependency of many of our libraries. A few use cases:
-
swift-case-paths defines
XCTUnwrap(case:)andXCTModifyhelpers for better testing enums and their associated values. -
swift-composable-architecture uses it to provide a
TestStorefor testing features written with the library without introducing another module. While we originally shipped a separateComposableArchitectureTestSupportmodule, it often led to downstream build issues in user land due to bugs in Apple's tooling. It's also less ergonomic to remind users toimport ComposableArchitectureTestSupport.In addition to the
TestStore, the library generally usesXCTFailto emit test failures and runtime warnings when we detect and communicate issues with how the library is being used to the developer. -
swift-custom-dump uses it to define its
XCTAssertNoDifferenceandXCTAssertDifferencefunctions. -
swift-dependencies invokes
XCTFailwhen an unstubbed dependency is called from a test, among other failures.
These are just a few high level issues that come to mind, and we'd be happy to dive deeper into any of the above topics if folks have questions.
39 Likes
Related to this - could the library provide a hook for formatting the test output of an #expect failure? Eg setting this globally or task-locally? This would then allow third parties to provide enhanced diffing, etc without the user needing to rewrite their tests.
This might already be covered by this:
Allow observing test events: Some use cases require an ability to observe test events—for example, to perform custom reporting or analysis of results.
3 Likes