First impressions adopting swift-testing

Related Projects swift-testing
1

recently, i’ve been working on upstreaming a totally-revamped JSON implementation to the swift-json project, an implementation which had previously been symlinked incessantly across many public and private repositories.

i figured this would be a great opportunity to adopt the new swift-testing library, as the project was previously using a testing framework i wrote myself long ago, which i was never really satisfied with. here are some of my first impressions of the swift-testing library.

Getting started documentation: A+

despite its early age, i think swift-testing has excellent “Getting started” documentation. i had zero experience using the library, but i was quickly able to pick up the basic patterns needed to use the library. the articles have good progressive disclosure, and i did not feel like i had a hard time understanding how the library is meant to be used. this reaction is atypical of my usual experience with Swift packages, so i felt it was worth highlighting.

one thing i think could be improved is the documentation should really be targeted towards Swift 5.10 users instead of nightly toolchain users, who probably make up a tiny fraction of the library’s potential audience. i often felt like the critical information i needed in order to use the library on Swift 5.10 was buried in asides and footnotes, and the articles were primarily addressed towards nightly users.

test writing workflow: C

in my opinion, this is not the fault of the swift-testing library, but i mention it anyway because it has a huge impact on anyone using the library: swift-testing depends on macros, macros depend on SwiftSyntax, and waiting for SwiftSyntax to resolve/clone/build sucks.

these are not new problems, they have been well-known for nearly a year now, and it has been independently-assessed that they are unlikely to be addressed under current language leadership.

because i was migrating away from a testing framework that did not use macros, i noticed a dramatic regression in how quickly i was able to iterate while writing tests.

again, i do not think this is the fault of the library, nor do i think the library should stop using macros. it is just an acknowledgement that using the library is going to surprisingly painful for those unaccustomed to building SwiftSyntax for test targets.

i should also note that if you already are using macros in your project, you will have a different probem, which is that swift-testing depends on version 509 and not 510, which will cause a dependency conflict. this conflict will be propogated to anyone using your project as well.

VSCode integrations: B

i don’t really have crazy expectations for tooling integrations in Swift, i’m generally happy as long as sourcekit-lsp works.

for reasons i don’t fully understand, compiling a project with swift build does not compile the Testing module in a way that is acceptable to sourcekit-lsp, so i always get the red squigglies with:

no such module 'Testing'sourcekitd

i can “fix” the problem by compiling Testing explicitly:

$ swift build --target Testing

but for some reason, this makes it even worse for sourcekit-lsp, as it suddenly loses the ability to digest the attached macros.

image

unknown attribute 'Test'sourcekitd

the “no such module” error is less of an impediment, so my recommendation is to avoid compiling the Testing module explicitly.

i found this very frustrating because mitigating the problem requires clearing the .build directory, and if you do that, you have to compile SwiftSyntax all over again.

test running workflow: N/A

running tests did not work for me at all; it crashes and you have to wait for backtracing to complete to get the terminal back.

[342/342] Linking swift-jsonPackageTests.xctest
Build complete! (57.35s)
Test Suite 'All tests' started at 2024-03-12 23:40:08.230
Test Suite 'debug.xctest' started at 2024-03-12 23:40:08.232
Test Suite 'AllTests' started at 2024-03-12 23:40:08.232
Test Case 'AllTests.testAll' started at 2024-03-12 23:40:08.232
_Testing/Tag+Macro.swift:37: Precondition failed: Tags must be specified as members of the Tag type or a nested type in Tag.
*** Signal 4: Backtracing from 0x77c3b8fc961f...
done ***

*** Program crashed: Illegal instruction at 0x000077c3b8fc961f ***

Thread 0 "swift-jsonPacka":
0  0x000077c3b811181e ppoll + 174 in libc.so.6
Thread 1 crashed:
0                  0x000077c3b8fc961f _assertionFailure(_:_:file:line:flags:) + 351 in libswiftCore.so
 1 [ra]             0x00006213d87a8008 static Tag.__fromStaticMember(of:_:) + 1031 in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Traits/Tag+Macro.swift:37:5
 2 [ra]             0x00006213d87aaa91 static Tag.red.getter + 48 in swift-jsonPackageTests.xctest at /tmp/swift-generated-sources/@__swiftmacro_8_Testing3TagV3redABfMa_.swift:3:19
 3 [ra]             0x00006213d86d2be2 one-time initialization function for _predefinedTagColors + 49 in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Events/Recorder/Event.ConsoleOutputRecorder.swift:93:8
 4 [ra]             0x000077c3b92fce81 swift::threading_impl::once_slow(swift::threading_impl::once_t&, void (*)(void*), void*) + 160 in libswiftCore.so
 5 [ra]             0x00006213d86d2df7 Event.ConsoleOutputRecorder._predefinedTagColors.unsafeMutableAddressor + 22 in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Events/Recorder/Event.ConsoleOutputRecorder.swift:92:24
 6 [ra] [thunk]     0x00006213d86d325b Event.ConsoleOutputRecorder.init(options:writingUsing:) + 282 in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Events/Recorder/Event.ConsoleOutputRecorder.swift:120:50
 7 [ra]             0x00006213d86ccc95 runTests(options:configuration:) + 148 in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Running/EntryPoint.swift:237:29
 8 [async]          0x00006213d87e3850 static XCTestScaffold.runAllTests(hostedBy:) in swift-jsonPackageTests.xctest at /swift/swift-json/.build/checkouts/swift-testing/Sources/Testing/Running/XCTestScaffold.swift:189
 9 [async]          0x00006213d86a60c0 AllTests.testAll() in swift-jsonPackageTests.xctest at /swift/swift-json/Sources/JSONTests/shims.swift:8
10 [async]          0x00006213d87e4900 implicit closure #2 in implicit closure #1 in variable initialization expression of static AllTests.__allTests__AllTests in swift-jsonPackageTests.xctest at /swift/swift-json/.build/x86_64-unknown-linux-gnu/debug/swift-jsonPackageDiscoveredTests.derived/JSONTests.swift:7
11 [async] [thunk]  0x00006213d87e4f00 partial apply for implicit closure #2 in implicit closure #1 in variable initialization expression of static AllTests.__allTests__AllTests in swift-jsonPackageTests.xctest at /swift/swift-json/<compiler-generated>
12 [async] [thunk]  0x00006213d87e4a20 thunk for @escaping @callee_guaranteed @async () -> () in swift-jsonPackageTests.xctest at /swift/swift-json/<compiler-generated>
13 [async] [thunk]  0x00006213d87e4e40 partial apply for thunk for @escaping @callee_guaranteed @async () -> () in swift-jsonPackageTests.xctest at /swift/swift-json/<compiler-generated>
14 [async]          0x000077c3b82e6e50 closure #1 in awaitUsingExpectation(_:) in libXCTest.so
15 [async] [thunk]  0x000077c3b82e76f0 partial apply for closure #1 in awaitUsingExpectation(_:) in libXCTest.so
16 [async]          0x000077c3b82e6fe0 specialized thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A) in libXCTest.so
17 [async] [thunk]  0x000077c3b82e7d60 partial apply for specialized thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A) in libXCTest.so
18 [async] [system] 0x000077c3b8e3e790 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) in libswift_Concurrency.so

Thread 2:

0  0x000077c3b811d86e epoll_wait + 94 in libc.so.6

Thread 3:

0  0x000077c3b806b38a __futex_abstimed_wait_common + 202 in libc.so.6


Registers:
rax 0x000077c3b9268510  55 48 89 e5 48 85 ff 7e 26 48 b9 00 00 00 00 fe  UH·åH·ÿ~&H¹····þ
rdx 0x00006213d881fe90  50 72 65 63 6f 6e 64 69 74 69 6f 6e 20 66 61 69  Precondition fai
rcx 0x000077c3b9268510  55 48 89 e5 48 85 ff 7e 26 48 b9 00 00 00 00 fe  UH·åH·ÿ~&H¹····þ
rbx 0x000000077c3b0005  32149012485
rsi 0x80006213d8828250  9223479874231108176
rdi 0x000077c3b00008d0  03 00 01 00 04 00 03 00 02 00 01 00 01 00 01 00  ················
rbp 0x000077c3b5dc26f0  60 29 dc b5 c3 77 00 00 08 80 7a d8 13 62 00 00  `)ܵÃw····zØ·b··
rsp 0x000077c3b5dc2650  10 00 00 00 00 00 00 00 e8 fe ff ff ff ff ff ff  ········èþÿÿÿÿÿÿ
r8 0x000077c3b00051e0  05 00 3b 7c 07 00 00 00 ad 6e 6a cd fe e3 53 00  ··;|····­njÍþãS·
r9 0x000077c3b5dc23c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ················
r10 0x000077c3b81803e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ················
r11 0x0053e3fecd6a6ead  23613106574487213
r12 0x00006213d8828250  5f 54 65 73 74 69 6e 67 2f 54 61 67 2b 4d 61 63  _Testing/Tag+Mac
r13 0x000000000000004a  74
r14 0x80006213d8828250  9223479874231108176
r15 0x0000000000000013  19
rip 0x000077c3b8fc961f  0f 0b 48 83 ec 08 48 8d 05 64 35 35 00 48 8d 3d  ··H·ì·H··d55·H·=
rflags 0x0000000000010286  SF PF
cs 0x0033  fs 0x0000  gs 0x0000
Images (15 omitted):
0x00006213d8613000–0x00006213d881251d <no build ID>                            swift-jsonPackageTests.xctest /swift/swift-json/.build/x86_64-unknown-linux-gnu/debug/swift-jsonPackageTests.xctest
0x000077c3b7fcf000–0x000077c3b816b04c a84b0b221620b4944c5940d3bed62056c7d7baa4 libc.so.6                     /usr/lib64/libc.so.6
0x000077c3b82b7000–0x000077c3b8310390 <no build ID>                            libXCTest.so                  /usr/lib/swift/linux/libXCTest.so
0x000077c3b8de4000–0x000077c3b8e5a5b0 <no build ID>                            libswift_Concurrency.so       /usr/lib/swift/linux/libswift_Concurrency.so
0x000077c3b8e65000–0x000077c3b947a2d0 <no build ID>                            libswiftCore.so               /usr/lib/swift/linux/libswiftCore.so
Backtrace took 0.24s
error: Exited with unexpected signal code 4

this was puzzling because i never got far enough with the library to use any Tags.

i considered that this might be due to the XCTest behavior described in that section’s documentation, but the --disable-xctest option mentioned there does not appear to exist at all, at least on Swift 5.10:

$ swift test --disable-xctest
error: Unknown option '--disable-xctest'
Usage: swift test <options> <subcommand>
  See 'test -help' for more information.

for completeness, i also tried passing the suggested --enable-experimental-swift-testing option, which also errored.

$ swift test --enable-experimental-swift-testing
error: Unknown option '--enable-experimental-swift-testing'
Usage: swift test <options> <subcommand>
  See 'test -help' for more information.

i checked in the crashing code to a branch of the swift-json repo here: re-initial commit · tayloraswift/swift-json@c5fe88b · GitHub

conclusion

i am annoyed that i was unable to actually run a test after sinking some amount of effort into migrating to the new library, but i assume there is a rational explanation for why i am getting this crash and i am eager to find out what i am doing wrong there.

the issues with sourcekit-lsp and SwiftSyntax are more profound and i’m not sure what the library could possibly do to address them, as they seem to be “environmental effects” of the Swift ecosystem. i think swift-testing has the right idea and the right API patterns, but its progress and utility is severely limited by the state of Swift macros. i hope this writeup serves as some motivation for Swift project leadership to engage with developers limited by Swift macros.

18 Likes
2

@taylorswift thank you for taking the time and effort to document your experience!

Massive +1 on the point regarding engagement with developers on issues with Swift Macros adoption (irregardless of whether the issues originate from Xcode or SwiftPM).

Just as a reminder to anyone visiting this thread, Swift Macros currently can add as much as 12 minutes of build time on Xcode Cloud primarily due to its dependency on Apple’s source distribution of SwifSyntax, and (to the best of my knowledge as of writing this post) there is no ETA for a fix at the moment.

4 Likes
3

This is a problem that we've brought up before and even tried to provide a solution to in an article here:

Unfortunately it does put the onus on every single macro library out there, and if one forgets to do so (which is easy to do, and as we see here even Apple packages are susceptible), you can quickly end up in a bad state w/r/t dependency resolution.

It looks like they recently updated to support version 510, but not in a way that is backwards-compatible with 509.

7 Likes
4

it’s not ideal as it might rob some folks of some flexibility (“now we need to bump all these upgrade swift-syntax tickets to P1”) but it’s still vastly preferable to depending on 509 only. let’s hope they tag a release with the updated dependency soon.

1 Like
5

Thanks for the feedback! A few notes:

As we've previously discussed, support for Swift 5.10 is temporary to aid developers who are not prepared to install a nightly main-branch toolchain.

We've actually updated to swift-syntax 5.10 on our main branch and our next tag will use it. That tag should become available soon.

My colleague @bnbarham knows a lot more about SourceKit-LSP than I do; I've asked him to stop by and reply. Regarding VSCode integration in general: we're only just starting to look at it, so it's not surprising the experience isn't stellar yet. For now, we recommend using Swift Package Manager from the command line.

This crash is new to us as we regularly test on Ubuntu 22. It does not reproduce on the main swift-testing branch as we've refactored the code that's causing a problem, so I expect our next tag (as mentioned above) will resolve the issue. If you're able to reliably reproduce the crash with the next tagged version of swift-testing, please file an issue. Thanks!

As discussed previously, Swift Package Manager integration is not available in Swift 5.10 or earlier. You will need to use a nightly main-branch Swift toolchain in order to use --enable-experimental-swift-testing or --disable-xctest (or any of the other SwiftPM changes we've made to add experimental support for swift-testing.)

6

i can confirm that with the new 0.6.0 release, the crash is no longer occurring :slight_smile:

1 Like
7

okay, not sure if i should start a new thread, but here are some more observations i had now that i can actually run tests :smiling_imp:

not printing full names of errors

when the testing framework catches errors, it only prints the last path component of the error type. in a code base that uses namespacing, this can produce some very generic outputs:

Caught error: ParsingError()

it would be nice if it printed something like

Caught error: JSON.StringLiteral.ParsingError()

instead.

drilling down into expected errors

here’s a common pattern in the JSON library:

extension JSON
{
    /// An error occurred while decoding a document field.
    @frozen public
    struct DecodingError<Location>:Error
    {
        /// The location (key or index) where the error occurred.
        public
        let location:Location
        /// The underlying error that occurred.
        public
        let underlying:any Error
    }
}

i am testing it with snippets like this:

@Test("UInt8")
func uint8() throws
{
    try #expect(throws: JSON.DecodingError<Never?>.self)
    {
        let _:UInt8 = try self.field.decode()
    }
}

but it would be really cool if i could expect something more-specific than that (like an IntegerOverflowError, perhaps with some yet-to-be-designed system for drilling down into the underlying error.

i can’t really #expect an actual value for the error, because the error does not conform to Equatable.

GitHub Copilot loves swift-testing

i don’t even know if this was an explicit design goal, but i’ve found that swift-testing gets along really well with GitHub Copilot, which i found surprising since i imagine the amount of training examples available to it so far is probably quite small. the AI just picks up the API and runs with it.

parameterizing tests with generics

i found myself writing a lot of tests that look like

@Test("Int16")
func int16() throws
{
    let _:Int16 = try self.field.decode()
}

@Test("Int32")
func int32() throws
{
    let _:Int32 = try self.field.decode()
}

@Test("Int64")
func int64() throws
{
    let _:Int64 = try self.field.decode()
}

@Test("Int")
func int() throws
{
    let _:Int = try self.field.decode()
}

but i couldn’t quite figure out how to fit this into Parameterized testing. it would be cool if we could parameterize tests over types and not just values.

anyways, these are more feature requests than actual complaints, so i am pretty satisfied with swift-testing 0.6.0 so far. hopefully the SwiftSyntax/sourcekit-lsp issues get resolved by their respective owners. great job!

4 Likes
8

It might be just me, but I run into this a lot. There are various ways around it using generic helper functions and the like, usually, but it always feels like I'm "holding it wrong" when I go that route. It feels like there should be a built-in, elegant way to parameterise based on types (and to test generic code, more broadly).

9

swift-testing relies on String(describing:) for most values, and this is a general constraint of the default implementation of String(describing:) for arbitrary values. If you have a type that can be described more precisely, consider adding conformance to CustomTestStringConvertible. This protocol allows you to provide more descriptive string representations of your errors.

I'm not sure this is exactly what you're looking for, but consider using #expect(performing:throws:):

#expect {
  // some work that's expected to throw
} throws: { error in
  guard let error = error as? JSON.DecodingError else {
    return false
  }
  guard let underlyingError = error.underlyingError as? IntegerOverflowError else {
    return false
  }
  return underlyingError == .tooManySevens
}

Generics are a hard problem to solve right now because generic test functions need to be fully specialized for them to be discoverable at runtime and for them to be invoked.

Right now, parameterized tests work best with homogenous inputs, but in the future we might be able to devise a syntax or macro expansion that allows heterogenous inputs to a generic function. That work would be a "future direction" for the testing library and may ultimately require language or compiler changes to implement correctly.

Your solution, having separate functions for different inputs, is the correct approach. If the body of the test is broadly similar and only the type of the input differs, consider factoring the body out into a common generic function and having the test functions invoke it. Something like:

func intCommon<T: BinaryInteger & Codable>(_ value: T) throws {
  let data = try value.encode()
  _ = try Self.decode(T.self, from: data)
}

@Test func int16() throws {
  intCommon(123 as Int16)
}

@Test func int32() throws {
  intCommon(123 as Int32)
}

// etc.

(I'm not familiar with your codebase, so that code probably doesn't compile as-is, but I'm sure you get the idea.)

Glad to hear it!

2 Likes
10

Could a slightly different macro be the answer here? Spit-balling ideas, would something like this be feasible:

@GenericTest("Int decoding", types: [Int16.self, Int32.self, Int64.self, Int.self])
func intDecoding<T: BinaryInteger & Codable>() throws {
  let _: T = try self.field.decode()
}

which would expand to multiple specialized peer declarations by stripping off the generic parameter clause and substituting any occurrences of T so that the result is effectively the same as if you had written the following by hand:

@Test func intDecoding_Int16() throws { let _: Int16 = try self.field.decode() }
@Test func intDecoding_Int32() throws { let _: Int32 = try self.field.decode() }
@Test func intDecoding_Int64() throws { let _: Int64 = try self.field.decode() }
@Test func intDecoding_Int() throws { let _: Int = try self.field.decode() }

The generic function would still be compiled even though it's never used, but that's fine, as it still serves as a nice way to verify that the code inside the generic version satisfies the constraints imposed by the generic requirements. And if any of the types passed in the macro don't satisfy a requirement, then you get a compiler error at that specific point in the expansion.

It's imaginable that this could even be expanded to work as a generic cartesian product, if you supported multiple generic parameters and an array of tuples of types.

2 Likes
11

Specialization doesn't seem necessary (except for performance, as usual), as long as there's a list of types to try out somewhere that the test runner can feed to the unspecialized implementation. For property testing it might even be fun to pick "random" conforming types from the process. (This is not to say that swift-testing absolutely needs to support generic test functions right away or anything, just an observation about the possible implementation.)

3 Likes
12

what’s preventing swift-testing from using String.init(reflecting:)?

13

Generic tests are a potential future enhancement to the package, but not something we're looking at just yet.

String(reflecting:) has less predictable output. By default, it produces a reflection of the value that can span multiple lines, and that doesn't really "fit" (for lack of a better word) where we need a description. The CustomTestStringConvertible protocol is provided so you can fine-tune descriptions of values for testing, but if a type conforms to CustomStringConvertible or CustomDebugStringConvertible instead, swift-testing will fall back to one or the other.

14

I don't think thats correct. String(reflecting:) is what is used by all collections to print their elements to have representations suitable for structural display. It is also used by default for printing nested values in structs and enums.
If the error includes any nested value, String(reflecting:) will likely be used for them.

For which types have you seen multiline output for String(reflecting:)? This isn't expected as it will result in non-ideal output for collections and other types. String does exactly the opposite and removes new lines and other non-visible characters for String(reflecting:) but includes them in String(describing:).

1 Like
15

Sorry, I miswrote. I should have said something to the effect of "it produces much longer output." I may have been confusing it with the dump(_:to:) API as I was writing it. My bad!

16

Since OP didn't use the library successfuly, the original post was mostly criticism. I'd like to share my different experience. I'm using an earlier version of the library to write test for my app. I like it. It's very simple to use and flexible. The test code is concise and has better readability. When test fails, the log helps to identify the cause quickly. Swift-testing is to XCTest as SwiftUI is to UIKit.

How I use it: I tried Swift-testing shortly after I installed Xcode 15.0. I found it worked. I was aware the library was being actively developed and would use new language or macro feature not available in Swift 5.9, so I saved a copy locally and never pulled new revisions.

I suspect the OP haven't used macros in his projects. I use a lot of macros in my code so I don't feel swift-test is much slower. There is one exception. If I use parameterized testing and modify test (or related code?), a simple change may cause a lot of file get recompiled and it becomes very slow. I didn't investigate the issue, nor am I sure how to reproduce it (I stopped using parameterized testing later).

I use Xcode to run test and it just works. It's not possible to run a single test yet, so it's a bit inconvenient at the moment. but overall I like the new test framework and think it's worth tring it.

1 Like