Introducing Spyable: A Swift Macro for Automatic Spies Generation

Hey folks!

I'm thrilled to introduce Spyable - a powerful Swift macro that simplifies and automates the process of creating spies for testing.

My goal with Spyable is to make it easier to write comprehensive, precise tests, and ensure our applications behave exactly as we expect them to.

I believe this macro can become a replacement for the most popular use-case of Sourcery - Mocks & Stubs. In order to ensure a smooth and seamless transition from Sourcery to the Spyable macro, I have taken steps in the initial version of Spyable. It's designed to generate spies for protocols in the same way as the AutoMockable template. Thanks to that, most projects using Sourcery with the basic AutoMockable template can be switched in minutes.

As many of you know, creating spies manually is tedious, and error-prone. With Spyable, you only need to annotate your protocol with @Spyable. The macro will then generate a corresponding spy class that implement this protocol. This generated class can track all interactions with its methods and properties. It records method invocations, their arguments, and returned values, which are then available for you to make precise assertions about the behavior of your system under test.

Here's a simple example of what this looks like:

Example
@Spyable
protocol ServiceProtocol {
    func fetchConfig(_ arg: UInt8) async throws -> [String: String]
}

This would generate a ServiceProtocolSpy class that implements ServiceProtocol , and includes properties and methods for tracking interactions.

class ServiceProtocolSpy: ServiceProtocol {
    var fetchConfigCallsCount = 0
    var fetchConfigCalled: Bool {
        return fetchConfigCallsCount > 0
    }
    var fetchConfigReceivedArg: UInt8?
    var fetchConfigReceivedInvocations: [UInt8] = []
    var fetchConfigReturnValue: [String: String]!
    var fetchConfigClosure: ((UInt8) async throws -> [String: String])?

    func fetchConfig(_ arg: UInt8) async throws -> [String: String] {
        fetchConfigCallsCount += 1
        fetchConfigReceivedArg = (arg)
        fetchConfigReceivedInvocations.append((arg))
        if fetchConfigClosure != nil {
            return try await fetchConfigClosure!(arg)
        } else {
            return fetchConfigReturnValue
        }
    }
}

And in your tests, you can use the spy to verify that your code is interacting correctly:

func testFetchConfig() async throws {
    let serviceSpy = ServiceProtocolSpy()
    let sut = ViewModel(service: serviceSpy)

    serviceSpy.fetchConfigWithArgReturnValue = ["key": "value"]

    try await sut.fetchConfig()

    XCTAssertEqual(serviceSpy.fetchConfigWithArgCallsCount, 1)
    XCTAssertEqual(serviceSpy.fetchConfigWithArgReceivedInvocations, [1])

    try await sut.saveConfig()

    XCTAssertEqual(serviceSpy.fetchConfigWithArgCallsCount, 2)
    XCTAssertEqual(serviceSpy.fetchConfigWithArgReceivedInvocations, [1, 1])
}

I encourage you all to try Spyable and to contribute to its development on GitHub. I would love to hear about your experiences, the pros and cons you've found, and any suggestions for improvements you might have. With your input, I believe we can make Spyable an even more robust tool for Swift testing.

Happy Coding! :man_technologist:

9 Likes

Looks very cool! Thank you :grin: It's awesome we can do this kind of stuff :star_struck:

Some projects may not want to ship the generated code on their app. Any thoughts on removing that for release builds?

Do you think there would be a way to generate the code on a different target?
What about protocol inheritance and protocol composition?

A lot of other edge cases, but the two issues above are the main things I'm interested in at this point. I'd love to get rid of external code generators.

2 Likes

That's an excellent question! I've given this some thought too. Personally, I wouldn't want to include the generated code in my main source code. Fortunately, macros generate code at compile time, so this code won't be visible in your git repository; you can't commit code generated by a macro to your repository (only usage of this code). This is a huge advantage for Spyable, particularly as in many of my projects where we've used Sourcery, we've often encountered issues with different AutoGenerated files in PR's, since not everyone enjoys running it every time they modify a protocol.

However, there is still one issue. Your auto-generated spy is visible from your source code, which means you could accidentally use it in production code. To mitigate this, during the design process of this macro, I decided to use internal modifiers to prevent accessing Spies from other targets. As we know, this requires the import {target_module_with_protocol} to be decorated with a @testable attribute. Ideally, as you suggested, it would be best to generate spies directly into the test target:

Regrettably, to the best of my knowledge, this is not currently possible.

Absolutely! There are indeed a lot of edge cases. If you have specific ones in mind, could you please create a new issue in the repo? I would truly appreciate it. :heart:

Would it be possible to generate the Spy between #if DEBUG ... #endif ?

So it's not included in the release binary

2 Likes

I can't understand why tests don't work on my Xcode 15.0

@testable import SpyableMacro

It says "No such module 'SpyableMacro'".

Any idea why?

It's interesting that you're facing this issue. I've just checked on my end and the tests are running fine both on my local machine and on the CI.

Here are some things that come to my mind that could be causing this:

  1. Make sure you don't have the swift-spyable SwiftPM project open at the same time as the Examples SwiftPM project. Xcode can only handle usage of the same SwiftPM for one project at a time.
  2. Ensure you're not running the tests on your iPhone. Logic testing is not available on iOS devices.
  3. Sometimes Xcode can get into a strange state. It can help to clean your project (Product -> Clean Build Folder or Shift + Command + K ) and then rebuild it (Product -> Build or Command +B ).
  4. Try deleting the Derived Data. This is another way to clean up state in Xcode, and can sometimes fix issues like this.

Let me know if any of these suggestions help. If not, we can continue troubleshooting.

Tried everything about clean, deleting derived data already and... turns out I was testing on iPhone simulator. Works great now, thanks!

It's great to hear that!

Let me know about your other thoughts about this macro! :heart:

1 Like

I wonder if there's a way to add this macro but only in a test environment. I don't think adding a macro that should be used for testing only should be added to production code.

Would it be possible to use this macro in a protocol extension perhaps? that extension could be defined in a file that lives only in the test sources.

For example

// In my library
protocol MyProtocol {}

and then

// On my tests
@Spyable
extension MyProtocol {}
1 Like

Thanks so much for your feedback!

As for the proposed example, I'm afraid it's not going to work as expected. Using a macro on an extension of a protocol in totally different file rather than the protocol declaration itself might seem like an elegant solution at first, but it's important to understand how macros actually work.

During compilation, Swift expands any macros in code before building code as usual. Macros operate on the Abstract Syntax Tree (AST), which is a tree representation of the syntactic structure of the code. Macros themselves do not have any knowledge about the types, classes, or interfaces - they only have access to this plain textual structure.

Therefore, attaching a macro to an extension won't have the effect you're hoping for, because the macro won't know anything about the extended protocol. The protocol and the extension are separate entities in the AST and the macro wouldn't be able to connect them.

As @mlienert mentioned:

We could indeed utilize compiler directives, specifically the #if directive, to check whether the DEBUG flag is enabled or not. This way, we can include the spy code only when the app is built in debug mode for testing. When the app is compiled for a release, the spy code would be ignored and wouldn't be a part of the final executable.

Here's an example of how this might look:

@Spyable 
protocol Foo { 
   func bar() 
} 
// expends to: 
#if DEBUG 
class FooSpy: Foo { ... } 
#endif

This strategy should help us keep our production codebase clean while still maintaining the convenience of having test spies during our development process. Please let me know if you have any thoughts or further questions on this approach!

3 Likes

Thanks! I've read about this but didn't quite click on my head until your explanation!!
I can totally see why I could't get my approach to work!
Thanks for taking the time really!

I'll try and add some ideas as PRs in your repo!

1 Like

I tried a few ideas over your code and I can't make them quite fit into your repo's way of facing the mocking, I ended up creating my library, as I already had something I used internally to manage mocks/callStacks/stubbing.
Open to anyone that wants to contribute as well GitHub - frugoman/SwiftMocks: Mocking framework for Swift using Macros

1 Like

I created a custom version of spyable in my project but I always get the following error:

See the elaborated problem description on my stack overflow

Could you please help?

Hi there!

It's great to see you on the Swift Forums! :heart:

Can you share on what platform are you building the SwiftPM?

Thanks for your quick response! :slightly_smiling_face:

platforms: [.iOS(.v14), .macOS(.v13)]

Hey hey, wanted to follow-up on my previous message. Do you have an idea what's going on? :slightly_smiling_face:

Hey @sharmin03,

Sorry, but I don't have the best news. I've been giving your issue a go, even got my buddies at work to give it a try too, but none of us can seem to run into the same problem. Also Spyable is building just fine on the CI.

I my pretty sure you have tried this thing, but if not there is a list of things that comes to my mind and potentially can help you:

  1. When you're picking your run destination, go with "My Mac".
  2. Ensure you're not running the tests on your iPhone. Logic testing is not available on iOS devices.
  3. Sometimes Xcode can get into a strange state. It can help to clean your project (Product -> Clean Build Folder or Shift + Command + K ) and then rebuild it (Product -> Build or Command +B ).
  4. Try deleting the Derived Data. This is another way to clean up state in Xcode, and can sometimes fix issues like this.
  5. Try creating a new macro from an Xcode template. You can do it by going to Xcode -> File -> New -> Package, then under the Multiplatform tab, choose Swift Macro. Make sure you can build it without any issues.

Hi @Matejkob,

Thanks a lot for your efforts and answering my question. :pray:
I tried for a couple of days every random thing and most of it didn't work.
At the end what worked was to upgrade to Xcode 15 beta 2 and disabling code coverage and now I am able to achieve what I want in the project. :tada:

1 Like

compiler does this for you

The purpose of writing all of those protocols (by hand) and all that "architecture" that we do is (mostly) to make tests possible. If you (hypothetically) remove the testing code, then all those protocols become obsolete.

And yet, you don't complain about all that test induced design damage but you complain about some generated code that will be removed by compiler anyway? Hm. Is it really a problem to stay there generated in the main target?

IMO considering the above, generated code is not even a problem.

1 Like