Generate Swift Testing code with Macros

I'm looking in to Macros and using them with the new Swift Testing.

Ideally I'd like to make a @SnapshotTest custom macro that does a lot of business logic and spits out some snapshots using the Pointfree library,

However, it looks like I've fallen at the first hurdle and @Test cannot be used when output my a Macro?

Below is my code in the Macro plugin's main.swift to try this out...

@Suite
@MainActor
class MySuite {
    @SnapshotTest
    func someTest() {
        // TODO: Implement
    }

    // Expanded macro output:
    //    @Test
    //    func someTestGenerated() {
    //        // TODO: Implement
    //    }

    // ❌ Compiler error:
    // @Suite
@MainActor
class MySuite {
    @SnapshotTest
    func someTest() {
        // TODO: Implement
    }

    // Expanded macro output:
//    @Test
//    func someTestGenerated() {
//        // TODO: Implement
//    }

    // ❌ Compiler error:
    // /var/folders/m3/0_3zpbjj1ks177l1p___zyy40000gq/T/swift-generated-sources/@__swiftmacro_19SnapshotSuiteClient02MyB0C17someTestGenerated0F0fMp_.swift:21:19
    // Instance member '$s19SnapshotSuiteClient02MyB0C17someTestGenerated0F0fMp_23funcsomeTestGenerated__fMu_' of type 'MySuite' cannot be used on instance of nested type 'MySuite.$s19SnapshotSuiteClient02MyB0C17someTestGenerated0F0fMp_53__🟠$test_container__function__funcsomeTestGenerated__fMu_'
}

}

Is there something I can do here to get what I want?

Or is this a current limitation of Testing/Macros?

For what it's worth, my Macro declaration is here and pretty simple at the moment:

@attached(peer, names: suffixed(Generated))
public macro SnapshotTest() = #externalMacro(module: "SnapshotSuiteMacros", type: "SnapshotTestMacro")

I think I'd need to see more of your macro implementation to understand exactly what's failing, but based on the compiler error you're seeing I am assuming that your macro tries to add a nested Thing type and then use the @Test macro from there, or something along those lines? I wouldn't expect that to work because the nested Thing type won't be able to call arbitrary instance methods on MySuite any more than it could if it were declared in "normal" Swift rather than a macro.

Happy to help you debug this issue and try to find a solution—feel free to DM me either here or on one of the various Swift Slack spaces that are floating around out there. :smile:

Hey @grynspan thanks for the quick reply,

My bad - ‘Thing’ should be ‘MySuite’, just updated to be right.

That teaches me for trying to make my code look nicer when posting it and trying to disguise my ugly naming of Thing to MySuite :joy:

In terms of my Macro implementation, for now it just returns a hardcoded string with nothing else going on there, so that output should always be the same.

I can post the code later though when I get back to my laptop if it helps.

Just to clarify, all the produced code from the macro should be inside of the ‘MySuite’ scope, so AFAIK there should be no scope issues, given that when I copy past the expanded macro directly in to MySuite this all works as expected :thinking:

Without seeing the actual macro core or expansion, I can really only guess at what's happening. Let's discuss in DMs.

1 Like

FYI for any future travellers here: Swift Testing Macros - empty MacroExpansionContext.lexicalContext · Issue #2967 · swiftlang/swift-syntax · GitHub

1 Like

Following on from this, I'm adding a configurations argument to my macro to pass down to the @Test's arguments parameter, but I'm seeing an issue possibly similar to the one above.

The expanded macro code is below and is pretty simple.

The idea is to create a Container type inside the suite to hold the configurations and pass them in to @Test compatible arguments.

I'll be using the configuration.name separately in my code to generate each individual image name (tbd), but for now the values should be piped through to the @Test's arguments by pulling them from Container.values().

I've tried a few container types and vars instead of funcs, but the issue seems deeper than that.

I can generate the code fine, but I'm getting a compiler error when the Macro code is being compiled in my unit tests:

Cannot find '$s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_' in scope

Below is all my code, I've added code expansion and compiler errors, I've also added a static name for the container type in case there was some issue with using unique names like I have (although I don't think this is the case).

@Suite
@MainActor
class MySnapshots {
    @SnapshotTest(
        "SwiftUI View",
        configurations: [
            SnapshotTestConfiguration(name: "Name 1", value: "One"),
            SnapshotTestConfiguration(name: "Name 2", value: "Two"),
            SnapshotTestConfiguration(name: "Name 3", value: "Three"),
        ]
    )
    func makeSwiftUISut(value: String) -> some View {
        Text(value)
    }

    // Expanded macro code (copy pasted from Expand Macro in Xcode) ...

    @MainActor
    @Suite("SwiftUI View")
    struct $s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_22SwiftUI_View_containerfMu_ {
        @Test("SwiftUI View", .tags(.snapshot), arguments: $s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_.values())
        func $s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_25SwiftUI_View_snapshotTestfMu_(value: String) throws {
            SnapshotSuite.assertSnapshotTest(
                name: "SwiftUI View",
                traits: [.theme(.all)],
                recording: false
            ) {
                Text(value)
            }
        }

        private enum $s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_ {
            static func testConfigurations() -> [SnapshotSuite.SnapshotTestConfiguration<String>] {
                [.init(name: "Name 1", value: "One"), .init(name: "Name 2", value: "Two"), .init(name: "Name 3", value: "Three")]
            }

            static func values() -> [String] {
                testConfigurations().map {
                    $0.value
                }
            }
        }
    }

    // ❌ Xcode error:

    // Cannot find '$s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_' in scope
    // In expansion of macro 'SnapshotTest' on instance method 'makeSwiftUISut(value:)' here


    // ❌ Build error details:

    /*

     @__swiftmacro_23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_.swift:4:56: error: cannot find '$s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_' in scope
     @Test("SwiftUI View", .tags(.snapshot), arguments: $s23SnapshottingSampleTests11MySnapshotsC14makeSwiftUISut12SnapshotTestfMp_36SwiftUI_View_configurationsContainerfMu_.values())
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     /Users/adam.carter/path/to/file/ButtonStyleSnapshotSpec.swift:95:5: note: in expansion of macro 'SnapshotTest' on instance method 'makeSwiftUISut(value:)' here
     @SnapshotTest(
     ^~~~~~~~~~~~~~
     /Users/adam.carter/path/to/file/ButtonStyleSnapshotSpec.swift:95:5: note: in expansion of macro 'SnapshotTest' on instance method 'makeSwiftUISut(value:)' here
     @SnapshotTest(
     ^~~~~~~~~~~~~~

     */
}

NOTE: When copying and pasting the expanded Macro (and manually changing the unique names to standard names) this code does all work as expected.

As far as I can see the container should sit in the same scope as @Test, both inside of MySuite, but I'm assuming there's some implementation details here that means this doesn't work in my Macro - having a glance at the Swift Testing library there's a __TestContainer enum generated which might not have the right access?

The only workaround I can see for now is to completely remove the container type and directly put the configurations code in to the @Test arguments to produce the expanded macro code like so:

@Test("Name", arguments: [SnapshotTestConfiguration(), SnapshotTestConfiguration()])

But aside from being kind of ugly, this could lead to some copy/pasting and bad code in future as features improve. It also prevents the user from doing .init() for the configurations as the array doesn't have a type so the compiler gets confused. Eg:

@Test("Name", arguments: [.init(), init()]) // ❌ Compiler doesn't know what type the array is

For completeness, the below does work when I copy and paste the expanded macro out (and manually update the unique names to more standard names).

Any ideas or input would be great as I'm not familiar with Swift Testing in depth, despite the open source project being available.

I've had a glance, but some more expertise on the project's internals would be great.

Thanks.

// ✅ Working code when expanded macro, copy paste, then manually change unique names to standard names

@MainActor
@Suite("SwiftUI View")
struct MySuite {
    @Test("SwiftUI View", .tags(.snapshot), arguments: MyContainer.values())
    func makeSwiftUISutExpanded(value: String) throws {
        SnapshotSuite.assertSnapshotTest(
            name: "SwiftUI View",
            traits: [.theme(.all)],
            recording: false
        ) {
            Text(value)
        }
    }

    private enum MyContainer {
        static func testConfigurations() -> [SnapshotSuite.SnapshotTestConfiguration<String>] {
            [.init(name: "Name 1", value: "One"), .init(name: "Name 2", value: "Two"), .init(name: "Name 3", value: "Three")]
        }

        static func values() -> [String] {
            testConfigurations().map {
                $0.value
            }
        }
    }
}

Update:
I should add, I've also tried adding my container to the same scope as MySuite instead of being inside of it, but no luck.

I believe this is a constraint, in some form, of how macros are expanded. @Douglas_Gregor are you able to provide any insight?

The __TestContainer-conforming enum is a side effect of how we currently emit test metadata into the binary. This type will go away in the future and be replaced by some in situ symbols, but we're not there yet.

Boring ABI plans/details here
  • I discussed the current design and future plans a bit in this older forum post.
  • We have documented our prospective "permanent" ABI here, which needs to be reviewed and approved by the testing workgroup once that's up and running. Note this design isn't finalized yet.
  • Emission of data conforming to the "new" ABI is dependent on the @section attribute which is being pitched here.
1 Like