[Pitch] Protocol Macros

Evolution Pitches
, , , ,
1

Introduction

Swift macros offers a variety of different macro types for performing code expansions for various use cases. One macro type that's discussed in the Macros Vision document but not yet implemented is the witness macro type, which is an attached macro that synthesizes requirements for members its attached to in protocol declarations.

While this attached macro type is certainly useful in its own right this document proposes a different, but related, macro type for declaring a protocol and its requirements where the invocation of the macro synthesizes all the requirements of the protocol.

Motivation

The kinds of macros that exist today make it simple to extend the Swift language in a highly customizable ways, which was previously only achievable through modification of the language/compiler itself. There's one feature, however, that's currently only available for select protocols that would be useful as its own macro type which we'll dub "Protocol Macros."

These macros are associated directly with a given protocol and invoked when that protocol is attached to a particular type. This macro then generates the requirements of the protocol it's associated with, fulfilling the protocol's requirements. Two standout examples of this kind of macro (which is currently implemented as a compiler/language feature, not via a macro like this) are the Encodable and Decodable protocols:

protocol Encodable {
    func encode(to encoder: Encoder) throws
}

protocol Decodable {
    init(from decoder: Decoder) throws
}

When either of these protocols are attached to a given type, the compiler synthesizes the protocol's requirements in the type its attached to. Not only that, but the compiler performs validation of the type to ensure that each of its stored members conform to the protocol which is required for being able to automatically synthesize the conformance. If the requirements can't be automatically synthesized, the requirements can be manually implemented on the type. Not only that but the protocol's requirements can be manually implemented even if the requirements could be synthesized by the compiler.

This kind of functionality can already be somewhat emulated using the 'conformance' field on the member or extension attached macro type. For example, we could create an attached member macro that conforms the attached type to a given protocol and synthesizes the requirements of the protocol:

// Declaration
protocol FooBar {
    associatedtype Foo
    associatedtype Bar

    func foo() -> Foo
    func bar() -> Bar
}

@attached(member, conformances: FooBar, named: named(foo()), named(bar()))
macro FooBarMacro() = #externalMacro(...)

...

// Implementation
struct EncodableMacro: ExtensionMacro {
    static func expansion(
        of node: AttributeSyntax,
        attachedTo declaration: some DeclGroupSyntax,
        providingExtensionsOf type: some TypeSyntaxProtocol,
        conformingTo protocols: [TypeSyntax],
        in context: some MacroExpansionContext
    ) throws -> [ExtensionDeclSyntax] {
        return [
            ExtensionDeclSyntax(
                ...
                inheritanceClause:
                    InheritanceClauseSyntax(inheritedTypes: 
                        InheritedTypeListSyntax(protocols.map {
                            InheritedTypeSyntax(type: $0)
                        })
                    ),
                ...
            )
        ]
    }
}

Although this works fine, it does lack some of the functionality that makes these kinds of protocols what they are today. Specifically, if one directly conforms to a given protocol (e.g. FooBar) then the protocol's requirements aren't automatically synthesized like they would have been had the macro been used intstead:

struct FooBarImplA: FooBar {
    // error: Missing implementations for `foo()` and `bar()`
}

@FooBarMacro
struct FooBarImplB {

}

// Conformance to `FooBar` + requirements synthesized in an extension of `FooBarImplB`:
//
// extension FooBarImplB: FooBar {
//     func foo() -> Self.Foo {
//         ...
//     }
//     func bar() -> Self.Bar {
//         ...
//     }
// }

Furthermore, this workaround doesn't allow for the associated protocol to be composed into other protocols or types like the way that Encodable and Decodable are composed in the Codable typealias:

typealias HashableFooBar = (FooBar & Hashable)
// OR: `protocol HashableFooBar: FooBar, Hashable { }`

struct FooBarImpl: HashableFooBar {
    func hash(into hasher: inout Hasher) {
        ...
    }

    // error: Missing implementations for `foo()` and `bar()`
}

Because of these drawbacks we are proposing a new protocol attached macro type that mirrors the behavior of these kind of special protocols where the macro is invoked by conforming a type to the protocol.

Proposed solution

The proposed solution is to create a new attached macro type conformance whose usage is restricted to protocol declarations:

@attached(conformance, ...)
protocol FooBar {
    ...
}

The implementation of the macro would then be invoked whenever the protocol is formally adopted on a conrete type, that is to say adopted by a class, struct, enum, or actor. When added to the inheritance clause of another protocol, the macro wouldn't be invoked. However, any conformance to the new protocol on a conrete would invoke the macro for that conrete type. For example:

@attached(conformance, ...)
protocol Foo {
    associatedtype Foo
    func foo() -> Foo
}

// `Foo`'s macro isn't invoked here
protocol FooBar: Foo {
    associatedtype Bar
    func bar() -> Bar
}

// `Foo`'s macro is invoked here and synthesizes the `foo()` requirement.
struct FooBarImpl: FooBar {
    func bar() -> Self.Bar {
        ...
    }
}

Detailed design

The new conformance attached macro type would accept two different parameters in the attribute's usage:

  1. additionalMembers: If provided, this field will accept a list of named parameters that specify any additional declarations that the macro will create. All of the declarations in the associated protocol are implicitly included and therefore it's not necessary to include them in this field. For example, if the Encodable protocol were declared as this kind of macro it would include an additional declaration for the CodingKeys enum that it creates:
@attached(conformance, additionalMembers: named(CodingKeys), macro: #externalMacro(...))
protocol Encodable {

    func encode(to encoder: Encoder) throws
}
  1. macro: This field is required to be include in the attribute and is provided with the implementation specification of the macro:
@attached(conformance, macro: #externalMacro(...))
protocol FooBar {
    ...
}

Next, a new macro protocol type would be created in the SwiftSyntax library for specifying the implementation of the macro:

public protocol ProtocolConformanceMacro: AttachedMacro {

    static func expansion(
        of protocol: ProtocolDeclSyntax,
        attachedTo declaration: some DeclGroupSyntax,
        in context: some MacroExpansionContext
    ) throws -> [DeclSyntax]
}

The parameters of the required expansion function are defined as follows:

  • protocol: In lieu of providing an AttributeSyntax like is done in all other attached macros, this function accepts the declaration of the protocol that this macro is being invoked.

  • declaration: The declaration field is provided with the concrete declaration type that the protocol is attached to. The AST this declaration would be equivalent to the declaration that would be provided to the invocation of a member or extension macro. That is to say that the full AST of the conforming type would be provided, not just the type of the declaration or a "skeleton" of the declaration.

  • context: The standard macro expansion context included in all other macro expansion functions.

Protocols declared in this manner effectively function as both a normal protocol and an attached macro. These protocols can be used identically to all other protocols and are semantically just protocols themselves, but with the addition of performing macro expansion on the types that conform to them.

12 Likes
2

How one would specify parameters for a macro? For your example suppose macros synthesising impls for Encodable and Decodable accept argument isInlinable which controls whether or not init(from:) and encode(to:) functions will be marked @inlinable.

1 Like
3

Very interesting pitch, thank you!

Encodable and Decodable can synthesize their conformance based on members that are declared in the same file as the conformance (not only in the declaration that declares the conformance):

// Synthesized
struct Foo: Codable { var name: String }

// Synthesized
struct Bar { var name: String }
extension Bar: Codable { }

// Not synthesized
extension DefinedInAnotherFile: Codable { }

It would be great if the protocol macros would support this feature. The expansion method would not only get the decl to which the protocol is attached, but also the decl for the original type (as long as it is declared in the same file).

Encodable and Decodable both synthesize a CodingKeys enum, without conflict:

// All these combinations synthesize a single `CodingKeys`
// enum in their target types:
struct A: Decodable { ... }

struct B: Codable { ... }

struct C: Decodable, Encodable { ... }

struct D { ... }
extension D: Decodable { }

struct E { ... }
extension E: Codable { }

struct F: Decodable { ... }
extension F: Encodable { }

struct G: { ... }
extension G: Decodable { }
extension G: Encodable { }

// I stop the enumeration of all cases here :-)
// With compiler-implemented Codable, all cases
// compile and do the right thing.

It would be great if the protocol macros would support this feature as well. This supposes that multiple protocol macros can play well together, when attached to the same type, on the type declaration or an extension, in any order.

4 Likes
4

Something that I admittedly don't have a great solution to. My thoughts went one of two ways:

  1. (Chosen for this proposal draft): Protocol macros just wouldn't accept arguments given that these are supposed to behave like protocols which don't accept parameters when included in inheritance clauses.

  2. The declaration of protocol macros would instead take the following form:

@attached(conformance, ...)
protocol FooBar {
    ... // protocol's requirements

    macro invoke(... /* macro parameters */) = #externalMacro(...)
}

or perhaps:

protocol FooBar {
    ... // protocol's requirements

    @attached(conformance, ...)
    macro invoke(... /* macro parameters */) = #externalMacro(...)
}

This form would instead have the requirement that the protocol have a macro with a specific title (e.g. invoke) that accepts the arguments that could be passed to the macro. The trouble that I had with this form was how this would be used in practice. My thoughts is that it would take one of the following two forms:

struct FooBarImplA: FooBar(inlinable: true) {
    ... // `FooBar`'s requirements are synthesized with the `@inlinable` attribute
}

struct FooBarImplB: @FooBar(inlinable: true) {
    ... // `FooBar`'s requirements are synthesized with the `@inlinable` attribute
}

Of the two forms I would think that the variation without the @ token (FooBar(inlinable: true)) would be the better choice, however, I don't really know how I feel about inheritances that accept parameters in this way. If you need parameters it would seem better to me to just create a member or extension macro that conforms the type to the desired protocol.

3 Likes
5

Something I definitely thought about but don't have the best answer to. IIRC the declarations that are passed into the macro implementations are completely detached from their parent context, which definitely makes these macros near impossible to implement exactly as currently proposes.

Specifically, the Decodable, Encodable, and Sendable protocols all need information about their stored variables to check that all stored variables on the type conform to the protocol as well. This information is entirely missing with the existing way in which macros are invoked.

In order to "recreate" these protocols using this proposal we'd need some kind of API that allows macro implementations to looks up information about a given type. For example:

struct TypeSyntaxInfo {
    struct TypeInheritanceInfo {
        let type: TypeSyntax
        let genericWhereClause: GenericWhereClauseSyntax?
        ...
    }

    var inheritances: [TypeInheritanceInfo]
    ...
}

extension MacroExpansionContext {
    func information(for type: TypeSyntax) -> TypeSyntaxInfo?
}

Or perhaps something akin to what's proposed here. In this way we would be able to get the inheritance information for each of the stored variables on the conforming type.

I haven't looked too deep into the exact point where macros are invoked by the compiler so it's entirely possible that compiler wouldn't have gather enough information to be able to supply the macro with this additional context, but this piece feels like it should be its own pitch given that querying the macro environment for additional information about provided types has much broader utility than just to this specific use case.

6

If macros had been implemented before automatic synthesis of Equatable, Hashable, Encodable and Decodable, would Swift even have automatic synthesis? Or would we just have conformance macros for them?

7

We agree on the diagnostic. I suggested another approach in my previous message, which was that the expansion method would also get the decl for the type declaration, as long as it belongs to the same file. Rough draft below with an extra typeDeclaration that is nil if and only if the type is declared in another file:

 public protocol ProtocolConformanceMacro: AttachedMacro {
     static func expansion(
         of protocol: ProtocolDeclSyntax,
         attachedTo declaration: some DeclGroupSyntax,
+        basedOn typeDeclaration: (some DeclGroupSyntax)?,
         in context: some MacroExpansionContext
     ) throws -> [DeclSyntax]
 }

This could provide a solution for a pattern that already has a solid tradition in the compiler (Equatable, Codable, etc. are only synthesised when conformed in the same file as the type declaration). Why not focus on this use case, since it has already been proven as very rich?

1 Like
8

I guess that would be larger up to the compiler devs, but assuming that each macro implementation could receive sufficient information about the conforming types' members it doesn't seem unreasonable that these could all be implemented as protocol macros

9

This seems like a very reasonable approach. It would allow for the protocol to be adopted on either the original type declaration or on a separate extension within the same file.

The only thing that this (along with the proposal itself) doesn't seem to address, at least for an implementation of one of the standard library protocols discussed above, is the matter of member introspection. If I were to want to automatically synthesize conformance to the Equatable protocol for a simple type:

struct Foo: Equatable {
    // ...
}

struct Bar: Equatable {
    let string: String
    let foo: Foo
}

For the Bar type, in the implementation of the macro I can inspect the type's members and see that it has two variable declarations, string and foo, but there's nothing that presently exists that can give me sufficient information about a given type for me to be able to determine whether or not the type(s) conform to the Equatable protocol, which needs to be known in order to synthesize the == operator.

The macro could just arbitrarily synthesize the operator's implementation anyways, but that would then lead to compilation errors of the synthesized declaration in lieu of the current error message that's generated today which indicates that the protocol's requirements aren't implemented. IMO this is a worse (albeit only slightly worse) experience in terms of diagnostics as it may confuse developers as to the root of the issue. Thoughts?

2 Likes
10

Thanks for pushing the exploration, and taking care of fellow developers :+1:

If macros could synthesize conditional declarations, we could avoid the compiler errors you mention, and mimic the current behavior of the compiler (when it does not synthesize user-provided declarations).

Let's try to mimic Equatable, and look at what an equivalent MacroPoweredEquatable protocol could do:

@attached(conformance, macro: ...)
protocol MacroPoweredEquatable {
    static func == (lhs: Self, rhs: Self) -> Bool
}

struct Foo: MacroPoweredEquatable {
    let name: String
    let score: Int
}

// --- Synthesized ---
extension Foo {    
    static func == (lhs: Foo, rhs: Foo) -> Bool
    where String: Equatable, Int: Equatable // ā¬…ļø
    {
        if lhs.name != rhs.name { return false }
        if lhs.score != rhs.score { return false }
        return true
    }
}

Of course this code does not compile today: we get a 'where' clause on non-generic member declaration requires a generic context on the where line.

But if I suppose that such an extension to SE-0267 were implemented, AND if the compiler disfavors such conditional declarations, then MacroPoweredEquatable behaves as expected:

// OK: compiles
struct Foo1: MacroPoweredEquatable {
    let name: String
    let score: Int
    
    // Conformance is synthesized
    // and enabled by the where clause.
}

// OK: compiles
struct Foo2: MacroPoweredEquatable {
    let name: String
    let score: Int
    
    // Conformance is synthesized
    // and enabled by the where clause,
    // but the compiler prefers the
    // non-conditional overload.
    static func == (lhs: Foo, rhs: Foo) -> Bool {
        // user-provided implementation
    }
}

// OK, because we have the expected compiler error
// Type 'Foo3' does not conform to protocol 'MacroPoweredEquatable'
struct Foo3: MacroPoweredEquatable {
    let name: NotEquable
    let score: Int

    // Conformance is synthesized
    // but not enabled due to its where clause:
    // the protocol conformance is incomplete.
}

So it looks like it is possible to improve the landscape with an extension to SE-0267, along with a sensible overload resolution. Such synthesized conditional conformances would fit well Equatable.==, Decodable.init(from:), and Encodable.encode(to:), all protected by a where clause that requires conformance of the involved properties.

Of course, this is not the end of the story :-)

  • Decodable needs to distinguish between optional and non-optional properties in order to synthesize decodeIfPresent or decode, and macros don't have access to this information :sweat_smile:
  • The type of properties declared with property wrappers is not always available for the macro, so it can't generate the where clause:
    struct Foo: MacroPoweredEquatable {
    @Surprise var lol
    // How to synthesize?
    static func == (lhs: Self, rhs: Self) -> Bool
    where ???: Equatable // šŸ˜¬
    { lhs.lol == rhs.lol }
}

Still, we may have ways to move forward.

Next, we could look for a way to synthesize "conditional new types", so that MacroPoweredDecodable and MacroPoweredEncodable can both generate non-conflicting CodingKeys enums :slight_smile:

1 Like
11

Thank you.

Thinking more about it, if it is cool to have access to the original type declaration in the same file, it is even more cool to have access to the declaration and all extensions :

 public protocol ProtocolConformanceMacro: AttachedMacro {
     static func expansion(
         of protocol: ProtocolDeclSyntax,
         attachedTo declaration: some DeclGroupSyntax,
+        basedOn otherDeclarations: [any DeclGroupSyntax],
         in context: some MacroExpansionContext
     ) throws -> [DeclSyntax]
 }

Having access to all extensions (in the same file) allows the macro to handle user-provided protocol requirements.

Let's consider Codable in the sample code below. See how it is necessary to have access to "Extension 1" in order to discover the customized CodingKeys:

struct Foo { ... } // Declaration
struct Foo {       // Extension 1
    enum CodingKeys: String, CodingKey { ... }
}
extension Foo: Decodable {
    // Synthesised code must know about the CodingKeys above.
}
extension Foo: Encodable {
    // Synthesised code must know about the CodingKeys above.
}
12

This is a solid extension on the existing syntax. In fact I think that this would be relevant outside of the current proposal. I have, on multiple occasions, found myself restricting macros that I've written from not being allowed to be applied to extension declarations since I didn't have the full context of the original declaration (e.g. the EnumSubset macro from the WWDC 2023 | Write Swift macros session).

@gwendal.roue IYO do you think that it would be worth splitting off into its own pitch?

1 Like
13

I really like this setup. It passes the responsibility of determining whether or not a setup like this could work to the compiler (arguably where it should be) and would provide a syntax, even if only usable in macro expansions like this, that can express this intent in a simple to understand way. Furthermore it allows the macro to provide an implementation without having to determine whether or not there's one already provided by the user and allowing the compiler to make that inference.

2 Likes
14

This seems like something that the macro would determine by iterating through all of the declarations for the provided type. One could imagine that after the first protocol macro is done creating its declarations the second protocol macro would be invoked and one of the declarations passed to otherDeclarations would be the extension (or extended declaration) that includes the just synthesized CodingKeys type.

Alternatively this could function comparable to how the extension or member macro function when adding conformances where there could be an additional parameter provided to the macro that contains the declarations that are named in the declaration of the protocol that haven't been fulfilled by other macro expansions or explicitly provided by the user:

@attached(conformance, ...)
protocol Encodable {
    func encode(to encoder: Encoder) throws
    associatedtype CodingKeys: CodingKey
}

@attached(conformance, ...)
protocol Decodable {
    init(from decoder: Decoder) throws
    associatedtype CodingKeys: CodingKey
}

The signature of for protocol macros would be amended in the following way:

 public protocol ProtocolConformanceMacro: AttachedMacro {
     static func expansion(
         of protocol: ProtocolDeclSyntax,
         attachedTo declaration: some DeclGroupSyntax,
         basedOn otherDeclarations: [any DeclGroupSyntax],
+        providingMembers: [any DeclSyntaxProtocol],
         in context: some MacroExpansionContext
     ) throws -> [DeclSyntax]
 }

Then let's say a type conformed to both protocols, expansion would happen in the following steps:

struct: FooBar: Encodable, Decodable {
    ...
}
  1. The Encodable protocol would be expanded first since it's first in the inheritance clause, but this detail wouldn't be guaranteed.
struct EncodableMacro: ProtocolConformanceMacro {
    static func expansion(
         of protocol: ProtocolDeclSyntax,
         attachedTo declaration: some DeclGroupSyntax,
         basedOn otherDeclarations: [any DeclGroupSyntax],
         providingMembers members: [any DeclSyntaxProtocol],
         in context: some MacroExpansionContext
     ) throws -> [DeclSyntax] {
         // `members` would be something to the effect of `[
         //     FunctionDeclSyntax(...) /* For the `encode(to:)` method */,
         //     AssociatedTypeDeclSyntax(...) /* for the `CodingKeys` type */,
         // ]

         ...
     }
}
  1. The Decodable protocol would then be expanded. Since the Encodable protocol was already expanded and already produced the CodingKeys type (or it was explicitly defined on the conforming type), it will be absent from the members parameter:
struct DecodableMacro: ProtocolConformanceMacro {
    static func expansion(
         of protocol: ProtocolDeclSyntax,
         attachedTo declaration: some DeclGroupSyntax,
         basedOn otherDeclarations: [any DeclGroupSyntax],
         providingMembers members: [any DeclSyntaxProtocol],
         in context: some MacroExpansionContext
     ) throws -> [DeclSyntax] {
         // `members` would be something to the effect of `[
         //     InitializerDeclSyntax(...) /* For the `init(from:)` initializer */,
         // ]

         ...
     }
}

Thoughts?

15

I don't know. A benefit of exploring what it means to give macros access to all extensions in a given file from this pitch thread is that it is possible to be focused. If members of the steering group feel like this feature should be generalized, they will chime in and provide guidance towards actual proposals.

There is still a lot to explore about giving access to extensions from a macro, because there are a lot of various kinds of extensions:

struct Foo<T> { ... }

extension Foo: P where T: Q { ... }

@available(...)
extension Foo: Q { ... }

extension Foo: MyMacroPoweredProtocol where ... { ... }

...
16

Looks correct :-)

17

Fair point. For the time being at least we could limit the scope of this to just this proposal and revisit it at a different time if it becomes relevant

18

Yes. I dare not pinging Doug Gregor or Holly Borla to get their feedback, but those are the people who seem like the most able to assess the relevance of this pitch. No doubt they have noticed this thread, so let's just be patient.

1 Like
19

I find this to be insanely useful in test mock generation if the protocol macro can be conditionally adopted on the implementation side.

If I want to make a macro that generates mocks in test only:

@attached(conformance, ...)
protocol FooBar {
  func requirement
}

In concrete class, we want to conform to FooBar ourselves

struct FooBarImpl: FooBar {
  func requirement(...) {
    ... (write your own impl)
  }
}

But with macro, we can generate test mock for the protocol

final class FooBarMock: FooBar {
  func requirement(...) {
    // record invocaion
  }
  // generated
  func stub_requirement(...) {
  }
}

The principle of a protocol is to leave the concrete implementation dynamic, and synthesizing the same implementation for every conformance seem to violate that best practice.

20

I truly love the idea! I believe it could offer significant value to GitHub - Matejkob/swift-spyable: Swift macro that simplifies and automates the process of creating spies for testing. However, I'm concerned we might encounter some constraints. I'd be keen to hear the thoughts of @Douglas_Gregor, @hborla, @ahoppen, and @beccadax on this matter

1 Like