[Pitch] Declaration macros

Can I clarify one thing regarding the dictionary storage example; it shows there are 2 declared macros at one point, one with a key and one without.

@declaration(.attached, members: [.accessors]) macro dictionaryStorage
@declaration(.attached, members: [.accessors]) macro dictionaryStorage(key: String)

Is this illustrating that both need to be declared to support what appears to be an optional key? If that is the case can we not simply make the key optional and give it a default value of nil just like we can with standard functions so the key can be omitted if not required?

@declaration(.attached, members: [.accessors]) macro dictionaryStorage(key: String? = nil)

This feels a bit nicer instead of having to redeclare the macro twice and will allow greater flexibility for which arguments may need to be passed in to a macro or omitted.

I may have misunderstood, but if that is the case then maybe this can be cleared up in the pitch.

3 Likes

Sorry if this question is answered in the proposals but I'm having some trouble seeing the forest for the trees, and it's probably because I haven't spent enough time with them.

My main interest for macros and reflection is declarative REST API generation. Being able to produce an OpenAPI specification at runtime from the implementation using reflection, for example.

A short example from Java:

@Path("/hello")
public class GreetingResource {

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        return "Hello from RESTEasy Reactive";
    }
}

This is more about reflection than macros, but there's probably some interaction there where the macro has to leave something there for reflection to find and act on so at runtime the API routing can be built from what reflection discovers.

Once both reflection and macros are implemented, will this be possible?

I think the reason to declare the macro twice is to allow it to be used without parentheses, as in

@dictionaryStorage var name: String

With an optional key, it would have to be written as

@dictionaryStorage() var name: String

I have a question. Would a function body with an attached macro be type-checked before the macro expansion? An example would be generating API code via macro:

@API(.GET, endpoint: "https://example.com/users")
func getUsers() async throws -> [User] {}

// would expand to:

func getUsers() async throws -> [User] {
    // Fetch and decode users
    ...

    return users
}

In this case, would not having a return statement in the original code be an error?

And thinking about it, it would even be nicer if one could leave out the {} when declaring the function:

@API(.GET, endpoint: "https://example.com/users")
func getUsers() async throws -> [User]
2 Likes

Thanks for the answers Doug!

Sorry, I’m a bit slow today :brain::dash:, but does that mean we won’t be able to use macros to implement protocol conformances?

Nice :joy:

That's a great question. I think all of the arguments in favor of type checking the macro arguments before expanding the macro apply to the function body just as well: better diagnostics, macros only get well-formed inputs, easier to reason about what macros do, etc.

However, your question and @Alejandro_Martinez 's question about order-of-operation bring up a really important point. Right now, the compiler will only type-check the body of the function when it needs the body for something, i.e., to generate code for the function. We don't want to pay the cost of type checking the function body if we're only doing so to expand a macro for its peer declarations.

This is yet more evidence for...

i.e., we should separate out the ability to add or replace a function body from the ability to add peer declarations, attributes, etc., because they run at different conceptual times: we need peer declarations to do things like name lookup, we need attributes to understand the full signature of a type for type checking, and we need the function body to generate code. This might even mean that "peer" and "member" macro implementation entry points should be separate in the design :thinking: .

Right, it should be fine for a function-body-producing macro to provide a function body for a function that doesn't have one.

I believe this will be achievable by having a macro generate custom metadata attributes, as pitched elsewhere.

If we can't implement at least some protocol conformances with macros, we've gotten macros wrong. The question is how best to do it. With this pitch, you could create an attached macro that you place on the type itself, and which generates member declarations that correspond to the requirements of the protocol. That macro can use syntactic information from the type, e.g., it can walk through and find the declared properties. However, it can't reason about (e.g.) the effects of macros or property wrappers on the declared properties, so it's going to have rough edges. Perhaps that's okay, or perhaps it means we need a different model for things that want access to stored properties (protocol conformances, member wise initializers, etc.).

Doug

8 Likes

The first feature that came to mind as a use case for declaration macros was "newtype" with automatic protocol forwarding. As a quick sketch, something like:

@newType(basedOn: Int)
public struct SomeIndex: Comparable, Hashable {}

would synthesize something like this:

public struct SomeIndex: Comparable, Hashable {
  private var rawValue: Int

  public init(rawValue: Int) { self.rawValue = rawValue }

  public static func == (lhs: SomeIndex, rhs: SomeIndex) { lhs.rawValue == rhs.rawValue }
  public static func < (lhs: SomeIndex, rhs: SomeIndex) { lhs.rawValue < rhs.rawValue }
  public func hash(into hasher: inout Hasher) { rawValue.hash(into: &hasher) }
}

The big blocker here is that AFAICT we don't have the capability in these macros yet to examine the members of the protocols we want to conform to; we only have the macro's syntactic context. So I can know that I want to do something with "things" named Comparable and Hashable, but nothing else about them (I don't know what module they came from, whether they're protocols or something else, etc.).

As a general feature, I really like where this is going, and I think that that there are certain examples (like those highlighted in the proposal) where purely syntactic introspection can be powerful enough for what users need, but I wonder how many times users would hit a wall where we need some semantic data. Do you have any more insight here? (If you've got another pitch in your back pocket, just tell me to wait :grin: )

13 Likes

While I fear that this may delay the ability to work with functions, I think this is a good decision if we get a nicer API this way.

I feel that it could be valuable to have a general idea how type information would be supplied to the macro. It may have consequences for the design of the basic feature (even if type information is added in a future proposal). While thinking about and implementing different possible macros, I have repeatedly seen the need to have type info. I think, there are (at least) two distinct needs:

  1. Get the type of a (sub-)expression supplied to the macro. This can especially be useful for expression macros, but also for macros modifying function bodies. And since the compiler has already type-checked the code, this info should be readily available. A macro implementing something like function builders would definitely need this. It may be possible to somehow do some hacks involving type inference to make it work, but it wouldn't be nice. ( SE-0380 (if and switch expressions) could help, but I am not sure.)

    I imagine this API such that the macro can supply a sub-expression of the expression passed to the macro to the MacroExpansionContext and get the type of the expression back (at least its name, possibly more).

  2. Get infos about any given type or protocol (conformances, protocol requirements, members, etc.). I see this as generally valuable for all sorts of macros. This would be some sort of reflection API. It would be very nice if it could mirror the runtime reflection API as mentioned already in that thread. However, the currently pitched reflection API would not be enough because it does not supply conformances, computed properties and methods.

I guess that makes sense but it's still a bit of a rough edge. I wonder if there is an opportunity to smooth it out at this stage as I can imagine as this feature gains wider adoption it will become a common use case.

Challenge accepted, I guess? I went ahead and did the full implementation of AddCompletionHandler, as my test case for the pull request that starts implementing peer declaration macros at a syntactic level.

It's about a hundred lines of syntax manipulation. There are some obvious little utility functions we could build to make this easier (e.g., "build a forwarding call to this function"), and we could drop all of the the trivia manipulation by having swift-format clean up after you at the end, but for the most part it's straightforward: form the completion-handler parameter, drop the result type, drop the attribute, form the call, etc. Iteration on this kind of syntax macro development is fast, because you're plucking the bits you care about from the existing syntax tree and interpolating them into strings to make more syntax nodes. The new parser catches any mistakes quickly and gives nice diagnostics.

Full implementation of `AddCompletionHandler`
public struct AddCompletionHandler: PeerDeclarationMacro {
   public static func expansion(
     of node: CustomAttributeSyntax,
     attachedTo declaration: DeclSyntax,
     in context: inout MacroExpansionContext
   ) throws -> [DeclSyntax] {
     // Only on functions at the moment. We could handle initializers as well
     // with a bit of work.
     guard let funcDecl = declaration.as(FunctionDeclSyntax.self) else {
       throw CustomError.message("@addCompletionHandler only works on functions")
     }

     // This only makes sense for async functions.
     if funcDecl.signature.asyncOrReasyncKeyword == nil {
       throw CustomError.message(
         "@addCompletionHandler requires an async function"
       )
     }

     // Form the completion handler parameter.
     let resultType: TypeSyntax? = funcDecl.signature.output?.returnType.withoutTrivia()

     let completionHandlerParam =
       FunctionParameterSyntax(
         firstName: .identifier("completionHandler"),
         colon: .colonToken(trailingTrivia: .space),
         type: "(\(resultType ?? "")) -> Void" as TypeSyntax
       )

     // Add the completion handler parameter to the parameter list.
     let parameterList = funcDecl.signature.input.parameterList
     let newParameterList: FunctionParameterListSyntax
     if let lastParam = parameterList.last {
       // We need to add a trailing comma to the preceding list.
       newParameterList = parameterList.removingLast()
         .appending(
           lastParam.withTrailingComma(
             .commaToken(trailingTrivia: .space)
           )
         )
         .appending(completionHandlerParam)
     } else {
       newParameterList = parameterList.appending(completionHandlerParam)
     }

     let callArguments: [String] = try parameterList.map { param in
       guard let argName = param.secondName ?? param.firstName else {
         throw CustomError.message(
           "@addCompletionHandler argument must have a name"
         )
       }

       if let paramName = param.firstName, paramName.text != "_" {
         return "\(paramName.withoutTrivia()): \(argName.withoutTrivia())"
       }

       return "\(argName.withoutTrivia())"
     }

     let call: ExprSyntax =
       "\(funcDecl.identifier)(\(raw: callArguments.joined(separator: ", ")))"

     // FIXME: We should make CodeBlockSyntax ExpressibleByStringInterpolation,
     // so that the full body could go here.
     let newBody: ExprSyntax =
       """

         Task {
           completionHandler(await \(call))
         }

       """

     // Drop the @addCompletionHandler attribute from the new declaration.
     let newAttributeList = AttributeListSyntax(
       funcDecl.attributes?.filter {
         guard case let .customAttribute(customAttr) = $0 else {
           return true
         }

         return customAttr != node
       } ?? []
     )

     let newFunc =
       funcDecl
       .withSignature(
         funcDecl.signature
           .withAsyncOrReasyncKeyword(nil)  // drop async
           .withOutput(nil)  // drop result type
           .withInput(  // add completion handler parameter
             funcDecl.signature.input.withParameterList(newParameterList)
               .withoutTrailingTrivia()
           )
       )
       .withBody(
         CodeBlockSyntax(
           leftBrace: .leftBraceToken(leadingTrivia: .space),
           statements: CodeBlockItemListSyntax(
             [CodeBlockItemSyntax(item: .expr(newBody))]
           ),
           rightBrace: .rightBraceToken(leadingTrivia: .newline)
         )
       )
       .withAttributes(newAttributeList)
       .withLeadingTrivia(.newlines(2))

     return [DeclSyntax(newFunc)]
   }
 }

Doug

14 Likes

If that is true that macros should be able to alter members it then begs a couple of associated questions:

  1. Does it make sense for declaration macros to be applicable to all declarations? e.g. You have an example for an enum, but what about structures? (this one 100% makes sense since enums are just structures in a trench coat) What about classes? If so then what about also applying to actors?
  2. If all of those in the previous question are a yes (which I would expect): Should declaration macros be applicable to protocols?
  3. Applying to a protocol would then be able to be done as an extension to then provide a default implementation
  4. Are the emissions from the macro controllable to the access control of the emission? e.g. can a macro emit things that would be private to the declaration it is applied to? For example can we use a macro to inject additional storage into a type?

If all of these cases are true, then this is definitely a superset of the type wrapper feature in my view and would likely solve some edge cases in type wrappers and allow some pretty powerful advancements beyond the current design.

Having these two features collapse into one would not only make it more approachable for developers to only need to learn 1 thing, it would also mean only one point of maintenance and behavior. In my view this would be distinctly desirable since it would ensure cohesiveness in the expectations developers would have in how things work.

2 Likes

I'm still fretting about this, because making arbitrary alterations to members seems like a significant non-local effect. Perhaps there's another way to spin this: some attributes that are placed on a type or extension are also implicitly applied to the members within that type or extension definition. We could say that some attached macros work this way---perhaps as an opt-in---so (say) an attached declaration attribute could apply to type and would also be run on members of that type, so something like:

@accessorizeMyProperties
struct MyType {
  var x: Int
  var y: Int
  func f() { }
}

would apply the macro accessorizeMyProperties to x, y, and f as well as MyType. The macro implementation itself would have to decide what declarations it cares about---so perhaps it does nothing to f, but does something else to x and y (apply a property wrapper attribute, or some other attribute, perhaps). Or maybe there's some other way to get this effect.

Pretty much everything. Some particular combinations won't make sense---you can't add a function body to a struct, or add members to a typealias---but especially for things like "peer declarations", most declarations can be alongside other declarations, so they're quite general.

Yes, protocols are declarations.

I think it's likely to be important for macros to be able to introduce extensions. I suspect a peer-declaration macro will be able to do it, but I want to dig into the implementation further to be sure.

Yes, that should be fine.

Doug

2 Likes

I think there's an unstated "if you're used to working on a compiler" after "straightforward", but maybe that's totally fine. If the goal here was to turn Swift into a Lisp and have writing macros be a normal thing that every developer does on a regular basis then I don't think this approach would work at all, but with macros intended to be a rarer thing then my concern is more around how easy it is to understand what the macro is doing without prior experience and this seems fine on that front.

Removing the need for the trivia manipulation would probably help quite a bit with that too; if you're used to looking at code using SwiftSyntax I assume your eyes just automatically skip all of the withoutTrivia() etc. calls, but for someone new to it that's a lot of extra incidental code to read on the way to understanding what's actually happening.

5 Likes

Definitely. I probably spent a quarter of my time on this messing with trivia :slight_smile:. It's something we can improve on via the swift-syntax APIs over time.

Doug

Hey all,

I appreciate all of the design discussion here! I've gone ahead and revised the pitch. The changes are, roughly:

  • Split peer/member/accessor macro implementations into separate protocols and attribute spellings, so the compiler can query them in a more fine-grained manner.
  • Removed function-body macros... for the moment. We'll come back to them.
  • Add example showing composition of different macro roles for the same macro to effect property-wrappers behavior.

That last example is really fun. Alongside this, @hborla and I have prototyped some of these syntactic transformations in swift-syntax, and @rxwei has made progress on an implementation of freestanding macros in the compiler:

EDIT: Lots more interesting cases to consider, so I've updated the document again with:

  • "Body" macros to supply or alter the body of a function/initializer/accessor/closure.
  • Default-witness macros to help with synthesis of protocols.
  • Member-attribute macros to allow one to add attributes to the members of a type/extension.

This is too big for one proposal, but I'd rather not trickle out ideas one-by-one. Rather, there's a lot in here covering a large space of what is possible, and we can tease it apart later into more-easily-reviewable chunks.

Doug

10 Likes

The changes look really great! Thank you! Some remarks:

  • The peersOf:, membersOf:, accessorsOf:, bodyOf: and memberAttributesOf: labels sound a bit clunky to me. They prevent the function to be read like a sentence. Maybe providingPeersOf: or generatingPeersOf: would be a better fit?

  • I would appreciate seeing an example of how a body macro applied to a closure would look like.

  • I think it would be valuable for default witness macros to get more information about the conforming type. Currently, the macro wouldn't even know if the type is a value or a reference type if I am not mistaken. Having at least the spine of the parents of the node where the generated witness would be expanded (as suggested by you for expression macros) would be a good start. Or getting the type definition or extension syntax including conformance list and the definitions inside just as a member attribute macro would get would be even better.

  • If multiple macros are attached to the same definition, how would they be expanded?

I tried to implement a custom type wrapper syntactic transform as a combination of attached macros in the current swift-syntax prototype, and I got pretty far!

The type wrapper macro struct conforms to MemberDeclarationMacro, MemberAttributeMacro, and AccessorDeclarationMacro. Each macro capability provides a different part of the type wrapper transform:

  • MemberDeclarationMacro adds the backing var _storage variable.
  • MemberAttributeMacro applies macro attributes to each stored property inside the wrapped type, which are recursively expanded.
  • AccessorDeclarationMacro uses the macros applied to stored properties to add get and set accessors, turning those stored properties into computed properties that indirect access through _storage.

Using this macro transform, I can transform this type:

@customTypeWrapper
struct Point {
  var x: Int
  var y: Int
}

into

struct Point {
  var x: Int {
    get {
      _storage[wrappedKeyPath: \.x]
    }
    set {
      _storage[wrappedKeyPath: \.x] = newValue
    }
  }

  var y: Int {
    get {
      _storage[wrappedKeyPath: \.y]
    }
    set {
      _storage[wrappedKeyPath: \.y] = newValue
    }
  }

  var _storage: Wrapper<Self>
}

Check out my progress here: [Macros] Implement a type wrapper transformation as a macro. by hborla · Pull Request #1225 · apple/swift-syntax · GitHub

4 Likes

Is there enough info to transform this:

@customCompressedTypeWrapper
struct Point {
  var x: Int
  var y: Int
}

Into this?

struct Point {
  private struct Storage {
    var x: Int
    var y: Int
  }
  private var _storage: Storage

  init(x: Int, y: Int) {
    _storage = Storage(x: x, y: y)
  }

  var x: Int {
    get { _storage.x }
    set { _storage.x  = newValue }
  }

  var y: Int {
    get { _storage.y }
    set { _storage.y  = newValue }
  }
}

That way we can avoid building key paths and also avoid internal storage types from leaking out accidentally. Overall this approach to me feels distinctly more flexible, more performant, and safer.

1 Like

Yes, I believe so. When adding nested members during the expansion of a MemberDeclarationMacro, the macro is able to iterate over all existing members in order to create new members. Once expansion is hooked up to the compiler, I think the compiler can provide limited type information for existing declarations so if you write e.g. var x = 0, you will still be able to generate var x: Int in that nested Storage type.

1 Like

I like this, because it makes it clear that's the purpose of the expansion.

Sure, it be something like { @MyResultBuilderMacro in ... }.

Right, we should have an extensible structure here so that we can provide whatever information is reasonable, and expand it over time as needed.

I think we'll end up going left-to-right, since that's how evaluation order works in Swift everywhere else. You're right that I need to specify that, and there are other interesting ordering constraints that need to be specified in this document.

Doug

5 Likes