[Pitch] Attached macros

Hey all,

We had some great discussion in the pitch on declaration macros. One of the things we realized in the discussion is that there is a big difference between "freestanding" and "attached" macros. I've pulled some discussion of that difference into the updated proposed vision, and am opening up different topics for the two.

This thread is about "attached" macros, which extends the custom attribute syntax (@something(args)) to enable macros to produce code that's associated with the declaration to which the attribute is attached. Attached macros are really powerful, because they're able to work with other existing source code to make additions. They can subsume much of the behavior of property wrappers, as well as being able to express operations like "add a completion-handler version of an async function".

Here is the full attached macros proposal. The prototype implementation is coming along but isn't really ready for much yet. I'll back when there's a toolchain that can demonstrate some of the things attached macros can do.

Doug

15 Likes

The new proposal looks good to me. I think that the last paragraph of the section ‘Member attribute macros' still needs to be updated.

I have thought about using macros to replace property wrappers. I believe we should allow attached macros to introduce declarations with $ prefixes to be on par with property wrappers.

Is there already an implementation we can use to test the proposed macro types (either for attached or freestanding macros)? I would love to get my hands on this to find out what is possible (if I find the time :sweat_smile:).

Will do!

Yes, I've been thinking about that. I think we could permit prefixed($) but not arbitrary $ names to stay within the spirit of the use of $ with property wrappers without throwing the doors open wide.

Why yes, there is! The February 2, 2023 snapshot can handle member, member-attribute, and accessor attached macros. I just merged a pull request that shows how to define some of these.

Peer attached macros are coming along, as are freestanding macros.

Looking forward to seeing what you build!

Doug

3 Likes

My feedback after playing with an attached macro implementation: I would like to be able to add attributes and protocol conformances with a macro.

Gory details…

One feature I've long wanted in Swift is a “newtype”-like facility to make it easier to get away from primitive obsession. Since there's little or no overhead to wrapping a single value in a struct, we can ‘manually’ implement a newtype; for example:

struct Hostname: RawRepresentable,
                 Equatable,
                 Hashable,
                 Comparable,
                 Codable,
                 CustomStringConvertible
{
  var rawValue: String
  init(rawValue: String) { self.rawValue = rawValue }

  static func == (lhs: Self, rhs: Self) -> Bool { lhs.rawValue == rhs.rawValue }
  static func < (lhs: Self, rhs: Self) -> Bool { lhs.rawValue < rhs.rawValue }
  func hash(into hasher: inout Hasher) { rawValue.hash(into: &hasher) }
  // etc. etc. tedious conformance boilerplate
}

But it's a lot of friction to write all that out. My dream syntax is

newtype Hostname: String,
                  Equatable,
                  Hashable,
                  Comparable,
                  Codable,
                  CustomStringConvertible

One way to reduce the boilerplate is to introduce a NewTypeProtocol that refines RawRepresentable and has extensions for other protocol conformances:

public protocol NewTypeProtocol: RawRepresentable {
  init(_ rawValue: RawValue)
}

extension NewTypeProtocol {
  public init(rawValue: RawValue) { self.init(rawValue) }
}

extension NewTypeProtocol where Self: Equatable, RawValue: Equatable {
  public static func == (lhs: Self, rhs: Self) -> Bool { lhs.rawValue == rhs.rawValue }
}

extension NewTypeProtocol where Self: Comparable, RawValue: Comparable {
  public static func < (lhs: Self, rhs: Self) -> Bool { lhs.rawValue < rhs.rawValue }
}

extension NewTypeProtocol where Self: Hashable, RawValue: Hashable {
  public func hash(into hasher: inout Hasher) { rawValue.hash(into: &hasher) }
}

// etc. etc.

This eliminates the boilerplate of the conformance requirements:

struct Hostname: NewTypeProtocol,
                 Equatable,
                 Hashable,
                 Comparable,
                 Codable,
                 CustomStringConvertible
{
  var rawValue: String
  init(_ rawValue: String) { self.rawValue = rawValue }

  // Most conformance boilerplate eliminated!
}

But the declarations of rawValue and init(_:) can't be eliminated this way.

As a workaround, Point-Free has the swift-tagged package. It still requires coming up with some extra type for each newtype to use as the tag, which can be boilerplate, and Tagged doesn't allow precise control of your newtype's conformances. I use my own variation that provides more precise control over protocol conformances than the swift-tagged package, but my scheme requires declaring a separate tag type for each newtype, which is definitely boilerplate!

So I took a stab at implementing newtype using an attached macro, declared as

@attached(member, names: arbitrary)
public macro NewType(_ type: Any.Type) = #externalMacro(module: "MacroExamplesPlugin", type: "NewTypeMacro")

and that gets my newtype declaration down to this:

@NewType(String.self)
struct Hostname: NewTypeProtocol,
                 Equatable,
                 Hashable,
                 Comparable,
                 Codable,
                 CustomStringConvertible
{ }

which is almost perfect, but:

  • The macro can't add the NewTypeProtocol conformance, so I have to declare it manually.

  • I have to specify String.self rather than just String, which is unfortunate. Maybe someday Swift will allow String by itself to mean what String.self does now. I tried to work around this by changing the declaration and use of the macro to

    @attached(member, names: arbitrary)
    public macro NewType<T>(_: T.Type = T.self) = #externalMacro(module: "MacroExamplesPlugin", type: "NewTypeMacro")
      
    @NewType<String>()
    struct Hostname: NewTypeProtocol,
      etc. etc.
    

    but swiftc crashed.

  • It would be nice, if vending a resilient library, to be able to add @frozen to the newtype declaration.

My macro implementation:

import SwiftSyntax
import SwiftSyntaxBuilder
import SwiftSyntaxMacros

public enum NewTypeMacro { }

extension NewTypeMacro: MemberMacro {
  public static func expansion<Declaration, Context>(
    of node: AttributeSyntax,
    providingMembersOf declaration: Declaration,
    in context: Context
  ) throws -> [DeclSyntax] where Declaration : DeclGroupSyntax, Context : MacroExpansionContext {
    do {
      guard
        case .argumentList(let arguments) = node.argument,
        arguments.count == 1,
        let memberAccessExn = arguments.first?
          .expression.as(MemberAccessExprSyntax.self),
        let rawType = memberAccessExn.base?.as(IdentifierExprSyntax.self)
      else {
        throw CustomError.message(#"@NewType requires the raw type as an argument, in the form "RawType.self"."#)
      }

      guard let declaration = declaration.as(StructDeclSyntax.self) else {
        throw CustomError.message("@NewType can only be applied to a struct declarations.")
      }

      let access = declaration.modifiers?.first(where: \.isNeededAccessLevelModifier)

      let decls = ("""
        {
          \(access)typealias RawValue = \(rawType)

          \(access)var rawValue: RawValue

          \(access)init(_ rawValue: RawValue) { self.rawValue = rawValue }
        }
        """ as MemberDeclBlockSyntax).members.map(\.decl)

      return decls
    } catch {
      print("--------------- throwing \(error)")
      throw error
    }
  }
}

extension DeclModifierSyntax {
  fileprivate var isNeededAccessLevelModifier: Bool {
    switch self.name.tokenKind {
    case .keyword(.public): return true
    default: return false
    }
  }
}

extension SyntaxStringInterpolation {
  // It would be nice for SwiftSyntaxBuilder to provide this out-of-the-box.
  fileprivate mutating func appendInterpolation<Node: SyntaxProtocol>(_ node: Node?) {
    if let node {
      appendInterpolation(node)
    }
  }
}
4 Likes

Protocols conformances seem like something we could handle, and they're in Future Directions because they should be straightforward.

Attributes are slightly trickier, because we have to cut off recursive expansion and think about ordering issues. OTOH, there are likely a lot of use cases for attributes, and we already have member attributes, so we should try to make it happen.

I think you can drop the parameter here entirely, and use

@attached(member, names: arbitrary)
public macro NewType<T>() = #externalMacro(module: "MacroExamplesPlugin", type: "NewTypeMacro")

but it appears that we need to implement this :)

error: generic parameter 'T' could not be inferred
@NewType<String>
^

Cool. Want to create a pull request to include this in swift-macro-examples?

Doug

3 Likes
1 Like

Thanks! Merged (and we'll look at fixing the bugs uncovered in this experiment).

Doug

3 Likes

I have played a little with the attached macros implementation. My goal was to see if it is feasible to implement Codable as a macro. I am aware that witness macros are a better fit for this job, but those are not implemented yet, so I used an attached macro. Here is the result:

@CustomCodable
struct Party: Codable {
  enum Theme: Codable {
    case halloween
    case birthday
    case piratesAndMonstersAndUnicornsAndEverythingElse
  }
  
  @CustomKey("host_name")
  var hostName: String
  var theme: Theme
  var numberOfGuests: Int = 0 {
    didSet {
      print("Someone came! Or went, I don't know...")
    }
  }
  
  var description: String {
    "\(hostName) has invited \(numberOfGuests) \(theme == .halloween ? "ghosts" : "guests")!"
  }
}

let party = Party(
  hostName: "Jim",
  theme: .halloween,
  numberOfGuests: 2
)
let encoder = JSONEncoder()
let partyData = try encoder.encode(party)
print(String(data: partyData, encoding: .utf8)!)
// Prints {"theme":{"halloween":{}},"host_name":"Jim","numberOfGuests":2}

let decoder = JSONDecoder()
let partyCopy = try decoder.decode(Party.self, from: partyData)
print(partyCopy.description) // Prints Jim has invited 2 ghosts!
Full implementation

In MacroExamplesLib:

@attached(member, names: named(encode), named(CodingKeys))
public macro CustomCodable() = #externalMacro(module: "MacroExamplesPlugin", type: "CustomCodableMacro")

/// This macro does nothing. It is only used as a hint for `@CustomCodable`.
@attached(accessor)
public macro CustomKey(_ key: String) = #externalMacro(module: "MacroExamplesPlugin", type: "CustomKeyDummyMacro")

public extension KeyedDecodingContainer {
  func _decodeInferredType<Value: Decodable>(forKey key: KeyedDecodingContainer<K>.Key) throws -> Value {
    try self.decode(Value.self, forKey: key)
  }
}

In MacroExamplesPlugin:

public enum CustomCodableMacro {}

extension CustomCodableMacro: MemberMacro {
  public static func expansion(
    of node: AttributeSyntax,
    providingMembersOf declaration: some DeclGroupSyntax,
    in context: some MacroExpansionContext
  ) throws -> [DeclSyntax] {
    
    let storedProperties = declaration.members.members.compactMap {
      StoredProperty(potentialPropertyDeclaration: $0.decl, diagnosticHandler: context.diagnose)
    }
    
    let codingKeyDefinitions = storedProperties.map { property in
      "\(property.name) = \(property.keyLiteral)"
    }
    
    let codingKeysSyntax: DeclSyntax = """
      enum CodingKeys: String, CodingKey {
        case \(raw: codingKeyDefinitions.joined(separator: ", "))
      }
      """
    
    let encodingCode = storedProperties
      .map { property in
        "  try container.encode(\(property.name), forKey: .\(property.name))"
      }
      .joined(separator: "\n")
    
    let encodeFunctionSyntax: DeclSyntax = """
      func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
      \(raw: encodingCode)
      }
      """
    
    let decodingCode = storedProperties
      .map { property in
        "  self.\(property.name) = try container._decodeInferredType(forKey: .\(property.name))"
      }
      .joined(separator: "\n")
    
    let decodingInitializerSyntax: DeclSyntax = """
      init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
      \(raw: decodingCode)
      }
      """
    
    let propertiesWithTypeAnnotations = storedProperties.filter { $0.typeName != nil }
    
    let parameterList = propertiesWithTypeAnnotations
      .map { property in
        "\(property.name): \(property.typeName!)\(property.initializer.map { " " + $0 } ?? "")"
      }
      .joined(separator: ", ")
    
    let propertyAssignmentList = propertiesWithTypeAnnotations
      .map { property in
        "self.\(property.name) = \(property.name)"
      }
      .joined(separator: "\n  ")
    
    let memberwiseInitializer: DeclSyntax = """
      init(
        \(raw: parameterList)
      ) {
        \(raw: propertyAssignmentList)
      }
      """
    
    return [codingKeysSyntax, encodeFunctionSyntax, decodingInitializerSyntax, memberwiseInitializer]
  }
}

// MARK: - StoredProperty

fileprivate struct StoredProperty {
  var name: String
  var typeName: String?
  var keyLiteral: String
  var initializer: String?
}

extension StoredProperty {
  init?(potentialPropertyDeclaration: DeclSyntax, diagnosticHandler: (Diagnostic) -> Void) {
    guard let property = potentialPropertyDeclaration.as(VariableDeclSyntax.self) else {
      return nil
    }
    
    self.init(property, diagnosticHandler: diagnosticHandler)
  }
  
  init?(_ property: VariableDeclSyntax, diagnosticHandler: (Diagnostic) -> Void) {
    guard
      property.isStoredProperty,
      let binding = property.bindings.first,
      let identifier = binding.pattern.as(IdentifierPatternSyntax.self)?.identifier
    else {
      return nil
    }
    
    let attributes = property.attributes ?? []
    let key = attributes
      .compactMap { syntax -> String? in
        guard
          let attributeSyntax = syntax.as(AttributeSyntax.self),
          attributeSyntax.attributeName.trimmedDescription == "CustomKey",
          let argument = attributeSyntax.argument
        else {
          return nil
        }
        
        return argument.trimmedDescription
      }
      .first
    
    self.name = identifier.text
    self.typeName = binding.typeAnnotation?.type.trimmedDescription
    self.keyLiteral = key ?? "\"\(identifier.text)\""
    self.initializer = binding.initializer?.trimmedDescription
    
    if typeName == nil {
      let message = SimpleDiagnosticMessage(
        message: "@CustomCodable requires that all properties have an explicit type annotation",
        diagnosticID: .init(domain: "test", id: "error"),
        severity: .warning
      )
      diagnosticHandler(.init(node: binding.initializer?.as(Syntax.self) ?? property.as(Syntax.self)!, message: message))
    }
  }
}

// MARK: - CustomKeyDummyMacro

/// This macro does nothing. It is only used as a hint for `CustomCodableMacro`.
public enum CustomKeyDummyMacro {}

extension CustomKeyDummyMacro: AccessorMacro {
  public static func expansion(
    of node: AttributeSyntax,
    providingAccessorsOf declaration: some DeclSyntaxProtocol,
    in context: some MacroExpansionContext
  ) throws -> [AccessorDeclSyntax] {
    return []
  }
}

The macro only replaces CodingKeys, encode(to:) and init(from:) and uses the other parts of Codable as is. However, I think this is enough to see if macros are feasible for the job. As a test case, I have added the ability to use a custom key using @CustomKey (see hostName in the example), which I believe is not possible with the default Codable implementation.

I have learned several things. First, the positive:

  • It works!

  • Implementing CodingKeys and encode(to:) was straightforward (apart from finding the types and properties I needed in SwiftSyntax).

  • When using the current snapshot, if there is a compilation error or crash, Xcode directly points to the expanded code in a buffer. This is really great for debugging!

Now the points that could be improved:

  • When generating init(from:), the macro can't just use KeyedDecodingContainer.decode(_:forKey:) because it does not know the type of properties that have a default value and use type inference. I have circumvented the issue by defining a helper method and (mis)using type inference:

    public extension KeyedDecodingContainer {
      func _decodeInferredType<Value: Decodable>(forKey key: KeyedDecodingContainer<K>.Key) throws -> Value {
        try self.decode(Value.self, forKey: key)
      }
    }
    

    This seems like a brittle solution.

  • Because the macro has to generate init(from:), the default memberwise initializer is no longer automatically generated. I think this is a problem. My macro could have used a peer macro for this if they had already been implemented. However, the canonical macro type for Codable would be witness macros, which are from my understanding not able to generate a peer. We definitely need a solution for this problem!

    I ended up needing to generate the memberwise initializer myself. Here, again, the problem were properties that have a default value and use type inference because the macro didn't know the types of these. I decided to only add initializer parameters for the properties with explicit types and generate warnings for those properties without. Generating the warnings was easy.

Implementing this macro reinforced my impressions that the proposed macro system is very powerful but what it misses most is type information. For this macro, it would have sufficed to provide a list of stored properties to the macro as proposed for witness macros. However, other macros may need other types of type information.

4 Likes

Yeah, it's pretty nice. Since this landed, I've also found that I no longer ever look at the output of -dump-macro-expansions, when things go wrong it's generally a type error and I can see it.

Oh, that's interesting! Should a macro-generated initializer suppress the default member wise initializer? I... don't really know. From the perspective that you should be able to expand the macro and have unchanged behavior, it's doing the right thing. But it's certainly annoying for your use case.

Yeah, the types of stored properties would be very, very useful.

Doug

1 Like

The implementation is much further along now. We have member, member-attribute, peer, and accessor macros all implemented, and there are examples in the example repository.

Doug

7 Likes

If the answer is “yes” it seems like it wouldn’t be too difficult to write a macro to synthesize a memberwise init (though synthesizing the same one that the compiler does might be tricky… there’s a lot of complexity around how the memberwise init interacts with property wrappers that I’m not sure macros could replicate).

Tangentially related, should macros be able to abstract over access control levels? One of the reasons that we've always given for not synthesizing a public memberwise init is that the user should have to explicitly opt-in, but if we had the memberwise init as a macro, should it be possible for a user to write something like:

@memberwiseInit(access: .public)
struct S {
  var x: Int
}

@memberwiseInit(access: .private)
struct R {
  var y: Int

  static var specialValue: Self { .init(y: 42) }
}

?

It seems like it would be broadly useful for a macro author to let clients control the visibility of whatever members it might introduce.

2 Likes

Yeah, sure. Your macro gets to generate the init declaration with whatever access modifier it wants. It can't change the access modifier of an existing declaration, though.

Doug

2 Likes

In a discussion elsewhere, I asked Doug what happens when an attached macro with multiple roles is applied to a declaration for which some, but not all, of those roles are valid. For example, suppose you used the pitch's Clamping macro like this:

struct MyStruct {
    @Clamping(min: 0, max: 255) func fn() { ... }
}

The peer role is valid on a func, so the compiler would presumably expand it. But the accessor role is not valid on a func. Does the compiler ignore the invalid role, or does it stop you from using the macro there? In other words, does it require any of the roles to be valid, or does it require all of them to be?

Doug clarified that it does the former—applies the valid roles and ignores the invalid ones—and added language to that effect:

Not every macro role applies to every kind of declaration: an accessor macro doesn't make sense of a function, nor does a member member make sense on a property. If a macro is attached to a declaration, the macro will only be expanded for those roles that are applicable to a declaration. For example, if the Clamping macro is applied to a function, it will only be expanded as a peer macro; it's role as an accessor macro is ignored. If a macro is attached to a declaration and none of its roles apply to that kind of declaration, the program is ill-formed.

I like this answer because it's way simpler than the alternative (which would lead me to ask whether macro declarations can be overloaded on combinations of roles—a possibility I don't exactly relish). But that leads me to wonder, would it be a good idea to allow a Macro implementation to somehow signal that a role could not be applied to a particular declaration so that the compiler could apply the same soft "at least one role must be valid" logic to those decisions? That seems a little difficult to get right otherwise—diagnosing an error will make the macro fail unconditionally, while returning a no-op expansion like an empty list of peers could lead to a macro accidentally being applicable in places where it isn't really valid but happens to expand to a no-op.

(We could signal the "whoops, this role wasn't applicable after all, but you can try the others" condition by adding Bool-returning canExpand equivalents to the expansion methods, or by making the expansion methods' return values Optional, or perhaps by allowing the expansion methods to throw.)


Tangentially, consider the peer role:

  public static func expansion(
    of node: CustomAttributeSyntax,
    providingPeersOf declaration: some DeclSyntaxProtocol,
    in context: some MacroExpansionContext
  ) throws -> [DeclSyntax]

Should this method at least receive information about the kind of context it's injecting peers into? The rules for members permitted in an enum, struct, class, protocol, extension (of what kind of decl?), function body, or top-level scope are quite different; for instance, a @Memoized macro which cached results in a stored property would probably not work correctly in an enum, and if it could be used in a protocol, it would have to add a requirement for a settable property instead of a stored property with an initializer. Providing this minimal amount of context would help macros to better validate that they were producing usable code.

(You could also make a different observation—"stored properties are special, so we should have a special peerStorage role that doesn't work/works differently on enums and can inject stored properties into the main body of a type when used from an extension"—but there are other subtleties, like convenience and mutating modifiers, which also vary depending on the kind of context you're adding members to.)

I think they should be able to. But I also think they shouldn't be forced to generate their own. Member macros and member attribute macros could just also use the peer macro role to add an additional initializer inside an extension. Accessor macros would probably not need to generate an initializer. This leaves freestanding macros used inside a type declaration and witness macros. I think they should be able to choose if a generated initializer suppresses the default one.

One idea I had was to introduce an attribute:

@doesNotSuppressDefaultInitializer
init(customArgument: String) {
  ...
}

Maybe this attribute could be helpful in other circumstances as well?


I'm not sure I understand you correctly, but given @Douglas_Gregor's rule you quoted, a macro would simply be able to diagnose an error using MacroExpansionContext.diagnose() if it was attached to a declaration it cannot handle.


I believe this information would be very helpful. However, there are other types of information (conformances, attributes on the type, other members, etc.) that would be equally helpful. I would prefer it if there was a holistic solution to this (sooner rather than later :sweat_smile:).

[Edit: What I wrote about macros being able to generate extensions is wrong. See my next post.]

Having just read the newest version of the proposal, I have to take that back. Now, the new attribute I mentioned really seems like the simplest solution.

The review for SE-0389: Attached Macros has begun in this thread; please continue any discussion over there. Thank you!

2 Likes