SE-0382: Expression Macros

I certainly appreciate that this gets rid of the -> vs : distinction in the grammar, so it feels simpler. It means that a no-parameter macro can be expanded with or without the (): but it's not ambiguous (as it is with functions), that's not going to change, and we've already done this now with property wrappers, which are similar in nature to macros.

Alright, I think I like it! Thank you.

Doug

9 Likes

Implementing macros is almost how to use SwiftSyntax. The same is true if the macro needs to be debugged by the user.
In other words, using macros requires knowledge of SwiftSyntax, regardless of implements or uses the macros.
Currently, however, there is very little documentation on the SwiftSyntax API or the structure of the syntax tree.
Is there any plan to organize such information for the release of macros?

12 Likes

Will it be possible to compartmentalize the dependency on the SwiftSyntax package to just the rewriting code? It would be unfortunate to need to compile that entire package for a project that just needs some small macro like #fileID.

Or, put another way, how much binary growth can we expect for a command-line tool or application that uses a macro declared in a package somewhere upstream?

SwiftSyntax needs to be compiled for the host machine, not the target machine, since the macro will be expanded as part of compilation. Sometimes (often!) those are the same, and it will take compile time, but it’s a good way to remember that it can’t possibly be linked into the output.

6 Likes

Before using Swift, I used Clojure a lot. The one thing I miss the most is a good macro system. Therefore, I am very excited about this. I think the general approach is great and the proposal looks good.

Here are some thoughts:

  • Having a good and powerful macro system to be able to implement many 'language features' in user code is exactly the right approach in my opinion. It prevents the compiler from getting too complex. A good example is Codable. While it is great to have a standardized way of encoding types that requires little boilerplate, having it baked into the compiler means that any changes need to go through Swift Evolution, which requires a lot of effort. Moving it to an 'officially sanctioned' library (similar to Swift Collections or Swift Algorithms) would mean that it can react better to developer requirements while still being the default coding framework. Furthermore, if the official library does not fit a given problem, developers can modify the library or create their own using macros.

  • I have spent quite some time using SwiftSyntax. While the package is very powerful, its developer experience is not that great. First of all, I found little useful documentation. Secondly, finding the right types or properties needed to solve a given problem often took me a lot of time and experimentation.

  • The use of an @-attribute in @expression macro ... looks odd to me. I think the reason is that (to my knowledge) every other @-attribute is used to modify existing code and is therefore optional. You can delete @MainActor or @escaping and the remaining code is still valid. Removing @expression leaves macro ..., which is invalid code. I think the proposed spelling is not self-explanatory and would prefer something like macro(expression) ...

  • I was surprised that so few of the built-in expression macros can be implemented using ExpressionMacro. It feels like this proposal misses the point a little in this regard. I would welcome macros having more information available to allow these types of macros to be implementable. I also don't understand the reasoning why source-location information is unavailable to macro definitions.

Using the example macro repository, I have implemented a simple version of the #printArguments macro proposed by @davedelong in the macro vision thread. Aside from the debugging experience (which will hopefully get much attention), the implementation was pretty straightforward. It can be used like this:

func doSomething(_ a: Int, b: Int) {
    #printArguments()
}

doSomething(42, b: 256) // prints 'doSomething(42, b: 256)'
Full implementation
public struct PrintArgumentsMacro: ExpressionMacro {
    public static func expansion(
        of node: MacroExpansionExprSyntax, in context: inout MacroExpansionContext
    ) -> ExprSyntax {
        var syntax = node.as(Syntax.self)
        while syntax != nil && syntax!.as(FunctionDeclSyntax.self) == nil {
            syntax = syntax!.parent
        }
        
        guard let functionSyntax = syntax!.as(FunctionDeclSyntax.self) else { return "" }
        let signature: FunctionSignatureSyntax = functionSyntax.signature
        let parameterList = signature.input.parameterList
        
        // For a function `doSomething(_ a: Int, b: String, c: Double)`, the expanded epression shall be `print("doSomething(\(a), b: \(b), c: \(c))")`
        
        let parameters = parameterList.map { parameter -> String in
            let potentialLabel = parameter.firstName!.withoutTrivia().description
            let label = potentialLabel == "_" ? nil : potentialLabel
            let potentialName = parameter.secondName?.withoutTrivia().description ?? potentialLabel
            let name = potentialName == "_" ? nil : potentialName
            var string: String
            if let label {
                string = "\(label): "
            } else {
                string = ""
            }
            if let name {
                string += "\\(\(name))"
            } else {
                string += "_"
            }
            return string
        }
        let parametersString = parameters.joined(separator: ", ")
        
        return "print(\"\(raw: functionSyntax.identifier.description)(\(raw: parametersString))\")"
    }
}

All in all, the proposal gets a +1 from me. While I think the other types of macros will be more useful to me, I think this is a great start. I am excited about the possibilities that macros bring.

And on a personal note, thank you @Douglas_Gregor for the great work you put into property wrappers and macros to put more power into the hands of developers!

6 Likes

and...

are great feedback, thank you. We are looking to improve the SwiftSyntax documentation, as well as providing more examples for folks to start from.

Most of time, removing a @MainActor or @escaping will cause type-checking failures. The same is true for @expression on macros.

Swift is consistent about having introducers (func, var, struct, protocol, etc.) be followed directly by the name of the thing being introduced. I think that's for good reason---if we put more between the introducer and the name, it's hard to find the name:

macro(expression) stringify(...)
macro(declaration(.peer, namePrefixes: ["_", "$"])) myLazy(...)

it gets hard to scan through the parenthesized bits to get to the name of the macro we're declaring. I think we're also boxed in a bit here by source compatibility---the @ from attributes gives us the ability to add new syntax without breaking existing code, but something like this:

macro(declaration(.peer, namePrefixes: ["_", "$"]))
myLazy(...)

is well-formed code today, and we shouldn't break that.

Yes, that's fair. While many of the built-in expression macros can be declared, they can't be implemented unless we allow the expansion context to access more information.

We're trying to keep macro definitions self-contained so that we can ensure that we understand and minimize their dependencies on other source code in the file/project. This post goes into some detail about the concerns here.

Very nice! If you're up for it, please go ahead and create a pull request to that repository, where we can collect various macro examples.

I agree that the bigger wins will come from other kinds of macros. Expression macros are the first step, and the easiest one because they are the most contained. Once we start generating declarations, issues around compile-time dependencies and incremental builds that we've only touched on with source locations will become a whole lot more interesting.

Doug

9 Likes

First of all, thanks for the detailed reply!

If there are good reasons to use an @-attribute, that is fine with me. Thanks for the explanation.

Would this mean that macros using node.parent risk inconsistent behavior with incremental builds? That would be a huge limitation in my opinion. For example, the #printArguments macro would not work without accessing the parent nodes in order to get the function name and signature.

2 Likes

It means that the implementation shouldn't give access to the parent node at all, and it is a big limitation. It's possible that we could find a more reasonable "cut" point for the syntax nodes that are passed to the macro that balances between macro expressivity and still preserved incremental builds. What if, for example, you could access up to the nearest enclosing function / type / variable / etc? That would be enough for #printArguments, and the built-in #function, without exposing the rest of the source file. It aligns fairly well with the granularity at which we could do incremental compilation, too.

I'll give that a try!

Doug

7 Likes

Accessing the nearest definition would help. However, if #printArguments also wanted to print the type name of self in case of a method, this would not be enough. Some other examples would be:

  1. detecting if the macro call is inside of a method or a free function
  2. detecting if self is of a value or reference type
  3. detecting or enumerating other members of self inside a method (This could be emulated using protocols. However, this would mean runtime checks making the code less efficient and necessitate runtime errors instead of build errors.)

There are some solutions I can think of. I am not sure if any of them are feasible:

  • Macros could declare the need to access the parent nodes, disabling incremental builds.

  • Maybe the compiler could provide some sort of declaration hierarchy to the macro, such as name, signature and attributes of the enclosing function, then attributes, type (class, struct, enum, etc.), name and inheritance of the type containing the function, maybe even the type containing this type, etc. This would solve example 3 only in part.

  • Maybe the whole syntax tree could be cached and the relevant parts be updated during incremental compilation.

I suspect that other macro types (propertyWrapper, requirement, conformance, memberDeclaration) would need access to all information about the type in question (members, attributes, name, etc.). Therefore, a solution to this must be found. It would be unfortunate if expression macros would miss out on this information only because they were proposed and accepted before the other macro types.

1 Like

Something that I’m noticing in all of the example macros is a pervasive use of (macro-)runtime verification of arguments. I feel like while doing all this checking explicitly in the macro body allows for maximum flexibility in macro interfaces, there could be some benefit to offering an alternative, more-strongly-typed API. This could be implemented in a similar way to how any Swift type can either define one or more callAsFunction methods with statically typed parameters, or be @dynamicCallable and define a dynamicallyCall method that does validation at runtime. Here’s one way this could work for macros, taking the example WarningMacro:

protocol StaticExpressionMacro: ExpressionMacro {}
extension StaticExpressionMacro {
  // compiler auto-generates this based on your `expandCall` implementations
  static func expansion(of macro: MacroExpansionExprSyntax, in context: inout MacroExpansionContext) throws -> ExprSyntax
}

struct StaticWarningMacro: StaticExpressionMacro {
  static func expandCall(
    // context (+ other args) provided by the compiler go here:
    to macro: MacroExpansionExprSyntax,
    in context: inout MacroExpansionContext,
    // arguments passed by the macro invocation go here
    _ stringLiteral: StringLiteralExprSyntax
  ) throws -> ExprSyntax {
    guard
      // these checks are still necessary once we have the `StringLiteralExprSyntax`
      stringLiteral.segments.count == 1,
      case let .stringSegment(messageString)? = stringLiteral.segments.first
    else {
      throw CustomError.message("#myWarning macro requires a string literal")
    }
    // [same macro implementation]
  }

  // version for value-like macros
  // static func expandReference(to macro: MacroExpansionExprSyntax, in context: inout MacroExpansionContext) throws → ExprSyntax {}
}

// potentially allow more concise declaration since the valid
// overloads can be inferred from the macro struct declaration?
public macro myWarning: StaticWarningMacro

Does this seem like a reasonable approach? I’m not entirely sold on allowing the parameters of the expandCall method to be declared as anything other than ExprSyntax since I imagine most macros will do semantic verification of their arguments rather than plain syntactic verification (but maybe it’s worth it to make very simple macros easier to write?)

Another reason I’m proposing this is that most of the checks in these macros are somewhat untestable — if you only declare a macro as taking one parameter, there’s no way to test that it behaves correctly when passed zero or more than one parameters, so that code would not be covered by unit tests (if the unit tests use the #blah syntax rather than invoking the macro directly).

1 Like

Absolutely! I agree that SwiftSyntax's documentation is lacking at the moment. I'm going to use this PR to collate a basic set of articles on working with SwiftSyntax. I'm hoping to also add an article on working with SyntaxVisitors, and a another on writing a refactoring pass with SwiftRefactor.

7 Likes

I tried to implement a very rough proof of concept for implementing the result builder transform using an expression macro. In theory, this should be possible. A result builder type and a closure to transform is passed to the macro and a transformed closure is returned. As I test it, however, the compiler gives me an error with the following output:

Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3 as (() -> String)
------------------------------
{ () -> String in
let __macro_local_0 = StringAppender.buildPartialBlock(first: StringAppender.buildExpression("This"));
let __macro_local_1 = StringAppender.buildPartialBlock(accumulated: __macro_local_0, next: StringAppender.buildExpression("is"));
let __macro_local_2 = StringAppender.buildPartialBlock(accumulated: __macro_local_1, next: StringAppender.buildExpression("a"));
let __macro_local_3 = StringAppender.buildPartialBlock(accumulated: __macro_local_2, next: StringAppender.buildExpression("sentence."));
return StringAppender.buildFinalResult(__macro_local_3)
}
------------------------------
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:3:69: error: cannot find '__macro_local_0' in scope
let __macro_local_1 = StringAppender.buildPartialBlock(accumulated: __macro_local_0, next: StringAppender.buildExpression("is"));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:4:69: error: cannot find '__macro_local_1' in scope
let __macro_local_2 = StringAppender.buildPartialBlock(accumulated: __macro_local_1, next: StringAppender.buildExpression("a"));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:5:69: error: cannot find '__macro_local_2' in scope
let __macro_local_3 = StringAppender.buildPartialBlock(accumulated: __macro_local_2, next: StringAppender.buildExpression("sentence."));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:6:40: error: cannot find '__macro_local_3' in scope
return StringAppender.buildFinalResult(__macro_local_3)
                                       ^~~~~~~~~~~~~~~

The error persists if I use other variable names.

Full code

In MacroExampleLib:

public macro apply<R: ResultBuilder>(resultBuilder: R.Type, to closure: () -> Void) -> (() -> String) = MacroExamplesPlugin.ResultBuilderMacro // If I use `(() -> R.FinalResult)` as the return type, I get another error

public protocol ResultBuilder {
    associatedtype Component
    associatedtype FinalResult
    
    static func buildPartialBlock(first: Component) -> Component
    
    static func buildPartialBlock(accumulated: Component, next: Component) -> Component
    
    static func buildFinalResult(_ component: Component) -> FinalResult
}

In MacroExamplesPlugin:

public struct ResultBuilderMacro: ExpressionMacro {
    public static func expansion(
        of node: MacroExpansionExprSyntax, in context: inout MacroExpansionContext
    ) throws -> ExprSyntax {
        guard
            let resultBuilderSelfExpr = node.argumentList.first?.expression.as(MemberAccessExprSyntax.self),
            let resultBuilderName = resultBuilderSelfExpr.base?.withoutTrivia().description,
            let originalClosure = node.argumentList.dropFirst().first?.expression.as(ClosureExprSyntax.self)
        else {
            throw SomeError()
        }
        
        let originalStatements: [CodeBlockItemSyntax] = Array(originalClosure.statements.map { $0.withoutTrivia() })
        guard let firstStatement = originalStatements.first else {
            throw SomeError()
        }
        
        var localName = context.createUniqueLocalName()
        var newStatements: [String] = []
        newStatements.append("let \(localName) = \(resultBuilderName).buildPartialBlock(first: \(resultBuilderName).buildExpression(\(firstStatement)));")
        
        for statement in originalStatements.dropFirst() {
            let newLocalName = context.createUniqueLocalName()
            newStatements.append("let \(newLocalName) = \(resultBuilderName).buildPartialBlock(accumulated: \(localName), next: \(resultBuilderName).buildExpression(\(statement)));")
            localName = newLocalName
        }
        
        newStatements.append("return \(resultBuilderName).buildFinalResult(\(localName))")
        
        let joinedStatements = newStatements.joined(separator: "\n")
        return "{ () -> String in\n\(raw: joinedStatements)\n}"
    }
}

struct SomeError: Error {}

Finally, in main.swift:

@resultBuilder
struct StringAppender: ResultBuilder {
    static func buildExpression(_ expression: String) -> String {
        expression
    }
    
    static func buildExpression<T>(_ expression: T) -> String {
        String(describing: expression)
    }
    
    static func buildPartialBlock(first: String) -> String {
        first
    }
    
    static func buildPartialBlock(accumulated: String, next: String) -> String {
        accumulated + " " + next
    }
    
    static func buildFinalResult(_ component: String) -> String {
        component
    }
}

@StringAppender
var string: String {
    "This"
    "is"
    "a"
    "sentence."
}

let stringClosure = #apply(resultBuilder: StringAppender.self, to: {
    "This"
    "is"
    "a"
    "sentence."
})

print(string)
print(stringClosure())

Macro arguments are type-checked against the parameter types of the macro prior to instantiating the macro.

macro stringify<T>(_: T) -> (T, String)

But what about scenario where I might want to construct type based on the input provided, i.e. building a type from json string:

macro jsonObj<T>(_: String) -> T

let obj = #jsonObj("""
{
"one": 1,
"two": 2
}
""")

Or registering routes in a server framework:

app.get("hello/:name") { req in
    // req could be inferred from the provided string as
    // SomeGenericStruct<(name: String)>
    let name = req.parameters.name
    return "Hello, \(name)!"
}

Although, such scenario can degrade compile-time performance by having to expand the macro for type inference, but having this as an option would be really beneficial.

Hello everyone!

The language workgroup discussed the initial review feedback for this proposal, and we felt that given its introduction over the previous holiday season, it'd be reasonable to extend the review for another two weeks (until January 15, 2023).

The workgroup (and proposal authors) would be particularly interested in hearing feedback based on hands-on user experience with the draft implementation. Are there sharp edges, unexpected limitations, or other ideas for improvement in the proposed design that arise based on attempts to make your own expression macros? This would all be very useful information that would help shape not just expression macros but, as the first in a series of these, potentially other macros as well.

As always, thank you for your participation in the Swift Evolution process!

Xiaodi Wu
Review Manager

2 Likes

@Douglas_Gregor I would like to experiment a bit more with the current implementation. For that, it would be very helpful if I could get some feedback on the error mentioned above. I would like to know if this is an inherent limitation of the current implementation when expanding closures or if I can somehow work around it.

Thanks in advance!

1 Like

Would expression macros support trailing closure syntax? I have the feeling that passing closures could be a major use-case.

1 Like

Big +1 to the proposal from me.

Here's one use case I'd like to highlight. With the example repository I've been able to implement a compile-time conversion from string literals to arrays of bytes. This seems like a reasonable solution for environments where String type with Unicode handling is too heavyweight. Here's the example code for MacroExamplesPlugin.

import SwiftSyntax
import _SwiftSyntaxMacros

public struct UTF8ArrayMacro: ExpressionMacro {
  public static func expansion(
    of macro: MacroExpansionExprSyntax, in context: inout MacroExpansionContext
  ) throws -> ExprSyntax {
      guard let firstElement = macro.argumentList.first,
        let stringLiteral = firstElement.expression
          .as(StringLiteralExprSyntax.self),
        stringLiteral.segments.count == 1,
        case let .stringSegment(messageString)? = stringLiteral.segments.first
      else {
        throw CustomError.message("#utf8 macro requires a string literal")
      }

      return "\(raw: messageString.syntaxTextBytes)"
  }
}

Then in MacroExamplesLib:

public macro utf8(_ string: String) -> [UInt8] = MacroExamplesPlugin.UTF8ArrayMacro

And at the place of use:

_ = #utf8("Hello, world!").withUnsafeBytes {
    fputs($0.baseAddress, stdout)
}

When dead code elimination becomes advanced enough to strip out unused Unicode tables on platforms where everything is linked statically, this would allow producing binaries as small as those you can get with Rust or C. Of course, there are downsides to this approach, as you're losing interpolation and other niceties of String. But for applications where binary size is a top priority, this is still a big improvement.

10 Likes

Your macro implementations looks good, thank you for working on it! I (very recently) fixed a compiler bug that caused the incorrect error messages, and with my locally-built compiler (and adding a bunch of @expression utterances), your result-builder macro works. My fix hasn't made it into a snapshot yet, but here's a one-off toolchain that contains it.

Yes, they do! I updated your example macro to this:

let stringClosure = #apply(resultBuilder: StringAppender.self) {
    "This"
    "is"
    "a"
    "sentence."
}

and taught the macro implementation to look for a trailing closure as well:

        guard
            let resultBuilderSelfExpr = node.argumentList.first?.expression.as(MemberAccessExprSyntax.self),
            let resultBuilderName = resultBuilderSelfExpr.base?.withoutTrivia().description,
            let originalClosure = node.argumentList.dropFirst().first?.expression.as(ClosureExprSyntax.self) ??
              node.trailingClosure
        else {
            throw SomeError()
        }

and it works fine.

Doug

4 Likes

I finally had an opportunity to try writing some macros. Some thoughts:

JSON literal conversion

I was inspired by the JSON example above, so I wrote one that converts a JSON string literal to nested calls to JSON case initializers. I don't think this is precisely what the author of that post had in mind (it sounds like they want to construct specific types from the literal, more like Codable), but since we don't have the semantic info for that, I tried this simpler idea instead.

Expand for implementation
// In the macro module
public enum JSON {
  case null
  case string(String)
  case number(Double)
  case array([JSON])
  case object([String: JSON])
}

public macro jsonLiteral(_ string: String) -> JSON = MacroExamplesPlugin.JSONLiteralMacro
// Plug-in implementation
import Foundation
import SwiftSyntax
import SwiftSyntaxBuilder
import _SwiftSyntaxMacros

private func jsonExpr(for jsonValue: Any) throws -> ExprSyntax {
  switch jsonValue {
  case is NSNull:
    return "JSON.null"

  case let string as String:
    return "JSON.string(\(literal: string))"

  case let number as Double:
    return "JSON.number(\(literal: number))"

  case let array as [Any]:
    var elements = [ArrayElementSyntax]()
    for element in array {
      let elementExpr = try jsonExpr(for: element)
      elements.append(
        ArrayElementSyntax(
          expression: elementExpr,
          trailingComma: .commaToken()))
    }
    let arrayLiteral = ArrayExprSyntax(
      elements: ArrayElementListSyntax(elements))
    return "JSON.array(\(arrayLiteral))"

  case let dictionary as [String: Any]:
    guard !dictionary.isEmpty else {
      return "JSON.object([:])"
    }

    var elements = [DictionaryElementSyntax]()
    for (key, value) in dictionary {
      let keyExpr = StringLiteralExprSyntax(content: key)
      let valueExpr = try jsonExpr(for: value)
      elements.append(
        DictionaryElementSyntax(
          keyExpression: keyExpr,
          valueExpression: valueExpr,
          trailingComma: .comma))
    }
    let dictionaryLiteral = DictionaryExprSyntax(
      content: .elements(DictionaryElementListSyntax(elements)))
    return "JSON.object(\(dictionaryLiteral))"

  default:
    throw CustomError.message("Invalid type in deserialized JSON: \(type(of: jsonValue))")
  }
}

public struct JSONLiteralMacro: ExpressionMacro {
  public static func expansion(
    of macro: MacroExpansionExprSyntax,
    in context: inout MacroExpansionContext
  ) throws -> ExprSyntax {
    guard let firstElement = macro.argumentList.first,
      let stringLiteral = firstElement.expression
        .as(StringLiteralExprSyntax.self),
      stringLiteral.segments.count == 1,
      case let .stringSegment(jsonString)? = stringLiteral.segments.first
    else {
      throw CustomError.message("#jsonLiteral macro requires a string literal")
    }

    let json = try JSONSerialization.jsonObject(
      with: jsonString.content.text.data(using: .utf8)!,
      options: [.fragmentsAllowed])
    let jsonCaseExpr = try jsonExpr(for: json)
    if let leadingTrivia = macro.leadingTrivia {
      return jsonCaseExpr.withLeadingTrivia(leadingTrivia)
    }
    return jsonCaseExpr
  }
}

This worked really nicely (after stumbling on some SwiftSyntax corruption issues :grimacing:). The following invocation produced the source code I expected:

let json: JSON = #jsonLiteral("""
  {
    "name": "Bojack Horseman",
    "species": "horse",
    "age": 59,
    "friends": [
      "Diane Nguyen",
      "Mr. Peanutbutter",
      "Todd Chavez"
    ],
    "selfControl": null
  }
  """)
// JSON.object(["selfControl":JSON.null,"name":JSON.string("Bojack Horseman"),
// "species":JSON.string("horse"),"friends":JSON.array([
// JSON.string("Diane Nguyen"),JSON.string("Mr. Peanutbutter"),
// JSON.string("Todd Chavez"),]),"age":JSON.number(59.0),])

(Naturally there are some problems like key ordering being different due to JSONSerialization and NSDictionary but that's not relevant here.)

My takeaways here are:

  • Using string interpolation to construct nodes from a combination of literal Swift code and substituted content is a joy to use.
  • When you have to drop down a level to raw initializers, SwiftSyntax/SwiftSyntaxBuilder still provides nice defaults in many places for fixed structural tokens. For example, when creating an ArrayLiteralExpr, you don't have to provide the [ and ] tokens manually.
  • But, we should extend the builder functionality in SwiftSyntax and/or provide additional helpers to make common functionality much more approachable for users. The average macro author shouldn't have to deal with subtle things like including trailing commas in arrays/dictionaries/argument lists, nor have to be aware of the different syntax node representation used for the content of an empty dictionary vs. a dictionary with elements. If SwiftSyntaxBuilder already has some of this, I've missed it, in which case it's a documentation problem instead. My code is probably not the simplest form possible; since SwiftSyntax is such a large API surface, we should figure out how to strongly push users toward the simplest/cleanest APIs.

Trivia handling

I observed that in your FontLiteralMacro implementation, you write this:

if let leadingTrivia = macro.leadingTrivia {
    return initSyntax.withLeadingTrivia(leadingTrivia)
}
return initSyntax

What is the purpose of retaining the leading trivia—to preserve any comments that may precede it if the macro expansion is printed for debugging? I can see this being more important for declaration macros where you'd want to be able to scrape for documentation comments after expansion.

If trivia had to be manually preserved, I would have expected something like this to fail:

let x = #someMacro(...) + y

Where if the expansion didn't preserve the space in #someMacro(...)'s trailing trivia, you'd end up with an expansion let x = VALUE+ y, which would fail to parse because + is now a postfix operator instead of an infix operator. But I tested that and it appears to be still parsed as though it was let x = VALUE + y, so I'm unclear on what the actual trivia behavior is here.

EDIT: I may be able to answer my own question here. Since the macro is applied to the already type-checked AST, it knows that #macro(...) + y must already be an infix operator even if the trivia for the expansion changes?

Can the macro infrastructure manage trivia automatically so that <leading trivia>#macro(...)<trailing trivia> is always transformed to <leading trivia><expanded node><trailing trivia>, so that the macro can never remove or replace trivia? I could see us wanting to merge the original trivia with the trivia attached to the expanded node, but not allow it to be completely replaced.

In fact, I wonder if it's problematic for macro expansions to be able to see trivia at all. I wrote a really stupid macro:

// Returns `string` as an integer, but also add the number in the preceding comment if there is one.
public macro theSameNumber(_ string: String) -> Int = MacroExamplesPlugin.TheSameNumberMacro

let n1 = #theSameNumber("123")
print(n1)  // 123

let n2 =
  // 5
  #theSameNumber("123")
print(n2)  // 128

Allowing folks to have semantically significant comments feels like a bad idea, but I can also see value in having access to trivia for other kinds of macros. For example, a declaration macro could treat a preceding doc comment as a template to splat out new doc comments for the declarations that it generates. I'm not sure what's the best way to square these goals; should we only provide trivia to certain types of macros (e.g., declarations but not expressions)? Should we require any kind of macro to explicitly opt-in if it wants the trivia? Or do we just accept that people can do bizarre things with it?

5 Likes

One curiosity I noticed here is that you can have macros which do something different if a closure is specified as an argument vs as a trailing closure. Not sure what conclusions to draw from this, just feels a bit off to me.