SE-0382: Expression Macros

Evolution Proposal Reviews
1

Hello Swift community,

The review of SE-0382: "Expression Macros" begins now and runs through December 30, 2022.

For context beyond the pitch threads linked in the proposal, there is a prospective vision document which has itself been pitched and is under consideration with the language workgroup. And, as noted in this proposal, a companion proposal will be put forward at a later time regarding the process of building macros generally and Swift package manager–related changes. For now, there is an example repository showing how to build your own expression macro without those package manager features.

As the holidays approach, we're mindful that there will be some for whom now is the time to wind down from Swift-related matters. Please rest assured, if you're in that category, that we expect ample opportunity to provide feedback on this feature going forward. However, we know that there are others for whom this time of year provides unique opportunities to experiment with new features, and for that reason, we felt it'd be important to make this proposal and its implementation available now.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager via the forum messaging feature. When contacting the review manager directly, please keep the proposal link at the top of the message.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

https://github.com/apple/swift-evolution/blob/main/process.md

Thank you,

Xiaodi Wu
Review Manager

24 Likes
2

What would happen if, in the body of a macro expansion(of:in:) method, the macro executed context = MacroExpansionContext(moduleName: "Foo", fileName: "bar")? Is this supported behavior? Would the change propagate out of the current macro in any way? (including down into macros contained within the current macro, or up into macros that contain the current macro.) Would it make sense to make MacroExpansionContext a reference type instead so it can maintain its own internal state?

3

Is it really necessary to have a difference between function-like macros and value-like macros?

When either a function-call-argument-clause or a trailing-closures term is present, the identifier must refer to a function-like macro. When neither is present, the identifier must refer to a value-like macro. There is no such thing as a value of macro type.

Is there a reason to not have this: all expression macros use the function-like syntax in their declaration, and when invoking the macro, the argument clauses and trailing closures can be omitted when no arguments are specified.
This would be similar to how property wrappers work today. What is currently a function-like expression macro could have parameters with default values, and use the value-like syntax for the expansion to use all default values.

Having a value-like macro expand to a closure and calling it immediately already seems forbidden, so I don't see a downside to this change.

5 Likes
4

No, each macro expansion will get its own MacroExpansionContext.

We could do that. I suppose it would make it easier for a macro to temporarily stash the context into another instance somewhere, e.g., to pass it down into a visitor.

Doug

3 Likes
5

I certainly appreciate that this gets rid of the -> vs : distinction in the grammar, so it feels simpler. It means that a no-parameter macro can be expanded with or without the (): but it's not ambiguous (as it is with functions), that's not going to change, and we've already done this now with property wrappers, which are similar in nature to macros.

Alright, I think I like it! Thank you.

Doug

9 Likes
6

Implementing macros is almost how to use SwiftSyntax. The same is true if the macro needs to be debugged by the user.
In other words, using macros requires knowledge of SwiftSyntax, regardless of implements or uses the macros.
Currently, however, there is very little documentation on the SwiftSyntax API or the structure of the syntax tree.
Is there any plan to organize such information for the release of macros?

12 Likes
7

Will it be possible to compartmentalize the dependency on the SwiftSyntax package to just the rewriting code? It would be unfortunate to need to compile that entire package for a project that just needs some small macro like #fileID.

Or, put another way, how much binary growth can we expect for a command-line tool or application that uses a macro declared in a package somewhere upstream?

8

SwiftSyntax needs to be compiled for the host machine, not the target machine, since the macro will be expanded as part of compilation. Sometimes (often!) those are the same, and it will take compile time, but it’s a good way to remember that it can’t possibly be linked into the output.

6 Likes
9

Before using Swift, I used Clojure a lot. The one thing I miss the most is a good macro system. Therefore, I am very excited about this. I think the general approach is great and the proposal looks good.

Here are some thoughts:

  • Having a good and powerful macro system to be able to implement many 'language features' in user code is exactly the right approach in my opinion. It prevents the compiler from getting too complex. A good example is Codable. While it is great to have a standardized way of encoding types that requires little boilerplate, having it baked into the compiler means that any changes need to go through Swift Evolution, which requires a lot of effort. Moving it to an 'officially sanctioned' library (similar to Swift Collections or Swift Algorithms) would mean that it can react better to developer requirements while still being the default coding framework. Furthermore, if the official library does not fit a given problem, developers can modify the library or create their own using macros.

  • I have spent quite some time using SwiftSyntax. While the package is very powerful, its developer experience is not that great. First of all, I found little useful documentation. Secondly, finding the right types or properties needed to solve a given problem often took me a lot of time and experimentation.

  • The use of an @-attribute in @expression macro ... looks odd to me. I think the reason is that (to my knowledge) every other @-attribute is used to modify existing code and is therefore optional. You can delete @MainActor or @escaping and the remaining code is still valid. Removing @expression leaves macro ..., which is invalid code. I think the proposed spelling is not self-explanatory and would prefer something like macro(expression) ...

  • I was surprised that so few of the built-in expression macros can be implemented using ExpressionMacro. It feels like this proposal misses the point a little in this regard. I would welcome macros having more information available to allow these types of macros to be implementable. I also don't understand the reasoning why source-location information is unavailable to macro definitions.

Using the example macro repository, I have implemented a simple version of the #printArguments macro proposed by @davedelong in the macro vision thread. Aside from the debugging experience (which will hopefully get much attention), the implementation was pretty straightforward. It can be used like this:

func doSomething(_ a: Int, b: Int) {
    #printArguments()
}

doSomething(42, b: 256) // prints 'doSomething(42, b: 256)'
Full implementation 
public struct PrintArgumentsMacro: ExpressionMacro {
    public static func expansion(
        of node: MacroExpansionExprSyntax, in context: inout MacroExpansionContext
    ) -> ExprSyntax {
        var syntax = node.as(Syntax.self)
        while syntax != nil && syntax!.as(FunctionDeclSyntax.self) == nil {
            syntax = syntax!.parent
        }
        
        guard let functionSyntax = syntax!.as(FunctionDeclSyntax.self) else { return "" }
        let signature: FunctionSignatureSyntax = functionSyntax.signature
        let parameterList = signature.input.parameterList
        
        // For a function `doSomething(_ a: Int, b: String, c: Double)`, the expanded epression shall be `print("doSomething(\(a), b: \(b), c: \(c))")`
        
        let parameters = parameterList.map { parameter -> String in
            let potentialLabel = parameter.firstName!.withoutTrivia().description
            let label = potentialLabel == "_" ? nil : potentialLabel
            let potentialName = parameter.secondName?.withoutTrivia().description ?? potentialLabel
            let name = potentialName == "_" ? nil : potentialName
            var string: String
            if let label {
                string = "\(label): "
            } else {
                string = ""
            }
            if let name {
                string += "\\(\(name))"
            } else {
                string += "_"
            }
            return string
        }
        let parametersString = parameters.joined(separator: ", ")
        
        return "print(\"\(raw: functionSyntax.identifier.description)(\(raw: parametersString))\")"
    }
}

All in all, the proposal gets a +1 from me. While I think the other types of macros will be more useful to me, I think this is a great start. I am excited about the possibilities that macros bring.

And on a personal note, thank you @Douglas_Gregor for the great work you put into property wrappers and macros to put more power into the hands of developers!

6 Likes
10

and...

are great feedback, thank you. We are looking to improve the SwiftSyntax documentation, as well as providing more examples for folks to start from.

Most of time, removing a @MainActor or @escaping will cause type-checking failures. The same is true for @expression on macros.

Swift is consistent about having introducers (func, var, struct, protocol, etc.) be followed directly by the name of the thing being introduced. I think that's for good reason---if we put more between the introducer and the name, it's hard to find the name:

macro(expression) stringify(...)
macro(declaration(.peer, namePrefixes: ["_", "$"])) myLazy(...)

it gets hard to scan through the parenthesized bits to get to the name of the macro we're declaring. I think we're also boxed in a bit here by source compatibility---the @ from attributes gives us the ability to add new syntax without breaking existing code, but something like this:

macro(declaration(.peer, namePrefixes: ["_", "$"]))
myLazy(...)

is well-formed code today, and we shouldn't break that.

Yes, that's fair. While many of the built-in expression macros can be declared, they can't be implemented unless we allow the expansion context to access more information.

We're trying to keep macro definitions self-contained so that we can ensure that we understand and minimize their dependencies on other source code in the file/project. This post goes into some detail about the concerns here.

Very nice! If you're up for it, please go ahead and create a pull request to that repository, where we can collect various macro examples.

I agree that the bigger wins will come from other kinds of macros. Expression macros are the first step, and the easiest one because they are the most contained. Once we start generating declarations, issues around compile-time dependencies and incremental builds that we've only touched on with source locations will become a whole lot more interesting.

Doug

9 Likes
11

First of all, thanks for the detailed reply!

If there are good reasons to use an @-attribute, that is fine with me. Thanks for the explanation.

Would this mean that macros using node.parent risk inconsistent behavior with incremental builds? That would be a huge limitation in my opinion. For example, the #printArguments macro would not work without accessing the parent nodes in order to get the function name and signature.

2 Likes
12

It means that the implementation shouldn't give access to the parent node at all, and it is a big limitation. It's possible that we could find a more reasonable "cut" point for the syntax nodes that are passed to the macro that balances between macro expressivity and still preserved incremental builds. What if, for example, you could access up to the nearest enclosing function / type / variable / etc? That would be enough for #printArguments, and the built-in #function, without exposing the rest of the source file. It aligns fairly well with the granularity at which we could do incremental compilation, too.

I'll give that a try!

Doug

7 Likes
13

Accessing the nearest definition would help. However, if #printArguments also wanted to print the type name of self in case of a method, this would not be enough. Some other examples would be:

  1. detecting if the macro call is inside of a method or a free function
  2. detecting if self is of a value or reference type
  3. detecting or enumerating other members of self inside a method (This could be emulated using protocols. However, this would mean runtime checks making the code less efficient and necessitate runtime errors instead of build errors.)

There are some solutions I can think of. I am not sure if any of them are feasible:

  • Macros could declare the need to access the parent nodes, disabling incremental builds.

  • Maybe the compiler could provide some sort of declaration hierarchy to the macro, such as name, signature and attributes of the enclosing function, then attributes, type (class, struct, enum, etc.), name and inheritance of the type containing the function, maybe even the type containing this type, etc. This would solve example 3 only in part.

  • Maybe the whole syntax tree could be cached and the relevant parts be updated during incremental compilation.

I suspect that other macro types (propertyWrapper, requirement, conformance, memberDeclaration) would need access to all information about the type in question (members, attributes, name, etc.). Therefore, a solution to this must be found. It would be unfortunate if expression macros would miss out on this information only because they were proposed and accepted before the other macro types.

1 Like
14

Something that I’m noticing in all of the example macros is a pervasive use of (macro-)runtime verification of arguments. I feel like while doing all this checking explicitly in the macro body allows for maximum flexibility in macro interfaces, there could be some benefit to offering an alternative, more-strongly-typed API. This could be implemented in a similar way to how any Swift type can either define one or more callAsFunction methods with statically typed parameters, or be @dynamicCallable and define a dynamicallyCall method that does validation at runtime. Here’s one way this could work for macros, taking the example WarningMacro:

protocol StaticExpressionMacro: ExpressionMacro {}
extension StaticExpressionMacro {
  // compiler auto-generates this based on your `expandCall` implementations
  static func expansion(of macro: MacroExpansionExprSyntax, in context: inout MacroExpansionContext) throws -> ExprSyntax
}

struct StaticWarningMacro: StaticExpressionMacro {
  static func expandCall(
    // context (+ other args) provided by the compiler go here:
    to macro: MacroExpansionExprSyntax,
    in context: inout MacroExpansionContext,
    // arguments passed by the macro invocation go here
    _ stringLiteral: StringLiteralExprSyntax
  ) throws -> ExprSyntax {
    guard
      // these checks are still necessary once we have the `StringLiteralExprSyntax`
      stringLiteral.segments.count == 1,
      case let .stringSegment(messageString)? = stringLiteral.segments.first
    else {
      throw CustomError.message("#myWarning macro requires a string literal")
    }
    // [same macro implementation]
  }

  // version for value-like macros
  // static func expandReference(to macro: MacroExpansionExprSyntax, in context: inout MacroExpansionContext) throws → ExprSyntax {}
}

// potentially allow more concise declaration since the valid
// overloads can be inferred from the macro struct declaration?
public macro myWarning: StaticWarningMacro

Does this seem like a reasonable approach? I’m not entirely sold on allowing the parameters of the expandCall method to be declared as anything other than ExprSyntax since I imagine most macros will do semantic verification of their arguments rather than plain syntactic verification (but maybe it’s worth it to make very simple macros easier to write?)

Another reason I’m proposing this is that most of the checks in these macros are somewhat untestable — if you only declare a macro as taking one parameter, there’s no way to test that it behaves correctly when passed zero or more than one parameters, so that code would not be covered by unit tests (if the unit tests use the #blah syntax rather than invoking the macro directly).

1 Like
15

Absolutely! I agree that SwiftSyntax's documentation is lacking at the moment. I'm going to use this PR to collate a basic set of articles on working with SwiftSyntax. I'm hoping to also add an article on working with SyntaxVisitors, and a another on writing a refactoring pass with SwiftRefactor.

7 Likes
16

I tried to implement a very rough proof of concept for implementing the result builder transform using an expression macro. In theory, this should be possible. A result builder type and a closure to transform is passed to the macro and a transformed closure is returned. As I test it, however, the compiler gives me an error with the following output:

Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3 as (() -> String)
------------------------------
{ () -> String in
let __macro_local_0 = StringAppender.buildPartialBlock(first: StringAppender.buildExpression("This"));
let __macro_local_1 = StringAppender.buildPartialBlock(accumulated: __macro_local_0, next: StringAppender.buildExpression("is"));
let __macro_local_2 = StringAppender.buildPartialBlock(accumulated: __macro_local_1, next: StringAppender.buildExpression("a"));
let __macro_local_3 = StringAppender.buildPartialBlock(accumulated: __macro_local_2, next: StringAppender.buildExpression("sentence."));
return StringAppender.buildFinalResult(__macro_local_3)
}
------------------------------
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:3:69: error: cannot find '__macro_local_0' in scope
let __macro_local_1 = StringAppender.buildPartialBlock(accumulated: __macro_local_0, next: StringAppender.buildExpression("is"));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:4:69: error: cannot find '__macro_local_1' in scope
let __macro_local_2 = StringAppender.buildPartialBlock(accumulated: __macro_local_1, next: StringAppender.buildExpression("a"));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:5:69: error: cannot find '__macro_local_2' in scope
let __macro_local_3 = StringAppender.buildPartialBlock(accumulated: __macro_local_2, next: StringAppender.buildExpression("sentence."));
                                                                    ^~~~~~~~~~~~~~~
Macro expansion of #apply(resultBuilder:to:) in /Users/kocki/Downloads/swift-macro-examples/MacroExamples/main.swift:67:21-72:3:6:40: error: cannot find '__macro_local_3' in scope
return StringAppender.buildFinalResult(__macro_local_3)
                                       ^~~~~~~~~~~~~~~

The error persists if I use other variable names.

Full code

In MacroExampleLib:

public macro apply<R: ResultBuilder>(resultBuilder: R.Type, to closure: () -> Void) -> (() -> String) = MacroExamplesPlugin.ResultBuilderMacro // If I use `(() -> R.FinalResult)` as the return type, I get another error

public protocol ResultBuilder {
    associatedtype Component
    associatedtype FinalResult
    
    static func buildPartialBlock(first: Component) -> Component
    
    static func buildPartialBlock(accumulated: Component, next: Component) -> Component
    
    static func buildFinalResult(_ component: Component) -> FinalResult
}

In MacroExamplesPlugin:

public struct ResultBuilderMacro: ExpressionMacro {
    public static func expansion(
        of node: MacroExpansionExprSyntax, in context: inout MacroExpansionContext
    ) throws -> ExprSyntax {
        guard
            let resultBuilderSelfExpr = node.argumentList.first?.expression.as(MemberAccessExprSyntax.self),
            let resultBuilderName = resultBuilderSelfExpr.base?.withoutTrivia().description,
            let originalClosure = node.argumentList.dropFirst().first?.expression.as(ClosureExprSyntax.self)
        else {
            throw SomeError()
        }
        
        let originalStatements: [CodeBlockItemSyntax] = Array(originalClosure.statements.map { $0.withoutTrivia() })
        guard let firstStatement = originalStatements.first else {
            throw SomeError()
        }
        
        var localName = context.createUniqueLocalName()
        var newStatements: [String] = []
        newStatements.append("let \(localName) = \(resultBuilderName).buildPartialBlock(first: \(resultBuilderName).buildExpression(\(firstStatement)));")
        
        for statement in originalStatements.dropFirst() {
            let newLocalName = context.createUniqueLocalName()
            newStatements.append("let \(newLocalName) = \(resultBuilderName).buildPartialBlock(accumulated: \(localName), next: \(resultBuilderName).buildExpression(\(statement)));")
            localName = newLocalName
        }
        
        newStatements.append("return \(resultBuilderName).buildFinalResult(\(localName))")
        
        let joinedStatements = newStatements.joined(separator: "\n")
        return "{ () -> String in\n\(raw: joinedStatements)\n}"
    }
}

struct SomeError: Error {}

Finally, in main.swift:

@resultBuilder
struct StringAppender: ResultBuilder {
    static func buildExpression(_ expression: String) -> String {
        expression
    }
    
    static func buildExpression<T>(_ expression: T) -> String {
        String(describing: expression)
    }
    
    static func buildPartialBlock(first: String) -> String {
        first
    }
    
    static func buildPartialBlock(accumulated: String, next: String) -> String {
        accumulated + " " + next
    }
    
    static func buildFinalResult(_ component: String) -> String {
        component
    }
}

@StringAppender
var string: String {
    "This"
    "is"
    "a"
    "sentence."
}

let stringClosure = #apply(resultBuilder: StringAppender.self, to: {
    "This"
    "is"
    "a"
    "sentence."
})

print(string)
print(stringClosure())
18

Macro arguments are type-checked against the parameter types of the macro prior to instantiating the macro.

macro stringify<T>(_: T) -> (T, String)

But what about scenario where I might want to construct type based on the input provided, i.e. building a type from json string:

macro jsonObj<T>(_: String) -> T

let obj = #jsonObj("""
{
"one": 1,
"two": 2
}
""")

Or registering routes in a server framework:

app.get("hello/:name") { req in
    // req could be inferred from the provided string as
    // SomeGenericStruct<(name: String)>
    let name = req.parameters.name
    return "Hello, \(name)!"
}

Although, such scenario can degrade compile-time performance by having to expand the macro for type inference, but having this as an option would be really beneficial.

20

Hello everyone!

The language workgroup discussed the initial review feedback for this proposal, and we felt that given its introduction over the previous holiday season, it'd be reasonable to extend the review for another two weeks (until January 15, 2023).

The workgroup (and proposal authors) would be particularly interested in hearing feedback based on hands-on user experience with the draft implementation. Are there sharp edges, unexpected limitations, or other ideas for improvement in the proposed design that arise based on attempts to make your own expression macros? This would all be very useful information that would help shape not just expression macros but, as the first in a series of these, potentially other macros as well.

As always, thank you for your participation in the Swift Evolution process!

Xiaodi Wu
Review Manager

2 Likes
21

@Douglas_Gregor I would like to experiment a bit more with the current implementation. For that, it would be very helpful if I could get some feedback on the error mentioned above. I would like to know if this is an inherent limitation of the current implementation when expanding closures or if I can somehow work around it.

Thanks in advance!

1 Like
22

Would expression macros support trailing closure syntax? I have the feeling that passing closures could be a major use-case.

1 Like