SE-0382: Expression Macros

A couple of things have come up in the review that warrant small proposal revisions. I've collected them in this pull request. They are:

  • Make MacroExpansionContext a class, because the state involving diagnostics and unique names needs to be shared.

  • Allow macro parameters to have default arguments, with restrictions on what can occur within a default argument.

  • Clarify that macro expansion cannot be recursive.

  • Rename createUniqueLocalName to createUniqueName; the names might not always be local in scope.

    Doug

3 Likes

Stopping people from doing weird ill-advised things with macros is pretty much impossible. I don't see a good use-case for accessing comments, but if not exposing comments requires any amount of effort at all then I think that effort is much better invested in making even the smallest amount of progress towards providing a proper way to do whatever they'd be doing with comments.

5 Likes

Cool stuff. Assuming function returning a value can this somehow print:

    enter: doSomething(42, b: 256)
    exit:  doSomething(42, b: 256) -> 53

where 53 is the function result?


Hope we end up with something much nicer than the often hated C macros.

1 Like

I worry that unless we invent a parallel type hierarchy for "incomplete" syntax nodes that do not provide the same interface, this will be an attractive nuisance.

  • It looks like you can access all children of a parent node, but you cannot
  • It looks like you could determine relative source locations within the parent chain, but you cannot (the lengths may be shortened due to children).
  • You can sometimes get more information than others; e.g. in your example you don't know it's a struct/class, but you would sometimes know that if you use a macro inside the original body. This is a somewhat general issue with syntax tress of course.
1 Like

This is late, and it seems like this proposal should have an official revision, but for posterity:

  • What is your evaluation of the proposal?

nil, as this proposal has far reaching impact that is impossible to assess without full development flow integration.

-1 as proposed due to no guarantee of said flow integration. As is, this proposal seems to introduce APIs that are rather hard to build and impossible to debug.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Given the motivation for the proposal seems to be "we should have expression macros", it, of course, addresses that problem. However, the larger, implicit issue of, some expressions can't be built in Swift may or may not be a problem. It doesn't seem to be a problem, per se. At most it's a missing feature with a variety of solutions. I would say some types of expressions are probably desirable to be expressed directly in code, but not all, so all I can say here is "maybe, :man_shrugging:".

There are probably better solutions for the vast majority of examples provided in this thread, including more limited versions of compile time evaluation.

  • Does this proposal fit well with the feel and direction of Swift?

In that result builders are precedent for enabling special language modes in Swift, perhaps, but overall I don't think macros fit well in Swift's previously espoused goals, at least at a high level. Given a macro's rather impenetrable nature, especially now, they're basically magic that can be introduced from any package. Is progressive disclosure really just "accept the magic until you want to learn about it"?

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I haven't used modern macros from Rust. I still have nightmares about Obj-C codebases that used C macros extensively. These are different and much better in many ways, but worse in others (textual macros were easily inspectable and could be reasoned about, if only because macro authors needed to reason about them in the same way).

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Can't say I have expertise here but I did try to read the pitch and proposal and tried the example repo. It was largely inscrutable given there's no documentation and no tooling help, as well as no debugger, so it doesn't seem super useful for anyone who isn't already intimately familiar with SwiftSyntax.

3 Likes

Thank you! I will try to make time to work on it some more and give feedback on what I learned.

I think this would need a declaration macro that must be applied to the function itself.

I'm +1 on the vision for macros so far.

I personally think this still fits in with progressive disclosure. Particularly since I don't see macro writing as something one will generally need to learn. They are in the domain of DSL writers where it is natural to work with syntax nodes. I'd rather see this then some sort of macro sub-language which I think is the only other alternative.

Swift needs magic for things like Codable and I'd rather see that magic in libraries than in the compiler. By making it difficult to write macros it makes them less likely to be abused or used in ways that don't result in straightforward transformation.

I think debugging for a macro creator will be reasonable, but I think it would be nice if IDEs like Xcode could somehow preview the syntax node transformation to make it easier to peek behind the curtain.

2 Likes

I was able to get surprisingly far with just an expression macro that would "wrap" the function body in a trailing closure, which at least makes it easy to capture the return value and do something with it before returning. But yeah, printing the arguments isn't possible unless you pass them to the macro explicitly. By abusing the heck out of closures and implicit single-expression returns, I was able to write this:

func adder(_ a: Int, _ b: Int) -> Int {
  #trace(a, b) {
    let result = a + b
    return result
  }
}

which gets transformed to this:

func adder(_ a: Int, _ b: Int) -> Int {
  {
    print("enter: \(#function) <- ", terminator: "");
    {print("a = \(a), ", terminator: "");print("b = \(b), ", terminator: "");}()
    print()
    let __macro_local_0 = {
      let result = a + b
      return result
    }()
    print("exit:  \(#function) -> \(__macro_local_0)")
    return __macro_local_0
  }()
}

with this macro:

@expression public macro trace<Result>(
  _ args: Any..., body: () -> Result
) -> Result = #externalMacro(module: "MacroExamplesPlugin", type: "TraceMacro")

Then, if I invoke let x = adder(100, 200), I get this output:

enter: adder(_:_:) <- a = 100, b = 200, 
exit:  adder(_:_:) -> 300
Expand for plug-in implementation
import SwiftSyntax
import SwiftSyntaxBuilder
import _SwiftSyntaxMacros

public struct TraceMacro: ExpressionMacro {
  public static func expansion(
    of macro: MacroExpansionExprSyntax,
    in context: inout MacroExpansionContext
  ) throws -> ExprSyntax {
    guard let closure = macro.trailingClosure else {
      throw CustomError.message("#trace must have a trailing closure")
    }

    let argPrinterClosure = ClosureExprSyntax(
      statements: CodeBlockItemListSyntax {
        for arg in macro.argumentList {
          let argString = String(describing: arg.withTrailingComma(false))
          let printExpr: ExprSyntax =
            """
            print("\(raw: argString) = \\(\(arg.expression)), ", terminator: "")
            """
          CodeBlockItemSyntax(item: .expr(printExpr), semicolon: .semicolonToken())
        }
      }
    )

    let resultIdentifier = context.createUniqueLocalName()
    return """
      {
        print("enter: \\(#function) <- ", terminator: "");
        \(argPrinterClosure)()
        print()
        let \(resultIdentifier) = \(closure)()
        print("exit:  \\(#function) -> \\(\(resultIdentifier))")
        return \(resultIdentifier)
      }()
      """ as ExprSyntax
  }
}

Note: I had to use the toolchain from this comment above from Doug to get around expanded variable declaration issues.

I'm not exactly proud of the implementation of that macro, but it was a really fun exercise to see how far I could get with just the expression features.

3 Likes

I agree that writing them will be rare (thankfully more rare than in C) but using them is also difficult since every macro will have different rules. At least with result builders the basic syntax within the builder is the same whether or no the builder has implemented supported for all of the build* methods.

The magic of Codable and Equatable synthesis is slight, as you can hand write that logic (and we do) quite easily. Not really possible with generalized macros.

What can you possibly be basing that on? Swift's current debugging experience is unreliable at best and macros would need a new type of compile time debugging as well as runtime support (writer and user debugging). Personally, I've run out of patience with Swift's tooling.

I think this is a good point, but probably one that makes more sense for declaration macros or some sort of auto-protocol conformance macro. I hope it will be best practice to look for a user implementation of any auto-generated code and allow that to override what the macro would have written. I think there is a strong need to establish good best practices for cases like this.

I guess I'm not clear on what form of debugging this refers to. If talking about lldb, I hope it will be able to step through the output of -dump-macro-expansions.

Macro writers need to be able to set breakpoints in a macro during the build process when it's invoked so we can actually investigate the context and help build the macro. Macro users, if they have access to the macro source, may need to be able to do the same thing at runtime. I'm not sure how useful that is and it may already be possible, but the macro writer case seems impossible right now. In that way writing and debugging macros seems even more difficult than result builders.

3 Likes

Ok. That seems easily solvable with advances to tooling though. I get that it might take awhile for tooling to improve if it isn't prioritized, but I don't see anything that would prevent creation of better tooling. Hopefully it would be implemented as a way to debug expansions regardless of if macro source is present, but that might be considered private API... It would be nice to see more about debugging in the proposal even if just as a future direction.

There may be some workarounds–admittedly tedious–like passing in a closure to hold a breakpoint during development.

Overall, I see a need for macros in some contexts and either manipulating syntax trees or using templates is what every modern macro system does. I personally prefer the former. I agree that good tooling should be part of the picture, but I still think the proposal is a step in the right direction.

EDIT: Small correction

1 Like

My point is that I've lost faith that Apple will properly prioritize such tooling, so proposals that require it are automatic no votes unless the tooling is already being built in some way. Even then, the longevity of such tooling is questionable, so I'm generally negative on anything that needs it at all.

2 Likes

While no doubt it is valid to want an excellent tooling experience, as part of the Swift Evolution process we consider the implementability of proposals but do not evaluate either the quality of any actual implementation nor review any engineering roadmap.

3 Likes

I'm aware. Personally, implementability should include tooling and the ecosystem as a whole, since no one uses a feature in a vacuum. Apple has especially lost my trust over the last two years, from implementation to tooling to maintenance. The language workgroup can consider what they want here, but this is how I'll vote from now on, at least for proposals like this.

3 Likes

having worked with SwiftSyntax extensively over the summer, i’ve learned that the version of SwiftSyntax you are using is quite important, and getting it right is more, not less difficult than getting normal build dependency resolution right. especially if you are distributing libraries and therefore (likely) targeting multiple toolchain versions per release.

i won’t comment on the pros or cons of macros themselves, but i really do not think it is a good idea to start exposing this feature until the SwiftSyntax versioning story is more fleshed out.

5 Likes

I strongly dislike Codable and Result Builders, and part of that dislike is that they remind me of early-2000s boost libraries that found "clever" ways to do things which seemed like they should be impossible which resulted in utterly alien code. They're even worse than those libraries, though, because they didn't actually find a clever way to make things like CodingKeys work, and instead had compiler magic added specifically to enable their weird misuse of the language. My assumption is that the unstated context behind the macro vision document is that other SDK teams at Apple want to do similar things for their APIs, and macros are a way to address that demand without a steady stream of niche special compiler magic. If those are indeed the two options, I would certainly pick a macro system every day.

I share many of the concerns around tooling. This proposal is implicitly promoting SwiftSyntex to being part of the language, and that's a very big deal for a library which is currently unstable, undocumented, and not really realistically usable by third-parties. I don't think it makes sense to send SwiftSyntax through the evolution process (I really can't imagine it getting much in the way of productive feedback), but there does need to be some sort of story for how SwiftSyntax is going to get to a state where it's reasonable for a language feature to depend on it.

Similarly, the question of if macros will be debuggable is very relevant to the question of if macros are a good idea or not. Macros which can be stepped through in a debugger and macros which can't be are categorically different in what sorts of problems they can cause, and it's very important to design a macro system which falls into the first category. "That's an implementation question" suggests that this macro system hasn't been designed to ensure that it's debuggable, and that's a pretty big problem.

10 Likes

To clarify, whether the design of a feature is conducive to or inhibits debugging is in scope for review, and any suggestions as to changes to the design which would improve the user debugging experience would be welcome.

“I’m only in favor of this feature if it can be implemented well”—in scope for Swift Evolution.

“I’m only in favor of this feature if it will be implemented well”—not up to us.

3 Likes

Does the dependence on SwiftSyntax mean that SwiftSyntax is now a part of official language specification?

6 Likes

MacroExpansionContext:

  • What are the moduleName and fileName properties for?

  • Does the public init(moduleName:fileName:) exist only for testing?