Passing Syntax directly to the macro without swift-syntax

Evolution Discussion
,
1

Building on Swift macros without requiring swift-syntax I wondered about the bare minimum changes required to persuade swift to export the syntax in a usable form.

How would y'all feel about the following to improve macro plugins that don't use swift-syntax (or may not even be written in Swift)?

This can be acheived by:

  1. making SwiftSyntax.Syntax itself Encodable (or some other mechanism to make the tree Encodable),
  2. extending SwiftCompilerPluginMessageHandling.PluginMessage.Syntax to allow Syntax to be included in the message to macro plugin,
  3. fix ASTGen to handle the extended SwiftCompilerPluginMessageHandling.PluginMessage.Syntax. (this removes the smelly "loop" of SwiftSyntax.Syntax being generated only to be stringified externally to SwiftSyntax)
extension Syntax: Encodable {
    enum CodingKeys: String, CodingKey{
        case kind
        case payload
    }
    
    public func encode(to encoder: any Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(kind.rawValue, forKey: .kind)
        if kind == .token {
            try container.encode(description, forKey: .payload)
        } else {
            try container.encode(Array(children(viewMode: .fixedUp)), forKey: .payload)
        }
    }
}

For Apple's default #stringify example the result is:

{
  "kind": "macroExpansionExpr",
  "payload": [
    { "kind": "token", "payload": "#" },
    { "kind": "token", "payload": "stringify" },
    { "kind": "token", "payload": "(" },
    {
      "kind": "labeledExprList",
      "payload": [
        {
          "kind": "labeledExpr",
          "payload": [
            {
              "kind": "infixOperatorExpr",
              "payload": [
                {
                  "kind": "declReferenceExpr",
                  "payload": [
                    { "kind": "token", "payload": "a " }
                  ]
                },
                {
                  "kind": "binaryOperatorExpr",
                  "payload": [
                    { "kind": "token", "payload": "+ " }
                  ]
                },
                {
                  "kind": "declReferenceExpr",
                  "payload": [
                    { "kind": "token", "payload": "b" }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    { "kind": "token", "payload": ")" },
    {
      "kind": "multipleTrailingClosureElementList",
      "payload": [ ]
    }
  ]
}
PoC: Improving macro build times with WebAssembly
2

Serializing the syntax tree into JSON for macros is not a viable option because it would mean that the compiler and the macro would need to use the exact same version of swift-syntax in order to communicate the syntax tree.

Even though it predates macros, as an example, let’s consider the introduction of if expressions, which changed the name of the if nodes from IfStmtSyntax to IfExprSyntax. Now, say your macro was written against a version of swift-syntax that still uses IfStmtSyntax but the compiler knows about if expressions (such a scenario is pretty common if you eg. use a Swift 5.10 compiler but your macro is written using swift-syntax 509.0.0). Then the compiler would generate JSON that produces a node with kind: "ifExpr", which the macro doesn’t know how to interpret. By just exchanging the source code, we work around this problem because the macro can interpret the source code with its own swift-syntax version. Also note that these problems aren’t solely for renamed syntax nodes, they also exist for more structural changes to the syntax tree, for which we’re able to provide compatibility shims in the swift-syntax library but which wouldn’t be possible in JSON.

2 Likes
3

Apologies for not being clear on the nature of extending SwiftCompilerPluginMessageHandling.PluginMessage.Syntax to allow Syntax to be included in the message to macro plugin. This would be a new parameter in addition to the current source, say syntax.

On further thought I would hope that the internal swift-syntax's version number would also be surfaced via `HostToPluginMessage.getCapability to all the plugin to decide if its compatible.

To reiterate, whether the plugin can interpret a specific HostToPluginMessage case is not the compiler's problem so long as PluginMessage.PROTOCOL_VERSION_NUMBER is set correctly.

4

Additionally, if we did replace source rather than adding syntax, the original string may be trivially reconstructed by visiting all the token kinds in order:

{ "kind": "token", "payload": "#" }
{ "kind": "token", "payload": "stringify" }
{ "kind": "token", "payload": "(" }
{ "kind": "token", "payload": "a " }
{ "kind": "token", "payload": "+ " }
{ "kind": "token", "payload": "b" }
{ "kind": "token", "payload": ")" }

This seems a guarentee we'd want anyway so isn't combersome.

5

I do understand where you’re coming from but it’s not something we are planning to support. Using swift-syntax is the recommended way of writing macros and while it is possible to write macros without swift-syntax, we currently don’t intend to extend the communication with the macro plugin to facilitate them.

PoC: Improving macro build times with WebAssembly
6

That is disappointing. To clarify, who is the "we" here?

1 Like