[Pitch] Package Manager Support for Custom Macros

Introduction

Macros provide a way to extend Swift by performing arbitary syntactic transformations on input source code to produce new code. One example for this are expression macros which were previously proposed in SE-0382. This proposal covers how custom macros are defined, built and distributed as part of a Swift package.

Motivation

SE-0382 and A Possible Vision for Macros in Swift covered the motivation for macros themselves, defining them as part of a package will offer a straightforward way to reuse and distribute macros as source code.

Proposed solution

Macros implemented in an external program can be declared as part of a package via a new macro target type, defined in
the CompilerPluginSupport library:

public extension Target {
    /// Creates a macro target.
    ///
    /// - Parameters:
    ///     - name: The name of the macro.
    ///     - dependencies: The macro's dependencies.
    ///     - path: The path of the macro, relative to the package root.
    ///     - exclude: The paths to source and resource files you want to exclude from the macro.
    ///     - sources: The source files in the macro.
    static func macro(
        name: String,
        dependencies: [Dependency] = [],
        path: String? = nil,
        exclude: [String] = [],
        sources: [String]? = nil
    ) -> Target {
}

Similar to package plugins (SE-0303 "Package Manager Extensible Build Tools"), macro plugins are built as executables for the host (i.e, where the compiler is run). The compiler receives the paths to these executables from the build system and will run them on demand as part of the compilation process. Macro executables are automatically available for any target that transitively depends on them via the package manifest.

A minimal package containing the implementation, definition and client of a macro would look like this:

import PackageDescription
import CompilerPluginSupport

let package = Package(
    name: "MacroPackage",
    targets: [
        .macro(name: "MacroImpl"),
        .target(name: "MacroDef", dependencies: ["MacroImpl"]),
        .executableTarget(name: "MacroClient", dependencies: ["MacroDef"]),
    ]
)

Macro implementations will be executed in a sandbox similar to package plugins, preventing file system and network access. This is a practical way of encouraging macros to not depend on any state other than the specific macro expansion node they are given to expand and its child nodes (but not its parent nodes), and the information specifically provided by the macro expansion context. If in the future macros need access to other information, this will be accomplished by extending the macro expansion context, which also provides a mechanism for the compiler to track what information the macro actually queried.

Detailed Design

SwiftPM builds each macro as an executable for the host platform, applying certain additional compiler flags such as search paths to the SwiftSyntax library supplied by the toolchain. Each target that transitively depends on a macro will have access to it, concretely this happens by SwiftPM passing -load-plugin-executable to the compiler to specify which executable contains the implementation of a certain macro module. The macro defintion refers to the module and concrete type via an #externalMacro declaration which allows any dependency of the defining target to have access to the concrete macro. If any target of a library product depends on a macro, clients of said library will also get access to any public macros. Macros can have dependencies like any other target, but product dependencies of macros need to be statically linked, so explicitly dynamic library products cannot be used by a macro target.

Concretely, the code for the macro package shown earlier would contain a macro implementation looking like this:

import SwiftSyntax
import SwiftCompilerPlugin
import SwiftSyntaxBuilder
import SwiftSyntaxMacros

@main
struct MyPlugin: CompilerPlugin {
  var providingMacros: [Macro.Type] = [FontLiteralMacro.self]
}

/// Implementation of the `#fontLiteral` macro, which is similar in spirit
/// to the built-in expressions `#colorLiteral`, `#imageLiteral`, etc., but in
/// a small macro.
public struct FontLiteralMacro: ExpressionMacro {
  public static func expansion(
    of macro: some FreestandingMacroExpansionSyntax,
    in context: some MacroExpansionContext
  ) -> ExprSyntax {
    let argList = replaceFirstLabel(
      of: macro.argumentList,
      with: "fontLiteralName"
    )
    let initSyntax: ExprSyntax = ".init(\(argList))"
    if let leadingTrivia = macro.leadingTrivia {
      return initSyntax.with(\.leadingTrivia, leadingTrivia)
    }
    return initSyntax
  }
}

/// Replace the label of the first element in the tuple with the given
/// new label.
private func replaceFirstLabel(
  of tuple: TupleExprElementListSyntax,
  with newLabel: String
) -> TupleExprElementListSyntax {
  guard let firstElement = tuple.first else {
    return tuple
  }

  return tuple.replacing(
    childAt: 0,
    with: firstElement.with(\.label, .identifier(newLabel))
  )
}

The macro definition would look like this:

public enum FontWeight {
  case thin
  case normal
  case medium
  case semiBold
  case bold
}

public protocol ExpressibleByFontLiteral {
  init(fontLiteralName: String, size: Int, weight: FontWeight)
}

/// Font literal similar to, e.g., #colorLiteral.
@freestanding(expression) public macro fontLiteral<T>(name: String, size: Int, weight: FontWeight) -> T = #externalMacro(module: "MacroImpl", type: "FontLiteralMacro")
  where T: ExpressibleByFontLiteral

And the client of the macro would look like this:

import MacroDef

struct Font: ExpressibleByFontLiteral {
  init(fontLiteralName: String, size: Int, weight: MacroDef.FontWeight) {
  }
}

let _: Font = #fontLiteral(name: "Comic Sans", size: 14, weight: .thin)

Impact on existing packages

Since macro plugins are entirely additive, there's no impact on existing packages.

Alternatives considered

The original pitch of expression macros considered declaring macros by introducing a new capability to package plugins, but since the execution model is significantly different and the APIs used for macros are external to SwiftPM, this idea was discarded.

Future Directions

Evolution of SwiftSyntaxMacros

Macro targets will get access to a version of SwiftSyntaxMacros and associated libraries which ships with the Swift toolchain. This means package authors aiming to support multiple Swift compiler versions may have to resort to conditional compilation based on the Swift compiler version to handle the evolving API of SwiftSyntax. It also means clients of packages which are utilizing macros may need to wait for their dependencies to adopt to new APIs in order to upgrade to a new version of the Swift tools. We expect these limitations to eventually be solved by stabilizing the relevant SwiftSyntax APIs at which point macros could depend on SwiftSyntax via a package dependency on the source repository.

Generalized support for additional manifest API

The macro target type is provided by a new library CompilerPluginSupport as a starting point for making package manifests themselves more extensible. Support for product and target type plugins should eventually be generalized to allow other types of externally defined specialized target types, such as, for example, a Windows application.

12 Likes

An initial implementation of this feature is already available behind the pre-release tools-version: https://github.com/apple/swift-package-manager/pull/6185 and Support macros as executables by neonichu Β· Pull Request #6200 Β· apple/swift-package-manager Β· GitHub

2 Likes

Great to see progress on this. I have some questions/comments for this:

These dependencies are for the actual macro declaration right?

This seems interesting to me. It kinda implies that the MacroDef target is depending on the MacroImpl and you could write import MacroImpl inside your MacroDef sources, but that doesn't work right? That dependency is mostly there for making sure the implementation is compiled before it is used so it can be loaded to the compiler? Should we use a separate macros parameter here on target similar to how we did it for plugins?

Can this actually ever happen? I was talking with @ahoppen about this at length recently and if I understand the architecture correctly then the compiler is talking with the macro executable through serialising the AST back and forth between the two executables. While I can see a way to make the source generation side of SwiftSyntax API stable I cannot see how we make the parser API stable which is also needed in this setup.

Since we are building all the macro implementations as separate executables it seems like another great place where we might want to allow separate dependency trees between the actual macro implementations and the other products that are build in the graph.

Plugins definition targets are not allowed to have any dependency at all. Here with macros this seems differently where both the declaration and the implementation can have arbitrary dependencies. Why is there a difference between the two now?

2 Likes

Is this the intended design? My understanding from previous vision/proposal documents was that the communication protocol between the compiler and the macros would not require coupling between the two and that the SwiftSyntax dependency would be chosen by the macro author, not the one provided by the toolchain.

Coupling to the toolchain's SwiftSyntax would make packages with macros fragile because they would have to compile with arbitrary versions of the toolchain's SwiftSyntax library instead of pinning their own version that they know they're compatible with, and would require stability guarantees that have not been made yet for SwiftSyntax.

cc @Douglas_Gregor

4 Likes

I don't think so, the same type of dependency is already possible with executable targets. Personally, I never understood why we made plugins special here.

A separate tree doesn't really scale, two different macros can still not have different dependencies.

1 Like

As far as I understood this requirement is in place because the compiler and macro are communicating the AST over stdin/stdout and have to serialize it. Since the AST is not in a stable format we need the versions to match up. Though I agree this makes macros incredibly brittle.

1 Like

Why can’t they have separated ones? We are building statically linked executables in the end right? I do understand that separate trees are not scaling nicely though!

My understanding was that the communication protocol would send source to the macro, which the macro would parse, transform, and then the macro would send it back to the compiler again as source text. Then the versions don't need to match up unless the macro author wants to use new syntax features that aren't in an older version of SwiftSyntax. (And even then, they would recognize unknown syntax and leave it untouched, if the nature of the macro allows for that.)

If this property is dropped, then the overall macro proposals effectively require source stability of SwiftSyntax going forward, which wasn't a requirement before and seems unrealistic. Language evolution would be made much more difficult if SwiftSyntax APIs had to remain 100% source-compatible with older AST representations and not be able to be a precise representation of the language at the version of the SwiftSyntax release.

4 Likes

You're right, in principle this would work, but it requires a complete redesign of SwiftPM's dependency resolution process if we want something better than a simple second list of dependencies for a particular type of targets. Basically each leaf node would need to have its own tree of dependencies and we'd need some way to deal with a potential explosion of redundant copies. Definitely something we could pursue in the future, though.

2 Likes

I might have misunderstood @ahoppen here when we last talked. If we send Strings back and forth we still have problems when we have source code that cannot be parsed by the SwiftSyntax version of the macro e.g. when new language features such as if expressions come around (always mix up expression and statements here)

1 Like

Individual macros pinning their own versions of swift-syntax isn't possible with SwiftPM today, since there can only be one version of a certain package in the entire graph. This lead us down the path of vending libraries in the toolchain, @Douglas_Gregor can elaborate.

1 Like

Yes, that's a possibility, but it would at least be a conscious choice by the macro author. A macro author pinning to let's say SwiftSyntax's 5.8 release will know for sure that they might not be able to process syntactic constructs that come in Swift versions after that.

The alternative, if we require the macro to be compiled with a specific version of the toolchain, is that macro authors can never vend a stable macro, because it makes semver meaningless; they have to ensure that their macro compiles with the version of SwiftSyntax that comes with any toolchain a user might use.

Consider situations like https://github.com/apple/swift-syntax/pull/1354 where a property name was changed in SwiftSyntax. A macro author could write code that references the letOrVar property to get that token. In a later semver of SwiftSyntax, that property is renamed bindingKeyword to better represent new features like inout. If a macro author can't pin to a specific version of SwiftSyntax, their macro will just fail to compile when users update to a new toolchain because the property name has changed, and therefore the macro won't work for any Swift source code. Instead, if the author they could pin to a specific version that they know is compatible with their macro, then they could still reference letOrVar keyword and their macro would compile successfully and work with existing Swift source code, even if compiling with a newer version of the toolchain. If someone wrote code that used a different keyword than let or var, the macro might not recognize that, but it would at least have the ability to do something graceful like preserve it by making no change.

Potentially breaking macros by coupling to the toolchain's version of SwiftSyntax seems like it would violate our source compatibility guarantees for new versions of the language. Stabilizing SwiftSyntax might resolve that, but at the same time it would make updating the language syntax very difficult if the API has to be able to preserve compatibility with older code, and my understanding from previous discussions has been that stabilizing SwiftSyntax in that way has not been in scope for the macros work (and is in fact not a goal at all).

This sounds like a limitation that needs to be lifted specifically for host tools by SwiftPM, or I fear that the experience of both writing and consuming macros is going to be a lot more difficult.

7 Likes

FWIW I resorted to only use precompiled binaries for all my plugins executables because for me it was not acceptable for my plugin to impose any restrictions on the dependency graph (in my case Swift Syntax) of the consumers of the plugin.
This effectively lets me choose a exact version of SwiftSyntax (which has undergone a few sourcebreaking changes)
A unintended benefit was improved build performance and also the executable can be built in release mode (spm currently builds them in debug)

4 Likes

Under the expectation that SwiftSyntax will stabilize and given the library’s prominence in macros (which fall under SE), I think changes to SwiftSyntax should be evaluated through SE. Namely, our collective expertise is much more likely to foresee and avoid problems before locking in a particular, stable API.

2 Likes

This is something I've been wondering about. The dependencies of macro targets don't necessarily have anything to do with other targets or dependencies, because they run on the host rather than the compilation target. They should really have an independent tree of dependencies, but clearly that would add a lot of complexity.

That's why I wish we'd started having discussions about build system support much earlier. IMO, the compiler implementation is actually secondary to build-system support, but it seems that's where the focus has been until now.


Anyway, since we do only have one tree of dependencies, that means every target may be built as part of a macro. I think we need an addition to BuildSettingCondition so macro builds can be customised.

For example, WebURL's parser supports reporting validation errors when it fails to parse something. This information is currently unused - I don't think this detailed failure information is really useful for the runtime parser, since that is for runtime strings. Even if I threw an Error which communicated that a URL failed to parse because it contains an unclosed IPv6 address (e.g. https://[::1), I don't expect anybody's code to actually be able to handle that. And actually throwing detailed errors has a significant cost for binary size and prohibits compiler optimisations.

But for the compile-time version of WebURL, things weigh differently - the "error handler" in this case is a human developer seeing a literal string rather than pre-written code parsing unknown data, so it is worth giving more precise diagnostics. Binary size is a secondary concern, and since the information is more realistically useful the hit to parsing performance becomes worth it.

So I want to conditionally compile some of this code based on whether the module is being built as part of a macro implementation or not. AFAICT there is no way to do this. If we had an addition to BuildSettingCondition, I could write something like:

.target(
  name: "WebURL",
  swiftSettings: [
    .define("ENABLE_ENHANCED_DIAGNOSTICS", .when(configuration: .macro))
  ]
),

If I understand this correctly, it is somewhat disappointing. It means that if I want to expose a macro as part of WebURL, I'll have to add a new "WebURLMacros" module to hold the declaration, and users will need to import that explicitly - which limits discoverability of the feature. Presumably they would also need to add the "WebURLMacros" target as a dependency of their own targets, making it even more difficult to start using.

What I want (the ideal situation) is to expose the macro declaration from the WebURL module itself, so no additional imports are necessary.

The macro implementation would be in another module, and depend on the compile-time version of the WebURL module (this would be a cycle, so the compile-time version of WebURL would need to exclude the macro declaration).

At the same time, all of the host-side stuff (the macro implementation and compile-time version of the parser) would not be built unless the client actually used the macro.

I'm not sure this is possible with just one dependency tree.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WebURL (client)              β”‚
β”‚                              β”‚
β”‚- WebURL parser               β”‚
β”‚                              β”‚    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚- #url(...) macro declaration β”œβ”€β”€β”€β–Ίβ”‚   #url(...) macro implementation (host)    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                  β”‚
                                                  β–Ό
                                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                    β”‚ WebURL (host)                β”‚
                                    β”‚                              β”‚
                                    β”‚- WebURL parser               β”‚
                                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Why not add support for all target settings?

  • Resources: It is possible that your macro implementation might depend on a binary resource, such as a canned database, JSON/XML files, ML models, or other data.

  • Plugins: Similarly, macros may need to invoke host tools and generate some code or resources. I think this makes a lot of sense, given that macros themselves are a kind of host tool.

  • Swift settings: Has various uses, including setting defines, enabling experimental language features, etc.

  • C/C++ settings (future): We may add support for mixed-language targets, and the C/C++ side may need compiler flags.


I think we will also need to address the inherently lack of scalability at some point. If every macro, in every one of your dependencies, can pin its own version of swift-syntax, building a project could easily require building dozens of different versions of that library.

For example, if WebURL's macro support uses swift-syntax version X, and your app/library uses WebURL, you'll need to download and build swift-syntax X as part of building your project.

And if you use a library which uses WebURL, again - download and build swift-syntax X. And so on, down the chain, everybody who depends on the library in any way takes a build-time hit. It can be very far removed from the code at the end of the chain, picking up many versions of swift-syntax along the way.

And if the idea is that every macro pins to a different version, that hit multiplies for every macro you use - WebURL was written to use version X, but some other library uses version Y, and another uses version Z, and if they are anywhere in the dependency graph, you take a hit from each of them.

6 Likes

I think it might make sense to have at least two dependency trees.

  • One for the "build" (any plugins/macros)
  • Another the actual code being built

Sorry i dont have any better terms to describe it.

I always imagined a separate counterpart to a Package.swift (i.e. Workspace.swift or Build.swift )

Which allows you to provide tools like plugins/macros (and maybe even typesafe config objects for them) that are usable/importable in the Package.swift

In my mind these would have two completely independent dependency-trees.

But this seems like something larger out of scope for this proposal.

1 Like

I think this is an interesting idea, but to actually support this in a meaningful way would require being able to build the module twice, once for the purpose of macros and again for the purpose of non-macros.

This should be possible and may just be unclear because the example focuses only on macros. The MacroDef target can contain any other non-macro related library code. The only "extra" component needed for macros is the macro target itself.

Mostly to keep things simple initially. I think plugins and Swift settings indeed make sense. I'm unsure about resources, but that's more a question for @Douglas_Gregor or @rintaro on whether that's something that'll work on the compiler side with the way resources currently work (using Foundation and bundles).

You're not wrong, but I don't think this solves any of our problems here (since multiple macros would still share a single version) and it seems more like a bandaid to me.

Two dependency trees makes sense when they're actually inherently different, e.g. tools needed to develop your library that aren't necessary for clients, like a command plugin that lints your source code. Clients will never even build that code, so it would make sense to keep those dependencies separate. This was something previously discussed when plugins were proposed and I am not sure it was made clear that this only really achieves anything for command plugins since they're not part of the "regular" build at all.

Macros and build-tool plugins still need to be build by anyone using the library, whether it is the package author or clients, so we would only be solving a very narrow problem where all the macros and plugins happen to agree on a version, but the library code doesn't. That can be useful in some cases, but it seems too limited and hard to explain to make it worth the effort to me.

I think the real thing we would want is for every leaf of the dependency tree to have the potential for diverging dependencies, e.g. if you're building two independent libraries, there's no technical reason for them to share versions either. I don't think macros or host-side tools are special in any way here, definitely not in terms of the design and implementation. Unfortunately, this isn't how SwiftPM was originally designed -- the majority of its current architecture and implementation hinges on the fact that each package can only exist once in the dependency tree, all the way from dependency resolution down to the built products being intermingled in a single directory.

In the fullness of time, this is certainly a problem worth solving, but without presupposing a design, I cannot see a way to achieve anything here without a potentially multi-year effort. Practically speaking, I don't see an alternative to accepting this limitation as a given for the purpose of the discussion around macros.

3 Likes

Right, but wouldn't this mean unconditionally downloading and compiling the macro implementation, including swift-syntax and the host versions of any targets it depends on?

My understanding is that separating the macro declaration in to its own module is what allows that stuff to potentially be built on-demand (because clients will need to depend on that declaration module in order to use it). Is that not the case? Does the compiler somehow call out to SwiftPM to build the external macro implementation when it is used, or is there some kind of scanner to detect which macro targets need to be built?

Yeah that's exactly what I'm after :slight_smile: Would that be particularly problematic?

Actually, I think it may even be required for this use-case. Otherwise we get this strange-looking dependency graph:

.macro(name: "WebURLMacroImpl", dependencies: ["WebURL"]),
.target(name: "WebURL", dependencies: ["WebURLMacroImpl"]),

Despite what it looks like, there isn't really a cycle here (or there shouldn't be) - the macro implementation does not actually depend on itself. This code doesn't even get linked together because one is a macro implementation. I think the only way out would be for WebURL to separate its implementation and API in to separate modules, which is actually an enormous amount of churn. Would SwiftPM's cycle detection be able to ignore macro targets?

1 Like

Yeah definitely can see that this is a really large undertaking.
I guess what i would really want is for each packages build tools to have their own dependency tree.

For plugins currently my workaround of vending prebuilt binaries works reasonably well. (This in the end gives me a completely isolated dependency-tree)
Would the current proposal allow me to do the same?

I dont think so, because i would need to build against the toolchains SwiftSyntax right?