A Possible Vision for Macros in Swift

Evolution Pitches
73

I’m curious how macros would work in practice with documentation comments and DocC in generated code. I assume swift-syntax could generate documentation comments automatically in some cases or comments could be included in Protocols instead of generated code.

I think macros would allow some interesting DSLs if combined with result builders. Such as a DSL to decode/encode binary data or ASN.1 encodings that isn’t 1:1 with a data model. The result builder would ideally be in a separate static library to avoid dynamic library bloat since it would be completely factored out at compile time, yet would still need public visibility.

3 Likes
74

After reading A possible vision for macros in Swift again (excellent breakdown by the way) I persuaded against introducing general macros. I'm minded that the urge for macros shows us where we need to improve Swift in ways that shouldn't need macros.

For example, one of the examples given - clamping values, is a runtime action. There’s nothing really in need of compile time behaviour, the optimiser should (either now or in the future) be able to handle static values to improve performance.

The only thing I think we’re in need of are compiler generated member-wise initialisers and handlers - eg a KeyPath equivalent of Codable (and TBH 99% of CodingKeys are one to one and on to KeyPaths anyway, and that's effectively what the Swift compiler generates).

10 Likes
75

I'd love these kinds of discussions to bring in the big picture more. In particular the opportunity costs of such a huge endeavour. So without being very technically versed, I want to give my two cents.

Macros by themselves might appeal to some group of users. So would many other potential extensions to Swift. But do they yield the best return on time and attention invested? Are there not features to round out that are more fundamental and add less complexity to the language as a whole? I'm thinking ownership, concurrency and cross-platform portability, just to name a few.

While expert Swift users still have trouble understanding and using a feature as fundamental as Swift's implementation of the actor model, here we're opening this huge can of worms that is a macro system. That feels slightly alienating to me.

Wanting to add a macro system is a sign of a mature and saturated language. A very comfortable but also delicate spot to be in. From there you can make the language more robust, consistent, diagnosable, portable, performant etc. Or you can inflate the language's surface and bloat it with the next best feature we can come up with.

To me personally and deep down, it feels like a design deficiency of language A if it inspires users to want another language B to produce code in A. And I don't think Swift is that deficient. I don't get the idea that this will free the evolution process from having to consider/implement subjective syntax sugar features. To the degree macros are just for subjective syntax sugar, they are by definition non-essential. And to the degree they fill gaps in the language, the language itself should do that.

Also, if Swift adds features, it should continue humbly learning from the languages that are most loved. And that is in particular since years and without close second: Rust.

8 Likes
76

I actually think Macros are one of the features that would benefit the widest subset of Swift developers. Things like custom literals, and the ability to synthesise conformances to library protocols, are huge.

Macros are far, far more than syntax sugar.

One of the things developers consistently say they love about Swift compared to other languages is how much it focuses on checking things at build-time rather than at run-time. Macros allow us to do much, much more at build-time; they are essentially build-time plugins.

8 Likes
77

So then I assume that the kind of things macros enable require some sort of meta level and could never be enabled by native language features? And that's the ultimate (almost mathematically given) reason many languages have macros?

1 Like
78

Take something like custom literals - as a maintainer of a URL library, I know that users have wanted build-time checking for URL literals for a long time. Something like this:

let goodURL = #url(http://example.com)

let badURL = #url(http:/example.com) // ❌ - compile time error: Invalid URL

Previously, when the Foundation team were considering adding a non-failable initializer for URL string literals, I outlined what my plans were for adding build-time checking to my own library:

Basically - when you consider what it would take to implement something like this (as a 'native language feature'), the result ends up looking exactly like the proposed macro system - the compiler takes the string value (http:example.com), passes it in to my library, my library can emit diagnostics.

It's not syntax sugar. It's a whole new feature, and greatly enhances the kinds of build-time processing we can do for Swift programs.

13 Likes
79

Anyone tried using C pre-processor with Swift? As stupid as it sounds it might actually work (as "C preprocessor doesn't know C language").

Preprocessors always seemed to me a band aid / handicapped solution to work around language limitations (and if there's an alternative way through the language itself I'd pick it any day), yet nevertheless it is very powerful and sometimes the only choice.

1 Like
80

I like this example, but don't see macros as the solution. For example, the following code has enough information for a future Swift compiler to run it at compile time rather than runtime:

extension URL {
    init(_ staticString: StaticString) {
        guard let url = Self.init(string: staticString.description) else {
            fatalError("Invalid URL")
        }
        self = url
    }
}

One improvement to this could be replacing fatalError with a compile time error (compileError?) which fails if found in the final product.

3 Likes
81

The whole point is that we want a guarantee that the input has been checked at build-time. There is a huge difference between:

  • Definitely checked at build-time, and
  • Maybe checked at build-time, otherwise crashes at runtime

You can read the previous thread if you want; what you are suggesting is what the Foundation team suggested, and I explained why I think a build-time plugin is actually superior and delivers something closer to what developers are actually asking for.

Also, it is worth noting: expecting the compiler's automatic constant-folding to statically evaluate an entire URL parser is unrealistic. Instead, we can just ship this little utility (call it a "macro" or "linting plugin" or whatever), and the compiler can call out to that for some automated checks.

In some cases, it may be able to avoids parsing the string at runtime at all, in other cases, it might just diagnose some obviously invalid inputs, and in yet other cases, it might be possible to do something in-between: to generate some kind of internal data structure which optimises the runtime parsing in a library-specific way (e.g. marking the locations of important sections within the string). There is no way that the compiler could automatically perform the latter.

4 Likes
82

The only actual alternatives to macros for this use case would be to implement an equivalent to C++'s constexpr/consteval functions. While I fully support doing so, eventually, it's a lot more complicated than adding a macro system. Especially since just copying the way C++ does it would be a bad idea.

There's a lot of design space that would need to be explored for (good) first-class build time execution of Swift code. A macro system not only gives us build time evaluation without that work, it can also be used to do a lot of that exploration.

3 Likes
83

It's always been a goal of Swift to be a good language to write great APIs in. I see macros primarily as a power tool for API development, which they achieve in two main ways:

  1. allowing API authors to better bridge the gap between generality (which often requires more abstraction, which often in turn adds circumlocution on the client side) and convenience; and
  2. allowing API authors to check preconditions that, for whatever reason, go beyond what can be expressed in the type system.

With that in mind, I think there are a lot of interesting potential interactions between macros and constant evaluation; but I do think we need to be a little more explicit about what we mean by constant evaluation.

Full constant evaluation means evaluating expressions all the way down to a normal form, which (glossing over some details) means a literal value of the expression's type. It requires all the values it sees to have this normal form, and it is blocked by any parts of the program that it doesn't understand. In Swift terms, the latter includes (at a minimum) calls to anything that isn't either @inlinable or non-resilient; if the constant evaluator sees such a call, it must fail.

This imposes some inherent limitations on what full constant evaluation can achieve. Expressions of resilient type, for example, cannot possibly be constant-evaluated (unless they throw) because they must ultimately produce a value by calling a non-delegating init, and the non-delegating inits of resilient types cannot be @inlinable. Expressions of optional resilient type can be constant-evaluated, but only if they produce nil (or throw). To make this concrete, we cannot fully constant-evaluate an expression of URL type unless it does not actually produce a URL.

Sometimes this is desirable. If you need a hard guarantee that a particular expression can be emitted as a compile-time constant, you really do need full constant evaluation. Otherwise, you need some way to avoid being blocked by code you can't understand statically. There are two basic ideas for doing that:

  • Work with abstract computations as completely opaque.
  • Separate some subset of the computation that can be reliably constant-evaluated while the remainder stays abstract.

Macros can be a tool for achieving both of these, within limits. A macro that decides not to analyze and break apart a sub-expression is treating it as an opaque computation, and a macro could certainly restrain itself to doing things that are consistent with constant-evaluation. For example, consider a macro that recognizes uses of + with string literals/interpolations and concatenates them. This is, effectively, treating the interpolation operands as opaque and doing an abstract constant-evaluation of the concatenation operator. The main limitations are that macros must work with source programs, and so they cannot acquire information that isn't obvious in the source (e.g. understanding that a variable referenced from the macro operand is initialized to 1 and not re-assigned prior to the point where the macro is used) or produce results that cannot be expressed in source. Procedural macros also require writing code in terms of expressions instead of values, which can be a significant conceptual leap from other programming tasks.

When a "constant" evaluator can work with opaque computations, that's usually called abstract interpretation. Abstract interpretation is able to work with opaque values, treat opaque calls as producing such values (and potentially leaving them in arbitrary memory), and so on. Unfortunately, it is inherently a best-effort analysis, because it is often very difficult for the interpreter to make basic decisions like whether to take a branch or not. (For example: suppose the program reads a stored property of an opaque value and compares it against a value previously read from that same property; when are these known to be the same?) Because of this, it is rarely (if ever) used in core language semantics; instead, it's mostly used in tools like static analysis engines, where gradual improvement of the tool over time is seen as a good thing.

Constant evaluation of subsets of computation is a more promising idea for cases where full constant evaluation is not possible but some kind of constant evaluation is still desired. A lot of these use cases boil down to doing some sort of precondition check statically, either purely for diagnostic purposes or as an optimization to avoid doing it at runtime. If the preconditions of a function can be identified statically, then in principle they can be constant-evaluated when the arguments are compile-time constants, and then the rest of the function can be executed normally. One advantage of this sort of design is that it can naturally degrade to a dynamic check in cases where the arguments aren't statically known; this can happen even with init(integerLiteral:) in several different situations.

One final, somewhat unrelated interaction between constant evaluation and macros that's worth calling out is that constant evaluation could conceivably be used from macros. If macros are integrated into the compiler, then in principle a macro could ask the compiler to try to constant-evaluate a particular expression, then do different things based on the result. For example, a macro could ask whether one of its argument expressions was the constant value false. This would require a lot of prerequisite work to enable, though.

21 Likes
84

The macro function seems to be very powerful. I am curious whether the following writing method becomes possible with this function?

public class Button {
    public var tapHandler: () -> Void
    
    public init(tapHandler: @escaping () -> Void) {
        self.tapHandler = tapHandler
    }
}

// with micros
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: #weakClosure self.onTapButton)
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}

// generated code
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: { [weak self] in
            guard let self else { return }
            self.onTapButton()
        })
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}
1 Like
85

I'm not trying to call anyone out, but I have to say that responses like the below in a recent thread about a possible new language feature by a member of the language workgroup concern me:

I'm not voicing any opinion on that particular feature, but dismissing or implying a higher bar for new language features once macros are available is something that should be avoided, at least to me. I think we can all agree that the discoverability and usability of custom macros is inherently worse than language-level features, and even if something could be implemented entirely with macros, I would hate for the mere existence of macros to deter the discussion and development of new language features.

6 Likes
86

It’s always reasonable to ask whether a potential feature can be implemented using existing features and what the trade-offs of that approach might be. As a result, any reasonably general feature does raise the bar for similar features.

Macros are intentionally very general, and they’ll get more general over time. That doesn’t mean we won’t consider adding a feature if it could possibly be implemented with macros — that could cover nearly the entire language — but if a feature doesn’t lose much as a macro application, that will definitely argue against adding it.

Discoverability in particular is not a very strong motivator on its own. Basically everything would be more discoverable as a language feature, even if only because it would show up in lists of language features. Of course, even that would start to lose its value if those lists were thousands of lines long.

16 Likes
87

The idea of AST to AST transformation by macro is excellent. It has very great potential. I want to show one practical example. I have created a proof of concept for the Power Assert library using currently publicly available code.

If you write a macro like the following:

let a = 4
let b = 7
let c = 12

#powerAssert(max(a, b) == c)
#powerAssert(a + b > c)

let john = Person(name: "John", age: 42)
let mike = Person(name: "Mike", age: 13)

#powerAssert(john.isTeenager)
#powerAssert(mike.isTeenager && john.age < mike.age)

It will generate the following code:

let a = 4
let b = 7
let c = 12

PowerAssert.Assertion(#"#powerAssert(max(a, b) == c)"#, line: 14).assert(max(a, b) == c).capture(expression: max(a, b), column: 13).capture(expression: a.self, column: 17).capture(expression: b.self, column: 20).capture(expression: max(a, b) == c, column: 23).capture(expression: c.self, column: 26).render()
PowerAssert.Assertion(#"#powerAssert(a + b > c)"#, line: 15).assert(a + b > c).capture(expression: a.self, column: 13).capture(expression: b.self, column: 17).capture(expression: a + b > c, column: 19).capture(expression: c.self, column: 21).render()

let john = Person(name: "John", age: 42)
let mike = Person(name: "Mike", age: 13)

PowerAssert.Assertion(#"#powerAssert(john.isTeenager)"#, line: 29).assert(john.isTeenager).capture(expression: john.isTeenager.self, column: 18).capture(expression: john.self, column: 13).render()
PowerAssert.Assertion(#"#powerAssert(mike.isTeenager && john.age < mike.age)"#, line: 30).assert(mike.isTeenager && john.age < mike.age).capture(expression: mike.isTeenager.self, column: 18).capture(expression: mike.self, column: 13).capture(expression: mike.isTeenager && john.age < mike.age, column: 29).capture(expression: john.age.self, column: 37).capture(expression: john.self, column: 32).capture(expression: mike.isTeenager && john.age < mike.age, column: 41).capture(expression: mike.age.self, column: 48).capture(expression: mike.self, column: 43).render()

Run this code produces the following output:

#powerAssert(max(a, b) == c)
             |   |  |  |  |
             7   4  7  |  12
                       false
#powerAssert(a + b > c)
             |   | | |
             4   7 | 12
                   false
#powerAssert(john.isTeenager)
             |    |
             |    false
             Person(name: "John", age: 42)
#powerAssert(mike.isTeenager && john.age < mike.age)
             |    |          |  |    |   | |    |
             |    false      |  |    42  | |    13
             |               |  |        | Person(name: "Mike", age: 13)
             |               |  |        false
             |               |  Person(name: "John", age: 42)
             |               false
             Person(name: "Mike", age: 13)

An executable project is available here.

Wouldn't it be great to use such rich assertions in the Swift testing framework? I look forward to a future where this kind of AST manipulation is in Swift.

Unfortunately, SwiftSyntax does not provide complete type information that does not appear in the source code, so there are some cases where code generation fails.

For example, the value cannot be captured if a type is omitted due to dot syntax or an anonymous closure argument is used.

It would be great if Swift macros could be passed the type information checked by the compiler as well as the source code structure to achieve this use case.

47 Likes
88

This is amazing! I would love to have these rich assertions for all of the tests I write. Thank you!

Yes, that's a great point. Since we are type-checking the arguments to a macro, it would be possible to annotate the various subexpressions with type information that could be provided to the macro. The big trick, as ever, is that we need to have a representation of the Swift type system that we can pass through.

Doug

19 Likes
89

+1 for the idea of figuring out how to get types of subexpressions over to macros.

However…. for a usecase like power asserts, maybe you could instead offer a way to get a fully-qualified name for references to a declaration? That seems like it would be easy to pass over a text-based protocol. No need to come up with a new format either, since it should all be spellable in Swift code.

I know we don’t really have fully-qualified names, but we can usually get pretty close to it in standard Swift.

This seems like a pretty useful feature for macros in general, regardless of type info.

90

Right, knowing which declaration a name refers to is another interesting bit of information that's determined by type checking and could be provided to macros.

Doug

91

Just following up, but with a few minor fixes to the compiler and pulling this into my local version of swift-syntax, it works as advertised:

% swiftc-dev -enable-experimental-feature Macros t.swift -sdk `xcrun --show-sdk-path -sdk macosx` && ./t
#powerAssert("hello".hasPrefix("h") && "goodbye".hasSuffix("y"))
             |       |         |    |  |         |         |
             "hello" true      "h"  |  "goodbye" false     "y"
                                    false

I did, however, change the signature and generic signature of the macro to the following:

  static var genericSignature: GenericParameterClauseSyntax? = nil
  static var signature: TypeSyntax = "(expression: Bool) ->

In the near future, it should become possible to integrate these macros into a development compiler without having to rebuild the whole compiler, so folks can experiment more.

Doug

14 Likes
92

Hey all, I've pitched expression macros in another thread, the first proposal that's part of this vision.

Doug

9 Likes