A Possible Vision for Macros in Swift

The whole point is that we want a guarantee that the input has been checked at build-time. There is a huge difference between:

  • Definitely checked at build-time, and
  • Maybe checked at build-time, otherwise crashes at runtime

You can read the previous thread if you want; what you are suggesting is what the Foundation team suggested, and I explained why I think a build-time plugin is actually superior and delivers something closer to what developers are actually asking for.

Also, it is worth noting: expecting the compiler's automatic constant-folding to statically evaluate an entire URL parser is unrealistic. Instead, we can just ship this little utility (call it a "macro" or "linting plugin" or whatever), and the compiler can call out to that for some automated checks.

In some cases, it may be able to avoids parsing the string at runtime at all, in other cases, it might just diagnose some obviously invalid inputs, and in yet other cases, it might be possible to do something in-between: to generate some kind of internal data structure which optimises the runtime parsing in a library-specific way (e.g. marking the locations of important sections within the string). There is no way that the compiler could automatically perform the latter.

4 Likes

The only actual alternatives to macros for this use case would be to implement an equivalent to C++'s constexpr/consteval functions. While I fully support doing so, eventually, it's a lot more complicated than adding a macro system. Especially since just copying the way C++ does it would be a bad idea.

There's a lot of design space that would need to be explored for (good) first-class build time execution of Swift code. A macro system not only gives us build time evaluation without that work, it can also be used to do a lot of that exploration.

3 Likes

It's always been a goal of Swift to be a good language to write great APIs in. I see macros primarily as a power tool for API development, which they achieve in two main ways:

  1. allowing API authors to better bridge the gap between generality (which often requires more abstraction, which often in turn adds circumlocution on the client side) and convenience; and
  2. allowing API authors to check preconditions that, for whatever reason, go beyond what can be expressed in the type system.

With that in mind, I think there are a lot of interesting potential interactions between macros and constant evaluation; but I do think we need to be a little more explicit about what we mean by constant evaluation.

Full constant evaluation means evaluating expressions all the way down to a normal form, which (glossing over some details) means a literal value of the expression's type. It requires all the values it sees to have this normal form, and it is blocked by any parts of the program that it doesn't understand. In Swift terms, the latter includes (at a minimum) calls to anything that isn't either @inlinable or non-resilient; if the constant evaluator sees such a call, it must fail.

This imposes some inherent limitations on what full constant evaluation can achieve. Expressions of resilient type, for example, cannot possibly be constant-evaluated (unless they throw) because they must ultimately produce a value by calling a non-delegating init, and the non-delegating inits of resilient types cannot be @inlinable. Expressions of optional resilient type can be constant-evaluated, but only if they produce nil (or throw). To make this concrete, we cannot fully constant-evaluate an expression of URL type unless it does not actually produce a URL.

Sometimes this is desirable. If you need a hard guarantee that a particular expression can be emitted as a compile-time constant, you really do need full constant evaluation. Otherwise, you need some way to avoid being blocked by code you can't understand statically. There are two basic ideas for doing that:

  • Work with abstract computations as completely opaque.
  • Separate some subset of the computation that can be reliably constant-evaluated while the remainder stays abstract.

Macros can be a tool for achieving both of these, within limits. A macro that decides not to analyze and break apart a sub-expression is treating it as an opaque computation, and a macro could certainly restrain itself to doing things that are consistent with constant-evaluation. For example, consider a macro that recognizes uses of + with string literals/interpolations and concatenates them. This is, effectively, treating the interpolation operands as opaque and doing an abstract constant-evaluation of the concatenation operator. The main limitations are that macros must work with source programs, and so they cannot acquire information that isn't obvious in the source (e.g. understanding that a variable referenced from the macro operand is initialized to 1 and not re-assigned prior to the point where the macro is used) or produce results that cannot be expressed in source. Procedural macros also require writing code in terms of expressions instead of values, which can be a significant conceptual leap from other programming tasks.

When a "constant" evaluator can work with opaque computations, that's usually called abstract interpretation. Abstract interpretation is able to work with opaque values, treat opaque calls as producing such values (and potentially leaving them in arbitrary memory), and so on. Unfortunately, it is inherently a best-effort analysis, because it is often very difficult for the interpreter to make basic decisions like whether to take a branch or not. (For example: suppose the program reads a stored property of an opaque value and compares it against a value previously read from that same property; when are these known to be the same?) Because of this, it is rarely (if ever) used in core language semantics; instead, it's mostly used in tools like static analysis engines, where gradual improvement of the tool over time is seen as a good thing.

Constant evaluation of subsets of computation is a more promising idea for cases where full constant evaluation is not possible but some kind of constant evaluation is still desired. A lot of these use cases boil down to doing some sort of precondition check statically, either purely for diagnostic purposes or as an optimization to avoid doing it at runtime. If the preconditions of a function can be identified statically, then in principle they can be constant-evaluated when the arguments are compile-time constants, and then the rest of the function can be executed normally. One advantage of this sort of design is that it can naturally degrade to a dynamic check in cases where the arguments aren't statically known; this can happen even with init(integerLiteral:) in several different situations.

One final, somewhat unrelated interaction between constant evaluation and macros that's worth calling out is that constant evaluation could conceivably be used from macros. If macros are integrated into the compiler, then in principle a macro could ask the compiler to try to constant-evaluate a particular expression, then do different things based on the result. For example, a macro could ask whether one of its argument expressions was the constant value false. This would require a lot of prerequisite work to enable, though.

21 Likes

The macro function seems to be very powerful. I am curious whether the following writing method becomes possible with this function?

public class Button {
    public var tapHandler: () -> Void
    
    public init(tapHandler: @escaping () -> Void) {
        self.tapHandler = tapHandler
    }
}

// with micros
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: #weakClosure self.onTapButton)
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}

// generated code
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: { [weak self] in
            guard let self else { return }
            self.onTapButton()
        })
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}
1 Like

I'm not trying to call anyone out, but I have to say that responses like the below in a recent thread about a possible new language feature by a member of the language workgroup concern me:

I'm not voicing any opinion on that particular feature, but dismissing or implying a higher bar for new language features once macros are available is something that should be avoided, at least to me. I think we can all agree that the discoverability and usability of custom macros is inherently worse than language-level features, and even if something could be implemented entirely with macros, I would hate for the mere existence of macros to deter the discussion and development of new language features.

6 Likes

It’s always reasonable to ask whether a potential feature can be implemented using existing features and what the trade-offs of that approach might be. As a result, any reasonably general feature does raise the bar for similar features.

Macros are intentionally very general, and they’ll get more general over time. That doesn’t mean we won’t consider adding a feature if it could possibly be implemented with macros — that could cover nearly the entire language — but if a feature doesn’t lose much as a macro application, that will definitely argue against adding it.

Discoverability in particular is not a very strong motivator on its own. Basically everything would be more discoverable as a language feature, even if only because it would show up in lists of language features. Of course, even that would start to lose its value if those lists were thousands of lines long.

16 Likes

The idea of AST to AST transformation by macro is excellent. It has very great potential. I want to show one practical example. I have created a proof of concept for the Power Assert library using currently publicly available code.

If you write a macro like the following:

let a = 4
let b = 7
let c = 12

#powerAssert(max(a, b) == c)
#powerAssert(a + b > c)

let john = Person(name: "John", age: 42)
let mike = Person(name: "Mike", age: 13)

#powerAssert(john.isTeenager)
#powerAssert(mike.isTeenager && john.age < mike.age)

It will generate the following code:

let a = 4
let b = 7
let c = 12

PowerAssert.Assertion(#"#powerAssert(max(a, b) == c)"#, line: 14).assert(max(a, b) == c).capture(expression: max(a, b), column: 13).capture(expression: a.self, column: 17).capture(expression: b.self, column: 20).capture(expression: max(a, b) == c, column: 23).capture(expression: c.self, column: 26).render()
PowerAssert.Assertion(#"#powerAssert(a + b > c)"#, line: 15).assert(a + b > c).capture(expression: a.self, column: 13).capture(expression: b.self, column: 17).capture(expression: a + b > c, column: 19).capture(expression: c.self, column: 21).render()

let john = Person(name: "John", age: 42)
let mike = Person(name: "Mike", age: 13)

PowerAssert.Assertion(#"#powerAssert(john.isTeenager)"#, line: 29).assert(john.isTeenager).capture(expression: john.isTeenager.self, column: 18).capture(expression: john.self, column: 13).render()
PowerAssert.Assertion(#"#powerAssert(mike.isTeenager && john.age < mike.age)"#, line: 30).assert(mike.isTeenager && john.age < mike.age).capture(expression: mike.isTeenager.self, column: 18).capture(expression: mike.self, column: 13).capture(expression: mike.isTeenager && john.age < mike.age, column: 29).capture(expression: john.age.self, column: 37).capture(expression: john.self, column: 32).capture(expression: mike.isTeenager && john.age < mike.age, column: 41).capture(expression: mike.age.self, column: 48).capture(expression: mike.self, column: 43).render()

Run this code produces the following output:

#powerAssert(max(a, b) == c)
             |   |  |  |  |
             7   4  7  |  12
                       false
#powerAssert(a + b > c)
             |   | | |
             4   7 | 12
                   false
#powerAssert(john.isTeenager)
             |    |
             |    false
             Person(name: "John", age: 42)
#powerAssert(mike.isTeenager && john.age < mike.age)
             |    |          |  |    |   | |    |
             |    false      |  |    42  | |    13
             |               |  |        | Person(name: "Mike", age: 13)
             |               |  |        false
             |               |  Person(name: "John", age: 42)
             |               false
             Person(name: "Mike", age: 13)

An executable project is available here.

Wouldn't it be great to use such rich assertions in the Swift testing framework? I look forward to a future where this kind of AST manipulation is in Swift.

Unfortunately, SwiftSyntax does not provide complete type information that does not appear in the source code, so there are some cases where code generation fails.

For example, the value cannot be captured if a type is omitted due to dot syntax or an anonymous closure argument is used.

It would be great if Swift macros could be passed the type information checked by the compiler as well as the source code structure to achieve this use case.

47 Likes

This is amazing! I would love to have these rich assertions for all of the tests I write. Thank you!

Yes, that's a great point. Since we are type-checking the arguments to a macro, it would be possible to annotate the various subexpressions with type information that could be provided to the macro. The big trick, as ever, is that we need to have a representation of the Swift type system that we can pass through.

Doug

19 Likes

+1 for the idea of figuring out how to get types of subexpressions over to macros.

However…. for a usecase like power asserts, maybe you could instead offer a way to get a fully-qualified name for references to a declaration? That seems like it would be easy to pass over a text-based protocol. No need to come up with a new format either, since it should all be spellable in Swift code.

I know we don’t really have fully-qualified names, but we can usually get pretty close to it in standard Swift.

This seems like a pretty useful feature for macros in general, regardless of type info.

Right, knowing which declaration a name refers to is another interesting bit of information that's determined by type checking and could be provided to macros.

Doug

Just following up, but with a few minor fixes to the compiler and pulling this into my local version of swift-syntax, it works as advertised:

% swiftc-dev -enable-experimental-feature Macros t.swift -sdk `xcrun --show-sdk-path -sdk macosx` && ./t
#powerAssert("hello".hasPrefix("h") && "goodbye".hasSuffix("y"))
             |       |         |    |  |         |         |
             "hello" true      "h"  |  "goodbye" false     "y"
                                    false

I did, however, change the signature and generic signature of the macro to the following:

  static var genericSignature: GenericParameterClauseSyntax? = nil
  static var signature: TypeSyntax = "(expression: Bool) ->   

In the near future, it should become possible to integrate these macros into a development compiler without having to rebuild the whole compiler, so folks can experiment more.

Doug

14 Likes

Hey all, I've pitched expression macros in another thread, the first proposal that's part of this vision.

Doug

9 Likes

I'd be interested in seeing more information provided in the MacroExpansionContext. When implementing a delegate method, I frequently find myself doing this:

func delegator(_ delegator: Delegator, willBegin operation: SomeOperation) {
    print(#function, delegator, operation)
}

func delegator(_ delegator: Delegator, didFinish operation: SomeOperation) {
    print(#function, delegator, operation)
}

This is a quick way for me to try out a new piece of code and quickly get a sense of the control flow between the delegator and the delegate. I was thinking that this would potentially be a nice macro:

func delegator(_ delegator: Delegator, willBegin operation: SomeOperation) {
    #printArguments
}

However, when I looked at the MacroExpansionContext, I didn't see a way I could implement this. Is there a way to get the containing scope and, if it's an argument-taking thing, get the list of arguments so I can splat out a print(...) call?

4 Likes

You can walk to the parent nodes of the MacroExpansionExprSyntax (just follow the “parent” property of each syntax node), and you’ll be able to get context information like that.

Doug

6 Likes

Ah, excellent! Sounds like that's sufficient already then :smile:

1 Like

If I were a wisenheimer, I'd say I want to write macros that generate macros.

Seriously, in my Swift programming, I often have to look at the build log to get enough information to just debug type issues. The compiler tries to output information, but Xcode doesn't reflect it back nicely. (And even more information would be better.) Point is, I would be very worried about the debug/error experience of macros unless there were enough resources available on the tooling side for really good support when things go wrong. The tooling effort is likely non-trivial.

7 Likes

Perhaps I missed it in the proposal or the thread, but would it be possible to use this system to capture the names of variables as they are declared? Or, to put it another way, to take some input in a macro and use it to declare the name of a let or var? I'm thinking of a macro I used in Obj-C in the past to declare views with accessibility identifiers to make debugging easier.

AXIDView(profileContainer);

This would be equivalent to:

UIView *profileContainer = [UIView new];
profileContainer.accessibilityIdentifier = @"profileContainer";

Here, I always want the accessibility identifier to be the same as the variable name so I can find it easily in the view debugger, layout constraint errors, or logs. I haven't thought much about it in SwiftUI, so I don't know if I would need an equivalent, but I believe I've wanted to be able to know the name of a variable at run-time in non-UIKit Swift code before.

3 Likes

In the system that I've described, all of the arguments to a macro need to be well-typed expressions before the macro will be expanded. That means profileContainer will need to already exist, so you won't be able to introduce it for the first time in AXIDView.

Now, it's possible that in the future we'll allow certain macro arguments to opt out of type checking, such they only need to be syntactically well-formed. Even so, we will want to be careful about allowing macros to declare new names, because that can have an impact on compile-time performance for incremental builds: specifically, we'd have to expand macros to figure out what names they declare, and those names could interfere with other non-macro code.

Doug

1 Like

Thanks. That makes sense. And one could presumably use the proposed macros to write something like autoAccessibilityID(someView), which is still an improvement.

This has probably been considered somewhere, but it just occurred to me that macros offer some interesting opportunities for interoperability with C++ templates. It's always nice when a C++ template can be imported as a first-class constrained generic, but without hints from the programmer, in general that can be arbitrarily hard… and likewise, writing those hints could be arbitrarily hard. Obviously using a template whose parameter is dependent on a Swift generic parameter would still erase type information, but it's something at least.