A Possible Vision for Macros in Swift

More on the implementation side, with a nod to Codable, I’d like to see any macro system cover all current gyb files in the standard library.

If I remember correctly tuple operators and Int128 are both generated through gyb and it would be great to see these migrated into Swift proper.

1 Like

We're likely to start with macros that work on expressions, so that won't be able to replace uses of gyb that generate a lot of different top-level declarations. That sort of top-level declaration-generating macro also has a lot of interesting but complex tooling interactions that we'd like time to think carefully about. For example, you really want syntax highlighting, code completion, and so on to work normally when editing the "body" that's repeatedly instantiated, almost as if the body was generic code that just happened to have the power to actually declare its generic parameters as structs or whatever.

7 Likes

I suggest you don't introduce any declarative macros at all but rather function macros if any. Mixing declarative/functional programming into an imperative language usually makes it unreadable, not to mention the mental stress switching between them. This is similar to the functional style template programming in C++ which usually considered as very hard and unreadable. This is one of biggest mistakes in C++ and Rust didn't learn from that mistake (which they later realized).

Not to derail the conversation, but it seems to me that this generalization is too broad. I personally don't find that using SwiftUI or any other declarative and functional features of Swift makes it unreadable. In fact, quite the opposite, being able to between declarative/functional and imperative paradigms where needed improves readability in my opinion, as I can pick an appropriate tool for a job instead of trying to shoehorn the only available paradigm everywhere.

If you personally find that mixing paradigms in Swift makes it unreadable, that's totally fine, but deserves a separate thread if it's not considered off-topic for this Forums category altogether?

If you're finding that declarative macros in Rust have specific issues, would you mind listing them with concrete examples to explain what exactly went wrong?

5 Likes

I suspect it would be possible to build declarative macros atop procedural macros, much like the various result-builder APIs that wrap procedural NSLayoutConstraint code in a declarative syntax.

This is a good idea. Macro-like things are also very related to the compile-time constant features as well. Unfortunately there have been many ideas for compile-time constants and they've all failed spectacularly (this is the most recent instance).

I don't want this feature to suffer from having to figure out compile-time constants first, but just want to mention it :slightly_smiling_face:

I’m curious how macros would work in practice with documentation comments and DocC in generated code. I assume swift-syntax could generate documentation comments automatically in some cases or comments could be included in Protocols instead of generated code.

I think macros would allow some interesting DSLs if combined with result builders. Such as a DSL to decode/encode binary data or ASN.1 encodings that isn’t 1:1 with a data model. The result builder would ideally be in a separate static library to avoid dynamic library bloat since it would be completely factored out at compile time, yet would still need public visibility.

4 Likes

After reading A possible vision for macros in Swift again (excellent breakdown by the way) I persuaded against introducing general macros. I'm minded that the urge for macros shows us where we need to improve Swift in ways that shouldn't need macros.

For example, one of the examples given - clamping values, is a runtime action. There’s nothing really in need of compile time behaviour, the optimiser should (either now or in the future) be able to handle static values to improve performance.

The only thing I think we’re in need of are compiler generated member-wise initialisers and handlers - eg a KeyPath equivalent of Codable (and TBH 99% of CodingKeys are one to one and on to KeyPaths anyway, and that's effectively what the Swift compiler generates).

10 Likes

I'd love these kinds of discussions to bring in the big picture more. In particular the opportunity costs of such a huge endeavour. So without being very technically versed, I want to give my two cents.

Macros by themselves might appeal to some group of users. So would many other potential extensions to Swift. But do they yield the best return on time and attention invested? Are there not features to round out that are more fundamental and add less complexity to the language as a whole? I'm thinking ownership, concurrency and cross-platform portability, just to name a few.

While expert Swift users still have trouble understanding and using a feature as fundamental as Swift's implementation of the actor model, here we're opening this huge can of worms that is a macro system. That feels slightly alienating to me.

Wanting to add a macro system is a sign of a mature and saturated language. A very comfortable but also delicate spot to be in. From there you can make the language more robust, consistent, diagnosable, portable, performant etc. Or you can inflate the language's surface and bloat it with the next best feature we can come up with.

To me personally and deep down, it feels like a design deficiency of language A if it inspires users to want another language B to produce code in A. And I don't think Swift is that deficient. I don't get the idea that this will free the evolution process from having to consider/implement subjective syntax sugar features. To the degree macros are just for subjective syntax sugar, they are by definition non-essential. And to the degree they fill gaps in the language, the language itself should do that.

Also, if Swift adds features, it should continue humbly learning from the languages that are most loved. And that is in particular since years and without close second: Rust.

8 Likes

I actually think Macros are one of the features that would benefit the widest subset of Swift developers. Things like custom literals, and the ability to synthesise conformances to library protocols, are huge.

Macros are far, far more than syntax sugar.

One of the things developers consistently say they love about Swift compared to other languages is how much it focuses on checking things at build-time rather than at run-time. Macros allow us to do much, much more at build-time; they are essentially build-time plugins.

8 Likes

So then I assume that the kind of things macros enable require some sort of meta level and could never be enabled by native language features? And that's the ultimate (almost mathematically given) reason many languages have macros?

1 Like

Take something like custom literals - as a maintainer of a URL library, I know that users have wanted build-time checking for URL literals for a long time. Something like this:

let goodURL = #url(http://example.com)

let badURL = #url(http:/example.com) // ❌ - compile time error: Invalid URL

Previously, when the Foundation team were considering adding a non-failable initializer for URL string literals, I outlined what my plans were for adding build-time checking to my own library:

Basically - when you consider what it would take to implement something like this (as a 'native language feature'), the result ends up looking exactly like the proposed macro system - the compiler takes the string value (http:example.com), passes it in to my library, my library can emit diagnostics.

It's not syntax sugar. It's a whole new feature, and greatly enhances the kinds of build-time processing we can do for Swift programs.

13 Likes

Anyone tried using C pre-processor with Swift? As stupid as it sounds it might actually work (as "C preprocessor doesn't know C language").

Preprocessors always seemed to me a band aid / handicapped solution to work around language limitations (and if there's an alternative way through the language itself I'd pick it any day), yet nevertheless it is very powerful and sometimes the only choice.

1 Like

I like this example, but don't see macros as the solution. For example, the following code has enough information for a future Swift compiler to run it at compile time rather than runtime:

extension URL {
    init(_ staticString: StaticString) {
        guard let url = Self.init(string: staticString.description) else {
            fatalError("Invalid URL")
        }
        self = url
    }
}

One improvement to this could be replacing fatalError with a compile time error (compileError?) which fails if found in the final product.

3 Likes

The whole point is that we want a guarantee that the input has been checked at build-time. There is a huge difference between:

  • Definitely checked at build-time, and
  • Maybe checked at build-time, otherwise crashes at runtime

You can read the previous thread if you want; what you are suggesting is what the Foundation team suggested, and I explained why I think a build-time plugin is actually superior and delivers something closer to what developers are actually asking for.

Also, it is worth noting: expecting the compiler's automatic constant-folding to statically evaluate an entire URL parser is unrealistic. Instead, we can just ship this little utility (call it a "macro" or "linting plugin" or whatever), and the compiler can call out to that for some automated checks.

In some cases, it may be able to avoids parsing the string at runtime at all, in other cases, it might just diagnose some obviously invalid inputs, and in yet other cases, it might be possible to do something in-between: to generate some kind of internal data structure which optimises the runtime parsing in a library-specific way (e.g. marking the locations of important sections within the string). There is no way that the compiler could automatically perform the latter.

4 Likes

The only actual alternatives to macros for this use case would be to implement an equivalent to C++'s constexpr/consteval functions. While I fully support doing so, eventually, it's a lot more complicated than adding a macro system. Especially since just copying the way C++ does it would be a bad idea.

There's a lot of design space that would need to be explored for (good) first-class build time execution of Swift code. A macro system not only gives us build time evaluation without that work, it can also be used to do a lot of that exploration.

3 Likes

It's always been a goal of Swift to be a good language to write great APIs in. I see macros primarily as a power tool for API development, which they achieve in two main ways:

  1. allowing API authors to better bridge the gap between generality (which often requires more abstraction, which often in turn adds circumlocution on the client side) and convenience; and
  2. allowing API authors to check preconditions that, for whatever reason, go beyond what can be expressed in the type system.

With that in mind, I think there are a lot of interesting potential interactions between macros and constant evaluation; but I do think we need to be a little more explicit about what we mean by constant evaluation.

Full constant evaluation means evaluating expressions all the way down to a normal form, which (glossing over some details) means a literal value of the expression's type. It requires all the values it sees to have this normal form, and it is blocked by any parts of the program that it doesn't understand. In Swift terms, the latter includes (at a minimum) calls to anything that isn't either @inlinable or non-resilient; if the constant evaluator sees such a call, it must fail.

This imposes some inherent limitations on what full constant evaluation can achieve. Expressions of resilient type, for example, cannot possibly be constant-evaluated (unless they throw) because they must ultimately produce a value by calling a non-delegating init, and the non-delegating inits of resilient types cannot be @inlinable. Expressions of optional resilient type can be constant-evaluated, but only if they produce nil (or throw). To make this concrete, we cannot fully constant-evaluate an expression of URL type unless it does not actually produce a URL.

Sometimes this is desirable. If you need a hard guarantee that a particular expression can be emitted as a compile-time constant, you really do need full constant evaluation. Otherwise, you need some way to avoid being blocked by code you can't understand statically. There are two basic ideas for doing that:

  • Work with abstract computations as completely opaque.
  • Separate some subset of the computation that can be reliably constant-evaluated while the remainder stays abstract.

Macros can be a tool for achieving both of these, within limits. A macro that decides not to analyze and break apart a sub-expression is treating it as an opaque computation, and a macro could certainly restrain itself to doing things that are consistent with constant-evaluation. For example, consider a macro that recognizes uses of + with string literals/interpolations and concatenates them. This is, effectively, treating the interpolation operands as opaque and doing an abstract constant-evaluation of the concatenation operator. The main limitations are that macros must work with source programs, and so they cannot acquire information that isn't obvious in the source (e.g. understanding that a variable referenced from the macro operand is initialized to 1 and not re-assigned prior to the point where the macro is used) or produce results that cannot be expressed in source. Procedural macros also require writing code in terms of expressions instead of values, which can be a significant conceptual leap from other programming tasks.

When a "constant" evaluator can work with opaque computations, that's usually called abstract interpretation. Abstract interpretation is able to work with opaque values, treat opaque calls as producing such values (and potentially leaving them in arbitrary memory), and so on. Unfortunately, it is inherently a best-effort analysis, because it is often very difficult for the interpreter to make basic decisions like whether to take a branch or not. (For example: suppose the program reads a stored property of an opaque value and compares it against a value previously read from that same property; when are these known to be the same?) Because of this, it is rarely (if ever) used in core language semantics; instead, it's mostly used in tools like static analysis engines, where gradual improvement of the tool over time is seen as a good thing.

Constant evaluation of subsets of computation is a more promising idea for cases where full constant evaluation is not possible but some kind of constant evaluation is still desired. A lot of these use cases boil down to doing some sort of precondition check statically, either purely for diagnostic purposes or as an optimization to avoid doing it at runtime. If the preconditions of a function can be identified statically, then in principle they can be constant-evaluated when the arguments are compile-time constants, and then the rest of the function can be executed normally. One advantage of this sort of design is that it can naturally degrade to a dynamic check in cases where the arguments aren't statically known; this can happen even with init(integerLiteral:) in several different situations.

One final, somewhat unrelated interaction between constant evaluation and macros that's worth calling out is that constant evaluation could conceivably be used from macros. If macros are integrated into the compiler, then in principle a macro could ask the compiler to try to constant-evaluate a particular expression, then do different things based on the result. For example, a macro could ask whether one of its argument expressions was the constant value false. This would require a lot of prerequisite work to enable, though.

21 Likes

The macro function seems to be very powerful. I am curious whether the following writing method becomes possible with this function?

public class Button {
    public var tapHandler: () -> Void
    
    public init(tapHandler: @escaping () -> Void) {
        self.tapHandler = tapHandler
    }
}

// with micros
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: #weakClosure self.onTapButton)
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}

// generated code
public class View {
    public lazy var button: Button = {
        return Button(tapHandler: { [weak self] in
            guard let self else { return }
            self.onTapButton()
        })
    }()
    
    public init() {}
    
    public func onTapButton() {
        print("did tapped")
    }
}
1 Like

I'm not trying to call anyone out, but I have to say that responses like the below in a recent thread about a possible new language feature by a member of the language workgroup concern me:

I'm not voicing any opinion on that particular feature, but dismissing or implying a higher bar for new language features once macros are available is something that should be avoided, at least to me. I think we can all agree that the discoverability and usability of custom macros is inherently worse than language-level features, and even if something could be implemented entirely with macros, I would hate for the mere existence of macros to deter the discussion and development of new language features.

6 Likes

It’s always reasonable to ask whether a potential feature can be implemented using existing features and what the trade-offs of that approach might be. As a result, any reasonably general feature does raise the bar for similar features.

Macros are intentionally very general, and they’ll get more general over time. That doesn’t mean we won’t consider adding a feature if it could possibly be implemented with macros — that could cover nearly the entire language — but if a feature doesn’t lose much as a macro application, that will definitely argue against adding it.

Discoverability in particular is not a very strong motivator on its own. Basically everything would be more discoverable as a language feature, even if only because it would show up in lists of language features. Of course, even that would start to lose its value if those lists were thousands of lines long.

16 Likes