A Possible Vision for Macros in Swift

Evolution Pitches
61

Right, but I think that's a reasonable trade-off to make. If we tried for something declarative like macro_rules, we're going to bump against its limitations and also want something like procedural macros (Rust's equivalent to what I'm proposing) in addition. I'd rather try have just one feature here, even if it means it's a little harder to get started, and bridge the gap with tutorials and tools like swift package init --type macro.

That's interesting. The implicit package would likely have very different dependencies than where the source code is written, and in many cases be built for a different architecture, both of which can be problematic for inline code. For example, the macro code will need access to swift-syntax to parse/manipulate syntactic ASTs, whereas the main program will not. The macro code will build for the host platform (wherever you do your builds) whereas the main program will build for the target platform (wherever you run the code). If we're going to do inline macro code, I think we'll need to make the macro inputs untyped---i.e., you need a well-formed syntax tree, but it isn't type checked.

Yes, it does mirror the conformance. I wrote it this way because the macro declaration is something that's visible to the main program, whereas the conformance is visible only to the package that's used by the compiler.

Yeah, this is a great idea. I went ahead and expanded the main Macro protocol in the prototype to have all of the information that was in the macro declaration: name, type signature, documentation. That way, the conformance becomes the single source of truth, and there's no declaration in the language.

There are downsides to this approach---where do you "jump to definition" on a macro? On the other hand, it eliminates a pile of repetition, and lets us defer defining macros in the surface language.

Yes, of course! I can't believe I forgot to write down the function use-case. I think we want such a macro to only affect the body of the function, and not the type signature. I'll add it to my next revision, thank you.

Picking up a whole bunch of use cases like this, without adding bespoke features for each, is exactly the goal of a macro system.

This is a good point. We already have attribute syntax for applying some additional semantics to a declaration; we should re-use that for applying macros to a declaration, rather than invent a new syntax like #traced func f(). I do think the # syntax makes sense for cases where we aren't annotating an existing thing, but generating a new thing, e.g., the expression cases like #line.

That's a really interesting use case. The way in which you need to extend several other types makes it a tricky one, because it could start to feel like spooky-action-at-a-distance. It's worth thinking through this one!

Yeah, this is a missing piece. I should have talked about macro hygiene in general, and some kind of gensym is probably the only effective strategy since we're dealing with parsing syntax.

Doug

7 Likes
62

Debuggability is an important issue here. @allevato broke it down into the various pieces, so I'll respond to those specifically:

I'm sure this is possible, but I don't have a sense of how much work it is. The macro-expanded source code could be recorded somewhere in debug info, with line/column information pointing into the expanded source code. I know DWARF has as support for expressing C macros, but I don't know if it's suitable or if we would need something else.

What I'd like here is a way to ask the compiler to emit a macro expansion description, for each macro that gets expanded. That macro expansion description would describe all of the inputs to the macro expansion in a manner that doesn't need the compiler to run---it should be easy to take that description and drop it into a test case within the macro package, so you can develop and debug the macro transform outside of the compiler, where it's easier to iterate and re-run all of your tests.

This is one of the advantages of having macro definitions in a separate package: you have the macro definition (which will be a library), and can have test targets that you use for most of your macro development, without ever needing it to be fed into the compiler. I've been working this way with the prototype, and it's really easy to iterate on a macro definition in that environment.

Possibly? I added an expand-macros operation into the prototype, but I've since never used it---it's far easier to work with individual tests, like I mentioned above. Moreover, providing this operation implies that the fully-expanded source must also compile and have the same semantics as the original source. That could be a goal, but it has downsides. For example, we would have to figure out how to deal with things like #line and #column in the expanded form to maintain the same semantics. It could also preclude some implicit expansions of macros (e.g., applying a macro to the argument of a function), unless we also add some kind of do-not-expand directive to avoid double-expanding the argument.

Doug

8 Likes
63

Maybe @Adrian_Prantl has some insight here?

To parallel @beccadax 's comment above, I think this is an extremely compelling reason (in addition to those she already mentioned) to focus entirely on macros-as-a-separate-package and not support declaring macros in the same module in any fashion. It would likely require significant additional effort to support debugging of macros during compilation, and instead of telling authors "you can have this great development experience if you use a separate package, or this lesser experience if you put your macro in the same module", let's only give them the great experience that's also the easiest to implement.

One place where this potentially falls down is if we want to support macros in Swift scripts run in the interpreter. For Swift scripts in general, we would need better support for importing source files anyway (some officially supported form of -enable-source-import), so maybe the answer to that use case is that you still have to keep the macro in a separate source file next to your script or on the search path and we import it like any other module.

1 Like
64

Would it make sense to tie in macros with #if conditional compilation feature somehow? Say, to allow Bool-returning macros to be used in this context:

macro func usesIntelCPU() -> Bool {
  #if arch(i386) || arch(x86_64)
  return true
  #else
  return false
  #endif
}

// ...
// contrived example to illustrate the place of use
#if usesIntelCPU()
// code compatible only with Intel CPUs here
#else
// code for other CPUs
#endif
8 Likes
65

This is particularly important if we want to let people design macros that read like normal expressions. To be unable to copy the currently executing line of source code into the debugger and run it, because that line happens to contain a macro evaluation, would be rather upsetting.

Hm, in this case the conditionals need to be evaluated with the context of the place where the macro is referenced, not in the context of building the macro module itself.

1 Like
66

To me, making the source for macros be separate from the ordinary source of Swift modules just means drawing a line between them so that macro definitions and the types and functions they use internally aren't conflated with the ordinary source of the program. It doesn't have to mean that macros can only be built and linked as separate executables with full SDK access, the way they likely will be in the first phase; we could still e.g. load a macro module and interpret the macro body directly within the compiler, and as long as we can do that, the tooling story doesn't seem doomed to be second-rate.

We may find that first-phase macros don't automatically migrate to allow that second-phase implementation approach, though.

3 Likes
67

More on the implementation side, with a nod to Codable, I’d like to see any macro system cover all current gyb files in the standard library.

If I remember correctly tuple operators and Int128 are both generated through gyb and it would be great to see these migrated into Swift proper.

1 Like
68

We're likely to start with macros that work on expressions, so that won't be able to replace uses of gyb that generate a lot of different top-level declarations. That sort of top-level declaration-generating macro also has a lot of interesting but complex tooling interactions that we'd like time to think carefully about. For example, you really want syntax highlighting, code completion, and so on to work normally when editing the "body" that's repeatedly instantiated, almost as if the body was generic code that just happened to have the power to actually declare its generic parameters as structs or whatever.

7 Likes
69

I suggest you don't introduce any declarative macros at all but rather function macros if any. Mixing declarative/functional programming into an imperative language usually makes it unreadable, not to mention the mental stress switching between them. This is similar to the functional style template programming in C++ which usually considered as very hard and unreadable. This is one of biggest mistakes in C++ and Rust didn't learn from that mistake (which they later realized).

70

Not to derail the conversation, but it seems to me that this generalization is too broad. I personally don't find that using SwiftUI or any other declarative and functional features of Swift makes it unreadable. In fact, quite the opposite, being able to between declarative/functional and imperative paradigms where needed improves readability in my opinion, as I can pick an appropriate tool for a job instead of trying to shoehorn the only available paradigm everywhere.

If you personally find that mixing paradigms in Swift makes it unreadable, that's totally fine, but deserves a separate thread if it's not considered off-topic for this Forums category altogether?

If you're finding that declarative macros in Rust have specific issues, would you mind listing them with concrete examples to explain what exactly went wrong?

5 Likes
71

I suspect it would be possible to build declarative macros atop procedural macros, much like the various result-builder APIs that wrap procedural NSLayoutConstraint code in a declarative syntax.

72

This is a good idea. Macro-like things are also very related to the compile-time constant features as well. Unfortunately there have been many ideas for compile-time constants and they've all failed spectacularly (this is the most recent instance).

I don't want this feature to suffer from having to figure out compile-time constants first, but just want to mention it :slightly_smiling_face:

73

I’m curious how macros would work in practice with documentation comments and DocC in generated code. I assume swift-syntax could generate documentation comments automatically in some cases or comments could be included in Protocols instead of generated code.

I think macros would allow some interesting DSLs if combined with result builders. Such as a DSL to decode/encode binary data or ASN.1 encodings that isn’t 1:1 with a data model. The result builder would ideally be in a separate static library to avoid dynamic library bloat since it would be completely factored out at compile time, yet would still need public visibility.

4 Likes
74

After reading A possible vision for macros in Swift again (excellent breakdown by the way) I persuaded against introducing general macros. I'm minded that the urge for macros shows us where we need to improve Swift in ways that shouldn't need macros.

For example, one of the examples given - clamping values, is a runtime action. There’s nothing really in need of compile time behaviour, the optimiser should (either now or in the future) be able to handle static values to improve performance.

The only thing I think we’re in need of are compiler generated member-wise initialisers and handlers - eg a KeyPath equivalent of Codable (and TBH 99% of CodingKeys are one to one and on to KeyPaths anyway, and that's effectively what the Swift compiler generates).

10 Likes
75

I'd love these kinds of discussions to bring in the big picture more. In particular the opportunity costs of such a huge endeavour. So without being very technically versed, I want to give my two cents.

Macros by themselves might appeal to some group of users. So would many other potential extensions to Swift. But do they yield the best return on time and attention invested? Are there not features to round out that are more fundamental and add less complexity to the language as a whole? I'm thinking ownership, concurrency and cross-platform portability, just to name a few.

While expert Swift users still have trouble understanding and using a feature as fundamental as Swift's implementation of the actor model, here we're opening this huge can of worms that is a macro system. That feels slightly alienating to me.

Wanting to add a macro system is a sign of a mature and saturated language. A very comfortable but also delicate spot to be in. From there you can make the language more robust, consistent, diagnosable, portable, performant etc. Or you can inflate the language's surface and bloat it with the next best feature we can come up with.

To me personally and deep down, it feels like a design deficiency of language A if it inspires users to want another language B to produce code in A. And I don't think Swift is that deficient. I don't get the idea that this will free the evolution process from having to consider/implement subjective syntax sugar features. To the degree macros are just for subjective syntax sugar, they are by definition non-essential. And to the degree they fill gaps in the language, the language itself should do that.

Also, if Swift adds features, it should continue humbly learning from the languages that are most loved. And that is in particular since years and without close second: Rust.

8 Likes
76

I actually think Macros are one of the features that would benefit the widest subset of Swift developers. Things like custom literals, and the ability to synthesise conformances to library protocols, are huge.

Macros are far, far more than syntax sugar.

One of the things developers consistently say they love about Swift compared to other languages is how much it focuses on checking things at build-time rather than at run-time. Macros allow us to do much, much more at build-time; they are essentially build-time plugins.

8 Likes
77

So then I assume that the kind of things macros enable require some sort of meta level and could never be enabled by native language features? And that's the ultimate (almost mathematically given) reason many languages have macros?

1 Like
78

Take something like custom literals - as a maintainer of a URL library, I know that users have wanted build-time checking for URL literals for a long time. Something like this:

let goodURL = #url(http://example.com)

let badURL = #url(http:/example.com) // ❌ - compile time error: Invalid URL

Previously, when the Foundation team were considering adding a non-failable initializer for URL string literals, I outlined what my plans were for adding build-time checking to my own library:

Basically - when you consider what it would take to implement something like this (as a 'native language feature'), the result ends up looking exactly like the proposed macro system - the compiler takes the string value (http:example.com), passes it in to my library, my library can emit diagnostics.

It's not syntax sugar. It's a whole new feature, and greatly enhances the kinds of build-time processing we can do for Swift programs.

13 Likes
79

Anyone tried using C pre-processor with Swift? As stupid as it sounds it might actually work (as "C preprocessor doesn't know C language").

Preprocessors always seemed to me a band aid / handicapped solution to work around language limitations (and if there's an alternative way through the language itself I'd pick it any day), yet nevertheless it is very powerful and sometimes the only choice.

1 Like
80

I like this example, but don't see macros as the solution. For example, the following code has enough information for a future Swift compiler to run it at compile time rather than runtime:

extension URL {
    init(_ staticString: StaticString) {
        guard let url = Self.init(string: staticString.description) else {
            fatalError("Invalid URL")
        }
        self = url
    }
}

One improvement to this could be replacing fatalError with a compile time error (compileError?) which fails if found in the final product.

3 Likes