SE-0382 (second review): Expression Macros

Evolution Proposal Reviews
1

Hello Swift community,

The second review of SE-0382: "Expression Macros" begins now and runs through February 20, 2023.

This version of the proposal incorporates changes requested after the first round of review. At the conclusion of that review, the language workgroup described the rationale for delineating the separation between language design through the Swift Evolution process and further improvements to the Swift Syntax library independently of this process. This second review is meant to be focused on questions of language design and, specifically, such questions as they relate to aspects which have undergone revision.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager via the forum messaging feature. When contacting the review manager directly, please keep the proposal link at the top of the message.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

https://github.com/apple/swift-evolution/blob/main/process.md

Thank you,

Xiaodi Wu
Review Manager

15 Likes
SE-0382: Expression Macros
Update on SE-0382 and SE-0389 (expression macros and attached macros)
[Accepted] SE-0382: Expression Macros
2

Object literals

Object literals

@freestanding(expression) macro colorLiteral<T: ExpressibleByColorLiteral>(red: Float, green: Float, blue: Float, alpha: Float) -> T
@freestanding(expression) macro imageLiteral<T: ExpressibleByImageLiteral>(resourceName: String) -> T
@freestanding(expression) macro fileLiteral<T: ExpressibleByFileReferenceLiteral>(resourceName: String) -> T

The object literals allow one to reference a resource in a program of various kinds. The three kinds of object literals (color, image, and file) can be described as expression macros. The type signatures provided above are not exactly how type checking currently works for object literals, because they aren't necessarily generic. Rather, when they are used, the compiler currently looks for a specially-named type (e.g., _ColorLiteralType) in the current module and uses that as the type of the corresponding color literal. To maintain that behavior, we propose to type-check macro expansions for object literals by performing the same lookup that is done today (e.g., for _ColorLiteralType) and then using that type as the generic argument for the corresponding macro. That way, the type checking behavior is unchanged when moving from special object literal expressions in the language to macro declarations with built-in implementations.

Given the way these things work (looking for a special type in a module and not really being generic), and the concepts they refer to, perhaps we should reconsider their role as part of the language. I understand that they needed direct support from the compiler in the past, but this seems like the perfect opportunity to break them out and make them library features.

Xcode Playgrounds could include these macro definitions in a support library, which is implicitly imported via a command-line flag. I think that's a nicer solution, and it encourages other platforms and libraries to define their own literals.

6 Likes
3

I'm not sure whether this is the right place but I would like to propose:

#standalone
or
#autonomous

as alternate names for #freestanding.

4

Yes, that's a good point. We could deprecate these object literals as language features (a la SE-0383) and have the libraries that _ColorLiteralType instead provide the macro declarations and implementations.

Doug

6 Likes
5

Dont know if t this is the appropriate thread.
A lot of the difficulty in providing useful functionality is that you can only operate on the syntax.

But Swift often relies on type-inference.

I read somewhere that the code is typechecked before expanding macros.

Would it be possible to "fully annotate" the code before letting the macro process it?

What i mean is turning something like this


func makeThing() -> String { "a string" }

class A {
  let a = makeThing()
}

Into this beforehand


func makeThing() -> String { "a string" }

class A {
  let a: String = makeThing() 
}
1 Like
6

My opinion stays largely the same compared to the first review. The general approach of the proposal is solid. I also like the changes.

I have implemented multiple macros using the partial implementation of this feature. It is great how far you can come with the proposed features. However, all of my macros had to (mis)use type inference in the generated code because of the lack of type information. This is two-fold:

  1. The macro often does not know the types of expressions passed to the macro. In most cases, the name of the type would be all that is needed.
  2. The macro has no way of knowing the capabilities (conformances, properties, methods, ...) of types that are used in the code passed to the macro.

The idea of @anreitersimon would solve the first issue. However, I am not sure if it is feasible.

While I think that the proposal as is can be very valuable on its own, many macros would for certain face the same problems as me.

1 Like
7

It's technically possible, but I feel that a better approach is the one outlined in future directions, where the macro expansion context gains a query that can ask for the types of subexpressions.

Doug

8

This is coming along nicely. I have some tiny quibbles with the source location APIs:

Could we choose reasonable defaults (presumably through an extension method) for some of these parameters? Maybe .afterLeadingTrivia for position and .fileID for filePathMode?

(By the way, what is the file path or file ID for code expanded from a macro? If you wanted to get fancy, you could probably encode locations of macro expansions in the unused disambiguator field of a #fileID.)

In the "Source-location macros" section, you say that #function can't be implemented as a macro because MacroExpansionContext doesn't have enough information to do it, but if these properties are returning some sort of placeholder expression node, I don't understand why function can't be added to AbstractSourceLocation. What gives?

(The same question goes for #file and even #dsohandle, come to think of it. Why can't they have opaque ExprSyntax nodes too?)

Edit: Wish I’d noticed this in the first review…

Should macro-expansion-expression allow a module name so macro names can be disambiguated?

print(#Swift.line)   // as opposed to some other macro called “line”
5 Likes
9

Yeah that probably makes more sense.

I really hope this sort of look up is something that can also be exposed to swift package plugins.

10

Yes, there is a such an extension method in the implementation, and those are the defaults. This is in the grey area where the swift-syntax APIs might change and provide more conveniences than what's specified here in the proposal.

It's a mangled name that demangles to something like "accessor macro expansion #1 of myPropertyWrapper in accessor_macros.MyStruct.name : Swift.String".

I suppose we could add it here.

They could. I suppose we could do them all, since we saw fit to make them part of the language.

@jrose brought this up before, and noted that this would create an ambiguity with existing code, because you can currently write:

``swift
#file.uppercased()


I think we're probably stuck waiting for some kind of module disambiguation (https://forums.swift.org/t/pitch-fully-qualified-name-syntax/28482) for these issues.

  Doug
1 Like
11

True. But I think that just means you can't know the full extent of a macro expansion at parse time, right? You'd have to wait until you looked up the macro name and discovered it was a module, and then rewrite the surrounding MemberExpr to interpret the member name as a macro. (Probably look for a surrounding CallExpr too so you can grab arguments out of it.) That sounds like something you could do in PreCheckExpr so the constraint solver wouldn't have to think about it.

But maybe that's too complex to be worth it. #Swift::file would be easy to support as long as we're parsing the macro name as a declaration base name instead of just a bare identifier.

12

Syntactic macros are not hygienic, meaning that the way in which a macro expansion is processed depends on the environment in which it is expanded, and can affect that environment.

Given Swift’s emphasis on safety, this strikes me as a strange choice. The community will grow a collection of best practices to avoid non-hygienic foot guns, but convention over compiler-guaranteed safety doesn’t seem very Swifty.

We propose to provide a builtin macro that names the module and the ExpressionMacro type name within the macro declaration following an =, e.g.,

@freestanding(expression)
macro stringify<T>(_: T) -> (T, String) =
  #externalMacro(module: "ExampleMacros", type: "StringifyMacro")

Given that we need to introduce macro to the language to support this, it seems strange to me that the right-hand side of this declaration is a stringly-typed macro.

Presumably this macro declaration appears in the module vending the macro, alongside the StringifyMacro type. If so, I’d expect the right-hand side to just be StringifyMacro.self. (In fact, the #colorLiteral example at the end of the proposal uses nearly this syntax.)

I suspect the choice of #externalMacro is motivated by the chained expansion of macros discussed in the detailed design. I’m afraid the choice strikes me as more clever than user friendly.

Also apropos to where the macro declaration appears, is the assumption that all macro declarations are public by default? It seems they must be in order to be useful, since macro declarations can’t be co-located with the code that uses them. Or is it just that StringifyMacro must be declared in a different module?

(I recognize that the module/plug-in structure is to appear in another proposal, but #externalMacro takes a module name and so doesn’t seem reviewable without some sense of what code is expected to live where.)

macro-declaration -> macro-head identifier generic-parameter-clause[opt] macro-signature macro-definition[opt] generic-where-clause[opt]

What does a macro-declaration that omits the macro-definition even mean? How could such a macro be expanded? Is such a macro-declaration a built-in macro? If so, why is macro-definition optional in the surface syntax? Users of the compiler couldn’t define such a thing, could they?

The section “Macros in the Standard Library” proposes lifting many compiler built-ins into the standard library as (almost) macros. Is the idea that the compiler will expose underscored methods that the standard library uses to implement these macros?

What types are proposed to define these macros? Are the macro definitions public but the Macro conforming types internal? The proposed language features don’t seem to support that.

I don’t know how to review the proposed standard library additions when the proposed language features aren’t adequate to define them.

  • What is your evaluation of the proposal?

I love the power offered. I’m concerned with how ad-hoc the approach seems. I think too many details regarding standard library macros are left unstated for a fair evaluation.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Definitely.

  • Does this proposal fit well with the feel and direction of Swift?

The fact that macro inputs and outputs are type-checked is appropriate to Swift. The fact that macros are non-hygienic seems like a purely pragmatic decision that we’ll regret.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I’ve used macros in Racket/Scheme and in C. The proposal compares favorably to C macros.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I studied the revised proposal and read the initial pitch and discussion thread, but missed the intervening bits.

(Apologies for coming late to this process. Health and family issues have had me sidelined.)

1 Like
13

In order to speak of the type StringifyMacro.self, that would require the macro library to depend on the macro implementation. But that's not desired, because the macro implementation will live in a completely different binary. The way to think of the two modules is as follows:

  1. Users import the module that declares the macro; this is how the compiler determines which macros in the user's source code can be substituted. This module provides no implementation (i.e., syntactic transformation) for the macro, but it can provide additional types that are used by the macro.
  2. The module referenced in #externalMacro is separately compiled into a different binary and passed to the compiler as a plug-in. It is not an import dependency of the code that uses the macro; it provides the syntactic transformations that should be applied when the macro is used.

You don't want #1 to import #2 because then the implementation of the syntactic transformations themselves would be compiled and linked into the user's application code. Of course, when defining the syntax for these things they could have chosen to express it as StringifyMacro.self, but the module name would still need to be specified, and nothing would actually be able to enforce that StringifyMacro is a valid macro implementation type. This would only give the illusion of safety, which I think would be worse. Using strings here enforces the reality that there's no dependency between the two modules.

2 Likes
14

But that's an implementation decision on the part of the compiler, is it not?

We could require that module 1 have visibility into module 2 for compiling module 1 without requiring module 1 to re-export symbols from module 2.

15

It's not about whether the symbols are re-exported or not. In order for #1 to import #2, #2 needs to be compiled in the same configuration as #1. But that will very often not be the case—module #1 is compiled in the target configuration while #2 is compiled in the host configuration (where swiftc will run). So a concrete example would be a macro used in an iOS app: module #1 will be compiled to target iOS but module #2 will be compiled to target macOS.

To support what you're suggesting, module #2 would need to be compiled for iOS, even though it will never be used or executed there.

2 Likes
16

Thanks. I think I understand now.

It’s unfortunate that neither this proposal, nor the Vision document, outline the expected structure for where macro definitions and declarations will appear.

I fear we’ve been too aggressive in subsetting out components of the macro system design. It makes sense to subset out the specific design of the relevant SPM changes. However, I don’t think we can effectively review the design of macro syntax/semantics without at least a sketch of how the code will be structured, like you’ve provided here.

Again, thanks for the enlightenment!

3 Likes
17

We can do all of these things, but a purely syntactic rule works better for more tools, and we know we need something like #Swift::file in the long run anyway.

No, the StringifyMacro type is in a different program entirely than the macro declaration. That's why it has to be stringly-typed, and the use of string literals reinforces that.

They are not public by default; they are internal like other declarations. One might define and use macros solely to eliminate code repetition within the implementation of a module, and never expose them.

It is a semantic restriction that any non-built-in macro must have a definition. The grammar permits it to be omitted, so that one can declare built-in macros. It's similar to the body of a function being optional in the grammar---sometimes it must be there (most functions), sometimes it can't be there (functions in protocols).

That's one option; or we might use the special Builtin module that we use for other compiler builtins within implementations; or we can leave them undefined and the compiler can recognize them by name.

I think this question comes from the original misunderstanding; the macro declarations are in the standard library. The actual implementations could be backed by Macro-conforming type or not, but they'd be in a separate program entirely and never visible to the user.

I had thought the discussion of macros being an entirely separate program would have made this clear, but I'll see about revising the vision to make it clear which declarations go where.

Doug

2 Likes
18

I think the first one should be solved with a query function on MacroExpansionContext that provides access to the types of the various subexpressions. I think it's important that the way in which it represents types be coordinated with any work on reflection, so I don't want to lump that in with this proposal.

The second one is something we need to take a lot of care in designing. This kind of query can kick off a lot of compile-time work, type-checking things that might not otherwise be necessary, and introducing cyclic dependencies in the compilation of the program. For example, one macro wants to introduce a stored property, but asks for type information in a manner that triggers another macro's expansion, which then asks for all of the stored properties... the compiler will catch these cyclic dependencies, but breaking them isn't always trivial and might require more fine-grained APIs than we would have thought to expose in MacroExpansionContext.

Doug

19

I wonder if a first version of this function could just return a token that can act as the type’s name in the generated code. I believe this could solve most of the problems I faced.

I think it is reasonable to leave this to a future proposal.

20

Should createUniqueName(_:) be named makeUniqueName(_:) instead to follow the "factory method" naming guideline ?

6 Likes