The review of SE-0382: Expression Macros ran from December 16, 2022 until January 15, 2023.
Overall, the feedback we received was positive. Modifications were incorporated to the proposal during review based on early feedback when the review period was extended. Having reviewed all of the points raised during the review period, the language workgroup has decided that the proposal as revised should be returned for revision and subsequent focused re-review centered on the following changes:
- To reflect reviewer feedback and further development work regarding the design of
MacroExpansionContext - To revise how macros are declared—that is, the spelling
@expression macro— to align with evolving discussions regarding declaration macros - To update the design so as to support opaque result types, and to define the rules surrounding such use of opaque types given that Swift presently does not support explicit opaque type coercion (
as some P) in the surface language
The language workgroup recognizes the inherent difficulty of reviewing a proposal that covers only a portion of a larger feature. As with other interlinked proposals in the past and present (such as concurrency or ownership), we leave open the possibility of revisiting aspects of accepted proposals in light of new information from follow-on proposals. Package manager-related questions have been subset into a future proposal, and pitches for freestanding and attached macros (originally, declaration macros) have already been posted.
Despite returning the proposal for further revision, the workgroup notes that a number of complex and novel issues covered in the proposal have been settled in this review process, and we consider them to be accepted in principle. We think a number of these issues deserve a more expansive exposition in these decision notes:
The scope of review and the role of Swift Syntax
Reviewers have correctly brought up that this proposal is unusual in naming a library that's not itself subject to the Swift Evolution process as a requisite, user-facing "ingredient." There hasn't been appetite on the part of the community to review the entire API surface of Swift Syntax, and the language workgroup agrees that it is not the intention of this proposal to bring that API into the scope of review.
Rather, the language workgroup is satisfied that the feature as proposed has been designed to decouple the specific version of Swift Syntax from the language in a sensible way:
- Swift Syntax is now fully standalone and written entirely in Swift.
- Each macro, being a separate program, can depend on its own version of Swift Syntax—indeed, a macro could theoretically depend on its own forked version of Swift Syntax; more on the packaging aspects of macros will (as mentioned above) be explored in later proposals specifically on that subtopic.
- An invariant of Swift Syntax is that it is able to faithfully preserve all the code it encounters even if it cannot parse it; therefore, future directions for the design of macros could expose a way for macros which depend on an old version of Swift Syntax that cannot parse some new code to opt into either handling it as a maybe-syntactically-broken "blob" or rejecting it with an error that the macro couldn't understand the syntax.
On the other hand, future improvements to MacroExpansionContext will be subject to Swift Evolution if they require compiler support, as they’d expose new functionality and raise language design questions. This is to be distinguished from changes exclusively scoped to Swift Syntax that are about how it represents the language; those are outside the realm of the review process here.
Macro authors will need to contend with the reality that the Swift Syntax dependency exposes an API without the stability guarantees of, say, the standard library. This is in some ways a positive, as rapid development to improve the user experience would answer reviewer feedback about ergonomic shortcomings of that library.
Applying an observation about the natural lifecycle of libraries in general, it is almost certainly the case that Swift Syntax will face stabilizing pressure as its clients grow over time. Running counter to that stabilizing trend, the language workgroup observes that we have a natural destabilizing force involved that is the language evolution process itself, as Swift Syntax would have to accommodate future approved additions to the language. As a corollary to the above separation between the Swift Evolution proposal and the Swift Syntax API, the language workgroup does not intend for considerations regarding how the Swift Syntax API may be destabilized (and its downstream effects on new macros) to weigh as a factor that could block future language evolution.
User experience
Several reviewers provided feedback about the user experience of creating a macro that were related to the state of Swift Syntax, including its documentation, the approachability of APIs that go beyond string interpolation to construct nodes, and boilerplate required to check macro arguments. The language workgroup acknowledges that these are valid sorts of observations to make: a section on macro development is now available to host discussions about developing and using macros, which can inform future improvements to Swift Syntax independently of the Swift Evolution process.
Other feedback concerned the debuggability of macros. Design-level considerations were addressed by authors in the review thread, and the language workgroup was satisfied that these discussions did not reveal fundamental changes which need to be made to the proposal that would hold up acceptance.
Macro expansion context
Reviewer feedback pointed out the lack of expansion context in the current proposal, limiting the kinds of macros which can be implemented. For example, a printArguments macro would be infeasible to implement without access to parent nodes.
The rationale for this limitation is to minimize dependencies on other source code. Alternative approaches to find some sort of reasonable "cut point" were explored, such as the possibility of presenting an incomplete "spine" of a hierarchy. However, it was pointed out that such an approach could be an attractive nuisance, giving the appearance of but not actually permitting accessing all the children of a parent node or determining relative source locations in the parent chain.
Although no resolution to this question was reached during the review period, it's notable that numerous reviewers were impressed with the variety of macros they are able to build with the existing proposal features. Returning the proposal for revision does offer an opportunity to incorporate further revisions of MacroExpansionContext, but not all questions need to be resolved before we have a sufficiently useful, and thus approvable, feature.
Trivia, trailing closures
Reviewers pointed out during the review period that macro implementers will be able to conditionalize behavior based on whitespace differences or the presence or absence of comments (in Swift Syntax parlance, 'trivia'), or whether an argument is passed using trailing closure syntax or not.
The language workgroup agrees with the consensus during review discussion: it's for the best that the design of the language doesn't attempt to prohibit macros from taking action based on whether there's a comment or trailing closure, though it is certainly the case that macros shouldn't do that, and of course Swift Syntax can offer affordances in the future that help macro authors to do the right thing. However, deciding that macros can only access a "normalized" form of an expression would involve defining such a normalized form for the whole language. Further, stripping out trivia across-the-board is not an acceptable normalized form because, in certain scenarios, it's actually necessary for Swift Syntax to preserve and/or for macro authors to insert trivia—for example, balanced whitespace when manipulating expressions containing operators—in order to ensure that the syntax tree is well-formed and roundtrips correctly when serialized and deserialized.
This brings up a foundational concept: macros are provided a correctly-parsed bit of program text and are expected to produce a syntax tree that will correctly print and parse again, because serialization/deserialization is inherent to decoupling of compiler versions from Swift Syntax versions. Since Swift Syntax provides for manipulation of the syntax tree, it's possible to create trees that cannot roundtrip correctly after printing and re-parsing. We felt that a more substantial exploration of this concept is more appropriately one of the subjects of a future proposal regarding macro infrastructure, build configuration, and related matters.
As always, thank you for your participation in Swift Evolution.
Xiaodi Wu
Review Manager