If that is true that macros should be able to alter members it then begs a couple of associated questions:
Does it make sense for declaration macros to be applicable to all declarations? e.g. You have an example for an enum, but what about structures? (this one 100% makes sense since enums are just structures in a trench coat) What about classes? If so then what about also applying to actors?
If all of those in the previous question are a yes (which I would expect): Should declaration macros be applicable to protocols?
Applying to a protocol would then be able to be done as an extension to then provide a default implementation
Are the emissions from the macro controllable to the access control of the emission? e.g. can a macro emit things that would be private to the declaration it is applied to? For example can we use a macro to inject additional storage into a type?
If all of these cases are true, then this is definitely a superset of the type wrapper feature in my view and would likely solve some edge cases in type wrappers and allow some pretty powerful advancements beyond the current design.
Having these two features collapse into one would not only make it more approachable for developers to only need to learn 1 thing, it would also mean only one point of maintenance and behavior. In my view this would be distinctly desirable since it would ensure cohesiveness in the expectations developers would have in how things work.
I'm still fretting about this, because making arbitrary alterations to members seems like a significant non-local effect. Perhaps there's another way to spin this: some attributes that are placed on a type or extension are also implicitly applied to the members within that type or extension definition. We could say that some attached macros work this way---perhaps as an opt-in---so (say) an attached declaration attribute could apply to type and would also be run on members of that type, so something like:
@accessorizeMyProperties
struct MyType {
var x: Int
var y: Int
func f() { }
}
would apply the macro accessorizeMyProperties to x, y, and f as well as MyType. The macro implementation itself would have to decide what declarations it cares about---so perhaps it does nothing to f, but does something else to x and y (apply a property wrapper attribute, or some other attribute, perhaps). Or maybe there's some other way to get this effect.
Pretty much everything. Some particular combinations won't make sense---you can't add a function body to a struct, or add members to a typealias---but especially for things like "peer declarations", most declarations can be alongside other declarations, so they're quite general.
Yes, protocols are declarations.
I think it's likely to be important for macros to be able to introduce extensions. I suspect a peer-declaration macro will be able to do it, but I want to dig into the implementation further to be sure.
I think there's an unstated "if you're used to working on a compiler" after "straightforward", but maybe that's totally fine. If the goal here was to turn Swift into a Lisp and have writing macros be a normal thing that every developer does on a regular basis then I don't think this approach would work at all, but with macros intended to be a rarer thing then my concern is more around how easy it is to understand what the macro is doing without prior experience and this seems fine on that front.
Removing the need for the trivia manipulation would probably help quite a bit with that too; if you're used to looking at code using SwiftSyntax I assume your eyes just automatically skip all of the withoutTrivia() etc. calls, but for someone new to it that's a lot of extra incidental code to read on the way to understanding what's actually happening.
I appreciate all of the design discussion here! I've gone ahead and revised the pitch. The changes are, roughly:
Split peer/member/accessor macro implementations into separate protocols and attribute spellings, so the compiler can query them in a more fine-grained manner.
Removed function-body macros... for the moment. We'll come back to them.
Add example showing composition of different macro roles for the same macro to effect property-wrappers behavior.
That last example is really fun. Alongside this, @hborla and I have prototyped some of these syntactic transformations in swift-syntax, and @rxwei has made progress on an implementation of freestanding macros in the compiler:
EDIT: Lots more interesting cases to consider, so I've updated the document again with:
"Body" macros to supply or alter the body of a function/initializer/accessor/closure.
Default-witness macros to help with synthesis of protocols.
Member-attribute macros to allow one to add attributes to the members of a type/extension.
This is too big for one proposal, but I'd rather not trickle out ideas one-by-one. Rather, there's a lot in here covering a large space of what is possible, and we can tease it apart later into more-easily-reviewable chunks.
The changes look really great! Thank you! Some remarks:
The peersOf:, membersOf:, accessorsOf:, bodyOf: and memberAttributesOf: labels sound a bit clunky to me. They prevent the function to be read like a sentence. Maybe providingPeersOf: or generatingPeersOf: would be a better fit?
I would appreciate seeing an example of how a body macro applied to a closure would look like.
I think it would be valuable for default witness macros to get more information about the conforming type. Currently, the macro wouldn't even know if the type is a value or a reference type if I am not mistaken. Having at least the spine of the parents of the node where the generated witness would be expanded (as suggested by you for expression macros) would be a good start. Or getting the type definition or extension syntax including conformance list and the definitions inside just as a member attribute macro would get would be even better.
If multiple macros are attached to the same definition, how would they be expanded?
I tried to implement a custom type wrapper syntactic transform as a combination of attached macros in the current swift-syntax prototype, and I got pretty far!
The type wrapper macro struct conforms to MemberDeclarationMacro, MemberAttributeMacro, and AccessorDeclarationMacro. Each macro capability provides a different part of the type wrapper transform:
MemberDeclarationMacro adds the backing var _storage variable.
MemberAttributeMacro applies macro attributes to each stored property inside the wrapped type, which are recursively expanded.
AccessorDeclarationMacro uses the macros applied to stored properties to add get and set accessors, turning those stored properties into computed properties that indirect access through _storage.
Using this macro transform, I can transform this type:
@customTypeWrapper
struct Point {
var x: Int
var y: Int
}
into
struct Point {
var x: Int {
get {
_storage[wrappedKeyPath: \.x]
}
set {
_storage[wrappedKeyPath: \.x] = newValue
}
}
var y: Int {
get {
_storage[wrappedKeyPath: \.y]
}
set {
_storage[wrappedKeyPath: \.y] = newValue
}
}
var _storage: Wrapper<Self>
}
@customCompressedTypeWrapper
struct Point {
var x: Int
var y: Int
}
Into this?
struct Point {
private struct Storage {
var x: Int
var y: Int
}
private var _storage: Storage
init(x: Int, y: Int) {
_storage = Storage(x: x, y: y)
}
var x: Int {
get { _storage.x }
set { _storage.x = newValue }
}
var y: Int {
get { _storage.y }
set { _storage.y = newValue }
}
}
That way we can avoid building key paths and also avoid internal storage types from leaking out accidentally. Overall this approach to me feels distinctly more flexible, more performant, and safer.
Yes, I believe so. When adding nested members during the expansion of a MemberDeclarationMacro, the macro is able to iterate over all existing members in order to create new members. Once expansion is hooked up to the compiler, I think the compiler can provide limited type information for existing declarations so if you write e.g. var x = 0, you will still be able to generate var x: Int in that nested Storage type.
I like this, because it makes it clear that's the purpose of the expansion.
Sure, it be something like { @MyResultBuilderMacro in ... }.
Right, we should have an extensible structure here so that we can provide whatever information is reasonable, and expand it over time as needed.
I think we'll end up going left-to-right, since that's how evaluation order works in Swift everywhere else. You're right that I need to specify that, and there are other interesting ordering constraints that need to be specified in this document.
Macros with arguments rightly read somewhat like functions (parentheses and argument labels and all that). However, the : Void syntax makes it seem like the addCompletionHandler macro itself (or whichever other zero-argument macro) has no implementation because it takes on the type of Void, so to speak. Would it be so bad to omit the : Void part? Is there an alternative that doesnāt make it look like addCompletionHandler is an identifier for a āinstanceā (to the extent that Void has āinstancesā) of Void?
I'm really excited about the direction for declaration macros ā it will undoubtedly have a huge effect on both the design for libraries and the process of writing and maintaining them. I'm still digesting what's in the pitch, so I don't have much feedback other than supporting any features that would let declaration macros subsume the type wrappers proposed elsewhere.
That said, could the proposal engage with how the result of macro transformations will be communicated, to both library authors and users? I come at this from the perspective of a library author, so these are the specific questions I'm wondering about:
Should there be some kind of communication that new symbols are created when I use a macro?
Should there be a way of seeing the result of a declaration macro's transformation (i.e. all the new code)?
Should there be a way of seeing the new public API generated by a macro?
Do we have sufficient tools to know if an update to a macro changes or adds to a library's public API?
From my experience in talking with people about property wrappers, there's a large group of Swift users who think of them as magic attributes that give a property special powers, rather than as syntactic sugar for a generated property and computed accessors. While this isn't a problem per se (it's good to have abstractions that people can use without full knowledge of what's happening under the hood), I'm a little worried that we're expanding the scope of what these magic attributes can do without thinking through how we're going to help people understand what is happening in their programs. Thank you!
I'm very excited about this, and the examples are compelling. In particular, I believe implementing macros in Swift, using swift-syntax, as a little program, is a flexible, elegant, and powerful approach. I prefer this approach over evaluating constant expressions, or some other restricted language mode.
This will be useful for gamedev/XR-dev tasks like serialization and synthesizing property UIs, and other boilerplate. Within the implementation of the macro, are there any restrictions on which modules we can import? Could we import SwiftUI?
This is well beyond the scope of the current macro proposals, but I'm intrigued about the possibility of transforming C++/MSL/other language types with Swift macros. For example, we could generate type-safe Swift bindings/invocations to Metal compute functions (programs like mtlswift can do something like this as a separate build phase). Perhaps there is a cpp-syntax analogue for swift-syntax, and some way to transform a cpp type within Swift code.
It's become very clear to me that this "declaration macros" pitch is really two separate proposals rolled into one, so I've gone ahead and split this document into two separate proposals that we can iterate on:
Freestanding macros covers the "freestanding" case, generalizing the # syntax to also introduce declarations and code items.
Attached macros covers the "attached" case, using the custom attribute syntax to introduce peers/members/etc.
Both proposals have been filled out with more details based on the discussion here, although we have more feedback to address.
Can an @attached(peer) macro, even without this future direction, add an extension declaration with a protocol conformance? (I understand this could only work at top-level, where extensions are legal.)
I've been fretting about extensions. If we could declare extensions, it would also subsume most[*] uses of the "member" role, since you can add members through extensions. However, that would side-step the fine-grained nature of member macros having to explicitly say what member names they introduce. You would end up having to expand a macro that can produce an extension quite eagerly, because extensions do so much.
Part of the reason I'm fretting about this is because the compiler's implementation of extension binding is trickier than most of the rest of type checking: it's an iterative process that resolves what extensions it can, then goes back and revisits other extensions that it couldn't resolve previously, because one can extend nested types:
struct A { }
extension A.B { // can't resolve this until we've resolved the extension A below
}
extension A {
struct B { }
}
This "extension binding" process happens very early in compilation. I'm concerned that we won't be able to implement support for macros that generate extensions, or that it will be inherently brittle. It certainly needs more thought.
[*] Most, not all, because there are things you can't do in an extension, such as add stored properties, overridable methods, designated initializers of non-final classes, cases, or protocol requirements.
If you're talking about communication from the compiler, that would be a warning or error, presumably. If we generate such a warning, we'd need a way to suppress that in the source code. That feels like it would be very noisy to me, if in addition to explicitly stating that I'm using a macro (@equatable or whatever) I also need to write something else in the source to say "yes, I know this generates names."
We've talked about this a bit in the other macros threads: yes, there should be a way to see the expansion of any macro.
In the worse case, generated Swift interfaces will show the result of a module after macro expansion, with all uses of macros removed. That's user-inspectable and diff'able output even if we did no additional tooling support for macros.
This is a subset of your first question, but it's an interesting subset because Swift does tend to draw a line around module boundaries, and require more explicitness for public API commitments. Perhaps we want some spelling that makes it clear that a particular macro use is permitted to introduce public API? I don't know what it would be... @public:macroname(macro-args) or something?
swift package diagnose-api-breaking-changes would handle this, right?
The fact that one can see the effect of a macro expansion as actual source code (and replace the use of the macro with that source code) is a huge leap forward over features like property wrappers, result builders, and Codable synthesis, where there is almost a source-to-source translation, but you can't see it.
The macro implementations are just packages, so no---there's no restrictions per se beyond the sandboxing requirement discussed in SE-0382. There is a semantic restriction that your macro should produce the same output every time it's called with the same inputs, or else compilation will be non-deterministic and correct incremental builds will be impossible. That's something where we have to trust macro implementers, because there's no restriction we can place on macro implementations to ensure that's true.
I think there's a lot one could do here. The main missing feature, as far as I can tell, is that there's no way right now to give a macro implementation access to the contents of another file. If we had that ability, one could (say) have a macro that takes a Metal file and produces declarations for the Swift bindings. We'd want to sort out how to deal with errors in the input file, and there are likely other usability issues to make it a decent experience,
I'm not sure macros are the right tool for this job. @timdecode: If I understand what you want to do correctly, Swift package plugins using code generation (e.g. with SwiftSyntax) can be used to do that today.
As I see it, package plugins are the right tool to generate new Swift code and macros are the right tool to transform existing Swift code. The only usability downside of the former is that you have to explicitly declare that you want to use the plugin in Package.swift. I would like to hear your opinions on this.
I haven't yet finished reading the new proposal for attached macros, but have some comments regarding freestanding macros:
Overall, the document looks good to me. But this got me thinking:
Code item macros can only introduce new declarations that have unique names, created with createUniqueName. They cannot introduce named declarations
I think I am okay with this rule. However, I can see that a macro would want to expand to something like:
let file = File("example.txt")
defer {
file.close()
}
With the current proposal, I believe the only way to make this work would be wo write:
let file = File("example.txt")
#deferClose(file)
which is not really better than the expanded version. I wonder if we could make this work with the following syntax:
let file = #autoClosedFile("example.txt")
The rule regarding the visibility of names worries me a bit. It looks to be very complicated and hard to teach. I am not even sure I understand it myself after reading the section multiple times.
To drill into this a little further, since imports are "declarations" as far as the compiler is concerned, can a freestanding macro generate an import declaration that influences type resolution for other code outside the macro invocation?
For example, would this work?
// Let's say this generates "import SwiftUI"
#totallyCoolAndLegitImportMacro(arguments)
// Is this fine even if we don't import SwiftUI explicitly, because the
// macro did it?
struct MyView: View {
var body: some View { ... }
}
. It's something we can improve on via the swift-syntax APIs over time.