[Pitch] Generalize `conformance` macros as `extension` macros

Ah good question, the compiler already handles this with conformance macros. Notice that the macro does not provide the name of the type it extends (both with conformance macros and with extension macros in this pitch) because the compiler binds the extension directly without doing name lookup of the extended type. The compiler also inserts the extension into the correct scope, not the potentially-nested scope where the macro is applied.

3 Likes

OK, so those conformances live at another level. Good to hear! :+1:

I asked because those reports about macros that don't work with nested types were suspicious:

2 Likes

I think this will need another pitch, but I find this very interesting as well. Some GRDB users are happy (1, 2) to retroactively conform their models to database record protocols, and thus I see a use case for such a pitch. Such "retroactive macros" will probably not be able to access the inner guts of the attached type, so they will be limited. Are we sure the freestanding macros aren't the correct tool for such a job?

2 Likes

According to this comment, this may require significant compiler work.

1 Like

Yeah, not being able to see the guts of the type is an expected limitation for extension-writing, even if it's done manually, so I don't think anyone's going to be upset about that. Also, a making an extension macro a declaration macro instead of an attached macro makes perfect sense, especially in light of the potential to have structural extensions in the future, where there is no one obvious type to extend.

The significant compiler rework may be a limiting factor, though. But given how powerful and useful retroactive conformances are, the compiler rework may end up being justified.

This looks like a good improvement! Would it also pave the way for lifting some of the restrictions on extensions produced by macros?

2 Likes

I think generalizing conformance macros to extension macros is a good idea, but it should also be possible to attach attributes and modifiers to the extension. Attributes in particular are crucial for declaring conformances with partial availability, a feature ConformanceMacro cannot currently support.

Better yet, the tuple should be replaced by a nominal type that would allow these kinds of changes in the future as new features are added to the language. The other expansion methods all return a complete syntax node per item, which means they effortlessly support future language features. But conformance/extension macros instead return a tuple of constituents of a syntax node; expanding that tuple will cause a source compatibility break. There are good reasons not to just return an ExtensionDeclSyntax, but that means we need to think about other ways to provide the flexibility it needs.

(Alternatively, just pass a TypeSyntax node to the expansion method and require it to return only ExtensionDeclSyntax nodes using that exact type syntax.)

5 Likes

Yeah, I agree with your points and I think this is the best approach. ExtensionDeclSyntax is already the type we have to describe extensions, including attached attributes, and we can verify that the resulting extension uses the right TypeSyntax for the extended type (and still bind the extension directly on the compiler side). I'll update the pitch, thanks!

Possibly. I could imagine a future addition to @attached(extension) that allows you to name the extended type in the macro attribute, e.g. @attached(extension, of: MyType), which defaults to the type the macro is attached to.

5 Likes

I'm not sure I understand the use case here. If you're adding functionality to a type outside of the primary type declaration, you're already writing an extension of that type. Why would you want to attach an extension macro that adds in another extension, instead of a macro that fills in the contents of an extension you already wrote?

No, that's not what I meant, sorry for the confusion. I merely meant that, as proposed, an extension macro can be used by attaching it to the primary type declaration:

@MyMacro
struct MyModel {
    // ...
}

This means that only the author of MyModel can use extension-generating macros.
What I was suggesting was the ability to use an extension-generating macro without having access to the primary declaration of the type (syntax bikesheddable):

#MyMacro(MyModel.self)

I'm asking why you wouldn't write this

@MyMacro
extension MyModel { /* filled in by MyMacro */ }

It's crucial that the compiler understands what type is being extended prior to macro expansion. That means it either needs to be specified in the macro attribute with the extension role, in which case it'd be fixed, or it must be the type that the macro is attached to. It cannot be specified by arbitrary macro arguments, because the compiler cannot know what the macro does with those arguments without expanding the macro. For your suggested syntax to work, we would need to invent a new way for a macro role attribute to identify type parameters that are used in the macro parameter list. But personally, I think this use case is better off using an attached macro on an explicit extension. In any case, I think this is out of scope for this pitch.

1 Like

Oh, that's what you meant! Sorry for the confusion! Even though it feels awkward to declare an extension with empty braces just so the macro can fill them in, it's still a reasonable approach, given the limitations you mentioned about the compiler having to know what is being declared.

I'm assuming, with this proposal, the macro will be able to fill in the extension and add a protocol conformance clause with a single role, right?

EDIT:

Would it be possible/practical to be able to omit the braces on an extension if it has an attached macro on it?

As @hborla has said, this is not part of this pitch:

1 Like

I think this is a great direction.

The only request I'd like to make for improvement is to take @beccadax's suggestion to have the macro implementation return the ExtensionDeclSyntax directly. Conformance macros were the only one that tried to return a tuple containing several different syntactic pieces, rather than a whole declaration/expression/statement/code-item or list thereof, and it became very limiting.

We might need to ban extension macros from being applied to local types, because we don't have a notion of extensions on local types and wouldn't be able to "expand" the macros to produce valid Swift code.

Doug

1 Like

Is it accurate? It looks like this contradicts this above comment. But maybe I misinterpreted what a "local type" is.

I was unclear. We could implement this in the compiler.

However, because it cannot currently be expressed in the base language (there is no syntax to extend a local type), I do not think we should allow macros to do it because it would mean that macros can do something, yet you couldn't expand the macro-generated code and still have it work.

Doug

1 Like

I wonder if it is a good idea. Let's consider a user of a library that exposes some macros. The user writes:

import ColorKit

func f() {
  // Define and use a local type
  @Blue struct Sky { ... }
}

@Blue does not add any extension, just some members.

So this code compiles. And this is nice, because the macro works identically for top-level types as well as local types. The user is not even aware that we're discussing their fate in this forum thread.

Now, if this code would stop compiling in a future release of ColorKit, just because @Blue now adds some extension to the attached type, this would be quite not nice at all.

The implementation details of a macro are what the user doesn't want to know until it's time to debug, and expand the macro. Macros are convenient because their inner work is only at the edge of the user's awareness. The macro author has worked very hard in order to make such a gift to the macro user. Any author of a nice and non-trivial macro knows that. I'm leaving non-nice macros out of the discussion, because well they're not interesting.

According to @hborla's message, the compiler is able to add extension to local types. I agree that this is not possible for "the rest of us", using the normal UI for interacting with the compiler (.swift text files). In my opinion, this missing feature is not a good reason for making macros less ergonomic.

4 Likes

That wouldn't be great, no. However, (a) this would indicate either that the author of ColorKit didn't consider the local type use case or implemented the macro poorly, and there are many, many ways in which a poorly implemented macro or a macro not meant to be used on a certain declaration can fall apart in the end user's hands; (b) I would consider the drawbacks of allowing this to be more severe than disallowing it.

One of the fundamental principles we've settled about Swift macros is that their output is expressible Swift code. Yes, it's true that macros allow users to not think about the implementation details when they don't need to; it's quite another thing to have inexpressible macro results that users can never reason about in terms of the Swift language. Supporting a good debugging experience is a key aspect, but only one part, of hewing to this principle which fundamentally is about allowing users to understand what's going on when they do need to do so.

Another issue specifically here is that there are invariants which users writing local types may currently rely upon regarding the accessibility of their members, which are broken when extensions are expressible—see my reply in the most recent pitch thread about allowing nested extensions. If we permit macros to do this, then we will have to deal with the fallout of allowing those invariants to be broken whether or not we allow such extensions in the surface language, which would expand the design questions in this proposal significantly.

2 Likes

@xwu, you're a member of the Language Steering Group, so your voice has a particular weight. Yet it looks like you are missing important information, and drawing incorrect conclusions.

Consider the recent SwiftData library that Apple has just shipped. If you hadn't checked it yet, please go to the WWDC23 videos, and have a special look at the one about data migrations (videos). In SwiftData, lightweight migrations need the developer to provide distinct versions of a model type. The only way to do it is to gather them in namespaces, which means to nest model types. See a quick sketch below:

enum SchemaV1: VersionedSchema {
  @Model final class Language {
    var name: String
  }
}

enum SchemaV2: VersionedSchema {
  @Model final class Language {
    var name: String
    var hasMacros: Bool = false
  }
}

Just look at this video slide. Quite an evidence.

Now, maybe you know it: the @Model macro adds protocol conformances:

// Expansion of `@Model` adds at least PersistentModel
// and Observable conformances:
enum SchemaV1: VersionedSchema {
  final class Language {
    var name: String
  }
  extension Language: PersistentModel { }
  extension Language: Observable { }
}

Hey, but this code is not supposed to compile: extensions are only valid at file scope, right? So it's not possible to nest models in enums? What about this SwiftData slide, then?

At this point, we have made a loop in this thread, which started there, with an answer you may have missed there. The information contained in this answer will allow you to adjust your conclusions.

Let's break this loop now, if everybody's ok.

IIUC we’re talking about banning extension macros applied to local types, i.e. types declared inside function bodies. Types that are nested inside other type declarations are fine.

EDIT: if the expand macro refactoring puts the extension inside the nested scope, that’s a bug in the refactoring action because that’s not where the expansion actually exists.

1 Like