[Pitch] Generalize `conformance` macros as `extension` macros

I wonder if it is a good idea. Let's consider a user of a library that exposes some macros. The user writes:

import ColorKit

func f() {
  // Define and use a local type
  @Blue struct Sky { ... }
}

@Blue does not add any extension, just some members.

So this code compiles. And this is nice, because the macro works identically for top-level types as well as local types. The user is not even aware that we're discussing their fate in this forum thread.

Now, if this code would stop compiling in a future release of ColorKit, just because @Blue now adds some extension to the attached type, this would be quite not nice at all.

The implementation details of a macro are what the user doesn't want to know until it's time to debug, and expand the macro. Macros are convenient because their inner work is only at the edge of the user's awareness. The macro author has worked very hard in order to make such a gift to the macro user. Any author of a nice and non-trivial macro knows that. I'm leaving non-nice macros out of the discussion, because well they're not interesting.

According to @hborla's message, the compiler is able to add extension to local types. I agree that this is not possible for "the rest of us", using the normal UI for interacting with the compiler (.swift text files). In my opinion, this missing feature is not a good reason for making macros less ergonomic.

4 Likes

That wouldn't be great, no. However, (a) this would indicate either that the author of ColorKit didn't consider the local type use case or implemented the macro poorly, and there are many, many ways in which a poorly implemented macro or a macro not meant to be used on a certain declaration can fall apart in the end user's hands; (b) I would consider the drawbacks of allowing this to be more severe than disallowing it.

One of the fundamental principles we've settled about Swift macros is that their output is expressible Swift code. Yes, it's true that macros allow users to not think about the implementation details when they don't need to; it's quite another thing to have inexpressible macro results that users can never reason about in terms of the Swift language. Supporting a good debugging experience is a key aspect, but only one part, of hewing to this principle which fundamentally is about allowing users to understand what's going on when they do need to do so.

Another issue specifically here is that there are invariants which users writing local types may currently rely upon regarding the accessibility of their members, which are broken when extensions are expressible—see my reply in the most recent pitch thread about allowing nested extensions. If we permit macros to do this, then we will have to deal with the fallout of allowing those invariants to be broken whether or not we allow such extensions in the surface language, which would expand the design questions in this proposal significantly.

2 Likes

@xwu, you're a member of the Language Steering Group, so your voice has a particular weight. Yet it looks like you are missing important information, and drawing incorrect conclusions.

Consider the recent SwiftData library that Apple has just shipped. If you hadn't checked it yet, please go to the WWDC23 videos, and have a special look at the one about data migrations (videos). In SwiftData, lightweight migrations need the developer to provide distinct versions of a model type. The only way to do it is to gather them in namespaces, which means to nest model types. See a quick sketch below:

enum SchemaV1: VersionedSchema {
  @Model final class Language {
    var name: String
  }
}

enum SchemaV2: VersionedSchema {
  @Model final class Language {
    var name: String
    var hasMacros: Bool = false
  }
}

Just look at this video slide. Quite an evidence.

Now, maybe you know it: the @Model macro adds protocol conformances:

// Expansion of `@Model` adds at least PersistentModel
// and Observable conformances:
enum SchemaV1: VersionedSchema {
  final class Language {
    var name: String
  }
  extension Language: PersistentModel { }
  extension Language: Observable { }
}

Hey, but this code is not supposed to compile: extensions are only valid at file scope, right? So it's not possible to nest models in enums? What about this SwiftData slide, then?

At this point, we have made a loop in this thread, which started there, with an answer you may have missed there. The information contained in this answer will allow you to adjust your conclusions.

Let's break this loop now, if everybody's ok.

IIUC we’re talking about banning extension macros applied to local types, i.e. types declared inside function bodies. Types that are nested inside other type declarations are fine.

EDIT: if the expand macro refactoring puts the extension inside the nested scope, that’s a bug in the refactoring action because that’s not where the expansion actually exists.

1 Like

Oh, local types were included in my original question, and I understood that your answer was applying to them as well. I'm sorry to hear they were not.

This won't prevent me from suggesting the language steering group to have an open mind here, and consider trade-offs. Local types are an important part of the language, and they do not fully overlap with file-private types. Banning extension macro from local types sounds like a gratuitous language inconsistency that will only create annoyance. "Poorly implemented macros", as I read above, sounds maybe cruel, but to the point.

As @xwu said above:

One of the fundamental principles we've settled about Swift macros is that their output is expressible Swift code

Granted, but now macros have shipped (in beta), and user feedback is flowing in, discussions are getting much more real. A perfect time for polishing the fundamentals, or adjust language expressibility (at some well-chosen level which may not be the .swift text files). If expressibility is uneasy to adjust, then bending the fundamentals until the expressibility follows is just a way to stage changes one after the other.

1 Like

If we're to highlight a gratuitous language inconsistency, it would be the inability to write an extension on a local type at all. Fix that, and extension macros would work for local types.

The problem with bending the fundamentals is that you end up with sharp corners in unexpected places. And then when you fix the underlying problem, you have a weird behavior change. I'd much rather ship a well-delineated limitation than something inconsistent with our fundamentals.

Doug

5 Likes

Fine, that's a pragmatic point of view on "fundamentals" that I agree with.

May I try to suggest that the steering group tries harder, though. We all deserve the best. Macros do not operate at the level of expressible Swift code. Pretending the opposite is flatly wrong. Pick up the opportunities, and don't let half-baked and leaky abstractions rule your designs. Thanks.

The correct formulation here is

// Expansion of `@Model` adds at least PersistentModel
// and Observable conformances:
enum SchemaV1: VersionedSchema {
  final class Language {
    var name: String
  }
}

extension SchemaV1.Language: PersistentModel { }
extension SchemaV1.Language: Observable { }

i.e., this is expressible in the language, but the way it's presented on a slide and in "Expand Macro" is misleading. Cool, we can fix the presentation. It doesn't change the feature.

I honestly have no notion of how to interpret this statement. I suspect you're extrapolating from the presentation error above regarding extensions on nested types, and also trying to refute my argument that macros shouldn't allow extensions on local types because you want that feature. But "Macros do not operate at the level of expressible Swift code" is a fairly sweeping statement that is at odds with the macro system vision and design, wholly unsubstantiated by the above. If you want to stick by that argument, please find some more substantive evidence and bring it to another thread: it's either a much bigger topic than extensions, or it's just those two things we talked about.

The above is patronizing and rude. Please tone it down so we can have a productive discussion.

Doug

6 Likes

I’m not entirely sure what you mean by this. I’m interpreting it as a criticism that the code a macro author writes operates “one level of abstraction up”, using the SwiftSyntax APIs to emit the AST of the Swift code they want the macro to by replaced by. Is that correct?

If so, it sounds like you’re demanding the Swift team cease any development or improvement to macros until they revisit that decision. That doesn’t seem like a good compromise to me.

1 Like

I don't see any criticism in my sentence. Yes the macro author interacts with the various syntax packages. My intent was to remind this plan fact. Macro authors do not work with Swift text files.

The macro runtime, fed from output from macros, can deal with extensions of nested types. That's pretty cool. I'm just wondering why this can't be applied to local types as well.

I consider this question pretty normal. The question of someone who wants to push the language towards more inner consistency. A question that ought to be expected.

It happens that this question hurts some "fundamentals" that I consider weak. Indeed no clear answer was provided, except that it may be difficult (ok, let's see), and that I'm rude because I express my disappointment with the answers given so far.

Rude or not, the disappointment is the same.

Why is this worth mentioning in a thread that is dedicated to the narrow question of extending conformance macros into extension macros? You didn’t use this point to compare the pitch against the status quo, but instead to complain about the design of macros in general. And you did so in an extremely rude way, accusing the steering team of not trying hard enough, of half-baking the design, and of missing opportunities to plug leaky abstractions.

It’s possible to be strongly critical of a feature without resorting to such vitriol, but this thread isn’t the place for it.

1 Like

This is another misguided view of my contributions here, maybe influenced by other members who have started an antagonist conversation that I never wished, and tried to make me look like a "bad guy".

I have mentioned extensions in a thread dedicated to extensions. What's odd, here?

I have mentioned current problems that some people have with extensions in a thread dedicated to extensions, with links.

And I have asked the steering group to work harder, with arguments, because it is not rude to suggest the steering group to improve the language.

Fine. Let's go back to the topic of this thread, then. Extensions. All extensions.

(Edit: this message is an answer to a deleted message)

I believe I already answered this here, but to reiterate:

  1. Nested types can be extended with or without macros. There appears to be some confusion because the "Expand Macro" output for a macro when a nested type is extended (via a conformance macro) produces invalid Swift code, when there is already a way to express this with valid code. This is a bug that should be fixed. Any inference from this bug to some larger notion that macros can do things that normal Swift code can't is misguided.

  2. Local types cannot be extended without macros, because extensions have to occur at the top level and there is no way to name a local entity from outside its scope. Conformance macros accidentally side-stepped this in a narrow case (where the conformance has constraints). This does not imply that generally adding extensions to local types would work.

There are both philosophical and practical reasons for keeping the "macros can only do what's expressible in normal Swift". The philosophical reason should be clear by now: it means we can expand away the macros to look underneath them, and understand exactly what they're doing. The practical reason is that we expand away macros when emitting Swift interface files, which are a key part of ABI-stable frameworks. To deal with local types that have expansion macros applied to them, we would have to invent an in-language way to extend local types to be able to emit a Swift interface file using this feature.

You're entitled to be disappointed that local types don't work with extension macros, but the answer isn't to dismiss the design goals or to try to carve out an exception to them. The answer is to make extensions themselves work with local types in general, and that unifying feature means they will work with extension macros.

Doug

10 Likes

FWIW, as an outsider, it looks like

  • We're combining the non-locality of extensions with the opacity of macros
  • Some are proposing abandoning the notion of source code, i.e. text as the ground truth
  • There's been little analysis or discussion of programmer's model writ large - how this plays in IDE's, static analysis tools, training, etc. - or the software lifecycle.
  • It's being discussed without a range of samples to evaluate
  • It's all happening in a rush at the last minute

In almost any other context, I'd call it crazy.

But the essential idea -- of simplifying variants of protocol extension macros -- is a good one that people seem to be earnestly and carefully adopting, and I trust the team, particularly as they follow longstanding principles in addressing what's coming up.

So realize: It's all good! And, be careful!

1 Like

I must apologize to all people here for several mistakes I made.

Yes, I wish macros would work as well with local types as with other types. But I said it very badly, and I sounded as if was accusing the steering group of not working hard enough in order to 1. achieve this goal, 2. in the context of this pitch.

My deepest apologies to Doug and Holly, for my very wrong way to put it. I'm as convinced as others of the great work you do. Holly, in particular, who drives and implements this pitch.

I'm very happy that we're having a better understanding of the difference between local and nested types, now.

If it's not too late, please let's keep on examining this great pitch together. Please don't let my mistakes have a too bad impact on Holly's work.

8 Likes

The latest draft has incorporated the return of ExtensionDeclSyntax nodes. It should probably address what the type syntax for the resulting extensions needs to look like---whether its passed in as you suggest, or should be the identifier and gets overridden by the compiler, or whatever.

Additionally, I think we need to prohibit the use of peer macros on the extensions generated by other macros. Otherwise, an extension macro applied to a nested type could, because extensions are module-scope, add new module-scope declarations. Aside from being problematic for the implementation, that would potentially be quite confusing.

Doug

1 Like

Whoops, you're right, thank you. To leave space to explore this suggestion in the future:

I think it makes sense for the requirement of ExtensionMacro to pass in a TypeSyntax for the providingExtensionsOf parameter separately from the "attached-to" declaration (similar to the requirement of MemberAttributeMacro):

  static func expansion(
    of node: AttributeSyntax,
    attachedTo declaration: some DeclGroupSyntax
    providingExtensionsOf type: TypeSyntax,
    in context: some MacroExpansionContext
  ) throws -> [ExtensionDeclSyntax]

I will also specify the restriction in the proposal.

Agreed, I will add this restriction to the proposal too.

3 Likes

apologies if this is a dumb question, but after looking over the pitch, i'm still not clear on one thing: will extension macros be able to declare conformances, and then implement them while still having access to the type members?

more concretely, right now i have a lot of kludgy codegen so that i can have custom Codable conformances for enums with associated types, for the purpose of better interop with TypeScript. when i saw macros demoed in the WWDC sessions it was immediately apparent that i could replace all the codegen with a macro.

so i'd like to convert my codegen code to a @TypeScriptCodable macro that would declare Codable conformance, and handle a custom implementation - but I would need to be able to inspect the type and iterate over the enum case members. If i got dropped into a bare extension context with just the ability to write arbitrary conformance methods, but without access to introspect the type, I'd be stuck.

2 Likes

Yes, this is exactly the point of extension macros.

1 Like