[Pitch] Member macros that know what conformances are missing

Evolution Pitches
1

Hi all,

The move from conformance macros to extension macros included the ability for extension macros to learn about which protocols the type already conformed to (e.g., because a superclass conformed), so that one could avoid adding declarations that aren't needed. It also meant that any new declarations added are part of an extension---not the original type---which is generally a good thing, because it means that (e.g.) a new initializer doesn't suppress the memberwise initializer, and it's usually considered good form to split protocol conformances out into their own extensions.

However, there are some times when the member used for the conformance really needs to be part of the original type definition. For example:

  • An initializer in a non-final class needs to be a required init to satisfy a protocol requirement
  • An overridable member of a non-final class
  • A stored property or case can only be in the primary type definition

A Codable-synthesizing macro is the most obvious case for this, because it hits both of the first two bullets: when we're in a non-final class, we need to make the init(from:) be required, and also call the superclass initializer, and we also want encode(to:) to be overridable.

My proposal is that member macros can be declared with a conformances argument that has the same meaning as the one for extension macros, e.g.,

@attached(member, conformances: Decodable, Encodable, names: named(init(from:), encode(to:)))
@attached(extension, conformances: Decodable, Encodable, names: named(init(from:), encode(to:)))
macro Codable() = ...

and that member macro implementations will use this new requirement to get the list of conformances that are "missing":

  /// Expand an attached declaration macro to produce a set of members.
  ///
  /// - Parameters:
  ///   - node: The custom attribute describing the attached macro.
  ///   - declaration: The declaration the macro attribute is attached to.
  ///   - missingConformancesTo: The set of protocols that were declared
  ///     in the set of conformances for the macro and to which the declaration
  ///     does not explicitly conform. The member macro itself cannot declare
  ///     conformances to these protocols (only an extension macro can do that),
  ///     but can provide supporting declarations, such as a required
  ///     initializer or stored property, that cannot be written in an
  ///     extension.
  ///   - context: The context in which to perform the macro expansion.
  ///
  /// - Returns: the set of member declarations introduced by this macro, which
  /// are nested inside the `attachedTo` declaration.
  static func expansion(
    of node: AttributeSyntax,
    providingMembersOf declaration: some DeclGroupSyntax,
    missingConformancesTo protocols: [TypeSyntax],
    in context: some MacroExpansionContext
  ) throws -> [DeclSyntax]

Member macros still cannot produce conformances themselves, but at least they can react to whether conformances are already there vs. could be synthesized by an extension macro.

I have cobbled together an implementation here.

Thoughts?

Doug

11 Likes
2

I'm having a lot of trouble naming this parameter. Perhaps I should call it preexistingConformances, because they are conformances that exist prior to macro expansions that can add conformances?

Doug

1 Like
3

I would just call it missingProtocols, especially considering you're already calling the argument protocols for the function body

4

I like the name :+1:

How would a macro implement Codable in this case? Would it provide the implementations using the member role and the conformance using the extension role? If so, this feels a bit awkward, but it‘s not a dealbreaker.

Is there a case (apart from inheritance) where the conformance itself must be declared in the original type definition? If so, it would be nice to be able to add those conformances using a macro. …come to think of it: It would be nice if we could find a way to add inheritances using macros.

5

It's the opposite of this, isn't it? They're the conformances that do not exist in the original source prior to macro expansion, so an extension macro is free to add those conformances. The ExtensionMacro protocol uses the label conformingTo because the expansion method will add extensions that state conformances to those protocols.

Maybe we could name the label based on what the protocols will be used for in the member macro expansion? Something like includingWitnessesFor, implementingRequirementsFor, etc?

6

Yes. I think if I were to write a Codable implementation using this, I'd have the extension role provide the conformance in all cases, and also provide the init(from:)/encode(to:) to for everything except a non-final class. That way, it wouldn't interfere with (e.g.) the synthesized memberwise initializer for structs, and the whole conformance would be nicely separated out into a self-contained extension. It's only for the non-final classes where we'd need init(from:)/encode(to:) to be members of the class.

The other case I called out is stored properties, i.e., if you want the macro to generate a stored property that helps satisfy a requirement.

Yes, you're right. preexistingConformances is wrong.

I think we should continue to avoid the word "witness" here in the surface language. implementingRequirementsFor is... really long, but accurate. I'm happy to use that as a slightly-better placeholder than missingConformancesTo, but it still feels like there's something better.

Doug

1 Like
7

This feels like it has the potential to become a pretty subtle sharp edge for macro implementors. The difficulty we're having in naming this parameter is indicative of what a narrow concept we're trying to describe, and I'm wondering if we might be better served by revisiting the discussion of whether extensions that appear in the same source file as the original type declaration ought to be fully privileged and behave exactly like being part of the original type.

In past discussions I've been against such a change, but if we're getting to the point of abstracting over these restrictions in the macro layer, I might be convinced that it's time to relax them.

6 Likes
8

I would greatly appreciate this approach. It would make conditional compilation of conformances so much easier, because I could put the conformance in an extension along with all the stored properties that only exist to support that conformance.

1 Like
9

This would have a number of effects that I can think of:

  1. Stored properties would be scattered across extensions, so it wouldn't be clear what "the storage" of a type actually is without tools
  2. Similarly, the implicit memberwise initializer involves looking across all of the extensions
  3. Classes would get many more overridable methods---it's everything in the file. This would pessimize existing code (larger virtual tables, more dispatch through them)
  4. Protocols get more requirements, where any member of a protocol extension is both requirement and its own default implementation (?)

Personally, I like that the type definition provides the whole "shape" of the type in a single place: its fields, requirements, superclass, required initializers, overridable members, etc. It's what the type "is". Everything provided in extensions is "extra" on top of that core shape, giving it what the type "does". And while it would make some things easier if an extension could extend that core shape, I think the loss of the ability to look in a single place to find a type's "shape" would make code overall harder to understand.

Doug

4 Likes
10

Hey all, thanks for the feedback so far. I turned this into a proper proposal document with a bit more detail, updated the implementation, and kicked off toolchain builds so we can play with it.

I added an Alternatives Considered section to capture the idea of "implementation extensions" that @Jumhyn brought up here. Having written more about it, I'm feeling a bit more positive on this approach than my earlier reply suggests... but I'm still thoroughly on the fence and would love more input.

Doug

2 Likes
11

Thanks for writing that up!

I definitely share this feeling, and it's why I've been apprehensive about imbuing extensions with these powers in the past. OTOH, I also think that in a world of macros we're likely to run up against situations like this in the future where 'what's ideal for humans to read and write' is at odds with 'what's easy for a macro to process in a generalized way,' and it's not obvious to me that it makes sense to require macros to adhere to the same expectations that we'd have for source 'on the page.'

For instance, the first drawback of 'implementation extensions' that you mention above:

is already out the window when macros are in the mix, so I don't think it really makes sense as an aesthetic principle to hold macro expansions to. Easily understanding what the memberwise initializer will look like is also defeated by macros.

I wonder if we ought to open a path for macros to generate code that we accept as valid, while still discouraging it for general use. I suppose this would look like downgrading some current errors to warnings, and then also providing a way for macro authors to swallow warnings from the code they generate. Applied to the current discussion, that would mean that, e.g., defining stored properties in a same-file extension would generate a warning ("stored properties should be defined in the main type body"), but macros would still be able to generate such code and prevent the warning from being 'passed through' to the end user (since there's nothing they could do about it).

I suppose formalizing the idea of 'implementation extensions' is another way to sidestep this issue and just invent a tool to solve the problem at-hand that we consider acceptable for general use.

I'm not seeing why this need have an effect on any existing code. Yes, classes would get more overridable members, but those members are definitionally not being overridden today, so it seems like the compiler should be able to see that and eliminate the dynamic dispatch in such cases, no? I suppose the only exception would be open methods which are today declared in extensions on open classes (apparently this is just a warning, not an error) where we can't necessarily see the universe of overrides.

If don't go all in on formalizing 'implementation extensions' as a concept, and instead just become more permissive about what we allow same-file extensions to do, then I don't see any reason why we must extend the behavior to protocols as well. I'd similarly consider it weird if we allowed protocols to have 'implementation extensions.'

12

Perhaps, but in this case the needs of humans and of macros are the same: having the declared stored properties all in the original type definition is what lets macros like the Codable in the pitch, or Observable, or SwiftData's Model, or any other "abstract the storage" macro to actually work. If we let implementation extensions add stored properties, these macros won't work in the presence of them. I hadn't realized it in my original replies, for me this is the killer argument against implementation extensions (at least ones that can add stored properties): it breaks the same macros we're trying to fix.

I've been resisting this since the beginning of the macro system, because I really want to preserve the ability to expand a macro out of existence without having to rework your code.

With whole-module optimization, yes. With incremental builds, no---those builds can't see all potential subclasses.

We could certainly say that this feature doesn't support protocols, if we don't think it has meaningful semantics. I'm trying to suss out what the feature would mean, because it's an interesting alternative to my proposal.

Doug

3 Likes
13

Ah, yes. This is a very good point.

14

If macros could access extensions, then this point would be moot. And it occurs to me that this might already be a problem for some macros that e.g. rewrite computed properties. It would be surprising for such a macro not to apply to computed properties declared in an extension!

15

Right, to really put implementation extensions on equivalent footing with the main declaration, we would have to expose a way for macros on a primary type definition to gain access to all the implementation extensions in the file as well. That might be a little odd since the macro is not, in fact, attached to the implementation macros syntactically.

16

This proposal is now in review as SE-0407.