Parameterized Extensions

I agree with this. A generic extension should not introduce the names of its parameters jnto the extended type.

6 Likes

Hi all,

I'm back with some more design questions that I believe needs to be hashed out.

I discussed earlier about the one-to-one mapping of extension parameters and the extended type and many have expressed that they expect those to canonicalize to the same thing. I agree. Generalizing this also means that one can swap between conditional conformance syntaxes like we've discussed way up this thread. However, although the one-to-one mapping would be a special case, how does this extend to other potential use cases?

// ok, children in this extension are treated as children like
// extension Array {}
extension<T> Array<T> {}

struct Foo<A, B> {}

// ok, this makes sense to also treat this extension as
// extension Foo {}
extension<T, U> Foo<T, U> {}

// this can rewritten to something more readable as
// extension Foo where B == A? {}
// the question is, should we rewrite the following extension
// as the more simpler one to allow users to swap between the two
// to preserve ABI compatibility?
extension<U> Foo<U, U?> {}

// ok, this is a one-to-one mapping, but the extension's parameter
// adds a requirement that is being imposed on Element.
// we could take these requirements and simply apply them to
// the parameter that's being one-to-oned.
extension<T: Equatable> Array<T>: Equatable {}

// this is an example of where we can't perform any sort of
// rewrite because we're making use of the introduced parameter
// in the signature without any same-type to relate T to.
// consider the third example in this code block, we were able to
// rewrite that extension because the first parameter had a one-to-one
// with A == U, so we could naturally express B in terms of A.
// this is not the case with the following because there is no other
// parameter to express Element in.
extension<T> Array<T?> {}

Obviously there are hundreds of examples I could give to demonstrate the fact that we (the compiler) can rewrite some parameterized extensions to be compatible with extensions we can already write today. In my opinion, I believe that it should perform this rewrite to as many extensions as it can (obviously it won't where it can't). I know @owenv expressed that swapping between compatible extension signatures shouldn't be compatible with each other, but others like @griotspeak and @jrose expressed that the one-to-one should be special and do expect those to be compatible. My question is, to what extent (assuming we're ok with some extensions being compatible with each other), should they be compatible? Should we aggressively rewrite all compatible parameterized extensions? Should we only do a one-to-one when there's a single generic parameter? Would love if others could chime in and express their opinion on this as well.

On the topic of these generic parameters being introduced into the extended type, I agree with others that these generic parameters should be private to the extension.

If there's anything here that you'd like explained more in detail, or maybe I explained something poorly, please let me know and I'll try to reword it to make it easier to understand. Please let me know your thoughts, concerns, or questions!

6 Likes

Here are my thoughts, in order of importance:

  1. Whatever rewrites you/we implement, it must be fixed by the time of the next release, because that's part of ABI stability.

  2. The completely unconstrained case is most important.

  3. The "only adds constraints" case seems easy enough to implement.

  4. Anything more and I'm worried we'd have trouble explaining the rules to people, which is still important even if we continue to productize the ABI checker. So, your Foo<U, U?> case would be nice if it were the same, but I'm not sure it's obvious how that example differs from the Array<T?> case. So whatever stopping place you/we pick, we should be able to describe it well. (But read on.)

So let's try to describe these cases:

  1. "An extension without generic parameters may not be rewritten as an extension with generic parameters or vice versa; doing so is not ABI-compatible."

  2. "An extension with generic parameters may only be rewritten as an extension without generic parameters or vice versa only when there are no constraints on any of the parameters, and the parameters map one-to-one to the original type's generic arguments. For example, extension <K, V> Dictionary<K, V> is equivalent to extension Dictionary, but extension <V, K> Dictionary<K, V> is not. Similarly, extension <T> Array<T> is equivalent to extension Array, but extension <T> Array<T> where T: Equatable is not equivalent to extension Array where Element: Equatable".

    [After describing this, the limitation on constraints seems weird.]

  3. "An extension written without generic parameters implicitly introduces generic parameters with the same names as the type's original generic parameters. That is, extension Array where Element: Equatable is shorthand for extension <Element> Array<Element> where Element: Equatable. Rewriting between these two forms is ABI-compatible."

    I like this because even if it isn't implemented this way it's very easy for people to reason about, and it makes it clear that you're just changing the names of generic parameters if you want to use T instead, which will always be allowed for extensions.

  4. "An extension with generic parameters where all parameters are used directly as generic arguments may be rewritten as an extension without generic parameters and vice versa without breaking ABI compatibility. However, if any parameters are not used as top-level generic arguments, the extension must remain parameterized."

    This is what you were asking about with the Foo<U, U?> rewrite, but I think it's actually the correct general principle for when it's possible to do a rewrite. It's a bit harder to explain but on the other hand I think it's 100% correct: that is, we know how to map every such parameterized extension to a non-parameterized extension automatically (through unification, I guess), and we know that if someone writes a parameterized extension that does introduce a new parameter, it wasn't possible to express the same thing using a non-parameterized extension.

    EDIT: I suppose equality constraints might make a newly-introduced parameter redundant, and therefore I think it's important that we'd still stick to the rule as written and not attempt additional unification.

Conclusion: I think #3 (very explainable) or #4 (correct and complete, but possibly harder to implement) would be good ways to go. But the most critical thing is that we can't add support for #4 later without breaking ABI.

6 Likes

Why is the last extension not the same? You said it's only important if there are no constraints. This extension has no constraints, it just has the order of the generic type parameters flipped. Is this also important?

Can you also elaborate a little more why these two are different?
What kind of 'difference' are we talking about in this case?

I'm more confused because with #3 you said that extension Array where Element: Equatable is the same as extension <Element> Array<Element> where Element: Equatable and then you also said "it makes it clear that you're just changing the names of generic parameters if you want to use T instead, which will always be allowed for extensions", which to me means that it's also equivalent to extension <T> Array<T> where T: Equatable.


@Alejandro I'm really happy to see this feature moving forward again. It would allow me to delete a huge chunk of duplicative code and make a lot of things simpler. (:crossed_fingers: this could land in Swift 5.3)

5 Likes

These are all attempts at "how we would explain a restriction to the user if we/Alejandro chose not to lift it", with the items in the second list corresponding to the items in the first list. You're right that a normal Swift developer probably won't think of the order of generic parameters as being ABI; they are, unfortunately. (Maybe we should have sorted by name and not allowed you to rename them instead, but we didn't.)

2 Likes

What is the plan for overlap detection? I looked at the PR and I don't see anything additional there for detecting overlap. For instance, if we have

struct S<A> { ... }
struct X { ... }
protocol PT { ... }
protocol PS { func f() }

extension<T : PT> S : PS where A == T { func f() { print("Generic") } }
extension         S : PS where A == X { func f() { print("Specialized") } }
extension X : PT { ... }

let s : S<X> = ...
s.f()
  1. What are the semantics of this? Do they depend on which modules the different declarations are in?
  2. Do we provide an error or warning about the ambiguity?
  3. Do we have syntax that lets the user manually pick which one they want?

This problem is already present with the existence of conditional conformances, and I am afraid by adding more ways of writing them (in this case, making equality constraints available, not just conformance constraints), we might exacerbate the issue. It seems even more of a problem with the "Future Directions" part where you allow blanket extensions over any generic type, because that can overlap with anything else (you can look at Haskell's OverlappingInstances extension, for example of the headaches this can cause). For now, at least we have one level of nesting in the extended type (e.g. S or Array or Pair) which makes it somewhat less likely.

3 Likes

sorry for the maybe super newbie/obvious question, but why do we need to declare the <T> twice?

isn’t

extension Array<T> {}

enough context to allow the feature to work?

—————

to be upfront, i’m just worried about the possibility of some really hard to parse (as a human) declarations like

extension <T, U: Codable, V: Optional> SomeObject<T, U, V>

when

extension SomeObject<T, U: Codable, V: Optional> 

should seem to work (to me anyways)
—————

if i’ve missed something, some pointers would be appreciated...

thanks
(ps. i really hope this fearure can get in soon ^^)

1 Like

The main issue is that you should be able to do things like extension Dictionary<String, T>. In this case the compiler needs to know which are existing types, and which are generic types. Making it infer which is which just by the existence of the type is prone to mistakes, and could have its behaviour modified by writing code elsewhere (e.g. in another file struct T {}).

4 Likes

to be upfront, i’m just worried about the possibility of some really hard to parse (as a human) declarations like

DeFrenZ explained the issue with your suggestion, but that doesn't answer this concern. The answer to this concern is that you can write out the constraints later, so

// Discouraged
extension <T, U: LongProtocolName1, V: LongProtocolName2> SomeObject<T, U, V> {}
// Better
extension<T, U, V> SomeObject<T, U, V> where
  U : LongProtocolName1, V : LongProtocolName2 {}

// Discouraged
extension <T: Foo & Bar & Baz, U: Foo & Bar & Qux> SomeObject<T, U> {}
// Better
extension<T, U> SomeObject<T, U> where
  T : Foo & Bar & Baz,
  U : Foo & Bar & Qux {}

I might've messed up the punctuation, but you get the idea.

It's the same problem as with functions with many generic parameters.

5 Likes

thanks for the reply

i guess not declaring the generic parts beforehand also slows down parsing and other checking...

one other idea though, what if you had to declare the generic parts just like a function on the object you extend, but can only constrain it using where afterwards?

like

extension SomeObject<T,U,V> where
  T: Decodable & SomeOtherProtocol
  U: Optional
  U.Wrapped == T
  V: Result<Any, Error>

in other words, require the declaration of the generic parts firstly (just like with a function), but instead of writing it potentially twice, just do it once and require the contraints to use where at the end

is that possible or does it still not meet the requirement as outlined by @DeFrenZ ?

In theory, perhaps. The problem with this approach, however, is that we're extending a type Array with element type T, but we haven't declared T anywhere in source. It'd be confusing to expect the compiler to synthesize generic types out of the blue like this. For this type of extension it's best to just use what's possible now with:

extension Array {}

where there's no confusion about what's happening.

Another aspect I see is that for every other context in Swift (structs, classes, enums, functions, subscripts, etc.) they have to explicitly declare new generic types that they're introducing:

// ok make generic type T for this struct
struct GenericStruct<T> {}

// ok make generic type T for this class
class GenericClass<T> {}

...

Again, it'd be really confusing to learn that extensions are the exception. It seems naturally that we'd need to explicitly declare the generic types for an extension too.

This type of extension is already possible in Swift.

// assuming SomeObject's generic types are named U and V
extension SomeObject where U: Codable, V: Codable {}

The parameterized extension version says that we're declaring three new generic types, T, U, and V. We're then extending a type SomeObject whose generic types align with T, U, and V.

I completely agree. When I read through this, it felt like your #4 was an extension on #3 (implementation wise). We can start with #3, evaluate if this meets most use cases or if we're satisfied with just this amount of rewrite, or extend it to support #4. (When I say extend support, I mean before we ship this feature, perhaps toolchains so others can learn the rewrite rules, etc.) I think it makes sense to start with #3, and see community input to check if #3 is enough for understanding, or if they prefer more complicated rules around rewriting.

2 Likes

thanks for the thoughtful reply, its appreicated!

hope this feature can get in soon enough ^^

I worry about one thing with parameterized extension, it give yet another syntax for generic extensions:

extension Array where Element: Codable { }
extension<T> Array<T> where T: Codable { }
extension<T: Codable> Array<T> { }

Shouldn't we allow the generic parameter to only be used with the where clause? This would reduce number of possible syntaxes and make code more consistent.

1 Like

Is this really a problem though? I think it‘s a implication of the general feature and the differences in syntax forms will be just personal preferences. It doesn‘t hurt you if your form is supported but it will hurt someone who prefers the other form if it‘s artificially unavailable.

4 Likes

I'd be in favour of consistency so that code is uniform and beginners don't have to ask themselves what's the difference between the two syntaxes.
I know I might be kind of alone in that thinking but wanted to say it anyway :slight_smile:

Apart from that... parameterized extensions would be really useful :slight_smile:

6 Likes

@Alejandro just respectfully and kindly checking if there is any progress with this feature? I doubt that it could land in 5.3 (not even with cherry-picking) even though I'd pay for it, but it would be cool if it could land on master soon-is.

4 Likes

I think I would too :smiley:

I'm working right now on an abstract data types library, and the lack of parametrized extensions is a major problem, that requires ugly workarounds.

1 Like

In my code I just want to re-build existing code, delete a lot of boilerplate and no longer to be forced to create boilerplate code. I've been waiting for this feature to land for many many years and am really glad that it's finally moving forward.

1 Like

What about putting generic types in their own brackets like this: extension Dictionary<String, <T>>? The syntax does look a bit weird to me, but so does extension <T> ... and it’d solve the problem.