[Discussion] Easing the learning curve for introducing generic parameters

Introduction

Swift’s generics system is highly expressive, but understanding the full generality of protocols with associated types, generic signatures with where clauses, and other generics features is a significant barrier to introducing generics into a Swift project. A major goal of a more approachable generics system is easing the learning curve of abstracting a concrete API into a generic one by improving the ergonomics of writing generic code in Swift. This discussion is to solicit feedback on possible directions toward achieving this goal, and gather other ideas surfaced by the community. Questions, comments, and ideas are all welcome!

Many of the ideas in this post were laid out by @Joe_Groff in Improving the UI of generics.

The problem

Without a deep understanding of existential types and generics, existential types appear to be the most natural solution when abstracting away concrete types. The confusion between existential types versus generics is a long-standing problem for Swift programmers. Joe shed a lot of clarity on the difference between these two concepts in his write-up, but the language, tools, and documentation still make it far too easy for programmers to reach for existentials when a better solution is to use generics. And worse — when programmers hit the fundamental limitations of value-level abstraction, they often don’t realize that abstracting at the type level is the right path forward. Finally, once programmers do realize that they need to introduce generics, there’s a giant learning curve of understanding angle-bracket syntax, associated types, where clauses, type constraints, and more, before they can make further progress on whatever they were trying to accomplish in the first place. These poor ergonomics of introducing generics are a significant productivity barrier for programmers, and the steep learning curve combined with the lack of guidance frame the generics system as a tool only for the highly experienced Swift guru.

What follows is a set of possible directions for the language and tools to make Swift’s existing generics system more ergonomic by designing more progressive disclosure and guidance into the generic programming experience. Note that the goal is not to add more expressive power to the generics system.

Possible directions

Type parameter inference via some

Part of the reason why programmers reach for existential types over type parameters is because existential types have a much more natural syntax. To bridge the syntactic complexity jump to using generics, we could introduce type parameter inference from certain declarations that use opaque some types.

Parameter declarations

Parameter declarations with opaque types could be sugar for a type parameter on the enclosing function that conforms to the specified protocol. For example:

func concatenate(a: some Collection, b: some Collection) -> some Collection { ... }

Using some Collection as a parameter type would be sugar for a type parameter conforming to Collection :

func concatenate<C1: Collection, C2: Collection>(a: C1, b: C2) -> some Collection { ... }

This feature combined with light-weight same-type constraint syntax would allow type parameters and their constraints to be expressed directly on the relevant parameters, instead of separating semantic information about the parameter from the parameter itself, making generic function signatures easier to comprehend. For example, the following function signature:

func maxValue<C: Collection>(in collection: C) -> C.Element where C.Element == Int

could be simplified to:

func maxValue(in collection: some Collection<Int>) -> Int

Stored property declarations

Similarly, using the some modifier on stored property types could allow type parameter inference on the enclosing type:

protocol Shape { ... }

struct Button {
  var shape: some Shape
}

// sugar for

struct Button<S: Shape> {
  var shape: S
}

To make the generic signature more explicit for clarity, we could require programmers to write explicit placeholder syntax, such as <_> , on the enclosing declaration to signal that type parameters are being inferred from the properties inside the body, e.g.

struct Button<_> {
  var shape: some Shape
}

Type parameter inference from stored properties is an interesting direction, but there are some significant downsides that are important to highlight:

  • This feature is a source breaking change; a stored property with an opaque some type is valid Swift code today, but the concrete type is inferred from the stored property’s initial value specified via = , rather than inferring a generic argument at the use site of the enclosing type.
  • This would introduce a subtle difference between var shape: some Shape and var shape: some Shape { ... } . Refactoring a stored property into a computed one (and vice versa) is a common action, and this feature would introduce a type system difference between stored and computed properties.
  • This feature raises challenging questions about what it means to spell the enclosing type name in other contexts, including:
    • Can you explicitly specify generic arguments for the implicit generic parameters in angle brackets? If so, is the order dependent on declaration order of the stored properties, or is there some other canonical order? In either case, the programmer cannot simply look at the declaration of the type to see what must be specified.
    • If you cannot explicitly specify generic arguments, they necessarily need to be inferred. In that case, what does it mean to write the plain type name? How can you use such a type as, say, a parameter type?
  • This feature is significantly limited to simple cases where you do not need to use the property type anywhere else in the struct. As soon as you need to use the type elsewhere, you either need to name the type parameter as usual, or you need a new syntax for referring to the type of the stored property. In either case, this is exposing more of the learning curve to the programmer pretty quickly, which may render this potential feature to be a shallow step toward easing the learning curve.

Though type parameter inference via some would give generics a more light-weight syntax, programmers will still reach for existential types first, and they may struggle to understand what some means in different contexts and how it’s different from writing the plain protocol name. One possible solution is to allow programmers to elide the some keyword in certain cases, such as when using the sugared same-type constraint syntax, e.g. Collection<Int> . However, this may introduce more confusion between existential types and generics, especially given the acceptance of SE-0309: Unlock Existentials for All Protocols. As long as existential types have a more natural spelling, any syntax for type parameter inference would not alleviate the need for more informative and easily accessible guidance toward generics.

Same-type constraint inference via default arguments

When abstracting the concrete type of a stored property, it’s common to start with a property that has an initial value. For example:

protocol Shape { ... }

struct Circle: Shape { ... }

struct Button {
  var shape = Circle()
}

If a programmer wants Button to work with any shape type, and they want the default shape to still be Circle() , they might try to write:

struct Button<ShapeType: Shape> {
  var shape: ShapeType = Circle()
}

For which the compiler produces the following error:

error: cannot convert value of type 'Circle' to specified type 'ShapeType'
var shape: ShapeType = Circle()
                       ^~~~~~~~
                                as! ShapeType

Upon this error, the programmer might try other ways of providing a default argument, such as writing an explicit initializer with a parameter of type Circle . None of the diagnostics surfaced in this process will lead the programmer to the right solution, which is to write the default argument in an initializer with a same-type constraint where ShapeType == Circle . In order to keep the synthesized member-wise initializer that can be called for any ShapeType , this initializer must be written in an extension:

extension NoteButtonStyle where ShapeType == Circle {
  init(circle: Circle = Circle()) {
    self.init(shape: circle)
  }
}

This code is using a same-type constraint to default ShapeType to Circle when the default argument Circle() is used. We can alleviate this boilerplate, and (more importantly) the depth of understanding required to write it, by allowing default arguments to be more concrete than the parameter type, implying a same-type constraint when that default argument is used:

struct NoteButtonStyle<ShapeType: Shape> {
  var shape: ShapeType

  // Implies that `ShapeType == Circle` when `shape`
  // is omitted at the call-site.
  init(shape: ShapeType = Circle()) { ... }
}

This not only decreases the cognitive overload for the programmer, but it also improves compile-time performance by decreasing the number of overloads required to achieve a default argument with a concrete type.

Guide programmers toward type parameterization with tooling

The directions above make generics easier to write, but the generic programming experience is still missing guidance toward using generics in the first place. For programmers with experience in languages where the primary tool for polymorphic behavior is subtype polymorphism, guidance toward parametric polymorphism with value types is crucial for making Swift easier to learn.

Diagnostics

It’s possible to guide the programmer toward type parameterization using diagnostic fix-its. For example, if the user reaches for an existential type in a way that doesn’t work:

protocol P {}
extension P {
  func method(_: Self) {}
}

func useMethod(p: P) {
  p.method // error: member 'method' cannot be used on value of protocol type 'P'; use a generic constraint instead
}

The error message attached to p.method can include a fix-it to insert a type parameter conforming to P , and change the type of p to be that type parameter, transforming the useMethod function into:

func useMethod<PType: P>(p: PType) {
  p.method // 👍
}

Furthermore, error messages about existential types could have educational notes to explain when the programmer might want to use generics instead of existential types.

This strategy suffers from allowing the user to go down the wrong path for some amount of time before they discover that they should use a type parameter instead of an existential type. Any expressive power added to existential types exacerbates this issue.

Code completion

One possible way to guide the user toward type parameterization before going down the existential type route is via code completion when the user types a protocol name. Imagine the programmer is adding a type annotation for a stored property, and they start to type Collection :

struct Document {
  var lines: Collect|
}

Code completion could suggest a completion to insert a type parameter <CollectionType: Collection> on the enclosing struct, and complete the property type with CollectionType .

Refactoring actions

We can also help programmers with compiler-aided source transformations via refactoring actions. For a given declaration with concrete type, one could imagine a refactoring action to abstract away that concrete type with a type parameter, perhaps with an option to constrain that type parameter to any of the protocols that the concrete type conforms to.

Other ideas?

If you have other ideas for how to make Swift’s generics system more approachable, please leave a comment below with your thoughts!

35 Likes

a quick raw idea by means of example:

func foo(a: Equatable) {  // sugar for foo<T: Equatable>(a: T)
	a == a // ok
}

func foo(a: Equatable, b: Equatable) { // sugar for foo<T: Equatable>(a: T, b: T)
	a == b // ok
}

func foo(a: Equatable Apple, b: Equatable Orange) { // sugar for foo<Apple: Equatable, Orange: Equatable>(a: Apple, b: Orange)
	a == a // ok
	a == b // error, can't compare apples and oranges
}

I think part of the problem is that existential syntax is too good. Most of the time, people should be reaching for generics, but no matter how much clutter we can clear, we're never going to beat:

func doSomething(items: Collection)

Even some Collection for a generic parameter can't quite beat the existential syntax.

Nothing against existentials - they're great and all, but their syntax is so minimal it's misleading, and leads to users reaching for them even though there's a different tool that better expresses their intent.

16 Likes

Nice summary! I don't know if this might change the outcome of this work but it might be worth discussing / considering what is more urgent/important: helping people learn how to write generic algorithms or generic data structures?

Personally, I think that just improving the generics algorithm learning curve would be a decent win and a good stepping stone to learning how to write generic data structures.

1 Like

I am suspicious of syntax sugar as a general principle, but even I have to admit this is much nicer :smiley:

I agree with @Karl about the existing existential syntax.

6 Likes

That's why I think it'd be worth a source break to deprecate the current existential syntax, introduce any Protocol as the new existential syntax, then remove the old syntax in the subsequent language version.

10 Likes

how about:

func maxValue(in collection: Collection of Int) -> Int

alternatively:

func maxValue(in collection: Collection(Int)) -> Int

I do agree that the minimal spelling of existential types is part of the problem, but even if we can justify a massive source break to make existential types harder to spell, I still think we would need some of the ideas in this post for this reason:

Generic programming is a new way of thinking for a lot of programmers, and there is still a learning curve and ergonomic issues even if type parameters have as verbose of a syntax as existential types.

7 Likes

If bare protocol names were forbidden, learners would be guided via code completion to choose either a generic parametrization of the type/function (as proposed), the addition of the some keyword (where allowed) or the addition of the any keyword (as a third option, of course). Requiring any results in a call for action for the programmer who will be forced to understand that there's the trichotomy generic/opaque/existential when working with protocols. A simple search on the web (or better, a link to the proper apple/swift/userdocs document) would then explain their differences.

5 Likes

I think we should do both. Having these two solutions be spelled out by any/some makes the distinction clearer and doesn’t favour one or the other.

1 Like

I never said we couldn't do both, I was only expanding on Karl's observation that existentials are still going to be the path of least resistance as long as they have the nicer syntax.

1 Like

The only thing that is really necessary for swift is better type inference, so people can focus on writing expressions and mind types later. It looks silly that after all years language has been around there is still a significant burden caused by the need to be explicit in the syntax of generics. In a reasonable language, like haskell, there is rarely a need to mention type at all so the experience of writing some code feels pleasant. Why haven't there been any progress done on this? The only thing that got implemented is better inference in closures, and that's pretty minor for the time passed since inception of the lang.

@hborla I had to sleep on this for a bit, so apologies for the somewhat delayed reply. Thank you for writing this document down; as you say, it's a continuation of sorts of @Joe_Groff's earlier document.

To start, I fundamentally agree with the problem statement. All the aspects of your post with respect to tooling are, in my view, essential.

We need to do more to diagnose and guide users to use these features correctly. Some learners will learn best by a careful exposition of the rules, and we can certainly work on improving the educational notes and TSPL sections for those folks, but others learn best by rolling up their sleeves and doing, and it's in the careful implementation of helpful diagnostics and fix-its that we can best reach those users.

Fundamentally, I think any time a user stumbles on a set of diagnostic messages that they can't address such that they find themselves having to ask on these forums or elsewhere we have an opportunity for improvement.


Now, as to the proposed language changes, the portion about using some T as a shorthand for a generic function parameter has been well explored before and, as I have said at those times, makes a lot of sense in my view. This would be because many have found the explanation of opaque types as "reverse generics" to be intuitive to them, and the expansion of this syntax adheres to that well worn principle that similar things should look similar and different things different. It naturally opens the door as well on going the other direction in generalizing the long-form generic syntax for more complicated constraints on opaque types (i.e., something like func() -> <T: Collection> where T.Element: Codable).

(Needless to repeat here, I also agree that the bare spelling of existentials is an attractive nuisance and that the way to fix it is to adopt the long mooted any T spelling, in line with what Rust has done, and that this could be done in concert with generalizing the some T spelling above to create that nice balance which steers users the right way.)


But as to the additional portions you outline here, I have to say that I am similarly wary of these as in the other pitch about lightweight syntax for protocol associated types. You very rightly point out a major caveat to the some Shape example:

This is a design "smell," as it were, that two things are being made to look alike which are dissimilar. I struggled to see why that would be the case but I think I can explain it now:

struct Button {
  var shape: some Shape
  var shape2: some Shape { ... }
}
// ...sugar for...
struct Button<S: Shape> {
  var shape: S
  var shape2: some Shape { ... }
}

// is analogous to blurring the difference between:

struct S<T: Shape> {
  func f(_: T) { ... }
}
// ...and...
struct S {
  func f<T: Shape>(_: T) { ... }
  // which we agree could have a nice shorthand as:
  // func f(_: some Shape) { ... }
}

I do not think that blurring these lines is advisable. This is an opportunity for the user to learn why they might want to use one or the other and then to be able to apply that to scenarios where the compiler can't so easily help correct the user; just silently accepting that the user wrote one thing but meant the other is a missed opportunity and not doing them any favors in the long run. For the same reason that I am wary about adopting <T> syntax to meant "generic constraint or associated type," so too am I wary of adopting some T syntax to mean "generic in some way."

I think my overall take is this:

The distinctions between opaque types, generic constraints, associated types, existentials and their ilk are meaningful ones, not there merely to frustrate the uninitiated. Therefore, our task as I see it is to design Swift in a way that discloses these differences where they arise and to guide users to the correct usage.

Where the distinction doesn't matter, I do not think that the compiler should be changed to accept just any plausible syntax as "do what I mean--you (the compiler, or human reader of my code) figure it out," because it is not helpful for user learning so that they can make the right choice in those scenarios where the distinctions are meaningful and the compiler can't just figure it out. I think such a "do what I mean" approach actually runs counter to the principle of progressive disclosure and delays rather than catalyzes mastery of the language.

(By contrast, if the distinction that we want the user to learn actually never matters and the compiler can always figure out what the user means, then my premise is wrong, and we should get rid of the distinction in the language entirely. But, as I say, my premise is that there's a useful reason why Swift has generic constraints and opaque types and existentials and associated types.)

Instead, I propose that if the compiler can figure out unambiguously that the user means to do X when they write Y, there should be a fix-it and a helpful message. Circling back concretely to the struct Button { ... some Shape } example, don't make it sugar for a generic Button type, emit a fix-it so that the user can learn the difference.

19 Likes

I believe existing responses mostly capture my feeling here so I'll keep it brief.

I agree with @Karl here that the fundamental issue is existentials receiving the "blessed" bare-name syntax. If we think that the language should guide users towards generics first (or at least put generics on equal footing with existentials), then IMO that's an argument that elevates the bare-name existential syntax to the level of "active harm" that would begin to justify what would presumably be a massive source break.

This would be a harder bar to clear, obviously, but IMO we should explore this direction first and only then consider alternatives if it's decided that the break would be too burdensome for developers.

1 Like

Existing use of some means “the function chooses the type”, but this proposed new use means “the caller chooses the type”. Consider:

func f(input: some RangeReplaceableCollection) -> some RangeReplaceableCollection {
    ...
}

The caller can pass in any conforming type it wants, but can't control the type it gets back. Yet the syntax is identical. And because it's RangeReplaceableCollection (which has an init() requirement), we really can write a function that returns any conforming type of the caller's choosing:

func f<In: RangeReplaceableCollection, Out: RangeReplaceableCollection>(input: In) -> Out {
    return Out()
}

I don't think we should allow the same syntax to have such different meanings.

1 Like

I strongly disagree with using some for generics. some in argument position and generics are dissimilar, but the syntax change will make them similar.

As discussed in the Structural opaque result types thread, (some P) -> (some P) would have following behavior.

// take opaque Numeric value and return opaque Numeric value.
let foo: (some Numeric) -> some Numeric = { (value: Double) in value * 2 }

let doubleValue = 4.2
foo(doubleValue)  // error (since some Numeric is opaque)
foo(4)            // OK (due to ExpressibleByIntegerLiteral conformance)
foo(.zero)        // OK (due to static member requirement)

However, if we adopt some for generics, the behaviors of closure and function get torn.

func foo(value: some Numeric) -> some Numeric { return value * 2 }
let doubleValue = 4.2
foo(doubleValue)  // OK (since value is generic)
foo(4)            // OK (since value is considered as Int)
foo(.zero)        // error (since there is no way to infer generic type parameter)

It would be really confusing for many developers. What is wrong here is using some for generics. some in the argument position should be also opaque in function.


If we want to add sugar, then we should use any for generics.

// two functions behave the same
func foo(value: any Numeric) -> some Numeric { return value * 2 }
func foo<T: Numeric>(value: T) -> some Numeric { return value * 2 }

Actually, Rust explained its impl as hybrid of generic any and reverse generic some in RFC 1951. I think this is logical direction.
As of existential types, we should search some other prefix like exist.

4 Likes

This is a good point.

IMO, if we extend some types to parameter positions (which, broadly, I think I am in favor of), I believe it should be an error to use some type in both parameter and result position. At that point, we should guide the user toward making the function explicitly generic to avoid any possible confusion.

See, on the contrary, I think some is appropriate even from this angle (besides the whole opaque types being reverse generics angle):

In plain language, the function takes an argument of some type and returns a value of some type. If I insert some money into a machine, I choose both the quantity (value) and denomination (type) of the bills I insert, and if the machine returns some amount of money, it (not I) chooses both the quantity and denomination of the bills.

Who chooses is determined by which side of the function arrow it’s on—both for the value and the type—and some here unifies the concepts in denoting that a specific choice has to be made (statically) as to the underlying type, as opposed to existentials.

In the absence of empiric evidence, I am not convinced that users will have trouble appreciating the locus of control here; it’s not as though users struggle to understand who controls the argument value versus the return value.

13 Likes

I would say it takes any type conforming to the protocol, and returns some specific type conforming to the protocol.

2 Likes

Sure, and you could also say that it returns any one type conforming to the protocol—from your perspective as the caller, you don’t get the choose, so it’s “any.” But in Swift we’ve established the use of Any* for existential boxes, so it would not be appropriate for this distinct concept.

The question, rather, is whether some is workable, and I argue that it is, and that it’s intuitive.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy