Newtype without automatic protocol forwarding


(Tom Lokhorst) #1

Goal: Two semantically different types, with the exact same shape, should have a different Swift type. Creating a new type should be as light-weight as possible.

Introduction

It is often useful to create a new distinct type, based on a previous type. We propose newtype as a language feature to make this possible in an easy and concise way:

newtype RecordId = Int

This should create a new type, that isn't equal to the original type on which it is based. In particular, this shouldn't compile:

func pow(_ n: Int) {}
func get(id: RecordId) {}

let someId: RecordId = RecordId(value: 42)
pow(someId) // Error, RecordId is not an Int

let someInt: Int = 42
get(id: someInt) // Error, Int is not a RecordId

Note that in this way, newtype is different from the existing typealias, because it creates a new type, that is not freely exchangeable for the original type.

Semantically, this newtype declaration would expand to what can be done currently with a new struct:

struct RecordId {
  var value: Int
}

Motivation

Using a new distinct type for domain models helps to prevent mistakes and improves clarity of the codebase. For example, when two identifiers have the same type, it is easy to accidentally use the wrong variable when calling a function:

struct Person {
  let id: String
  let name: String
}

struct Building {
  let id: String
  let owner: Person
  let title: String
}

func scrollToPerson(withId: String) {}

scrollToPerson(withId: mainBuilding.id) // Uncaught accidental error

When we manually create a distinct type, it becomes possible for the compiler to prevent this mistake:

struct Person {
  struct Identifier { // Without newtype
    var value: String
  }

  let id: Identifier
  let name: String
}

struct Building {
  struct Identifier { // Without newtype
    var value: String
  }

  let id: Identifier
  let owner: Person
  let title: String
}

func scrollToPerson(withId: Person.Identifier) {}

scrollToPerson(withId: mainBuilding.id) // Error, incompatible type
scrollToPerson(withId: mainBuilding.owner.id) // Correct

Manually creating new types in cases like these would improve safety and clarity at the point of use. But manually creating a new type using a nested struct creates a lot of visual overhead, preventing it from becoming a common pattern. This is especially true for types that can just as easily be represented using an existing type, like the identifiers in the example above.

Lowering the cost of declaring new types from existing types will encourage users to write more strongly typed code, which improves code quality and readability.

Proposed solution

To make it easier to create a new distinct type based on an existing type, we propose the newtype feature. With this feature the example becomes:

struct Person {
  newtype Identifier = String

  let id: Identifier
  let name: String
}

struct Building {
  newtype Identifier = String

  let id: Identifier
  let owner: Person
  let title: String
}

This newly created type generates a struct box around the existing type in a value field. It can be used in the following way:

let personId = Person.Identifier(value: "Jane")
let stringValue = personId.value

Just like any other type in Swift, it is possible to write an extension on Person.Identifier to implement protocols and add members.

No automatic protocol implementations

Note that the newly created type doesn’t implement any of the protocols implemented by the original type. Nor does it implement any of the members on the original type. This fully separates the new and original types and prevents semantically incorrect behaviour. In the Person identifier example, this means it doesn’t implement CustomStringConvertable and two Person.Identifiers can’t be concatenated using +.

There are valid use cases for automatic forwarding of (some) protocols or members from the original type to the new type, but this is (explicitly) not part of this proposal. Several different approaches to implement this functionality are discussed in the paragraph Possible Future Extensions.

We believe the newtype feature to be a useful addition to the Swift language, even without automatic forwarding.

Use cases

We see 3 primary use cases that we’ve extracted from existing codebases and public discussions.

1. Identifiers

As described above:

struct Person {
  newtype Identifier = String

  let id: Identifier
  let name: String
}

Identifiers are used in a lot of applications, these often are numbers or strings and important not to mix up. Example of a online discussion about creating a distinct type for identifiers: https://twitter.com/cocoaphony/status/756538935707365376

2. Separating semantically distinct fields

In an app dealing with shipments of packages, it is important to never mixup the sender and receiver fields, even though they’re both of the same Address type:

struct Shipment {
  newtype Identifier = String
  newtype Sender = Address
  newtype Receiver = Address

  var id: Identifier
  var sender: Sender
  var receiver: Receiver
  var contents: String
}

This example is taken from our app for the Dutch Postal Services. See this gist for how this is implemented without the newtype feature. The version with manually written types is considerably harder to understand.

3. Phantom types

Phantom types can also easily be created using newtype, taking the example from objc.io:

newtype FileHandle<T> = Foundation.FileHandle

Source: https://www.objc.io/blog/2014/12/29/functional-snippet-13-phantom-types/

Possible Future Extensions

Previous discussions about newtype have gotten stuck on the topic of protocol implementations/forwarding. This is why we have kept it out of scope for now. However adding protocol forwarding would make this a more useful feature.

As a motivating example, say we have a Point with separate X and Y types (from LoĂŻc Lecrenier):

struct Point {
  newtype X = CGFloat
  newtype Y = CGFloat

  var x: X = 0.0
  var y: Y = 0.0
}

With this type, we prevent accidentally mixing up the Point.X and Point.Y types, but we also can’t calculate with them. We’d need to add an extension with implementations for all numeric operators. Here, some form of automatic protocol implementation would be useful.

Approaches:

1. Automatic implementation of Hashable & Equatable

Just hardcode these two protocols in the compiler, for all newtypes. For the same reason that these are already hardcoded for value types.

2. Add a Newtypable protocol

Have all newtypes automatically implement a new protocol Newtypable. That way users only have to write a manual implementation once using conditional conformance. See example gist.

3. Add a Haskell-like deriving syntax

Similar to Haskell, we could add a new language feature where the user can explicitly opt-in to certain protocols by specifing them:

newtype X = CGFloat deriving (FloatingPoint)

In the future, this could also be extended to included the latest state-of-the-art in Haskell, deriving-via: https://www.kosmikus.org/DerivingVia/deriving-via-paper.pdf

Previous discussions

--
Tom Lokhorst, Q42 (https://twitter.com/tomlokhorst)
Mathijs Kadijk, Q42 (https://github.com/mac-cain13)


(Dave DeLong) #2

YES YES YES YES YES YES YES

I have a Newtype protocol in my core frameworks to mimic this, but it's a poor substitute for an official language feature:

public protocol Newtype {
    associatedtype RawValue
    
    var rawValue: RawValue { get }
    init(rawValue: RawValue)
}

extension Newtype where Self: Equatable, RawValue: Equatable {
    public static func ==(left: Self, right: Self) -> Bool {
        return left.rawValue == right.rawValue
    }
}

extension Newtype where Self: Comparable, RawValue: Comparable {
    public static func <(left: Self, right: Self) -> Bool {
        return left.rawValue < right.rawValue
    }
}

(Johannes Weiss) #3

Yes, please!


(Jacob Williams) #4

Definitely a +1 from me too. A feature similar to this has been brought up before but no official pitch/proposal has ever been made. This feature could solve a large number of issues where reusing types for things like identifiers or file handles leads to hard to track bugs.


(Jordan Rose) #5

Any reason not to use the existing RawRepresentable protocol automatically? That would also give you a default implementation for Equatable already (though not an implicitly-added one).


(Dave DeLong) #6

How would using RawRepresentable work with the "Address" example in the OP?


(Jordan Rose) #7

I just mean making the default synthesized struct be

struct Recipient: RawRepresentable {
  var rawValue: Address
  init(rawValue: Address) { … }
}

instead of

struct Recipient {
  var value: Address
}

But I can potentially answer my own question here with another question: what's the access level of value/rawValue? Does it match the access of the newtype, or is it always internal or private? If the latter, it's not suitable for RawRepresentable.


(Douglas Gregor) #8

Given the prevalence of protocols for expressing common functionality in Swift, I consider a newtype-like facility without some form of (opt-in) forwarding of protocol conformances to be too small to be useful. Indeed, I think the right feature for Swift is to add opt-in forwarding of protocol conformances without adding newtype, per my comments in Opaque result types.

Doug


(Jacob Williams) #9

The OP did state it would be 'semantically' similar to a struct with a value, but it doesn't have to be exactly equivalent to that. I'm no expert, but perhaps it would be possible for the compiler to fudge the access level so that we could have the RawRepresentable conformance while still hiding the internally stored value?

Of course, part of the pitch does state Int is not a RecordId and having RawRepresentable with the copied type would break that wouldn't it?

It also may be against the goals/design of this proposal to do too much compiler magic.


(Jordan Rose) #10

It's not a compiler restriction. Conforming to a protocol means exposing the members that satisfy the requirement because anyone could just write a generic function that does the same thing. In this case, conforming to RawRepresentable on a public newtype means that the newtype has a publicly-accessible rawValue method.


(Jon Shier) #11

This seems like a huge surprise and yet has no justifying explanation. What's the point of basing one type on another without forwarding conformances and functionality? What use is newtype Identifier = String when I can't do anything with it?


(Nobody1707) #12

If you just want a newtype without the forwarding mechanism, then couldn't you just use Tagged? I don't see how this rates syntactic sugar if there's no forwarding mechanism.

If Tagged's pre-existing conditional conformances bother you then you can always roll your own, as the overwhelming majority of that (single source file) library is the conditional conformances.

public struct Tagged<Wrapped, Tag> {
    var wrapped: Wrapped
    public init(_ wrapped: Wrapped) { self.wrapped = wrapped }
}

// For when you don't need a tag.
public typealias Newtype<T> = Tagged<T, Void>

Edit: Sorry, I meant to reply directly to the OP, not to @jrose


(Brent Royal-Gordon) #13

I think this would be a good idea. We can recommend that types provide conditional extensions to RawRepresentable which forward to the underlying types, and these will be available to both enums and structs. We might even provide sugar on the protocol to provide the appropriate defaults, forwarding any Self parameter to the rawValue:

@rawForwarding public protocol Equatable {
  static func == (lhs: Self, rhs: Self) -> Bool
}
// Compiler synthesizes:
// 
//   extension RawRepresentable where Self: Equatable, RawValue: Equatable {
//     public static func == (lhs: Self, rhs: Self) -> Bool {
//       return lhs.rawValue == rhs.rawValue
//     }
//   }
// 
// So you can write something like this and automatically get a conformance:
// 
//   newtype Identifier: Equatable = Int

If we do go with RawRepresentable, we might also replace the newtype keyword with a syntax similar to the existing enum feature:

struct RecordID: Int, Equatable {}

You would, of course, be free to provide your own init?(rawValue: RawValue) if you needed to validate.

I'm not perturbed by having to make the rawValue public. Enums are already forced to provide a public rawValue if they want the convenient sugar; this is no different. In both cases, you can always forgo the convenience if you really need to control public use.

(But this does make me wonder if we could support some sort of "opaque associated type"—that would make it more difficult to use the rawValue in a way the type didn't want you to. For instance—to assume one of the proposed opaque syntaxes—if you made the raw type Int as Equatable, most code would only know that the raw value was Equatable unless it started fishing for the actual type with as?.)


(Jon Hull) #14

+1 with opt-in protocol forwarding

I would also like to suggest something for the "Future Directions" section. Allow newtype to make thunks for PATs. For example:

protocol A {
    associatedtype Element
    var element:Element
}

newtype GenericA<T> = A where Element == T 

would generate at compile time:

struct GenericA<T>:A {
    typealias A = T
    private var box:GenericABox<T>
    init<P:A>(_ boxed:P) {
        self.box = GenericAConcrete(boxed)
    }
    var element:A {
        return self.box.element
    }
}

private class GenericABox<T>:A {
     typealias A = T
     var element:A {fatalError()}
}

final private class GenericAConcrete<P:A>:GenericABox<P.A> {
    var concrete:P
    init(_ concrete:P){
        self.concrete = concrete
    }
    override var element:A {
        return self.concrete.element
    }
}

You could also specialize without generics:

newtype IntA = A where Element == Int

or completely erase a type:

newtype AnyA = A where Element == _

The last one would omit any setters of the Element type, and all getters/returns of Element would return Any instead. For example:

newtype AnyCollection = Collection where Element == _

would allow calls to things like count, and would return Any from things like first and the subscript getter. Setting values via subscript, etc... would not be possible in the wrapper.

Two optional (but super nice to have) pieces of compiler magic:

  1. The ability to unbox using as? or as!
  2. The ability to automatically box when assigning a conforming type to a variable of one of these newtyped thunks
let c:AnyCollection = myCollection //Auto Box
//vs
let c:AnyCollection = AnyCollection(myCollection)

guard let a = c as? MyCollection else {...}  //Unbox using as
//vs
let a:MyCollection = try c.unbox()

(Michael) #15

My initial thought seeing this was, +1! It is something I came across recently in a project I am working on.

I mean, this proposal would simplify this code:

internal struct Device: Encodable {
    internal let id: UUID

    internal init(id: UUID) {
        self.id = id
    }
}

to something like this:

newtype Device = UUID

My only concern with this is that it indirectly competes with the existing syntax (not a deal breaker for me personally).

typealias Something = UUID

What I would be more interested in seeing is what @jrose suggested above - having synthesized structs seems like a very reasonable approach.


#16

“Considerably harder” seems like an exaggeration to me. I understand the temptation to pare this pitch down to something uncontroversial, but I don't think it clears the bar if all it does is remove a few lines of very straightforward code (most of which only exist because of acknowledged deficiencies in automatic initialiser generation and will often not be required). In the common case I'm left comparing

struct Identifier { var value: String }
newtype Identifier = String

which is a hard sell.


(Charles Constant) #17

I'm moderately +1 on this and I can see how this proposal could come in handy. I think the name is really confusing, though, and might collide with a fuller featured "new type" in the future.

With the name "newType", I'd expect it to behave like a subclass, if a subclass were passed by value instead of reference. Like I'd assume the compiler would let me pass a MyInt to myfunc(foo:Int), but not an Int to myfunc(foo:MyInt)

This seems more like it should be "WrappedType" or "UniqueType" or something.


(Tim Vermeulen) #18

The failable initialiser seems problematic to me. How would forwarding init() of e.g. the SetAlgebra protocol work if the newtype's initialiser can fail?


(Tom Lokhorst) #19

I made a last-minute change to the title and forgot to include one important word: "automatic". I've updated the title: "Newtype without automatic protocol forwarding".

Protocol forwarding is a great feature, and newtype could definitly use it in some form. But I think automatic protocol forwarding would go against the principle of a new type, if it isn't truly distinct.

Like any other normal type, the new type should only implement protocols on a opt-in basis. The implementation of a protocol could also differ from the implementation of the underlying type.

Perhaps with one exception: Automatically implementing of Equatable and Hashable (if the underlying type implements those) would be very useful. Those two protocols in particular for the same reasons they can already be generated by the Swift compiler.

With Equatable and Hashable implemented, all three of our original use cases are handled: Identifiers can be compared and used as keys in dictionary. Sender and Receiver addresses are separated by their types, until the underlying Address value is actually needed.

Newtype does become more useful with optional protocol forwarding. But that's also true for normal structs.


(Tom Lokhorst) #20

Not really, I wasn't sure if it's semantically correct. The optional initializer for RawRepresentable is also bit weird in this context.

In one of our possible future enhancements, we use a Newtypable protocol, that could also be the existing RawRepresentable.

As for the access of the newtype. I would imagine to use the access modifier of the newtype for the value as well:

// A public newtype
public newtype Identifier = String

// Becomes something like:
public struct Identifier {
  public let value: String
  public init(value: String) { self.value = value }
}

Although the private version should probabibly use fileprivate for the members:

// A private newtype
private newtype Identifier = String

// Becomes something like:
private struct Identifier {
  fileprivate let value: String
  fileprivate init(value: String) { self.value = value }
}