[Pitch] Nonescapable Standard Library Primitives

Hello all!

Now that the language has gained support for nonescapable types, we need to begin the work of adapting the core Standard Library to support them. Of course, we also need to continue deepening the support for noncopyable types that we started in SE-0436.

This draft proposal takes the next few steps towards fully integrating these new language features into the stdlib. Its primary goal is to add support for nonescapable types to the core Optional, Result, and MemoryLayout types. It also resolves minor API omissions that have come to light since we landed SE-0436.

Like before, it is crucial that we perform these generalizations in a manner that does not break existing Swift code nor binaries.

Note that fully generalizing the core Standard Library will take multiple proposals; this document is merely the second in a series. This draft proposal includes changes that I feel we will be ready to ship soon. I'd like if we could focus the discussion mostly around the specific changes proposed.

Some obvious and highly desirable generalizations still remain on the to do list. (This includes tasks such as starting to generalize standard protocols like Equatable, Comparable or CustomStringConvertible, and allowing pointer types to address nonescapable values.) We're working on unblocking these tasks, and I expect we'll address them in followup proposals, as soon as possible.

32 Likes

We can also generalize the convenient get() function, which is roughly equivalent to optional unwrapping:

extension Result where Success: ~Copyable & ~Escapable {
  @lifetime(self)
  consuming func get() throws(Failure) -> Success
}

In the non-escapable case, this function returns a value with a lifetime that precisely matches the original Result.

***FIXME: Is consuming expected to cause any problems? (E.g., what if the Result is a noncopyable+nonescapable variant that was yielded by a coroutine?)

Maybe it is time to add a property to Result that is similar to Task<>.value which could have one of the future accessors (I guess that would be borrow?) to allow non-copyable types that do not consume self. Something like this:

extension Result where Success: ~Copyable & ~Escapable {
  @lifetime(self)
  var value: Success {
    borrow throws(Failure)
  }
}
1 Like

Would make sense to harmonize with Task's API surface here, IMHO.

1 Like

That FIXME was just a reminder to self to double check we have the correct behavior. (We do! Noncopyable types yielded from a coroutine cannot be consumed.)

I agree it would be useful to have a property for this with a borrowing getter. (Or, ideally, with pair of getters, one borrowing and one consuming.) IIIRC, get() was defined before Swift started supporting throwing from property getters.

We cannot define a property like that though: (1) we do not have borrow accessors yet, and (2) computed properties with noncopyable results cannot currently throw. I believe we'll first need to resolve these; in the meantime, I think this generalization of get() is the right move.

For now, we are able to manually borrow the payload using a pattern match binding in a borrowing switch:

switch result {
  case .success(let value):
    // `value` is borrowed here
}
3 Likes

Would it make sense to spell extendLifetime(_:) as extendLifetime(of:) to align with type(of:), or as extendLifetime(ofValue:) to align with MemoryLayout.size(ofValue:)?

5 Likes

I see, thanks! Very happy with the proposal as is otherwise. I'm really looking forward to start using non-escapable types!

I just noticed the draft pitch includes an extension to ObjectIdentifier, but does not extend the == and != operators for metatypes. Assuming it is technically feasible, I don't see any reason not to update them too.

2 Likes

Yes! If we do one, ideally we should do the other too. Alas, I don't think we can do that quite yet. The reasons have to do with frustratingly technical minutia, but I'll try to convey them.

The bit gotcha right now is that the existing definitions heavily rely on type existentials:

// Current definitions as of Swift 6:
extension ObjectIdentifier {
  public init(_ x: Any.Type) { ... }
}

public func == (t0: Any.Type?, t1: Any.Type?) -> Bool { ... }
public func != (t0: Any.Type?, t1: Any.Type?) -> Bool { ... }

Ideally we'd generalize these using generalized existential types:

// Desired generalizations:
extension ObjectIdentifier {
  public init(_ x: any (~Copyable & ~Escapable).Type) { ... }
}

public func == (
  left: (any (~Copyable & ~Escapable).Type)?, 
  right: (any (~Copyable & ~Escapable).Type)?
) -> Bool { ... }
public func != (
  left: (any (~Copyable & ~Escapable).Type)?, 
  right: (any (~Copyable & ~Escapable).Type)?
) -> Bool { ... }

Unfortunately, these any (~Copyable & ~Escapable).Type existentials aren't fully operational yet. (The syntax seems to work, but the overloads never trigger. The correct syntax results in an ill-formed .swiftinterface due to a small issue.) The pitch works around this by adding a generic initializer for ObjectIdentifier as a temporary measure:

extension ObjectIdentifier {
  public init(_ x: Any.Type) { ... }

  @_disfavoredOverload // Unstable hack to avoid ambiguities
  public init<T: ~Copyable & ~Escapable>(_ x: T.Type) { ... }
}

The generic isn't a perfect substitute for the original Any-based version, though, as it isn't dynamic enough:

func test1(_ t: Any.Type) {}
func test2<T>(_ t: T.Type) {}

let t: Any.Type = Bool.self
test1(t) // OK
test2(t) // error: generic parameter 'T' could not be inferred

So unfortunately we need to have both variants as a workaround, with extra hacks to avoid ambiguous expressions in cases where both variants would apply. (My expectation is that we would be able to merge the two variants once generalized existentials become a thing, without breaking newly written code.)

Unfortunately, applying the same workaround on the ==/!= functions would result in the addition of new generic overloads for infix operators:

public func == (t0: Any.Type?, t1: Any.Type?) -> Bool { ... }
public func != (t0: Any.Type?, t1: Any.Type?) -> Bool { ... }

@_disfavoredOverload
public func ==<
  T1: ~Copyable & ~Escapable,
  T2: ~Copyable & ~Escapable
> (left: T1.Type?, right: T2.Type?) -> Bool { ... }

@_disfavoredOverload
public func !=<
  T1: ~Copyable & ~Escapable,
  T2: ~Copyable & ~Escapable
> (left: T1.Type?, right: T2.Type?) -> Bool { ... }

This would be extremely messy! Adding new generic overloads to infix operators increases the complexity of overload resolution, and that inevitably leads to previously working code running into "expression too complex" errors -- an unacceptable outcome.

The pitch therefore draws a line at the ObjectIdentifier initializer, deferring to generalize the metatype == operators until generalized existentials become usable.

We do have the option to delay generalizing ObjectIdentifier too, for consistency. (Or we could fix type existentials, of course -- but that would unnecessarily delay the rest of the proposal.)

(Edit: I corrected the syntax of generalized type existentials. (any ~Copyable & ~Escapable).Type is obviously very different from any ((~Copyable & ~Escapable).Type), and naturally we need the latter. any (~Copyable & ~Escapable).Type parses to this second interpretation. :face_with_spiral_eyes:)

2 Likes

Thanks for the explanation. That's disappointing! Please make sure to discuss this in the Future Directions section so folks know it wasn't just an oversight?

3 Likes

We can do one better -- @Slava_Pestov has kindly fixed the issue blocking the use of generalized existential metatypes in library evolution mode, so I put metatype comparisons back on the table.

Unfortunately, I've also discovered an issue with C++ interop where the generalization of Optional.unsafelyUnwrapped unexpectedly breaks the C++ interop overlay. I've removed this specific generalization for now.

I updated both the pitch and the implementation to incorporate these changes.

Keep these comments coming! They are useful.

4 Likes

Seems like something we should be able to fix. My inclination is to keep it in the proposal, since it's obvious we want to do it, and we can land that generalization along with its fix.

I'd love to see unsafeBitCast generalized in this proposal as well. It's something we're using within the macro that generates safe versions of unsafely-imported C/C++ APIs. Was there a technical reason for omitting it?

Doug

5 Likes

I’d like to suggest separating the ability to transform lifetimes out of unsafeBitCast. There’s a lot of regret in other languages surrounding almighty cast operations. If you mean to change ā€œjust oneā€ of the type or its lifetime, we shouldn’t make it easy to accidentally change both, even with unsafe operations. I think there’s also a decent argument to make that changing properties of a type orthogonal to its bit representation using a function called unsafeBitCast will not sound correct for some people. I understand that the lifetime is formally part of the type, but years of C and C++ have created a mental model where a ā€œtypeā€ is about operations and bit representation.

One factoid about xnu is that it enables -Werror=everything, which prevents removing const qualifiers from pointees, and it has a __DECONST(new-type, expr) macro to remove it without a diagnostic. The macro expands to something like (new-type)(intptr_t)(expr); in other words, it can actually cast anything that converts to an integer to any pointer type. Over time, this has led to the accumulation of a non-trivial number of misuses of __DECONST that actually wouldn’t build if it was implemented in a way that only allowed you to remove const qualifiers.

1 Like

Yep, during discussions we decided to defer the ~Escapable generalization of unsafeBitCast, based on @fclout's objection and our general inability to precisely reason about lifetimes in these API proposals.

I'm already quite alarmed about finding myself having to introduce the idea of immortal nil values and (implicit and explicit) enum wrapping/unwrapping semantics, without stable syntax or an accepted spec for such things. unsafeBitCast and generalized pointer types would conjure borrowing dependencies out of thin air; I don't think it would be a good idea to introduce such things this manner.

3 Likes