[Pitch] Distributed Actors

Hello everyone,

With the recent introduction of Swift concurrency, and most notably actors to the language, Swift gained powerful and foundational building blocks for expressing thread-safe concurrent programs. This proposal aims to extend Swift's actors with the ability to work equally well for distributed systems thanks to the introduction of distributed actors and location transparency associated with them.

  • The proposal is fairly long, so please continue reading the full version linked below.
    • You may skip reading the "appendix" section, it is not necessary to read unless one wants to understand the deeper implementation details of the proposal.
  • We also include a sample application, showcasing this feature in action. We suggest giving it a try after reading the proposal. The provided transport implementation is only an example and is fairly incomplete, however the sample shows off all the interesting interactions with the language feature.

Distributed Actors

Introduction

Swift's actors are a relatively new building block for creating concurrent programs. The mutable state isolated by actors can only be interacted with by one task at any given time, eliminating a whole class of data races. But, the general actor model works equally well for distributed systems too.

This proposal introduces distributed actors, which are an extension of regular Swift actors that allows developers to take full advantage of the general actor model of computation. Distributed actors allow developers to scale their actor-based programs beyond a single process or node, without having to learn many new concepts.

Unlike regular actors, distributed actors take advantage of location transparency, which relies on a robust, sharable handle (or name) for each actor instance. These handles can reference actor instances located in a different program process, which may even be running on a different machine that is accessible over a network. After resolving an actor handle to a specific instance located somewhere, the basic interactions with the actor instance closely mirrors that of regular actors. Distributed actors abstract over the communication mechanism that enables location transparency, so that the implementation can be extended to different domains.

distributed actor Greeter { 
  distributed func greet(name: String) -> String {
    "Hello, \(name)!"
  }
}

Please continue reading in the full proposal.

Please post editorial changes, such as fixing typos or small mistakes should be pointed out in this swift-evolution PR.

In-depth discussions about transport implementation details are a huge topic on their own and likely to be better split off into their own dedicated threads. Thank you in advance for keeping this review focused on the language and runtime features.


Table of Contents

// edit 1: fix links a little bit, to well formatted proposal

60 Likes

This is super exciting! I’ve only given it a quick read, but one thing stands out already:

It feels like distributed actors are begging for typed throws. If Swift supported typed throws, in addition to the other restrictions on distributed methods, you could also have the restriction that any errors thrown must conform to Codable.

That would avoid the weird situation where a thrown error can’t be sent over the wire, or is unsafe to send over the wire. Assumptions around which errors are safe to send would then be clearly documented in the method’s signature: if it’s an error thrown from a distributed method, it’s going to be sent, and so it must be safe to send, just like arguments to distributed methods and return values from distributed methods must be safe to send.

(Personally, I’ve been skeptical of typed throws before, but this feels like an extremely compelling case for them.)

9 Likes

This proposal reads very nicely, and I love the possibilities that will be opened up once we get the functionality of distributed actors. I have a couple of open questions after giving it a full read:

Usage of Codable

Using Codable for serializing the message on the wire seems reasonable; however, I am wondering if the performance implications and usability of Codable will live up to the expectations of distributed actors. One thing, that comes to my mind, is that SwiftProtobuf intentionally did not adopt Codable because of various intricacies and the performance issues that come along with it. Since Protobuf is a very common wire format, I would love to get your opinion on that.

ActorIdentities with multiple replicas

In a system where we model a remote service as a DistributedActor and that service has multiple replicas running. Would the ActorIdentity be the same, or would that depend on the service discovery mechanism used in the end?

Polyglot communication

Today's distributed systems are often running on a number of different languages. To allow communication between these services, oftentimes communication protocols such as gRPC are deployed. Since DistributedActors are something very Swift, but the transport is nicely decoupled, do you think it is possible to use them also in communication with services of other languages? Maybe using the DistributedActors in the generated code of something like grpc-swift?

7 Likes

I agree that relying on Codable creates an unavoidable performance toll. I would foster a design that supports Codable as a convenience, and lets people who aim for the best performance provide efficient and customized decoding and encoding.

This was the strategy chosen by GRDB. Codable support is there, super handy, but people who look for the best performance can implement a much more efficient protocol. Practically speaking, database rows are decoded through the FetchableRecord protocol, which grants super-efficient raw access to SQLite, with great support for generics specialization. A default implementation of this protocol is ready-made for Decodable types. A similar mechanism exists for encoding.

19 Likes

I first thought that usage of @_dynamicReplacement would imply that we can load only one transport implementation per distributed actor type. But actually this does not need to be the case. Multiple transports can be supported by using chaining:

extension Greeter { 
  // TODO: introduce `distributed` specific annotation, e.g. @remoteReplacement(for:greet(_:))
  @_dynamicReplacement(for :_remote_greet(_:))
  nonisolated func _cluster_greet(_ greeting: String) async throws -> String { 
    // Step 0: Check if replacement is applicable
    guard let transport = self.transport as? KnownTransport else {
        // Forward to next transport's dynamic replacement
        return try await _remote_greet(greeting)
    }
    // Step 1: Form a message
    let message: _Message = _Message.greet(greeting)
    
    // Step 2: Dispatch the message to the underlying transport
    return try await transport.send(message, to: self.id)
  }
}

Would it be possible to send distributed actor references as existentials? Currently existentials cannot conform to protocols, including Codable.

Calling generic methods would require serialising types. Curious to see how it would look like. Are there any limitations on types used?

Was protocol-based solution considered? Where distributed is an attribute of a protocol, local actor conforms to that protocol, and generated code contains proxy classes conforming to that protocol, instead of @_dynamicReplacement?

NB:

2 Likes

Thanks @rjchatfield, as @xwu quoted, please direct such small edits at the PR directly -- it'll keep the noise out of the discussion thread :slight_smile:

Having that said, I'll apply those three changes right away -- thanks a lot for spotting them!

1 Like

I see multiple references to @_dynamicReplacement, but I'm struggling to find reliable documentation for it.

Should @_dynamicReplacement go through its own SE first? Or maybe the distributed actors proposal should give more context for it, explaining why that's underscored, or what are the implications of future changes to @_dynamicReplacement in the light of it not being standardized yet?

Same could be appropriate for @_marker attribute.

2 Likes

I remember seeing a discussion about this a while back Dynamic method replacement.

1 Like

I saw that too, and that's why I'm puzzled by the fact that this proposal depends on an underscored feature that could change without warning or disappear completely in future versions of the compiler.

1 Like

The proposal has a comment about this:

NOTE: The @_dynamicReplacement(for:) will be replaced with a distributed actor specific attribute. The proposal and use of dynamic member replacement reflects the current state of the implementation.

We are likely going to propose some form of @distributedRemoteImplementation(for:greet(_:)) in order to completely hide the existence of the _remote functions from users. The implementation mechanism would be the same, by making use of dynamic member replacement. This change will be introduced in an upcoming revision of the proposal.

4 Likes

Sorry, I somehow missed that. My worry about underscored attributes still applies to the use of @_marker though.

Since @_marker isn't (I believe) intended for use in user code, I think the situation is somewhat different. Implicit in the "underscored language features may be removed at any time" policy is that we won't break the source compatibility promise, so to the extent that any language/standard library features depend on a given attribute, a suitable replacement would have to be developed.

3 Likes

Thanks for the early replies and feedback everyone! I'll go "topic by topic":

Typed throws

On a personal note, and coming from years of work both with checked exceptions as well as actor runtimes, I really don't really think typed throws are as great as forums threads often make them out to be. They look nice in small examples, but in long running, evolving projects, they're more of a trap than help to be honest IMHO since they become part of the API and can never change; while error conditions or underlying transports and implementations do change all the time.

Ok, but typed-errors rant aside: you are correct, if Swift had typed errors this indeed would be possible to enforce that errors thrown by distributed functions must satisfy the Codable (or similar) requirement.

Do note however that signatures would become a union of YourError | ActorTransportError and be may be somewhat annoying to deal with...

Having that said, there are currently no plans for typed throws in Swift so this design cannot assume them.

Codable

Being an avid protocol buffers user for many years myself... we do have to realize though that the two are at odds here -- since both protocols (or someday maybe "service definitions") and protobufs Interface Definition Language serve the same purpose: to be the source of truth for the types involved. We are interested in making Swift protocols be the interface definitions, rather than using external languages to do so.

Related note: The reason for SwiftProtobuf not adopting Codable isn't just performance, it is because the entire model of working with those types is completely different - they're IDL driven and have their own quirks.

This is what the future direction Future Direction: Ability to customize parameter/return type requirements is all about. It's a relatively simple future feature and it may even just happen during the initial work on this feature.

It would enable specifying any other type requirement, other than Codable that parameters and returns are checked against. The example uses a "SuperSafe" enum of trusted values, but equally well one could imagine using ProtobufMessage there if one really wanted to.

ActorIdentities with multiple replicas

To answer your question about "actor is a service, has multiple replicas" -- they'd have different identities, because an identity must uniquely identity a specific actor.

Think of ActorIdentity similar to ObjectIdentifier however that can persist over time and process boundaries. This matters for example, a "long lived" and "well known" distributed actors which can be resolved a well-known identity such as /virtual/game/lobby and the system ensures there is always one such actor and it can always be resolved, regardless where it is physically located.

Typically though actor identities are more specific, for example following URL-ish patterns like this: sact://127.0.0.1:8888/user/worker#123545 where the last bit is the unique identity of it.

How specifically identities are used it up to the transports though -- some may choose to just use an integer (process IDs perhaps?) and perform mapping of those integers to actual instances.

This is why the identity is a protocol.

Polyglot communication

So while it is absolutely possible: you can easily implement a _remote function in a way that a remote other-language process may understand, this is not a goal of this work. We are aiming for simple and native feeling Swift actor systems that inter-operate the best with each-other.

In a server world it is best to think about this as best used within a single clustered service. You wouldn't really mix languages within the same system; going across languages can of course continue to be done with gRPC and HTTP and other tools. A similar approach has been taken by Akka and other actor runtimes in the past.

In a multi process world, we are interested in making xpc services and other cross process work on Apple platforms "feel great". Think how NSXPC works great in objective-c but feels pretty weird when used with Swift; this is a similar situation here, we intend to provide a solution that feels best for Swift, and it may be accessible to other languages, but it is not a primary goal of this work.

6 Likes

The use of _dynamicReplacement

As mentioned in the proposal, this will be replaced by some non-underscored attribute of similar capabilities.

We do not consider having to stabilize the dynamic member replacement feature in order to land this feature. Note that the existence of any dynamic functions is completely hidden from users already; it only leaks through this @_dynamicReplacement attribute; Our goal would be to not have users worry about how the remote function is "getting there", it may be the _dynamicMember replacement or some other underlying mechanism, it doesn't really matter as far as transport implementors are concerned.

Another future direction which impacts this is Synthesis of _remote and well-defined Envelope<Message> functions because if we were able to get away with not having to rely on the source generation to fill in _remote functions, there would not be any need for those replacements, as a transport could completely implement it all within library code.

This is a very tricky problem though, and it'd likely both need Variadic Generics as well as some more general and powerful meta-programming tools to make this a reality. There are also concerns about forming a "general message type" as we don't know yet about some of the specific requirements of a few transports we are investigating. We don't want to lock in this part of the design into the language unless we are 100% confident it would support all future transport implementations - we are working on figuring this out though.

4 Likes

I am still digesting the pitch and hope to come with more feedback later, but one short comment:

Being an avid protocol buffers user for many years myself... we do have to realize though that the two are at odds here -- since both protocols (or someday maybe "service definitions") and protobufs Interface Definition Language serve the same purpose: to be the source of truth for the types involved. We are interested in making Swift protocols be the interface definitions, rather than using external languages to do so.

This is super nice if realised, not having to keep a separate interface definition and all that entails in codegen and interface would be very convenient.

I do share the concern of @gwendal.roue about performance though, I'd echo it'd be nicer to formulate a more generic approach where Codable is a convenience. Looking at the future direction of customised parameter/return types, maybe it should be surfaced such that Codable needs to be specified also in the common case?

Also, a related concern - have there been thoughts of how many copies/transformations that will be needed from the raw transport data read until it's accessible by the distributed actor? For some use cases (e.g. distribution of small(ish) structs with a handful of plain data types, or small vectors of such) this will be compared to non serialisable format that one can access similar to Cap'n'proto, SBE or Flatbuffers - where it is possible to have fundamentally zero copies/transformations if done right. It'd be great to consider such use cases too.

Overall, I'm very glad to see this work in progress, just want to ensure we'll be able to use it in practice :-) .

3 Likes

One thing that strikes me as odd is the ActorTransport parameter to the actor initializer. Wouldn’t it be better to also require a specific parameter name? That would be consistent with property wrappers where wrappedValue and projectedValue parameters? That would also enable some syntax sugar (just like property wrappers) to make it more idiomatic.

3 Likes

I think the Actor protocol should be renamed LocalActor if this is to be accepted. Having Actor inherit from AnyActor is confusing, and none of these names have been formally released yet anyway.

Swift 5.5 likely ships by the end of the month, so it's pretty well locked now.

I feel like this is going to be a common programming error forever if it isn’t changed now, though.

Then again, I suppose StringProtocol already has this problem. Maybe just name it ActorProtocol?