RediStack future plans

Server
,
1

RediStack future plans

Joannis Orlandos (@Joannis) and I (@fabianfett) have taken over the maintenance for RediStack from Nathan Harris (@mordil), as described in the RediStack maintainer update posted on the Swift forums. Joannis and I want to thank Nathan again, for creating a fantastic library that is an important building block for the Swift on server ecosystem. We hope to keep and improve its quality and feature set into the future.

In this post we want to layout our plan for RediStack’s future.

Hosting RediStack

Nathan had hosted the RediStack repository at GitLab. As part of taking over the maintenance for the repository, we moved active development of RediStack to GitHub. It lives in the swift-server organization.

We have setup automation to keep the existing GitLab repository in the swift-server-community organization up to date. The previous GitHub mirror forwards to the new repository https://github.com/swift-server/RediStack.

Versions and branches

Version 1.0 was released in October 2020. After 1.1 in November 2020, a release/1.x branch was created of master around December 2020, which has been the source branch for all 1.x releases after 1.1.

After creating the release/1.x branch, breaking changes were made on master and work towards a RediStack 2.0 release was well underway. However, even though there were multiple 2.0.0-alpha, beta, experimental and gamma releases a stable 2.0 has not been shipped. This means that the current master branch contains commits, that are up to two and half years old, that were never shipped in a stable release.

After having gone through all the commits on the master branch, we think we can apply the added features in a non-breaking fashion. While some features add new API surface or change existing one, we think that we can work with deprecations and new types instead.

Based on this, Joannis and I agreed to proceed as follows:

  1. We will not make any further commits to the master branch
  2. The current release/1.x branch has been renamed to main and has become the new default branch.
  3. We will try to cherry pick features, that are only on the master branch, into main and release them in a non breaking manner.

This means that we will embrace improving the released stable 1.x instead of releasing a breaking version 2.0 soon. We will be building out new functionality next to existing functionality and guiding users to the new implementations by using deprecation warnings.

I have experience with this procedure as PostgresNIO has used a similar approach in the last years.

Eventually we will release a breaking version 3.0 that will remove all deprecated APIs.

I want to emphasize that this means that version 2.0 will not become stable. If you are currently using a pre 2.0 release, we advise to migrate back to 1.x. If you depend on features currently not present in 1.x please reach out (by creating an issue) and inform us which features you rely on, so that we can ensure those are ported back to 1.0 soon.

Development areas

Initially we want to focus our RediStack efforts on three major areas:

  • Structured concurrency
  • Typesafe Redis commands
  • Support for Redis clustering

Structured concurrency

We want to support structured concurrency. This means that all new APIs should support async/await and eventually cancellation. While adding async/await APIs is quite simple, thanks to NIOs concurrency helpers, supporting cancellation will be more involved.

Early API drafts:

  • Sending commands:
let response = try await client.get("bar")
  • Supporting pub/sub using AsyncSequences:
for try await event in client.subscribe("foo") {
  // do something with the received event. Automatically cancel the subscription, once the loop is exited
}

Typesafe Redis commands

RediStack currently has a generic RedisCommand struct, used for sending commands to the Redis server.

public struct RedisCommand {
  /// An encoded Redis Message in the RESP array encoding
  public let message: RESPValue
  /// A promise to be fulfilled with the commands response 
  public let responsePromise: EventLoopPromise<RESPValue>

  public init(message: RESPValue, responsePromise promise: EventLoopPromise<RESPValue>)
}

This has been great abstraction that served RediStack well for many years. Sadly though we lose lots information as soon as we encode a command into a RedisCommand. For example we need to iterate the message RESPValue array to learn which Key a command was sent for.

Because of this, we want to introduce a new Command protocol that abstracts the commands that are sent to Redis. The idea is that each Redis command can get its own Swift type, which then conforms to the Command protocol. This way we can preserve more Command information in properties and don't need to parse already encoded information.

/// A redis response that is decoded from a `RESPValue`. We use this protocol to create typesafe responses
/// to queries.
public protocol CommandResponse {
  /// Decode the type from a `RESPValue`.
  /// - Parameters:
  ///   - respValue: The RESPValue to decode the type from
  init(respValue: RESPValue) throws
}

/// A redis command that can be executed on a connection.
public protocol Command {
  /// The response type. Must be decoded from a `RESPValue`.
  associatedtype Response: CommandResponse

  // encode the command to the wire protocol
  func encode(_ into: RESPArrayEncoder)
}

RediStack will implement the most common commands directly. But we want to note that full coverage of all RediStack commands is currently a non-goal for us. All required protocols will be public, so that users can easily implement the missing commands for themselves. (Contributions with new Command types are of course always welcome.)

Support Redis Clustering

We want to introduce a new RedisClusterClient that supports sending messages to a sharded Redis cluster. For this we will need to extend the proposed Command protocol:

protocol ClusterCommand: Command {
  var key: String { get } // key to compute the shard to connect to
}

We will then introduce a RedisClusterClient that will support node discovery by using swift-service-discovery.

Other smaller features on our list:

Get involved

We are excited about the work ahead. As always, we welcome contributions in all shapes and forms. If you are interested in helping please reach out.

28 Likes
2

This is fantastic news! :partying_face:

4 Likes