[Discussion] MongoDB Swift driver

MongoDB Swift Driver

Package Description

MongoDB driver for Swift.

Package Name mongo-swift-driver
Module Name MongoSwift
Proposed Maturity Level Sandbox
License Apache 2.0
Dependencies libmongoc 1.15.3 (vendored), SwiftNIO 2, Nimble 8 (for tests)

Introduction

mongo-swift-driver is a client library for using MongoDB from Swift. Its module MongoSwift provides a SwiftNIO-based asynchronous API for interacting with the database. The core type, MongoClient, maintains a pool of connections to servers in a MongoDB deployment and provides an interface for querying, inserting, and updating data in the deployment. In addition, the client handles authentication, TLS, topology monitoring, and automatically retrying failed commands. The driver also contains a BSON implementation allowing users to create and manipulate MongoDB documents and to convert between documents and native Swift types.

Motivation

This driver represents an official effort by MongoDB itself to provide first-class support for using the database from server-side Swift applications. It is developed by the team that writes the official MongoDB drivers for several programming languages and is written in accordance with the official driver specifications, meaning it provides a user experience consistent with what developers who have worked with other MongoDB in other languages expect.

Proposed solution

The package mongo-swift-driver contains two modules, MongoSwift and MongoSwiftSync. MongoSwift contains an asynchronous, SwiftNIO-based API, and MongoSwiftSync contains a synchronous wrapper of the asynchronous API.

The driver depends on a vendored copy of the MongoDB C driver, libmongoc. As libmongoc is a synchronous driver, the asynchronous API is implemented by running all blocking code within a SwiftNIO NIOThreadPool.

The main entry point to the API is a MongoClient, which handles interactions with a single MongoDB deployment. Clients are thread-safe and automatically pool connections to the deployment's server(s) under the hood. The client also automatically discovers the full topology of the deployment and dynamically tracks its state over time. For most use cases, sharing a single MongoClient for an entire application should be sufficient.

Child MongoDatabases and MongoCollections are retrieved from MongoClients, and provide APIs to perform CRUD and various other operations on corresponding databases and collections in the MongoDB deployment.

MongoDB stores data in a format called BSON, i.e. binary JSON. To support working with this format, the driver also contains a BSON library. This includes a Document type corresponding to a MongoDB document, as well as a BSONEncoder and BSONDecoder to support conversion back and forth between Codable Swift types and Documents.

The driver works with MongoDB 3.6+ and Swift 5.0+. It is tested against all supported MongoDB and Swift versions on both macOS 10.14 and Ubuntu 18.04, as well as a variety of MongoDB topologies (standalone server, replica set, sharded cluster, TLS and authentication on/off).

This library has been primarily developed by the three authors of this proposal. Kaitlin and Matt began developing it about two years ago, and Patrick has been working on the project for about a year. The library was initially developed with a synchronous API to support use of the mobile/embedded version of MongoDB on iOS, but over time its focus and primary intended use case has shifted toward server-side Swift usage.

We've recently tagged a release candidate for an upcoming 1.0 release, which we plan to release once we've received and incorporated feedback from the community. The long term-plan following 1.0 is to:

  1. Catch up on MongoDB features introduced in recent server versions. While the driver works with all versions of MongoDB 3.6+, it lacks APIs for some newer features such as transactions.
  2. Decouple the BSON library from the driver, pulling it out into a separate package and repository.
  3. Rewrite the BSON library and driver internals in pure Swift, culminating in a 2.0 release.

Detailed design

Each MongoClient is backed by:

  • A libmongoc mongoc_client_pool_t, which can be thought of as a pool of connections to the MongoDB deployment. This is wrapped in an internal type called ConnectionPool which supports checking Connections in and out.
  • A SwiftNIO NIOThreadPool, whose size may be customized by the user at the time of client creation.
  • A SwiftNIO EventLoopGroup, provided by the user at the time of client creation.

Each I/O-performing method on MongoClient as well as its child objects (MongoDatabase, MongoCollection) has an internal Operation type which encapsulates all of the blocking code and has an execute(using conn: Connection) method.

Whenever an I/O-performing API method is called on MongoClient or one of its child MongoDatabase/MongoCollections, we call the NIOThreadPool's runIfActive method to generate the future we return (this is a simplification, but demonstrates the general idea):

return threadPool.runIfActive(eventLoop: elg.next()) {
    let connection = connectionPool.checkOut()
    defer { connectionPool.checkIn(connection) }
    return try operation.execute(using: connection)
}

The driver has a very large API surface, which we have not included in full for the sake of brevity. Instead, we provide a representative sample of the API. The full API may be viewed on our documentation website.

A large portion of our API is Swift translations of APIs defined in MongoDB specifications such as CRUD, Change Streams, Driver Sessions, Index Management, Enumerate Indexes, Enumerate Databases, Enumerate Collections, etc.

MongoClient

/// A MongoDB Client providing an asynchronous, SwiftNIO-based API.
public class MongoClient {
    /**
     * Create a new client for a MongoDB deployment. For options that included in both the connection string URI
     * and the ClientOptions struct, the final value is set in descending order of priority: the value specified in
     * ClientOptions (if non-nil), the value specified in the URI, or the default value if both are unset. This client
     * will lazily establish connections to the MongoDB deployment the first time it is used for I/O.
     *
     * - Parameters:
     *   - connectionString: the connection string to connect to.
     *   - eventLoopGroup: A SwiftNIO `EventLoopGroup` which the client will use for executing operations. It is the
     *                     user's responsibility to ensure the group remains active for as long as the client does, and
     *                     to ensure the group is properly shut down when it is no longer in use.
     *   - options: optional `ClientOptions` to use for this client
     *
     * - SeeAlso: https://docs.mongodb.com/manual/reference/connection-string/
     *
     * - Throws:
     *   - A `InvalidArgumentError` if the connection string passed in is improperly formatted.
     */
    public init(
        _ connectionString: String = "mongodb://localhost:27017",
        using eventLoopGroup: EventLoopGroup,
        options: ClientOptions? = nil
    ) throws

    /**
     * Shuts this `MongoClient` down, closing all connections to the server and cleaning up internal state.
     *
     * Call this method exactly once when you are finished using the client. You must ensure that all operations
     * using the client have completed before calling this.
     * 
     * The returned future must be fulfilled before the `EventLoopGroup` provided to this client's constructor
     * is shut down.
     */
    public func shutdown() -> EventLoopFuture<Void>

    /**
     * Shuts this `MongoClient` down in a blocking fashion, closing all connections to the server and cleaning up
     * internal state.
     *
     * Call this method exactly once when you are finished using the client. You must ensure that all operations
     * using the client have completed before calling this.
     *
     * This method must complete before the `EventLoopGroup` provided to this client's constructor is shut down.
     */
    public func syncShutdown()

    /**
     * Gets a `MongoDatabase` instance for the given database name. If an option is not specified in the optional
     * `DatabaseOptions` param, the database will inherit the value from the parent client or the default if
     * the client’s option is not set. To override an option inherited from the client (e.g. a read concern) with the
     * default value, it must be explicitly specified in the options param (e.g. ReadConcern(), not nil).
     *
     * - Parameters:
     *   - name: the name of the database to retrieve
     *   - options: Optional `DatabaseOptions` to use for the retrieved database
     *
     * - Returns: a `MongoDatabase` corresponding to the provided database name.
     */
    public func db(_ name: String, options: DatabaseOptions? = nil) -> MongoDatabase
}

MongoDatabase

MongoDatabases are instantiated via the MongoClient.db method.

/// A MongoDB Database.
public struct MongoDatabase {
    /**
     * Access a collection within this database. If an option is not specified in the `CollectionOptions` param, the
     * collection will inherit the value from the parent database or the default if the db's option is not set.
     * To override an option inherited from the db (e.g. a read concern) with the default value, it must be explicitly
     * specified in the options param (e.g. ReadConcern(), not nil).
     *
     * - Parameters:
     *   - name: the name of the collection to get
     *   - options: options to set on the returned collection
     *
     * - Returns: the requested `MongoCollection<Document>`
     */
    public func collection(_ name: String, options: CollectionOptions? = nil) -> MongoCollection<Document>

    /**
     * Access a collection within this database, and associates the specified `Codable` type `T` with the
     * returned `MongoCollection`. This association only exists in the context of this particular
     * `MongoCollection` instance. If an option is not specified in the `CollectionOptions` param, the
     * collection will inherit the value from the parent database or the default if the db's option is not set.
     * To override an option inherited from the db (e.g. a read concern) with the default value, it must be explicitly
     * specified in the options param (e.g. ReadConcern(), not nil).
     *
     * - Parameters:
     *   - name: the name of the collection to get
     *   - options: options to set on the returned collection
     *
     * - Returns: the requested `MongoCollection<T>`
     */
    public func collection<T: Codable>(
        _ name: String,
        withType _: T.Type,
        options: CollectionOptions? = nil
    ) -> MongoCollection<T>
    
    /**
     * Issues a MongoDB command against this database.
     *
     * - Parameters:
     *   - command: a `Document` containing the command to issue against the database
     *   - options: Optional `RunCommandOptions` to use when executing this command
     *   - session: Optional `ClientSession` to use when executing this command
     *
     * - Returns:
     *    An `EventLoopFuture<Document>`. On success, contains the server response to the command.
     *
     *    If the future fails, the error is likely one of the following:
     *    - `InvalidArgumentError` if `requests` is empty.
     *    - `LogicError` if the provided session is inactive.
     *    - `LogicError` if this databases's parent client has already been closed.
     *    - `WriteError` if any error occurs while the command was performing a write.
     *    - `CommandError` if an error occurs that prevents the command from being performed.
     *    - `EncodingError` if an error occurs while encoding the options to BSON.
     */
    public func runCommand(
        _ command: Document,
        options: RunCommandOptions? = nil,
        session: ClientSession? = nil
    ) -> EventLoopFuture<Document>
}

MongoCollection

MongoCollections are initialized by calling one of the collection or createCollection methods on MongoDatabase. They are generic over a Codable type T. This type can be a BSON Document or a Codable type that matches up with the document structure in the server's corresponding collection.

/// A MongoDB collection.
public struct MongoCollection<T: Codable> {
    /**
     * Finds the documents in this collection which match the provided filter.
     *
     * - Parameters:
     *   - filter: A `Document` that should match the query
     *   - options: Optional `FindOptions` to use when executing the command
     *   - session: Optional `ClientSession` to use when executing this command
     *
     * - Returns:
     *    An `EventLoopFuture<MongoCursor<CollectionType>`. On success, contains a cursor over the resulting documents.
     *
     *    If the future fails, the error is likely one of the following:
     *    - `InvalidArgumentError` if the options passed are an invalid combination.
     *    - `LogicError` if the provided session is inactive.
     *    - `LogicError` if this collection's parent client has already been closed.
     *    - `EncodingError` if an error occurs while encoding the options to BSON.
     */
    public func find(
        _ filter: Document = [:],
        options: FindOptions? = nil,
        session: ClientSession? = nil
    ) -> EventLoopFuture<MongoCursor<CollectionType>>

    /**
     * Counts the number of documents in this collection matching the provided filter. Note that an empty filter will
     * force a scan of the entire collection. For a fast count of the total documents in a collection see
     * `estimatedDocumentCount`.
     *
     * - Parameters:
     *   - filter: a `Document`, the filter that documents must match in order to be counted
     *   - options: Optional `CountDocumentsOptions` to use when executing the command
     *   - session: Optional `ClientSession` to use when executing this command
     *
     * - Returns:
     *    An `EventLoopFuture<Int>`. On success, contains the count of the documents that matched the filter.
     *
     *    If the future fails, the error is likely one of the following:
     *    - `CommandError` if an error occurs that prevents the command from executing.
     *    - `InvalidArgumentError` if the options passed in form an invalid combination.
     *    - `LogicError` if the provided session is inactive.
     *    - `LogicError` if this collection's parent client has already been closed.
     *    - `EncodingError` if an error occurs while encoding the options to BSON.
     */
    public func countDocuments(
        _ filter: Document = [:],
        options: CountDocumentsOptions? = nil,
        session: ClientSession? = nil
    ) -> EventLoopFuture<Int>

    /**
     * Encodes the provided value to BSON and inserts it. If the value is missing an identifier, one will be
     * generated for it.
     *
     * - Parameters:
     *   - value: A `CollectionType` value to encode and insert
     *   - options: Optional `InsertOneOptions` to use when executing the command
     *   - session: Optional `ClientSession` to use when executing this command
     *
     * - Returns:
     *    An `EventLoopFuture<InsertOneResult?>`. On success, contains the result of performing the insert, or contains
     *    `nil` if the write concern is unacknowledged.
     *
     *    If the future fails, the error is likely one of the following:
     *    - `WriteError` if an error occurs while performing the command.
     *    - `CommandError` if an error occurs that prevents the command from executing.
     *    - `InvalidArgumentError` if the options passed in form an invalid combination.
     *    - `LogicError` if the provided session is inactive.
     *    - `LogicError` if this collection's parent client has already been closed.
     *    - `EncodingError` if an error occurs while encoding the `CollectionType` to BSON.
     */
    public func insertOne(
        _ value: CollectionType,
        options: InsertOneOptions? = nil,
        session: ClientSession? = nil
    ) -> EventLoopFuture<InsertOneResult?>
}

MongoCursor

MongoCursors are initialized via API methods that return them - for example, MongoCollection.find.

MongoCursor is generic over a Codable type T , which varies depending on the particular API method that returned the cursor. For example, MongoCollection<CollectionType>.find returns a MongoCursor<CollectionType>, while MongoDatabase.listCollections returns a MongoCursor<CollectionSpecification>.

public class MongoCursor<T: Codable> {
    /**
     * Indicates whether this cursor has the potential to return more data.
     *
     * This method is mainly useful if this cursor is tailable, since in that case `tryNext` may return more results
     * even after returning `nil`.
     *
     * If this cursor is non-tailable, it will always be dead as soon as either `tryNext` returns `nil` or an error.
     *
     * This cursor will be dead as soon as `next` returns `nil` or an error, regardless of the `CursorType`.
     */
    public func isAlive() -> EventLoopFuture<Bool>

    /**
     * Attempt to get the next `T` from the cursor, returning `nil` if there are no results.
     *
     * If this cursor is tailable, this method may be called repeatedly while `isAlive` is true to retrieve new data.
     *
     * If this cursor is a tailable await cursor, it will wait for results server side for a maximum of `maxAwaitTimeMS`
     * before evaluating to `nil`. This option can be configured via options passed to the method that created this
     * cursor (e.g. the `maxAwaitTimeMS` option on the `FindOptions` passed to `find`).
     *
     * Note: You *must not* call any cursor methods besides `kill` and `isAlive` while the future returned from this
     * method is unresolved. Doing so will result in undefined behavior.
     *
     * - Returns:
     *    An `EventLoopFuture<T?>` containing the next `T` in this cursor, an error if one occurred, or `nil` if
     *    there was no data.
     *
     *    If the future evaluates to an error, it is likely one of the following:
     *      - `CommandError` if an error occurs while fetching more results from the server.
     *      - `LogicError` if this function is called after the cursor has died.
     *      - `LogicError` if this function is called and the session associated with this cursor is inactive.
     *      - `LogicError` if this cursor's parent client has already been closed.
     *      - `DecodingError` if an error occurs decoding the server's response.
     */
    public func tryNext() -> EventLoopFuture<T?>

    /**
     * Get the next `T` from the cursor.
     *
     * If this cursor is tailable, this method will continue polling until a non-empty batch is returned from the server
     * or the cursor is closed.
     *
     * A thread from the driver's internal thread pool will be occupied until the returned future is completed, so
     * performance degradation is possible if the number of polling cursors is too close to the total number of threads
     * in the thread pool. To configure the total number of threads in the pool, set the `ClientOptions.threadPoolSize`
     * option during client creation.
     *
     * Note: You *must not* call any cursor methods besides `kill` and `isAlive` while the future returned from this
     * method is unresolved. Doing so will result in undefined behavior.
     *
     * - Returns:
     *   An `EventLoopFuture<T?>` evaluating to the next `T` in this cursor, or `nil` if the cursor is exhausted. If
     *   the underlying cursor is tailable, the future will not resolve until data is returned (potentially after
     *   multiple requests to the server), the cursor is closed, or an error occurs.
     *
     *   If the future fails, the error is likely one of the following:
     *     - `CommandError` if an error occurs while fetching more results from the server.
     *     - `LogicError` if this function is called after the cursor has died.
     *     - `LogicError` if this function is called and the session associated with this cursor is inactive.
     *     - `DecodingError` if an error occurs decoding the server's response.
     */
    public func next() -> EventLoopFuture<T?>

    /**
     * Consolidate the currently available results of the cursor into an array of type `T`.
     *
     * If this cursor is not tailable, this method will exhaust it.
     *
     * If this cursor is tailable, `toArray` will only fetch the currently available results, and it
     * may return more data if it is called again while the cursor is still alive.
     *
     * Note: You *must not* call any cursor methods besides `kill` and `isAlive` while the future returned from this
     * method is unresolved. Doing so will result in undefined behavior.
     *
     * - Returns:
     *    An `EventLoopFuture<[T]>` evaluating to the results currently available in this cursor, or an error.
     *
     *    If the future evaluates to an error, that error is likely one of the following:
     *      - `CommandError` if an error occurs while fetching more results from the server.
     *      - `LogicError` if this function is called after the cursor has died.
     *      - `LogicError` if this function is called and the session associated with this cursor is inactive.
     *      - `DecodingError` if an error occurs decoding the server's responses.
     */
    public func toArray() -> EventLoopFuture<[T]>

    /**
     * Calls the provided closure with each element in the cursor.
     *
     * If the cursor is not tailable, this method will exhaust it, calling the closure with every document.
     *
     * If the cursor is tailable, the method will call the closure with each new document as it arrives.
     *
     * A thread from the driver's internal thread pool will be occupied until the returned future is completed, so
     * performance degradation is possible if the number of polling cursors is too close to the total number of threads
     * in the thread pool. To configure the total number of threads in the pool, set the `ClientOptions.threadPoolSize`
     * option during client creation.
     *
     * Note: You *must not* call any cursor methods besides `kill` and `isAlive` while the future returned from this
     * method is unresolved. Doing so will result in undefined behavior.
     *
     * - Returns:
     *     An `EventLoopFuture<Void>` which will succeed when the end of the cursor is reached, or in the case of a
     *     tailable cursor, when the cursor is killed via `kill`.
     *
     *     If the future evaluates to an error, that error is likely one of the following:
     *     - `CommandError` if an error occurs while fetching more results from the server.
     *     - `LogicError` if this function is called after the cursor has died.
     *     - `LogicError` if this function is called and the session associated with this cursor is inactive.
     *     - `DecodingError` if an error occurs decoding the server's responses.
     */
    public func forEach(_ body: @escaping (T) throws -> Void) -> EventLoopFuture<Void>

    /**
     * Kill this cursor.
     *
     * This method MUST be called before this cursor goes out of scope to prevent leaking resources.
     * This method may be called even if there are unresolved futures created from other `MongoCursor` methods.
     *
     * - Returns:
     *   An `EventLoopFuture` that evaluates when the cursor has completed closing. This future should not fail.
     */
    public func kill() -> EventLoopFuture<Void>
}

Errors

All driver errors are marked by their conformance to an empty protocol MongoError. This protocol is conformed to by:

  • protocol ServerError, encompassing errors that occur in the database and are returned to the driver in a command response. Conformed to by:
    • struct CommandError: occurs when a command encounters an error on the server side that prevents execution (e.g. due to the server being unable to parse a command)
    • struct WriteError: occurs when a single write fails on the server
    • struct BulkWriteError: occurs when a bulk write fails on the server
  • protocol UserError, encompassing errors caused by incorrect usage of the driver. Conformed to by:
    • struct LogicError: occurs when the user makes a logical error, e.g. advancing a cursor that has already been killed
      * struct InvalidArgumentError: occurs when the user provides an invalid argument e.g. an empty array is passed to bulkWrite.
  • protocol RuntimeError, encompassing unexpected errors occurring at runtime that do not fit neatly into either of the above protocols. Conformed to by:
    • struct InternalError: occurs when e.g. something is nil that should never be nil, or the server sends a response the driver cannot understand. This generally indicates an error in the driver or a system failure (e.g. a memory allocation failure).
    • struct ConnectionError: occurs due to connection establishment / socket-related errors.
    • struct AuthenticationError: occurs when the driver isn't authorized to perform a command, e.g. due to invalid credentials.
    • struct ServerSelectionError: occurs when the driver is unable to select a server for an operation, e.g. due to reaching a timeout or an unsatisfiable read preference.

The driver's BSON library also uses EncodingError and DecodingError from the standard library when appropriate, and will propagate any errors encountered in its use of SwiftNIO to the user.

Why not use enums?

It's common to use enumerations to model errors in Swift, and in fact we did this in the driver until not long ago. We switched to the protocol and struct-based approach recently, after realizing enums are a poor fit for error types that evolve over time:

  1. Adding an additional associated value to an enum case is a breaking change for anyone who is writing case let statements to match that case and its values.
  2. Adding a new case to an enum is a breaking change, as a user's previously exhaustive switch statement will no longer compile. (If non-frozen enums were available for general use this wouldn't be an issue, but for now we can't use them.)

Over time, the MongoDB server has added more and more information to the errors it returns, and has added various new categories of errors. For example, MongoDB 4.0 introduced the concept of "error labels", containing labels we expose to the user when present. Using protocols and structs gives us the ability to change our errors additively without breaking backwards compatibility.

Example Usage

import MongoSwift
import NIO

struct Cat: Codable {
    let name: String
    let color: String
    let _id: ObjectId
}

let elg = MultiThreadedEventLoopGroup(numberOfThreads: 4)
let client = try MongoClient("mongodb://localhost:27017", using: elg)
defer {
    client.syncShutdown()
    try? elg.syncShutdownGracefully()
}

let db = client.db("test")
let cats = db.collection("cats", withType: Cat.self)

let data = [
    Cat(name: "Chester", color: "tan", _id: ObjectId()),
    Cat(name: "Roscoe", color: "orange", _id: ObjectId())
]

// insert values into the collection
let insert = cats.insertMany(data)

insert.whenSuccess { result in
    print(result?.insertedCount) // prints 2
}

insert.whenFailure { error in
    if let err = error as? MongoError {
        switch err {
        case let runtimeError as RuntimeError:
            // handle RuntimeError
        case let serverError as ServerError:
            // handle ServerError
        case let userError as UserError:
            // handle UserError
        default:
            // currently would never get here, but will handle
            // any new types of MongoError added in the future
        }
    } else if let err = error as? DecodingError {
        // handle DecodingError
    } else {
        // handle any other error types
    }
}

// after insert create a cursor over the collection to read back the documents
let result = insert.flatMap { _ in
	cats.find()
}.flatMap { cursor in
	cursor.forEach { cat in
		print(cat) // prints out a `Cat` struct
	}
}

Maturity Justification

To our knowledge there is no significant production usage of the library; therefore, Sandbox is most appropriate.

Alternatives considered

An alternative solution would be to write a pure Swift driver from the ground up, rather than initially writing a driver on top of libmongoc. However, this would have significantly delayed the development of a stable, feature-complete API which developers can start to depend on while we implement pure Swift driver internals.

15 Likes

This is very exciting to see @kmahar! I've only done a quick read through the proposal thus far but it seems like the foundation is very solid and there's good work here.

1 Like

Just as an FYI; we are actually using an older version of the library in production using Vapor with several thousands users and have been for some time. We originally used MongoKitten but had some issues with it sustaining connections to Mongo Cloud. After moving to the official driver we haven't had any issues at all.

I'm very much in favour of seeing this being pushed forward as the official mongo driver for swift. Great job to all involved :+1:t2:

5 Likes

Awesome to hear you've had such success @dlbuckley! We knew you were using it but didn't realize it was at that scale :slight_smile:

1 Like

Hi Kaitlin, thanks for your work on the official MongoDB driver and for your talk about it at try! Swift last year that tipped me off on the NIO implementation.

FYI we are about to complete the first iteration of a project that uses the driver in an AWS Lambda function, also thanks to the work of @fabianfett. It’s an internal project for R&D purposes but I think that also counts as “production use”.

You mentioned the C driver is synchronous, which the Swift driver circumvents by using an NIOThreadPool. Does that mean the driver essentially uses a connection per thread, currently? Would rewriting the internals in Swift, as you mention is a goal for 2.0, likely give the driver better performance characteristics due to “real” non-blocking IO? I would be interested to hear if you have an intuition how that would compare to the node.js implementation, performance-wise.

2 Likes

Hi @Geordie_J, there already is a non blocking MongoDB driver available. It's by @Joannis_Orlandos: https://github.com/OpenKitten/MongoKitten

I haven't worked with either of the implementations (I'm more of a DynamoDB guy :wink:) and can't find a performance comparison anywhere (which doesn't mean that there isn't one). Maybe you can run some tests and let everyone know?

Though I might assume that the performance gain shouldn't be too big within lambda (except when you are making a ton of request concurrently). But only tests will tell if that "feeling" is right.

Thanks, we initially tried MongoKitten but had crashes with it on Lambda. It’s probably because we were accidentally calling synchronous code in an EventLoop but the error reporting wouldn’t tell us anything about it.

The question about “real” NIO was more for use outside of Lambda, and out of interest. I have a pretty good grasp on what these concurrency primitives mean but haven’t fully understood how traditionally synchronous APIs interact with NIO / Futures in general.

@Geordie_J If you've got questions regarding using MongoKitten synchronously or experience unexpected problems in general, definitely do reach out. I'm happy to help.

1 Like

As I'm using MongoKitten for my current project, I'm now wondering, what will be the future of MongoKitten. It's cool, that there is now an official support, because as far as I remember, MongoKitten doesn't support all the stuff, that a MongoDB could do (what actually was never a problem for my project). @Joannis_Orlandos: Do you plan to stop working on MongoKitten? If so, I hope that all your hard work will now be used for replacing libmongoc. Would you suggest changing my project to the official driver now?

@kmahar: I will need to take a closer look to the API, but for the view sight I'm thinking about the fact, that you use protocols for errors and not enums. I personally would use enums and if there are new error types that breaks the API, I would make a change in the version number and bring it to a new major version (at least that is how I understood the version numbers in SwiftPM).

That is exciting to hear @Geordie_J! Please keep us posted on how it goes and let us know if you run into any questions or issues.

Our ConnectionPool is not quite an accurate mapping to a traditional connection pool, due to the way libmongoc designed pooling - each mongoc_client_t actually maintains one connection per host specified in the connection string. So in a standalone configuration, one of our Connections is truly one connection, but if you were connected to e.g. a 3-node replica set each Connection would be in actuality 3 connections.

Since the number of concurrent operations happening is capped by the number of threads in the NIOThreadPool, we should expect the number of Connections to max out around the number of threads in the NIOThreadPool. Though due to some libmongoc quirks types such as MongoCursor and ClientSession also hold onto their own Connections until they are killed/ended.

Yes, I definitely expect the performance to improve in a pure Swift implementation. The NIOThreadPool usage means we have to cross threads, which could have impact on performance. As I mentioned in the "alternatives considered" section above, we went with this approach so we could make sure we had a solid API before making structural changes to our existing work.

I don't really have a good sense of how our eventual perf will compare to the Node.js driver (which my co-author @mbroadst works on in addition to this driver :slightly_smiling_face:) but when the time comes we do have an official benchmarks spec that will come in handy.

1 Like

Thanks for your feedback! Based on our experience maintaining other MongoDB drivers, we've learned that users value the API stability of the drivers greatly, and that updating across major version releases of a driver can be quite difficult for a lot of organizations. Given how frequently both the MongoDB server and the driver require new error types, we decided that using a non-frozen enum would force us to make major releases too frequently and require our users to upgrade too often. By opting for a protocol-oriented error hierarchy, we allow ourselves to introduce new error types as necessary without worrying about API stability. If you haven't already, check out the section of the proposal that discusses this in more depth.

Another thing to note is that we document each public method that returns a future with the types of errors the future could fail with. This way, users can more easily discover which types of errors to catch and handle specifically and which ones they can probably ignore safely. For an example of such documentation, see the API docs for insertOne.

Edit: I forgot to introduce myself! I'm Patrick and I also work with @kmahar and @mbroadst at MongoDB on the Swift driver.

Hey @Lupurus, I'm not planning on stopping the development of MongoKitten, and I won't for the forseeable future. I am, however, looking to get more community involvement as a requirement for being a SSWG accepted project.

libmongoc is a shared core, and that provides a number of benefits, 90% of which is guaranteeing stability. They can focus all their company's effort on testing and improving the C driver with it having direct benefits to all language implementations. MongoKitten doesn't benefit from this, so I need to test it separately from their ecosystem and automated testing/tooling.

On the flip-side, there's downsides to using a shared C driver. One has been stability as well, initially, because C doesn't simply translate 1:1 to a (good) Swift API. The wrappers downside is that it needs to be both a C implementation and a Swift driver, so the API needs to be adapted to how C works. This ends up creating bottlenecks in the API design as well as performance.

With all due respect to the MongoDB team, but I wouldn't even consider using their blocking C driver if I had to start a MongoKitten driver from scratch. I've been in a solid fight for over four years to create great Swift APIs. I know it cannot be done as well if you built on top of a huge C codebase. Swifts language features just don't translate well with C, and it doesn't allow deep integration with Swift libraries such as the sswg logging, metrics, channels & event loops. Also performance wise there are bottlenecks in Swift as a language that prevents C libraries from achieving their full potential.

That being said I appreciate the effort for official support because it makes the Swift ecosystem as a whole directly supported for enterprise. If a company has a support contract with MongoDB, I bet they can at least expect some help as a part of that. With MongoKitten there's no way I can help as part of a MongoDB service contract.

Finally, MongoKitten does support all features MongoDB has to offer in its raw APIs. But I didn't create helpers for each edge case. This is something the MongoDB driver doesn't offer either. Missing helpers can be added in "feature" requests, although nothing stops you from adding a helper for some feature yourself. In the contrary, there are less limitations in a sense that the MongoCore and MongoKitten core can be used to built much different clients in usability/API. The API & socket layer in the C library are dead set, while if you wanted you could use MongoKitten together with a future SSH library built on NIO and use that to build a proxy that way.

1 Like

@Joannis_Orlandos: Thank you for your detailled answer. I really like MongoKitten (especially the fact, that it is fully Swift) and I also already made a pull request for a small helper (if you may remember ;)).

What I don't understand from the MongoDB-team: Why starting a new driver instead of helping at MongoKitten?

I remember your PR, although I don't remember what it was specifically :slight_smile: We discussed collaborating, but their demands in a cooperation weren't exactly cooperative. So under those demands it's not just them passing the opportunity.

Hi @Lupurus! Thanks for your interest in our proposal, I can help give some context here and hopefully answer your questions.

Background
The original purpose for MongoSwift was to provide a solid foundation for something we called MongoMobile, which was an embedded version of the MongoDB database running in memory on your iOS device (similar to sqlite). In order to accomplish this task in a very limited time we felt our best approach was to wrap the rock-solid existing libmongoc with a lightweight Swift wrapper. Since the embedded server was written in C++, and we already had a shim for libmongoc to speak directly to the embedded server, it seemed prudent to reuse that work to get something in users hands as quickly as possible. Language interoperability, in particular with C-like languages, is a core feature of the Swift language and so in a relatively short period of time we were able to develop a full featured, stable synchronous MongoDB driver for Swift, even though we only intended to make something useful for MongoMobile!

Around this time you may recall that MongoDB acquired Realm, so efforts on the MongoMobile project were rolled into that project. As a company, we still strongly believe in the promise of Server Side Swift, and so decided to press on with development of MongoSwift.

Approach
Using libmongoc gave us the confidence and security to focus our efforts completely on making the API as idiomatic as possible while providing all the features of a modern MongoDB driver. The 1.0 version of the driver achieves two major goals: API stability for a major version, and an asynchronous version of the driver. libmongoc is treated as an internal implementation detail, none of its API is leaked in our implementation.

What does this mean for users? Given that our goal post-1.0 is to migrate the internals of the driver away from libmongoc to pure Swift, you can safely adopt this version of the driver and gain progressive performance enhancements without needing to alter your code.

Performance
@Geordie_J asked about performance on AWS Lambda compared to the Node.js driver, so I ran a small benchmark this weekend attempting to simulate realistic load on a local REST api against an M10 instance hosted on MongoDB Atlas. This relatively unscientific benchmark shows that MongoSwift 1.0.0-rc0 is ~10% slower than the Node.js driver. We expect that a migration to using SwiftNIO directly will unlock new levels of performance, but considering we haven’t spent any time at all on performance tuning I would say this is a good start for us.

What I don't understand from the MongoDB-team: Why starting a new driver instead of helping at MongoKitten?

@kmahar touched on this when we discussed it during the pitch phase, but the short answer to this question is that years of experience have shown us that owning the driver is the best way we can provide a high quality user experience. The advantage of MongoDB owning the driver is in our expertise and commitment to supporting, enhancing, improving, and maintaining the driver consistently for years to come.

We are strongly committed to the ideals of the FOSS community, and we must balance that with the concerns of our users. We are happy to collaborate on components the driver uses (a DNS library was brought up as an example), or projects which use the driver (maybe a Fluent adapter), and we also need to track current server releases in an agile manner. Not being in control over our code base could become complicated, and risk jeopardizing user experience.

I hope this answers most of the outstanding questions. We’ve worked very hard on this release and are eager to hear technical feedback on our proposal.

3 Likes

I'm not experienced enough to estimate, what problems in the API design can be there, if you build up the API on a c-driver. Could you make an example?

Okay, that's surely a good reason. I just really feel sorry for Joannis, because he did a great work with MongoKitten and in the end, he may not have a chance against a bigger (and paid) team.

For me (as a private and so far non-relevant person) it may be a big plus, that the code of the MongoDB-team is perfectly documented, what really helps. I hope I will have soon time to make some tests, then I can give further feedback.

hey @kmahar would you please open a PR at https://github.com/swift-server/sswg/tree/master/proposals with the proposal markdown, the proposal number is #10 and the review manager is @tanner0101

1 Like

hey @kmahar only skimmed the proposal, and looking forward to exploring it in more details.

couple of outstanding questions:

  1. the initializer signature suggests you need to pass in an EventLoopGroup:
public init(
        _ connectionString: String = "mongodb://localhost:27017",
        using eventLoopGroup: EventLoopGroup,
        options: ClientOptions? = nil
    )

in other parts of the ecosystem we follow a pattern where you pass in an EventLoopGroupPovider to make this a bit easier in cases the user does not already have an ELG at hand. you may want to consider following the same pattern

  1. could the BSON module be useful outside the context of the mongo client? if so, is it worth extracting it to a separate library?
2 Likes

Yes I'll put together a PR shortly!

Thanks for this suggestion! We like the idea a lot and are happy to incorporate it into the API. (For anyone who would like to follow along with that change being implemented I've opened SWIFT-749.)

The library could be useful on its own, e.g. you might use it to convert between files containing raw BSON data and JSON files.

Moving this out into a separate package altogether is definitely a goal of ours, though as of now we've been planning to handle that separation post-1.0 as part of the pure Swift BSON rewrite (same public API, new internals) I alluded to in the proposal. Correct me if I'm wrong, but I believe we can do that in a non-breaking manner for driver users by having the driver re-export all the public types we've moved out.

I think the main reason we've held off on this so far is that, for the purpose of the standalone library efficiently interoperating with the driver, we'd need the BSON API to use several C types in its public API, which is something we've worked to avoid in our APIs.

For example, Document would need to expose a pointer to its backing bson_t so that MongoSwift methods accepting Documents would have a bson_t to pass to the corresponding libmongoc method, and would also need an initializer accepting a pointer to a bson_t, and so on. Right now these can be internal since the code lives in one module. In a pure Swift implementation Document would be wrapping a ByteBuffer which we'd be fine to expose in the public API.

1 Like

I think the main reason we've held off on this so far is that, for the purpose of the standalone library efficiently interoperating with the driver, we'd need the BSON API to use several C types in its public API, which is something we've worked to avoid in our APIs.

this makes sense, thanks for additional details

1 Like
Terms of Service

Privacy Policy

Cookie Policy