Officially supported MongoDB Driver

Introduction

Hi all! My name is Kaitlin Mahar, and I’m a software engineer at MongoDB on the drivers team. We build the official client libraries for using MongoDB in just about every popular programming language.

I work on our driver for Swift, MongoSwift. My team (@mbroadst, @patrick) and I would like to pitch the driver to SSWG and the community for consideration.

We’re currently working on adding an asynchronous SwiftNIO-based version of our API to meet the SSWG incubation process minimal requirements. This is still in the design phase, and we’d love to hear feedback from the community on what we’ve come up with so far.

Current State and Future Plans

We initially developed MongoSwift about 18 months ago with a synchronous API for use with the mobile/embedded version of MongoDB. However, we’ve increasingly shifted our focus toward providing a great server-side experience. We did work over the summer to implement automatic connection pooling in the driver and are now tackling the crucial asynchronous API.

The driver complies with nearly all of the MongoDB driver specifications, which are documents our team writes detailing everything from what a driver’s CRUD API should look like to how it should select which server to send a command to. These specs ensure consistent experiences using MongoDB across languages, while still allowing room for implementations and public APIs to vary in language-appropriate ways.

MongoDB stores data in BSON (binary JSON) documents, and the driver includes a Document type which supports inserting and retrieving a number of built-in Swift types (e.g. Int, String, Bool) as well as various BSON-specific types we’ve implemented such as ObjectId. We also have a BSONEncoder and BSONDecoder that support converting between Codable types and Documents.

The latest release we’ve tagged is 0.1.3. We are aiming to do a 1.0 release once the async API is ready.

libmongoc

The driver currently wraps the MongoDB C driver, a.k.a. libmongoc. So far, this choice has allowed us to focus a lot on the API and ergonomics of the driver and less on internals. After our API is stabilized we plan to start on rewriting the internals in pure Swift using SwiftNIO.

In the meantime, we’ll be able to provide an asynchronous API that uses SwiftNIO primitives by wrapping all blocking libmongoc calls in an NIOThreadPool. We plan to continue maintaining the synchronous API to provide a great experience when using the driver in scripts or the Swift REPL.

At the moment, users are required to install libmongoc separately on their system, but we have work in progress to vendor libmongoc and build it with SwiftPM instead to provide a better user experience.

Testing/Supported Usage

MongoSwift supports Swift 5.0+ and MongoDB 3.6+, and is tested in CI on MongoDB's in-house CI system Evergreen on MacOS and Linux against a variety of MongoDB deployments (standalone, replica set, sharded cluster) both with and without SSL enabled.

Sample Usage of Async API

// Option 1: initialize a client that will lazily connect upon first I/O
let client = try MongoClient("mongodb://127.0.0.1/", using: myEventLoop)

// Option 2: initialize a client and attempt to ping the server with it.
// this provides consistency with other DB drivers e.g. PostgresNIO
let futureClient = try MongoClient.connect("mongodb://127.0.0.1/", on: myEventLoop)

// Create a collection and use it with a `Codable` type of your choice
struct Cat {
    let _id: ObjectId
    let name: String
    let color: String
}

// these methods perform no I/O, they just set up local objects containing
// the state necessary to construct commands
let db = client.db("home")
let collection = db.collection("cats", withType: Cat.self)

// insert a `Cat` with a randomly generated _id and print the 
// resulting `InsertOneResult`
let chester = Cat(_id: ObjectId(), name: "Chester", color: "tan")
try collection.insertOne(chester).whenSuccess { result in
    print("result: \(result)") // InsertOneResult(insertedId: .objectId("..."))
}

// Get a count of documents in a collection
try collection.countDocuments().whenSuccess { count in
    print("count: \(count)") // 1
}

// Retrieve all documents from a collection
let result = try collection.find().flatMap { cursor in
    cursor.all()
}
result.whenSuccess { documents in
    print(documents) // [Cat(_id: "...", name: "Chester", color: "tan")]
}

// Perform an operation for each document in a collection, querying
// the server as-needed rather than holding all the documents in memory
let result = try collection.find().flatMap { cursor in
    cursor.forEach { cat in
        // do something with cat
    }
}
result.whenSuccess {
    print("finished processing all documents")
}
result.whenFailure { error in
	print("error processing documents: \(error)")
}

Thanks for reading, we very much look forward to hearing any thoughts or feedback you have on the driver so far and our future plans!

29 Likes

In the meantime, we’ll be able to provide an asynchronous API that uses SwiftNIO primitives by wrapping all blocking libmongoc calls in an NIOThreadPool .

I think I have already done what you just described - here:
https://github.com/asensei/vapor-fluent-mongo/tree/feature/fluent-4/Sources/FluentMongo

Perhaps it helps to get an idea.

1 Like

Thanks very much for your pitch! A MongoDB driver is definitely in scope for the SSWG and therefore I very much support going forward towards a proposal.

I also think your approach of starting with asynchronous APIs right now (wrapping the blocking C library on a background thread) and then doing an ‘undercover’ rewrite in pure SwiftNIO is sensible. I’m sure @Joannis_Orlandos could give some valuable input here too and maybe you could even collaborate.

One question I have about the C driver you’re wrapping right now is: is that driver being compiled with SwiftPM or do you have to install the C driver into your system before you can use MongoSwift.

1 Like

@johannesweiss

2 Likes

+1!

I'm thrilled to see this pitch come forwards from MongoDB themselves.

It would also be great to hear your thoughts on connection pooling design and whether there is any potential to reuse/share code in this area between MongoSwift and other database drivers.

1 Like

Whoops, thank you. Sounds great!

1 Like

I'd be open to a collaboration if it would be a Swift driver (like MongoKitten). I wouldn't be interested in collaborating on a C based driver. I've got a load of feedback on API design & internals design. Mostly based on the official MongoDB specification and my experience implementing it.

As an added note:
I like the new API design, although I have my concerns with the enum for BSON Values. MongoDB tends to add new types to BSON when needed. Decimal128 being a recent example of this. That's why I think an enum is a burden to the driver and user.

Thanks for the link, @valeriomazzeo!

I think there is definitely potential for this!

Earlier this year our team published a connection monitoring and pooling specification. This defines a standardized approach for how all MongoDB drivers should pool connections. (The monitoring bit involves publishing events in some language-specific manner when e.g. a connection is checked out or back into a pool.)

As libmongoc doesn't comply with this specification right now, we don't either, though we've mostly retrofitted the desired pool API atop their version of pooling. When we get to writing the SwiftNIO-based implementation, we'll want it to fully comply with this spec.

We'd be very interested in taking part in any discussions on this going forward and seeing where pooling needs overlap and differ across drivers and whether it makes sense to share any/all of a pooling implementation.

As I noted in the pitch above, the pure Swift implementation won't be coming for a while longer, and when it does it will occur in pieces rather than being a complete rewrite all at once.

Along the way, we'd certainly like to collaborate with you and the rest of the community on components that are useful to DB drivers and the server-side ecosystem as a whole, e.g. the DNS client you pitched a while back, connection pooling, etc. On the driver code itself, in general the official MongoDB drivers are owned by our team, but welcome collaboration via pull requests, GitHub issues, JIRA, etc.

We considered this, however, adding new BSON types is a pretty rare occurrence (e.g. the last addition Decimal128 was 3.5 years ago). We felt that the benefits of the new approach (being able to exhaustively switch on BSON types, less verbose construction of document literals, being able to fully hide implementation details that were previously on the public protocol, removal of "placeholder" structs containing no data such as BSONNull) far outweighed the potential risk of some potential breaking change.

2 Likes

I’m very +1 on this.

I have used the official driver in the past and MongoKitten more recently due to the Swift-NIO interop requirement in the project I was working on.

From my stand point as a driver user, the async aspect is more crucial than pure Swift internals.

1 Like

late +1, the gradual approach is sensible

2 Likes
  • 1

Strong +1

Hi all! Just wanted to provide a status update: we've just released 1.0.0-rc0, which includes the new async API, vendored libmongoc, and a bunch of other improvements. We'd love for you all to check it out!

Now that the code is written, we are putting together a proposal to post on the forums in the very near future :slightly_smiling_face:

12 Likes

Your message seems to be a copy-paste from the documentation for the MongoDB Java driver. Not sure why you're posting it here (seems like maybe you're a bot) but it is certainly not relevant to the Swift driver.

1 Like