MongoDB Client using SwiftNIO

There are currently two established drivers for MongoDB usable in the Swift ecosystem.

One of the libraries is the official package by MongoDB and the other is my own package called MongoKitten.

Using this pitch, I'm hoping to make a step to bringing the support for MongoDB to a higher level within the (Server-Side) Swift ecosystem.

The current situation

There are two big primary drivers that are usable, as listed above:

The first (official) one wraps around the MongoDB C library. It currently only supports synchronous requests. Aside from the synchronous implementation of executing queries, the library does not run on a NIO EventLoop.

The second library is an implementation based on SwiftNIO. It only supports asynchronous requests which has a small impact on the friendliness of the API for MongoDB's mobile database users that use the MongoKitten driver.

Proposal

The currently envisioned plan is to create two modules.

  • BSON
  • MongoCore

BSON is an implementation of the bson spec to read and write MongoDB's data format.

MongoCore is an objectively correct implementation of the MongoDB database protocol specification. It should handle initiating a connection, maintaining a connection and provide an API for MongoDB client libraries to send queries and receive results.

Motivation

MongoKitten is technically feature-complete, although some of the less-commonly used APIs are not implemented in a helper function.

The incentive for the MongoKitten project is to:

  • Open up to more engagement with the Swift community

  • Separate networking code from the public APIs to allow for better tests

  • Create a publically available test suite that is actively maintained and covers more edge cases

The proposed solution

There is an implementation of BSON available at OpenKitten/BSON. There is no implementation of MongoCore available, but a MongoCore + Client implementation can be found at OpenKitten/BSON.

Example APIs

let settings = try ConnectionSettings(uri: "mongodb://localhost")

// MongoCluster manages a ConnectionPool and initiates + maintains connection with all hosts
// It also runs a heartbeat and applies DNS Seedlist Discovery
// Note that MongoDB's protocol is multiplexed, so only 1 connection is made for each host
// Unlike SQL where you need many connections to the same database server
let mongoCluster = try MongoCluster.connect(with: settings).wait()

// Databases and collections are created after the first query and are a placeholder until then
// MongoKitten has a different more subjective syntax, which is slightly off this example
let database = mongoCluster.databases["app"]
let users = database.collections["users"]

struct User: Codable {
    let _id: ObjectId
    let name: String
}

// Reads all entities in this collection into an `Array<User>`
let users = try users.find().decode(User.self).fetchAllResults().wait()

Example implementation

I've created an example implementation in the MongoCore and MongoClient folders in the MongoKitten 6 branch:

7 Likes

Ping @kmahar.

Is a combined effort possible?

For users, it will be quite confusing to have two "official" drivers (one from the SSWG and one from Mongo).

While NIO is great technology, is there really a necessity to rebuild everything ontop of it? We are using the official mongo driver with the swift wrapper solely for the reason that its battle tested and we know it's going to work.

@dlbuckley there's no need to rebuilt everything because I already built it with MongoKitten.

By building these libraries in Swift with SwiftNIO, our ecosystem is able to implement features with a better API than is otherwise possible. At the same time, this allows us for better performance and a better integrated ecosystem.

The primary reason why I wouldn't consider using the official driver with SwiftNIO is because it's got a blocking API. The only two options are to either build a SwiftNIO driver or to wrap the C library with async APIs and expose a NIO interface for that. The second solution has far more limitations than the first one.

1 Like

as stated in the sswg incubation process, all initiatives are welcome, and the sswg can and will vote and recommend more than one solution when such exist

@Joannis_Orlandos I think there are two very different approaches between the drivers and both have their pro's and con's. I can see you that you feel very strongly that your solution is the right one and should be the way to go, but @tomerd is right, it should go through the SSWG proposal process.

I think my main question is what were the main reasons that the official driver went the way that it did (maybe @kmahar can answer that) and is there a reason why both teams can't work on a combined solution to combat the problem?

From a personal POV, we chose the official driver due to it having known reliability in many production environments. Reliability and being a known entity was a bigger driving force for us when it came to such a critical part of our infrastructure than an asynchronous/better interface from a business perspective.

I can understand the preference for a driver that's backed officially. I'd be open and interested in working together on a solution. It would be far more effective than working on two implementations of the same functionality.

It's interesting to consider recommending more than one solution, depending on the applications' needs.

2 Likes

Hi!

@dlbuckley points out a large part of our motivation for wrapping the C driver — it’s battle-tested. It’s reliable and is compliant with all of the crucial MongoDB driver specifications for things like server discovery and selection, and provided a solid foundation on which to build a new driver. Wrapping the C driver is not new for us — our official C++ and PHP drivers also wrap it. This decision has allowed us to thus far focus less on internals and more on providing a really great, spec-compliant synchronous API.

We plan to pitch MongoSwift to SSWG sometime this summer, and will provide concrete details then about our plans to provide great server-side support in the driver. As @tomerd points out there can be more than one SSWG-recommended driver, and we are happy to see MongoKitten go through the process. There’s definitely room for more than one MongoDB Swift driver to exist, especially given the differences in our drivers and the variety of user needs.

3 Likes

I concur, and I think it's a good step forward to see official MongoDB support and to provide to the different needs of different users.

I do also want to clarify that just because the C driver is very battle tested and compliant, this doesn't necessarily manifest itself in the Swift wrapper due to a difference in ecosystems. I think a good example of this was the UTF-8 encoding bug that I found early this year. The driver looks solid on almost every front, but I think there are different challenges when it comes to a wrapper that shouldn't be ignored. As far as I've seen, you're not ignoring them. So my compliments for that!

I've created an example implementation. I think this is a good way to separate the protocol from the client behaviour and helpers. This should work nicely with MongoDB's official specifications and respective tests.

I think that @kmahar and her colleagues would be best suited to advice on how to approach testing. I'd love to see collaboration and community around this driver, outside of @Obbut and myself.

@tanner0101 and @Mordil, do you have suggestions for testing the protocol and client behaviour? Since you built out MySQL, PostgreSQL and Redis drivers recently.

For the Redis library, a great majority of the test suite are integration tests as there are dozens of convenience methods for individual commands. For those I connect to a Redis instance through Docker to execute them.

For verifying byte parsing, I do have unit tests where I pre-fill ByteBuffers and pass them to the parsing functions. NIO also provides a helper test object for ByteToMessageHandler testing (I haven’t used it yet, but plan too)

I also plan on writing a CLI app or two that executes a series of tasks against Redis using the library for performance testing - which I believe NIO and Vapor both do

1 Like

Thanks for the feedback, I didn't know NIO had their own testing helper. Great goodie. I wrote those myself so far. A CLI for performance testing is also a great idea!

Happy to share some insight on how we test the official drivers over here:

As you noted, many of the specifications come with accompanying tests that ensure particular features/components are implemented in accordance with the specification. That said, all of our drivers implement a number of additional unit and integration tests to cover what the spec tests don't, which you can get a feel for by looking their GitHub repos.

Generally, drivers run all of their tests against a matrix consisting of supported OSes, supported MongoDB server versions, various server configurations (standalone, replica set, sharded cluster, with and without auth enabled, etc.), and supported language versions. To assist with this we use a publicly available tool called mongo-orchestration that allows easy configuration of various MongoDB topologies.

Hope that helps!

1 Like