Server work group, new focus areas

As described in Next steps for the Swift Server work group, we are marching on with this effort. Going forward, the work group will focus on creating a robust, healthy ecosystem, where developers can easily find high quality libraries and tools that enable them to effectively build and run server applications built with Swift.

As described on Swift.org - Swift on Server the goal of the work group is to eventually recommend libraries and tools for server application development with Swift. The difference between this work group and the Swift Evolution process is that server-oriented libraries and tools that are produced as a result of work group efforts will exist outside of the Swift language project itself, and will be distributed across different code bases.

The teams at Apple, IBM and Vapor have engineers that will actively participate in the development of such libraries and tools, and we would love to see the community joining in this effort. To that end, the work group will define and launch an incubation process where anyone can pitch, propose, develop and contribute to such libraries and tools. The workgroup will then help nurture and mature them ensuring standardization, quality and longevity. The exact details of the incubation process are yet to be defined, so stay tuned for a follow up announcement on that.

To bootstrap the process, the work group came up with the following focus areas which we feel can make a real and immediate impact on the ecosystem. The list reflects what we previously heard from the community as well as our own personal experience writing server applications with Swift, and is designed to solve tangible needs that do not require new features from the language or core libraries.

Our goal in publishing this list is to collect feedback, and then work with the community to make pitches and proposals to the @server-workgroup around each one of these libraries and tools, then make progress on them individually. We encourage everyone to review and provide feedback on the content of this list, as well as where they think they can contribute.

Database Drivers and Storage Clients

As many as possible, some critical items include:

  • postgres
  • mysql
  • redis
  • foundationdb
  • cassandra
  • mongodb
  • zookeeper
  • kafka

Tooling and Production Readiness

Official docker support
Official build and runtime docker images. This effort was launched and tracked in https://forums.swift.org/c/development/docker.

Logging
Abstract logging API that allows the concrete logger to be pluggable at compile or run time.

Metrics
Abstract metrics API that allows the concrete metrics collector to be pluggable at compile or run time.

Distributed Tracing
Abstract tracing API that allows the concrete tracing collector to be pluggable at compile or run time.

Crash Handling and Debugging

  • Better debuggability of crashes, for example: capture stack traces and ship them through the logging API before process exits.
  • Recommending best practices and tools for minimizing the impact of exceptions.

Distributed Systems, Microservices and Event Driven Architecture

Http Client
Feature complete Http client that works on Linux without dependency on curl.

File System Library
Non-blocking file system abstraction.

Logging
See above

Metrics
See above

Distributed Tracing
See above

Auth and Crypto
JWT, JOSE, macaroons and others. Requires a basic common crypto library.

33 Likes

+1 from me on this. At the time I started my project, about a year and a half ago, my choice of database was dictacted solely by driver support, not features.

I ended up using MongoDB because MongoKitten was by far the best driver I could find. It was/is a full-featured native Swift driver, whereas most other drivers were simple wrappers around the C driver. We need the same level of support for other databases.

1 Like

Vapor has NIO-based MySQL (+ MariaDB), Redis, and PostgreSQL (+ CockroachDB) clients. I'd be happy to submit these through the SSWG proposal and incubation (once those processes have been determined).

We also have a JWT library that I would be happy to submit. There needs to be some discussion around how to handle crypto though (either make the crypto opaque or do it the right way and create a shared crypto lib first).

Overall, I think Logging and Crash Handling and Debugging should be given high priority. I don't have anything to submit there, but I definitely want to help anyone with more expertise to push those efforts forward.

10 Likes

TIL MongoDB has an official Swift driver: GitHub - mongodb/mongo-swift-driver: The official MongoDB driver for Swift

@kmahar Have you tried MongoKitten? I'd like to know how the two compare.

All of these are good but for me including JavaScriptCore would be a priority since you could then generate HTML with the same code on the server as on the client (using plain DOM methods). This way a Swift-server could both cater the needs of the regular web and iOS apps. Also perhaps even run Swift instead of Node (whenever you need to run JS-code you can just send it to JavaScriptCore).

It actually seems strange to me that JavaScriptCore is not included/ported to linux since Webkit already is. Perhaps is there a special reason for this?

I've just seen so many examples of duplicated server code, where they first generate HTML by building a DOM-tree themselves (with own classes) and then sending the result to the web-client - which will need to basically do the same thing again (whenever the layout change). Instead of just using the same code in both places. Having JavaScriptCore would have lots other benefits as well. E.g. there would be no need to build your own HTML/CSS parsers and when preflighting your react-pages (as this person is doing: Universal/Isomorphic React on a Swift Backend | mko.re).

Also note that Node isn't capable of generating HTML using DOM-apis since it is using v8, which doesn't have DOM-apis (also very strange, but true).

Anyhow, that is my suggestion for a focus area, and I do believe others would also find it very useful.

4 Likes

Yes, a Swift package providing cross-platform JavaScriptCore (or even a higher level interop wrapper, JavaScriptKit?) would be hugely appreciated.

Currently Swift already has some very good interop stories (Objective-C on Darwin, C in all platforms and the Python one being developed as part of the Swift for TensorFlow project), SSWG-backed JavaScript support would be a great addition.

3 Likes

I would also add to the list some very popular ones like ElasticSearch and Microsoft SQL Server.

4 Likes

I am working on a persistence framework which includes a Mongo Db adapter based on the mongo-swift-driver. The framework manages the movement of data between an application process and external persistent storage with a media agnostic API. It also provides application developers with thread safe access to retrieved model objects (as long as they adhere to some straightforward coding conventions).

I haven't even started using this framework in my application yet so I'm definitely not certain the approaches I've taken with it are any good. However, standard APIs or libraries in this area (the space between application/model code and drivers which actually write to persistent media) would also be of use to developers writing server based applications.

1 Like

Kellan Cummings and I have been working on a client for Kafka called Franz. It's currently available through Swift PM on macOS. Hopefully we can get it working on Linux as well soon.

2 Likes

FWIW, I second this. I'm not sure what the process is for this yet, but I'd love to get involved on the logging/monitoring fronts.

For building HTML etc. I'd rather not have "call through to JS" be the recommended/common way. The other direction sounds much more appealing to me: Adding wasm support to Swift and use Swift on the client side as well :partying_face:

10 Likes

Though this is client side Swift and doesn’t belong here ;-)

1 Like

wasm cannot access the DOM (yet), so we are probably many years away until that is an alternative. (and then we have some old-browsers to support too).

A really good list. I think along with Kafka, a pure Swift RabbitMQ package would be really nice to get as well :)

4 Likes

Thanks to everyone for their inputs so far. Please keep them coming.

Although the process by which packages will move from definition to recommendation is not yet defined, one key requirement we expect to be included is that packages should provide an async API, and integrate with SwiftNIO's event loop.

There are a number of libraries already out there like Kitura's PostgreSQL driver which do not provide this kind of integration. The official MongoDB driver mentioned above is another example. We can perhaps consider these to be "first generation" database libraries.

The focus of the Swift Server Work Group is towards "next generation" libraries, like the ones @tanner0101 linked to. This does require a shift in programming model and mindset.

For more examples of the kind of SwiftNIO integration I'm talking about, take a look at how MongoKitten's latest connect() API takes an EventLoop, and returns an EventLoopFuture<Database>. Or how @Helge_Hess1's Redis client makes sure to run its callbacks on the event loop.

If you want to learn more about SwiftNIO and its design, I highly recommend @normanmaurer's presentation at try! Swift Tokyo 2018, where it was announced.

4 Likes

AWS/IBM/Google Cloud SDKs would be great to have, i guess those will be planned after Swift 5/ABI?

6 Likes

In my vision, MongoKitten is both the first generation and next generation driver since MongoKitten has been out since january 2016 and is still going strong.

What would a "next generation" library imply to be? Would it be part of an "official" organisation of sorts? Or still be self-steering?

Yes I personally think it is totally fine for libraries to provide both kinds of API - I was citing MongoKitten as an example of a driver that already provides integration with SwiftNIO.

As @tomerd said at the start of the thread, the details of the incubation process and what would be required for a library to be "recommended" by the working group are still to be decided. I expect there will be a thread to gather people's views on that in due course.

I would like to add NATS support.

Having just built a fully distributed, event based ecommerce platform entirely in Swift and migrated every Redis interface to NATS, this would be a boost to a huge project.

We also took the Vapor Postgres driver and modified it to improve a few specific things we needed whilst using CockroachDB.

The changes have been submitted as part of a PR for Vapor, but we’d be happy to work with @Tanner as our use case is probably one of the largest, and specifically focuses on Cockroach interop. with transaction isolation.

We also have a custom logging API, so I’m happy to work closely with anyone on that.

Additionally, we’ve built libraries to support multiple standard web APIs including:

  • ActivityPub / ActivityStreams
  • PWAs / Web App Manifests
  • RSS
  • WebFinger

We always wrote these to be easy for any project to utilize so we’d be keen to push them through the Swift Server process.

I’m also wondering about DAT support, which is something we’ve been implementing in Rust and using via FFI.

Finally I’m working heavily with OpenFaaS to make everything work serverless. I’m not sure how this would fit in with the workgroup but I’m happy to contribute what I can. Our ecommerce services feature entirely serverless APIs now.

3 Likes

I’m curious if other people think there is a need for a background Jobs library, similar to ruby’s Sidekiq. Something that would allow asynchronous tasks to easily be created, managing prioritized queues, retrying and reliability in the case of the process being killed. Not sure if that’s the kind of thing the server work group aims to solve.

1 Like