We're excited to announce the first beta release of swift-kafka-client — 1.0.0-beta.1!
What is swift-kafka-client?
swift-kafka-client is a Swift package for producing and consuming messages with Apache Kafka built on top of librdkafka.
It's designed for server-side Swift, with support for:
Swift async/await and AsyncSequence
Integration with swift-service-lifecycle
Logging through swift-log
Metrics through swift-metrics
The README includes producer and consumer examples, along with guides for getting started and integrating the client.
Feedback
We'd love to get your feedback before we tag 1.0. Please give it a try and let us know what you think, either here or by opening an issue. We're especially interested in your thoughts on the API, how it fits into your services and anything we should improve before 1.0.
I'm excited to see this reach version 1.0 soon! I haven't studied everything in depth yet, so please forgive me if this is obvious, but I was wondering about:
What is the purpose of the product KafkaFoundationCompat?
Is the package designed to use FoundationEssential wherever possible when referencing Foundation types?
Is there an API that can consume all the events of a particular topic, then close the connection and signal that the end of the topic has been reached or just continue without blocking for new events? This would be useful as a mechanism for restarting an entity after failure.
thanks, yeah we are excited to announce the 1.0 release pretty soon (sometime this week). Let me allow to answer your queries as follows-
KafkaFoundationCompat is a small, optional module that lets you use Foundation.Data directly as a message key or value when producing messages. Import it if your payloads are already Data (e.g. from JSONEncoder). If you use String or [UInt8], you don't need it.
We're taking a low-overhead approach to Foundation. We haven't switched the whole project to a FoundationEssentials requirement yet, but the core Kafka module's use of Foundation is lean: it uses precise individual imports for the few types it needs, such as import struct Foundation.UUID for unique identifiers and import class Foundation.JSONDecoder for decoding librdkafka's statistics. We'd like to move the remaining imports to FoundationEssentials where possible, and we'll track that as a follow-up
Today, KafkaConsumer.Messages is a continuous stream: it waits for new messages and only ends when the consumer shuts down. There's no API yet to detect that you've reached the end of a topic so your catch-up use case isn't natively covered. This is a great feature to have. Would you be able to submit a GitHub issue and we will scope it for a future release.
I am happy that kafka client is moving forward. Thank you!
We have a fork of this client that moved quite away from the origin.
One of the things that we eventually did differently was propagating events and messages through the single stream (or rather call) - that is very useful for synchronization, especially for rebalance events.
That turned to be the most convenient API for us: synchronized, back pressured (by librdkafka) and performant at the same time. Plus EOFs for partitions.
That is also possible to wrap into AsyncSequence similarly to how it is currently done for messages on main.
Unfortunately, we did not have much time to port the changes in a proper way and open PRs but I would love to help if some of our ideas or code might be helpful API- or performance -wise.
I haven't studied everything in depth yet, so please forgive me if this is obvious, but I was wondering about: