Hello everyone,
Thanks for voicing your interest in this project! Let me try to explain a bit more what this project is about.
Kafka plays a crucial part in many modern software environments, and almost every modern language has clients available. In the past, a lot of different groups have worked on Kafka clients for Swift (Franz, PerfectKafka, SwiftKafka). Some of them are already using librdkafka and can be used as a reference point for how to interop between Swift and the lib. Additionally, a good starting point is to read up on the official documentation of librdkafka, which can be found here. There are also recommendations for developers of language bindings at the bottom.
This project aims to wrap this lib and vend a native Swift API using the new Concurrency features. The two major APIs that need to be exposed are the Consumer and Producer API that librdkafka offers.
As a GSoC participant, you need to develop a proposal on how you want to achieve the end goal. Some rough ideas that I have how you can get started:
First, I would focus on the currency types that a Kafka library needs, regardless of the underlying implementation, e.g., a KafkaClient
, KafkaTopic
, KafkaMessage
, etc. . Next, I would focus on the APIs that adpoters are using to produce and consume messages. Lastly, you need to figure out how to wrap the actual C library.
Kafka has a large surface area and can be very complex in some parts; I think it is essential to state this here. This project’s scope should be to produce a starting point with the basic APIs needed to consume, produce, and configure librdkafka. Additionally, we want to follow the Swift on Server guides best practices around logging and adopting Concurrency, which can be found here.
Another point of reference that I think is great to look at is the go library kafka-go which natively implements the Kafka protocols and vends both a high and low-level API.
If you have more detailed questions around specific areas, please ask questions here or reach out to me through the forums or at f.busch at apple.com.
Franz
PS: Also, thanks for the link @hassila, having a running Kafka backend makes everything easier. I can also recommend trying to run Kafka inside minikube (tutorial how-to).