I've been experimenting with swift-embedded and went through the vision document, but I’m still a bit unclear on how the project aims to address some fundamental tasks, especially when it comes to Networking (URLSession or low-level equivalents), Filesystem operations, or sys-features and calls like dynamic library loading (utterly important in memory & power constraint environments).
For now, I’ve started to wrap esp-idf, FreeRTOS and Zephyr APIs along with other c/c++ libraries. However, I’m beginning to feel that this approach introduces a considerable amount of work and introduces unsafe and inconvenient code.
What makes this especially challenging is that I frequently find myself having to jump across different OS abstraction layers to clean up incomplete or inconsistent interfaces. These layers often blur the line between hardware and OS specifics, making things tedious and prone to mistakes.
I’m wondering if the Swift-Embedded working group has a strategy or vision for handling this common pain point. Specifically:
Will swift-embedded include abstractions for core functionality, similar to SwiftIO (or even build on / merge with SwiftIO)?
Are there plans to build on a specific (default) real-time OS e.g., Zephyr?
Is there an abstraction approach that handles everything above the HAL and lets us implement drivers where needed?
Right now, I feel that Zephyr has the most thought-through ecosystem compared to its competitors, but I’m not yet deep enough in the embedded space to fully understand all the potential pitfalls of using it as a core dependency.
I’d appreciate any advice on whether continuing to wrap Zephyr APIs is a solid long-term strategy or if I should consider alternatives, and on whats the vision/strategy for swift-embedded on a Foundation for embedded.
For Networking, we do need to be a bit careful. In general SwiftNIO is not embedded compatible, and can’t be made to be.
However, we could subset out the lowest-level features into something like LowLevelNIO. This would provide limited wrappers over sockets and selectors, in a no-existentials no-allocations subset of the code.
That gets us part of the way there, but then we have part two: most embedded platforms aren’t Linux. In particular, they lack our preferred selector technology, epoll, or its future replacement, io_uring.
That means we’d need some way to subset out which features of LLNIO are built. This would let you decide: do you want only sockets, sockets + select, sockets + poll, sockets + epoll, etc. This is crying out for the package traits feature in SwiftPM.
But even that isn’t enough. Many embedded platforms lack sockets at all. Wasm can be a good example: up until recently even WASI didn’t have sockets, and not all Wasm environments are WASI.
So I don’t expect Embedded to have a “one size fits all” networking solution. We should extend NIO to offer a small core that works for the common case of “has a libc”, but many other cases will need custom, per-environment handling.
I wonder if a common set of wrapper packages like swift-log could at least set a standard for what the API surface could look like, then individual components (as in, physical hardware) could be backed by an adapter to their SDK or raw utilization to get a nicer Swift API. Then, LLNIO would just need to adopt the role of an adapter between an available libc and whatever the higher level, more general API is, allowing others to fill the gaps for different systems with more or less hardware capabilities.
Yeah, so to @dimi's point, "looks a bit like" is not the same as "is", and poses a real problem for libraries like hypothetical LLNIO in embedded-land.
Embedded is the wild-west. Everyone defines their own interfaces to their hardware. This is great for hobbyists and for eking out every bit of performance from constrained systems, but horrible for trying to write general purpose interfaces.
@dimi's suggestion is mostly a good one: basically define a pluggable abstraction for "everything else". The problem with this is that the natural pluggable abstraction is socket.
@lukasa Thank you for sharing your thoughts and your detailed answer on networking, but where does this leave us? Will each one continue developing its own solution independently, or is there any plan to form a working group or community effort to address this shared challenge collaboratively?
I believe having a swift network stack that offers the strengths of NIO, URLSession, and similar—while allowing developers to easily integrate hardware-specific APIs through a simple interface—could greatly benefit embedded development. Especially, if all you needed is to link to the hardware-interrupt API and wrap drivers for Wi-Fi, BLE, or Ethernet as a generic swift socket (and maybe link the task/thread API :=)). At least I would love to push such a project :)
I think there is scope for a working group at some stage, yeah. For now, I think the most useful thing for folks to do is to proactively share, perhaps in this thread, any networking abstractions or bindings they have built for embedded platforms. I think of particular interest would be platforms that do not have a socket/poll layer.
Such a thing is absolutely a great idea. What it relies upon is having enough examples for us to make confident generalisations.
NIO is very highly abstracted: in principle, you can build a Channel and an EventLoop out of nearly anything. However, it also relies heavily on existentials and the ability to perform arbitrary heap allocations. That makes NIO as it exists today is probably not a great fit, unless we get a version of embedded Swift that enables existentials. @kubamracek could probably weigh in on how practical that would be.
But a more common slightly-lower-level abstraction is definitely possible, as is a version of NIO with a slightly different design. Similarly URLSession et al.
Well, I’ve been using IORingSwift on a CM4 to handle socket, file, UART, SPI and I2C I/O for an audio device. Works nicely, in conjunction with SwiftIO and LinuxHalSwiftIO. I’ll port to Zephyr at some point.
There are some non-Swift projects that might be instructive?
For example, in the Arduino ecosystem, people write board cores that can be plugged into the Arduino IDE (JSON profile), which then uses its own compiler setup with its it own HAL layer that can access the selected board profile.
In all of these it is still up to the community/manufacturer to make sure that the layer between the specific hardware and the abstraction exist, which I would not personally expect the core Swift project to be able to do! There. are. so. many. boards.
These ecosystems are largely focused on boards without Linux/don't “have a libc”, but not exclusively.
None of these are C or C++ or Python itself providing a Vision, FWIW. But it will be interesting to see how high and how low Swift decides to go.
ETA: compre working with an espressif board to a pico using Arduino
(also FWIW, to the degree I've been writing stuff for swift-embedded yet I've been using a Services/Protocols model for offering up hardware interfaces for even pretty low level stuff because the hardware itself can change out from under you, so even a socket would be protocol Socket)