Exploring Multicore Concurrency for Embedded Swift

Platform Embedded
, ,
1

Hello :waving_hand:

I’ve been experimenting with running Swift Concurrency across multiple cores in Embedded Swift, specifically on RP2350-class microcontrollers.

Today, Embedded Swift defaults to NoThreads. That works well for single-core or cooperative environments, but it also means the Swift runtime assumes it is not being executed concurrently across multiple cores. That becomes a limitation when trying to schedule Concurrency tasks on multiple cores.

To explore this, I opened a PR that introduces a new threading mode: SWIFT_THREADING_DEFER.

Instead of choosing pthreads, C11 threads, or the existing no-threaded runtime model, SWIFT_THREADING_DEFER maps Swift’s threading abstraction to a small C hook surface. Embedded platforms can then provide their own implementations for mutexes, condition variables, once operations, thread identity, stack bounds, and TLS.

The PR also includes weak default implementations. Those defaults keep simple single-threaded embedded configurations linkable, while allowing a platform to override individual hooks with strong definitions at link time.

There is an intentional tradeoff: the actual threading operations are no longer fully header-only inline implementations in this mode. ThreadDeferImpl.h still provides inline wrappers for Swift’s threading abstraction, but those wrappers call external C symbols. That symbol boundary is what allows the weak defaults to be replaced by platform-specific implementations.

The PR is here:

I’d love feedback from the community. In particular, I’m interested in whether this hook surface feels like the right abstraction boundary, and whether the weak-default approach is a reasonable way to keep simple embedded configurations working while enabling multicore platforms to opt in.

13 Likes
[Pitch 4] Custom Main and Global Executors
2

Yes, this is a fantastic direction to go for Embedded Swift. Thank you for working on this!

I've been doing something that's in the same spirit for the standard library itself, which is captured by the platform abstraction layer header. It's taking the same approach that you are, with a bunch of swift-prefixed declarations (although I used _swift_ to stay away from the Swift runtime) that each platform is required to define. Some functions are optional, if you aren't using the corresponding functionality. That header is dealing with only what the standard library cares about, e.g., memory allocation, random number generation, and standard output.

My thinking is that we would extend this header to cover the entire set of potential platform dependencies for Embedded Swift. That way, to bring up Embedded Swift on any given platform, you would go through and implement each function from that one header, and you'd be done. For your pull request, that means I'd prefer the contents of ThreadDeferImpl.h‎ to go into the EmbeddedPlatform.h where we document each entrypoint clearly for folks doing the porting. What do you think of that direction?

I debated this for the header I was working on, and took a bit of a different direction. I added an EmbeddedPlatformPOSIX library that folks could link in to implement the header definitions in terms of the same underlying POSIX functions we've already required (e.g., posix_memalign, putchar) to make it easier to make the move. For concurrency, we'd likely want different "single threaded" and "multi-threaded" implementations with different implementations on top of POSIX. I'm not opposed to providing weak definitions, but I wonder if being explicit is better for folks, so long as it is very well documented what they need to do.

// Expected: initialize a non-recursive mutex; checked may request misuse checks.
void swift_threading_defer_mutex_init(uintptr_t *handle, bool checked);
// Expected: destroy a mutex initialized by swift_threading_defer_mutex_init.
void swift_threading_defer_mutex_destroy(uintptr_t *handle);
// Expected: acquire a non-recursive mutex, blocking or spinning until success.
void swift_threading_defer_mutex_lock(uintptr_t *handle);
// Expected: release a non-recursive mutex held by the current context.
void swift_threading_defer_mutex_unlock(uintptr_t *handle);
// Expected: try to acquire a non-recursive mutex without blocking.
bool swift_threading_defer_mutex_try_lock(uintptr_t *handle);
// Optional: acquire from error paths; default calls mutex_lock.
void swift_threading_defer_mutex_unsafe_lock(uintptr_t *handle);
// Optional: release from error paths; default calls mutex_unlock.
void swift_threading_defer_mutex_unsafe_unlock(uintptr_t *handle);

I think I'd call these _swift_mutex_. We could also use them directly from the Synchronization library to make Mutex available in Embedded Swift.

Do we actually need the swift_threading_defer_lazy_mutex_* and swift_threading_defer_recursive_mutex_* entrypoints for the concurrency library? We shouldn't be using recursive mutexes, for sure.

void *swift_threading_defer_tls_get(uintptr_t key);
void swift_threading_defer_tls_set(uintptr_t key, void *value);

Hmm, these would be useful as default implementations for the "exclusivity TLS" get and set operations I have in the platform abstraction header.

I’d love feedback from the community. In particular, I’m interested in whether this hook surface feels like the right abstraction boundary

In general, I'd like us to pare down the number of functions that someone needs to implement to bring up Embedded Swift on a platform, to the minimum reasonable set.

This is an excellent direction for Embedded Swift, let's refine it and get it landed!

Doug

13 Likes
3

My thinking is that we would extend this header to cover the entire set of potential platform dependencies for Embedded Swift. That way, to bring up Embedded Swift on any given platform, you would go through and implement each function from that one header, and you'd be done.

Yes, this definitely sounds like the right direction, not just for threading but for all platform-dependent code. I’m glad this experiment aligns with that direction!

For your pull request, that means I'd prefer the contents of ThreadDeferImpl.h‎ to go into the EmbeddedPlatform.h where we document each entrypoint clearly for folks doing the porting. What do you think of that direction?

That makes sense. I’ve updated the PR in that direction: the hook surface now lives in `EmbeddedPlatform.h`, using `_swift_*` names, and `ThreadEmbeddedImpl.h` is just the adapter from Swift’s internal threading abstraction to those platform hooks

I debated this for the header I was working on, and took a bit of a different direction. I added an EmbeddedPlatformPOSIX library that folks could link in to implement the header definitions in terms of the same underlying POSIX functions we've already required (e.g., posix_memalign, putchar) to make it easier to make the move. For concurrency, we'd likely want different "single threaded" and "multi-threaded" implementations with different implementations on top of POSIX. I'm not opposed to providing weak definitions, but I wonder if being explicit is better for folks, so long as it is very well documented what they need to do.

That's really nice! Let me play this back to make sure I’m interpreting the model correctly.

Today, the Swift threading package is effectively selected as a build-time choice for the stdlib/runtime. For Embedded, the direction would be to move more of that decision to link time through the selected EmbeddedPlatform implementation. For example, there could be multiple implementations of the same EmbeddedPlatform.h hook surface:

  • a single-threaded implementation
  • a multithreaded POSIX/pthreads implementation
  • possibly target-specific implementations provided by board/platform packages

The consumer would choose which implementation to link, and the Swift libraries would just depend on the documented _swift_* platform hooks. For convenience, the toolchain could provide a conservative single-threaded implementation, while platforms that need multicore scheduling would link a stronger/threaded implementation.

Is that the shape you have in mind?

If so, I can take a look at how to materialize this in the PR. One possible approach would be to provide multiple EmbeddedPlatform implementation libraries with the same library name and the same hook surface, but different behavior, for example single-threaded and POSIX-threaded variants. Then the consumer could select the implementation by putting the desired library earlier in the linker search path, or provide its own same-named implementation library. Does that match the model you’re thinking of?

Do we actually need the swift_threading_defer_lazy_mutex_* and swift_threading_defer_recursive_mutex_* entrypoints for the concurrency library? We shouldn't be using recursive mutexes, for sure.

void *swift_threading_defer_tls_get(uintptr_t key);
void swift_threading_defer_tls_set(uintptr_t key, void *value);

I kept the recursive and lazy mutex hooks because they preserve the current Threading contract. I agree we should reduce the required surface over time, especially if Concurrency should not rely on recursive mutexes, but as it does rely on them at the moment I think removing those assumptions could be a separate cleanup. In the meantime, single-threaded/default implementations can make those hooks cheap aliases, while platforms that need to honor the full current contract still can.

For TLS, I also moved toward making allocation explicit rather than treating it as optional, because the existing abstraction does have dynamic TLS keys. The embedded implementation can still reserve fixed keys for known runtime/stdlib uses, but if the abstraction exposes dynamic allocation, the platform layer should probably model that directly.

This is an excellent direction for Embedded Swift, let's refine it and get it landed!

Thank you for taking the time to review this, and for all the context and insight! I can take another pass over the PR once we’re aligned on the specifics.

4

Yes, that's the idea.

I think we should have separately-named libraries for each. I've had bad experiences with configuration problems where a slight mistake in the linker search path causes a lot of head-scratching. If we use separate library names, it's very clear which one is getting used, and if somehow multiple libraries get linked in, it becomes very obvious.

Hmm. If we can avoid making these part of the embedded platform, it would be good, because every platform has to implement the hooks.

I see that there's just one recursive lock (statusLock) in the concurrency runtime. @ktoso or @John_McCall, do we need this lock to be recursive?

There's only one LazyMutex, and it's dynamically dead in the embedded build (see Task.cpp's continuationChecking::isEnabled. We can shift the #ifs around so we never utter a lazy mutex in the embedded concurrency build, and that abstraction layer surface area can go away.

Doug

1 Like
5

I know we’ve had trouble with cancellation handlers. We probably ought to have a better approach than just using a recursive lock, but it’s difficult because we do need to synchronize appropriately with leaving the handler scope. So I can’t promise that we can easily eliminate this problem.

6

@Gonzalo_Larralde I wanted to provide some perspective from the debugger side here and ask a couple of questions.

While the debugger today doesn't have a happy story for embedded swift async debugging, this is something that should be relatively easy to address in the current single-threaded model. One of the questions the debugger has is: "Given an operating system thread, which Task is it executing?". This is a core piece to support debugging async code. In this many-concurrency-libraries world, there are two challenges in answering that question:

  1. How does the debugger know which concurrency library is being used? I initially thought separately-named libraries would help, but because we statically link everything for embedded, I am no longer sure. A symbol name maybe?
  2. Once the debugger knows which concurrency library is being used, how does that library keep track of which Task is currently being run on a Thread? In "full swift", the runtime uses thread local storage to save a pointer to the Task. In the current embedded library, my understanding is that we have a simple global variable with a pointer to the Task (I'm still trying to confirm this). How would the new library address this?

The debugger also relies a lot on the layout of the Task data structure. Based on the PR, I believe this would be fixed across the many embedded variants?

7

Hello @felipepiovezan ,

Thanks for bringing this up. I am not very familiar with LLDB internals, but I spent some time gathering information about the Task debugging path. Please correct anything I’m misunderstanding.

How does the debugger know which concurrency library is being used? I initially thought separately-named libraries would help, but because we statically link everything for embedded, I am no longer sure. A symbol name maybe?

From what I can see, the SwiftTasks OS plugin first tries to find the Swift concurrency runtime through SwiftLanguageRuntime::FindConcurrencyDebugVersion. That eventually looks for the concurrency runtime/module and the _swift_concurrency_debug_internal_layout_version symbol. An option could be moving to only test for this symbol, although I wonder if it may have some side effects?

Once the debugger knows which concurrency library is being used, how does that library keep track of which Task is currently being run on a Thread? In "full swift", the runtime uses thread local storage to save a pointer to the Task. In the current embedded library, my understanding is that we have a simple global variable with a pointer to the Task (I'm still trying to confirm this). How would the new library address this?

Yes, I think this is the key point. From what I understand, the plugin starts from an LLDB Thread representing the current execution context, gets that thread’s extended info, and looks for a Thread-Specific Data address. Then TaskInspector uses that to find the current task pointer. (GetTaskAddrFromThreadLocalStorage, ComputeTaskAddrLocationFromThreadLocalStorage)

The interesting problem is what to do when TLS is emulated, or when there is no OS thread in the usual sense. In the current implementation, LLDB expects a tsd_address for each real/backing “thread” it is inspecting. For bare-metal cases this might be relatively easy: the execution contexts may just be CPU cores, so the runtime/debugger contract could be something like a small per-core table or exported symbols indexed by CPU.

RTOS cases like Zephyr, where there are real scheduler threads, probably need a different path. There the debugger or an LLDB OS/process plugin would need to know how to map an LLDB Thread to the RTOS thread state and obtain the corresponding tsd_address, or otherwise expose the current Swift task pointer for that RTOS thread directly.

Any thoughts on possible approaches?

The debugger also relies a lot on the layout of the Task data structure. Based on the PR, I believe this would be fixed across the many embedded variants?

I believe the task layout would remain fixed across the embedded variants in this PR. I do not anticipate changes to Task itself as part of this work.

This is a really interesting area! It would be very useful to make multithreaded embedded debugging work with Swift Concurrency.

8

I realise in hindsight that I might have sent you on a quest to read a lot of code I wrote myself :sweat_smile:Your general assessment is correct. Today, LLDB assumes it deals with the "traditional" swift runtime, and all of the async support is written with that in mind.

However, as you discovered, there are only a few places where this information matters, and it should be easy to change those pieces once we know how a given runtime works. My questions were trying to:

  • Help me understand how the embedded concurrency runtime works (today and in the future, with your proposal), as I was about to try improving support for the embedded case (as it exists today).
  • make sure your proposal helps the debugger answer those two questions (which concurrency runtime is loaded, which task is currently running on every thread). I don't think your proposal needs to prescribe an exact path for this, merely point out that it is crucial for debugability.

Inevitably, I think the debugger will need to understand whichever version of the concurrency runtime is used, and that means having specialised code for each of them.

For the "Which task is executing on this thread" question:

  • The normal runtime writes a Task* on a TLS variable.
  • The embedded runtime writes a Task* on a global variable.
  • This proposal could document that multicore runtimes (if they are interested in being debuggable) need to provide a mechanism for the debugger. Maybe also use the existing runtimes as an example.

For the "which runtime is used in this program":

  • The regular runtime is a dylib we can test for. The "version" symbol you mentioned is used in case some details of the library change, and for mixing old LLDBs with new concurrency libraries (or vice versa).
  • I still need to figure out how to detect the current embedded runtime.
  • For what's being proposed here, also document this need and maybe provide an example of the current embedded runtime (once we have a strategy for detecting it)
9

I realise in hindsight that I might have sent you on a quest to read a lot of code I wrote myself :sweat_smile:

Oh, no worries at all! This came out of pure interest in the topic you brought up. FWIW, these days it is much easier to read and explore code.

I am more than happy to document this. I also wonder whether it would be useful to formalize, at least loosely, the shape of the mechanism that platform/runtime code is expected to provide.

One thing that seems worth considering is that an embedded target may be single-threaded, multi-threaded, or multi-core without a conventional threading model, and that may affect the right detection strategy.

From what I saw, the Darwin path can surface tsd_address through debugserver’s thread extended info. That path is Mach-specific, so something more generic may be needed for embedded. There are FreeRTOS-aware debugging paths in the broader GDB/OpenOCD/J-Link ecosystem, but I do not know whether upstream LLDB has an equivalent built-in path here. In any case, it would not expose tsd_address in the same Darwin/Mach-specific way, so there may be some additional wiring required.

I am not familiar with debugger integrations, but would it be possible or appropriate for the runtime/platform code to expose a small hook that lets the debugger derive the task storage address for the “current thread” or execution context? In a non-threaded multicore environment, I assume that context might effectively be the current core.

I can imagine that calling into target code from the debugger could perturb execution state, so maybe that is undesirable. Are there other established ways for a program/runtime to describe how it wants to be inspected? For example, is there any precedent for debugger-interpreted metadata or expressions that avoid executing target code?

For the “which runtime is used in this program” question, I wonder whether embedded could expose a debugger-visible runtime identity symbol next to _swift_concurrency_debug_internal_layout_version. My understanding is that _swift_concurrency_debug_internal_layout_version currently describes the version of debugger-visible internal layouts, so it may be better not to overload that value with runtime identity. But perhaps a companion symbol, or a small set of well-known symbols, could let LLDB distinguish the regular runtime from the embedded runtime, and later distinguish different embedded runtime strategies if needed.

Again, I am mostly exploring ideas out of interest here. I am happy to document as much as you think is viable today. Just let me know what level of specificity would be useful, and I can add it to one of the PRs.

Thanks for your feedback!

10

For the “which runtime is used in this program” question, I wonder whether embedded could expose a debugger-visible runtime identity symbol next to _swift_concurrency_debug_internal_layout_version.

I think this is the crucial piece. More importantly, the "which runtime" is not exactly important, the crucial part is "how does this runtime -- whatever it may be -- stores its currently-executing-task-pointers?"

For that end, I prototyped another variable we expose in the Concurrency/Debug.h header capturing that very notion. Since this is a build-time configuration (for the target), LLDB can't really guess what happened there; a variable seems the only solution.

[Concurrency] Add _concurrency_current_task_storage_kind to Debug.h

[lldb] Detect how the swift runtime stores currently executing task

Possibly, but note that this is unfortunately quite tightly integrated into LLDB's abstractions (e.g. how it represents threads, etc). In the PR linked above, I tried to abstract away some of those details and expose the bare minimum required.

An important consideration: this mapping of Threads -> Task has to be fast. It happens on every single stop (there are usually many non-visible stops when a user is stepping over lines of code, and there are many non-visible stops during program startup). This is particularly important when communication between the debugger and the target is slower than local (e.g. over a wire, even a fast wire).

Concurrency-aware debugging with Embedded Swift platform runtimes
11

Hi all,

I wanted to post a short update here, because we've made a bunch of progress on this thanks to the great work of @Gonzalo_Larralde. We now have Mutex enabled in Embedded Swift builds using new entrypoints in the platform abstraction layer. There's currently only a single-threaded implementation (by linking in swiftEmbeddedPlatformSingleThreaded), with a pthreads-based library coming soon.

A follow-up pull request extended this to recursive mutexes, which don't surface in the Synchronization library but are used inside the Concurrency implementation. We're still sorting out some issues with thread-local storage and the main thread/main actor for the concurrency library, but we're getting closer to getting the full Concurrency library building for Embedded Swift with a clear, well-defined, small set of platform requirements that should be easily adaptable.

Doug

21 Likes
12

Hi all!

Another update, because things are going well. @Gonzalo_Larralde landed support for TLS in the platform library (pull request) the "embedded" threading library that underpins the C++ parts of the concurrency runtime (pull request). I got the concurrency library building with -ffreestanding (pull request).

The coolest thing is @Gonzalo_Larralde 's multi-threaded concurrency implementation for the Pico SDK, which builds on the soon-to-be-merged next step of adopting the embedded threading library for embedded concurrency ([Embedded] Enable platform-backed threading in Concurrency by gonzalolarralde · Pull Request #91047 · swiftlang/swift · GitHub).

There are still a bunch of odds and ends to tackle, but we're making great progress on this.

Doug

11 Likes
13

Really looking forward to those merges. I have been playing a lot with embedded Swift lately, even porting some games to Nintendo 3DS using Embedded Swift. I had to rewrite my Bluetooth GATT server library to get it compiling on the Pico W and ESP32, and make it synchronous to compile with embedded Swift. TLDR, having support for concurrency will unlock a lot of possibilities.

2 Likes
14

@Douglas_Gregor, thank you again for all the support and guidance throughout this work!

We discussed several possible follow-ups in swiftlang/swift#91047, including the idea of collecting the remaining work into a checklist. Here is my current view of what remains.

Core Embedded Concurrency follow-ups

  • Replace the specialized _swift_getExclusivityTLS and _swift_setExclusivityTLS PAL hooks with the general _swift_tls_*mechanism. This landed in swiftlang/swift#91157.

  • Define the Embedded MainActor and async main story. A conventional main thread does not necessarily exist, so this likely needs explicit contracts for startup, shutdown, and scheduling work onto the platform’s main execution context.

  • Move clocks into the platform abstraction layer. Clock and sleep behavior is currently handled ad hoc by platform executors. PAL hooks could enable the standard ContinuousClock and Task.sleep APIs using hardware timers or RTOS facilities. Pull request opened swiftlang/swift#91187

  • Make custom SerialExecutors fully supported in Embedded. This would let actors express CPU, peripheral, interrupt, or other platform-specific affinity requirements.

  • Audit concurrency APIs to determine what ones are @_unavailableInEmbedded that don't need to be. For example, I tripped over assumeIsolated that could, I think, be enabled in Embedded.

  • Review the documentation PR from @BoisyPitre so we can make it as easy as possible to bring up a new platform.

  • Work on an approach to fix the now-broken _swift_concurrency_debug_internal_layout_version value, that helps LLDB figure out what the current active Task is when debugging an Concurrency enabled Embedded Swift program. Opened a new thread here

Hosted implementations and testing

  • Add hosted Embedded executors for testing. I currently have a Darwin/libdispatch implementation in progress, with Linux as a possible follow-up. These are not intended as microcontroller executors; they provide practical multithreaded environments for exercising the Embedded runtime in CI. Pull request opened swiftlang/swift#91338

  • Enable selected existing Concurrency tests under Embedded. Rather than duplicate the full suite, the initial coverage could reuse tests involving tasks, actors, task groups, cancellation, and suspension, plus one focused test that proves jobs execute in parallel. Pull request opened swiftlang/swift#91338

Broader platform work

These feel useful but separable from finishing the core integration, and would be good areas for community involvement in case anyone is interested:

  • Make Embedded Concurrency independently buildable. It would be useful to iterate on Concurrency and platform implementations without rebuilding a complete toolchain. This might fit naturally into the standalone standard-library package using traits or conditions to select the required libraries.

  • Implement Embedded executors for FreeRTOS and Zephyr. Both provide the worker, queue, timer, and synchronization primitives that could be enough to integrate Swift Concurrency without libdispatch.

  • Explore a small family of executor implementations for platforms without libdispatch. The existing CooperativeGlobalExecutor already covers the single-threaded, cooperatively drained case. Two additional models seem useful: a fixed-worker executor for systems with a known number of cores or execution contexts, where a fixed number of long-lived workers consume jobs from shared queues; and a thread-backed executor for platforms that provide basic thread creation and synchronization but no higher-level scheduling library. Both need enough machinery for queueing, wakeups, priorities, and timers, but should remain substantially smaller than reimplementing or embedding libdispatch. The goal would be a few adaptable executor models spanning single-threaded, fixed-core, and general threaded environments.

  • Explore a wasi-threads executor. This is platform-specific and likely involves shared memory, Wasm atomics, worker startup, and TLS. Some executor design lessons may overlap with bare metal, although the host-provided threading model is quite different.

I’m happy to keep working through these, but several are independent enough that others could pick them up. Feedback on priorities, missing items, or areas where the abstraction still needs refinement would be very welcome!

7 Likes
15

Thanks for collecting all of these!

There's a pull request here that started on this. I've left a number of comments there.

A couple of other things I know of:

Audit concurrency APIs to determine what ones are @_unavailableInEmbedded that don't need to be. For example, I tripped over assumeIsolated that could, I think, be enabled in Embedded.
Review the documentation PR from @BoisyPitre so we can make it as easy as possible to bring up a new platform.

Doug

5 Likes
16

I commented on the PR, but I believe the direction this went makes it difficult for the debugger to support concurrency in embedded swift, as the way threading works is no longer a property of the concurrency library, where all the "debug ABI" variables live today, but rather some other shim library that is chosen when a user's binary is being linked. And this mechanism is only done for embedded swift, so we can't move those ABI variables into the shims, as they don't exist for "regular" swift.

17

This is exciting to follow and amazing progress so far.

For those two in particular, I would love if we could add them to GitHub - swiftlang/swift-platform-executors: This package provides platform-native executors for Swift Concurrency. · GitHub. The repo is intended to provide executors for Swift's supported platforms. Ideally, at some point in the future we would start using them as the default executors for Swift. I am mostly focused on the Linux executors here which are implemented without any Dispatch dependency using pthreads. Decoupling the Concurrency runtime entirely from Dispatch is a larger thing though and requires exposing more hooks as protocols that executors can implement such as the ongoing [Pitch 4] Delayed Enqueuing for Executors.

1 Like
18

Hello Felipe, that’s completely fair. Sorry for not getting back to you sooner!

I see your point. I missed your reference to this change back then, so when Embedded switched from NoThreads to the Embedded threading backend it erroneously started reporting pthread_reserved_key.

Would it make sense to start with a PR to add an unavailable storage kind? That would at least give LLDB a clear and deterministic failure path instead of selecting a lookup strategy that might be incorrect. I could put together that PR if this direction makes sense to you.

I opened this thread to analyze the broader topic in more depth. We have discussed a few possible approaches but none seems like an obvious solution, so I think a separate thread might help organize the conversation!

I'm also adding this topic to the ongoing list of pending items to work on.

Thanks!

19

That’s great! I wasn’t aware of this effort.

Removing the dependency on libdispatch is a great step forward. For bare-metal platforms, I think the main motivation will be finding the right scheduler for each platform, its constraints, and the specific use case. So I really like the idea of not trying to build the executor, but instead providing a menu of options.

So far, I’ve been experimenting with a vibe-coded executor, although its performance on the RP2350 is quite meh. There are likely multiple factors contributing to that. I hope to have time to come back to it in a couple of weeks.

2 Likes