Multi-Process Parallelism

Using Swift
1

Hi all,

I have a dynamic library I want to use from Swift that is an emulator for a platform. The library is written in such a way that all state is static and I can only have one such emulator running per process. In Swift, I have written a class called Emulator that provides an API to that library. The issue is that. I want to have multiple instances of that Emulator class created and running in parallel. I cannot use multi-threading due to the underlying dynamic library limitation and so I was wondering if there is a clean way to create these class instances in separate processes and have them communicate with my main process. Is there a simple/nice way to do that in Swift?

In general, are there any common design patterns for how situations like this should be handled?

Thanks,
Anthony

2

I think that Process and Pipe might be what you are looking for, if they have to be separate processes.

3

Thanks @Diggory! The problem with process and pipe is that: (1) you can't pass e.g. a closure for execution in a separate process -- you should compiled separately the child processes, and most importantly (2) there is no simple way to share memory between the processes so I don't have to keep copying data around. Is there any other option?

4

Process, based on looking at the source at swift-corelibs-foundation, relies on posix_spawn to create the new process, which is a combination of fork and execute, which are Unix/Linux (maybe Posix?) primitives to create a new process and execute a new executable image. However, as I have done many times in the past on a variety of Unix variants, you can use fork only, which will create a new process which is an exact copy of the calling process. Using fork only will allow your new child process to call into whatever you want.

This is all based on low-level operating system calls, and you will need to be aware of C-interfaces with Swift, etc.. You can use the operating system shared memory facilities to setup shared memory regions if you want prior to forking, or you can pipes (Pipe) to move between processes, or both.

There may be a Swift library out there that puts a Swifty face on all of this, but, I have not done a serious look

5

Is there a simple/nice way to do that in Swift?

What platforms are you targeting?

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

6

MacOS and Linux.

7

Alas, your Linux requirement rules out NSXPCConnection, which is my preferred mechanism for this sort of thing. This means you really are stuck with UNIX-y primitives, processes, pipes, and so on. Those primitives are… well… primitive, that is, they are the place to start for problems like this but you’ll need to layer a bunch of extra stuff on top. For example, when you send a message to another process over a pipe, you are responsible for serialising that message to a byte stream. There are lots of ways to do this — and some of them, like Codable and protobuf, are quite Swift friendly — but it’s still your responsibility.

You wrote:

(1) you can't pass e.g. a closure for execution in a separate process

This is not feasible in any multi-process system. If that’s an absolute requirement, the only way forward is to run your core emulator code in process, changing it to not rely on global variables.

(2) there is no simple way to share memory between the processes so I
don't have to keep copying data around.

It’s certainly possible to share memory between processes (for example, using shm_open man page). However, I think it’s fair to say that this is not “simple”.

Apropos UNIX-y primitives, one word of warning. jonprescott wrote:

However, as I have done many times in the past on a variety of Unix
variants, you can use fork only, which will create a new process which
is an exact copy of the calling process.

If you’re creating a library, and thus have no control over the type of program that your library is loaded in, this isn’t viable given your platform requirements. That’s because, on Apple platforms, doing a fork without an exec is only safe if you stick with the very lowest level frameworks. High-level frameworks lean heavily into Mach, and run into problems because the Mach port namespace [1] is not copied to the child process.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

[1] Apart from the special ports.