Simple example involving structured concurrency

Using Swift
1

I'm struggling to get structured concurrency to work and have two questions:
I'm generating a package with "swift package init --type executable" and do a "swift run" which works.
Trying to run with "swift run -Xfrontend -enable-experimental-concurrency" gives me an error:
"error: Missing value for '-c <configuration'"
So question number one: How to correctly add these options?

I managed to get around this with building with swiftc and running the resulting executable.
The synchronous code looks like this:

func doSomeCPUIntensiveStuff() -> Int {
return 33
}

func doSomeStuff() {
var result = Int
result.append(doSomeCPUIntensiveStuff())
result.append(doSomeCPUIntensiveStuff())
print(result)
}

doSomeStuff()

How to get this to work with Task.withGroup, async let or similar? Everything I tried ran into problems like
"error: 'async' in a function that does not support concurrency" or similar.

Thanks

2

If you want to use the concurrency support through SwiftPM, you'll need to use

swift run -Xswiftc -Xfrontend -Xswiftc -enable-experimental-concurrency

The meaning of -Xswiftc is "SwiftPM, please hand the following argument through to the Swift compiler without interpreting it". And similarly, -Xfrontend asks the Swift compiler to hand the next argument into the compiler's frontend.

Oh, and to actually run the binaries produces with the experimental concurrency support, you'll need a

export DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift/macosx/

(adapt the path to your Swift toolchain as necessary).

3 Likes
3

As the error suggests, you must be inside of an async function to use async features.

The "entry point" to enter an async context if you're not in one already is Task.runDetached, please read this proposal to learn about it: https://github.com/DougGregor/swift-evolution/blob/structured-concurrency/proposals/nnnn-structured-concurrency.md specifically, you could do this:

func slow() async -> Int { 42 } 
func someFunc() async -> Int { 
  async let x = slow()
  async let y = slow()
  return await x + y
}

Task.runDetached { // : () async -> T
  await someFunc()
}
1 Like
4

Thanks, that works.

This seems to be unnecessary, it runs fine on Ubuntu without that export.

Thanks, that was very helpful. What I have now is:

func slow() async -> Int { return 42 }

struct A {
        func run() {
                var y: [Int] = []
                _ = Task.runDetached {
                        let x = await slow()
                        y.append(x)
                }
                print(y)
        }
}

let a = A()
a.run()

which gives me the following warning;

warning: local var 'y' is unsafe to reference in code that may execute concurrently"

and prints an empty array.
My last question would be: How can I safely get x into y?

Thanks for the pointer. I did read it and I am trying to get a simple "Hello world" to run.

Again, thanks for all the helpful comments.

5

Sure, since you never waited for anything and the code continues right away and prints y before the detached task even had a chance to run and append.

Yeah the warning is correct -- what you're doing here is not generally thread-safe; you potentially modify the variable from a different task (thread). So... don't do that, and instead what you're trying to do is much simpler:

func slow() async -> Int { return 42 }

struct A {
  func run() async {
    var y: [Int] = []
      let x = await slow()
      y.append(x)
      print(y)
  }
}

let a = A()
Task.runDetached { 
  await a.run()
}

sleep(10)

Hope this helps.

I recommend giving a read to all the proposals and APIs - the async/await one is here: https://github.com/apple/swift-evolution/blob/main/proposals/0296-async-await.md and the structured concurrency linked above.

6

This prints nothing.

I did read (or at least glanced over) all of them. And I'm trying to get something that works. It may be clear for people who had worked on it some time but to a new user it is hard to understand when to use async/task groups/etc.
I have a working application that is basically doing the following:

  1. Generate some read-only global state.
  2. Spawn many threads using said state.
  3. Merge the results from the threads back.

The results are not Int and the merged result is not an Array, the code above is just to teach me how it works.
I have working code using (concurrent) DispatchQueue.async, DispatchGroup.enter and leave, and DispatchSemaphore.wait and signal, and am trying to port it to the new world order.

I have to problems at that point:

  1. Why do I see no print?
  2. I don't want to use sleep. (Trying to use handle.get() results in an "error: 'async' in a function that does not support concurrency" again.)
7

I absolutely do not mean to come from the perspective of “this is so clear!”, and I’m a bit sad you think that’s how I approached this thread... I honestly want to help out from first principles here, but the reality is that it is still early development of these features, and a lot of stuff is missing - which is what you’re hitting.

Specifically to your point of:

... not wanting to put a sleep there; Today, there’s no way for it. And that’s because there are missing features — the missing feature here specifically is that main() will be able to be async and then you won’t run into this trouble. Today though, you need to wait somehow for the detached task to perform it’s work, and that’s either by some locking/blocking or the simple wait I proposed to you here.

In the (near) future you’ll be able to:


@main
struct A {
  static func main() async {
    await ... 
  }
}

but that’s missing today AFAIR, thus the sleep workaround.

Please keep in mind that you’re looking at early development versions where everything is in-flight and many things are missing.

—

If you’re happy and confident with semaphores—as you mention—you can do the same from the detached task to avoid the sleep(). As soon as async-main lands (I think it has not landed yet, perhaps I’m mistaken though and it works already, you can check), the simpler version will just work™.

8

I understand that and I'm totally happy to wait to try things out. I was under the impression that things are somewhat working now and I'd just give it a try. I thought maybe early feedback from developers is helpful in giving you a different use case that you didn't have in mind.

That's not what I had in mind; I'm totally happy with main() being single threaded. What I'd love to do is:

  1. Single threaded main, setup some things, then...
  2. Spawn lot of threads to do work.
  3. Collect the results and work single threaded from then to the end.

I was surprised by a few things because I thought I could do just:

  1. Spawn work by something like Task.withGroup
  2. Somehow collect the results.

1 doesn't work because Task.withGroup can only be called from async functions,
2 is unclear to me.

Specifically I don't want main to be "async" and I wanted "async" to be confined in one place/struct/file.
Also after reading Notes on structured concurrency, or: Go statement considered harmful — njs blog I was under the impression that I don't have to do Task.runDetached.

Maybe it's just to early, maybe I have the wrong picture.

Anyway, thanks for your help.

9

This is resurrecting Java's public static void main problem (i.e., that you have to know what types are, what static functions are, etc., just to use concurrency features).

One of Swift's demonstrations of progressive disclosure was that you can write print("Hello, World!") and nothing else, and it's a working program.

I think the main message of this thread--and the numerous other threads about "I understand how to await from an async function, but how do I start?"--is that the async/await features are very lacking in terms of progressive disclosure. Is there no plan to allow top-level await?

6 Likes
10

Don’t shoot the messenger :slight_smile:

I hope so, but it’s up to @Douglas_Gregor et al. Status quo though is that not even the async main exists. I don’t know in which proposal this would be part of.

1 Like
11

:+1:

12

Isn't it specified in the Asynchronous programs section of the Structured Concurrency proposal?

3 Likes
13

Huh I see, that’s new and been added during the x-mas break (which is still ongoing, so I’ve not followed the pitch super-closely). So that’s nice that that’s being pitched there.

But in any case, for this thread of “get something working now” that doesn’t really change anything, that does not work yet.

14

This was always the plan, but finally got into the second draft of Structured Concurrency.

For now, you can use

runAsyncAndBlock { 
  /* async code */
}

at the top level. I've been doing this for my little example programs like the parallel map.

Doug

3 Likes
15

Thanks. I will try again in a few weeks. If I understand correctly it will be possible to do something like "runAsyncAndBlock" from within func/structs/classes, not only from top level.

16

I tried this countless times with different toolchains and different macOS versions and could never get it to work. I would export DYLD_LIBRARY_PATH=…, but then the DYLD_LIBRARY_PATH variable would not show up in the environment when I ran env.

It turns out that I had to disable System Integrity Protection because macOS doesn't pass DYLD_LIBRARY_PATH to SIP-protected processes when SIP is on.

You probably don't have to disable SIP completely. It should be enough to disable only SIP's debugging restrictions, as described in this comment on an rbenv GitHub issue from 2017: reboot into recovery mode, then type csrutil enable --without debug in Terminal. But I haven't tried that myself.

1 Like
17

To clarify, calling swift run from a toolchain directly does not work when SIP is enabled (presumably because swift qualifies as a SIP-protected process):

env DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-01-10-a.xctoolchain/usr/lib/swift/macosx/ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-01-10-a.xctoolchain/usr/bin/swift run

But building the binary first and then running the binary directly does work when SIP is enabled:

/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-01-10-a.xctoolchain/usr/bin/swift build
env DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2021-01-10-a.xctoolchain/usr/lib/swift/macosx/ .build/debug/AsyncAwait
4 Likes
18

It may be a good idea to set things up so that in swift.org toolchains, swift run itself would (optionally!) be able to set up the right DYLD_LIBRARY_PATH.

This would probably better be enabled via an explicit command-line option rather than turned on by default though. Stdlib binaries downloaded from swift.org (or compiled manually) generally aren't an exact match for the ones that come with any particular OS version -- so for example, some crucial frameworks may fail to load with dyld errors, or exhibit subtle (or not so subtle!) runtime issues. Sometimes these issues would only start triggering after installing a minor OS update.

(Overriding the OS-provided core Swift libraries is an extremely dangerous power tool. While this is definitely very useful while testing/trying new language features in tiny throwaway projects (or trying out a potential fix for a bug, learning about how Swift works, etc.), the OS is not designed to have one of its core components arbitrarily replaced -- it would be practically impossible to make sure that these custom binaries interoperate perfectly with the rest of the OS. It's a bit like installing a custom build of the Objective-C runtime -- I can't recall ever needing/wanting to do that as a Cocoa developer.

To be clear, we aren't breaking things intentionally, and for the most part simple test projects do work, but even tiny discrepancies in the list of exposed symbols or their behavior can have an outsized impact on interoperability, and the more things the project imports, the higher the chance of triggering something. The Swift stdlib is a living codebase, while OS releases are always tied to a very particular snapshot of it. E.g., consider how Foundation and the Swift stdlib cooperate on implementing bridging.)