Structured concurrency bug in Xcode 14.3

makadaw · April 25, 2023, 6:40pm

Hello everyone,

We've been using structured concurrency successfully for the last two months, but after upgrading to Xcode 14.3, we've been experiencing crashes on iOS 15.0/15.1 that we suspect may be related to the new release.

Specifically, we've noticed that some of our code that previously worked well with structured concurrency is now causing crashes on iOS 15.0. We've tried debugging the issue but have been unable to find a solution. As crashes happen only in iOS 15.x our guess was that there possible issue with a back-deploying.

We wanted to reach out to the community to see if anyone else has experienced similar issues or if anyone has any suggestions on how to resolve the problem.

The stack traces are also not super detailed:

EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000017
Crashed: com.apple.root.user-initiated-qos.cooperative
0  libdispatch.dylib              0x2a6d4 _voucher_xref_dispose + 108
1  libdispatch.dylib              0x6ed8 _dispatch_set_priority_and_voucher_slow + 272
2  libdispatch.dylib              0x16638 _dispatch_worker_thread2 + 228
3  libsystem_pthread.dylib        0x10b8 _pthread_wqthread + 228
4  libsystem_pthread.dylib        0xe94 start_wqthread + 8

EXC_BAD_ACCESS KERN_PROTECTION_FAILURE 0x000000016ddec000
Crashed: com.apple.root.user-initiated-qos.cooperative
0  libdispatch.dylib              0x15d38 _dispatch_root_queue_drain + 244
1  libdispatch.dylib              0x165f8 _dispatch_worker_thread2 + 164
2  libsystem_pthread.dylib        0x10b8 _pthread_wqthread + 228
3  libsystem_pthread.dylib        0xe94 start_wqthread + 8

Another crash that point to back deploying code is this:

EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000010
Crashed: com.apple.root.user-initiated-qos.cooperative
0  libswiftCore.dylib             0x3128ac swift_task_exitThreadLocalContext + 48
1  RedditApp                      0x3ef18c8 swift::AsyncTask::waitFuture(swift::AsyncTask*, swift::AsyncContext*, void (swift::AsyncContext* swift_async_context) swiftasynccall*, swift::AsyncContext*, swift::OpaqueValue*) + 782476
2  RedditApp                      0x3ef1b18 swift::swift56override_swift_task_future_wait_throwing(swift::OpaqueValue*, swift::AsyncContext*, swift::AsyncTask*, void (swift::AsyncContext* swift_async_context, void* swift_context) swiftasynccall*, swift::AsyncContext*, void (swift::OpaqueValue*, swift::AsyncContext* swift_async_context, swift::AsyncTask*, void* swift_context, swift::AsyncContext*) swiftasynccall*) + 783068
3  libswift_Concurrency.dylib     0x2e910 swift::runJobInEstablishedExecutorContext(swift::Job*) + 336
4  libswift_Concurrency.dylib     0x2f694 swift_job_runImpl(swift::Job*, swift::ExecutorRef) + 68
5  libdispatch.dylib              0x48d5c _dispatch_root_queue_drain + 328
6  libdispatch.dylib              0x4951c _dispatch_worker_thread2 + 160
7  libsystem_pthread.dylib        0x1d40 _pthread_wqthread + 224
8  libsystem_pthread.dylib        0x18a8 start_wqthread + 8

Is anyone saw something similar?

sjavora · April 26, 2023, 11:00am

We're seeing this too, iOS 15.0, 15.1, similarly unhelpful stack traces.

KERN_PROTECTION_FAILURE at 0x16fec4000.

Thread 8 Crashed:
0   libdispatch.dylib               0x300d23d38         _dispatch_root_queue_drain
1   libdispatch.dylib               0x300d23d2c         _dispatch_root_queue_drain
2   libdispatch.dylib               0x300d245f4         _dispatch_worker_thread2
3   libsystem_pthread.dylib         0x3e2b430b4         _pthread_wqthread

One of the stack traces points to a URLSession shim that we're using since we support iOS 14, but still that only tells us that it has to do with concurrency...

Attempted to dereference null pointer.

Thread 5 Crashed:
0   libobjc.A.dylib                 0x3337b702c         objc_msgSend
1   libsystem_trace.dylib           0x3376f5760         _os_log_fmt_flatten_object
2   libsystem_trace.dylib           0x3376f3650         _os_log_impl_flatten_and_send
3   libsystem_trace.dylib           0x3376f1824         _os_log
4   libsystem_trace.dylib           0x3376f6b84         _os_log_impl
5   CFNetwork                       0x306313ddc         CFURLCacheSetMemoryCapacity
6   appname                         0x20156cab4         [inlined] URLSessionTaskActor.start (URLSession+Async.swift:38)
7   appname                         0x20156cab4         NSURLSession.data (URLSession+Async.swift:16)
8   libswift_Concurrency.dylib      0x3d12eca58         swift::runJobInEstablishedExecutorContext

sjavora · April 26, 2023, 11:02am

I thought that was only for iOS 13 and 14...

pcifani · May 1, 2023, 4:17pm

Same thing here, we had to bump the deployment target to iOS 15.2 because of this. And it looks like there’s no way to work around it

makadaw · May 1, 2023, 5:11pm

Interesting that iOS 14 is fine. We also saw some crashes on iOS16 betas, but not on a stable versions.

lukeredpath · May 3, 2023, 12:22am

Just adding to the list of people seeing this crash in our logs, similar stack traces only affecting 15.1 users. Fortunately only a small minority of our users, I may have to consider bumping our deployment target to 15.2.

ctp · May 3, 2023, 3:02am

Do you (or anyone else) have a Feedback Request that's been filed with Apple that we could reference?

waltflanagan · May 3, 2023, 1:15pm

Hi Chris, I filed FB12163887 with a few of our own crash reports also with links back to this thread.

yyarashevich · May 4, 2023, 8:08am

Could someone who experiences random crashes when compiling with Swift 5.8 please try running the application on the iOS simulator using libgmalloc? With the help of libgmalloc, I have previously managed to trace down the exact function in libswiftConcurrency that corrupts memory, which led to further random crashes. Perhaps it can also help this time.

etcwilde · May 5, 2023, 3:44pm

I'm not certain that all of the crashes listed here are from the same cause.

The original crash is happening while awaiting a swift task.
@makadaw, are you able to show the code around the crash, or provide a minimal reproducer?
I'm interested in seeing how the task that is crashing was created. (is it an async-let, taskGroup, Task, or detached task?)

I'm not entirely sure what is happening in the second stack trace, though we have URLSessionTask involved here, and I don't see much in the way of the swift concurrency runtime involvement, though more context would be helpful. It's sort of the same for the screenshot of xcode. It's saying it's crashing in the URLSessionTask resume, which isn't a Swift Task, it's a URLSessionTask.

I thought that was only for iOS 13 and 14...

Interesting that iOS 14 is fine.

iOS 14 has the backdeploy concurrency library. iOS 15 has the OS runtime concurrency library. They're not quite the same, unfortunately. Both have runtime fixes that come from the 5.6 compatibility library though, which is where the swift56override_... part is coming from in the initial stack-trace.

sjavora · May 5, 2023, 4:32pm

Most of the crashes (taken from Xcode's Organizer) look like this:

Thread 11 Crashed:
0   libdispatch.dylib             	0x00000001802b4d38 _dispatch_root_queue_drain + 244 (inline_internal.h:2639)
1   libdispatch.dylib             	0x00000001802b4d30 _dispatch_root_queue_drain + 236 (inline_internal.h:603)
2   libdispatch.dylib             	0x00000001802b55f8 _dispatch_worker_thread2 + 164 (queue.c:6930)
3   libsystem_pthread.dylib       	0x00000001f11ba0b8 _pthread_wqthread + 228 (pthread.c:2554)
4   libsystem_pthread.dylib       	0x00000001f11b9e94 start_wqthread + 8 (:-1)

But there are other significant crashes that look kind of like:

Thread 5 Crashed:
0   libobjc.A.dylib               	0x000000019baad230 objc_autorelease + 16 (objc-object.h:899)
1   CFNetwork                     	0x0000000183d351c4 -[NSURLSessionTask resume] + 148 (SessionTask.mm:786)
2   app name                      	0x0000000104868ab8 URLSessionTaskActor.start(_:) + 28 (URLSession+Async.swift:38)
3   app name                      	0x0000000104868ab8 closure #1 in closure #1 in closure #1 in NSURLSession.data(for:) + 64 (URLSession+Async.swift:16)
4   app name                      	0x0000000104869a39 partial apply for closure #1 in closure #1 in closure #1 in NSURLSession.data(for:) + 1
5   app name                      	0x00000001048691a5 specialized thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A) + 1 (<compiler-generated>:0)
6   app name                      	0x0000000104869a45 partial apply for specialized thunk for @escaping @callee_guaranteed @Sendable @async () -> (@out A) + 1
7   libswift_Concurrency.dylib    	0x00000001ff1715e5 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1 (Task.cpp:369)

URLSessionTaskActor is something we use since we need to support iOS 14, but wanted to use async with URLSession.

makadaw · May 5, 2023, 4:45pm

Hey @etcwilde

For the first two stacktraces - no. We can see this reports only in the Crashalytics and can't reproduce it locally. For the third one with swift_task_exitThreadLocalContext this is a normal Task.

MontakOleg · May 6, 2023, 1:47pm

Hey @etcwilde
We have similar crashes.

From Crashlytics:

Crashed: com.apple.root.utility-qos.cooperative
0  libdispatch.dylib              0x15d38 _dispatch_root_queue_drain + 244
1  libdispatch.dylib              0x165f8 _dispatch_worker_thread2 + 164
2  libsystem_pthread.dylib        0x10b8 _pthread_wqthread + 228
3  libsystem_pthread.dylib        0xe94 start_wqthread + 8

Crashed: com.apple.root.utility-qos.cooperative
0  libobjc.A.dylib                0x5230 objc_autorelease + 16
1  CFNetwork                      0x151c4 CFURLCacheSetMemoryCapacity + 19068
2  libswiftFoundation.dylib       0x7ac2c closure #1 in closure #2 in NSURLSession.data(for:delegate:) + 584
3  libswift_Concurrency.dylib     0x5ea8 partial apply for closure #1 in withCheckedThrowingContinuation<A>(function:_:) + 112
4  libswift_Concurrency.dylib     0x5fa4 withUnsafeThrowingContinuation<A>(_:) + 208
5  libswift_Concurrency.dylib     0x3cfec swift::runJobInEstablishedExecutorContext(swift::Job*) + 276
6  libswift_Concurrency.dylib     0x3d890 swift_job_run + 188
7  libdispatch.dylib              0x14f48 _dispatch_root_queue_drain + 388
8  libdispatch.dylib              0x15768 _dispatch_worker_thread2 + 164
9  libsystem_pthread.dylib        0x1174 _pthread_wqthread + 228
10 libsystem_pthread.dylib        0xf50 start_wqthread + 8

Crashed: com.apple.root.utility-qos.cooperative
0  libdispatch.dylib                   0x54fd0 voucher_activity_create_with_data$VARIANT$armv81 + 276
1  libsystem_trace.dylib               0x208c _os_activity_create_addr + 892
2  libsystem_containermanager.dylib    0x2480 container_create_or_lookup_app_group_path_by_app_group_identifier + 108
3  Foundation                          0xbd4c8 -[NSFileManager containerURLForSecurityApplicationGroupIdentifier:] + 36
4  appname                             0xcac53c specialized static FileDownloaderStorage.cacheDiskPath.getter + 4316415292
5  appname                             0xcac9a8 specialized static FileDownloaderStorage.getCacheFilePath(for:) + 4316416424
6  appname                             0xcab4cc specialized static FileDownloader.dataAsync(for:cachePolicy:storage:) + 4316411084
7  libswift_Concurrency.dylib          0x2aa40 swift::runJobInEstablishedExecutorContext(swift::Job*) + 196
8  libswift_Concurrency.dylib          0x2b138 swift_job_run + 152
9  libdispatch.dylib                   0x48488 _dispatch_root_queue_drain + 320
10 libdispatch.dylib                   0x48c5c _dispatch_worker_thread2 + 160
11 libsystem_pthread.dylib             0xefc _pthread_wqthread + 224
12 libsystem_pthread.dylib             0xaa4 start_wqthread + 8

From Xcode Organizer:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000020
Exception Codes: 0x0000000000000001, 0x0000000000000020
VM Region Info: 0x20 is not in any region.  Bytes before following region: 4377247712
      REGION TYPE                 START - END      [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                   104e78000-104e7c000 [   16K] r-x/r-x SM=COW  ...le.app/Simple
Exception Note:  EXC_CORPSE_NOTIFY
Termination Reason: SIGNAL 11 Segmentation fault: 11
Terminating Process: exc handler [20029]

Triggered by Thread:  11


Thread 11 name:
Thread 11 Crashed:
0   libobjc.A.dylib               	0x000000019ac12230 objc_autorelease + 16 (objc-object.h:899)
1   CFNetwork                     	0x0000000182e9a1c4 -[NSURLSessionTask resume] + 148 (SessionTask.mm:786)
2   libswiftFoundation.dylib      	0x0000000186daec2c $sSo12NSURLSessionC10FoundationE4data3for8delegateAC4DataV_So13NSURLResponseCtAC10URLRequestV_So0A12TaskDelegate_pSgtYaKFAH_AJtyYaKXEfU0_yScCyAH_AJts5Error_pGXEfU_ + 584 (URLSession.swift:283)
3   libswift_Concurrency.dylib    	0x00000001fe960ea8 $ss31withCheckedThrowingContinuation8function_xSS_yScCyxs5Error_pGXEtYaKlFySccyxsAC_pGXEfU_TA + 112 (CheckedContinuation.swift:292)
4   libswift_Concurrency.dylib    	0x00000001fe960fa4 $ss30withUnsafeThrowingContinuationyxySccyxs5Error_pGXEYaKlF + 208 (PartialAsyncTask.swift:283)
5   libswift_Concurrency.dylib    	0x00000001fe960d19 $ss31withCheckedThrowingContinuation8function_xSS_yScCyxs5Error_pGXEtYaKlFTQ0_ + 1 (CheckedContinuation.swift:291)
6   libswiftFoundation.dylib      	0x0000000186dae859 $sSo12NSURLSessionC10FoundationE4data3for8delegateAC4DataV_So13NSURLResponseCtAC10URLRequestV_So0A12TaskDelegate_pSgtYaKFAH_AJtyYaKXEfU0_TQ0_ + 1 (URLSession.swift:273)
7   libswiftFoundation.dylib      	0x0000000186db3871 $sSo12NSURLSessionC10FoundationE4data3for8delegateAC4DataV_So13NSURLResponseCtAC10URLRequestV_So0A12TaskDelegate_pSgtYaKFAH_AJtyYaKXEfU0_TATQ0_ + 1 (<compiler-generated>:0)
8   libswiftFoundation.dylib      	0x0000000186daee61 $s10Foundation4DataVSo13NSURLResponseCs5Error_pIgHoozo_AC_AEtsAF_pIegHrzo_TRTQ0_ + 1 (<compiler-generated>:0)
9   libswiftFoundation.dylib      	0x0000000186db7a61 $s10Foundation4DataVSo13NSURLResponseCs5Error_pIgHoozo_AC_AEtsAF_pIegHrzo_TRTATQ0_ + 1
10  libswift_Concurrency.dylib    	0x00000001fe975f71 $ss27withTaskCancellationHandler9operation8onCancelxxyYaKXE_yyYbXEtYaKlFTQ0_ + 1 (TaskCancellation.swift:42)
11  libswiftFoundation.dylib      	0x0000000186dae525 $sSo12NSURLSessionC10FoundationE4data3for8delegateAC4DataV_So13NSURLResponseCtAC10URLRequestV_So0A12TaskDelegate_pSgtYaKFTQ0_ + 1 (URLSession.swift:268)
12  appname                        	0x000000010509c331 specialized static FileDownloader.makeRequest(url:etagValue:cachedContent:storage:) + 1
13  appname                        	0x000000010509ce45 specialized static FileDownloader.dataAsync(for:cachePolicy:storage:) + 1
14  appname                        	0x0000000105098a9d closure #1 in closure #2 in static FileDownloader.data(for:cachePolicy:storage:completion:) + 1
15  appname                        	0x000000010509bd79 partial apply for closure #1 in closure #2 in static FileDownloader.data(for:cachePolicy:storage:completion:) + 1
16  appname                        	0x000000010509d3d5 specialized closure #1 in DeduplicatedPool<>.async(key:operation:) + 1
17  appname                        	0x000000010509d3e9 partial apply for specialized closure #1 in DeduplicatedPool<>.async(key:operation:) + 1
18  libswift_Concurrency.dylib    	0x00000001fe99d5e5 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1 (Task.cpp:369)

We have only one class where we use async/await, and crashes started after updating to Xcode 14.3.

To not miss any details, I will provide full code, please look at it if it can help: https://gist.github.com/MontakOleg/9939c35ff7882f86b15991059d642b8b

Douglas_Gregor · May 8, 2023, 5:30pm

Hey all,

Thanks for the reports. We've tracked down the issue to an incompatibility with very specific version ranges. If you're curious about the details about what happened and what the fix looks like, see the pull request here.

I've kicked off a toolchain build. Once that's done, I'll point to its artifacts and provide instructions here on how you can try out the fix.

[EDIT: The toolchain build is available. All you really need from it are the libswiftCompatibility56.a files, where the bug fix is]

Doug

Kyle-Ye · May 8, 2023, 6:03pm

After it was merged, will there be a new Xcode version we can use? eg. Xcode 14.3.1

Or we'll have to wait for the WWDC 2023 / Xcode 15.

makadaw · May 9, 2023, 9:06am

Thanks Doug for a fast response!

lukeredpath · May 9, 2023, 10:43am

@Douglas_Gregor thanks for the update.

As well as the spike in crashes with backtraces similar to the above, I've also noticed a spike in WatchdogTermination errors in our Sentry crash reports with our last release - before that release they happened very infrequently (maybe half a dozen reports a month?). Since the last release we've had around a dozen per day, all iOS 15.1, 15.1.1 and 15.2 and there seems to be a correlation in user IDs between these errors and the other crashes.

Could this be related?

Douglas_Gregor · May 10, 2023, 9:32pm

iOS 15.2 was not affected by this bug; only iOS [15.0, 15.2).

Doug

tim1724 · May 10, 2023, 10:07pm

I'm not sure I'd assume all Swift programmers have studied enough math to be familiar with the standard mathematical notation for intervals. Perhaps it would more helpful in this context to use Swift half-open range syntax: 15..<15.2

august · May 14, 2023, 3:32pm

@Douglas_Gregor
Please advise on how we can use the updated toolchain. Do we replace the libswiftCompatibility56.a that comes with the Xcode toolchain with the one from the new toolchain? There is quite a size difference between the two (4.7 MB vs 78 KB).