# Data race when concurrently calling C function using global variables **URL:** \ **Category:** Using Swift\ **Tags:** concurrency, c, thread-sanitizer\ **Created:** [August 2, 2023, 8:52am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527 "2023-08-02T08:52:44Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![thebluepotato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/thebluepotato/32/9010_2.png) [@thebluepotato](https://forums.swift.org/u/thebluepotato)\ **Post date:** [August 2, 2023, 8:52am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/1 "2023-08-02T08:52:44Z")
I've been using a binary written in C ([pngnq-s9](https://github.com/ImageProcessing-ElectronicPublications/pngnq-s9)) to quantize PNG files to drastically reduce their size. Up to now, I've been calling the binary itself using Process/NSTask. Since multiple PNG files have to be processed, I've parallelized these tasks using a TaskGroup. Now, I want to see if calling the underlying C function directly from Swift would be faster. I have written a very thin Swift wrapper that imports the C library and wraps calls to the imported global C function (`pngnq()`) that quantizes a PNG file. However, this function calls another that heavily relies on global variables (to be found [here](https://github.com/ImageProcessing-ElectronicPublications/pngnq-s9/blob/master/src/neuquant32.c)), causing data races and EXC\_BAD\_ACCESS. Now, my question: is it possible to "isolate" or "prefix" calls to a C global function with global variables? I mean this in the sense that each call to the C function from my Swift wrapper would create its own context (thread? process?) and address space where the global variables are not in conflict with other threads? Anything short of rewriting the C library would be nice. Thanks in advance! EDIT: in short, is it possible to create a distinct memory space (like a subprocess) here? ---
**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\ **Post date:** [August 2, 2023, 9:54am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/2 "2023-08-02T09:54:38Z")
You can call those C functions on the single thread/queue but that would undo any parallelisation you've done above via the TaskGroups. It would be possible to spawn a new process for each PNG file, could be a bit heavy though. ---
**Author:** ![thebluepotato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/thebluepotato/32/9010_2.png) [@thebluepotato](https://forums.swift.org/u/thebluepotato)\ **Post date:** [August 2, 2023, 10:02am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/3 "2023-08-02T10:02:16Z")
The approach using the C binary directly does use Process/NSTask, so it's indeed spawning separate processes (and that's why they don't run into data races). However, are you saying there would be a way to spawn a process without calling an external executable or are you suggesting compiling the Swift wrapper as an executable and then calling it as a separate process? The second option (calling a binary built around the Swift wrapper) would not be very efficient, and I can't sacrifice performance by undoing parallelization. The performance gains I measured when calling the Swift wrapper instead of the external C binary were minute anyway, so I think I'll need to rewrite the whole thing (in Swift though) if I want things to run on iOS, where Process/NSTask is not available. ---
**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\ **Post date:** [August 2, 2023, 10:18am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/4 "2023-08-02T10:18:39Z")
I reread what you've said and it indeed looks that processes are unavoidable in your case unless of course you can tolerate processing all PNGs sequentially in your main app process. You can try using "thumbnail extension" (a subclass of QLThumbnailProvider) – that runs in a separate process and is availble on both macOS and iOS. A bit roundabout way but could do the trick. ---
**Author:** ![ibex](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ibex/32/30439_2.png) [@ibex](https://forums.swift.org/u/ibex)\ **Post date:** [August 2, 2023, 12:25pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/5 "2023-08-02T12:25:13Z")
> [@thebluepotato](#): > > EDIT: in short, is it possible to create a distinct memory space (like a subprocess) here? The short answer is no. In addition to spawning off a process, you can also utilise an XPC service (on macOS only.) Just in case you are not familiar with it, an XPC service also runs in its own process and is easier to communicate with. ---
**Author:** ![thebluepotato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/thebluepotato/32/9010_2.png) [@thebluepotato](https://forums.swift.org/u/thebluepotato)\ **Post date:** [August 2, 2023, 12:36pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/6 "2023-08-02T12:36:24Z")
I think in the end it’ll be easier to rewrite this using Swift and the Accelerate framework (the original C implementation is quite bare-bones), which might lead to even more optimizations! ---
**Author:** ![wadetregaskis](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wadetregaskis/32/8494_2.png) [@wadetregaskis](https://forums.swift.org/u/wadetregaskis)\ **Post date:** [August 2, 2023, 2:31pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/7 "2023-08-02T14:31:13Z")
Rewriting it in Swift would be great for the Swift community, but looks like it'd be quite a bit of work. An easier alternative is to fix the C library - e.g. move all those globals into a context structure that the caller provides to the library. Are there other PNG libraries you could use, though? That one hasn't been touched in over four years. Perhaps there's a more active equivalent, that might already have removed the globals? ---
**Author:** ![thebluepotato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/thebluepotato/32/9010_2.png) [@thebluepotato](https://forums.swift.org/u/thebluepotato)\ **Post date:** [August 2, 2023, 3:13pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/8 "2023-08-02T15:13:39Z")
AFAIK `pngnq-s9` is a descendant (or cousin?) of `pngquant` and the related `libimagequant`, which are still actively maintained. They used to be written in C, but now are written in Rust, with a C interface. While usable from a Swift program, I don’t think could use the Rust version (pre-compiled or source) within my Swift Package. But from brief testing that I had actually already done in the past, mistakenly thinking that `pngquant` was less powerful, it’s actually way more efficient. ---
**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\ **Post date:** [August 2, 2023, 4:02pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/9 "2023-08-02T16:02:51Z")
> [@wadetregaskis](#): > > move all those globals into a context structure that the caller provides to the library. I like this approach. Make a global variable: ```swift typedef struct { } G; G gv; G* g = &gv; ``` Then move the first global variable into it: ```swift // static int search_cache[SEARCH_CACHE_SIZE]; ... typedef struct { int search_cache[SEARCH_CACHE_SIZE]; } G; ``` and change the code to use `g->search_cache` instead of `search_cache` – it should still compile and work as before. Repeat for all global variables. After the end of this exercise you'll only have a single global variable `g`. Then comment that global variable out and whenever you need `g` pass it as a parameter. Note there could be static variables inside functions – those should be treated as global variables (and if there are name collisions between them resolve those collisions e.g. by prefixing those names appropriately. ---
**Author:** ![odmir](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/odmir/32/4784_2.png) [@odmir](https://forums.swift.org/u/odmir)\ **Post date:** [August 2, 2023, 4:58pm UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/10 "2023-08-02T16:58:40Z")
you could try using @taylorswift’s pure swift [PNG library](https://github.com/tayloraswift/swift-png), it might fulfill your needs to reduce file size and be fast ---
**Author:** ![tera](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/tera/32/38201_2.png) [@tera](https://forums.swift.org/u/tera)\ **Post date:** [August 3, 2023, 12:44am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/11 "2023-08-03T00:44:50Z")
I wonder how the following sample code compares (speed wise) to what you are doing now. Ignore the poor quantisation quality. It works in both iOS and macOS and should be thread safe (parallelizable). > **Sample code that uses platform PNG conversion via ImageIO** > > ```swift > extension Data { > // works on Apple platforms like macOS and iOS > func quantizedImageData(quality /*TODO*/: Int = 0) -> Data { > let imageData = self > let source = CGImageSourceCreateWithData(imageData as CFData, nil)! > let type = CGImageSourceGetType(source)! > let count = CGImageSourceGetCount(source) > precondition(count == 1) // TODO: suport multi images later > let image = CGImageSourceCreateImageAtIndex(source, 0, nil)! > let bitmapInfo = image.bitmapInfo > let alpha = image.alphaInfo > let newAlphaInfo: CGImageAlphaInfo = switch alpha { > case .last: .noneSkipLast // should it be premultipliedLast ? > default: fatalError("TODO") > } > let (width, height, stride) = (image.width, image.height, image.bytesPerRow) > let newBitmapInfo = CGBitmapInfo(rawValue: (bitmapInfo.rawValue & ~CGBitmapInfo.alphaInfoMask.rawValue) | newAlphaInfo.rawValue) > > let context = CGContext(data: nil, width: width, height: height, bitsPerComponent: image.bitsPerComponent, bytesPerRow: stride, space: image.colorSpace!, bitmapInfo: newBitmapInfo.rawValue)! > context.draw(image, in: CGRect(x: 0, y: 0, width: width, height: height)) > > let p = context.data!.assumingMemoryBound(to: UInt32.self) > > precondition(image.bitsPerComponent == 8 && stride >= width*4, "TODO other formats") > let pixelStride = stride/4 > > // TODO: do something better (dithering, halftoning, accelerate, CoreImage, etc) > for y in 0 ..< height { > let line = y * pixelStride > for x in 0 ..< width { > p[line + x] &= 0xFFE0E0E0 // some simple colour quantization > } > } > > let newImage = context.makeImage()! > let newCFData = CFDataCreateMutable(kCFAllocatorDefault, 0)! > let destination = CGImageDestinationCreateWithData(newCFData, type, 1, nil)! > CGImageDestinationAddImage(destination, newImage, nil) > let done = CGImageDestinationFinalize(destination) > precondition(done) > let newData = newCFData as CFData as Data > return newData > } > } > > ``` ---
**Author:** ![thebluepotato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/thebluepotato/32/9010_2.png) [@thebluepotato](https://forums.swift.org/u/thebluepotato)\ **Post date:** [August 3, 2023, 8:30am UTC](https://forums.swift.org/t/data-race-when-concurrently-calling-c-function-using-global-variables/66527/12 "2023-08-03T08:30:22Z")
I checked it out and it's mostly a replacement for `libpng`, so reading and writing PNGs (with compression), not quantization. The API also feels very un-Swifty to me, even though it's 100% Swift.