Bundle stdlib with swift binary?

I have a swift binary I am packaging in an electron app. The binary simply connects to AVFoundation and serves as a wrapper to enable recording screen and microphone (GitHub - slashauth/aperture-node: Record the screen on macOS from Node.js forked from GitHub - wulkano/aperture-node: Record the screen on macOS from Node.js). I then bundle this binary into an electron app and distribute it. As such, I attempt to compile and link the swift stdlib to the binary so that I do not require the consumer to have the stdlib installed. This appears to not work with a warning:

warning: Swift compiler no longer supports statically linking the Swift libraries. They're included in the OS by default starting with macOS Mojave 10.14.4 beta 3. For macOS Mojave 10.14.3 and earlier, there's an optional Swift library package that can be downloaded from "More Downloads" for Apple Developers at https://developer.apple.com/download/more/

When distributing the app, I am blocked from running the code. Specifically, when the app starts I receive:

XprotectService: [com.apple.xprotect:xprotect] File /Applications/Debrief.app/Contents/Resources/app.asar.unpacked/node_modules/@getdebrief/aperture/aperture failed on rPathCmd /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libswiftAVFoundation.dylib (rpath resolved to: (path not found), bundleURL: /Applications/Debrief.app)

This made me infer that the binary is actually dynamically linked, and using otool, I found it sure is:

otool -L aperture
aperture:
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2022.20.117)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1463.2.1)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreMediaIO.framework/Versions/A/CoreMediaIO (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1770.255.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/usr/lib/swift/libswiftCoreMIDI.dylib (compatibility version 1.0.0, current version 2.0.0, weak)
	/usr/lib/swift/libswiftUniformTypeIdentifiers.dylib (compatibility version 1.0.0, current version 633.0.2, weak)
	@rpath/libswiftAVFoundation.dylib (compatibility version 1.0.0, current version 2000.5.4, weak)
	@rpath/libswiftAppKit.dylib (compatibility version 1.0.0, current version 103.10.0, weak)
	@rpath/libswiftCloudKit.dylib (compatibility version 1.0.0, current version 962.0.0, weak)
	@rpath/libswiftCore.dylib (compatibility version 1.0.0, current version 1200.2.41)
	@rpath/libswiftCoreAudio.dylib (compatibility version 1.0.0, current version 1.1.0, weak)
	@rpath/libswiftCoreData.dylib (compatibility version 1.0.0, current version 3.0.0, weak)
	@rpath/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 1.6.0, weak)
	@rpath/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 2.0.0)
	@rpath/libswiftCoreImage.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftCoreLocation.dylib (compatibility version 1.0.0, current version 5.0.0, weak)
	@rpath/libswiftCoreMedia.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
	@rpath/libswiftDispatch.dylib (compatibility version 1.0.0, current version 4.40.2, weak)
	@rpath/libswiftFoundation.dylib (compatibility version 1.0.0, current version 20.0.0)
	@rpath/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftMetal.dylib (compatibility version 1.0.0, current version 1.3.1, weak)
	@rpath/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftQuartzCore.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	@rpath/libswiftXPC.dylib (compatibility version 1.0.0, current version 1.1.0, weak)
	@rpath/libswiftsimd.dylib (compatibility version 1.0.0, current version 1.3.0, weak)

My question is: Does a way exist to bundle the stdlib with the app for distribution? If not, can anyone suggest another path to create a native binary for interacting with AVFoundation, please? Failing that, any other threads you could suggest for me to pull on?

1 Like

On Apple platforms, statically linking the standard library is not supported and may lead to runtime failures in the future even if you can get it working on a currently-shipping OS.

If you're willing to accept the binary-size impact of statically linking the standard library, why is the supported approach of dynamically linking it unacceptable? What's the overhead you're measuring?

2 Likes

If i understand correctly, the last few MacOS and iOS operating system upgrades install the Swift runtime support as part of the system, starting with Swift 5.1, which I think is Mojave on MacOS. Unless you're trying to support an older operating system version, the stdlib should be loaded as part of the system. That was the point, I think, of ABI stability. Your customers should have it already installed as a dynamic library.

1 Like

Ah very interesting!

The binary size is something I'm not concerned with right now (although I'll need to address it down the road.)

So if I'm understanding you correctly you're recommending to package the swift stdlib in my Frameworks deployed with the app? This sounds like it might work. I'm going to give it a shot!

That makes complete sense. For whatever reason though compiling from the commandline leaves the @rpath directive pointing do the stdlib resources rather than going to the system available ones. This feels strange to me and there may be a way around it but I haven't found it yet. Any thoughts on this one?

Just wanted to say I found a fix here thanks to some of the thoughts in this thread (sorry I can't mention you because I'm a new user and it blocks me from doing so for spam reasons, presumably)

The issue was in the rpath not being updated to refer to a relative path (in my case based on @executable_path). I did some reverse engineering of other electron apps to see how they handled this and they indeed package swift dylib with their app. Their LC_RPATH is then updated to refer to a relative path from the executable.

In order to accomplish the latter part, I used install_name_tool with the -rparth command to replace the reference to my system's dylib path to the relative path:

install_name_tool -rpath /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx @executable_path/../../../swift-libs <binary_name>

Once this was complete and the package setup properly, everything operates as expected.

1 Like

There's a standard way this is done when building Swift executables so that it uses the system library when available and otherwise falls back on the library from your bundle. Essentially, the libraries are linked with @rpath-relative paths, and the executable's rpath is configured to search first in /usr/lib/swift and then relative to the executable. Please try building a trial Swift executable with an older deployment target (one prior to Swift being included in the OS) and verify that your executable seems to match. If you accidentally set up your executable to always use the standard library from your bundle, it may stop working on future operating systems.

There is an option in Xcode build settings: "Always embed swift standard libraries"
But it should still run fine on macOS 10.14.4+ without that.

Maybe I'm missing something on my system (or I installed something incorrectly) but my /usr/lib/swift contains the following:

% ls -al /usr/lib/swift
total 7440
drwxr-xr-x   7 root  wheel      224 Jan  1  2020 .
drwxr-xr-x  37 root  wheel     1184 Jan  1  2020 ..
drwxr-xr-x   7 root  wheel      224 Jan  1  2020 CreateMLInternal
-rwxr-xr-x   1 root  wheel  8176144 Jan  1  2020 libswiftCreateML.dylib
-rwxr-xr-x   1 root  wheel   648384 Jan  1  2020 libswiftDemangle.dylib
-rwxr-xr-x   1 root  wheel  1219312 Jan  1  2020 libswiftRemoteMirror.dylib
-rwxr-xr-x   1 root  wheel   342848 Jan  1  2020 libswiftXCTest.dylib

Perhaps am I missing something here? My fear is if my system can be in this state, any of my users could end up in this state too!

The Swift standard library itself is actually in the shared cache. I don't know exactly what's going with libswiftAVFoundation, but it probably just has a different deployment rule.

Again, you need to set this up so that it picks up libraries from the system when present, which is mostly a matter of matching Swift's normal linking behavior, assuming you can't actually just use Swift to link.

I just hit this same issue with Aperture and Apple Notarization as well.

On Xcode 12.3 the option to Always embed swift standard libraries seems to be gone. Building from the swift command line used to work but as the OP mentioned, that functionality has been removed as well.

Is the guidance just that, if you use a swift library that doesn't live in /usr/lib/swift then you have to package it yourself?

I just hit this same issue with Aperture and Apple Notarization as
well.

Can you be more specific about the exact symptoms you’re seeing? Presumably you’re building an Aperture plug-in? If so, supporting deployment targets prior to 10.14.4 is tricky because a plug-in can’t carry around its own runtime [1].

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

[1] In 10.14.4 the runtime is built in to the OS and every Swift program must use it. On prior systems each Swift program carries its own runtime. This is fine for apps and app extensions, where you control the main executable. However, plug-ins run into problems because, if your host app uses Swift itself, or loads another plug-ins built with a different runtime, you end up with two runtimes loaded in the same process and… well… that ends badly.

So our workflow is we use GitHub - wulkano/aperture-node: Record the screen on macOS from Node.js to build a binary. Under the hood this runs swift build --configuration=release. When I built this on Big Sur, what used to be a 10MB binary became about 300kb.

We distribute this binary via an Electron app, and on Big Sur, when trying to run it, we got the 'Unidentified Developer' popup from Gatekeeper, although looking at the console.app output it was the same XCodeProtect issue that the OP linked above, with the AVFoundation dylib.

I've resolved this for now by shipping every dylib that uses @rpath from the output of otool -L aperture and switching the Xcode @rpath from otool -l aperture to a relative path from where the binary gets unpacked to where the swift libraries get unpacked.

But it does seem that things like libSwiftAVFoundation.dylib do not exist under /usr/lib/swift so it's dependent on the Xcode rpath?

But it does seem that things like libswiftAVFoundation.dylib do not
exist under /usr/lib/swift so it’s dependent on the Xcode rpath?

No. On systems with the built-in Swift runtime, the Swift libraries, including libswiftAVFoundation.dylib, are part of the OS itself. On macOS 11 you don’t see them on disk because they’re part of the dyld shared cached (see the discussion in macOS Big Sur 11.0.1 Release Notes; search for 62986286). However, they do show up at runtime. Consider this:

  1. On macOS 11, there’s no libswiftAVFoundation.dylib in /usr/lib/swift:

     % sw_vers
     ProductName:	macOS
     ProductVersion:	11.1
     BuildVersion:	20C69
     % ls /usr/lib/swift | cat
     CreateMLInternal
     libswiftCreateML.dylib
     libswiftDemangle.dylib
     libswiftRemoteMirror.dylib
     libswiftXCTest.dylib
    
  2. But, if you build a Swift app that uses AVFoundation and then run it, you’ll see it loaded into memory:

     (lldb) image list 
     …
     [  8] … /usr/lib/swift/libswiftAVFoundation.dylib 
    

As to your big picture problem, I’ve seen this before and there are a bunch of inter-related issues in play:

  • Swift Package Manager builds are leaving a meaningless rpath entry in the library. I’m pretty sure there’s a bug on file about this but I don’t have the number handy.

  • Gatekeeper (on modern systems, 10.15 and above IIRC) gets grumpy when it sees that rpath entry.

  • This Gatekeeper check is intended to prevent a process from accidentally loading code from an unexpected location. It’s only meant to apply to executables without library validation (because library validation would prevent the process from loading such rogue code). However, a bug in early 10.15.x systems causes Gatekeeper to not recognise that library validation is enabled implicitly via the hardened runtime.

My experience is that most folks who see this issue do so because they’ve explicitly disabled library validation via the com.apple.security.cs.disable-library-validation entitlement. Is that the case here?

If so, the easiest fix is to not do that. Unless you actually need to load code from other third-party developers. And, to be clear, I’m not talking about third-party code that you build and incorporate into your app, because that should be signed as you. The most common reason to disable library validation is that you need to load in-process plug-ins from other developers.


Finally, what’s your deployment target?

This really matters because it informs how best to structure your product. For example, if you intend to support 10.14 and above, it’s actually best to raise that to 10.14.4 and then you get to rely entirely on the built-in Swift runtime (and, frankly, 10.14 users should be updating to the latest 10.14.x in order to get their security patches, so this isn’t exactly onerous). OTOH, if you have to support 10.13 then things get trickier (-:

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

5 Likes

This was a very helpful and clear explanation. As you predicted, we now have some 10.13 users writing in saying that they are having issues - what's the strategy here for 10.13 support?

How does this interact with the disable-library-validation entitlement? We do have the library validation entitlement but I think that is in error and we could probably remove it.

You could ask your 10.13 users to install the Swift 5 Runtime Support package:

https://support.apple.com/kb/DL1998?locale=en_US

2 Likes

This is my first time reading about this, neat! Why doesn't Software Update push these? Seems pretty crucial ...

It is not normally necessary, since Swift app bundles normally include the backward deployment libraries in the bundle. You only need to install the package above to use Swift binaries that can't bundle the libraries, such as command-line tools, or plugins for an app that doesn't itself use Swift.

1 Like

what's the strategy here for 10.13 support?

What Joe_Groff said.

How does this interact with the disable-library-validation entitlement?

There are three sources of the Swift runtime libraries:

  • Built in to the system

  • Swift 5 Runtime Support for Command Line Tools

  • Bundled within the app that’s using them

In the first two cases the libraries are signed by Apple and thus pass library validation like any other system framework. In the second case the libraries are signed by you and thus pass library validation like any other framework you’ve bundled inside your app. Either way, you won’t need to disable library validation.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple