Why does Xcode require an unsafeFlag to find certain headers but `swift build` doesn't?

Using Swift
, ,
1

As mentioned in other topics, I'm currently designing a Swift wrapper for a C++ library (which already provides a C wrapper). I've been able to set up the whole thing and it compiles and runs but I have to rely on multiple unsafeFlags which make distribution impossible.
Some background: the C++ library depends on Boost, which can be installed via Homebrew so declaring a .systemLibrary target came to mind. However, Boost does not provide a .pc file so pkg-config is out of the picture (note: I've actually used a tool to produce one "automatically" and Xcode was able to find the headers, but I cannot ask package users to install a .pc file). So last option seems to be a modulemap, but this is completely out of my depth (and even with #include <boost/config.hpp> in a shim.h header did not work).
I've therefore resorted the following: since headers and libs are in /usr/local/opt/boost, symlinked to /usr/local/include and /usr/local/lib, I've set cxxSettings: [.unsafeFlags("-I", "/usr/local/include")] on both the C++ and the C target (because the C target is actually C++ with extern "C"). Without those flags, Xcode is unable to find the Boost headers.
To my surprise though, while tinkering with swift build, the latter compiles the whole package even when I remove the unsafeFlags! Based on some research with clang -v, swift build includes /usr/local/include in its default header search paths, whereas Xcode's invocation doesn't.

My question is: why doesn't Xcode look at the same header search paths as swift build?
Follow-up: what's the best solution? Why does a .systemTarget with an (admittedly bad) modulemap not find the headers for Boost? but

EDIT: I've found some tentative modulemaps for boost (here and here) but they're both just over 9000+ lines long, and point to the headers directly, not to an umbrella...

2

Xcode sets the sysroot to the SDK that conforms to the platform/OS specified in the Xcode build settings for both clang and swiftc (the raw Swift compiler), and the toolchain directory which include the C++ standard library headers and Swift support headers for C (and libs, tools, etc.). There is no local directory in any of those usr directories, which is why you have to add /usr/local/include and /usr/local/lib to default search paths in the build settings. This is accomplished by using the -isysroot command line parameter for both clang and swiftc when it compiles source code. Using clang/swift from the command line doesn't change the sysroot unless you add the command line parameter. The default sysroot includes /usr/local in the search paths.

Look at the build logs in Xcode to see the compiler invocation.

3

Boost is a set of separate libraries, and are intended to be used separately (although, the libraries often depend on each other). There is no umbrella header provided by default in the Boost distribution. There are often umbrella headers in each Boost library that you could use to write an overall umbrella header if you wanted to.

If your Boost libraries are header-only, you might consider just copying over the required header directories (as well as any dependent libraries) into your source code architecture within a boost directory. If you need libraries with binaries, you might think about incorporating the source code into build architecture. That would alleviate any worries about where did a user put the Boost libraries, the need to install Homebrew or Boost itself, and you could make sure the Boost version is compatible with you software. Whether this is feasible depends on what libraries you are using.

boost/config.h just sets up some environment variables based on your operating system, compiler, etc.. It does not include any of the library umbrella headers.

Distributing Boost as part of a distribution of your own software is painful!

Clang module map functionality requires that module maps reside in the same directory as the header files. For your Boost example, it would be in /usr/local/include/boost, or, in /usr/local/include/boost/<library name> depending on the library header architecture, unless re-directed using a command line parameter. You might want to look at the Clang module map documentation in the Clang documentation at clang.llvm.org

4

Well, I was mistaken. I had to do some investigation on a project, and when I did a clang -v, I noticed that clang sets the sysroot as well. The only difference seems to be that clang from the command line adds a -I /usr/local/include and clang invoked within Xcode doesn't.

5

Is it possible that the extra -I is a side effect of something else in your graph? I don't think it is there by default.

6

The only places in my environment where /usr/local is used is in my PATH, my DYLD_FALLBACK_LIBRARY_PATH, INFO_PATH (for GNU info), MANPATH, and PKG_CONFIG_PATH. None of those paths reference include, just lib, share, and bin. A question would be: "does clang infer anything from these path definitions?" Note that in the simple invocation of clang I was doing to investigate my problem, I was just compiling a single file, using -c, and no header or lib specifications.

7

I've made a test package here: GitHub - thebluepotato/BoostTest
Please excuse my level of C/C++, it's been years... Regardless, it shows the issues I'm having:

  1. Different header search behavior: swift build; swift run can compile and run the Package without unsafe flags. Xcode needs the unsafe flags to look for Boost's headers in /usr/local/include. This means that swift build can build as long as the headers are present on the system, but there's no trivial way of telling the Package user that they need to be installed beforehand.
  2. .systemLibrary targets are documented as follows: "Use system library targets to adapt a library installed on the system to work with Swift packages." However, they seem to be limited to adapting C libraries as Swift modules (and not generally making those targets "available" in terms of headers and such for other (C-family) targets within a Package). Accordingly, those targets only have an effect when depend on by a Swift target. SPM should at least warn that having such a target as dependency in C/C++ targets is useless. (Or do they also expose headers but only for C targets?)

Feel free to play around with the test package. All in all, it seems like an Xcode issue here but I still understand the reason why swift build is "allowed" to search further. This makes it hard to have a "safe" package if SPM doesn't provide a way to manage a system-installed C/C++ library required by the C library or wrapper we're trying to have in Swift.

8

According to @NeoNacho's response to my earlier post, having /usr/local/include included by default may be an oversight, or was not originally intended. He seemed surprised at my experiment's results. So, not sure which way is intended behavior.

Boost is not a system-installed library. It has to be installed by the user, either through something like Homebrew, or MacPorts, Fink, or directly by downloading and building. Homebrew will take over /usr/local, but it doesn't have to, it's all decided by the user when the package manager is installed. Same with MacPorts and Fink. If the user builds the library directly, it can go anywhere the user wishes. Homebrew installation co-opting /usr/local may be a de facto default, but it's not guaranteed.

I've played with your test case, and I put together a version that actually builds a subset of the Boost libraries as part of the project, GitHub - jprescott/BoostTestWithLib: Example project using Boost with Swift package. It's actually much more that what is required for you particular header (e.g., I included all of the algorithms library, and all the supporting libraries that you wouldn't need if you were more strategic). Not knowing if you example was artificially limited because it was for illustrative purposes, I choose to try out a more extensive set of the libraries. I also included the date_time and exception libraries as examples of how you might include source code from Boost that has to be compiled.

You can't really treat Boost as a single library, as I've mentioned before. It's a series of (possibly) inter-related libraries. My project includes much of what might be considered the "core" code, but, there is going to be more needed depending on whether you use the more sophisticated libraries, like serialization, logging, and others.

This method, although requires a more in-depth knowledge of Boost, is probably the cleanest way of including Boost in your distribution that does not require the user to install anything external. Also, you know the user is accessing the code that is version-compatible with your distribution, and does not subject the user of keeping multiple versions of the libraries around.

When SPM supports distributing binary libraries, you could distribute the header files and the binaries you need from the Boost distribution, which would save having to compile Boost a second time.

9

I believe this behavior is actually part of the xcrun tool (which swift goes through by default on macOS) if you don't explicitly provide an SDK. Xcode always builds against an SDK, though, so it doesn't add any outside-of-SDK paths by default (as mentioned above). See https://twitter.com/kastiglione/status/1174419772651368448.

1 Like
10

It seems that you're saying that Xcode will only recognize libraries that are already installed in the SDK. Then I don't really see the point of providers like .brew() .systemLibrary(name:providers:)... If they have to be already installed withn the SDK, installing them via Homebrew will not make them available within the SDK; and if they actually are already installed, you don't need a provider in the first place! The purpose is getting less and less clear IMO...

Yes, it was very much limited for illustrative purpose! I'm very unfamiliar with Boost, that's why it was already an ordeal to get it to compile in the first place. I don't think I want to have to "bundle" all the headers with the package and clang still needs to link them somehow, no? Or is all if not most of boost declared already within the headers? As a side note, most of your "-D" unsafe flags could be replaced with .define("YOUR_DEFINE").

Actually the current version (5.3 - still under development) supports it! That's also an option indeed.

TL/DR: I think I'll end up having to make a Boost "package" (binary or not) to get what I want, even though it does make the package 120 Mo heavier with all the headers (!).
However, I'm still convinced that .systemLibrary is not behaving as it should here, especially if providers is a non-feature.

11

Regarding that, I find it particularly odd that swift build -v finds the headers and compiles the package but does not mention /usr/local/include even once in the output. Freakier still: the commands shown in the output of swift build -v, taken individually, can't find the headers either because they don't have -I/usr/local/include !!! It goes even further: swift build -v -Xcxx -v actually shows that -I/usr/local/include is defined as per the verbose output, but I have no idea where. Here's the full logs: Outputs from `swift build -v` and `swift build -v -Xcxx -v` · GitHub

EDIT: since -I/usr/local/include is only defined in the "verbose" version of the clang invocation and that one also contains -triple x86_64-apple-macosx10.10.0, I suspect /usr/local/include is taken as include dir by BuildPlan.swift, but it's unclear where. The difference with Xcode could probably be that when generating of the containing xcodeproj/pbxproj, this flag is not added (since it doesn't use swift build).

12

Xcode doesn't look at external paths, by default. However if you set the HEADER_SEARCH_PATHS or USER_HEADER_SEARCH_PATHS build settings, you can put in all the external paths you want. That's why Homebrew, etc., work and are greatly appreciated. As @jrose noted about, Xcode only compiles against an SDK be default , which means you have to provide the external paths you also want to include in your project via the build settings.

However, I don't know enough about SPM to know what would happen if you specified a system library that is provided by Homebrew and you don't have Homebrew installed? Does it automatically install it? Does it put up a message to the user that you have to install it? As I said, they don't have to be installed in the SDK, but, you have to add build settings to Xcode to setup the search paths (or use command line parameters like -I /usr/local/include, as you originally did).

The behavior about verbosity makes sense. clang has -I /usr/local/include as a default search path (probably because scrum inserts it as part of a command line invocation, as mentioned by @jrose). However, when swift invokes clang as a subsidiary tool to compile C/C++ programs, the -v argument doesn't apply to the clang invocation, only the swift invocation. -Xcxx -v actually passes a -v to the clang compiler, which is where you get the clang verbose output.

Most of Boost is header only. In my example project, I included a the date_time package to show how you might incorporate a library that does have to be compiled. You could tailor what I included (which is not nearly the whole of Boost) to specifically fit your needs to reduce the number of header files. What I included are all those files needed for the complete algorithms, date_time, and exception libraries. However, I also included a bunch of stuff that you might not need, for expediency's sake. For example, some of the date-time formatting code requires the sign function defined in the math library. To be expedient, I copied in the entire math library, which include math functions like Bessel functions, that you probably don't need and could be eliminated to reduce the size.

Header files are not linked. They are complled into object code by clang as part of the object generated by the code that actually includes the header files(s). The linker is not involved

swift build plans compilations and other tool usage as part of the overall task. If there are C++ files (C files, Obj-C files, etc.) that are specified as part of the task, swift build schedules sub-taks for clang/clang++ to compile those files to object code. From then on, it's clang that processes the files. As far as I know, header files are really strictly the province of the subsidiary tools, not swift build itself. swift build probably only looks at header files for changes since they are marked as dependencies for affected source code files (.cpp,.c files) by clang, and uses that information to help schedule what source code files need to be compiled.

13

What Boost libraries are you actually interested in? If you are using the "big" libraries like serialization, regex, logging, filesystem (actually filesystem wouldn't be too bad, it only has a couple of source code files to compile, depending on machine architecture), python, mpi that have significant source code files, those would be laborious to include.

Are you concerned about version compatibility? Boost publishes updates 1-2 times a year.

14

I honestly don't really know either, but from my current understanding, they are more tailored toward exposing more or less any C library that can be installed via Homebrew and friends to Swift targets, optimally when there's a pkg-config file. I would hope they'd be more capable, but unless I'm missing something big, I don't think they're meant for my use case.

That's what I'll end up doing. I have no needs regarding compatibility for now, so I can indeed bundle the headers, selectively. Thanks for guiding me in that direction!

I'm still perplexed regarding my initial question: I understand that Xcode has different searching behavior now, but I still don't the reason behind it, or rather the reason why swift build seems to be more "permissive". It seems peculiar that you could build and test a package with swift build but the exact same configuration would fail for Xcode.

15

It's an historical thing. When Apple started providing developer support with Xcode to iOS, iPadOS, watchOS, tvOS, there were more platforms to support, and they developed the idea of a System Development Kit (SDK) which would be platform and O/S specific. I think the third-party default search paths in Xcode are a casualty of the expansion to new devices, new operating systems, and the new development architecture..

swift build, however, is descended and works the same as the clang family of tools on the command line, and the xcrun architecture. Since clang is cross-platform, and swift being cross-platform, some of the standard Unix conventions still remain, including /usr/local/include in the search path. However, based on both the @jrose and @NeoNacho posts, /usr/local/include may be because xcrun puts in, or its been inadvertently retained. Not sure its actually intended, but, it's probably not worth worrying about for the general case.

1 Like