Swift package manager, linker flags, and dependencies

Hi,

I have a package (GitHub - dowobeha/Foma) that wraps a C library (foma/foma at master · mhulden/foma · GitHub). On my machines, this C library was installed by hand (using ./configure, make, make install) rather than through apt or brew. As such, there is no corresponding .pc file. The C library lives in /usr/local/lib.

In order to get my Foma package to compile, I started off writing a Makefile that passed -Xlinker -L/usr/local/lib to swift build. That worked.

But when I tried opening my package in XCode, XCode didn't know where to find the C library. So, in Package.swift, I modified the .target with linkerSettings: [LinkerSetting.unsafeFlags(["-Xlinker", "-L/usr/local/lib"])]),. That worked. Now my Foma project compiles cleanly in XCode.

But I need to have my Foma package serve as a dependency for another package of mine. In my other project, I added the dependency as .package(url: "GitHub - dowobeha/Foma", from: "0.0.7"),.

This causes XCode to complain that "the target 'Foma' in product 'Foma' contains unsafe build flags." That is true. I searched online and found documentation (swift-evolution/0238-package-manager-build-settings.md at master · apple/swift-evolution · GitHub) explaining this behavior by XCode.

I understand what is going on. But I'm stuck. To summarize:

  1. I have a Swift package (Foma) that wraps a C library that lives in /usr/local/lib that was not installed through a package manager
  2. I need to tell SPM how to find the C library
  3. I need other Swift packages to depend on my Foma package
  4. The only way I have found to resolve point 1 and 2 (using LinkerSetting.unsafeFlags) prevents me from doing point 3

What is "the right way" to resolve this situation?

1 Like

You have two options:

  • Create a package for the C library so that it can be depended on as a package. This is preferable because it'll be easier to use for clients and could work for platforms where there isn't really a place for external libraries to be installed, e.g. iOS.

  • Use a system library target: Apple Developer Documentation

Boris,

Thanks. Eventually I do plan on doing the first option (making a package for the C library). But that will involve considerably more effort than just wrapping the C library, and so right now I need a solution that works with the C library wrapper.

In the Package.swift for my Foma package that wraps the C library, I am already doing declaring a system library. But as far as I can tell, .systemLibrary has no mechanism for specifying a path. There is a path argument, but the docs state:

"The custom path for the target. By default, a targets sources are expected to be located in the predefined search paths, such as [PackageRoot]/Sources/[TargetName] . Do not escape the package root; that is, values like ../Foo or /Foo are invalid."

And indeed, when I try doing .systemLibrary(name: "CFomaSystem", path: "/usr/local/lib"), XCode complains that "target path '/usr/local/lib' is not supported. It should be relative to the package root.

I am concerned that what I am trying to do is currently impossible. If so, that's a real shame. There should be some supported mechanism for this use case. It is especially important in terms of Swift on Linux. The Swift team is doing great work in expanding the number of supported distros. That is fantastic.

I hope that I'm missing something, and there is a way to accomplish what I'm trying to do.

The other option is to create a .pc file. This is a good idea anyway, especially on Linux where png-config is the standard way to locate a library and provide instructions on how to use it on Linux. However:

That's because the path doesn't do what you want it to do. That path points to somewhere inside the package where you provide a modulemap and potentially a separate header file, not to where the system library itself lives.

Here's a quick example that I found by searching Github: GitHub - tokyovigilante/Harness: Swift utility library for macOS/Linux platform harmonisation. Here's the relevant part of Package.swift:

        .systemLibrary(
            name: "CInotify"
        ),

This corresponds to the path Sources/CInotify. In that directory are a module.modulemap and a .h. The module.modulemap defines a Clang module for the given target. It basically just says "import the header file in this directory". That header file (cinotify.h) then imports the relevant headers.

Given that your library is apparently stored in /usr/local/lib, it will be found on the standard linker search path. As a result, you should only have to replicate what this user has done for inotify for your library.

Cory,

Thanks. I'll do some investigation to see if it's feasible to create a .pc file.

The example with Harness and CInotify is helpful. But after looking into it, I believe that still assumes that the C library inotify is in /usr/lib, not /usr/local/lib. I have the following:

.systemLibrary(
        name: "CFomaSystem"),

and in Sources/CFomaSystem/module.modulemap, I have tried this:

module CFomaSystem [system] {
  header "cfomasystem.h"
  link "foma"
  export *
}

with cfomasystem.h containing:

#include <fomalib.h>

I have also tried having module.modulemap contain the following:

module CFomaSystem [system] {
  header "/usr/local/include/fomalib.h"
  link "foma"
  export *
}

Either way, swift build succeeds, but swift test fails with linker error message:

ld: warning: Could not find or use auto-linked library 'foma'

The same error occurs at the command line and in XCode. This issue goes away if I manually add the linker flags:

swift test -Xlinker -L/usr/local/lib

Unfortunately, I can't figure out any mechanism for providing these linker tags to XCode.

Set the Other Linker Flags build setting for project/target. The build setting would be

-L/usr/local/lib,

don't have to include the -Xlinker.

Yup, so I reproduce this behaviour. I tried an equivalent build with c-ares (just the first thing I spotted in /usr/local/lib on my machine) and I got the same issue with failing to find the library. Seems like for some reason SwiftPM ignores the default build path. @NeoNacho any ideas there?

Are you sure /usr/local/lib is on the linker default search path? If the OP has to provide -L/usr/local/lib as a linker flag, it would seem that that may not be the case. Can't be ignoring search path completely since the system libraries are being linked without specifying /usr/lib. That also squares with my experience with Xcode at least where /usr/local/lib is concerned.

On my machine ld -v 2 says /usr/local/lib is on the default search path. It is also on the default search path for clang, so yeah, it certainly should be.

1 Like

I can confirm that on my machine, /usr/local/lib is known to ld:

$ ld -v 2
@(#)PROGRAM:ld PROJECT:ld64-530
BUILD 18:57:17 Dec 13 2019
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em

Library search paths:
/usr/lib
/usr/local/lib

Framework search paths:
/Library/Frameworks/
/System/Library/Frameworks/

I can also confirm that despite this fact, SPM does not link against my library that is in /usr/local/lib unless I explicitly provide -L/usr/local/lib.

We are always building using an SDK, which will end up passing -syslibroot to the linker. I think all default paths will be relative to that, so we are probably looking in /usr/local/lib, but not in the one on the host but the one in the SDK.

1 Like

Huh, that’s a pretty profound limitation, given that system packages will often not be in the SDK. Do we have a bugs.swift.org for addressing that?

Not that I know of, I wonder if that's because using pkg-config often masks this?

Also I haven't tried this on Linux, but I don't think we have an equivalent to SDKs there and the default paths will likely work.

I suspect it is masked by pkg-config, yeah. Do you consider this a bug? If so I'm happy to file a bugs.swift.org for it.

I'm not sure if it's a bug, but it was certainly unexpected for several folks on this thread, so would be great if you could file it. At the very least, this seems like something that should be documented.

I'm not sure about others, but from my perspective the current behavior feels like a bug.

@NeoNacho, I'm not sure I follow. What do you mean by an SDK?

While I might eventually deploy on iOS or macOS with a GUI, right now I'm developing for command-line only use on Linux and macOS. So in my case there is only one system - the one I'm coding on.

On macOS, /usr/include does not exist. /usr/lib contains only the libraries necessary for the operating system and supplied Apple applications. Most of the support expected by a developer actually resides in either:
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs,

and/or,

/Library/Developer/CommandLineTools/Developer/Platforms/MacOSX.platform/Developer/SDKs.

That's the SDK that is being referred to. Unless otherwise directed, clang/swiftc use one of those as the system root.

@jonprescott Thanks. That's very helpful.

As a long-time macOS user and long-time coder used to POSIX-style OSes, that is also fairly disturbing.

When developing programs for the command line, my first instinct is to assume that my programming language compiler will look in the standard locations, such as /usr/lib and /usr/local/lib, and the corresponding include/ subdirs.

If swiftc is instead looking somewhere else (like /Library/Developer/CommandLineTools/usr), that is very unexpected behavior. At a minimum, that behavior should be very prominently documented, as it will almost certainly confuse developers used to working on Linux and similar OSes.

1 Like

/usr/local is included in the default search paths for both header and libraries, so that is there. Catalina is the first macOS that locked down parts of /usr (or /, for that matter), for security reasons. The compilers (clang/swiftc) do the appropriate mappings to all of the SDK /usr directories so that it looks like what would expect from a POSIX/Unix developer experience when using the compilers. There are compiler flags like -isystemroot that allow you to alter that convention, if you want.

It is a challenge when you want to look at a system header file outside of something like Xcode that follows the mappings.

1 Like