OpenSSL "pc" file missing

Update 3

Okay, I've created a minimally reproducible SPM project. All it is is a default SPM project with the Swift-JWT from Kitura dependency added.

After cloning of the project, you can run this Docker command from the root of the project directory to see the error:

docker run \
        --rm \
        --volume "$(pwd)/:/src" \
        --workdir "/src/" \
        swift:5.3.1-amazonlinux2 \
        swift build --product MissingPCFile -c release -Xswiftc -static-stdlib

If anyone has any insight to lend on the matter it would be greatly appreciated!

Update 2

I found the problem dependency. It is Swift-JWT from Kitura. Project compiles fine before the commit that incorporates that dependency, but following it, experiences the couldn't find pc file error.

Strangely, whenever I run the docker command to compile the project for AWS on any commit from this one on, and then do a git status my Package.resolved file has been altered. The alteration is the addition of the OpenSSL package:

            "version": "1.9.200"
         }
       },
       {
+        "package": "OpenSSL",
+        "repositoryURL": "https://github.com/Kitura/OpenSSL.git",
+        "state": {
+          "branch": null,
+          "revision": "80b04f33b086fc90e28d9ae159d43705fb348e16",
+          "version": "2.2.200"
+        }
+      },
       {
         "package": "postgres-kit",
         "repositoryURL": "https://github.com/vapor/postgres-kit.git",

However, this appears to be something different than the system OpenSSL; which I imagine is why the installations of openssl-devel and openssl didn't resolve the issue — as the error is not referring to the pc file of the system's OpenSSL, but perhaps to that of Kitura's OpenSSL.

Update 1

This appears to be an issue with the pkg-config utility in the Amazon Linux 2 Docker container. I've checked the openssl-devel installation in the container and openssl.pc exists inside /usr/lib64/pkgconfig. When I run pkg-config --variable pc_path pkg-config to see where the utility is checking for these pc files, the following is output:

/usr/lib64/pkgconfig:/usr/share/pkgconfig

The first item being the very directory where openssl.pc exists. I also read talk of a $PKG_CONFIG_PATH environment variable, so I figured it may be possible that this value is being used instead. Low and behold after echoing the value it was completely empty. So I set its value to that which was output above, but it didn't change anything.

Original

I posted the following question on StackOverflow first. But it hasn't gotten a response, and being that it appears to be an issue with the Swift Package Manager on Linux, I decided to come and see if anyone has any light to shed on the matter here at Swift HQ. For convenience, I've copied it below.

I basically followed this tutorial initially in order to create a basic AWS Swift Lambda. As a part of it you are meant to package the Swift project in an Amazon Linux 2 Docker image. I'm using the below command to do so:

docker run \
    --rm \
    --volume "$(pwd)/:/src" \
    --workdir "/src/" \
    swift:5.3.1-amazonlinux2 \
    swift build --product API -c release -Xswiftc -static-stdlib

And initially, it always worked. But I've been working on a particular component of the project for a while that has required adding some new dependencies. Ready to test it I reissued the above command but this time was met with this as the first line of the output:

'OpenSSL' openssl.pc: warning: couldn't find pc file

Then the command proceeds to compile different package components. It gets to [14/1013] which is CryptorRSA of the BlueRSA library and outputs this error:

<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "shim.h"
         ^
/src/.build/checkouts/OpenSSL/Sources/OpenSSL/shim.h:20:10: error: 'openssl/conf.h' file not found
#include <openssl/conf.h>
         ^
/src/.build/checkouts/BlueRSA/Sources/CryptorRSA/CryptorRSA.swift:25:12: error: could not build C module 'OpenSSL'
    import OpenSSL
           ^

It then proceeds to compile 5 more items ([15-19/1013]), until terminating, presumably due to the missing "pc" file.

I've tried running yum -y install openssl-devel on the Docker image, but to no avail. Also, I only have a single Docker image on my machine according to the Docker desktop app, so I'm fairly certain I'm not installing openssl-devel on some other image; because as far as I can tell there aren't any.

I've tried installing openssl (without "-devel"), but the problem persists.

How do you resolve this issue?

Not sure I can follow here. I just tried it and it seems to work for me. So from my observation this should work for you:

docker run \
        --rm \
        --volume "$(pwd)/:/src" \
        --workdir "/src/" \
        swift:5.5-amazonlinux2 \
        /bin/bash -c "yum install -y openssl-devel && swift build --product MissingPCFile -c release --static-swift-stdlib"

An alternative, that does not depend on Openssl is: GitHub - vapor/jwt-kit: 🔑 JSON Web Token signing and verification (HMAC, RSA, ECDSA) using BoringSSL.

You're saying you cloned the minimally reproduced repo I provided, ran that command, and everything was packaged successfully?

Noticed you used "swift:5.5" instead of "swift:5.3.1", I'm gonna see if using the newer version is the differentiating factor here.

Nope, problem persists.

Okay, actually noticing that that command is actually a little different from what I've been using — even disregarding the openssl-devel install. Trying it out now, but will replace "swift:5.5" with "swift:5.3.1".
...
Son of a gun! It's actually working! I don't understand. I installed openssl-devel in the the Docker container for this image. I'm green when it comes to Docker, are these containers like transient — as in not like virtual machines at all? Because I started a Docker container from the Docker desktop app, and installed openssl-devel. Now I'm figuring that that was just some transient instance, and despite actively running, every time I ran the package command it wasn't actually using that particular container; it would spin a new one up, and openssl-devel would always be missing on it because the container would instantly be deleted at the conclusion of the command.

Sound about right?

Unfortunately, still got one more error, the issue regarding the pc file has apparently resolved. But the swift compilation of the project appears to be failing on the last step. The exact error is:

error: link command failed with exit code 254 (use -v to see invocation)
clang-10: error: unable to execute command: Killed
clang-10: error: linker command failed due to signal (use -v to see invocation)

I tried using -v and the last command before the error is this:

/usr/bin/swiftc -lstdc++ -g -L /src/.build/x86_64-unknown-linux-gnu/release -o /src/.build/x86_64-unknown-linux-gnu/release/API -module-name API -static-stdlib -emit-executable -Xlinker '-rpath=$ORIGIN' @/src/.build/x86_64-unknown-linux-gnu/release/API.product/Objects.LinkFileList -target x86_64-unknown-linux-gnu -L /usr/lib

Any obvious/typical reason as to why it would fail?

could you check which version of 5.5 you are using? there was an issue with static linking FoundationNetworking that was resolved in 5.5.3

I'd also like to echo @fabianfett point about looking for alternatives to the dependency that pulls in OpenSSL. The latter is known to cause issues in the connext of moving code around machines, and its worth looking for an alternative solution that does not include a dependency on it (e.g GitHub - vapor/jwt-kit: 🔑 JSON Web Token signing and verification (HMAC, RSA, ECDSA) using BoringSSL.)

I ran:

docker run swift:5.5-amazonlinux2 /bin/bash -c "swift --version"

And the output was:

Swift version 5.5.3 (swift-5.5.3-RELEASE)
Target: x86_64-unknown-linux-gnu

I also tried explicitly executing the Docker packaging command using a Swift 5.5.3 image:

docker run \
        --rm \
        --volume "$(pwd)/:/src" \
        --workdir "/src/" \
        swift:5.5.3-amazonlinux2 \
        /bin/bash -c "swift build --product API -c release --static-swift-stdlib"

But got the same error (linking error).

Also, I've since removed Kitura's SwiftJWT completely — replacing it with Vapor's JWTKit — but the linking error persists.

is there code / reproducer you can share?

Got it. It's basically just the default SPM project structure with the dependencies that my project is using (the main.swift file is completely empty).

Running this Docker command results in the error:

docker run \
        --rm \
        --volume "$(pwd)/:/src" \
        --workdir "/src/" \
        swift:5.5.3-amazonlinux2 \
        /bin/bash -c "swift build --product LinkerErrorExample -c release --static-swift-stdlib"
1 Like

If you're getting the linking error still you need to give the Docker engine more RAM. It needs at least 4GB, preferably more. Static linking takes up a ton of memory

3 Likes

Hit the nail right on the head! How could I have discovered that on my own? Is there a setting that would have indicated that the failure was due to insufficient memory allocated?

1 Like

It's a compiler bug - it should at least report that it ran out of memory rather than crashing silently. Unfortunately I've seen this error so many times I know what to look for :sweat_smile:

This should probably be raised as an SR on bugs.swift.org

1 Like

First time doing so. How's it look?

Looks good!

cc @tomerd to channel to the right people