Issue with Swift 6 and aws-sdk-swift in Linux

marcelo-paredes · August 10, 2024, 12:28am

Hi, I'm trying to build a "small" command line tool that requires calling an AWS Lambda function. Originally I was producing the text for calling the aws client, but now that there's the aws sdk for swift I wanted to make a new version that used it. It builds and runs OK when on macOS (Swif5 and Swift6 by using /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift build

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let xName = "My-Project"

let package = Package(
    name: xName,
    platforms: [
        .macOS(.v10_15),
        .iOS(.v14),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.5.0"),
        .package(url: "https://github.com/awslabs/aws-sdk-swift.git", from: "0.16.0")
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: xName,
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
                .product(name: "AWSLambda", package: "aws-sdk-swift")
            ],
            swiftSettings: [
                .unsafeFlags([
                    "-parse-as-library"
                ])
            ]
        ),
    ]
)

For Linux, I'm running on Ubuntu 22.04 (x86_64 and arm64 on a Raspberry Pi 4, too) and it builds OK with Swift5, but I get this weird issue when building with swift6:

/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:85:103: error: incomplete definition of type 'struct openssl_hmac_ctx_table'
   85 |         (int (*)(HMAC_CTX *, const void *, int, const EVP_MD *, ENGINE *))g_aws_openssl_hmac_ctx_table->impl.init_ex_fn;
      |                                                                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:29:15: note: forward declaration of 'struct openssl_hmac_ctx_table'
   29 | static struct openssl_hmac_ctx_table hmac_ctx_table;
      |               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:108:37: error: incomplete definition of type 'struct openssl_hmac_ctx_table'
  108 |         g_aws_openssl_hmac_ctx_table->init_fn != s_hmac_ctx_init_noop &&
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:173:63: note: expanded from macro 'AWS_PRECONDITION'
  173 | #define AWS_PRECONDITION(...) CALL_OVERLOAD(AWS_PRECONDITION, __VA_ARGS__)
      |                                                               ^~~~~~~~~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/macros.h:54:92: note: expanded from macro 'CALL_OVERLOAD'
   54 | #define CALL_OVERLOAD(name, ...) GLUE(OVERLOAD_MACRO(name, COUNT_ARGS_MAX5(__VA_ARGS__)), (__VA_ARGS__))
      |                                                                                            ^~~~~~~~~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/macros.h:44:22: note: expanded from macro 'GLUE'
   44 | #define GLUE(x, y) x y
      |                      ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:118:48: note: expanded from macro 'AWS_PRECONDITION1'
  118 | #    define AWS_PRECONDITION1(cond) AWS_ASSERT(cond)
      |                                                ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:64:47: note: expanded from macro 'AWS_ASSERT'
   64 | #    define AWS_ASSERT(cond) AWS_FATAL_ASSERT(cond)
      |                                               ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:87:23: note: expanded from macro 'AWS_FATAL_ASSERT'
   87 |                 if (!(cond)) {                                                                                         \
      |                       ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:29:15: note: forward declaration of 'struct openssl_hmac_ctx_table'
   29 | static struct openssl_hmac_ctx_table hmac_ctx_table;
      |               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:112:33: error: incomplete definition of type 'struct openssl_hmac_ctx_table'
  112 |     g_aws_openssl_hmac_ctx_table->init_fn(ctx);
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:29:15: note: forward declaration of 'struct openssl_hmac_ctx_table'
   29 | static struct openssl_hmac_ctx_table hmac_ctx_table;
      |               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:120:37: error: incomplete definition of type 'struct openssl_hmac_ctx_table'
  120 |         g_aws_openssl_hmac_ctx_table->clean_up_fn != s_hmac_ctx_clean_up_noop &&
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:173:63: note: expanded from macro 'AWS_PRECONDITION'
  173 | #define AWS_PRECONDITION(...) CALL_OVERLOAD(AWS_PRECONDITION, __VA_ARGS__)
      |                                                               ^~~~~~~~~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/macros.h:54:92: note: expanded from macro 'CALL_OVERLOAD'
   54 | #define CALL_OVERLOAD(name, ...) GLUE(OVERLOAD_MACRO(name, COUNT_ARGS_MAX5(__VA_ARGS__)), (__VA_ARGS__))
      |                                                                                            ^~~~~~~~~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/macros.h:44:22: note: expanded from macro 'GLUE'
   44 | #define GLUE(x, y) x y
      |                      ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:118:48: note: expanded from macro 'AWS_PRECONDITION1'
  118 | #    define AWS_PRECONDITION1(cond) AWS_ASSERT(cond)
      |                                                ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:64:47: note: expanded from macro 'AWS_ASSERT'
   64 | #    define AWS_ASSERT(cond) AWS_FATAL_ASSERT(cond)
      |                                               ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-common/include/aws/common/assert.h:87:23: note: expanded from macro 'AWS_FATAL_ASSERT'
   87 |                 if (!(cond)) {                                                                                         \
      |                       ^~~~
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:29:15: note: forward declaration of 'struct openssl_hmac_ctx_table'
   29 | static struct openssl_hmac_ctx_table hmac_ctx_table;
      |               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:122:33: error: incomplete definition of type 'struct openssl_hmac_ctx_table'
  122 |     g_aws_openssl_hmac_ctx_table->clean_up_fn(ctx);
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:29:15: note: forward declaration of 'struct openssl_hmac_ctx_table'
   29 | static struct openssl_hmac_ctx_table hmac_ctx_table;
      |               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:138:5: error: use of undeclared identifier 'hmac_ctx_init'
  138 |     hmac_ctx_init init_fn = (hmac_ctx_init)HMAC_CTX_init;
      |     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:139:5: error: use of undeclared identifier 'hmac_ctx_clean_up'
  139 |     hmac_ctx_clean_up clean_up_fn = (hmac_ctx_clean_up)HMAC_CTX_cleanup;
      |     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:140:5: error: use of undeclared identifier 'hmac_update'
  140 |     hmac_update update_fn = (hmac_update)HMAC_Update;
      |     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:141:5: error: use of undeclared identifier 'hmac_final'
  141 |     hmac_final final_fn = (hmac_final)HMAC_Final;
      |     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:142:5: error: use of undeclared identifier 'hmac_init_ex'
  142 |     hmac_init_ex init_ex_fn = (hmac_init_ex)HMAC_Init_ex;
      |     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:145:28: error: use of undeclared identifier 'init_fn'
  145 |     bool has_102_symbols = init_fn && clean_up_fn && update_fn && final_fn && init_ex_fn;
      |                            ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:145:39: error: use of undeclared identifier 'clean_up_fn'
  145 |     bool has_102_symbols = init_fn && clean_up_fn && update_fn && final_fn && init_ex_fn;
      |                                       ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:145:54: error: use of undeclared identifier 'update_fn'
  145 |     bool has_102_symbols = init_fn && clean_up_fn && update_fn && final_fn && init_ex_fn;
      |                                                      ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:145:67: error: use of undeclared identifier 'final_fn'
  145 |     bool has_102_symbols = init_fn && clean_up_fn && update_fn && final_fn && init_ex_fn;
      |                                                                   ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:145:79: error: use of undeclared identifier 'init_ex_fn'
  145 |     bool has_102_symbols = init_fn && clean_up_fn && update_fn && final_fn && init_ex_fn;
      |                                                                               ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:150:21: error: use of undeclared identifier 'init_fn'
  150 |         *(void **)(&init_fn) = dlsym(module, "HMAC_CTX_init");
      |                     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:151:21: error: use of undeclared identifier 'clean_up_fn'
  151 |         *(void **)(&clean_up_fn) = dlsym(module, "HMAC_CTX_cleanup");
      |                     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:152:21: error: use of undeclared identifier 'update_fn'
  152 |         *(void **)(&update_fn) = dlsym(module, "HMAC_Update");
      |                     ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/openssl_platform_init.c:153:21: error: use of undeclared identifier 'final_fn'
  153 |         *(void **)(&final_fn) = dlsym(module, "HMAC_Final");
      |                     ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/opensslcrypto_hmac.c:30:9: error: use of undeclared identifier 'g_aws_openssl_hmac_ctx_table'
   30 |         g_aws_openssl_hmac_ctx_table->free_fn(ctx);
      |         ^
/home/marcelo/repo/my-project/.build/checkouts/aws-crt-swift/aws-common-runtime/aws-c-cal/source/unix/opensslcrypto_hmac.c:63:11: error: use of undeclared identifier 'g_aws_openssl_hmac_ctx_table'
   63 |     ctx = g_aws_openssl_hmac_ctx_table->new_fn();
      |           ^
[11/428] Write swift-version-6A52383ED409C6A7.txt
marcelo:my-project %

any help figuring out what's wrong would be great.

Thanks!

marcelo-paredes · August 12, 2024, 12:56pm

Hi @thomas945

It's not solved yet. Last time I tried was with swift-6 from 2024-08-09 and aws-sdk-swift 0.55.0. It builds OK with Swift 5.10.1, or if I build in a Mac (macOS 14.6, Swift-6 from Xcode 16 beta 5)

0xTim · August 12, 2024, 1:22pm

I recommend trying Soto (GitHub - soto-project/soto: Swift SDK for AWS that works on Linux, macOS and iOS) instead - that's tested on Linux with Swift 6 and doesn't require an installation of OpenSSL either

marcelo-paredes · August 12, 2024, 4:39pm

hi @0xTim

soto gives me this instead

.build/checkouts/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_base.h:429:10: fatal error: 'memory' file not found
  429 | #include <memory>
      |          ^~~~~~~~

0xTim · August 12, 2024, 4:53pm

Which version of SwiftNIOSSL are you using? Are you on the latest packages for everything? That shouldn't be happening unless something has broken in one of the nightlies (which is definitely a possibility)

marcelo-paredes · August 12, 2024, 5:04pm

I believe they're the latest

        .package(url: "https://github.com/soto-project/soto.git", from: "6.0.0"),
        .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.5.0")

0xTim · August 12, 2024, 5:28pm

FWIW I've just tested the main branch of Soto with Beta 4 and Beta 5 and they both compile fine. Have you tried Soto v7?

marcelo-paredes · August 12, 2024, 7:03pm

but are you also on Ubuntu 22.04 or 24.04? I haven't tried another distro because that's what I'm using here, maybe worth a try.

Update: building with the latest(5.10.1) docker image builds correctly for Soto.

marcelo-paredes · August 14, 2024, 4:16am

started from scratch with a new installation of ubuntu 24.04 and swift-6 from 22.04 in a raspberry pi 5, and it built correctly with soto now. It took me a bit to understand how to pass the payload to a lambda function but it worked in the end.

thanks!

(btw, the aws-swift-sdk version still fails to build w/ swift-6 on ubuntu 22.04 x86_64 and 24.04 aarch64)