So I've been puzzling through something in Swift for AWS Lambda, and I think I got it almost nailed except for one thing.
I am trying to create a Lambda WebSockets handler using AWS for Swift Lambda. The AWS API Gateway allows a lambda function to handle WebSockets clients. The way it works is that API Gateway handles the actual WebSockets sessions…but it packages up WebSockets client requests as APIGatewayV2Requests. The problem is that replies to clients have to be made through the AWS SDK (there is a special PostToConnectionInput function…part of the ApiGatewayManagementApiClient… in the SDK for it…given a WebSocket connectionID, it handles the reply). I did need to tweak the APIGatewayV2Request type for WebSockets to get it to "go"…but I got that sorted out and it's all good. But I can't get this to compile for AmazonLinux2.
Long story short, I need to be able to include the AWS SDK for Swift in my lambda function…I need it to be a package dependancy. I can do this on my Mac, running the lambda function locally, and it works just fine…it actually replies to WebSockets clients…it is a beautiful thing! I cannot, however, build for AmazonLinux2…when I do, the compiler complains that it can't find OpenSSL headers. I thought it would be sufficient to munge the Dockerfile this way:
# dependencies
RUN yum install -y wget perl-Digest-SHA
RUN yum install -y lsof dnsutils netcat-openbsd net-tools curl jq # used by integration tests
RUN yum install -y openssl11 # needed if the AWS SDK is to be used in a lambda project
…but, alas, that doesn't seem to work. It looks like this when I attempt to compile via Docker:
swift package --disable-sandbox archive
-------------------------------------------------------------------------
building "quizzler-lambda" in docker
-------------------------------------------------------------------------
updating "swift:amazonlinux2" docker image
amazonlinux2: Pulling from library/swift
Digest: sha256:aef39deb652529675228375b62a57cf6297524d596e7e11eeb5ea29da2c7e09b
Status: Image is up to date for swift:amazonlinux2
docker.io/library/swift:amazonlinux2
What's Next?
View summary of image vulnerabilities and recommendations → docker scout quickview swift:amazonlinux2
building "quizzler-lambda"
Building for production...
/workspace/.build/checkouts/aws-crt-swift/aws-common-runtime/s2n/utils/s2n_random.c:19:10: fatal error: 'openssl/engine.h' file not found
#include <openssl/engine.h>
^~~~~~~~~~~~~~~~~~
1 error generated.
Basically, it bails trying to find OpenSSL.
Does any kind, knowledgable soul know how I might fix the AmazonLinux2 image AWS Lambda for Swift uses so that I can include the AWS SDK for Swift (which Amazon does say will work). I hope this makes some kind of sense.
A small update. Apparently, OpenSSL is just OpenSSL…to get dev things, you also have to yum install openssl-dev. I tried that in the Dockerfile, but still no luck. Only other thing I can think of is the compiler can't find the header files even if they're actually there…in Linux, they're supposed to be in /usr/include/openssl…at least, if StackExchange opinion is correct. No such place on my dev Mac. I guess I could maybe try to include that path in this line in the Dockerfile:
RUN echo 'export PATH="$HOME/.tools:$PATH"' >> $HOME/.profile
I'm basically at the limits of my Docker knowledge. I'll figure this out eventually…but hoping someone can kind of point me in the right direction.
While not the solution you are looking for, but I think the community driven Soto project does not have a runtime dependency on OpenSSL and might "just work".
@t089…thank you very much for this suggestion! I am absolutely not married to the AWS SDK, so I will definitely give this a try. It looks to have a similar functionality in the specific area of interest to me (posting via an ApiGatewayManagement client)…and those folks have created specific instructions for getting it to work with AWS Lambda for Swift. Lastly, "Soto" is just an amazingly clever name…Swift's answer to Boto. Thanks again for this suggestion!
@t089, in answer to your question about using the correct base image…no, I am not sure, but I "think" it is working…at least at some level. If I have Docker Desktop show me all 200-odd packages in the Docker image, OpenSSL is in there:
# tools
RUN mkdir -p $HOME/.tools
RUN echo 'export PATH="$HOME/.tools:/usr/include/openssl:$PATH"' >> $HOME/.profile
and it didn't work. I'm gonna give @t089's suggestion a try this afternoon…I suspect it's gonna work great. I'm still kind of curious if I can get Amazon's SDK to work…but I'll definitely have to move beyond my "throwing ideas against a wall" approach… I guess I could use this as an excuse to learn how AWS for Swift Lambda packages up the image it uses to compile code for AmazonLinux2. I basically "get" what it does…but I admit my understanding about the details is really vague.
Okay, I made about a billion mistakes along the way, but after following up on @t089's awesome suggestion, I got my thingy to work…an AWS APIGateway/Lambda WebSockets server running 100% in Swift. Rabbit hole plumbed…funky item checked off the coding bucket list.
I usually use a Amazon Linux 2 Docker image to compile my server side code (i.e. Lambdas with AWS SDK and Lambda Runtime). The container installs the following packages:
RUN yum -y install git jq tar zip openssl-devel
Have you tried using openssl-devel? it sounds like you got it working.
Thanks for your reply @davmoser! I really, really appreciate your feedback.
So I typically use swift package --disable-sandbox archive to build for Lambda…and basically go with whatever Docker image the archive plugin decides is best (swift-aws-lambda f21e92895c91 the last time I tried). Whenever I use the Docker image that the archive plugin picks, I get the compile failure when building with the AWS SDK. I hadn't thought to try a different image…but I see the archive plugin lets me specifically pick an image rather than rely on the default. I could try that and see if some other image works.
The way that I solved this, for the nonce, was by taking up @t089's suggestion and using a "3rd party" SDK…instead of the AWS SDK package, I used the "Soto" package. Soto does not rely on OpenSSL, so it worked fine (after I worked out how to use it correctly). Maybe I have this wrong, but to replicate my error, compile with swift package --disable-sandbox archive…basically, go with the default. :)
Sorry I missed this over the holidays…but whoah, that looks like a great suggestion! I am going to give it a try…just because I am curious. Kind of the cool thing is that it gives me a bit more insight into how this is all put together. Many thanks!
