AWS Lambda Runtime API

Hello Georgie. Short answer : no. Long answer : AWS Lambda is running on Amazon Linux. Apple provides the Swift runtime for Ubuntu only. So to execute a Swift binary on Amazon Linux, the trick is to import into the runtime environment all dependencies from Ubuntu, including the libgnutls library from Ubuntu. Problem is that library has some hard coded path to configuration file. I am working on a solution to recompile libgnutls on ubuntu but this time using the Amazon Linux path for configuration files. I'll keep this thread posted with my results

3 Likes

Hello Toni,

No, there is no need for VPC. Any HTTPS call from the lambda function will fail.

This looks like it’s already at least 80% there, great work!

Architecture-wise I’m wondering whether the handler can be further decoupled from the handler by linking it as a prebuilt shared library (in the base layer).

As it stands this would be compiling (and statically linking) its own handler for each packaged set. I don’t know the runtime API well enough to be sure this is avoidable but seems like it should be able to have one binary running the runtime as called initially by bootstrap and running the infinite loop. And then maybe use dlopen to open the handlers dynamically in a sort of “plugin” pattern. Or just run each handler as invocations of separately-built handler binaries. I’m not sure what the performance implications would be of running a binary vs opening a shared library- if it makes no difference just running the handlers as individual binaries would be easiest to reason about and test.

That architecture has the disadvantage of having no static safety that the handler actually exists, but this is no different to js/python etc. But it has the advantage of having faster compile times, smaller binaries, less overhead, faster starts, and also running multiple handlers on a single layer - as it stands it seems like the handleName is hardcoded on init?

The only uncertain part about this architecture is that it would still need a way of sharing datatypes for event input and output between runtime and handler, or a manual coding/decoding of the input and return types in the handler. It could also be a function imported from the runtime shared library- runLambdaFunctionHandler(_ handler: (Codable) -> Codable) which just thinly decodes stdin and encodes the handler’s return value to an IOStream (stderr?). But I’m not sure- I’m pretty new to IPC. Architecting handlers as “plugins” would make these decisions unnecessary and allow the handlers to be called directly with real error handling etc.

Sorry for the long message, I don’t expect you to actually do any of this. It’s just as much a notepad for my own ideas about it. I’m excited about the prospect of using Swift on Lambda but I don’t have a lot of time to play around with it myself at the moment so this is in case I do. But feel free of course to try any of these ideas out of you’re interested.

I see, if they’re not bundling it then no I wouldn’t expect them to fix it ;)

Smoke Framework looks really promising. Are there any example projects to help getting started?

@mrmikemon
@tachyonics

I’d like to know this too. Would really like to see some information on getting started so we can take it for a spin and help move it forward.

@mrmikemon
@GalCohen

Thanks guys, appreciate your interest!

We actually have a full example showing a service with a REST-like API with some operations using DynamoDB as a persistence layer coded up and waiting approval. I don't have a firm timeline when we will be able to release it but in the past it hasn't taken long from this point.

Thanks @tachyonics ! Looking forward to it. Has compiling for the lambda execution environment been worked out yet?

@GalCohen Not yet. My team has a build script for building the Swift Toolchain for Amazon Linux 2 (which is what we run our services on). One of the next things I have planned for us is to finish adapting the script for Amazon Linux 2017.03 which is what Lambda uses.

I forked @tonisuter 's great implementation of the Lambda runtime API and was able to add support for async tasks/operations as well.

I was having a crash related to (what I believe) DNS resolution, but adding the /lib/x86_64-linux-gnu/libnss_dns.so.2 lib to the layer fixed it.

However, there is still the HTTPS error to sort out.

Toni, good job on what you did!

1 Like

Wanted to highlight that @jasonzurita posted about this at https://www.jasonzurita.com/websites-using-swift-and-aws-lambda :tada:

2 Likes

natanrolnik had the idea of trying to build the Swift source using the AWS Lambda linux (Amazon Linux) environment to resolve the HTTPS issue. I tried this out, by using this docker image, but I run into the following error (since it looks like you are trying something similar, I am not sure if you ran into the same thing @sebsto):

bash-4.2# ./swift/utils/build-script -r
./swift/utils/build-script: note: Using toolchain default
./swift/utils/build-script: fatal error: can't find clang (please install clang-3.5 or a later version)

Here is what I did to get the above:

docker run -it --name swiftawslinux amazonlinux /bin/bash
yum install cmake ninja-build clang python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev libcurl4-openssl-dev systemtap-sdt-dev tzdata rsync
mkdir swift-source
cd !$
git clone https://github.com/apple/swift.git
./swift/utils/update-checkout --clone
./swift/utils/build-script -r

If anyone has any suggestions of what to try, that would be great!

1 Like

IMHO, there are two possibilities to address the GNUTLS/HTTPS issue

  • a/ compile GNUTls on Ubuntu with compile-time options to point to Amazon Linux's cacerts
  • b/ compile SWIFT on Amazon Linux.

I naively thought that a/ was easiest as I would deal with less dependencies issues. I am struggling on this since several days. GNUTls has so many dependencies and I am still blocked by a lib or other not being installed or correctly identified.

I tried b/ yesterday. The issue reported above by Jason is easy to work around. You need clang, cmake3 installed. I softlinked cmake -> cmake3 etc ... I am blocked further
The problem with the approcah above is that the yum install command can not find most of these dependencies :

No package uuid-dev available.
No package libicu-dev available.
No package icu-devtools available.
No package libbsd-dev available.
No package libedit-dev available.
No package libxml2-dev available.
No package libsqlite3-dev available.
Package swig-3.0.12-11.amzn2.0.3.x86_64 already installed and latest version
No package libpython-dev available.
No package libncurses5-dev available.
No package pkg-config available.
No package libblocksruntime-dev available.
No package libcurl4-openssl-dev available.
No package systemtap-sdt-dev available.

At this stage, I would love to receive help by someone use to build native Linux projects to solve teh dependency issues on a/ or b/

3 Likes

Wondering if it is a dead end on the Swift on AWS Lambda? I would like to help out on getting this to work. I have no idea where to start with this. Like all of you I can write the AWS Lambda Runtime API but getting it to work with API is not there

1 Like

Regarding the SSL path issue discussed above by @natanrolnik and @sebsto. One potential solution might be to make a small change in swift-corelibs-foundation to allow for setting the path with an environment variable rather than relying on the default location that the Ubuntu based docker image uses. There already is support for this feature but it is limited to Android.

#if os(Android) || os(Linux)
...
  if let caInfo = getenv("URLSessionCertificateAuthorityInfoFile")  {
...
    try! CFURLSession_easy_setopt_ptr(rawHandle, CFURLSessionOptionCAINFO, caInfo).asError()
  }
#endif

We could change this line to also allow for Linux to configure the environment variable.

5 Likes

This would be even more convenient in a Lambda context than on Android- there it’s quite hard to provide the correct file path, in practice.

I've found a way to solve the HTTPS issue by using swift-nio 2.
I prepared an open-source framework https://github.com/swift-sprinter to support the development of the solution I've found.

Solving the HTTPS issue allows interacting with the AWS ecosystem.
I prepared documentation and three examples you can find here LambdaSwiftSprinter.

S3Test shows how to interact with an S3 bucket by using an early version of an open-source version of aws-sdk based on nio 2.

The project is in an early stage and it's open to collaboration and I hope someone will join me.

10 Likes

Wow, that sounds great! CC @georgebarnett, @tomerd, @kevints

2 Likes

Good news! Can't wait to use it myself :slight_smile:

It could be interesting once it's stable to do some benchmarks to see if it's a competitive option for production use.

Update:
Well we already have some elements there about general Swift performance.

@Damien_Petrilli , I like your idea.
I've opened an issue on the library, I would be very grateful if you can help me on this.


Feel free to open a new issue if this doesn't describe your idea.

You can consider the core LambdaSwiftSprinter package quite stable as it was ~100% tested (HelloWorld Example).

The unstable part is the one related to the nio 2.0 plugin and the aws-sdk as the related libraries are under development.

1 Like
Terms of Service

Privacy Policy

Cookie Policy