What would help you use Server Side Swift with AWS

Server
1

Hi all, I am looking to get some feedback/suggestions on using Server Side Swift with AWS.

If you are currently using Server Side Swift with AWS or are interested in using Server Side Swift with AWS but aren't sure how to, is there something that is currently missing that would make it easier to use the two together?

  1. Is there some documentation that would be useful? If so, what topics?
  2. Are there code examples or tutorials that would be useful? If so, what would be useful to have?
  3. Are there helper libraries that are available for other languages that don't exist for Swift?
  4. Are there any common patterns that are difficult to implement using Server Side Swift?
  5. Anything else?
14 Likes
2

swift on AWS could always use more documentation. i can’t really think of any topics in particular, since the resources are so sparse to begin with.

we could definitely use more EC2 tutorials; Lambda seems to be a bit more well-trodden.

however most of the issues i am currently dealing with aren’t specific to AWS/EC2, but are more general issues with using swift on linux.

for example, memory/heap profiling swift applications just doesn’t work on linux (1, 2). this affects all swift on linux users, but especially EC2 users, since EC2 is a very memory-constrained environment. my deployment target (a t2 nano instance) has just ~700 MB of memory available for swift, and a lot of that is eaten up by Foundation and the runtime.

because i don’t have access to heap profiling tools, i have to code very defensively for EC2, which hampers my productivity, since i have to preemptively optimize just about everything.

swift doesn’t have a native compression library, so we have to rely on zlib wrappers. it’s worth mentioning that the swift-png library actually has a high-performance LZ77 implementation, but i have not had time to spin that off into a consumable package. (contributors welcome!)

the (poor) state of swift-system is also a major obstacle for server-side swift applications. however the general vibe i am getting is that people in Apple are aware of this, are prioritizing it, and it may improve over the next 1–2 years.

SwiftNIO doesn’t speak async/await. its reliance on ByteBuffer and the low uptake of generics in the library ecosystem also make it hard to integrate SwiftNIO with other packages.

my main issue with Vapor is it evolves rapidly and most of the documentation that Google has indexed is severely out of date. its API reference is also somewhat labyrinthine and hard to navigate.

13 Likes
3

Speaking for myself, as someone who has never used AWS at all, but would like to try it with Swift, it would help me to have an absolute basic level step-by-step tutorial, that explains exactly what to do and where to click for the entire process of going from “I have literally never touched AWS before” to “I now have a functional Swift program running on AWS”, without skipping anything along the way.

16 Likes
4

Honestly my problems are less with Swift particularly but with AWS in general. Making sure that the program I'm running on this EC2 instance right here is allowed to make the appropriate calls to that database over there and that it can also read and write to that one bucket on S3 but not any of the others...come on, man, I'm a lone developer. It's enough that I have to be really smart about Swift, but I also have to be deeply knowledgeable about the way AWS handles access permissions, how my code is (and isn't) supposed to interact with other AWS resources, and I have to keep up to date on all these policies and know whenever Amazon changes their preferred way of doing things.

When I worked at a huge company, we had two entire engineering teams (on the order of 20 people) that did nothing but write scaffolding around this stuff so that the rest of engineering (around 280 people) could write business logic.

Scenario: I'm a single developer or maybe a small team of 5 people, and I want to build an application that consists of an API server, a database, some file storage that can be served out to clients directly, and a set of client applications that communicate with the API server and the storage. Let's say the application is something generic, like maybe a blog or a photo sharing service or something -- users register, authenticate, send data to the API server, fetch data from the API server and from the file storage. The API server writes files out to the file storage, and it persists user accounts (preferences, names, etc.) and file metadata in the database. This is a really straightforward kind of demo, and it'd make a decent example of how to get all the bits of AWS infrastructure to be used.

6 Likes
5

For me, native Swift support in Lambda would go a long way in making me use Swift on AWS.

9 Likes
6

For AWS Lambda only — I would see the biggest challenge coming from deploying, where developers need to use various tools with various configuration files or supporting scripts, which is much more complex than the one-click Heroku build-pack experience.

Probably any kind of CI/CD can solve this, and I feel that fewer additional files for deployment in the project will definitely lead to better experience.

4 Likes
7

I know the Serverless framework isn't really owned by Amazon, but it would be nice if there was an up-to-date tutorial on how to use Swift with AWS Lambda using the Serverless framework.

I believe there Is an example in the swift-aws-lambda-runtime repo, but it doesn't get indexed effectively by google and it's a little hard to tell how up-to-date it is.

4 Likes
8

I'm extremely interested in this topic. I am about to launch a startup that uses S3 for the assets and, for now, CloudKit as the backend, but I previously started writing my own backend in Vapor. The ideal scenario would be to use Vapor for the metadata, presumably with postgres, and S3 for the assets.

The barrier for me is the extreme amount of overhead. It took me longer to set up all the infrastructure than writing the Vapor code. I don't know what docker, kybernetes etc are, and ideally I want to remain that way. Is it really necessary to deal with all of that?

2 Likes
9

Something like what SST is doing /w other languages. They're currently offering JavaScript and TypeScript /w support for Go, Python, C#, and F# coming soon. Not seen any timelines on the "coming soon".

https://docs.serverless-stack.com/installation

1 Like
10

Have you tried using Heroku? I'm using it to host a Vapor + Postgres stack, and it required hardly any setup. It relies on a buildpack, so there's no containers, Docker, Kubernetes, ...

2 Likes
11

docker is useful for server-side development, but you shouldn’t have to deploy it on the server. it just makes local iteration and testing easier. you can certainly start developing on AWS without docker, but eventually you will reach for a tool that lets you emulate an EC2 instance locally, and you will discover that this tool already exists and that it is called docker.

you do not need to know anything about kubernetes to get started with AWS.

many people incorrectly assume you need familiarity with a whole bunch of disparate technologies to develop on AWS, and this sadly discourages many from giving it a try.

1 Like
12

Yes I think there was a dash of heroku in the mix there as well. Perhaps I didn't use kubernetes, but I'm pretty sure docker was involved. Glad to hear that it's superfluous, one less thing to worry about.

I wish there was a simple guide for people that don't know any of this stuff. Instead if I google it, there will be lines like "if you want to use Vapor with postgres in a docker container then do X", and I have no idea if I want to use that, I don't know what any of it is. I really don't want to use any of these technologies, I just want to set up a backend server.

1 Like
13

If only there was documentation telling us exactly what we need. It does seem that we need a separate postgres server running for example, which is a nightmare to get working, but that information isn't really anywhere. I want a guide for people who aren't already backend developers.

And I'm sorry, but Vapor + Postgres + SQL + Docker + AWS + S3 + Heroku does seem like "a lot of disparate technologies" to me.

14

to address these one by one

  1. Vapor: optional if you know SwiftNIO, and i would argue you’re better off learning SwiftNIO since Vapor is built atop of SwiftNIO.

  2. Postgres: skip unless you need to be persisting things (like user accounts) that would require a database. many web services do not actually need a database.

  3. SQL: same as above.

  4. Docker: optional, but you are much better off learning this early, as it will just make your workflow so much easier.

  5. AWS: mandatory, but that’s already a given.

  6. S3: skip unless you know ahead of time you need to be storing large amounts of static content (eg images, video, ML training sets, etc.). you will probably need a database before you need S3 storage.

  7. Heroku: i have never used Heroku, so i don’t have anything to say about it.

most of these technologies exist to solve specific problems that crop up once a service starts to scale. they are not prerequisites by any means.

if you use swift on linux regularly, and have some basic understanding of networking, you are in a good position to get started with AWS.

1 Like
15

Thank you very much for the explanations. I don't know if I need AWS, I guess that's where I put the server code? Since the purpose of my backend is to store large numbers of asset files, plus structured metadata that needs to be queryable somehow - can't just download all of it, it seems I definitely need at least Vapor, Postgres, AWS and S3. Heroku also seems necessary for deploying it, I wouldn't even know where to start deploying it manually.

I absolutely wouldn't want to write my own server framework in SwiftNIO.

I want maximum simplicity, I want to learn as little as humanly possible about all this infrastructure stuff, it's just overhead for me. I'm not worried about performance, or server costs, for now. If things scale I will hire someone that enjoys this stuff, and want to tinker with it low level.

16

AWS is a collection of technologies, S3 is part of AWS. based on your description, you probably do not need to use Postgres, S3 should suffice.

Vapor and AWS have nothing to do with one another. AWS (EC2) is just a place you can run a Vapor app; you could also run a Vapor app locally. in fact this is why Docker is useful, because it lets you emulate an EC2 instance locally.

i really do not think Heroku needs to be involved at any stage of this process.

if you want to go through the march of progress yourself, start by writing a SwiftNIO server that runs locally on your laptop and then try accessing it from other devices on the same wifi.

then if you have a VPN with port forwarding, try accessing it from cellular data, which should teach you the basics about accessing things over the internet.

then realize that it’s annoying to maintain a laptop running 24/7 with a VPN, and that it would be much easier and cheaper to rent a CPU and a network connection from a large corporation that delivers this service at scale. this is what AWS EC2 is. instead of running your SwiftNIO app on your laptop, connect to an EC2 instance over SSH, transfer the app over, and run it there.

then realize that your EC2 instance does not have a lot of disk space to store all your asset files, and that it would make sense to offload some of that to a service that’s optimized for storing lots of static data. this is what AWS S3 is.

in general, you do not “learn” AWS the way you learn SwiftNIO/Vapor/Postgres. you “learn” AWS the way you “learn” GitHub, to use an analogy. if you get familiar with AWS, a lot of that knowledge will transfer over to other cloud providers (Heroku, Google, etc.), just like being familiar with GitHub would help you use GitLab, BitBucket, etc.

4 Likes
17

That’s the thing, I don’t want to start low level, I don’t want to learn any transferable skills. I want to just write my backend code, define the database schema and press play.

Like you can pretty much do with a mobile app. At least there are guides that will take you through everything else.

I know AWS and Vapor are different things, I have written and deployed a Vapor app, and I absolutely don’t want to write a SwiftNIO app, that seems like writing a mobile app in CoreGraphics instead of SwiftUI.

And I don’t see how I could get by without postgres, or some form of database, since I have structured data that need to be queried. I presume it needs to be persisted.

As I said, I will happily sacrifice performance, precision and price, but I don’t want to have to separately learn about all these technologies, and try to figure pit what I need and how to install it.

There should be step by step guides for different use cases. Backend app without database, backend app with database, backend app with database and assets, etc.

2 Likes
18

It's clear from the above that basic 'getting started' guides would be useful, both for the very basic (Elastic Beanstalk or App Runner) and more complex (ECS/EKS/EC2 etc)

@GreatApe both App Runner and Heroic would be useful to you, you don't really need to know any thing about how to configure it all, it's just a few commands and go. Heroic has add ons for the DB to make it easy to add - check out the Vapor docs

19

Thanks, I will read up on those. Seems they are utilities provided by Amazon, probably filling a role similar to heroku?

Yes guides would be incredibly useful, but even just a high level guide that goes through all the components would be very useful. All the things that are needed to get the backend code deployed and connected to a db, and maybe to some asset storage. Once we know what parts we need, we can always research them separately.

As a newcomer to backend there are loads of things we have no idea about, that server veterans take for granted. Those are the most important things to document. Once I’ve figured out that I need postgres, I can always find the details elsewhere.

20

Is it correct to say that aws-sdk-swift currently does not have a working custom endpoints so that could connect to MinIO or LocalStack ? Providing custom EndpointResolver like on #533 doesn't lead to succesfull connections to S3 API.

2 Likes