Since the server side group is now entering a new phase, I was wondering if the following was a problem that has been thought about before or that other server side developers see a need for.
Namely, it's about the fact that Swift apps can crash in many ways. Some of them are preventable, e.g. by avoiding force unwraps, but others aren't always: for example, it might be a bug in a library, it might be an integer overflow (where it might be difficult to strictly guarantee it won't ever happen), an invalid array access, some concurrency bug, ... Basically, programming errors will happen and invalid assumptions will be made, and so crashes will occur.
Swift doesn't have a way of recovering from those crashes, by default. This is probably ok for app development. After all, if an app really does crash, the user can just restart it (and there's probably some Apple tools that can report crash logs to the developers). However, for server side applications that won't work, as a bad request could bring down the whole server.
The solution we're using now is to just have multiple instances running and have the load balancer take control in the case of a crash (e.g. by restarting); but that's additional work and also somewhat inflexible.
I was wondering if some tooling could be built than can handle these kinds of situations more seamlessly, so not everyone has to reinvent the wheel. I'm not sure exactly what the best solution would be (maybe supervisor trees like Erlang has?), but maybe some other people have ideas.
That's interesting, can you share some more details about those kinds of features and how they can be used for Swift?
@zoul: Yes, I've seen the manifesto before (even though I've never managed to fully read it, I probably should...). It paints a really bright future, and I hope that Swift will have this one day (the earlier the better), but of course I can't hope that this is something that is going to happen quickly. My question is whether there could be any community solutions for the meantime. It could be as simple as a tutorial that is linked from swift.org, for that matter, if there's solutions that can already be used (e.g. Apache, or maybe some supervisor tools, or some docker solution...). Even just a mention of the issue might be enough, as I suspect that quite a lot of people wouldn't think of this problem initially.
I'm going to write out my doubts here just so that they're written up; please don't take it as complete dismissal of the idea or anything.
If a thread goes down in-process—as in, insta-exit, not an exception—you've irretrievably lost the memory and resources that thread was holding unless you've registered some way to recover them from outside. For memory that might mean a leak; for file handles that might mean a broken connection or a truncated file on disk; for synchronization mechanisms across threads/actors, you've got a pending deadlock. So you'd need some exception-handling-like way to register cleanups, only with an outside thread running them.
(Swift is very unlikely to get true exception-handling; it mucks with the entire rest of the language model when every call can fail.)
If there's any memory-sharing, the memory might have been corrupted, so if we want to keep safety in the Swift sense, we need something like Go or Rust has to prevent sharing across actors.
Swift talks to C a fair amount. We'd have to figure out what uses of C in the runtime and in corelibs are not compatible with this kind of recovery, and either avoid them or document the Swift-side API as "recovery-unsafe" or whatever. (Libraries that people use on their own, including Libc, are the responsibility of the developer using them…)
These are the big things coming to mind right now.
Thank you for the detail. I'm not involved enough with the internals of the language to comment on them, so I'm not sure what the best solution would be.
Regardless, this is a problem that all server-side applications will run into, so this will need to be handled at some point. Currently, everyone is probably doing it their own way (Apache, load balancer, docker, systemd, ...), so it would be nice to have some kind of standard solution. This might be (long-term) an extension of the language of some sorts (as outlined in the manifesto above), but it could also just be additional, officially maintained tooling. Or even just a recommendation to use something that already exists.
As long as it is not possible to "catch exceptions" in Swift, it would seem our only option is to standardize some best practices for minimizing the impact of crashes.
Vapor currently recommends using a combination of nginx and supervisor when deploying your app to production. Nginx should help to deliver a proper 5xx if the reverse proxied server goes down unexpectedly (much better than just getting a TCP / socket error on the client side). Supervisor should instantly restart the app minimizing downtime after a crash.
I'm very interested to hear what others think is the best practice here.
I'm interested to learn more about what Apache offers. Are there things it can do that Nginx can't?
Sorry, I'll clarify the execption-handling comment a bit. What I mean by "Swift is unlikely to get true exception-handling" is
No way to stop a fault from taking out the current thread. (How this interacts with higher-level parallelism abstractions like Dispatch is a different and complicated problem.)
No way to perform cleanups that run on the same thread when a fault occurs. In general this probably scales up to "within the same actor", but I won't make that claim right now.
No way for an arbitrary function call to end without
Returning
Throwing (in the Swift.Error sense)
Exiting the thread or process
Trapping / faulting
"Catching exceptions" usually encompasses all three of these, but note that even if you didn't want to catch the exception and had absolutely no custom cleanup to perform (finally blocks), you'd still have pieces of the problems I mentioned above.
Even in these situations, there's usually an opportunity for improvement over sudden death that doesn't require perfect recovery or isolation. A server wants a trap handling one request to not immediately take down other requests; although it's formally possible for badly behaved code to leave any shared resources in inconsistent states, concurrent requests hopefully already have relatively isolated state by necessity of their execution environment, so in practice it's probably OK for the supervising event loop to stop taking new requests and try to let the surviving requests complete before killing itself. There are other places where it'd be nice to have "soft landing" crash handling without perfect recovery; for example, test runners would like to test behavior that may trap, and test should be simple and short-lived enough that leaked resources aren't an issue. StdlibUnittest forks to test crashes, but that's pretty heavyweight, and not all of our target platforms even support forking.
To that end, it wouldn't be unreasonable to have a setjmp-ish mechanism for introducing a last resort crash handler that at least was guaranteed to run in response to the safety check traps Swift itself emits, if not arbitrary signals raised by any C code. This wouldn't have to be a beautiful or composable API, since we don't want user code using it for traps as mundane control flow as happens too often with runtime exceptions in Java. Maybe we could require that the handler still end the process instead of being allowed to return. Maybe "actors will solve this" in the fullness of time, but that's just a beautiful idea, and it wouldn't hurt to have a less beautiful practical alternative in the short term.
Nginx returning a 500 is good, as is being able to bring back a process through supervisor. However, that still results in all other in flight requests dying (unless you also retry them somehow), which IMO is still a huge problem.
Thanks everyone for all the helpful replies. Clearly you are all much more knowledgeable than I am. If I understand correctly, I can see that the following solutions have already been tried out and found to work:
Swift server code as an Apache module with mod_swift
nginx + supervisord
for our own service, we use a docker container that runs supervisord internally to restart the service if it died; additionally a load balancer sits in front of the docker cluster and will detect the health of the containers periodically (unfortunately, this means that if supervisord can't restart the service for some reason, we have a time window in which we might lose requests); I'm not sure if that's the best solution.
In addition, the following possible future solutions have been brought up:
A more resilient, maybe actor-based way of writing Swift code (if I understand @jrose correctly, that will mean that any recovery mechanism would have to run at least in a different thread)—long term vision, of course
a more short-term solution where there is a (purposefully limited) last-resort crash handler that could be used in Swift code, as proposed by @Joe_Groff
maybe a standard community solution based on external tooling
Is there interest in setting up a page somewhere that outlines some of these ideas in more detail as a reference for all server-side developers?
Some additional questions:
Do you have a resource for that? I looked at the vapor docs, but haven't found anything about this. It would be very useful.
It looks like it installs a Mach exception handler to respond to the Mach exceptions that happen to get raised by the code current Swift compilers and LLVM backends generate for traps. These aren't stable; there is currently no guarantee as to how functions like precondition or fatalError really end the process. An official crash handler mechanism would have to be coordinated with Swift's code generation patterns for traps.
If I understand correctly, we have two somewhat separate issues then (might be worth splitting one of these off into a different thread):
1: Long-term.
We should continue to discuss the concept of “soft landing” crashes. If / when the mechanism for doing this becomes available in Swift, the @server-workgroup can adopt them and recommend best practices for using them.
2: Short-term.
The @server-workgroup should consider recommending best practices for minimizing the impact of exceptions. We could agree upon what we think those best practices are (be it Nginx, Apache, whatever) and make that information available. Even if the recommendation is simply to consider this potential pitfall, that is better than nothing.
Establishing and documenting current best practices while discussing both near- and far-term improvements sounds like a good approach. Even with soft landing crash handlers or in-process actors, I suspect many people would still want to rely on process isolation with existing mitigation practices for the added security and reliability.
Probably 99% of web services out there run in interpreted languages of some sort (including JVM languages and of course also PHP, Ruby, Python, Perl, Javascript, ...). With these kinds of languages hard crashes are very rare; you'll still have to protect yourself against it in the long run, but it's definitely not something you need to deal with right away. So yes, I think in that sense this is "language specific" in the sense that people used to doing web services in these languages wouldn't think of this problem in the same way.
Plus, Swift is supposed to be easy to learn, so I'm not sold on the idea of saying "they should know this already and it's documented elsewhere".
I'd like to suggest that fast fail is intact a feature rather then a bug.
In todays container based world where bootstrapping a new "docker instance" is almost immediate you might be wanting the service to crash when reaching a state of fault. You want the cluster manager to bring a new instance into existence connect it to the load balancer and continue running without requiring manual intervention.
You might also want to old instance to remain dead so that you can investigate what happened and debug.
How would these connections be handled in case of a rollout? How would this work in environments where CI is pushing new server version into productions multiple times a day?
I'm imagine some retry logic, either at load balancer or client side would be implemented to make this work.
I think you raise a good architectural point, would be interested to learn about solutions.
The combination of Swift's "unsafe" behaviors and the "one high-capacity process" design of NIO seems to all but demand a "recovery" model. One potential solution in the short term (absent deeper language/runtime support for dealing with these concerns) might be to adopt the "terminate a crashing thread" model, let the failed thread's resources leak (individual threads should be fairly well isolated in NIO's universe), and transition all remaining threads to an "orderly shutdown" state while also triggering spin-up of a new process. This allows for existing connections to finish servicing while minimizing downtime by leveraging the annoying truth that the majority of "crashing" crashes, while they do conceptually compromise the entire process' state, don't in practice corrupt all of the state, or at least not all at once.
The issue then becomes the security implications of the approach - not immediately terminating the entire process when a fault occurs potentially opens exploitation opportunities, even though the individual faulted thread is stopped.
This solution might come at the cost of increased memory footprint when scaling up. Just heard from a JS guy one of the major drawback of Node is the single-threaded nature coupled with browser sized memory need (~1 GB). Scaling up process wise is not a cheap option.
