Handling production server crashes

Hey folks, wondering what the best/favored methods of logging crashes for production server apps people have used are.

I've got error logs being sent to Datadog, but ideally would love to log any crash stack trace to there (as well as alert in slack) if the app dies.

I'm using supervisor to restart the app if it does die, so maybe it's best to just tap into that somehow. Or perhaps something in the server that detects if it's coming back from a crash and uploads the stack trace (like crashlytics / bugsnag on iOS). Not sure what solutions are out there.

Are you using Swift Backtrace? That'll give you a full stack trace if your app crashes.

For capturing logs it entirely depends on your deployment scenario. We're predominantly Fargate on AWS so we log to the console and AWS take care of capturing logs and pushing it to CloudWatch. How do you push your logs to DataDog? You don't want to be doing it in app because a) your app has to know about DataDog and b) if it crashes you get no logs. You'll want to either stream the logs to a collector running in a sidecar (a seperate process/container) or get the collector to read the console output. The collector can then push the logs to DataDog

Yeah using swift backtrace.

Okay, so basically have a collector that's watching the output of the server and have that log to DataDog? Any examples of out of the box solutions? Or would I have to create a separate swift app and run it.

Right now yeah, the app itself is logging to Datadog with GitHub - jagreenwood/swift-log-datadog: Send logs to Datadog with Swift.

Looking at their site they have an agent that should do what you need. You need to configure it to read logs from wherever your swift app puts them

1 Like