Collecting Design Requirements for a General Purpose Logging Backend

Server Development
1

After discussing this somewhat on slack, we decided it would be a good idea to collect design requirements from the server community for what you would need or like to see out of a "default choice" general-purpose logging backend.

swift-log provides an excellent API surface that many server-based deployments and packages use, but choosing a backend can be less than obvious. However, everyone's requirements for what an ideal backend should support is different, so I would like to use this space to collect everyone's thoughts so that we may design both a general purpose implementation that satisfies as many of these as possible, and an extensible API so we can grow its capabilities over time.

Right off the bat, I can think of a number of features that would be part of the ideal backend, but I'm sure there are many more:

  • Support for multiple destinations.
  • Asynchronous dispatch of log events so calling code is not slowed down by logging.
  • Destinations for console output, files with caps and log rotations, syslog logging servers, os_log on Darwin.
  • Optionally dropping logs of certain types under high volume, and logging how many are dropped.
  • Conformance to Service so log events are not dropped on server shut down.
  • Customizable formatting for log messages.

With that, I would love to open the floor with two particular questions:

  • What features would be part of your ideal general purpose logging backend?
  • What features, if available right now, would be enough to drop the current backend you are using and why?

(Note: although this discussion could easily be augmented to include apps and their requirements, I would like to purposefully limit the discussion to the server-based use cases to start, as that is the

5 Likes
2

I'll go first:

For my own projects, I would honestly be happy with:

  • a logger that can point to both the console and a file,
  • a file logging destination with a maximum size and log rotation enabled by default, so I can know with confidence I'm not going to fill the disk with logs.

Bonus points if it did the logging outside of my code's hot path, and more bonus points if it also outputted to os_log since I deploy on macOS, but neither are really necessary for me switch to a bespoke backend should one have been available.

2 Likes
3

Many thanks for bringing this up, a couple of thoughts:

‘Asynchronous dispatch of log events”

I read that as non blocking log submit api and that the logs then are asynchronously written out to the destinations - just to clarify?

Another thing I learned the hard way, is that one should be careful with how log entries are enqueued such that the running thread isn’t stolen by the os scheduler and end up processing the log entries inline. Which leads to:

Solid benchmarks for both latency and throughput would be great - one what’s to know that it’s cheap (relatively speaking) to log such that it’s possible to run with significant logging enabled in production when needed.

Another question is structured logging performance - not sure if it directly affects the backend implementation, but that’s something we’d be interest in in general.

2 Likes
4

Also being able to add compression (e.g. Lz4/zstd) for the rotated logs automatically would be nice. As would moving them to subdirectories per month. As would configuration of rotation schedule, e.g. both time (hourly, nightly, …) and/or size (e.g. max 1GB before rotation).

1 Like
5

Yes, absolutely! Ideally, the log destination should be async, ie. it can go over the network, or to a file, and shouldn't ever block, nor suspend, the thread that is calling the logger.

2 Likes
6

Thanks for kicking this off! Not sure if it's super relevant but I left some thoughts on this topic over here a few years ago.

2 Likes
7

Thank you for including this context so it is easy to find for everyone! I still think this is likely the right call for swift-log, though I’m not sure any package has stepped up to the quality standard we expect out of even the most simple of backends, so I thought now is as good a time as any to seriously consider what a community-led effort might look like. If anything, it could serve as a great starting poing for more powerful logging backends, but the current lack of a great starting point is unfortunately a sore spot at the moment.

1 Like
8

Thanks for opening this thread! I think your initial collections of features are good.

The most important feature that any logging backend needs is performance. Logs need to be as cheap as possible. Right now if in any log handler that I implemented I always had to make sure I amortise the cost of metadata formatting across log messages and make the time calculations for the log message as cheap as possible. In the end, the logging thread should be able to just quickly send off the log message and metadata and then resume its logic again.

Another feature that I have implemented in a previous log handler is customisable log levels. Right now the log level comes from the setting on the Logger itself. This is often not enough and you want to be able to dynamically customise the log level during the runtime for live troubleshooting. This includes:

  • Setting the log level on a per file, function, module basis
  • Being able to sample individual log messages from a low log level i.e. a certain trace log that would trigger 100 times per second only gets emitted once.

The last one is quite tricky and might potentially need new scoped APIs in the swift-log itself. To say something like:

withSampledLogging(logLevel: .info) {
  // every nested log statement from here on gets elevated to info
}

Taking a step back. I think the first steps to tackling this are as you did gathering requirements and then taking a stab at implementing a high performance console logger and build up from there.

4 Likes