Aren't we missing a swift-monitoring library?

Server
1

I believe the current ecosystem is missing an easy plug and play solution to receive real time notifications about server-side events.

The solution I usually implement is the following (example based on AWS CloudWatch but I am sure it can be applied to any other metric systems as well):

  • Setup a CloudWatch Alarm.
  • swift-metrics pushes metrics to CloudWatch using a custom EMF backend.
  • The alarm publishes relevant events on SNS, which then notifies our team.

While this solution works great for most technical or diagnostic oriented scenarios, I would argue it easily becomes an (expensive) overkill approach for very simple cases where we just want to know that something happened. This doesn't necessarily mean an error, in those cases the alarm based solution is definitely better suited, but there are cases where you are just "curious" about an event.

For example when an invoice has not being paid for a few days, you might want to receive a notification so that you can drop an email to your client and ask them if there is any problem, or when a user opens a ticket that requires immediate attention. Another example might be receiving a notification when someone signs up to your newly deployed app. These scenarios are not technical and IMHO, using metrics and alarms to monitor them just doesn't feel right.

AFAIK there currently isn't any well suited solution other than implementing a custom logging backend, which then carries the problem of too much noise or inelegant hacky approaches like "Logs this on Slack also, but only if the x-slack metadata is present".

Now, of course this would not be useful to very big companies with advanced alarming setup and heavy traffic, but I believe many small-medium projects and startups will largely benefit from having a simple plug and play library like this, where they could easily choose their own backend, be it Slack, Discord, Telegram, SNS, APNS or whatever.

Thoughts?

1 Like
2

I find it hard to visualize what specifically of a library you’re looking for from this description? Is it an email scheduler or some front end in front of a bunch of notification APIs perhaps? I could see that, like an API hiding various SMS or email or other notification APIs I suppose, is this the idea? That sounds like a notification library, not necessarily monitoring but yeah it could be useful :)

Clarifying this and if you’re looking for interest and are building such or are you looking to clarify what such library would even be…?

1 Like
3

Of course, here is a snippet of how I imagine it

// entry point
MonitoringSystem.bootstrap { label in
    TelegramMonitoringHandler(
        label: label,
        botToken: "",
        channelID: "",
        topicID: 1
    )
}

// somewhere else
let monitor = Monitor(label: "Unpaid Invoice")
monitor.recordEvent("Invoice 1234 has not been paid")

EDIT

If there is interest in it, I'd be willing to propose it as part of the server ecosystem. I'll probably build one anyway.

4

Is the idea that you'd have a monitoring system that listens for these events and handles them? Why can't this be done with the metrics API?

5

But your examples are domain logic, that don't require real-time notice and aren't detectable via infrastructure logic. Normally that's handled with middleware cognizant of domain logic and entity relations, sent not via logging but via messaging/event queues or db triggers.

When server-side application middleware evolves in Swift, the features you want will be included.

The middle ground is tracing tags that can associate multiple log messages with the same transaction, but that's more for higher-level systems debugging/forensics/evaluation than for business.

6

Technically speaking there isn't anything preventing the metrics API to be used. This is what we currently use right now and it does work. I don't feel like this is the right approach though.

  1. Metrics are more oriented to the overall health of the system, I don't really think anyone would want to check their CloudWatch or Grafana dashboard do see if there are unpaid invoices.
  2. If you wish to get notified about an event, you need to configure an additional layer with alarms or rules.
  3. Metrics do not carries information about an event. The only way to handle the above invoice example with metrics, would be to create a new dimension value for each invoice ID, and high cardinality metrics aren't much useful (nor cheap depending on which backend you are using).

Indeed they are, which is one more reason why I believe none of the current pillars (swift-log, swift-metrics, swift-distributed-tracing) are well suited for this specific scenario, and hence why I opened this thread.

For what concerns middlewares, I can see them as a viable solution, even though it would feel strange to have the code sending the event in a completely different file compared to the code that detects the event. Also, I have always considered middlewares more as agnostic decorators, and not much as business logic aware pipelines, but this is just a personal preference I guess. If there is already a conversation in place about the evolution of server-side application middlewares in Swift, I'd love to see it.

I definitely picked the wrong name for this topic, as @ktoso suggested this is more a notification library. I did not want to use the word notification to avoid confusion with push notifications, but it is indeed a notification system.

1 Like
7

One solution to this would be something like Vapor's Queues Package which schedules the jobs for the due date, and when it's run, checks it's been paid and if not fires off some notifications (or publishes a topic on an event system for different subsystems to pick up depending on the kind of architecture you have)

2 Likes
8

Vapor's queues is one of the many possible approaches, not exactly ideal for any system that isn't relying on Vapor though, as, like most of Vapor's packages, it forces you to import the whole Vapor stack.

I've probably failed in explaining why I have opened up this discussion, my point wasn't about asking for solutions, it was about understanding why there isn't an official server side Swift library for such scenarios.

My point is, why do we have official support for the following

LoggingSystem.bootstrap { CloudWatchBackend($0) }
MetricsSystem.bootstrap(EMFBackend())
InstrumentationSystem.bootstrap(XRayBackend())

but not for this?

NotificationSystem.bootstrap { SNSBackend($0) }
9

This makes me wonder if swift supports opentelemetry yet which is the standard at my employer

10

Yes, actually there's a quite nice (and still evolving) opentelemetry implementation for server-side swift - each of logging, metrics, and tracing.

And there's a CNCF library that's independent, but (IMO) a little more difficult to embrace as a library because it doesn't break up the API from the SDK, so to use theirs, you end up having to take a massive dependency hit (for example, when adding any of that feature set to a library)

2 Likes
11

Thx I’ll take a look!

12

Yes, via two separate approaches.

If there's anything missing please ping on either of those repositories and start a forums thread and we'll figure it out.

1 Like
13

Perhaps the community just didn't come around to it yet. Also, I can imagine that this one is a little more complex to make into a general solution.

Sending a notification is off course one thing you might want to do, but perhaps:

  • If an invoice remains unpaid, perhaps you want to kick off a seperate business process: hand over to specific team, create a reminder, hand over to debt collection. This one probably also has its own deadlines with escalations.
  • If an invoice remains unpaid, perhaps you want to block new orders from the related business partner.
  • If an invoice remains unpaid, perhaps you want to pause the service you provide.
  • ...

In more general terms, you have a business process step that "breaches" for one reason or the other (in the example, a deadline). Do you only want to be notified of the "breach", or do you want (in more general terms) want to kick off another business process?

14

Thanks I'll take a look.

1 Like