Metrics Extras

One of the benefits of swift-metrics being an API-only package is that it doesn't contain anything you don't need. The drawback being that it may not contain some of the things you do want!

As such I'd like to pitch a small package containing a set of tools related to swift-metrics.

Proposed tools:

  • Logging factory: a MetricsFactory which logs any changes to the underlying instruments. Logging would be limited to swift-log or print. This is helpful for general debugging and sanity-checking during development.
  • Delegating factory: MetricsFactory which delegates to a "current" factory. Since it is only possible to bootstrap the metrics system once per process (testable imports aside) it makes testing more difficult as noted in https://github.com/apple/swift-metrics/issues/60. The factory would be a singleton which would delegate to a swappable, user-provided factory. It would also have to keep track of where each instrument originates in order to correctly destroy them. This would allow the factory to be swapped out depending on test requirements. It could also handle the awkwardness of installing itself since the user would otherwise have to do keep track of this in tests.
  • Capture factory: a MetricsFactory which records structured events in a way that is suitable for testing. The rough idea being "I executed this code path and I want to check that this counter was incremented by this value and has this label and these dimensions". This requires some more exploration as to what shape this would take to be maximally useful here.

I'm curious as to whether the community thinks there's enough value in having a such a package, whether the proposed tools would be useful, and if there are any other suggestions for factories or otherwise which could be included here.

8 Likes

I'm supportive of such utils and package :+1:

The 2nd (a way to change backend "per test") and 3rd one (a way to capture and assert on metrics/logs) I had to (re-)invent myself in a project I work on basically immediately as we shipped swift-metrics, so yes, I agree they're valuable in certain use-cases. :smile: It is an unusual form of testing, however can be pretty useful if one wants to ensure some effects that otherwise would be hard to observe (without exposing tons of internals), so I'm +1 on that.

For what it's worth I'm also using the same approach for logs and find it tremendously valuable. Specifically, each test gets it's "LogCapture" and executes, if the tests passed -> no output, however if they failed all logs (including trace, debug, ... level logs) are dumped which makes for a good experience when a flaky test happens -- no need to ever adjust "increase logging" and be worried it'll pollute the successful executions.

Similarly, if a library expects that the main way others will diagnose problems when things go wrong is metrics and logs, it makes sense to sometimes for the lib to add tests that "yup, the log we make here reads well and always happens in this weird situation" etc.

So... swift-metrics-extras and swift-log-extras?
Or should we rather phrase and focus those more specifically around testing -- swift-metrics-testkit and swift-log-testkit?

That's a great idea!

These are intended for testing (of sorts) so I'm happy to set expectations upfront and include "test" in the name. I'd be okay with the -testkit names unless better alternatives are suggested.

1 Like

It's also worth noting explicitly that 2. and 3. also apply to swift-log as you allude to.

1 Like

Tagging @tomerd @weissi for ideas where this could end up living (eventually might be nice for those to be close to where the API packages live or swift-server hm...) and how to get there :rocket:

Maybe we PoC them out somewhere, do a pitch and they could graduate them eventually?

I'd be a fan of developing it as a separate product in each project, as *TestUtils - but that might require a whole SSWG proposal (or several).

1 Like

It's a difficult one; having a separate product obviously makes it much easier to adopt. It also makes it much harder to say no to other factories being included in each project ("why is this one allowed in the package but this one isn't?").

1 Like

Oh I think I misread @Mordil's response here. To clarify (seems I can't "un-like" :slight_smile:) my position on this is in line with what George is saying:

I do not think it's a good idea to add any implementations to the API projects.

As @georgebarnett says, the API projects then become harder to defend from feature creep -- they really should remain API projects, and nothing more to keep them as light and flexible as possible, without any creep. (That the logging one has a default impl is still a bit annoying IMHO... it's stopped people from implementing a really-great stdout one since the one shipping by default is good enough™ for simple stuff...).

Sounds like we can try to PoC something quickly soon out and come up with a pitch soon :slight_smile:

2 Likes

As far as pitching and where to let these live goes. I think we can start with hosting it on the GH of whoever starts working on it, and discuss in the pitch/proposal stage where to let it land. Could be apple/ or swift-server/ for both, but I'm personally also not opposed to having it at a single person's GH (like SwiftPrometheus, APNSwift and RediStack)

However I think getting a pitch & PoC up are more important than figuring out where exactly to host the repo :slight_smile:

2 Likes

+1 on the idea and +1 to start something as a personal repo for the purpose of a pitch / proposal and as it matures we can decide on a good home for it. this is what we did when we originally developer swift-log and swift-metrics proposals.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy