[GSoC 2024] AsyncHTTPClient Distributed Tracing Instrument Proposal

Development Standard Library
1

Hello Swift Community,

I am Heng Zhou, I am now a graduate student at University Florida in Computer Science.

I am interesting in the project of adding Distributed Tracing Instrument for AsyncHTTPClient. I took the distributed system course last semester, where I read the paper of Google Dapper and generally know how distributed tracing works. Now I want get some hands-on experience.

I did some homework and write an initial version of proposal.

The major question I have now is whether to adopt OpenTelemetry or OpenTracing as the tracing specification. They both provide Swift support.

@ktoso Looking forward to connect with you and all the Swift community! Comments/suggestions on my proposal are welcomed.

Best,
Heng

1 Like
2

Hello and thanks for reaching out.

"Open tracing" is pretty much dead; and open telemetry has replaced it.

We also have two implementations for the swift ecosystem: GitHub - open-telemetry/opentelemetry-swift: OpenTelemetry API for Swift and GitHub - slashmo/swift-otel: OpenTelemetry client built for server-side Swift (a minimal backend to emit OTLP data from swift-metrics/log/distributed-tracing).

You'd probably be best off collaborating with @slashmo on using the otelp lib for the demos.

On the HTTP client side, @fabianfett and @lukasa I know had ideas how to integrate. Note that you'd depend on GitHub - apple/swift-distributed-tracing: Instrumentation library for Swift server applications to do so, rather than any of the above specific backends.

We do wonder though if this project isn't too small, and if it would not be over within a few days... Perhaps it may help the proposal's changes to explore what else you could do with tracing in the server ecosystem, and perhaps expand this project to add to a few more things maybe?

Thanks and while fuzzy I hope these hints help a bit in shaping up the proposal.

1 Like
3

Thank you for the clarification. Your reply is very informative and helpful. I will revise my proposal.

We do wonder though if this project isn't too small, and if it would not be over within a few days... Perhaps it may help the proposal's changes to explore what else you could do with tracing in the server ecosystem, and perhaps expand this project to add to a few more things maybe?

A few days is too aggressive. I need 1-2 weeks to get familiar with Swift. But I do feel it's a 1 month project, not a 3 month project. I lean to expand the project to add more things so that I can get more hands-on experience. But is there any ideas? I see at server side, the hummingbird framework already have the TracingMiddleware, gRPC instrument is also done at: Add client and server tracing interceptors by gjcairo · Pull Request #1756 · grpc/grpc-swift · GitHub

But I see vapor framework does not support tracing yet, see: Adopt Swift Tracing · Issue #3033 · vapor/vapor · GitHub

Probably we can expand the project by adding instrument to both async-http-client and vapor, WDYT? Or do you have any other good ideas?

1 Like
4

Hi @ktoso,

I have revised the proposal according to your suggestions. If you have any further suggestions, please let me know. You can comment in the google docs.

5

It seems people like discussion in the forum, instead of externally at Google Docs. Just copy my proposal draft here to increase visibility.

@ktoso Let me know if you have any further comments/suggestions.

Distributed Tracing to AsyncHTTPClient and Vapor

Synopsis

This project aims at providing an out-of-box distributed tracing instrumentation for async-http-client and Vapor:

  • async-http-client is the recommended HTTP client for server applications in Swift

  • Vapor is a widely used HTTP web framework for Swift. With powerful built-in modules including intelligent request routing, RESTful support and middlewares for intercepting HTTP requests. Vapor makes it very convenient to build a web application from scratch.

This proposal aims at instrumenting async-http-client and Vapor with the swift-distributed-tracing library.

Benefits to Community

With tracing instrumentation, swift developers can add (or switch) tracing implementations/backends with an O(1) configuration change. More importantly, when Tracing is instrumented in async-http-client and Vapor, a low enough level in the software stack, even large applications could be traced without additional annotations. Based on our contributions, the application programmers even do not need to be aware of the tracing system.

Deliverables

Client side instrumentation

A PR to async-http-client will be raised to add the tracing instrumentation. Unit test will be provided to verify that the expected tracing HTTP header has been added to HTTP request if tracing is enabled.

Server side instrumentation

A PR to Vapor will be raised to add the tracing instrumentation. It will be a Vapor Middleware, and implement the following Middleware protocol.

@preconcurrency
public protocol Middleware: Sendable {
func respond(to request: Request, chainingTo next: Responder) -> EventLoopFuture<Response>
}

Inside the respond() function, it will:

  • Mutate the Request object by adding tracing information. If the request is already attached with a Span created by client side, reuse it, otherwise create a new one and record the tracing metrics.
  • Mutate the Response object by adding tracing information

Please refer to Middleware.swift for more details about the Vapor Middleware protocol.

Demo App

A Demo app will be provided to showcase the instrumentation works

Implementation

Information to be emitted

Tentatively the following information will be emitted

  • component name
  • operation name
  • HTTP method
  • HTTP URL
  • HTTP status code

Tracing specification

We will adopt OpenTelemetry as the tracing specification/protocol. Please refer to the opentelemetry-specification for more details.

Demo App

Demo App client side

Client side will send an async http request to server side using the instrumented async-http-client. It will be a simple HTTP GET request to http://127.0.0.1:8080

Demo App server side

Server side will use the Vapor framework and enable the newly added Tracing Middleware. The server will listen to 8080 port

It will use the swift-otel library to create a OTel span exporter, exporting the metrics/spans info via port 4317

Demo App Tracing backend

We will launch a Jaeger as the tracing backend, which collects the metrics/spans info exposed at port 4317 by the OTel span exporter. The jaeger-ui will be used to visualize all the information being emitted.

Demo App bundle

We will provide a Dockerfile for the demo app client side and server side respectively, then use Docker Compose to bundle the demo app as well as the Jaeger and jaeger-ui, so that developers can quickly try out the functionality.

Alternatives considered

Opentracing

While the Opentracing project used to be actively developed and maintained, the project is archived now, see: OpenTracing has been Archived. We have decided to adopt OpenTelemetry as the tracing specification

opentelemetry-swift

While OpenTelemetry community provide a opentelemetry-swift library, we have decide to use the swift-distributed-tracing library to to the instrumentation, as ​​swift-distributed-tracing library is more suitable for instrumenting multithreaded and distributed systems with Distributed Traces

Related Work

6

@0xTim I wonder if you could comment on the viability of instrumenting the existing Vapor with tracing -- think this could be a 2 week project for someone? Or would traces end up broken due to the future based internals -- or is it still possible to do somehow you think in the current shape of middleware, before moving more into all-async? :thinking:

7

I don't think it will be that long tbh - but there are a number of routes you could take. One is finish the work to migrate Vapor's responder chain to fully async (last time I tried I was hitting some NIO assertion errors before other stuff took precedence) in which case we have a simple middleware that relies on talk locals. Option 2 is do that Hummingbird does and copy the context across boundaries, both work though I'd lean towards option 1 at this point.

Happy to help advise over the course of GSoC though

2 Likes