How to manage multiple data streams (NIO/Logging/Debugging) over UART/USB?

Development Embedded
1

Hi everyone,

I'm working on an embedded project (ESP32-C6 Dev-Kit, nrf9161-DK). The goal is to create a custom TCP server on the Mac that interacts with the MCU through a non-blocking network stack, allowing me to open multiple data streams over the same connection and to support other use cases, like linking the MCU to the Mac’s network stack to provide internet access and interact with a custom shell and services on the MCU via TCP.

However, I’m running into an issue: when the ESP32 is connected via the USB/UART port, it sends all the output (e.g., printf, debug, fatalError) to the same port. This creates noise and potential data corruption, making it difficult for my TCP server to parse or interact properly.

I’m looking for advice on how to cleanly separate system logs/debug information from custom TCP data streams. I’ve considered a few options but would love input on which is most viable or if there’s a better approach:

  1. Software Multiplexing with a Virtual Stream Mapper:
  • Implement a framing protocol that distinguishes between streams by adding headers (e.g., [Stream ID][Length][Payload]).
  • Stream IDs could separate system logs (printf) from application data, allowing the Mac-side TCP server to demux them accordingly.
  1. Redirect ESP Logs to Another UART or Buffer:
  • ESP-IDF supports redirecting logs via esp_log_set_vprintf(). I could direct logs to a secondary UART or an internal buffer instead of the default UART0, keeping the main port clean for my custom streams.
  • How about development then?
  1. USB-CDC (Virtual COM Port):
  • If supported, I could switch from raw UART over USB to a USB-CDC implementation, creating virtual COM ports. This way:
    • One virtual port could handle system logs/debug messages.
    • Another virtual port could be reserved for custom TCP streams.
  • However, I'm not sure if every USB-Controller Chip supports that.

I'm happy about any input - thanks in advance for your help!

1 Like
2

Separate UART for debugging would be my suggestion if you don’t need it in production.

3

TCP framing should also be very easy if you use fixed-size integers to build a TLV format. UInt32 length and tag gives you a fixed 8-byte overhead, with enough room to send any valid buffer on a 32-bit platform and identify 2 billion data types.

I recommend fixed-width encoding for the integers because it’s cheap to encode and easy to pre-allocate space for. Variable width encodings can save bytes, but in this case that’s probably the wrong optimisation. If you really wanted to save bytes, you could send a 32-bit length and an 8-bit tag, which would save 3 bytes.

2 Likes