What is wrong
A captured HTTP body that is not valid UTF-8 is destroyed before it reaches ClickHouse. Rows are written as JSONEachRow, and Go's encoding/json replaces every invalid byte with U+FFFD. Measured on the Zstandard frame magic:
in 12 bytes: 28 b5 2f fd 64 8f 17 e5 35 00 5a 6d
out 20 bytes: 28 ef bf bd 2f ef bf bd 64 ef bf bd 17 ef bf bd 35 00 5a 6d
Four distinct bytes collapse to the same three-byte sequence, so the conversion is lossy and not invertible. The row still reports the original req_body_size, so the stored value silently disagrees with the length beside it.
What it costs
Observed on edge4, tailscale-system/ts-traefik-…, an ordinary captured row:
POST /c/tailnode.log.tailscale.io/<id> HTTP/1.1 200 OK
req_body_size 1740, latency 130 ms, trace_role 1 (client)
req_body begins 28 b5 2f fd → a zstd frame
The body is compressed, not encrypted: a plain zstd decoder would yield the JSON log lines. Nothing about the payload prevents decoding — only the storage path does. Every compressed, protobuf, gzip or otherwise binary body in the evidence tables is in the same state today.
What to do
- Encode at capture, before the value becomes a JSON string. base64 (or hex) the body columns on the way in, with a companion column naming the encoding, so a reader knows what it holds. This is the whole fix for the loss; it must happen before
json.Marshal, not after.
- Make the disagreement detectable meanwhile. Store the byte length actually written beside the reported
req_body_size; a mismatch is then a readable state rather than an invisible one.
- Decode on read, not on write. Detect the frame from its magic (zstd
28 b5 2f fd, gzip 1f 8b, …) in the view or the script, so the stored bytes stay exactly what crossed the wire and the interpretation is a reader's choice.
Decision needed before this is built
Encoding bodies faithfully turns the evidence store into a store of decodable third-party payloads. The row above is a neighbouring pod's log upload; the same path captures any plaintext request body that is traced, not only the subject of an investigation. Two questions for the owner of that decision, and I would rather they were answered than assumed:
- Should bodies be captured for every traced pod, or only for the pod under investigation?
- Should a body be stored whole, or truncated to a bounded prefix with the full length recorded?
Whatever the answer, it should be stated in the trace record rather than implied by what happens to be present, so a reader can tell "no body" from "body not kept".
What is wrong
A captured HTTP body that is not valid UTF-8 is destroyed before it reaches ClickHouse. Rows are written as JSONEachRow, and Go's
encoding/jsonreplaces every invalid byte with U+FFFD. Measured on the Zstandard frame magic:Four distinct bytes collapse to the same three-byte sequence, so the conversion is lossy and not invertible. The row still reports the original
req_body_size, so the stored value silently disagrees with the length beside it.What it costs
Observed on edge4,
tailscale-system/ts-traefik-…, an ordinary captured row:The body is compressed, not encrypted: a plain zstd decoder would yield the JSON log lines. Nothing about the payload prevents decoding — only the storage path does. Every compressed, protobuf, gzip or otherwise binary body in the evidence tables is in the same state today.
What to do
json.Marshal, not after.req_body_size; a mismatch is then a readable state rather than an invisible one.28 b5 2f fd, gzip1f 8b, …) in the view or the script, so the stored bytes stay exactly what crossed the wire and the interpretation is a reader's choice.Decision needed before this is built
Encoding bodies faithfully turns the evidence store into a store of decodable third-party payloads. The row above is a neighbouring pod's log upload; the same path captures any plaintext request body that is traced, not only the subject of an investigation. Two questions for the owner of that decision, and I would rather they were answered than assumed:
Whatever the answer, it should be stated in the trace record rather than implied by what happens to be present, so a reader can tell "no body" from "body not kept".