Concepts

Understanding Trace-Log Correlation in Distributed Systems

How it actually works, why it silently breaks, and how to verify it's genuinely working in your own pipeline.

"Trace-log correlation" gets mentioned constantly in observability marketing and rarely explained precisely. The mechanism is simple once you see it, but that simplicity is exactly why it's easy to silently break without any error, warning, or dropped-data metric telling you it happened.

What's actually happening under the hood

When a span starts, OpenTelemetry generates a trace_id and span_id and stores them in the current execution context. That context is a specific, addressable piece of state — not just metadata floating nearby. Any log statement emitted while that context is active can read those same IDs and attach them to the log record.

That's the entire mechanism. There's no special "linking" step, no separate correlation database, no matching logic on the backend. A log either had the right context active when it was written, or it didn't — and if it didn't, there's nothing downstream that can retroactively fix it.

Why it breaks silently, specifically

This is the part that catches people off guard: a pipeline with broken correlation looks completely healthy by every normal metric. Traces export fine. Logs export fine. Volume looks normal. Nothing errors. The only symptom is that searching for a trace ID in your logs backend returns nothing — and you usually only discover that during an actual incident, when you need it most.

Three specific, common causes:

No context manager registered. In Node.js specifically, if AsyncHooksContextManager (or an equivalent) was never explicitly registered, the SDK has nothing that reliably tracks "what context is active" across asynchronous boundaries. Spans still get created fine — they're built with an explicit parent reference — but logs emitted afterward have nothing correct to read from.
Logging happens outside the span's lifetime. If a log statement fires after a span has already ended, or in a callback that executes outside the original request's async context, it silently has no trace context available at all.
A processor strips the fields. A redaction or attribute-scrubbing rule in your collector that isn't scoped carefully can remove trace_id/span_id along with whatever it was actually meant to redact.

A real example of cause #1: a Node.js service correctly exported traces and logs separately, with logs even showing up correctly in the application's own internal UI — because that UI generated its own display-layer trace ID reference, unrelated to what actually got attached to the exported OTLP log record. The two looked identical in testing. Only checking the raw log record in the actual backend revealed trace_id was empty. The fix was one missing SDK registration call, but finding it required checking the actual exported data, not a UI that happened to display something plausible-looking.

How to actually verify it — not just assume it

Because this breaks silently, "traces work and logs work" tells you nothing about whether they're correlated. The only real test:

Generate a request that produces both a trace and at least one log
Copy the exact trace ID from the trace
Search your logs backend for that specific ID
Confirm the log record's raw fields — not a UI summary — actually contain a populated trace_id

Step 4 matters specifically because some tools display a trace reference in their own UI even when the underlying exported data doesn't actually carry it — the display and the data are not always the same thing. Check the raw record, not just what a dashboard renders.

This is also why testing correlation needs realistic, multi-service traffic rather than one hand-crafted example — see our guide on testing your collector configuration for the full pre-production checklist, including this exact check.