How to Test Your OpenTelemetry Collector Configuration Before Production
A misconfigured OpenTelemetry Collector rarely fails loudly. It fails quietly — a processor silently drops spans that don't match its expected shape, an exporter retries into a void because of a typo'd endpoint, a sampling rule filters out exactly the traces you needed during an incident. By the time anyone notices, it's usually because they went looking for data that should have been there and wasn't.
Testing a collector config properly means checking each stage of the pipeline independently, then checking that the whole chain works together — not just sending one test span and calling it done.
1. Test the receiver first, in isolation
Before anything else, confirm the collector is actually listening where you think it is. This catches the most common early mistake: a receiver bound to the wrong port, protocol, or interface.
- Check the collector's own logs on startup — it will log which receivers started and on which address.
- For gRPC receivers, a plain TCP connection test is often more useful than trying to send real telemetry — if the port refuses connections, nothing downstream matters yet.
- For HTTP receivers, confirm the exact path (
/v1/traces,/v1/logs,/v1/metrics) — a receiver can be "up" while still rejecting every request because of a path mismatch.
2. Test processors with data that actually varies
This is where most configs get under-tested. A single hand-crafted test span rarely exercises the processors that matter most:
- Batch processors — do they actually batch, or is every span going out individually because the batch size/timeout never gets hit under low test volume?
- Sampling processors — test with both a normal-looking trace and an explicitly error-tagged one. Tail-based sampling in particular needs traces with varying attributes to prove it's making the right keep/drop decisions.
- Attribute processors — confirm renamed or redacted fields actually show up correctly (or don't show up, if you're testing PII scrubbing) on the far side, not just in the collector's own debug output.
http.status_code >= 500 will pass a manual test with a 200 response and never reveal itself as broken — you only find out the rule was inverted when a real error trace goes missing during an actual incident.3. Test exporters against the real backend, not just "it connected"
A successful TCP handshake to your backend doesn't mean data is actually being accepted. Confirm:
- Auth headers are correct — many backends accept the connection but silently reject unauthenticated payloads
- TLS/mTLS certs are valid for the actual hostname being used, not just present
- Data genuinely appears in the backend's UI or query interface, searchable by something specific (a trace ID, a distinctive attribute) — not just "the export count went up"
4. Test the whole chain together — with correlation intact
The most common gap: each stage tests fine independently, but trace-log correlation breaks somewhere in the pipeline. A log processor that strips trace_id during redaction, or a batch that separates related signals across export cycles, can silently disconnect logs from the traces that produced them — without any error, warning, or dropped-data metric to flag it.
The only reliable way to catch this: generate a trace and its correlated logs together, then confirm in your backend that searching by that trace ID actually surfaces both. This has to be an explicit check — collector metrics alone won't tell you correlation broke.
Generating the test data itself
You have a few real options here, each with a different tradeoff:
- Manual OTLP payloads (hand-crafted JSON, or tools like
telemetrygen) — fast, precise, good for testing one specific rule in isolation. Weak for realistic volume or multi-signal correlation testing. - Full demo applications (e.g., the OpenTelemetry Astronomy Shop) — realistic, but heavy to deploy just to validate a collector config change.
- A dedicated generator — something that produces realistic, correlated traces/logs/metrics across many simulated services at once, so you're testing against traffic that actually resembles production rather than one clean example.
This is the exact gap OTel Studio is built for — generating correlated, multi-service telemetry against your actual collector config, so you can run through this whole checklist against realistic traffic patterns rather than a single synthetic span.
A minimal pre-production checklist
- Receiver reachable on the expected protocol/port/path
- Batch processor actually batching under realistic volume
- Sampling rules tested against both normal and error-flagged traces
- Attribute redaction/renaming verified on the exported side, not just collector debug logs
- Exporter auth/TLS confirmed against the real backend, not just a TCP check
- Trace-log correlation confirmed by searching a specific trace ID in your backend and finding both signal types
None of this replaces real production traffic eventually finding the gaps you missed. But catching the obvious ones — a wrong path, an inverted sampling rule, redaction that doesn't redact — before that traffic arrives is the difference between a quiet config change and a 2am page.
Related guides
- OTLP gRPC vs HTTP: Which Should You Use — for the receiver connectivity questions above
- Understanding Trace-Log Correlation — the full detail on checklist item 4
- Testing Your Grafana Stack With Synthetic Telemetry — this checklist applied to Tempo/Loki/Mimir specifically