Guide

Testing Your Grafana Stack With Synthetic Telemetry

A practical walkthrough for validating Tempo, Loki, and Mimir dashboards and alerts before real traffic arrives.

Standing up Grafana with Tempo (traces), Loki (logs), and Mimir (metrics) is the easy part — most of the actual setup guides stop there. The harder, less-documented part is knowing whether it actually works before you're relying on it during an incident. This walks through validating each piece specifically, using OTel Studio to generate realistic traffic against your stack.

1. Point OTel Studio at your Grafana stack

Grafana Alloy is the modern recommended OTLP receiver for the Grafana stack, replacing the older Grafana Agent. OTel Studio has a one-click preset for it:

Endpoints page → Stack Presets → Grafana Alloy
Pick gRPC (port 4317) or HTTP (port 4318) depending on your network path — see our OTLP gRPC vs HTTP guide if you're unsure which fits your setup
If Alloy is running in Docker alongside OTel Studio, remember both containers need to share a Docker network for the hostname to resolve

2. Validate Tempo (traces) specifically

Start a scenario, then in Grafana's Explore view, switch to the Tempo data source and search by service name. What to actually check, not just "did anything show up":

Does the trace waterfall show the correct parent-child span structure, or does it look flattened/disconnected?
Are span durations reasonable, or is everything showing 0ms (a common sign of a timestamp/clock issue between the generator and collector)?
Click into a span's attributes — are they actually populated, or empty despite the span existing?

3. Validate Loki (logs) — and the correlation specifically

This is the step most setups skip, and the one most likely to be silently broken. Copy a trace ID from a Tempo trace, then search Loki directly for that same ID (not through Grafana's "correlated logs" panel — search Loki's raw query interface directly, so you're testing the actual data, not a UI feature that might mask missing correlation). If nothing comes back, correlation is broken somewhere in the pipeline — see our trace-log correlation guide for exactly why this happens and how to find the specific cause.

4. Validate Mimir (metrics) and actually test your alert rules

Metrics validation is often reduced to "did the counter go up," which misses the real point — you're not just checking that metrics arrive, you're checking that your alert rules fire correctly against them. OTel Studio's Anomaly Injection is built for exactly this: a 30-second burst of 10× traffic and 80% error rate, controllable on demand, so you can watch your actual alert rules trigger against realistic conditions rather than trusting they're configured correctly without ever seeing them fire.

A specific thing worth testing deliberately: error-budget and SLO burn-rate alerts are usually the least-tested part of a new observability setup, precisely because they need sustained abnormal conditions to trigger — conditions real traffic rarely provides on a predictable schedule during setup. A controllable synthetic burst is the practical way to confirm the alert rule's math is actually correct before you're depending on it.

A minimal validation checklist

Traces appear in Tempo with correct span hierarchy and non-zero durations
Log records in Loki are searchable by the exact trace ID from a Tempo trace
Metrics in Mimir reflect the actual scenario configuration (rate, error percentage)
At least one alert rule has been deliberately triggered and confirmed to fire
Dashboards render correctly across a range of traffic patterns — not just the default "everything healthy" case

None of this replaces validating against real production traffic eventually. But confirming the stack actually works — correlation intact, alerts firing correctly — before that traffic exists is what turns "we think it's configured right" into something you've actually verified.