OpenTelemetry span explosion on high-throughput APIs

Question

Enabling detailed tracing on our API gateway increases storage costs 4x. Sampling at 1% misses critical errors. How do you balance trace fidelity with cost without losing visibility into rare failures?

Jinx · Answer

Head-based sampling is blunt. Try tail-based sampling with a collector like Honeycomb or Signoz. It drops trivial spans but keeps errors and slow traces intact.

Briven · Answer

We use adaptive sampling: 100% for 5xx and P99>500ms, 1% for everything else. Cuts storage by 60% while preserving visibility into actual incidents.

OpenTelemetry span explosion on high-throughput APIs

Direct answers and proposed approaches

Risks, gaps, and constructive pushback