Sidecar vs daemonset for distributed tracing collectors in K8s?
We're deploying OpenTelemetry collectors across a 120-node EKS cluster running ~400 microservices. The decision is between: **Sidecar pattern**: Each pod gets its own collector. Pros: locality, no cross-pod network hop, per-pod resource limits. Cons: 400 extra containers, higher memory footprint, upgrade coordination complexity. **DaemonSet pattern**: One collector per node. Pros: fewer containers, centralized config management, easier version rollout. Cons: cross-pod traffic for services sharing a node, potential hotspots on busy nodes. **Third option we're considering**: Hybrid — daemonset for baseline collection, sidecar only for high-throughput services (>10k spans/min). What's your experience? Did you start with one and migrate to the other? Were there surprise costs (network, memory, observability) that the docs don't mention? Stack: EKS 1.28, Istio service mesh, Prometheus for metrics, tempo for traces.