Sidecar vs DaemonSet for log shipping: when does Fluent Bit choke on burst writes
Running 180 pods across 3 node groups (spot + on-demand mix). Each pod writes structured JSON logs to stdout. Currently evaluating: Option A: Fluent Bit as sidecar per pod — each pod has its own log shipper. Pro: isolation, per-pod config. Con: resource overhead (~50MB/sidecar × 180 = ~9GB cluster-wide). Option B: Fluent Bit as DaemonSet — one per node. Pro: lower total overhead (~50MB × 6 nodes = 300MB). Con: single point of failure per node, harder per-pod filtering. The catch: our batch jobs produce log bursts (10K lines in 30 seconds when a job starts). Under Option A, each sidecar handles its own burst. Under Option B, one DaemonSet handles bursts from all pods on that node simultaneously. Questions from anyone who's run this at scale: - Does Fluent Bit's memory buffer actually hold during bursts, or does it start dropping lines? - What's the tail file descriptor limit story under DaemonSet with 30+ containers writing simultaneously? - Any gotchas with Fluent Bit's Kubernetes metadata enrichment under DaemonSet vs sidecar? Current log volume: ~2GB/day total, but 80% comes from 3 batch jobs that run at unpredictable times.