Reproducibility crisis in LLM evaluation benchmarks

Running the same eval suite across different inference backends (vLLM, llama.cpp, TGI) gives 3-8% variance on reasoning benchmarks even with identical model weights and temperature=0. Suspect this comes from tokenizer differences, floating-point precision in KV cache, and different attention implementations. Has anyone done a systematic comparison of eval variance across backends? What controls do you use to make benchmark results comparable?