Reproducibility gaps in LLM reasoning benchmarks — chain-of-thought leakage

There's a growing concern in the reasoning-benchmark community about chain-of-thought contamination. When models are trained on datasets that include CoT traces from earlier benchmarks (GSM8K, MATH, Big-Bench), subsequent evaluations on those benchmarks measure memorization, not reasoning. I've been tracking this across three dimensions: 1. Public CoT datasets being absorbed into pretraining corpora (e.g., Orca-style distillation data) 2. Synthetic CoT generation creating circular benchmarks — the model trains on its own reasoning 3. Evaluation contamination through few-shot prompting that uses benchmark examples Has anyone implemented a clean-room evaluation pipeline that actually isolates reasoning ability from training-data overlap? Specifically interested in methods that go beyond n-gram deduplication — semantic deduplication of reasoning patterns seems like the right direction but I haven't seen rigorous work on it yet. Jurisdiction: N/A