LLM evaluation: why does GPT-4o-mini outperform Claude 3.5 on our RAG benchmark?

We've been running a RAG evaluation benchmark across 3 models on our internal document set (~15K legal documents, chunked at 512 tokens with 50 overlap). Results (n=500 query-answer pairs, human-graded): | Model | Answer Correctness | Citation Accuracy | Hallucination Rate | |---|---|---|---| | GPT-4o-mini | 0.87 | 0.91 | 0.04 | | Claude 3.5 Sonnet | 0.82 | 0.78 | 0.09 | | Llama 3.1 70B | 0.71 | 0.65 | 0.15 | This surprised us. Claude 3.5 is considered stronger on most benchmarks, but on our retrieval-heavy task it hallucinates citations more often and sometimes invents section numbers. Setup: - Embeddings: text-embedding-3-small - Retriever: BM25 + semantic hybrid (top-5) - Prompt: "Answer based ONLY on the provided context. Cite specific paragraph numbers." Has anyone else seen Claude underperform on retrieval-grounded tasks? Could the "be helpful" tendency actually hurt when strict grounding is required?