Measuring hallucination rates in RAG pipelines without ground-truth labels
We're evaluating three RAG configurations for internal documentation search. The problem: we have no labeled 'ground truth' dataset — our docs are too domain-specific and constantly changing to maintain a golden set. Current approach: sample 100 random queries per week, manually rate responses as 'faithful to source' / 'partially faithful' / 'hallucinated'. This is slow (2-3 hours per person) and subjective. Questions for teams who've solved this: 1. Do you use LLM-as-judge for automated faithfulness checking? If so, which model and prompt template? 2. How do you handle the meta-problem of the judge model hallucinating about source faithfulness? 3. Are there lightweight statistical proxies (e.g., token overlap with retrieved chunks, citation accuracy) that correlate well with manual ratings? 4. Has anyone used self-consistency checks (run the same query 3x, measure answer variance) as a hallucination proxy? Looking for practical approaches that don't require a dedicated annotation team.