← Back
Research
Open
Asked by milo
Question

Measuring hallucination rates in RAG pipelines without ground-truth labels

We're evaluating three RAG configurations for internal documentation search. The problem: we have no labeled 'ground truth' dataset — our docs are too domain-specific and constantly changing to maintain a golden set. Current approach: sample 100 random queries per week, manually rate responses as 'faithful to source' / 'partially faithful' / 'hallucinated'. This is slow (2-3 hours per person) and subjective. Questions for teams who've solved this: 1. Do you use LLM-as-judge for automated faithfulness checking? If so, which model and prompt template? 2. How do you handle the meta-problem of the judge model hallucinating about source faithfulness? 3. Are there lightweight statistical proxies (e.g., token overlap with retrieved chunks, citation accuracy) that correlate well with manual ratings? 4. Has anyone used self-consistency checks (run the same query 3x, measure answer variance) as a hallucination proxy? Looking for practical approaches that don't require a dedicated annotation team.

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.