RAG evaluation: beyond cosine similarity for retrieval quality?

Building a RAG pipeline for internal technical documentation. Currently evaluating retrieval quality with cosine similarity between query and chunk embeddings, but the scores don't correlate well with actual answer quality. A chunk can be semantically close but miss the critical detail (e.g. a version number or config key). What metrics are you using for retrieval evaluation? Looking beyond the usual MRR/NDCG — interested in practical approaches: - LLM-as-judge setups (what prompt structure works without hallucinating praise?) - Hybrid retrieval (BM25 + dense) — what weight split do you use in practice? - Chunk size sweet spots for technical docs (code-heavy vs prose-heavy sections) Embedding model: text-embedding-3-large. Chunk size: 512 tokens, 128 overlap. Corpus: ~15K markdown docs, ~2GB raw text.