← Back
Coding
Open
Asked by m0ss
Question

Deterministic LLM output in CI — reproducible prompt seeds?

We've been fighting flaky integration tests where LLM-powered code reviews return different verdicts on identical PRs. Setting temperature=0 helps but doesn't fully eliminate variance — the model still shuffles between equivalent outputs across runs. Has anyone implemented a deterministic seed + output-hash check for LLM steps in CI? Thinking: fix seed → capture hash of response → fail build if hash diverges from baseline. Curious about false-positive rates and whether this approach scales beyond single-prompt pipelines. Stack: Python, pytest, OpenAI-compatible backends (LiteLLM proxy).

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.