Clinical LLM evaluation
ClinEval measures hallucinations, failure modes, and demographic equity for LLM-generated medical notes. Synthetic doctor–patient dialogues are turned into SOAP notes, decomposed into atomic claims, and each claim is grounded against the transcript — supported, unsupported, or contradicted.
Unsupported claim rate
5.8%
unsupported + contradicted ÷ claims
Contradiction rate
0.1%
directly contradicted ÷ claims
Planted-fact recall
98.6%
hybrid deterministic + LLM entailment match
Dialogues evaluated
80
904 atomic claims judged
Deterministic findings
65
rule-based violations (no LLM)
What this measures
- Unsupported claim rate — share of generated claims the transcript does not support (unsupported + contradicted).
- Contradiction rate — claims the transcript directly contradicts: the most dangerous failure mode.
- Planted-fact recall — of the facts deliberately seeded into each dialogue, how many the note recovered.
- Equity — every metric sliced by language profile, age band, sex, and complexity, with bootstrap CIs.
Methodology
- Dialogues and notes are fully synthetic — no PHI, no real patients.
- The grounding judge scores all of a dialogue's claims together, batched by index against the transcript, but never sees the SOAP note body — so a verdict can't leak from the note. Runs at temperature 0.
- Deterministic rule checks (drug/dose match, ICD-10 validity, dangling dates) run without an LLM.
- Per-slice deltas carry a 1,000-resample cluster bootstrap CI so gaps aren't overclaimed.
Generated 2026-06-10T07:57:31.599736Z · model llama-3.3-70b-versatile · contract v2