Key findings
- Without tools, Muse Spark 1.3 got 25.4% of its citations wrong and Gemini 3.8 Flash 16.7%. Claude Opus 5.5 got 1.1% wrong, the fewest of any model.
- Without tools, Muse invented at least one authority in 45% of its answers, and Gemini 3.8 Flash in 41%.
- Web search helped the two weakest models most. Muse fell to 1.2% wrong and Gemini 3.8 Flash to 6.3%. Claude Opus moved from 1.1% to 1.7%.
- With web search, neither GPT model invented an authority, but both still got details wrong. GPT 5.6 Sol miscited 3.1% of its citations and GPT 6 Luna 6.2%.
- Google's Gemini 4 models cited more accurately than Gemini 3.8 Flash. Gemini 4 Carbon got 2.9% wrong without tools, second only to Claude Opus 5.5, and 3.7% with web search. Gemini 4 Argon got 11.4% wrong without tools and 5.2% with web search.
| # | Setup | Correct | Wrong |
|---|---|---|---|
| 1 | Claude Opus 5.5 (no tools) | 98.9% | 1.1% |
| 2 | Muse Spark 1.3 (web search) | 98.8% | 1.2% |
| 3 | Claude Opus 5.5 (web search) | 98.3% | 1.7% |
| 4 | Gemini 4 Carbon (no tools) | 97.1% | 2.9% |
| 5 | GPT 5.6 Sol (web search) | 96.9% | 3.1% |
| 6 | Claude Haiku 5.5 (web search) | 96.7% | 3.3% |
| 7 | Claude Haiku 5.5 (no tools) | 96.5% | 3.5% |
| 8 | Gemini 4 Carbon (web search) | 96.3% | 3.7% |
| 9 | Gemini 4 Argon (web search) | 94.8% | 5.2% |
| 10 | GPT 6 Luna (web search) | 93.8% | 6.2% |
| 11 | Gemini 3.8 Flash (web search) | 93.7% | 6.3% |
| 12 | GPT 5.6 Sol (no tools) | 91.0% | 9.0% |
| 13 | GPT 6 Luna (no tools) | 90.0% | 10.0% |
| 14 | Gemini 4 Argon (no tools) | 88.6% | 11.4% |
| 15 | Gemini 3.8 Flash (no tools) | 83.3% | 16.7% |
| 16 | Muse Spark 1.3 (no tools) | 74.6% | 25.4% |
What a hallucination looks like
A hallucinated citation reads like a real one. A fake case has a plausible name, a real reporter and a page number. A fake statute has a section number that fits the code around it. The only way to catch either is to look it up.
What we measured
- Hallucinated
- No case, statute or rule matching the citation exists.
- Miscited
- The authority is real, but a detail is wrong, such as the page, the year or the court.
- Correct
- Neither hallucinated nor miscited, as a share of the citations we could check. The few we could settle neither way are left out.
How we ran it
We wrote 120 questions of the kind a practicing attorney asks, each built around a real decision from a named American court. Every model answered every question twice: once with no tools, and once with its maker's own web search, at its maker's default reasoning level.
A checker with web access looked up each citation in court websites, official reporters, the Caselaw Access Project and other open sources. A second, separate pass re-checked every citation the first pass marked as hallucinated.
Limitations
- OpenCase, which makes AI software for legal research, designed and ran this benchmark. We publish every question, citation and verdict so anyone can re-check our work.
- A few questions got no answer, and each is left out of that setup's rate. Muse refused 4 questions in both modes under its content filter, and its web search failed on 2 more. Gemini 3.8 Flash with web search returned no answer to 9, even after repeated retries. Gemini 4 Carbon timed out or returned no answer to 1 of 120 questions without tools and 7 with web search.
- Gemini 4 Argon often dropped or lowercased the first letters of a case name, writing “jasper v. State” for Jasper v. State. It did so in about one answer in five without tools. Gemini 4 Carbon did it in a handful of answers. Where the rest of the citation matched a real case, we counted the clipped name as a typo, not a hallucination. Where it matched no real case, we counted a hallucination. A reader could not always look these cases up as written.
- Each answer could use up to 16,000 output tokens, including the model's reasoning. Gemini 4 Carbon used them all on 11 answers without tools, which stop mid-answer. We checked the citations they gave before the cut. No other model reached the limit.
- Each model ran at its default reasoning level. A higher setting may cite more accurately, at more cost and delay.
- The rate is per citation. It does not measure whether an answer cited enough authority, or whether a real case supports the point it was cited for. The few citations the checker could settle neither way are left out.