Long text (~1594 words), small codex (11 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Z.AI GLM 5 Turbo	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5.4 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
GPT-5.4 (Reasoning, Low)	100%
Z.AI GLM 5	100%
GPT-5.4 Mini (Reasoning)	100%
o4 Mini High	100%
GPT-4.1	100%
Stealth: Healer Alpha	100%
GPT-5.4 Mini (Reasoning, Low)	100%
Nemotron 3 Super	100%
Inception Mercury 2	100%
Inception Mercury	100%
GPT-5.4 Nano (Reasoning, Low)	100%
GPT-4o Mini (temp=1)	100%
GPT-4o Mini (temp=0)	100%
ByteDance Seed 1.6 Flash	100%

	Score	Cost	Time
LFM2 24B	100%	$0.0002	1.3s
GPT-4.1 Nano	78%	$0.0003	1.4s
GPT-4o Mini (temp=1)	100%	$0.0004	597ms
GPT-4o Mini (temp=0)	100%	$0.0005	670ms
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.0s
Inception Mercury	100%	$0.0003	4.6s
Inception Mercury 2	100%	$0.0018	2.7s
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0029	3.0s
GPT-5.4 Nano (Reasoning)	93%	$0.0014	6.7s
GPT-4.1	100%	$0.0045	728ms
ByteDance Seed 1.6 Flash	100%	$0.0008	9.1s
Stealth: Healer Alpha	100%	$0.0000	15.3s
Grok 4.1 Fast	85%	$0.0022	20.0s
Z.AI GLM 5 Turbo	100%	$0.0064	16.6s
MiniMax M2.5	73%	$0.0029	22.1s
GPT-5.4 Mini (Reasoning)	100%	$0.0084	8.9s
Mistral Small 4 (Reasoning)	78%	$0.0030	31.2s
Gemini 2.5 Flash Lite (Reasoning)	78%	$0.0030	24.9s
GPT-5.4 (Reasoning, Low)	100%	$0.010	9.1s
Cohere Command R+ (Aug. 2024)	65%	$0.013	2.6s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
31	Claude Opus 4.6 (Reasoning)	$0.144	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Z.AI GLM 5 Turbo	$0.0064	16.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
38	Claude Sonnet 4.6 (Reasoning)	$0.150	2.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5.4 (Reasoning)	$0.029	27.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	GPT-5 Mini	$0.0060	40.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	GPT-5.1	$0.028	32.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5.4 (Reasoning, Low)	$0.010	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Z.AI GLM 5	$0.014	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
11	GPT-5.4 Mini (Reasoning)	$0.0084	8.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	o4 Mini High	$0.022	48.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	GPT-4.1	$0.0045	728ms	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Stealth: Healer Alpha	$0.0000	15.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Mini (Reasoning, Low)	$0.0029	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Nemotron 3 Super	$0.0000	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury 2	$0.0018	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Inception Mercury	$0.0003	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	GPT-4o Mini (temp=1)	$0.0004	597ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	GPT-4o Mini (temp=0)	$0.0005	670ms	100%	100	100	100	100	100	100	100	100	100	100	100%
8	ByteDance Seed 1.6 Flash	$0.0008	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	LFM2 24B	$0.0002	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
55	Gemini 3.1 Pro (Preview)	$0.144	1.7m	55%	100	100	100	100	100	100	100	100	100	25	93%
29	GPT-5	$0.045	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
77	Gemini 3 Pro (Preview)	$0.181	2.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
21	o4 Mini	$0.0094	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
23	ByteDance Seed 2.0 Mini	$0.0025	2.0m	55%	100	100	100	100	100	100	100	100	100	25	93%
26	Z.AI GLM 4.7 Flash	$0.0038	2.5m	55%	100	100	100	100	100	100	100	100	100	25	93%
22	GPT-5 Nano	$0.0044	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
20	GPT-5.4 Nano (Reasoning)	$0.0014	6.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Qwen 3.5 9B	$0.0038	4.5m	60%	100	100	100	100	100	100	100	100	50	50	90%
27	Claude Sonnet 4.6	$0.029	15.0s	43%	100	100	100	100	100	100	100	100	33	25	86%
28	Grok 4.20 (Beta, Reasoning)	$0.027	16.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
24	Grok 4.1 Fast	$0.0022	20.0s	40%	100	100	100	100	100	100	100	100	25	25	85%
25	Qwen 3 32B	$0.0015	48.1s	40%	100	100	100	100	100	100	100	100	25	25	85%
36	Nemotron 3 Nano	$0.0022	2.1m	37%	100	100	100	100	100	100	100	100	25	17	84%
30	GPT-4.1 Nano	$0.0003	1.4s	29%	100	100	100	100	100	100	100	50	25	3	78%
34	MiniMax M2.7	$0.0066	29.8s	31%	100	100	100	100	100	100	100	25	25	25	78%
41	Qwen 3.5 Flash	$0.0070	1.8m	31%	100	100	100	100	100	100	100	25	25	25	78%
32	Gemini 2.5 Flash Lite (Reasoning)	$0.0030	24.9s	31%	100	100	100	100	100	100	100	25	25	25	78%
33	Mistral Small 4 (Reasoning)	$0.0030	31.2s	31%	100	100	100	100	100	100	100	25	25	25	78%
35	MiniMax M2.5	$0.0029	22.1s	31%	100	100	100	100	100	100	50	25	25	25	73%
39	Z.AI GLM 4.5	$0.0047	42.7s	28%	100	100	100	100	100	100	50	33	26	8	72%
112	Qwen 3.5 27B	$0.056	5.6m	27%	100	100	100	100	100	100	25	25	25	25	70%
93	Qwen 3.5 35B	$0.064	3.9m	27%	100	100	100	100	100	100	25	25	25	25	70%
42	Stealth: Hunter Alpha	$0.0000	1.3m	27%	100	100	100	100	100	100	25	25	25	25	70%
49	Z.AI GLM 4.6	$0.023	1.8m	22%	100	100	100	100	100	100	42	25	25	0	69%
43	Cohere Command R+ (Aug. 2024)	$0.013	2.6s	15%	100	100	100	100	100	100	17	13	13	10	65%
67	Z.AI GLM 4.7	$0.029	2.4m	16%	100	100	100	100	100	25	25	25	25	25	63%
45	Grok 4 Fast	$0.0020	24.0s	16%	100	100	100	100	100	25	25	25	25	25	63%
48	GPT-5.2	$0.019	20.2s	14%	100	100	100	100	100	25	25	25	25	17	62%
44	GPT-4o, Aug. 6th (temp=1)	$0.0075	1.3s	15%	100	100	100	100	100	42	38	17	10	10	62%
47	Mistral Medium 3.1	$0.0027	4.4s	11%	100	100	100	100	42	33	33	33	30	6	58%
46	Hermes 3 70B	$0.0016	12.9s	19%	100	100	100	50	50	50	38	33	32	8	56%
64	Aion 2.0	$0.011	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
52	Arcee AI: Trinity Large (Preview)	$0.0000	20.5s	5%	100	100	100	100	29	28	28	10	8	4	51%
58	ByteDance Seed 1.6	$0.0056	46.9s	8%	100	100	100	25	25	25	25	25	25	25	48%
40	Claude Haiku 4.5	$0.0076	5.4s	44%	50	50	50	50	50	50	50	42	38	35	46%
114	Grok 4	$0.108	2.5m	7%	100	100	100	25	25	25	25	17	17	17	45%
71	Gemini 3 Flash (Preview, Reasoning)	$0.025	46.9s	7%	100	100	100	25	25	25	25	17	17	13	45%
116	Qwen 3.5 122B	$0.166	7.7m	7%	100	100	100	25	25	25	25	17	13	13	44%
99	Gemini 2.5 Pro	$0.060	45.6s	10%	100	100	25	25	25	25	25	25	25	25	40%
60	Gemini 2.5 Flash (Reasoning)	$0.0094	15.1s	10%	100	100	25	25	25	25	25	25	25	25	40%
50	Claude Opus 4.5	$0.033	5.9s	36%	42	42	42	42	38	38	38	38	38	35	39%
59	Claude Opus 4.6	$0.043	14.0s	30%	50	50	50	50	42	42	42	38	13	10	38%
61	Ministral 8B	$0.0007	9.6s	4%	100	100	100	27	26	26	2	1	1	1	38%
56	Qwen 2.5 72B	$0.0007	6.7s	10%	100	100	38	35	31	31	10	10	8	7	37%
62	Llama 3.1 Nemotron 70B	$0.0068	17.7s	9%	100	100	42	35	31	27	10	7	6	6	36%
54	GPT-5.4 Nano	$0.0004	1.7s	14%	100	50	50	35	29	25	25	25	17	6	36%
65	Claude Sonnet 4	$0.020	5.8s	13%	100	50	42	38	38	17	13	13	13	10	33%
51	GPT-4.1 Mini	$0.0023	7.3s	24%	50	50	33	33	33	32	31	29	28	2	32%
104	MoonshotAI: Kimi K2.5	$0.019	2.4m	14%	100	25	25	25	25	25	25	25	25	17	32%
72	Arcee AI: Trinity Mini	$0.0014	45.3s	6%	100	100	25	25	25	17	13	10	2	0	32%
69	Hermes 3 405B	$0.0052	6.8s	4%	100	100	17	13	13	13	13	10	10	10	30%
109	Claude Opus 4	$0.098	8.1s	27%	50	42	42	38	38	38	17	13	13	7	29%
70	Llama 3.1 70B	$0.0024	9.1s	3%	100	100	25	17	13	10	8	6	6	6	29%
100	Qwen 3.5 Plus (2026-02-15)	$0.024	2.2m	20%	50	50	50	32	31	30	28	8	6	6	29%
63	Mistral Small 4	$0.0013	8.8s	13%	100	35	28	28	27	27	27	2	2	2	28%
53	Ministral 3B	$0.0003	7.5s	26%	32	31	29	28	27	27	26	26	26	26	28%
76	Grok 4.20 (Beta)	$0.0060	3.1s	1%	100	100	30	5	5	5	4	4	4	4	26%
89	Ministral 3 8B	$0.0020	53.4s	0%	100	100	26	26	2	2	1	1	1	0	26%
115	Qwen 3.5 397B A17B	$0.037	7.1m	25%	25	25	25	25	25	25	25	25	25	25	25%
68	WizardLM 2 8x22b	$0.0037	57.8s	21%	50	31	29	29	29	28	28	13	5	4	25%
75	Gemini 2.5 Flash Lite	$0.0007	2.1s	1%	100	100	8	7	7	5	5	4	3	1	24%
57	GPT-5.4 Mini	$0.0015	930ms	23%	25	25	25	25	25	25	25	25	17	17	23%
82	GPT-4o, May 13th (temp=1)	$0.029	3.2s	14%	38	33	33	33	25	13	8	8	8	7	21%
66	Llama 3.1 8B	$0.0002	16.1s	17%	42	31	30	30	28	17	10	8	6	5	21%
81	Claude 3.7 Sonnet	$0.019	4.1s	11%	50	42	17	17	17	13	13	13	13	13	20%
74	Claude 3 Haiku	$0.0018	3.8s	9%	38	38	33	33	13	13	10	8	6	5	20%
87	GPT-5.4	$0.0089	8.3s	4%	100	17	17	10	8	8	5	4	4	4	18%
91	ByteDance Seed 2.0 Lite	$0.0079	1.3m	16%	25	17	17	17	17	17	17	17	17	17	18%
73	DeepSeek V3 (2024-12-26)	$0.0019	6.1s	12%	33	25	25	25	17	13	13	8	8	8	18%
85	Mistral Small 3.2 24B	$0.0010	13.9s	2%	100	32	6	6	6	6	5	4	4	3	17%
79	Mistral Large 3	$0.0043	19.7s	12%	29	29	29	29	28	4	3	3	3	3	16%
94	DeepSeek V3.1	$0.0016	38.9s	6%	50	29	25	10	10	7	7	5	5	1	15%
92	Ministral 3 3B	$0.0017	49.7s	10%	28	28	26	26	25	2	2	1	0	0	14%
106	Claude 3.5 Sonnet	$0.041	8.2s	12%	17	17	17	13	13	13	13	13	13	10	14%
78	Gemini 3.1 Flash Lite (Preview)	$0.0019	1.9s	10%	33	13	13	13	13	10	10	10	10	8	13%
80	DeepSeek V3 (2025-03-24)	$0.0012	12.4s	11%	25	17	13	13	13	13	13	10	8	8	13%
90	DeepSeek V3.2	$0.0015	31.6s	8%	28	26	13	13	10	10	7	7	6	5	12%
84	DeepSeek-V2 Chat	$0.0021	8.9s	8%	35	13	13	13	10	10	8	8	6	6	12%
108	Mistral Large 2	$0.017	17.6s	3%	29	28	27	4	4	4	4	4	4	3	11%
107	Claude Sonnet 4.5	$0.025	8.9s	7%	33	8	8	8	8	7	7	7	7	6	10%
83	Gemini 3 Flash (Preview)	$0.0023	3.7s	10%	13	13	13	10	10	10	8	8	8	7	10%
102	Ministral 3 14B	$0.0015	20.5s	2%	29	28	25	3	3	2	2	1	1	1	9%
98	Mistral Small Creative	$0.0011	10.7s	2%	27	27	27	2	2	2	2	2	2	1	9%
86	Gemma 3 12B	$0.0002	8.5s	8%	17	13	13	10	10	7	6	6	6	6	9%
103	Qwen3 235B A22B Instruct 2507	$0.0012	39.6s	4%	28	28	6	5	5	5	5	4	2	2	9%
96	Rocinante 12B	$0.0014	20.1s	6%	30	13	10	7	7	6	4	4	4	3	9%
111	Mistral Large	$0.017	16.5s	3%	29	28	5	4	4	4	4	4	3	3	9%
110	Writer: Palmyra X5	$0.013	22.2s	4%	28	27	5	5	5	4	4	3	2	2	8%
88	Gemini 2.5 Flash	$0.0021	2.0s	8%	10	10	8	8	8	8	8	8	7	5	8%
97	GPT-4o, Aug. 6th (temp=0)	$0.011	2.8s	7%	10	10	10	10	8	7	7	6	5	5	8%
95	Gemma 3 27B	$0.0006	11.4s	7%	10	8	8	8	7	7	7	7	6	5	7%
113	GPT-4o, May 13th (temp=0)	$0.033	4.9s	6%	8	7	7	7	6	6	6	6	5	5	6%
101	Mistral NeMO	$0.0009	7.8s	4%	7	7	6	5	4	4	4	4	3	2	5%
105	Gemma 3 4B	$0.0003	9.9s	3%	4	3	3	3	3	3	3	3	2	2	3%
52.36%

Median	Evaluator	Top 3	Flop 3
45.0%	Correct "no violations" response	100GPT-4o Mini (temp=1) 100Inception Mercury 100Claude Opus 4.6 (Reasoning)	0Qwen 3.5 397B A17B 0GPT-5.4 Mini 0Gemma 3 27B
55.8%	No hallucinated violations	100ByteDance Seed 1.6 Flash 100GPT-5.4 Nano (Reasoning, Low) 100GPT-5.1	3Ministral 3 3B 4Mistral Small Creative 4Ministral 3 14B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)