Long text (~1594 words), small codex (11 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Gemini 3.5 Flash (Reasoning)	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5.4 (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
Gemma 4 26B (Reasoning)	100%
GPT-5.4 (Reasoning, Low)	100%
Z.AI GLM 5	100%
GPT-5.4 Mini (Reasoning)	100%
o4 Mini High	100%
GPT-4.1	100%
Stealth: Healer Alpha	100%
GPT-5.4 Mini (Reasoning, Low)	100%
Nemotron 3 Super	100%
Inception Mercury 2	100%
Inception Mercury	100%

	Score	Cost	Time
LFM2 24B	100%	$0.0002	1.3s
GPT-4.1 Nano	78%	$0.0003	1.4s
GPT-4o Mini (temp=1)	100%	$0.0004	597ms
GPT-4o Mini (temp=0)	100%	$0.0005	670ms
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.0s
Inception Mercury	100%	$0.0003	4.6s
Inception Mercury 2	100%	$0.0018	2.7s
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0029	3.0s
GPT-5.4 Nano (Reasoning)	93%	$0.0014	6.7s
GPT-4.1	100%	$0.0045	728ms
ByteDance Seed 1.6 Flash	100%	$0.0008	9.1s
Stealth: Healer Alpha	100%	$0.0000	15.3s
Grok 4.1 Fast	85%	$0.0022	20.0s
GPT-OSS 120B	93%	$0.0008	34.8s
DeepSeek V4 Flash (Reasoning)	85%	$0.0010	35.4s
Z.AI GLM 5 Turbo	100%	$0.0064	16.6s
MiniMax M2.5	73%	$0.0029	22.1s
GPT-5.4 Mini (Reasoning)	100%	$0.0084	8.9s
Mistral Small 4 (Reasoning)	78%	$0.0030	31.2s
Gemini 2.5 Flash Lite (Reasoning)	78%	$0.0030	24.9s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
45	Claude Opus 4.6 (Reasoning)	$0.144	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Z.AI GLM 5.1	$0.021	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Z.AI GLM 5 Turbo	$0.0064	16.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Gemini 3.5 Flash (Reasoning)	$0.057	25.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
53	Claude Sonnet 4.6 (Reasoning)	$0.150	2.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5.4 (Reasoning)	$0.029	27.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	Claude Opus 4.7 (Reasoning)	$0.073	15.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	GPT-5 Mini	$0.0060	40.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	GPT-5.1	$0.028	32.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Gemma 4 26B (Reasoning)	$0.0031	3.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5.4 (Reasoning, Low)	$0.010	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Z.AI GLM 5	$0.014	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
11	GPT-5.4 Mini (Reasoning)	$0.0084	8.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	o4 Mini High	$0.022	48.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	GPT-4.1	$0.0045	728ms	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Stealth: Healer Alpha	$0.0000	15.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Mini (Reasoning, Low)	$0.0029	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Nemotron 3 Super	$0.0000	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury 2	$0.0018	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Inception Mercury	$0.0003	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	GPT-4o Mini (temp=1)	$0.0004	597ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	GPT-4o Mini (temp=0)	$0.0005	670ms	100%	100	100	100	100	100	100	100	100	100	100	100%
8	ByteDance Seed 1.6 Flash	$0.0008	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	LFM2 24B	$0.0002	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
77	Gemini 3.1 Pro (Preview)	$0.144	1.7m	55%	100	100	100	100	100	100	100	100	100	25	93%
27	Grok 4.3 (Reasoning)	$0.012	51.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
33	GPT-5.5 (Reasoning)	$0.042	20.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
42	GPT-5	$0.045	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
29	Grok 4.20 (Reasoning)	$0.017	48.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
28	Qwen 3.6 Flash	$0.015	45.1s	55%	100	100	100	100	100	100	100	100	100	25	93%
44	Qwen 3.6 27B	$0.031	2.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
31	Qwen 3.6 35B	$0.015	1.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
106	Gemini 3 Pro (Preview)	$0.181	2.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
26	o4 Mini	$0.0094	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
32	ByteDance Seed 2.0 Mini	$0.0025	2.0m	55%	100	100	100	100	100	100	100	100	100	25	93%
25	GPT-OSS 120B	$0.0008	34.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Z.AI GLM 4.7 Flash	$0.0038	2.5m	55%	100	100	100	100	100	100	100	100	100	25	93%
30	GPT-5 Nano	$0.0044	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
24	GPT-5.4 Nano (Reasoning)	$0.0014	6.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
52	Qwen 3.5 9B	$0.0038	4.5m	60%	100	100	100	100	100	100	100	100	50	50	90%
39	Claude Sonnet 4.6	$0.029	15.0s	43%	100	100	100	100	100	100	100	100	33	25	86%
41	Grok 4.20 (Beta, Reasoning)	$0.027	16.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
34	Grok 4.1 Fast	$0.0022	20.0s	40%	100	100	100	100	100	100	100	100	25	25	85%
35	DeepSeek V4 Flash (Reasoning)	$0.0010	35.4s	40%	100	100	100	100	100	100	100	100	25	25	85%
38	Xiaomi MIMO v2.5	$0.0092	38.1s	40%	100	100	100	100	100	100	100	100	25	25	85%
40	Z.AI GLM 4.5 Air	$0.0025	1.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
36	Qwen 3 32B	$0.0015	48.1s	40%	100	100	100	100	100	100	100	100	25	25	85%
51	Nemotron 3 Nano	$0.0022	2.1m	37%	100	100	100	100	100	100	100	100	25	17	84%
43	GPT-4.1 Nano	$0.0003	1.4s	29%	100	100	100	100	100	100	100	50	25	3	78%
50	GPT-5.5 (Reasoning, Low)	$0.021	10.6s	31%	100	100	100	100	100	100	100	25	25	25	78%
59	DeepSeek V4 Pro (Reasoning)	$0.0045	2.3m	31%	100	100	100	100	100	100	100	25	25	25	78%
48	MiniMax M2.7	$0.0066	29.8s	31%	100	100	100	100	100	100	100	25	25	25	78%
56	Xiaomi MIMO v2.5 Pro	$0.018	1.4m	31%	100	100	100	100	100	100	100	25	25	25	78%
57	Qwen 3.5 Flash	$0.0070	1.8m	31%	100	100	100	100	100	100	100	25	25	25	78%
46	Gemini 2.5 Flash Lite (Reasoning)	$0.0030	24.9s	31%	100	100	100	100	100	100	100	25	25	25	78%
47	Mistral Small 4 (Reasoning)	$0.0030	31.2s	31%	100	100	100	100	100	100	100	25	25	25	78%
49	MiniMax M2.5	$0.0029	22.1s	31%	100	100	100	100	100	100	50	25	25	25	73%
54	Z.AI GLM 4.5	$0.0047	42.7s	28%	100	100	100	100	100	100	50	33	26	8	72%
72	Qwen 3.5 Plus (2026-04-20)	$0.025	2.4m	27%	100	100	100	100	100	100	25	25	25	25	70%
144	Qwen 3.5 27B	$0.056	5.6m	27%	100	100	100	100	100	100	25	25	25	25	70%
124	Qwen 3.5 35B	$0.064	3.9m	27%	100	100	100	100	100	100	25	25	25	25	70%
58	Stealth: Hunter Alpha	$0.0000	1.3m	27%	100	100	100	100	100	100	25	25	25	25	70%
69	Z.AI GLM 4.6	$0.023	1.8m	22%	100	100	100	100	100	100	42	25	25	0	69%
60	Cohere Command R+ (Aug. 2024)	$0.013	2.6s	15%	100	100	100	100	100	100	17	13	13	10	65%
102	Qwen3.7 Max	$0.058	1.7m	16%	100	100	100	100	100	25	25	25	25	25	63%
92	Z.AI GLM 4.7	$0.029	2.4m	16%	100	100	100	100	100	25	25	25	25	25	63%
62	Grok 4 Fast	$0.0020	24.0s	16%	100	100	100	100	100	25	25	25	25	25	63%
67	GPT-5.2	$0.019	20.2s	14%	100	100	100	100	100	25	25	25	25	17	62%
61	GPT-4o, Aug. 6th (temp=1)	$0.0075	1.3s	15%	100	100	100	100	100	42	38	17	10	10	62%
64	Skyfall 36B V2	$0.0017	5.2s	13%	100	100	100	100	100	50	31	10	7	5	60%
65	Gemma 4 26B	$0.0008	27.0s	14%	100	100	100	100	42	42	38	32	32	13	60%
66	Mistral Medium 3.1	$0.0027	4.4s	11%	100	100	100	100	42	33	33	33	30	6	58%
63	Hermes 3 70B	$0.0016	12.9s	19%	100	100	100	50	50	50	38	33	32	8	56%
146	Gemma 4 31B (Reasoning)	$0.0030	5.5m	7%	100	100	100	100	25	25	25	25	25	25	55%
88	Aion 2.0	$0.011	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
68	Cydonia 24B V4.1	$0.0012	7.0s	7%	100	100	100	100	35	31	29	25	8	7	54%
73	Arcee AI: Trinity Large (Preview)	$0.0000	20.5s	5%	100	100	100	100	29	28	28	10	8	4	51%
81	ByteDance Seed 1.6	$0.0056	46.9s	8%	100	100	100	25	25	25	25	25	25	25	48%
55	Claude Haiku 4.5	$0.0076	5.4s	44%	50	50	50	50	50	50	50	42	38	35	46%
147	Grok 4	$0.108	2.5m	7%	100	100	100	25	25	25	25	17	17	17	45%
149	Qwen3.6 Max Preview	$0.079	4.2m	7%	100	100	100	25	25	25	25	17	17	13	45%
96	Gemini 3 Flash (Preview, Reasoning)	$0.025	46.9s	7%	100	100	100	25	25	25	25	17	17	13	45%
151	Qwen 3.5 122B	$0.166	7.7m	7%	100	100	100	25	25	25	25	17	13	13	44%
130	Gemini 2.5 Pro	$0.060	45.6s	10%	100	100	25	25	25	25	25	25	25	25	40%
83	Gemini 2.5 Flash (Reasoning)	$0.0094	15.1s	10%	100	100	25	25	25	25	25	25	25	25	40%
70	Claude Opus 4.5	$0.033	5.9s	36%	42	42	42	42	38	38	38	38	38	35	39%
82	Claude Opus 4.6	$0.043	14.0s	30%	50	50	50	50	42	42	42	38	13	10	38%
85	Ministral 8B	$0.0007	9.6s	4%	100	100	100	27	26	26	2	1	1	1	38%
78	Qwen 2.5 72B	$0.0007	6.7s	10%	100	100	38	35	31	31	10	10	8	7	37%
80	Claude Opus 4.7	$0.046	4.6s	32%	42	42	42	42	42	38	38	38	38	10	37%
86	Llama 3.1 Nemotron 70B	$0.0068	17.7s	9%	100	100	42	35	31	27	10	7	6	6	36%
75	GPT-5.4 Nano	$0.0004	1.7s	14%	100	50	50	35	29	25	25	25	17	6	36%
89	Claude Sonnet 4	$0.020	5.8s	13%	100	50	42	38	38	17	13	13	13	10	33%
84	DeepSeek V4 Pro	$0.0064	56.4s	25%	50	50	50	50	42	35	28	10	8	4	33%
148	MoonshotAI: Kimi K2.6	$0.052	4.5m	14%	100	25	25	25	25	25	25	25	25	25	33%
71	GPT-4.1 Mini	$0.0023	7.3s	24%	50	50	33	33	33	32	31	29	28	2	32%
135	MoonshotAI: Kimi K2.5	$0.019	2.4m	14%	100	25	25	25	25	25	25	25	25	17	32%
97	Arcee AI: Trinity Mini	$0.0014	45.3s	6%	100	100	25	25	25	17	13	10	2	0	32%
94	Hermes 3 405B	$0.0052	6.8s	4%	100	100	17	13	13	13	13	10	10	10	30%
141	Claude Opus 4	$0.098	8.1s	27%	50	42	42	38	38	38	17	13	13	7	29%
95	Llama 3.1 70B	$0.0024	9.1s	3%	100	100	25	17	13	10	8	6	6	6	29%
131	Qwen 3.5 Plus (2026-02-15)	$0.024	2.2m	20%	50	50	50	32	31	30	28	8	6	6	29%
87	Mistral Small 4	$0.0013	8.8s	13%	100	35	28	28	27	27	27	2	2	2	28%
74	Ministral 3B	$0.0003	7.5s	26%	32	31	29	28	27	27	26	26	26	26	28%
101	GPT-5.5	$0.015	3.0s	8%	100	25	25	25	17	17	17	13	13	13	26%
105	Grok 4.20 (Beta)	$0.0060	3.1s	1%	100	100	30	5	5	5	4	4	4	4	26%
120	Ministral 3 8B	$0.0020	53.4s	0%	100	100	26	26	2	2	1	1	1	0	26%
150	Qwen 3.5 397B A17B	$0.037	7.1m	25%	25	25	25	25	25	25	25	25	25	25	25%
76	Gemma 4 31B	$0.0008	5.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
93	WizardLM 2 8x22b	$0.0037	57.8s	21%	50	31	29	29	29	28	28	13	5	4	25%
104	Gemini 2.5 Flash Lite	$0.0007	2.1s	1%	100	100	8	7	7	5	5	4	3	1	24%
79	GPT-5.4 Mini	$0.0015	930ms	23%	25	25	25	25	25	25	25	25	17	17	23%
113	GPT-4o, May 13th (temp=1)	$0.029	3.2s	14%	38	33	33	33	25	13	8	8	8	7	21%
90	Llama 3.1 8B	$0.0002	16.1s	17%	42	31	30	30	28	17	10	8	6	5	21%
112	Grok 4.3	$0.0065	6.5s	3%	100	38	29	8	6	6	6	5	4	4	21%
110	Claude 3.7 Sonnet	$0.019	4.1s	11%	50	42	17	17	17	13	13	13	13	13	20%
91	Gemini 3.1 Flash Lite	$0.0017	5.0s	14%	38	33	17	17	17	17	17	17	17	10	20%
99	Claude 3 Haiku	$0.0018	3.8s	9%	38	38	33	33	13	13	10	8	6	5	20%
100	DeepSeek V4 Flash	$0.0004	13.6s	11%	42	42	31	30	28	5	5	4	2	2	19%
118	GPT-5.4	$0.0089	8.3s	4%	100	17	17	10	8	8	5	4	4	4	18%
122	ByteDance Seed 2.0 Lite	$0.0079	1.3m	16%	25	17	17	17	17	17	17	17	17	17	18%
98	DeepSeek V3 (2024-12-26)	$0.0019	6.1s	12%	33	25	25	25	17	13	13	8	8	8	18%
116	Mistral Small 3.2 24B	$0.0010	13.9s	2%	100	32	6	6	6	6	5	4	4	3	17%
103	Gemini 3.1 Flash Lite (Reasoning)	$0.0016	2.1s	11%	38	17	17	17	13	13	13	13	13	10	16%
108	Mistral Large 3	$0.0043	19.7s	12%	29	29	29	29	28	4	3	3	3	3	16%
125	DeepSeek V3.1	$0.0016	38.9s	6%	50	29	25	10	10	7	7	5	5	1	15%
123	Ministral 3 3B	$0.0017	49.7s	10%	28	28	26	26	25	2	2	1	0	0	14%
138	Claude 3.5 Sonnet	$0.041	8.2s	12%	17	17	17	13	13	13	13	13	13	10	14%
107	Gemini 3.1 Flash Lite (Preview)	$0.0019	1.9s	10%	33	13	13	13	13	10	10	10	10	8	13%
109	DeepSeek V3 (2025-03-24)	$0.0012	12.4s	11%	25	17	13	13	13	13	13	10	8	8	13%
121	DeepSeek V3.2	$0.0015	31.6s	8%	28	26	13	13	10	10	7	7	6	5	12%
115	DeepSeek-V2 Chat	$0.0021	8.9s	8%	35	13	13	13	10	10	8	8	6	6	12%
111	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0063	2.5s	12%	17	17	13	13	13	13	10	10	10	6	12%
140	Mistral Large 2	$0.017	17.6s	3%	29	28	27	4	4	4	4	4	4	3	11%
139	Claude Sonnet 4.5	$0.025	8.9s	7%	33	8	8	8	8	7	7	7	7	6	10%
114	Gemini 3 Flash (Preview)	$0.0023	3.7s	10%	13	13	13	10	10	10	8	8	8	7	10%
133	Ministral 3 14B	$0.0015	20.5s	2%	29	28	25	3	3	2	2	1	1	1	9%
129	Mistral Small Creative	$0.0011	10.7s	2%	27	27	27	2	2	2	2	2	2	1	9%
117	Gemma 3 12B	$0.0002	8.5s	8%	17	13	13	10	10	7	6	6	6	6	9%
134	Qwen3 235B A22B Instruct 2507	$0.0012	39.6s	4%	28	28	6	5	5	5	5	4	2	2	9%
127	Rocinante 12B	$0.0014	20.1s	6%	30	13	10	7	7	6	4	4	4	3	9%
143	Mistral Large	$0.017	16.5s	3%	29	28	5	4	4	4	4	4	3	3	9%
142	Writer: Palmyra X5	$0.013	22.2s	4%	28	27	5	5	5	4	4	3	2	2	8%
119	Gemini 2.5 Flash	$0.0021	2.0s	8%	10	10	8	8	8	8	8	8	7	5	8%
128	GPT-4o, Aug. 6th (temp=0)	$0.011	2.8s	7%	10	10	10	10	8	7	7	6	5	5	8%
126	Gemma 3 27B	$0.0006	11.4s	7%	10	8	8	8	7	7	7	7	6	5	7%
145	GPT-4o, May 13th (temp=0)	$0.033	4.9s	6%	8	7	7	7	6	6	6	6	5	5	6%
132	Mistral NeMO	$0.0009	7.8s	4%	7	7	6	5	4	4	4	4	3	2	5%
137	Grok 4.20	$0.0083	12.3s	4%	6	6	5	5	5	5	5	4	2	2	4%
136	Gemma 3 4B	$0.0003	9.9s	3%	4	3	3	3	3	3	3	3	2	2	3%
54.69%

Median	Evaluator	Top 3	Flop 3
50.0%	Correct "no violations" response	100LFM2 24B 100Claude Sonnet 4.6 (Reasoning) 100Z.AI GLM 5	0Claude 3.5 Sonnet 0GPT-4o, Aug. 6th (temp=0) 0Gemma 4 31B
60.0%	No hallucinated violations	100Inception Mercury 100GPT-5.4 Mini (Reasoning, Low) 100Z.AI GLM 5 Turbo	3Ministral 3 3B 4Mistral Small Creative 4Ministral 3 14B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)