Long text (~1594 words), big codex (51 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Qwen3.6 Max Preview	100%
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Grok 4.3 (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
Claude Opus 4.6	100%
MoonshotAI: Kimi K2.6	100%
GPT-5	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 26B (Reasoning)	100%
Grok 4.20 (Beta, Reasoning)	100%
Grok 4.20 (Reasoning)	100%
Z.AI GLM 5	100%
o4 Mini High	100%
Qwen 3.6 27B	100%
Grok 4.1 Fast	100%
Aion 2.0	100%

	Score	Cost	Time
GPT-5.4 Nano	78%	$0.0008	1.0s
Ministral 8B	100%	$0.0014	525ms
Ministral 3 8B	100%	$0.0022	678ms
Ministral 3 14B	93%	$0.0029	2.7s
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0014	6.3s
Arcee AI: Trinity Mini	90%	$0.0018	45.5s
GPT-4.1	100%	$0.0097	972ms
Grok 4.1 Fast	100%	$0.0031	13.4s
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0073	5.9s
Inception Mercury 2	85%	$0.0056	7.7s
ByteDance Seed 1.6 Flash	84%	$0.0016	13.8s
Grok 4 Fast	93%	$0.0032	25.0s
Inception Mercury	100%	$0.0007	20.5s
GPT-5.4 Nano (Reasoning)	100%	$0.0035	40.9s
Stealth: Healer Alpha	100%	$0.0000	27.7s
Gemini 2.5 Flash Lite (Reasoning)	93%	$0.0038	27.3s
ByteDance Seed 2.0 Lite	100%	$0.0056	25.4s
Gemini 2.5 Flash (Reasoning)	85%	$0.015	19.0s
GPT-5.4 (Reasoning, Low)	92%	$0.019	14.5s
GPT-5.2	93%	$0.021	20.9s

	Score	Cost	Speed	Stability
Ministral 8B	100%	$0.0014	525ms	100%
Ministral 3 8B	100%	$0.0022	678ms	100%
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0014	6.3s	100%
GPT-4.1	100%	$0.0097	972ms	100%
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0073	5.9s	100%
Grok 4.1 Fast	100%	$0.0031	13.4s	100%
Inception Mercury	100%	$0.0007	20.5s	100%
Stealth: Healer Alpha	100%	$0.0000	27.7s	100%
ByteDance Seed 2.0 Lite	100%	$0.0056	25.4s	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0035	40.9s	100%
Z.AI GLM 5 Turbo	100%	$0.016	33.5s	100%
Grok 4.20 (Beta, Reasoning)	100%	$0.033	18.8s	100%
GPT-5 Mini	100%	$0.0090	53.0s	100%
Stealth: Hunter Alpha	100%	$0.0000	1.1m	100%
DeepSeek V4 Flash (Reasoning)	100%	$0.0021	1.2m	100%
o4 Mini	100%	$0.026	43.6s	100%
Xiaomi MIMO v2.5	100%	$0.017	1.1m	100%
Grok 4.20 (Reasoning)	100%	$0.027	57.4s	100%
Grok 4.3 (Reasoning)	100%	$0.027	1.1m	100%
GPT-5.1	100%	$0.051	44.5s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
56	Claude Opus 4.6 (Reasoning)	$0.322	2.7m	100%	100	100	100	100	100	100	100	100	100	100	100%
48	Qwen3.6 Max Preview	$0.117	5.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
32	Z.AI GLM 5.1	$0.050	3.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Z.AI GLM 5 Turbo	$0.016	33.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Grok 4.3 (Reasoning)	$0.027	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
36	Claude Opus 4.7 (Reasoning)	$0.215	38.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5 Mini	$0.0090	53.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	GPT-5.1	$0.051	44.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Claude Opus 4.6	$0.104	19.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
51	MoonshotAI: Kimi K2.6	$0.067	7.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
28	GPT-5	$0.066	2.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Qwen 3.5 Plus (2026-04-20)	$0.036	3.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
31	Gemma 4 26B (Reasoning)	$0.0051	4.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Grok 4.20 (Beta, Reasoning)	$0.033	18.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Grok 4.20 (Reasoning)	$0.027	57.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Z.AI GLM 5	$0.031	2.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
23	o4 Mini High	$0.048	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Qwen 3.6 27B	$0.045	3.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Grok 4.1 Fast	$0.0031	13.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Aion 2.0	$0.017	1.7m	100%	100	100	100	100	100	100	100	100	100	100	100%
15	DeepSeek V4 Flash (Reasoning)	$0.0021	1.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-4.1	$0.0097	972ms	100%	100	100	100	100	100	100	100	100	100	100	100%
16	o4 Mini	$0.026	43.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Xiaomi MIMO v2.5 Pro	$0.028	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Stealth: Hunter Alpha	$0.0000	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
27	ByteDance Seed 2.0 Mini	$0.0043	3.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Stealth: Healer Alpha	$0.0000	27.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-5.4 Mini (Reasoning, Low)	$0.0073	5.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Xiaomi MIMO v2.5	$0.017	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
9	ByteDance Seed 2.0 Lite	$0.0056	25.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Nemotron 3 Super	$0.0000	2.6m	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5.4 Nano (Reasoning)	$0.0035	40.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Inception Mercury	$0.0007	20.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	GPT-5.4 Nano (Reasoning, Low)	$0.0014	6.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0022	678ms	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0014	525ms	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Ministral 3 14B	$0.0029	2.7s	56%	100	100	100	100	100	100	100	100	100	28	93%
146	Claude Sonnet 4.6 (Reasoning)	$0.404	6.7m	55%	100	100	100	100	100	100	100	100	100	25	93%
37	GPT-5.4 Mini (Reasoning)	$0.018	22.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
38	GPT-5.2	$0.021	20.9s	55%	100	100	100	100	100	100	100	100	100	25	93%
69	DeepSeek V4 Pro (Reasoning)	$0.048	7.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
41	MiniMax M2.7	$0.014	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
34	Grok 4 Fast	$0.0032	25.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
35	Gemini 2.5 Flash Lite (Reasoning)	$0.0038	27.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
40	GPT-5 Nano	$0.0046	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
59	Qwen3.7 Max	$0.116	2.5m	50%	100	100	100	100	100	100	100	100	100	17	92%
54	Gemini 3.5 Flash (Reasoning)	$0.145	1.1m	50%	100	100	100	100	100	100	100	100	100	17	92%
39	GPT-5.4 (Reasoning, Low)	$0.019	14.5s	50%	100	100	100	100	100	100	100	100	100	17	92%
53	MoonshotAI: Kimi K2.5	$0.024	3.2m	50%	100	100	100	100	100	100	100	100	100	17	92%
44	Gemini 3 Flash (Preview, Reasoning)	$0.031	52.9s	46%	100	100	100	100	100	100	100	100	100	10	91%
43	Arcee AI: Trinity Mini	$0.0018	45.5s	40%	100	100	100	100	100	100	100	100	100	0	90%
60	Gemma 4 31B (Reasoning)	$0.0047	3.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
47	ByteDance Seed 1.6	$0.0074	45.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
50	Qwen 3.6 Flash	$0.026	1.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
45	Gemini 2.5 Flash (Reasoning)	$0.015	19.0s	40%	100	100	100	100	100	100	100	100	25	25	85%
42	Inception Mercury 2	$0.0056	7.7s	40%	100	100	100	100	100	100	100	100	25	25	85%
71	Nemotron 3 Nano	$0.0055	6.2m	40%	100	100	100	100	100	100	100	100	25	25	85%
147	Qwen 3.5 122B	$0.211	8.8m	37%	100	100	100	100	100	100	100	100	25	17	84%
46	ByteDance Seed 1.6 Flash	$0.0016	13.8s	37%	100	100	100	100	100	100	100	100	25	17	84%
61	Qwen 3.5 Flash	$0.010	2.8m	35%	100	100	100	100	100	100	100	100	25	13	84%
52	MiniMax M2.5	$0.0064	50.3s	30%	100	100	100	100	100	100	100	100	25	2	83%
49	GPT-5.4 Nano	$0.0008	1.0s	31%	100	100	100	100	100	100	100	25	25	25	78%
63	Qwen 3.6 35B	$0.033	2.5m	29%	100	100	100	100	100	100	100	25	25	17	77%
58	GPT-5.5 (Reasoning, Low)	$0.045	17.7s	26%	100	100	100	100	100	100	100	25	17	17	76%
70	Grok 4	$0.099	1.7m	24%	100	100	100	100	100	100	100	25	17	10	75%
57	Z.AI GLM 4.5	$0.0061	34.6s	27%	100	100	100	100	100	100	31	29	25	17	70%
74	Z.AI GLM 4.7	$0.037	3.0m	23%	100	100	100	100	100	100	25	25	25	13	69%
120	Qwen 3.5 35B	$0.094	5.0m	22%	100	100	100	100	100	100	25	25	17	17	68%
66	GPT-5.4 (Reasoning)	$0.069	59.0s	21%	100	100	100	100	100	100	25	25	17	13	68%
143	Gemini 3.1 Pro (Preview)	$0.259	3.3m	17%	100	100	100	100	100	100	17	17	17	13	66%
62	Qwen 2.5 72B	$0.0018	5.5s	17%	100	100	100	100	100	38	33	27	25	6	63%
67	Z.AI GLM 4.6	$0.013	1.3m	15%	100	100	100	100	100	25	25	25	25	17	62%
64	Mistral Small 4 (Reasoning)	$0.0048	48.0s	13%	100	100	100	100	100	25	25	25	17	17	61%
65	Qwen 3 32B	$0.0027	1.1m	14%	100	100	100	100	100	29	25	25	17	13	61%
78	GPT-OSS 120B	$0.0027	1.3m	7%	100	100	100	100	25	25	25	25	25	25	55%
79	Claude Sonnet 4.6	$0.096	57.8s	26%	100	100	50	50	50	50	50	35	32	31	55%
77	Cohere Command R+ (Aug. 2024)	$0.040	12.8s	5%	100	100	100	100	100	17	13	13	3	1	55%
150	Qwen 3.5 397B A17B	$0.041	7.4m	6%	100	100	100	100	25	25	25	25	17	13	53%
141	Gemini 3 Pro (Preview)	$0.197	2.2m	5%	100	100	100	100	25	25	25	17	13	13	52%
126	GPT-5.5 (Reasoning)	$0.143	1.1m	3%	100	100	100	100	17	17	17	17	17	17	50%
55	Claude Haiku 4.5	$0.018	5.5s	48%	50	50	50	50	50	50	50	50	50	42	49%
72	GPT-4.1 Nano	$0.0010	8.3s	4%	100	100	100	100	26	26	25	3	2	1	48%
76	Claude Sonnet 4.5	$0.051	5.8s	16%	100	100	42	42	38	38	38	33	17	17	46%
122	Arcee AI: Trinity Large (Preview)	$0.0000	3.8m	7%	100	100	100	26	26	26	26	26	26	0	46%
89	Z.AI GLM 4.5 Air	$0.0041	48.3s	2%	100	100	100	100	17	17	10	8	3	1	46%
145	Qwen 3.5 9B	$0.0061	6.8m	10%	100	100	50	50	42	17	17	17	13	3	41%
81	GPT-5.4 Mini	$0.0045	1.2s	9%	100	100	25	25	25	25	25	25	17	13	38%
84	GPT-4o, Aug. 6th (temp=1)	$0.019	2.0s	9%	100	100	35	35	25	25	17	17	17	8	38%
73	DeepSeek V4 Pro	$0.015	1.3m	30%	50	50	50	50	50	38	38	26	25	1	38%
93	Gemini 2.5 Flash Lite	$0.0038	18.0s	2%	100	100	100	26	25	3	2	2	0	0	36%
68	Mistral Medium 3.1	$0.0070	6.2s	29%	42	38	33	33	31	31	30	29	29	27	32%
112	Mistral Small 3.2 24B	$0.0038	1.0m	0%	100	100	100	2	2	2	1	1	1	0	31%
108	WizardLM 2 8x22b	$0.0094	58.3s	4%	100	100	38	26	17	10	7	4	3	3	31%
87	Qwen3 235B A22B Instruct 2507	$0.0021	31.8s	14%	100	38	35	29	28	28	26	10	8	5	31%
88	Cydonia 24B V4.1	$0.0033	25.4s	14%	100	33	33	31	31	30	26	5	2	2	29%
90	Skyfall 36B V2	$0.0049	10.3s	10%	100	42	35	31	28	17	17	13	5	4	29%
133	Z.AI GLM 4.7 Flash	$0.0066	3.3m	11%	100	25	25	25	25	17	17	17	17	13	28%
103	Grok 4.3	$0.019	7.5s	2%	100	100	26	17	10	8	6	5	2	1	28%
105	Claude Opus 4.7	$0.120	7.2s	26%	38	35	35	35	33	33	31	13	13	8	27%
75	GPT-4.1 Mini	$0.0039	19.8s	27%	30	28	28	28	27	27	27	27	27	25	27%
83	DeepSeek V4 Flash	$0.0007	16.8s	21%	50	38	29	29	27	26	26	17	10	10	26%
95	GPT-4o, May 13th (temp=1)	$0.074	2.9s	20%	50	42	38	33	31	25	17	10	8	7	26%
85	Mistral Small Creative	$0.0025	19.9s	20%	50	26	26	26	26	26	26	26	25	1	26%
97	GPT-5.5	$0.028	4.0s	8%	100	25	25	17	17	17	17	13	13	10	25%
80	Gemma 4 31B	$0.0021	16.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
86	Rocinante 12B	$0.0032	17.1s	18%	50	50	32	31	31	25	6	6	5	4	24%
82	GPT-5.4	$0.0058	2.0s	23%	25	25	25	25	25	25	25	25	25	13	24%
149	Claude Opus 4	$0.254	10.0s	9%	50	42	38	35	13	13	13	10	10	7	23%
115	Claude Opus 4.5	$0.087	8.3s	17%	35	33	33	25	25	17	17	17	10	8	22%
91	Writer: Palmyra X5	$0.015	17.0s	21%	42	28	27	27	26	26	17	13	6	5	22%
113	Grok 4.20 (Beta)	$0.015	7.5s	1%	100	42	33	27	5	2	2	1	1	1	21%
96	GPT-4o, Aug. 6th (temp=0)	$0.027	3.9s	13%	42	42	42	35	31	7	6	5	3	2	21%
119	Claude 3.7 Sonnet	$0.052	6.8s	5%	100	35	13	10	10	10	10	8	7	7	21%
92	Gemini 2.5 Flash	$0.0036	1.1s	15%	25	25	25	25	17	17	17	17	17	17	20%
94	Llama 3.1 8B	$0.0005	38.9s	20%	33	33	33	29	27	27	6	6	2	0	20%
151	Qwen 3.5 27B	$0.064	4.8m	14%	35	25	25	25	17	17	17	13	13	10	20%
110	Gemma 4 26B	$0.0017	1.1m	14%	38	33	17	17	17	17	17	17	13	8	19%
98	Ministral 3B	$0.0012	1.1m	20%	27	27	27	26	26	26	26	2	1	0	19%
116	Hermes 3 70B	$0.0044	24.6s	3%	100	30	10	8	8	7	6	5	5	4	18%
127	DeepSeek V3.2	$0.0049	2.1m	12%	50	27	26	25	25	10	7	3	1	0	17%
99	Claude 3 Haiku	$0.0043	4.6s	9%	42	38	33	13	13	13	6	6	6	6	17%
114	DeepSeek-V2 Chat	$0.0056	21.8s	6%	50	30	25	25	10	7	7	6	4	1	16%
102	Mistral Small 4	$0.0024	9.7s	10%	35	31	27	27	17	10	7	6	2	1	16%
107	GPT-4o Mini (temp=1)	$0.0017	8.0s	8%	35	32	31	10	10	10	8	8	8	4	16%
100	Gemini 3.5 Flash (Reasoning, Minimal)	$0.024	2.9s	15%	25	17	17	17	17	17	13	13	13	10	16%
117	Ministral 3 3B	$0.0042	1.8m	19%	27	26	26	25	25	25	1	0	0	0	16%
106	Gemini 3 Flash (Preview)	$0.0086	4.7s	10%	35	31	13	13	13	13	13	10	10	5	15%
118	Claude Sonnet 4	$0.051	6.4s	10%	42	13	13	13	13	13	13	13	13	10	15%
128	Gemini 2.5 Pro	$0.064	40.8s	13%	25	17	17	17	17	13	13	13	10	10	15%
109	Hermes 3 405B	$0.014	7.7s	10%	38	17	17	17	13	13	10	10	7	6	15%
134	DeepSeek V3.1	$0.0048	2.2m	11%	27	26	26	25	25	4	3	2	2	1	14%
101	Gemini 3.1 Flash Lite (Reasoning)	$0.0030	2.1s	12%	17	17	17	13	13	13	13	13	13	10	14%
104	Gemini 3.1 Flash Lite	$0.0033	6.1s	12%	17	17	17	17	13	13	13	10	10	10	13%
125	Llama 3.1 Nemotron 70B	$0.018	29.9s	4%	33	31	28	6	6	6	6	6	4	3	13%
111	Gemini 3.1 Flash Lite (Preview)	$0.0035	2.6s	9%	29	13	10	10	10	10	10	10	8	7	12%
132	Mistral Large	$0.037	28.1s	4%	27	27	27	6	5	5	5	4	4	2	11%
121	Llama 3.1 70B	$0.0076	18.2s	6%	28	17	10	10	8	6	5	5	5	3	10%
123	DeepSeek V3 (2024-12-26)	$0.0058	16.0s	5%	25	17	10	7	6	6	5	5	4	3	9%
140	Claude 3.5 Sonnet	$0.105	9.6s	8%	10	8	8	8	8	8	7	7	7	7	8%
139	LFM2 24B	$0.0009	1.4m	1%	27	25	13	6	2	1	1	0	0	0	7%
142	Qwen 3.5 Plus (2026-02-15)	$0.025	2.2m	7%	13	13	10	10	8	7	5	3	2	1	7%
130	Mistral Large 3	$0.0091	28.3s	4%	27	8	6	6	5	4	4	3	3	2	7%
137	Mistral Large 2	$0.036	26.7s	3%	27	8	6	5	4	4	4	4	3	2	7%
129	GPT-4o Mini (temp=0)	$0.0016	45.6s	6%	8	8	7	7	7	6	6	5	4	4	6%
124	Gemma 3 12B	$0.0011	16.5s	5%	8	8	8	6	6	5	4	4	4	1	5%
135	Grok 4.20	$0.017	22.6s	2%	29	4	3	3	3	2	2	2	2	1	5%
136	DeepSeek V3 (2025-03-24)	$0.0038	54.7s	5%	8	8	6	5	5	5	5	5	1	1	5%
131	Gemma 3 27B	$0.0018	21.0s	3%	7	4	4	3	3	3	3	3	2	2	3%
148	Mistral NeMO	$0.0054	3.1m	2%	10	7	5	4	3	1	1	1	0	0	3%
144	GPT-4o, May 13th (temp=0)	$0.094	10.4s	3%	5	5	4	3	3	3	3	3	3	2	3%
138	Gemma 3 4B	$0.0008	30.9s	1%	2	1	1	1	1	1	1	1	1	1	1%
56.39%

Median	Evaluator	Top 3	Flop 3
50.0%	Correct "no violations" response	100Claude Opus 4.7 (Reasoning) 100Qwen 3.6 27B 100Grok 4.20 (Reasoning)	0DeepSeek V3 (2025-03-24) 0Gemini 2.5 Pro 0Gemma 3 12B
63.3%	No hallucinated violations	100Stealth: Hunter Alpha 100MoonshotAI: Kimi K2.6 100GPT-5.4 Nano (Reasoning, Low)	1Ministral 3 3B 2Gemma 3 4B 2Ministral 3B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), big codex (51 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)