Short text (~524 words), small codex (11 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Qwen3.7 Max	100%
Claude Opus 4.6 (Reasoning)	100%
Qwen3.6 Max Preview	100%
Gemini 3.1 Pro (Preview)	100%
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Gemini 3.5 Flash (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5 Mini	100%
Claude Opus 4.6	100%
Gemma 4 31B (Reasoning)	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 26B (Reasoning)	100%
Grok 4.20 (Beta, Reasoning)	100%
Grok 4.20 (Reasoning)	100%
ByteDance Seed 1.6	100%
GPT-5.4 Mini (Reasoning)	100%
Claude Opus 4.7	100%
Grok 4.1 Fast	100%
MiniMax M2.7	100%

	Score	Cost	Time
LFM2 24B	100%	$0.0000	1.3s
Gemma 4 26B	100%	$0.0002	3.8s
GPT-5.4 Nano	80%	$0.0003	1.0s
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0005	2.8s
Inception Mercury	100%	$0.0001	3.6s
Arcee AI: Trinity Mini	81%	$0.0012	36.6s
Gemma 4 31B	100%	$0.0002	12.4s
GPT-5.4 Nano (Reasoning)	100%	$0.0008	3.9s
ByteDance Seed 1.6 Flash	100%	$0.0004	5.3s
Inception Mercury 2	100%	$0.0013	2.0s
GPT-5.5	100%	$0.0041	1.3s
Gemini 3.5 Flash (Reasoning, Minimal)	100%	$0.0024	850ms
Grok 4 Fast	77%	$0.0008	8.7s
Gemini 2.5 Flash Lite (Reasoning)	93%	$0.0012	8.6s
Grok 4.1 Fast	100%	$0.0007	7.5s
GPT-5.4 Mini (Reasoning, Low)	93%	$0.0019	3.6s
Llama 3.1 Nemotron 70B	67%	$0.0021	9.0s
Mistral Small 4 (Reasoning)	100%	$0.0009	9.3s
Qwen 3 32B	78%	$0.0004	12.3s
Cohere Command R+ (Aug. 2024)	65%	$0.0046	2.5s

	Score	Cost	Speed	Stability
LFM2 24B	100%	$0.0000	1.3s	100%
Inception Mercury	100%	$0.0001	3.6s	100%
Gemma 4 26B	100%	$0.0002	3.8s	100%
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0005	2.8s	100%
Inception Mercury 2	100%	$0.0013	2.0s	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0008	3.9s	100%
ByteDance Seed 1.6 Flash	100%	$0.0004	5.3s	100%
Gemini 3.5 Flash (Reasoning, Minimal)	100%	$0.0024	850ms	100%
Grok 4.1 Fast	100%	$0.0007	7.5s	100%
Mistral Small 4 (Reasoning)	100%	$0.0009	9.3s	100%
Gemma 4 31B	100%	$0.0002	12.4s	100%
GPT-5.5	100%	$0.0041	1.3s	100%
Z.AI GLM 5 Turbo	100%	$0.0036	9.8s	100%
GPT-5.4 Mini (Reasoning)	100%	$0.0039	9.0s	100%
ByteDance Seed 1.6	100%	$0.0017	16.9s	100%
Z.AI GLM 4.5 Air	100%	$0.0012	19.2s	100%
Gemini 2.5 Flash (Reasoning)	100%	$0.0051	8.5s	100%
MiniMax M2.7	100%	$0.0014	22.4s	100%
Claude Opus 4.7	100%	$0.011	940ms	100%
Claude Opus 4.7 (Reasoning)	100%	$0.011	2.0s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
33	Qwen3.7 Max	$0.026	47.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
32	Claude Opus 4.6 (Reasoning)	$0.033	20.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
41	Qwen3.6 Max Preview	$0.039	2.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
35	Gemini 3.1 Pro (Preview)	$0.039	27.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	Z.AI GLM 5.1	$0.012	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
13	Z.AI GLM 5 Turbo	$0.0036	9.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
34	Gemini 3.5 Flash (Reasoning)	$0.039	17.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Claude Opus 4.7 (Reasoning)	$0.011	2.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	GPT-5 Mini	$0.0044	29.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Claude Opus 4.6	$0.022	9.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Gemma 4 31B (Reasoning)	$0.0011	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Qwen 3.5 Plus (2026-04-20)	$0.010	1.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Gemma 4 26B (Reasoning)	$0.0012	52.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Grok 4.20 (Beta, Reasoning)	$0.016	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Grok 4.20 (Reasoning)	$0.0089	28.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	ByteDance Seed 1.6	$0.0017	16.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	GPT-5.4 Mini (Reasoning)	$0.0039	9.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Claude Opus 4.7	$0.011	940ms	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Grok 4.1 Fast	$0.0007	7.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	MiniMax M2.7	$0.0014	22.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	GPT-5.5	$0.0041	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	o4 Mini	$0.0097	18.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Gemma 4 31B	$0.0002	12.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Gemini 2.5 Flash (Reasoning)	$0.0051	8.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0024	850ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Gemma 4 26B	$0.0002	3.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Z.AI GLM 4.7 Flash	$0.0012	47.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Nemotron 3 Super	$0.0000	47.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Inception Mercury 2	$0.0013	2.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Z.AI GLM 4.5 Air	$0.0012	19.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Mistral Small 4 (Reasoning)	$0.0009	9.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.4 Nano (Reasoning)	$0.0008	3.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Inception Mercury	$0.0001	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning, Low)	$0.0005	2.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	ByteDance Seed 1.6 Flash	$0.0004	5.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0000	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Z.AI GLM 4.6	$0.0029	18.0s	56%	100	100	100	100	100	100	100	100	100	27	93%
42	GPT-5.1	$0.018	22.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
47	Z.AI GLM 5	$0.013	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
40	Qwen 3.6 Flash	$0.011	34.9s	55%	100	100	100	100	100	100	100	100	100	25	93%
43	o4 Mini High	$0.019	35.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
38	MiniMax M2.5	$0.0015	12.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
46	ByteDance Seed 2.0 Mini	$0.0018	1.8m	55%	100	100	100	100	100	100	100	100	100	25	93%
36	GPT-5.4 Mini (Reasoning, Low)	$0.0019	3.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Gemini 2.5 Flash Lite (Reasoning)	$0.0012	8.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
45	Nemotron 3 Nano	$0.0016	1.7m	55%	100	100	100	100	100	100	100	100	100	25	93%
70	Claude Sonnet 4.6 (Reasoning)	$0.060	54.9s	40%	100	100	100	100	100	100	100	100	25	25	85%
68	Qwen 3.5 27B	$0.028	2.3m	40%	100	100	100	100	100	100	100	100	25	25	85%
48	GPT-OSS 120B	$0.0006	49.8s	40%	100	100	100	100	100	100	100	100	25	25	85%
53	GPT-5 Nano	$0.0036	1.2m	40%	100	100	100	100	100	100	100	100	25	25	85%
51	Grok 4.3 (Reasoning)	$0.0073	41.2s	37%	100	100	100	100	100	100	100	100	25	17	84%
56	Arcee AI: Trinity Mini	$0.0012	36.6s	22%	100	100	100	100	100	100	100	100	6	0	81%
44	GPT-5.4 Nano	$0.0003	1.0s	38%	100	100	100	100	100	100	100	50	25	25	80%
49	Z.AI GLM 4.5	$0.0018	13.0s	32%	100	100	100	100	100	100	100	29	25	25	78%
58	Aion 2.0	$0.0054	59.6s	31%	100	100	100	100	100	100	100	25	25	25	78%
57	Qwen 3.6 35B	$0.0080	47.0s	31%	100	100	100	100	100	100	100	25	25	25	78%
146	Gemini 3 Pro (Preview)	$0.114	1.4m	31%	100	100	100	100	100	100	100	25	25	25	78%
50	Qwen 3 32B	$0.0004	12.3s	31%	100	100	100	100	100	100	100	25	25	25	78%
54	DeepSeek V4 Flash (Reasoning)	$0.0004	17.6s	29%	100	100	100	100	100	100	100	25	25	17	77%
52	Grok 4 Fast	$0.0008	8.7s	29%	100	100	100	100	100	100	100	25	25	17	77%
55	GPT-5.2	$0.0091	10.8s	27%	100	100	100	100	100	100	100	25	25	13	76%
80	Qwen 3.6 27B	$0.025	1.8m	27%	100	100	100	100	100	100	100	25	25	13	76%
150	Qwen 3.5 122B	$0.086	3.8m	27%	100	100	100	100	100	100	25	25	25	25	70%
63	Xiaomi MIMO v2.5 Pro	$0.011	53.1s	27%	100	100	100	100	100	100	25	25	25	25	70%
61	Llama 3.1 Nemotron 70B	$0.0021	9.0s	18%	100	100	100	100	100	100	31	29	7	4	67%
120	Qwen 3.5 9B	$0.0025	3.4m	21%	100	100	100	100	50	50	50	50	50	17	67%
62	Cohere Command R+ (Aug. 2024)	$0.0046	2.5s	14%	100	100	100	100	100	100	25	10	8	8	65%
134	Gemini 2.5 Pro	$0.072	54.5s	21%	100	100	100	100	100	50	25	25	25	25	65%
60	GPT-4.1 Nano	$0.0003	3.7s	18%	100	100	100	100	100	31	29	28	27	26	64%
151	MoonshotAI: Kimi K2.6	$0.037	5.6m	16%	100	100	100	100	100	25	25	25	25	25	63%
64	Xiaomi MIMO v2.5	$0.0050	22.9s	16%	100	100	100	100	100	25	25	25	25	25	63%
148	Qwen 3.5 397B A17B	$0.022	4.2m	14%	100	100	100	100	100	25	25	25	25	17	62%
111	Gemini 3 Flash (Preview, Reasoning)	$0.030	54.0s	14%	100	100	100	100	100	25	25	25	25	17	62%
66	GPT-5.4 (Reasoning, Low)	$0.0085	7.6s	13%	100	100	100	100	100	25	25	25	17	17	61%
59	Claude Haiku 4.5	$0.0029	3.2s	30%	100	100	50	50	50	50	50	50	50	50	60%
76	Llama 3.1 70B	$0.0032	38.5s	11%	100	100	100	100	100	35	30	17	5	4	59%
145	Grok 4	$0.060	1.6m	11%	100	100	100	100	100	25	17	17	17	13	59%
65	Qwen 2.5 72B	$0.0003	5.9s	11%	100	100	100	100	42	33	32	29	25	17	58%
69	Claude Sonnet 4	$0.0056	2.1s	10%	100	100	100	100	50	25	25	25	25	17	57%
67	GPT-4.1 Mini	$0.0009	3.8s	9%	100	100	100	100	35	33	31	31	30	7	57%
74	Stealth: Healer Alpha	$0.0000	19.2s	7%	100	100	100	100	25	25	25	25	25	25	55%
73	Skyfall 36B V2	$0.0006	2.7s	6%	100	100	100	100	50	25	13	13	7	6	51%
83	Hermes 3 405B	$0.0016	2.9s	3%	100	100	100	100	17	17	17	17	13	8	49%
71	Hermes 3 70B	$0.0005	5.9s	11%	100	100	100	50	42	38	33	13	7	5	49%
75	Grok 4.20 (Beta)	$0.0015	904ms	9%	100	100	100	42	35	25	25	25	17	17	49%
78	Cydonia 24B V4.1	$0.0004	6.3s	9%	100	100	100	50	35	29	28	17	10	8	48%
105	GPT-5.5 (Reasoning, Low)	$0.017	10.5s	8%	100	100	100	25	25	25	25	25	25	25	48%
139	Z.AI GLM 4.7	$0.014	2.0m	8%	100	100	100	25	25	25	25	25	25	25	48%
99	Claude 3.5 Sonnet	$0.014	5.7s	8%	100	100	100	42	38	25	17	13	13	10	46%
82	GPT-4.1	$0.0028	2.1s	12%	100	100	42	42	38	25	17	17	17	17	41%
149	GPT-5	$0.048	1.6m	10%	100	100	25	25	25	25	25	25	25	25	40%
114	Stealth: Hunter Alpha	$0.0000	1.0m	10%	100	100	25	25	25	25	25	25	25	25	40%
95	Claude Sonnet 4.5	$0.0069	3.5s	10%	100	100	42	25	25	25	25	25	17	17	40%
107	Rocinante 12B	$0.0005	11.7s	2%	100	100	100	50	10	10	10	6	5	4	39%
85	DeepSeek-V2 Chat	$0.0006	2.5s	10%	100	100	25	25	25	25	25	25	25	17	39%
86	GPT-5.4 Mini	$0.0007	3.4s	10%	100	100	25	25	25	25	25	25	25	17	39%
79	Claude Sonnet 4.6	$0.0099	6.2s	30%	42	38	38	38	38	35	35	35	35	8	34%
109	Grok 4.3	$0.0022	2.3s	5%	100	100	38	17	17	17	17	13	13	6	33%
77	Ministral 3 14B	$0.0004	5.3s	24%	50	50	30	30	30	29	29	29	29	28	33%
141	GPT-5.5 (Reasoning)	$0.041	19.5s	14%	100	25	25	25	25	25	25	25	25	25	33%
132	DeepSeek V4 Pro (Reasoning)	$0.0053	1.7m	14%	100	25	25	25	25	25	25	25	25	25	33%
142	Qwen 3.5 35B	$0.023	1.2m	14%	100	25	25	25	25	25	25	25	25	25	33%
72	Mistral Medium 3.1	$0.0011	2.8s	27%	42	42	42	35	32	31	30	30	25	17	32%
92	DeepSeek V3 (2024-12-26)	$0.0006	3.5s	14%	100	28	25	25	25	25	25	25	25	17	32%
96	GPT-5.4	$0.0026	1.7s	14%	100	25	25	25	25	25	25	25	25	17	32%
97	DeepSeek V4 Pro	$0.0022	31.9s	22%	50	50	50	35	31	30	25	17	13	13	31%
94	Grok 4.20	$0.0014	1.3s	13%	100	25	25	25	25	25	25	25	25	13	31%
101	GPT-4o, Aug. 6th (temp=1)	$0.0044	2.5s	13%	100	38	35	35	35	17	17	13	13	10	31%
131	GPT-5.4 (Reasoning)	$0.022	36.4s	13%	100	25	25	25	25	25	17	17	17	17	29%
133	Arcee AI: Trinity Large (Preview)	$0.0000	1.8m	13%	100	35	29	29	29	28	28	6	3	0	29%
81	Ministral 3B	$0.0001	3.5s	23%	50	30	30	29	28	28	28	28	27	8	29%
84	Ministral 3 3B	$0.0002	3.4s	24%	31	31	29	29	29	29	29	28	27	3	26%
87	Mistral Large 3	$0.0015	8.4s	25%	32	32	32	31	31	31	31	30	6	6	26%
103	DeepSeek V3.2	$0.0006	22.6s	21%	50	29	29	28	28	25	25	17	17	7	25%
119	GPT-4o Mini (temp=1)	$0.0003	3.7s	6%	100	33	32	31	13	13	10	8	7	7	25%
112	WizardLM 2 8x22b	$0.0011	26.8s	18%	50	50	33	25	25	25	17	13	8	6	25%
89	DeepSeek V4 Flash	$0.0002	6.9s	23%	33	32	31	30	29	28	25	25	10	7	25%
147	MoonshotAI: Kimi K2.5	$0.016	2.9m	25%	25	25	25	25	25	25	25	25	25	25	25%
100	Claude Opus 4.5	$0.0099	3.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
125	Qwen 3.5 Flash	$0.0046	1.4m	25%	25	25	25	25	25	25	25	25	25	25	25%
116	ByteDance Seed 2.0 Lite	$0.0047	51.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
90	Ministral 8B	$0.0003	7.2s	23%	28	28	28	27	27	27	27	27	26	2	25%
88	Mistral Small Creative	$0.0004	4.0s	24%	30	30	29	29	29	29	28	28	7	6	24%
102	Writer: Palmyra X5	$0.0025	12.5s	22%	29	28	25	25	25	25	25	17	17	13	23%
104	Mistral Large 2	$0.0059	7.8s	23%	35	33	31	31	31	30	10	10	8	7	23%
93	DeepSeek V3 (2025-03-24)	$0.0005	5.7s	23%	25	25	25	25	25	25	25	17	17	17	23%
91	Gemini 2.5 Flash	$0.0006	875ms	23%	25	25	25	25	25	25	25	17	17	17	23%
106	Mistral Large	$0.0059	7.7s	23%	33	32	32	31	31	30	10	10	8	6	22%
98	GPT-4o, Aug. 6th (temp=0)	$0.0032	1.3s	23%	25	25	25	25	25	25	25	17	17	13	22%
123	GPT-4o, May 13th (temp=1)	$0.010	1.9s	13%	42	35	31	17	17	17	17	13	13	8	21%
113	Qwen3 235B A22B Instruct 2507	$0.0002	4.6s	15%	30	25	25	25	17	17	17	17	17	17	20%
143	Qwen 3.5 Plus (2026-02-15)	$0.010	1.1m	13%	50	32	25	25	25	10	10	10	8	5	20%
110	Gemini 3.1 Flash Lite	$0.0007	1.3s	18%	35	25	25	25	25	17	13	13	13	10	20%
108	Gemini 3.1 Flash Lite (Reasoning)	$0.0006	1.4s	18%	25	25	25	25	25	17	17	13	13	13	20%
121	Claude 3 Haiku	$0.0008	2.9s	8%	38	38	35	33	13	10	8	7	7	6	19%
129	Ministral 3 8B	$0.0022	1.4m	20%	28	28	28	27	27	27	26	2	0	0	19%
122	GPT-4o Mini (temp=0)	$0.0003	3.8s	8%	35	35	35	31	10	10	10	8	8	8	19%
144	Claude Opus 4	$0.034	5.8s	15%	25	25	25	17	17	17	17	17	13	13	18%
124	Claude 3.7 Sonnet	$0.0073	3.4s	12%	42	25	17	17	17	13	13	13	13	13	18%
115	DeepSeek V3.1	$0.0006	9.2s	17%	29	25	25	25	25	17	10	8	7	6	18%
118	Gemini 3 Flash (Preview)	$0.0012	1.6s	16%	25	17	17	17	17	17	17	17	13	13	17%
117	Gemini 3.1 Flash Lite (Preview)	$0.0007	1.2s	15%	25	25	17	17	17	17	13	13	13	13	17%
127	Mistral Small 4	$0.0005	4.8s	6%	30	29	28	28	8	7	6	6	5	5	15%
126	GPT-4o, May 13th (temp=0)	$0.0097	1.8s	14%	17	17	17	17	17	13	13	13	13	10	14%
137	Llama 3.1 8B	$0.0001	20.2s	4%	32	30	29	6	5	5	5	4	4	2	12%
128	Gemini 2.5 Flash Lite	$0.0003	1.1s	9%	17	17	13	10	10	10	6	6	4	3	9%
130	Gemma 3 27B	$0.0002	8.7s	8%	13	10	10	8	8	8	7	6	6	6	8%
136	Mistral Small 3.2 24B	$0.0003	2.6s	5%	10	10	7	6	6	6	5	4	4	4	6%
138	Gemma 3 4B	$0.0001	4.8s	5%	7	7	7	6	6	6	6	6	6	4	6%
135	Mistral NeMO	$0.0003	2.8s	6%	8	7	7	7	6	6	5	5	4	3	6%
140	Gemma 3 12B	$0.0002	10.5s	5%	7	6	6	5	5	5	5	5	4	3	5%
58.87%

Median	Evaluator	Top 3	Flop 3
50.0%	Correct "no violations" response	100GPT-5.4 Mini (Reasoning) 100Nemotron 3 Super 100Grok 4.1 Fast	0Claude Opus 4.5 0Gemini 2.5 Flash Lite 0Gemini 3.1 Flash Lite (Reasoning)
65.0%	No hallucinated violations	100Claude Opus 4.6 (Reasoning) 100GPT-5.4 Mini (Reasoning) 100Z.AI GLM 4.7 Flash	3Ministral 3 8B 4Ministral 8B 8Ministral 3 3B

Codex Red Herring (False Positive Detection)

Short text (~524 words), small codex (11 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)