Short text (~524 words), small codex (11 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5 Mini	100%
Gemma 4 31B (Reasoning)	100%
Gemma 4 26B (Reasoning)	100%
Grok 4.20 (Beta, Reasoning)	100%
GPT-5.4 (Reasoning, Low)	100%
Z.AI GLM 5	100%
Claude Sonnet 4.6	100%
Qwen 3.5 27B	100%
ByteDance Seed 1.6	100%
GPT-5.4 Mini (Reasoning)	100%
o4 Mini High	100%
Claude Opus 4.7	100%
Claude Opus 4.5	100%
Grok 4.1 Fast	100%
GPT-4.1	100%
Grok 4	100%
Gemma 4 31B	100%

	Score	Cost	Time
LFM2 24B	62%	$0.0004	56.5s
Ministral 8B	100%	$0.0004	338ms
Mistral Small 3.2 24B	63%	$0.0004	1.7s
GPT-5.4 Nano	83%	$0.0005	1.3s
Ministral 3 8B	100%	$0.0006	339ms
Gemma 4 26B	100%	$0.0004	3.9s
Gemma 4 31B	100%	$0.0005	7.1s
GPT-4.1 Mini	93%	$0.0007	1.6s
Cydonia 24B V4.1	83%	$0.0007	1.4s
Ministral 3 14B	100%	$0.0008	531ms
GPT-5.4 Nano (Reasoning, Low)	93%	$0.0006	2.8s
Arcee AI: Trinity Mini	93%	$0.0013	41.5s
Inception Mercury	100%	$0.0003	6.3s
Mistral Large 3	100%	$0.0020	774ms
ByteDance Seed 1.6 Flash	84%	$0.0006	7.9s
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0023	3.9s
GPT-5.4 Nano (Reasoning)	100%	$0.0012	7.5s
Grok 4.1 Fast	100%	$0.0011	8.6s
GPT-4.1	100%	$0.0035	613ms
Inception Mercury 2	85%	$0.0023	3.4s

	Score	Cost	Speed	Stability
Ministral 8B	100%	$0.0004	338ms	100%
Ministral 3 8B	100%	$0.0006	339ms	100%
Ministral 3 14B	100%	$0.0008	531ms	100%
Gemma 4 26B	100%	$0.0004	3.9s	100%
Mistral Large 3	100%	$0.0020	774ms	100%
Inception Mercury	100%	$0.0003	6.3s	100%
Gemma 4 31B	100%	$0.0005	7.1s	100%
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0023	3.9s	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0012	7.5s	100%
GPT-4.1	100%	$0.0035	613ms	100%
Grok 4.1 Fast	100%	$0.0011	8.6s	100%
GPT-5.4 Mini (Reasoning)	100%	$0.0050	6.1s	100%
Gemini 2.5 Flash Lite (Reasoning)	100%	$0.0021	15.1s	100%
Mistral Large 2	100%	$0.0078	650ms	100%
Mistral Large	100%	$0.0078	1.0s	100%
GPT-5.4 (Reasoning, Low)	100%	$0.0068	5.9s	100%
Z.AI GLM 5 Turbo	100%	$0.0055	11.5s	100%
Gemini 2.5 Flash (Reasoning)	100%	$0.0060	21.5s	100%
Claude Sonnet 4.6	100%	$0.013	1.0s	100%
ByteDance Seed 1.6	100%	$0.0035	29.7s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
32	Z.AI GLM 5.1	$0.020	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Z.AI GLM 5 Turbo	$0.0055	11.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Claude Opus 4.7 (Reasoning)	$0.028	1.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	GPT-5 Mini	$0.0047	29.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Gemma 4 31B (Reasoning)	$0.0017	2.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Gemma 4 26B (Reasoning)	$0.0026	1.9m	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Grok 4.20 (Beta, Reasoning)	$0.022	13.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5.4 (Reasoning, Low)	$0.0068	5.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Z.AI GLM 5	$0.012	57.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Claude Sonnet 4.6	$0.013	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Qwen 3.5 27B	$0.037	3.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
20	ByteDance Seed 1.6	$0.0035	29.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	GPT-5.4 Mini (Reasoning)	$0.0050	6.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	o4 Mini High	$0.023	51.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Claude Opus 4.7	$0.028	1.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	Claude Opus 4.5	$0.021	2.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Grok 4.1 Fast	$0.0011	8.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-4.1	$0.0035	613ms	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Grok 4	$0.043	52.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Gemma 4 31B	$0.0005	7.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Gemini 2.5 Flash (Reasoning)	$0.0060	21.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	GPT-OSS 120B	$0.0008	1.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Gemma 4 26B	$0.0004	3.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	GPT-5.4 Mini (Reasoning, Low)	$0.0023	3.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	Gemini 2.5 Flash Lite (Reasoning)	$0.0021	15.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Mistral Large 3	$0.0020	774ms	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Nemotron 3 Super	$0.0000	52.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Mistral Large 2	$0.0078	650ms	100%	100	100	100	100	100	100	100	100	100	100	100%
9	GPT-5.4 Nano (Reasoning)	$0.0012	7.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Mistral Large	$0.0078	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury	$0.0003	6.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Ministral 3 14B	$0.0008	531ms	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0006	339ms	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0004	338ms	100%	100	100	100	100	100	100	100	100	100	100	100%
34	Claude Haiku 4.5	$0.0044	1.3s	70%	100	100	100	100	100	100	100	100	100	50	95%
35	GPT-5.4 Nano (Reasoning, Low)	$0.0006	2.8s	60%	100	100	100	100	100	100	100	100	100	33	93%
36	GPT-4.1 Mini	$0.0007	1.6s	59%	100	100	100	100	100	100	100	100	100	32	93%
62	Qwen3.7 Max	$0.047	1.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
47	GPT-5.1	$0.022	27.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
52	Qwen 3.5 Plus (2026-04-20)	$0.016	1.6m	55%	100	100	100	100	100	100	100	100	100	25	93%
46	Grok 4.20 (Reasoning)	$0.012	39.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
49	Aion 2.0	$0.0088	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
38	MiniMax M2.7	$0.0033	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
43	o4 Mini	$0.012	22.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
45	Xiaomi MIMO v2.5 Pro	$0.0085	36.9s	55%	100	100	100	100	100	100	100	100	100	25	93%
41	Stealth: Hunter Alpha	$0.0000	45.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Grok 4 Fast	$0.0013	13.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
42	GPT-5 Nano	$0.0023	43.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
40	Arcee AI: Trinity Mini	$0.0013	41.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
44	Xiaomi MIMO v2.5	$0.0042	17.1s	50%	100	100	100	100	100	100	100	100	100	17	92%
58	Claude Opus 4.6	$0.036	11.3s	49%	100	100	100	100	100	100	100	100	50	25	88%
73	Claude Opus 4.6 (Reasoning)	$0.079	37.1s	40%	100	100	100	100	100	100	100	100	25	25	85%
60	Grok 4.3 (Reasoning)	$0.013	50.7s	40%	100	100	100	100	100	100	100	100	25	25	85%
59	GPT-5.4 (Reasoning)	$0.020	20.9s	40%	100	100	100	100	100	100	100	100	25	25	85%
151	Qwen 3.5 122B	$0.166	8.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
56	GPT-5.2	$0.013	16.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
70	Qwen 3.6 27B	$0.028	1.9m	40%	100	100	100	100	100	100	100	100	25	25	85%
54	Z.AI GLM 4.6	$0.0038	29.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
53	DeepSeek V4 Flash (Reasoning)	$0.0007	26.8s	40%	100	100	100	100	100	100	100	100	25	25	85%
66	Z.AI GLM 4.7	$0.018	1.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
51	Stealth: Healer Alpha	$0.0000	25.2s	40%	100	100	100	100	100	100	100	100	25	25	85%
50	Inception Mercury 2	$0.0023	3.4s	40%	100	100	100	100	100	100	100	100	25	25	85%
65	Nemotron 3 Nano	$0.0022	2.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
61	Z.AI GLM 4.5 Air	$0.0028	1.0m	37%	100	100	100	100	100	100	100	100	25	17	84%
55	ByteDance Seed 1.6 Flash	$0.0006	7.9s	35%	100	100	100	100	100	100	100	100	25	13	84%
57	Cydonia 24B V4.1	$0.0007	1.4s	32%	100	100	100	100	100	100	100	100	17	13	83%
48	GPT-5.4 Nano	$0.0005	1.3s	47%	100	100	100	100	100	100	100	50	50	29	83%
63	Cohere Command R+ (Aug. 2024)	$0.010	2.1s	27%	100	100	100	100	100	100	100	100	10	8	82%
127	Gemini 3 Pro (Preview)	$0.118	1.3m	31%	100	100	100	100	100	100	100	25	25	25	78%
64	MiniMax M2.5	$0.0029	22.7s	31%	100	100	100	100	100	100	100	25	25	25	78%
69	Z.AI GLM 4.7 Flash	$0.0024	1.5m	31%	100	100	100	100	100	100	100	25	25	25	78%
67	Mistral Small 4 (Reasoning)	$0.0022	18.2s	26%	100	100	100	100	100	100	100	25	17	17	76%
68	Hermes 3 405B	$0.0039	2.5s	21%	100	100	100	100	100	100	100	17	13	13	74%
71	Qwen 3.6 Flash	$0.013	39.2s	27%	100	100	100	100	100	100	25	25	25	25	70%
108	Qwen 3.5 9B	$0.0034	4.5m	22%	100	100	100	100	50	50	50	50	50	25	68%
75	Mistral Small 3.2 24B	$0.0004	1.7s	8%	100	100	100	100	100	100	8	6	6	5	63%
128	MoonshotAI: Kimi K2.5	$0.017	3.9m	16%	100	100	100	100	100	25	25	25	25	25	63%
83	Gemini 3 Flash (Preview, Reasoning)	$0.024	43.4s	16%	100	100	100	100	100	25	25	25	25	25	63%
122	Gemini 2.5 Pro	$0.069	49.8s	16%	100	100	100	100	100	25	25	25	25	25	63%
72	Skyfall 36B V2	$0.0014	6.3s	16%	100	100	100	100	100	38	38	17	17	13	62%
82	LFM2 24B	$0.0004	56.5s	6%	100	100	100	100	100	100	6	5	5	0	62%
74	Llama 3.1 70B	$0.0019	3.5s	14%	100	100	100	100	42	42	38	25	25	25	60%
76	GPT-4o, Aug. 6th (temp=1)	$0.0065	1.1s	13%	100	100	100	100	42	42	25	25	25	25	58%
79	Z.AI GLM 4.5	$0.0032	21.3s	9%	100	100	100	100	100	29	29	13	4	3	58%
138	Claude Sonnet 4.6 (Reasoning)	$0.065	51.6s	7%	100	100	100	100	25	25	25	25	25	25	55%
133	GPT-5	$0.045	1.3m	7%	100	100	100	100	25	25	25	25	25	25	55%
147	DeepSeek V4 Pro (Reasoning)	$0.029	5.7m	7%	100	100	100	100	25	25	25	25	25	25	55%
114	Qwen 3.5 Flash	$0.0070	2.0m	6%	100	100	100	100	25	25	25	25	25	17	54%
81	GPT-4o, Aug. 6th (temp=0)	$0.0055	1.2s	6%	100	100	100	100	25	25	25	25	25	17	54%
84	Qwen 3 32B	$0.0010	33.7s	6%	100	100	100	100	25	25	25	25	25	17	54%
146	Qwen 3.5 397B A17B	$0.024	5.4m	6%	100	100	100	100	25	25	25	25	17	17	53%
78	Mistral Medium 3.1	$0.0018	2.6s	13%	100	100	100	50	42	32	25	25	25	17	52%
100	Claude Sonnet 4	$0.014	2.9s	3%	100	100	100	100	17	17	17	17	17	17	50%
80	GPT-4.1 Nano	$0.0004	4.9s	10%	100	100	100	30	29	28	28	27	27	27	50%
134	Qwen 3.5 35B	$0.029	1.6m	7%	100	100	100	25	25	25	25	25	25	17	47%
97	Rocinante 12B	$0.0009	8.3s	2%	100	100	100	100	25	10	6	4	4	3	45%
86	Hermes 3 70B	$0.0012	8.3s	7%	100	100	100	50	31	25	17	13	10	6	45%
125	Claude 3.5 Sonnet	$0.031	6.5s	4%	100	100	100	25	17	17	17	17	13	10	41%
145	Gemini 3.1 Pro (Preview)	$0.084	58.8s	10%	100	100	25	25	25	25	25	25	25	25	40%
135	Gemini 3.5 Flash (Reasoning)	$0.050	20.6s	10%	100	100	25	25	25	25	25	25	25	25	40%
115	ByteDance Seed 2.0 Lite	$0.0068	1.1m	10%	100	100	25	25	25	25	25	25	25	25	40%
150	Qwen3.6 Max Preview	$0.068	3.7m	10%	100	100	25	25	25	25	25	25	25	17	39%
148	MoonshotAI: Kimi K2.6	$0.053	4.1m	10%	100	100	25	25	25	25	25	25	25	17	39%
106	Grok 4.3	$0.0052	2.2s	2%	100	100	100	25	13	13	13	10	10	8	39%
92	Mistral Small Creative	$0.0005	2.6s	10%	100	100	35	29	29	29	29	17	8	7	38%
113	Claude 3.7 Sonnet	$0.015	3.7s	7%	100	100	38	35	25	17	13	13	13	13	36%
144	Claude Opus 4	$0.071	5.2s	7%	100	100	25	25	25	17	17	17	17	17	36%
87	Qwen 2.5 72B	$0.0007	10.8s	17%	100	31	31	30	30	29	29	25	25	25	35%
77	Claude 3 Haiku	$0.0014	2.9s	31%	42	42	42	38	38	38	35	35	35	10	35%
130	GPT-5.5 (Reasoning)	$0.036	15.8s	14%	100	25	25	25	25	25	25	25	25	25	33%
121	Qwen 3.6 35B	$0.013	53.2s	14%	100	25	25	25	25	25	25	25	25	25	33%
96	GPT-5.4 Mini	$0.0015	653ms	14%	100	25	25	25	25	25	25	25	25	25	33%
98	DeepSeek V3 (2025-03-24)	$0.0008	4.3s	14%	100	25	25	25	25	25	25	25	25	25	33%
85	Ministral 3B	$0.0002	3.8s	23%	50	42	33	32	30	29	29	29	29	4	31%
119	Grok 4.20	$0.0046	4.1s	4%	100	100	13	13	13	13	10	10	10	10	29%
149	ByteDance Seed 2.0 Mini	$0.0052	5.8m	11%	100	25	25	25	25	17	17	17	17	17	28%
109	DeepSeek V4 Pro	$0.0040	30.4s	17%	50	50	50	33	28	25	25	17	3	3	28%
126	Arcee AI: Trinity Large (Preview)	$0.0000	2.0m	21%	50	35	28	27	27	27	27	26	26	4	28%
91	Llama 3.1 Nemotron 70B	$0.0051	12.9s	26%	38	38	38	35	35	35	33	17	6	4	28%
140	Qwen 3.5 Plus (2026-02-15)	$0.023	2.2m	21%	50	42	38	35	33	25	17	13	10	10	27%
93	Gemini 3.1 Flash Lite (Preview)	$0.0011	1.2s	21%	50	25	25	25	25	25	25	25	25	17	27%
105	DeepSeek V4 Flash	$0.0003	9.7s	19%	50	50	33	28	28	28	25	13	2	2	26%
99	Ministral 3 3B	$0.0006	8.5s	21%	42	38	33	29	28	28	27	26	3	1	26%
111	GPT-5.5 (Reasoning, Low)	$0.023	11.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
102	GPT-5.5	$0.0093	2.9s	25%	25	25	25	25	25	25	25	25	25	25	25%
94	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0067	1.4s	25%	25	25	25	25	25	25	25	25	25	25	25%
88	Gemini 3.1 Flash Lite (Reasoning)	$0.0011	1.4s	25%	25	25	25	25	25	25	25	25	25	25	25%
89	Gemini 3.1 Flash Lite	$0.0011	1.8s	25%	25	25	25	25	25	25	25	25	25	25	25%
90	DeepSeek V3 (2024-12-26)	$0.0014	2.4s	25%	25	25	25	25	25	25	25	25	25	25	25%
123	Grok 4.20 (Beta)	$0.0029	1.3s	6%	100	42	17	17	13	13	13	10	10	7	24%
101	GPT-4o Mini (temp=1)	$0.0007	6.7s	23%	35	33	32	31	31	30	29	5	4	4	23%
95	DeepSeek-V2 Chat	$0.0013	2.1s	23%	25	25	25	25	25	25	25	25	17	17	23%
103	GPT-5.4	$0.0031	2.2s	23%	25	25	25	25	25	25	25	17	17	17	23%
104	DeepSeek V3.1	$0.0008	8.0s	23%	25	25	25	25	25	25	25	25	13	13	23%
107	Llama 3.1 8B	$0.0002	31.4s	21%	38	30	29	28	28	27	27	8	6	4	22%
129	GPT-4o, May 13th (temp=1)	$0.023	3.3s	16%	42	35	33	29	25	17	17	13	10	4	22%
110	Qwen3 235B A22B Instruct 2507	$0.0007	17.5s	20%	42	28	25	25	25	25	25	10	8	3	22%
117	DeepSeek V3.2	$0.0010	14.7s	14%	38	33	25	17	17	17	17	17	13	10	20%
112	Mistral Small 4	$0.0008	4.1s	16%	31	31	30	29	28	13	13	8	6	6	19%
124	WizardLM 2 8x22b	$0.0027	26.8s	15%	42	25	25	25	25	13	13	10	8	4	19%
120	Writer: Palmyra X5	$0.0038	5.3s	15%	25	25	25	25	17	17	17	10	10	10	18%
131	Claude Sonnet 4.5	$0.016	4.9s	15%	25	25	17	17	17	17	17	17	13	10	17%
116	Gemini 2.5 Flash	$0.0013	1.7s	17%	25	25	25	25	25	17	13	13	3	2	17%
118	Gemini 3 Flash (Preview)	$0.0017	2.3s	15%	25	17	17	17	17	17	17	17	10	10	16%
136	GPT-4o, May 13th (temp=0)	$0.023	2.4s	14%	32	17	17	17	17	17	10	8	7	7	15%
137	Gemini 2.5 Flash Lite	$0.0006	2.2s	2%	100	6	6	5	4	4	3	3	3	3	13%
132	Gemma 3 12B	$0.0003	5.8s	12%	17	17	13	13	13	13	13	8	7	6	12%
139	Gemma 3 27B	$0.0005	9.5s	7%	13	10	8	8	7	7	6	6	6	5	8%
141	Mistral NeMO	$0.0007	4.7s	5%	8	6	6	5	5	5	4	4	4	3	5%
142	Gemma 3 4B	$0.0002	6.2s	4%	6	6	5	5	5	5	5	4	4	3	5%
143	GPT-4o Mini (temp=0)	$0.0007	9.9s	4%	4	4	4	4	4	4	4	4	4	4	4%
63.12%

Median	Evaluator	Top 3	Flop 3
60.0%	Correct "no violations" response	100Gemma 4 31B 100o4 Mini High 100GPT-4.1	0Gemini 3.1 Flash Lite 0Gemma 3 27B 0GPT-5.5 (Reasoning, Low)
75.0%	No hallucinated violations	100Gemma 4 31B (Reasoning) 100Grok 4.20 (Beta, Reasoning) 100Ministral 3 8B	8GPT-4o Mini (temp=0) 9Gemma 3 4B 10Mistral NeMO

Codex Red Herring (False Positive Detection)

Short text (~524 words), small codex (11 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)