Long text (~1594 words), small codex (11 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Qwen3.7 Max	100%
Claude Opus 4.6 (Reasoning)	100%
Qwen3.6 Max Preview	100%
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Claude Sonnet 4.6 (Reasoning)	100%
Grok 4.3 (Reasoning)	100%
GPT-5.4 (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5.5 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.5 (Reasoning, Low)	100%
GPT-5.1	100%
MoonshotAI: Kimi K2.6	100%
GPT-5	100%
Gemma 4 31B (Reasoning)	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 26B (Reasoning)	100%
Grok 4.20 (Beta, Reasoning)	100%
GPT-5.4 (Reasoning, Low)	100%

	Score	Cost	Time
GPT-4.1 Nano	93%	$0.0001	962ms
LFM2 24B	100%	$0.0001	1.8s
Gemini 2.5 Flash Lite	74%	$0.0003	516ms
Mistral Small 3.2 24B	72%	$0.0003	1.0s
Ministral 3 8B	90%	$0.0004	1.4s
Grok 4.20 (Beta)	77%	$0.0018	984ms
GPT-5.4 Nano	66%	$0.0004	1.6s
Gemini 3.1 Flash Lite	85%	$0.0007	741ms
Gemini 3.1 Flash Lite (Reasoning)	79%	$0.0008	874ms
Gemini 3.1 Flash Lite (Preview)	70%	$0.0008	882ms
Gemma 4 31B	100%	$0.0004	20.6s
Inception Mercury	100%	$0.0002	3.7s
Mistral Medium 3.1	95%	$0.0011	764ms
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.6s
Skyfall 36B V2	69%	$0.0010	1.9s
GPT-4.1	100%	$0.0022	815ms
Inception Mercury 2	100%	$0.0013	2.0s
Cydonia 24B V4.1	62%	$0.0008	9.4s
GPT-5.4 Nano (Reasoning)	100%	$0.0010	3.7s
ByteDance Seed 1.6 Flash	100%	$0.0005	7.4s

	Score	Cost	Speed	Stability
LFM2 24B	100%	$0.0001	1.8s	100%
Inception Mercury	100%	$0.0002	3.7s	100%
Inception Mercury 2	100%	$0.0013	2.0s	100%
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.6s	100%
GPT-4.1	100%	$0.0022	815ms	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0010	3.7s	100%
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0021	3.0s	100%
Grok 4.1 Fast	100%	$0.0009	6.1s	100%
ByteDance Seed 1.6 Flash	100%	$0.0005	7.4s	100%
Stealth: Healer Alpha	100%	$0.0000	15.5s	100%
Grok 4 Fast	100%	$0.0010	13.9s	100%
Z.AI GLM 5 Turbo	100%	$0.0039	9.7s	100%
DeepSeek V4 Flash (Reasoning)	100%	$0.0005	19.3s	100%
Gemma 4 31B	100%	$0.0004	20.6s	100%
GPT-5.4 Mini (Reasoning)	100%	$0.0065	7.8s	100%
GPT-5.4 (Reasoning, Low)	100%	$0.0077	5.5s	100%
Xiaomi MIMO v2.5	100%	$0.0035	14.7s	100%
GPT-5.2	100%	$0.0083	9.6s	100%
Gemini 2.5 Flash (Reasoning)	100%	$0.0074	11.8s	100%
MiniMax M2.7	100%	$0.0020	27.5s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
47	Qwen3.7 Max	$0.036	59.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
60	Claude Opus 4.6 (Reasoning)	$0.083	50.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
79	Qwen3.6 Max Preview	$0.065	4.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
37	Z.AI GLM 5.1	$0.012	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Z.AI GLM 5 Turbo	$0.0039	9.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
69	Claude Sonnet 4.6 (Reasoning)	$0.101	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
36	Grok 4.3 (Reasoning)	$0.011	51.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	GPT-5.4 (Reasoning)	$0.015	14.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
38	Claude Opus 4.7 (Reasoning)	$0.038	9.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	GPT-5.5 (Reasoning)	$0.019	10.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	GPT-5 Mini	$0.0044	29.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	GPT-5.5 (Reasoning, Low)	$0.016	7.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	GPT-5.1	$0.012	13.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
63	MoonshotAI: Kimi K2.6	$0.024	3.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
45	GPT-5	$0.030	57.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
42	Gemma 4 31B (Reasoning)	$0.0014	1.9m	100%	100	100	100	100	100	100	100	100	100	100	100%
46	Qwen 3.5 Plus (2026-04-20)	$0.016	1.6m	100%	100	100	100	100	100	100	100	100	100	100	100%
49	Gemma 4 26B (Reasoning)	$0.0018	2.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Grok 4.20 (Beta, Reasoning)	$0.015	10.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5.4 (Reasoning, Low)	$0.0077	5.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	Grok 4.20 (Reasoning)	$0.011	35.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Z.AI GLM 5	$0.0087	48.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
44	MoonshotAI: Kimi K2.5	$0.012	1.6m	100%	100	100	100	100	100	100	100	100	100	100	100%
23	ByteDance Seed 1.6	$0.0033	31.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	GPT-5.4 Mini (Reasoning)	$0.0065	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
35	o4 Mini High	$0.018	33.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	GPT-5.2	$0.0083	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
56	Qwen 3.6 27B	$0.032	2.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Grok 4.1 Fast	$0.0009	6.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	MiniMax M2.7	$0.0020	27.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	DeepSeek V4 Flash (Reasoning)	$0.0005	19.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
43	Z.AI GLM 4.7	$0.015	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-4.1	$0.0022	815ms	100%	100	100	100	100	100	100	100	100	100	100	100%
22	o4 Mini	$0.0085	15.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
32	Xiaomi MIMO v2.5 Pro	$0.0095	45.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	ByteDance Seed 2.0 Mini	$0.0018	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Gemma 4 31B	$0.0004	20.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Gemini 2.5 Flash (Reasoning)	$0.0074	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	GPT-OSS 120B	$0.0006	32.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Grok 4 Fast	$0.0010	13.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Stealth: Healer Alpha	$0.0000	15.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Mini (Reasoning, Low)	$0.0021	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Xiaomi MIMO v2.5	$0.0035	14.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Nemotron 3 Super	$0.0000	47.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Inception Mercury 2	$0.0013	2.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
34	GPT-5 Nano	$0.0033	1.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.4 Nano (Reasoning)	$0.0010	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Inception Mercury	$0.0002	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
41	Nemotron 3 Nano	$0.0016	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
9	ByteDance Seed 1.6 Flash	$0.0005	7.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0001	1.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
40	Mistral Medium 3.1	$0.0011	764ms	70%	100	100	100	100	100	100	100	100	100	50	95%
51	Claude Sonnet 4.6	$0.019	12.6s	59%	100	100	100	100	100	100	100	100	100	32	93%
48	GPT-4.1 Nano	$0.0001	962ms	58%	100	100	100	100	100	100	100	100	100	30	93%
52	Stealth: Hunter Alpha	$0.0000	51.9s	57%	100	100	100	100	100	100	100	100	100	28	93%
55	Qwen 3.6 Flash	$0.013	38.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
61	DeepSeek V4 Pro (Reasoning)	$0.0041	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
59	Aion 2.0	$0.0062	58.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
50	Gemini 2.5 Flash Lite (Reasoning)	$0.0020	14.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
58	Z.AI GLM 4.5 Air	$0.0041	1.0m	55%	100	100	100	100	100	100	100	100	100	25	93%
53	Ministral 3 8B	$0.0004	1.4s	41%	100	100	100	100	100	100	100	100	100	2	90%
54	MiniMax M2.5	$0.0020	24.5s	49%	100	100	100	100	100	100	100	100	50	25	88%
65	Qwen 3.6 35B	$0.0070	51.0s	40%	100	100	100	100	100	100	100	100	25	25	85%
57	Gemini 3.1 Flash Lite	$0.0007	741ms	40%	100	100	100	100	100	100	100	100	25	25	85%
76	Gemini 2.5 Pro	$0.045	32.3s	37%	100	100	100	100	100	100	100	100	25	17	84%
98	Grok 4	$0.063	1.6m	35%	100	100	100	100	100	100	100	100	25	13	84%
62	Gemini 3.1 Flash Lite (Reasoning)	$0.0008	874ms	34%	100	100	100	100	100	100	100	50	25	17	79%
83	Gemini 3.5 Flash (Reasoning)	$0.059	26.0s	31%	100	100	100	100	100	100	100	25	25	25	78%
64	Grok 4.20 (Beta)	$0.0018	984ms	29%	100	100	100	100	100	100	100	33	33	6	77%
67	Gemini 2.5 Flash Lite	$0.0003	516ms	20%	100	100	100	100	100	100	100	17	17	6	74%
74	Mistral Small 3.2 24B	$0.0003	1.0s	14%	100	100	100	100	100	100	100	6	6	6	72%
66	Gemini 3.1 Flash Lite (Preview)	$0.0008	882ms	26%	100	100	100	100	100	100	42	25	25	13	70%
72	Mistral Small 4 (Reasoning)	$0.0018	16.7s	24%	100	100	100	100	100	100	25	25	25	17	69%
70	Hermes 3 70B	$0.0009	6.2s	21%	100	100	100	100	100	100	42	33	7	6	69%
71	Skyfall 36B V2	$0.0010	1.9s	20%	100	100	100	100	100	100	50	13	13	10	69%
81	Qwen 3 32B	$0.0015	1.3m	21%	100	100	100	100	100	100	25	25	17	13	68%
75	GPT-5.4 Nano	$0.0004	1.6s	17%	100	100	100	100	100	100	31	17	8	7	66%
73	Ministral 8B	$0.0004	4.4s	22%	100	100	100	100	100	50	27	27	26	26	66%
149	Qwen 3.5 27B	$0.052	4.7m	18%	100	100	100	100	100	50	25	25	17	13	63%
80	Cydonia 24B V4.1	$0.0008	9.4s	8%	100	100	100	100	100	100	8	7	6	4	62%
77	Z.AI GLM 4.5	$0.0022	16.0s	15%	100	100	100	100	100	32	30	29	25	3	62%
133	Z.AI GLM 4.7 Flash	$0.0040	3.1m	14%	100	100	100	100	100	25	25	25	25	17	62%
143	Qwen 3.5 9B	$0.0034	4.5m	22%	100	100	100	50	50	50	50	50	25	25	60%
120	Z.AI GLM 4.6	$0.033	1.6m	14%	100	100	100	100	50	50	29	25	25	7	59%
148	Gemini 3.1 Pro (Preview)	$0.110	1.3m	7%	100	100	100	100	25	25	25	25	25	25	55%
147	Qwen 3.5 397B A17B	$0.018	3.9m	5%	100	100	100	100	25	25	25	17	13	13	52%
92	Grok 4.20	$0.0059	14.1s	4%	100	100	100	100	29	27	27	6	6	2	50%
88	Arcee AI: Trinity Mini	$0.0003	8.2s	3%	100	100	100	100	25	17	17	17	13	6	49%
68	Claude Haiku 4.5	$0.0042	3.4s	48%	50	50	50	50	50	50	50	50	50	42	49%
151	Qwen 3.5 122B	$0.101	4.5m	7%	100	100	100	25	25	25	25	25	17	17	46%
78	Gemma 4 26B	$0.0004	10.2s	27%	100	50	50	50	50	42	42	25	25	25	46%
146	Qwen 3.5 35B	$0.047	2.4m	5%	100	100	100	25	25	17	17	17	13	13	43%
82	DeepSeek V4 Pro	$0.0047	29.4s	33%	50	50	50	42	42	42	29	28	25	25	38%
89	Claude Opus 4.7	$0.030	6.3s	37%	38	38	38	38	38	38	38	38	35	35	37%
95	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0099	3.9s	15%	100	50	50	35	35	25	25	17	17	10	36%
84	Claude Sonnet 4	$0.012	4.5s	30%	50	42	42	42	42	38	35	17	17	13	34%
87	DeepSeek V4 Flash	$0.0003	9.8s	21%	50	50	50	30	29	29	28	28	28	6	33%
130	Gemini 3 Flash (Preview, Reasoning)	$0.025	44.3s	14%	100	25	25	25	25	25	25	25	25	17	32%
150	Gemini 3 Pro (Preview)	$0.131	1.5m	14%	100	25	25	25	25	25	25	25	25	17	32%
91	Claude Sonnet 4.5	$0.016	7.6s	29%	35	35	33	33	33	33	33	32	32	10	31%
126	Claude Opus 4	$0.056	6.7s	27%	50	50	42	42	42	38	13	13	13	10	31%
112	WizardLM 2 8x22b	$0.0021	47.3s	8%	100	50	50	50	30	8	6	6	5	4	31%
100	Llama 3.1 Nemotron 70B	$0.0040	22.1s	15%	100	38	35	32	31	31	28	5	5	3	31%
96	GPT-5.4 Mini	$0.0013	787ms	13%	100	25	25	25	25	25	25	25	17	13	30%
94	GPT-4o Mini (temp=1)	$0.0004	4.8s	15%	100	38	32	32	31	29	13	13	8	6	30%
85	Ministral 3B	$0.0001	3.0s	25%	42	38	31	31	31	31	30	29	29	3	29%
103	GPT-4o, Aug. 6th (temp=1)	$0.0065	2.7s	8%	100	42	42	42	17	17	13	8	7	7	29%
93	Writer: Palmyra X5	$0.0066	13.0s	25%	42	35	32	32	31	30	30	30	29	2	29%
86	GPT-4.1 Mini	$0.0015	7.2s	25%	35	35	35	33	31	30	30	29	28	5	29%
90	Ministral 3 3B	$0.0004	5.0s	24%	38	32	30	30	29	29	29	29	27	1	27%
108	Claude Opus 4.6	$0.030	12.3s	23%	42	42	38	35	32	32	32	6	6	6	27%
99	Mistral Small 4	$0.0008	5.6s	12%	100	32	32	30	28	26	6	5	5	4	27%
102	Qwen 2.5 72B	$0.0006	9.9s	10%	100	35	31	29	25	17	13	6	6	6	27%
140	Qwen 3.5 Flash	$0.0061	1.6m	8%	100	25	25	25	17	17	17	17	13	13	27%
104	DeepSeek V3.1	$0.0011	18.9s	9%	100	31	30	29	25	13	13	8	8	7	26%
101	Qwen3 235B A22B Instruct 2507	$0.0009	27.7s	21%	42	35	32	31	30	29	28	5	4	4	24%
111	Mistral NeMO	$0.0005	7.8s	1%	100	100	6	4	4	4	4	3	3	1	23%
122	ByteDance Seed 2.0 Lite	$0.0070	1.2m	23%	25	25	25	25	25	25	25	17	17	17	23%
109	Rocinante 12B	$0.0008	9.3s	4%	100	33	31	17	10	8	7	6	4	3	22%
107	Hermes 3 405B	$0.0029	6.3s	6%	100	25	17	13	13	13	10	10	10	8	22%
97	GPT-5.4	$0.0034	2.2s	22%	25	25	25	25	25	25	25	17	13	7	21%
144	Qwen 3.5 Plus (2026-02-15)	$0.020	1.9m	14%	50	31	31	31	30	10	10	8	6	2	21%
116	GPT-4o, May 13th (temp=1)	$0.018	3.4s	13%	42	32	32	29	25	10	8	8	7	6	20%
117	Grok 4.3	$0.0036	5.4s	3%	100	32	17	13	7	6	6	4	4	3	19%
110	DeepSeek V3.2	$0.0009	27.4s	13%	42	30	29	28	17	17	13	5	5	4	19%
135	Cohere Command R+ (Aug. 2024)	$0.012	15.0s	4%	100	17	17	10	10	10	8	7	6	1	19%
128	Claude Opus 4.5	$0.021	5.2s	12%	38	38	17	17	17	13	10	10	10	10	18%
114	DeepSeek V3 (2025-03-24)	$0.0007	15.9s	7%	42	35	33	10	10	10	10	10	8	7	18%
105	DeepSeek-V2 Chat	$0.0011	4.8s	9%	50	25	17	17	13	13	10	10	10	6	17%
129	Claude 3.5 Sonnet	$0.024	6.3s	16%	25	17	17	17	17	17	17	13	13	13	16%
113	GPT-5.5	$0.011	3.1s	13%	25	25	25	17	17	13	10	10	10	8	16%
106	DeepSeek V3 (2024-12-26)	$0.0012	5.5s	11%	31	25	17	13	13	13	13	13	13	8	16%
134	Llama 3.1 8B	$0.0001	28.5s	5%	29	28	28	28	7	6	6	5	5	4	15%
136	Arcee AI: Trinity Large (Preview)	$0.0000	31.2s	4%	29	28	28	27	5	5	5	4	3	3	14%
119	Llama 3.1 70B	$0.0014	12.4s	8%	31	25	17	17	10	10	10	10	5	3	14%
124	Claude 3.7 Sonnet	$0.012	4.1s	12%	17	17	13	13	13	13	13	10	10	10	13%
115	Gemini 3 Flash (Preview)	$0.0018	3.3s	8%	35	13	13	13	10	10	8	8	8	8	13%
131	Mistral Small Creative	$0.0008	9.4s	3%	27	26	26	26	4	3	3	2	2	2	12%
121	Claude 3 Haiku	$0.0011	3.0s	7%	33	10	10	8	8	7	7	7	6	6	10%
137	Ministral 3 14B	$0.0009	12.9s	2%	27	26	26	3	3	3	3	2	2	1	10%
118	Gemini 2.5 Flash	$0.0015	1.7s	9%	13	13	13	10	10	10	8	7	6	5	9%
123	Gemma 3 12B	$0.0001	6.8s	8%	17	8	8	8	8	8	7	7	7	7	9%
127	Gemma 3 27B	$0.0003	8.8s	8%	10	10	10	10	8	8	8	8	7	5	9%
132	GPT-4o, Aug. 6th (temp=0)	$0.0071	2.9s	8%	10	10	8	8	8	8	7	7	6	6	8%
125	GPT-4o Mini (temp=0)	$0.0005	5.4s	8%	8	8	8	8	8	8	8	8	6	6	8%
141	Mistral Large 2	$0.011	15.1s	4%	29	6	5	5	5	4	4	3	3	3	6%
138	Mistral Large 3	$0.0026	12.1s	5%	6	6	6	6	6	5	5	5	3	3	5%
145	GPT-4o, May 13th (temp=0)	$0.035	9.4s	4%	8	8	7	7	6	3	1	1	1	1	4%
142	Mistral Large	$0.011	15.0s	4%	6	6	5	5	4	4	4	4	3	3	4%
139	Gemma 3 4B	$0.0002	11.0s	3%	4	3	3	3	3	3	3	2	2	2	3%
62.01%

Median	Evaluator	Top 3	Flop 3
65.0%	Correct "no violations" response	100o4 Mini High 100Grok 4.3 (Reasoning) 100ByteDance Seed 2.0 Mini	0Gemma 3 27B 0GPT-5.4 0Claude 3.7 Sonnet
73.5%	No hallucinated violations	100o4 Mini High 100GPT-5.4 (Reasoning, Low) 100Qwen3.6 Max Preview	4Ministral 3 14B 4Mistral Small Creative 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)