Long text (~1594 words), big codex (51 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
GPT-5.1	100%
DeepSeek V4 Pro (Reasoning)	100%
Aion 2.0	100%
Stealth: Healer Alpha	100%
Inception Mercury 2	100%
GPT-5 Nano	100%
GPT-5.4 Nano (Reasoning, Low)	100%
LFM2 24B	100%
Claude Opus 4.6	94%
Ministral 3 8B	93%
GPT-4.1 Nano	93%
Gemini 2.5 Flash Lite (Reasoning)	93%
Xiaomi MIMO v2.5	93%
Nemotron 3 Super	93%
Inception Mercury	93%
Nemotron 3 Nano	93%
ByteDance Seed 1.6 Flash	93%
Grok 4.3 (Reasoning)	92%
GPT-5	92%

	Score	Cost	Time
GPT-4.1 Nano	93%	$0.0003	2.2s
LFM2 24B	100%	$0.0001	1.3s
Ministral 3 8B	93%	$0.0006	1.9s
Cydonia 24B V4.1	71%	$0.0008	4.4s
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0011	6.4s
Inception Mercury	93%	$0.0002	11.6s
ByteDance Seed 1.6 Flash	93%	$0.0010	14.8s
Inception Mercury 2	100%	$0.0039	5.5s
GPT-5.4 Mini (Reasoning, Low)	85%	$0.0048	6.5s
Stealth: Healer Alpha	100%	$0.0000	27.0s
Mistral Small 4 (Reasoning)	92%	$0.0026	21.1s
Gemini 2.5 Flash Lite (Reasoning)	93%	$0.0033	22.0s
Grok 4.1 Fast	92%	$0.0028	27.0s
DeepSeek V4 Flash (Reasoning)	78%	$0.0010	39.3s
Z.AI GLM 5 Turbo	68%	$0.0096	25.3s
Gemini 2.5 Flash (Reasoning)	85%	$0.012	18.1s
GPT-5.2	92%	$0.015	17.4s
Xiaomi MIMO v2.5	93%	$0.011	47.4s
GPT-OSS 120B	85%	$0.0013	1.2m
GPT-5 Nano	100%	$0.0038	1.3m

	Score	Cost	Speed	Stability
LFM2 24B	100%	$0.0001	1.3s	100%
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0011	6.4s	100%
Inception Mercury 2	100%	$0.0039	5.5s	100%
Stealth: Healer Alpha	100%	$0.0000	27.0s	100%
GPT-5 Nano	100%	$0.0038	1.3m	100%
GPT-5.1	100%	$0.029	28.6s	100%
Aion 2.0	100%	$0.0086	1.3m	100%
Ministral 3 8B	93%	$0.0006	1.9s	56%
GPT-4.1 Nano	93%	$0.0003	2.2s	55%
Inception Mercury	93%	$0.0002	11.6s	55%
ByteDance Seed 1.6 Flash	93%	$0.0010	14.8s	55%
Gemini 2.5 Flash Lite (Reasoning)	93%	$0.0033	22.0s	55%
Mistral Small 4 (Reasoning)	92%	$0.0026	21.1s	50%
DeepSeek V4 Pro (Reasoning)	100%	$0.013	4.9m	100%
Grok 4.1 Fast	92%	$0.0028	27.0s	50%
Xiaomi MIMO v2.5	93%	$0.011	47.4s	55%
GPT-5.2	92%	$0.015	17.4s	50%
Claude Opus 4.6	94%	$0.048	16.8s	61%
GPT-5.4 Mini (Reasoning, Low)	85%	$0.0048	6.5s	40%
Nemotron 3 Super	93%	$0.0000	2.3m	55%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
22	Claude Opus 4.6 (Reasoning)	$0.143	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.1	$0.029	28.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	DeepSeek V4 Pro (Reasoning)	$0.013	4.9m	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Aion 2.0	$0.0086	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Stealth: Healer Alpha	$0.0000	27.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Inception Mercury 2	$0.0039	5.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-5 Nano	$0.0038	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
2	GPT-5.4 Nano (Reasoning, Low)	$0.0011	6.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0001	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Claude Opus 4.6	$0.048	16.8s	61%	100	100	100	100	100	100	100	100	100	35	94%
8	Ministral 3 8B	$0.0006	1.9s	56%	100	100	100	100	100	100	100	100	100	26	93%
9	GPT-4.1 Nano	$0.0003	2.2s	55%	100	100	100	100	100	100	100	100	100	26	93%
12	Gemini 2.5 Flash Lite (Reasoning)	$0.0033	22.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
16	Xiaomi MIMO v2.5	$0.011	47.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
20	Nemotron 3 Super	$0.0000	2.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
10	Inception Mercury	$0.0002	11.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
32	Nemotron 3 Nano	$0.0044	5.8m	55%	100	100	100	100	100	100	100	100	100	25	93%
11	ByteDance Seed 1.6 Flash	$0.0010	14.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
21	Grok 4.3 (Reasoning)	$0.018	1.2m	50%	100	100	100	100	100	100	100	100	100	17	92%
31	GPT-5	$0.074	2.5m	50%	100	100	100	100	100	100	100	100	100	17	92%
17	GPT-5.2	$0.015	17.4s	50%	100	100	100	100	100	100	100	100	100	17	92%
15	Grok 4.1 Fast	$0.0028	27.0s	50%	100	100	100	100	100	100	100	100	100	17	92%
13	Mistral Small 4 (Reasoning)	$0.0026	21.1s	50%	100	100	100	100	100	100	100	100	100	17	92%
29	o4 Mini High	$0.045	1.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
25	o4 Mini	$0.021	41.4s	40%	100	100	100	100	100	100	100	100	25	25	85%
23	Gemini 2.5 Flash (Reasoning)	$0.012	18.1s	40%	100	100	100	100	100	100	100	100	25	25	85%
24	GPT-OSS 120B	$0.0013	1.2m	40%	100	100	100	100	100	100	100	100	25	25	85%
19	GPT-5.4 Mini (Reasoning, Low)	$0.0048	6.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
140	Claude Sonnet 4.6 (Reasoning)	$0.221	3.4m	33%	100	100	100	100	100	100	100	100	17	17	83%
28	Z.AI GLM 5	$0.019	1.7m	33%	100	100	100	100	100	100	100	100	17	17	83%
35	Qwen 3.5 Plus (2026-04-20)	$0.027	2.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
26	DeepSeek V4 Flash (Reasoning)	$0.0010	39.3s	31%	100	100	100	100	100	100	100	25	25	25	78%
38	MoonshotAI: Kimi K2.5	$0.019	2.8m	29%	100	100	100	100	100	100	100	25	25	17	77%
45	Z.AI GLM 4.7 Flash	$0.0058	4.4m	29%	100	100	100	100	100	100	100	25	25	17	77%
27	Cydonia 24B V4.1	$0.0008	4.4s	27%	100	100	100	100	100	100	38	38	38	2	71%
30	Z.AI GLM 5 Turbo	$0.0096	25.3s	20%	100	100	100	100	100	100	25	17	17	17	68%
145	MoonshotAI: Kimi K2.6	$0.068	8.4m	18%	100	100	100	100	100	100	17	17	17	17	67%
36	MiniMax M2.7	$0.0042	56.4s	16%	100	100	100	100	100	25	25	25	25	25	63%
34	Qwen 3 32B	$0.0011	28.7s	14%	100	100	100	100	100	25	25	25	25	13	61%
42	Stealth: Hunter Alpha	$0.0000	1.4m	12%	100	100	100	100	100	25	25	17	17	17	60%
33	Arcee AI: Trinity Mini	$0.0004	10.5s	16%	100	100	100	100	50	50	25	25	25	25	60%
39	MiniMax M2.5	$0.0035	36.5s	11%	100	100	100	100	100	25	25	17	17	10	59%
109	Z.AI GLM 5.1	$0.037	3.7m	11%	100	100	100	100	100	25	17	17	17	17	59%
112	Qwen 3.6 27B	$0.045	3.6m	11%	100	100	100	100	100	25	17	17	17	17	59%
56	Xiaomi MIMO v2.5 Pro	$0.021	1.7m	11%	100	100	100	100	100	25	17	17	17	17	59%
50	Claude Sonnet 4.6	$0.044	37.5s	13%	100	100	100	100	50	42	42	29	17	10	59%
54	GPT-5.4 (Reasoning)	$0.040	45.8s	10%	100	100	100	100	100	17	17	17	17	17	58%
37	GPT-5.4 Nano (Reasoning)	$0.0024	15.5s	10%	100	100	100	100	100	17	17	17	17	17	58%
40	Z.AI GLM 4.5 Air	$0.0027	41.1s	10%	100	100	100	100	100	25	25	17	13	4	58%
66	Z.AI GLM 4.6	$0.025	1.4m	8%	100	100	100	100	50	25	25	25	17	0	54%
41	GPT-5.4 Nano	$0.0007	2.5s	7%	100	100	100	100	50	27	17	17	10	2	52%
53	ByteDance Seed 1.6	$0.0060	55.9s	4%	100	100	100	100	25	17	17	17	17	17	51%
60	GPT-5.5 (Reasoning, Low)	$0.026	15.6s	3%	100	100	100	100	17	17	17	17	17	17	50%
76	Grok 4.20 (Reasoning)	$0.025	1.4m	3%	100	100	100	100	17	17	17	17	17	17	50%
88	Qwen 3.6 35B	$0.028	1.9m	7%	100	100	100	25	25	25	25	25	25	17	47%
122	Claude Opus 4.7 (Reasoning)	$0.098	26.9s	7%	100	100	100	25	25	25	25	17	17	17	45%
65	GPT-5 Mini	$0.0098	1.0m	7%	100	100	100	25	25	25	25	17	17	17	45%
44	GPT-4.1	$0.0053	2.7s	14%	100	100	50	42	35	33	33	30	17	8	45%
48	Ministral 8B	$0.0006	9.4s	6%	100	100	100	28	27	26	26	26	1	1	44%
49	Rocinante 12B	$0.0010	7.7s	7%	100	100	100	35	35	25	13	10	5	3	43%
129	Gemma 4 26B (Reasoning)	$0.0032	4.3m	4%	100	100	100	25	17	17	17	17	17	17	43%
81	Grok 4.20 (Beta, Reasoning)	$0.041	25.8s	4%	100	100	100	25	17	17	17	17	17	17	43%
61	GPT-5.4 (Reasoning, Low)	$0.014	10.3s	4%	100	100	100	25	17	17	17	17	17	17	43%
106	GPT-5.5 (Reasoning)	$0.062	31.3s	4%	100	100	100	17	17	17	17	17	17	17	42%
151	Gemini 3.1 Pro (Preview)	$0.203	2.6m	9%	100	100	25	25	25	25	17	17	17	13	36%
47	Claude Opus 4.7	$0.037	5.2s	30%	42	42	42	38	38	35	35	35	35	13	35%
77	Qwen 3.6 Flash	$0.023	1.1m	13%	100	50	25	25	25	25	25	25	25	17	34%
46	Claude Opus 4.5	$0.028	6.0s	30%	50	42	42	42	42	42	38	17	13	8	33%
142	Gemma 4 31B (Reasoning)	$0.0034	5.6m	6%	100	100	17	17	17	17	17	17	17	17	33%
147	Gemini 3 Pro (Preview)	$0.161	1.9m	14%	100	25	25	25	25	25	25	25	25	17	32%
70	Gemini 2.5 Flash Lite	$0.0010	4.5s	0%	100	100	100	4	4	2	2	2	1	1	31%
136	Mistral NeMO	$0.0007	4.1m	0%	100	100	100	4	3	2	2	2	0	0	31%
52	GPT-5.4 Mini	$0.0014	901ms	13%	100	25	25	25	25	25	25	25	17	17	31%
43	Claude 3 Haiku	$0.0014	3.1s	28%	38	38	38	38	38	38	35	13	13	6	29%
72	Hermes 3 405B	$0.0040	6.6s	3%	100	100	17	17	13	10	10	8	6	6	29%
134	Z.AI GLM 4.7	$0.028	3.5m	11%	100	25	25	25	25	17	17	17	17	17	28%
69	Grok 4.20 (Beta)	$0.0087	16.8s	12%	100	31	31	29	27	26	6	3	3	1	26%
148	Qwen3.6 Max Preview	$0.106	5.7m	25%	25	25	25	25	25	25	25	25	25	25	25%
132	Gemini 3.5 Flash (Reasoning)	$0.088	38.1s	8%	100	17	17	17	17	17	17	17	17	17	25%
144	Grok 4	$0.097	2.5m	8%	100	17	17	17	17	17	17	17	17	17	25%
55	Qwen 2.5 72B	$0.0009	31.4s	23%	31	29	29	28	28	28	27	26	10	7	24%
64	GPT-5.4 Mini (Reasoning)	$0.018	19.7s	24%	25	25	25	25	25	25	25	25	25	17	24%
51	Ministral 3B	$0.0002	6.2s	21%	38	32	29	28	28	27	26	26	5	3	24%
68	Writer: Palmyra X5	$0.014	26.5s	21%	42	30	29	27	27	27	27	26	2	2	24%
74	Skyfall 36B V2	$0.0018	6.8s	6%	100	38	32	17	13	13	10	10	3	3	24%
99	WizardLM 2 8x22b	$0.0037	2.2m	21%	35	28	27	27	26	25	25	25	13	1	23%
63	Claude Sonnet 4.5	$0.020	7.8s	24%	35	33	33	32	32	32	8	7	7	6	23%
143	Gemini 2.5 Pro	$0.128	1.5m	23%	25	25	25	25	25	25	25	17	17	17	23%
58	Mistral Small 4	$0.0013	8.3s	19%	50	29	29	28	27	27	26	4	3	2	22%
62	DeepSeek V4 Flash	$0.0007	24.5s	21%	33	28	28	27	27	26	26	26	1	1	22%
79	Z.AI GLM 4.5	$0.0028	18.9s	7%	100	28	25	17	17	13	10	6	4	1	22%
59	Mistral Small Creative	$0.0011	13.5s	21%	27	27	27	26	26	26	26	26	2	1	21%
57	GPT-4.1 Mini	$0.0016	6.2s	20%	29	29	25	25	25	25	25	25	4	2	21%
103	Arcee AI: Trinity Large (Preview)	$0.0000	2.5m	21%	26	26	26	26	26	26	25	25	6	0	21%
111	Qwen 3.5 35B	$0.033	1.7m	23%	25	25	25	25	25	25	17	17	17	13	21%
141	Qwen 3.5 9B	$0.0036	5.8m	23%	25	25	25	25	25	25	17	17	17	13	21%
82	DeepSeek V4 Pro	$0.0073	48.7s	15%	50	35	28	27	25	17	17	8	4	1	21%
75	Claude Haiku 4.5	$0.0062	4.5s	9%	50	38	38	17	13	13	13	10	10	10	21%
67	GPT-4o, Aug. 6th (temp=1)	$0.0092	5.9s	19%	35	33	30	29	25	25	8	7	7	6	21%
150	Qwen 3.5 27B	$0.074	6.4m	14%	38	25	25	17	17	17	17	17	17	17	20%
86	Claude Sonnet 4	$0.018	7.0s	10%	38	35	35	33	17	10	10	8	8	8	20%
92	Qwen3 235B A22B Instruct 2507	$0.0015	1.9m	21%	28	27	27	27	27	27	26	8	3	2	20%
116	Qwen 3.5 Flash	$0.0079	2.2m	15%	25	25	25	25	17	17	17	17	17	17	20%
91	Hermes 3 70B	$0.0013	17.8s	2%	100	50	13	7	6	6	5	5	4	4	20%
95	Llama 3.1 70B	$0.0018	21.1s	3%	100	29	17	17	8	6	5	4	3	3	19%
73	Llama 3.1 8B	$0.0003	54.9s	20%	31	30	28	28	27	26	17	3	3	0	19%
149	Qwen 3.5 122B	$0.099	5.0m	14%	42	17	17	17	17	17	17	17	17	13	19%
138	ByteDance Seed 2.0 Mini	$0.0041	4.3m	15%	35	17	17	17	17	17	17	17	17	17	19%
90	Ministral 3 14B	$0.0029	1.5m	20%	26	26	26	26	25	25	25	2	2	1	18%
100	Cohere Command R+ (Aug. 2024)	$0.013	6.3s	4%	100	17	17	10	8	8	7	7	5	5	18%
71	Ministral 3 3B	$0.0013	35.1s	20%	28	27	27	26	25	25	17	4	2	1	18%
130	Qwen3.7 Max	$0.052	1.4m	16%	25	17	17	17	17	17	17	17	17	17	18%
127	Gemini 3 Flash (Preview, Reasoning)	$0.043	1.2m	12%	38	25	17	17	17	13	13	13	13	10	17%
84	Llama 3.1 Nemotron 70B	$0.0057	28.1s	13%	31	30	28	28	28	6	6	5	5	5	17%
146	Qwen 3.5 397B A17B	$0.029	5.6m	16%	25	17	17	17	17	17	17	17	17	13	17%
135	Claude Opus 4	$0.087	10.8s	9%	38	35	31	13	13	10	8	8	8	5	17%
102	ByteDance Seed 2.0 Lite	$0.0079	1.4m	16%	25	17	17	17	17	17	17	17	13	13	17%
78	GPT-5.5	$0.015	3.6s	15%	25	25	17	17	17	17	13	13	13	8	16%
97	GPT-4o Mini (temp=1)	$0.0007	9.6s	4%	38	31	31	30	6	5	4	4	4	4	16%
105	GPT-4o, May 13th (temp=1)	$0.025	3.9s	9%	35	33	31	13	13	10	7	6	4	3	15%
137	Gemma 4 26B	$0.0049	3.2m	6%	42	31	31	13	8	7	6	6	6	6	15%
80	Grok 4 Fast	$0.0029	33.5s	16%	17	17	17	17	17	17	17	13	13	13	15%
101	Grok 4.3	$0.0062	6.8s	2%	100	8	8	7	6	5	5	5	3	2	15%
85	DeepSeek V3 (2024-12-26)	$0.0018	11.1s	10%	30	25	25	17	13	13	10	5	4	2	14%
98	Gemma 4 31B	$0.0007	26.8s	8%	35	33	13	13	10	10	8	8	6	6	14%
83	GPT-5.4	$0.0053	3.8s	12%	17	17	17	17	13	13	13	13	10	10	14%
115	Claude 3.5 Sonnet	$0.034	8.0s	8%	35	17	17	13	10	10	10	10	8	7	14%
108	Mistral Medium 3.1	$0.0043	12.9s	3%	28	28	28	27	3	3	3	3	2	2	13%
114	DeepSeek V3.1	$0.0025	56.9s	6%	31	26	25	8	8	6	5	5	4	4	12%
89	Gemini 3.5 Flash (Reasoning, Minimal)	$0.011	3.2s	12%	13	13	13	13	13	13	13	13	10	8	12%
96	Gemini 3.1 Flash Lite	$0.0017	6.0s	8%	35	10	10	10	10	8	8	8	7	7	11%
94	Gemini 3.1 Flash Lite (Reasoning)	$0.0017	2.7s	8%	31	10	10	10	10	8	8	8	8	6	11%
87	Gemini 2.5 Flash	$0.0021	2.0s	10%	17	17	13	13	13	10	8	7	5	5	11%
93	Gemini 3.1 Flash Lite (Preview)	$0.0017	2.2s	10%	13	10	10	10	10	10	10	8	8	5	9%
104	Gemini 3 Flash (Preview)	$0.0031	4.9s	6%	33	8	7	7	7	7	5	5	5	3	9%
107	DeepSeek V3 (2025-03-24)	$0.0011	19.7s	7%	13	13	10	10	8	7	7	6	6	4	8%
110	DeepSeek-V2 Chat	$0.0018	12.6s	5%	25	17	8	6	6	6	6	5	3	2	8%
131	DeepSeek V3.2	$0.0016	1.5m	3%	26	26	6	5	3	3	3	3	2	1	8%
124	Grok 4.20	$0.011	29.2s	2%	33	27	3	3	3	3	2	2	1	1	8%
113	Claude 3.7 Sonnet	$0.019	6.4s	8%	10	8	8	8	8	8	7	7	6	5	8%
117	GPT-4o, Aug. 6th (temp=0)	$0.011	4.7s	5%	7	7	6	6	6	6	5	4	4	4	5%
139	Qwen 3.5 Plus (2026-02-15)	$0.018	1.9m	5%	8	7	6	6	6	5	4	4	4	1	5%
123	Mistral Large 2	$0.013	12.9s	4%	8	6	6	6	5	4	4	3	3	2	5%
118	Gemma 3 27B	$0.0005	18.5s	4%	6	6	6	5	5	5	4	4	3	2	4%
119	Gemma 3 12B	$0.0006	16.9s	3%	13	5	4	4	4	4	3	3	3	2	4%
121	Mistral Large 3	$0.0034	14.1s	3%	7	6	5	5	4	3	3	3	3	2	4%
128	Mistral Large	$0.015	17.6s	3%	7	4	3	3	3	3	3	2	2	2	3%
133	GPT-4o, May 13th (temp=0)	$0.039	8.0s	2%	5	5	4	4	3	2	2	2	1	1	3%
120	Mistral Small 3.2 24B	$0.0007	7.5s	3%	4	4	3	3	3	3	3	2	2	1	3%
126	GPT-4o Mini (temp=0)	$0.0016	38.4s	3%	4	3	3	3	3	3	3	2	2	0	3%
125	Gemma 3 4B	$0.0003	23.6s	1%	1	1	1	1	1	1	1	1	1	1	1%
41.49%

Median	Evaluator	Top 3	Flop 3
30.0%	Correct "no violations" response	100GPT-5.1 100GPT-5.4 Nano (Reasoning, Low) 100Inception Mercury 2	0Gemini 3.5 Flash (Reasoning, Minimal) 0Qwen3.6 Max Preview 0GPT-4o Mini (temp=0)
40.0%	No hallucinated violations	100LFM2 24B 100Inception Mercury 2 100GPT-5 Nano	2Ministral 3 14B 2Gemma 3 4B 3Arcee AI: Trinity Large (Preview)

Codex Red Herring (False Positive Detection)

Long text (~1594 words), big codex (51 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)