Short text (~524 words), big codex (51 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Qwen3.7 Max	100%
Claude Opus 4.6 (Reasoning)	100%
Qwen3.6 Max Preview	100%
Gemini 3.1 Pro (Preview)	100%
Z.AI GLM 5.1	100%
Z.AI GLM 5 Turbo	100%
Gemini 3.5 Flash (Reasoning)	100%
Claude Sonnet 4.6 (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5 Mini	100%
Claude Opus 4.6	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 26B (Reasoning)	100%
Grok 4.20 (Reasoning)	100%
ByteDance Seed 1.6	100%
o4 Mini High	100%
Qwen 3.6 27B	100%
Grok 4.1 Fast	100%
GPT-5.5	100%
GPT-4.1	100%

	Score	Cost	Time
LFM2 24B	100%	$0.0001	2.4s
Ministral 8B	68%	$0.0003	3.4s
Gemma 4 26B	95%	$0.0003	2.8s
Ministral 3 8B	93%	$0.0004	1.2s
GPT-5.4 Nano	73%	$0.0004	1.4s
Gemma 4 31B	100%	$0.0004	1.7s
Inception Mercury	93%	$0.0002	5.4s
GPT-5.4 Nano (Reasoning, Low)	90%	$0.0006	3.2s
GPT-4.1	100%	$0.0020	737ms
ByteDance Seed 1.6 Flash	100%	$0.0005	6.7s
GPT-5.4 Nano (Reasoning)	100%	$0.0010	5.1s
Arcee AI: Trinity Mini	83%	$0.0003	15.8s
Grok 4.1 Fast	100%	$0.0012	9.6s
Gemini 2.5 Flash Lite (Reasoning)	100%	$0.0011	6.6s
Hermes 3 405B	75%	$0.0026	1.7s
Qwen 3 32B	93%	$0.0004	10.6s
Inception Mercury 2	100%	$0.0024	3.6s
GPT-5.4 Mini (Reasoning, Low)	85%	$0.0023	3.5s
DeepSeek V4 Flash (Reasoning)	85%	$0.0005	18.9s
GPT-5.5	100%	$0.0069	1.6s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
23	Qwen3.7 Max	$0.031	48.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Claude Opus 4.6 (Reasoning)	$0.041	24.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
40	Qwen3.6 Max Preview	$0.046	2.7m	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Gemini 3.1 Pro (Preview)	$0.045	32.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Z.AI GLM 5.1	$0.013	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Z.AI GLM 5 Turbo	$0.0034	7.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Gemini 3.5 Flash (Reasoning)	$0.040	17.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Claude Sonnet 4.6 (Reasoning)	$0.044	35.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Claude Opus 4.7 (Reasoning)	$0.019	997ms	100%	100	100	100	100	100	100	100	100	100	100	100%
12	GPT-5 Mini	$0.0044	28.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Claude Opus 4.6	$0.031	12.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Qwen 3.5 Plus (2026-04-20)	$0.011	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Gemma 4 26B (Reasoning)	$0.0015	1.9m	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Grok 4.20 (Reasoning)	$0.0095	31.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	ByteDance Seed 1.6	$0.0021	16.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	o4 Mini High	$0.020	41.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Qwen 3.6 27B	$0.022	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Grok 4.1 Fast	$0.0012	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5.5	$0.0069	1.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	GPT-4.1	$0.0020	737ms	100%	100	100	100	100	100	100	100	100	100	100	100%
13	o4 Mini	$0.010	20.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Gemma 4 31B	$0.0004	1.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Gemini 2.5 Flash Lite (Reasoning)	$0.0011	6.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Nemotron 3 Super	$0.0000	51.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Inception Mercury 2	$0.0024	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning)	$0.0010	5.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	ByteDance Seed 1.6 Flash	$0.0005	6.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0001	2.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Gemma 4 26B	$0.0003	2.8s	70%	100	100	100	100	100	100	100	100	100	50	95%
29	Ministral 3 8B	$0.0004	1.2s	56%	100	100	100	100	100	100	100	100	100	27	93%
36	GPT-5.1	$0.014	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
48	Gemma 4 31B (Reasoning)	$0.0018	2.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Qwen 3.6 Flash	$0.011	31.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
34	GPT-5.4 Mini (Reasoning)	$0.0056	7.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
39	Aion 2.0	$0.0053	49.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
35	MiniMax M2.7	$0.0022	26.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
33	Gemini 2.5 Flash (Reasoning)	$0.0047	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
32	Mistral Small 4 (Reasoning)	$0.0015	12.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
31	Qwen 3 32B	$0.0004	10.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
30	Inception Mercury	$0.0002	5.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
41	Grok 4.3 (Reasoning)	$0.010	42.7s	50%	100	100	100	100	100	100	100	100	100	17	92%
43	Grok 4.20 (Beta, Reasoning)	$0.021	14.0s	50%	100	100	100	100	100	100	100	100	100	17	92%
46	ByteDance Seed 2.0 Mini	$0.0021	2.0m	50%	100	100	100	100	100	100	100	100	100	17	92%
38	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.2s	42%	100	100	100	100	100	100	100	100	100	4	90%
42	Z.AI GLM 4.5 Air	$0.0012	21.5s	44%	100	100	100	100	100	100	100	100	30	30	86%
52	Qwen 3.6 35B	$0.014	51.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
45	DeepSeek V4 Flash (Reasoning)	$0.0005	18.9s	40%	100	100	100	100	100	100	100	100	25	25	85%
75	Gemini 2.5 Pro	$0.080	58.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
49	Xiaomi MIMO v2.5 Pro	$0.0092	43.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
44	GPT-5.4 Mini (Reasoning, Low)	$0.0023	3.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
61	Z.AI GLM 4.7 Flash	$0.0034	2.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
62	Nemotron 3 Nano	$0.0021	2.3m	37%	100	100	100	100	100	100	100	100	25	17	84%
47	Arcee AI: Trinity Mini	$0.0003	15.8s	31%	100	100	100	100	100	100	100	100	25	5	83%
67	Z.AI GLM 5	$0.015	1.7m	31%	100	100	100	100	100	100	100	25	25	25	78%
146	Gemini 3 Pro (Preview)	$0.140	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
53	Xiaomi MIMO v2.5	$0.0040	17.3s	31%	100	100	100	100	100	100	100	25	25	25	78%
60	GPT-5 Nano	$0.0040	1.3m	31%	100	100	100	100	100	100	100	25	25	25	78%
58	GPT-5.2	$0.0087	9.4s	24%	100	100	100	100	100	100	100	25	17	10	75%
55	Hermes 3 405B	$0.0026	1.7s	24%	100	100	100	100	100	100	100	25	17	10	75%
51	Claude Sonnet 4	$0.0096	2.1s	37%	100	100	100	100	100	100	50	42	42	17	75%
54	Claude Opus 4.7	$0.020	2.0s	38%	100	100	100	100	100	50	50	50	50	50	75%
50	GPT-5.4 Nano	$0.0004	1.4s	34%	100	100	100	100	100	100	50	31	28	25	73%
87	Qwen 3.5 9B	$0.0031	3.9m	32%	100	100	100	100	100	50	50	50	50	25	73%
57	MiniMax M2.5	$0.0016	17.5s	27%	100	100	100	100	100	100	25	25	25	25	70%
64	GPT-OSS 120B	$0.0006	52.6s	27%	100	100	100	100	100	100	25	25	25	25	70%
56	Stealth: Healer Alpha	$0.0000	19.6s	27%	100	100	100	100	100	100	25	25	25	25	70%
59	Ministral 8B	$0.0003	3.4s	21%	100	100	100	100	100	100	28	27	26	3	68%
66	Cohere Command R+ (Aug. 2024)	$0.0080	3.2s	14%	100	100	100	100	100	100	25	13	10	5	65%
65	Hermes 3 70B	$0.0008	7.2s	16%	100	100	100	100	100	42	38	35	8	7	63%
82	Z.AI GLM 4.6	$0.017	48.9s	11%	100	100	100	100	100	50	28	13	4	0	59%
137	Grok 4	$0.056	1.3m	11%	100	100	100	100	100	25	17	17	17	17	59%
149	MoonshotAI: Kimi K2.6	$0.041	5.8m	7%	100	100	100	100	25	25	25	25	25	25	55%
138	GPT-5	$0.042	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
148	Qwen 3.5 397B A17B	$0.022	4.5m	7%	100	100	100	100	25	25	25	25	25	25	55%
115	MoonshotAI: Kimi K2.5	$0.013	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
113	DeepSeek V4 Pro (Reasoning)	$0.0078	1.8m	7%	100	100	100	100	25	25	25	25	25	25	55%
112	Z.AI GLM 4.7	$0.016	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
96	Gemini 3 Flash (Preview, Reasoning)	$0.021	38.5s	6%	100	100	100	100	25	25	25	25	25	17	54%
63	Claude Haiku 4.5	$0.0046	5.1s	35%	100	50	50	50	50	50	50	50	50	42	54%
90	Claude 3.5 Sonnet	$0.020	3.7s	5%	100	100	100	100	25	25	25	17	17	13	52%
150	Qwen 3.5 27B	$0.075	6.6m	11%	100	100	100	42	42	25	25	25	25	17	50%
68	DeepSeek V4 Pro	$0.0018	27.1s	19%	100	100	50	50	50	42	35	29	25	6	49%
73	GPT-4o, Aug. 6th (temp=1)	$0.0048	2.0s	11%	100	100	100	42	38	33	25	25	13	10	49%
78	Cydonia 24B V4.1	$0.0007	7.4s	4%	100	100	100	100	31	30	8	4	3	3	48%
117	GPT-5.5 (Reasoning)	$0.033	17.7s	8%	100	100	100	25	25	25	25	25	25	25	48%
92	Stealth: Hunter Alpha	$0.0000	55.9s	7%	100	100	100	25	25	25	25	25	25	17	47%
79	Grok 4 Fast	$0.0014	19.2s	7%	100	100	100	25	25	25	25	25	25	17	47%
151	Qwen 3.5 122B	$0.111	6.2m	7%	100	100	100	25	25	25	25	17	17	17	45%
81	Z.AI GLM 4.5	$0.0019	11.5s	7%	100	100	100	38	29	25	25	25	4	2	45%
69	WizardLM 2 8x22b	$0.0018	23.8s	24%	100	50	50	50	42	42	25	25	25	25	43%
91	GPT-4.1 Nano	$0.0005	6.5s	1%	100	100	100	100	25	2	1	1	1	1	43%
88	Skyfall 36B V2	$0.0010	4.9s	4%	100	100	100	50	30	13	8	7	6	6	42%
140	Qwen 3.5 35B	$0.026	1.5m	10%	100	100	25	25	25	25	25	25	25	17	39%
86	Writer: Palmyra X5	$0.0026	4.4s	9%	100	100	50	29	25	25	17	17	17	13	39%
102	Claude 3.7 Sonnet	$0.011	3.1s	6%	100	100	42	17	17	17	17	17	17	17	36%
80	Qwen 2.5 72B	$0.0005	9.4s	16%	100	42	35	31	31	30	29	29	13	13	35%
77	Claude Sonnet 4.6	$0.014	7.6s	29%	42	38	38	35	35	33	33	33	32	13	33%
99	Claude Opus 4.5	$0.016	2.9s	14%	100	25	25	25	25	25	25	25	25	25	33%
133	Qwen 3.5 Flash	$0.0062	1.8m	14%	100	25	25	25	25	25	25	25	25	25	33%
108	ByteDance Seed 2.0 Lite	$0.0049	49.0s	14%	100	25	25	25	25	25	25	25	25	25	33%
85	GPT-5.4 Mini	$0.0011	683ms	14%	100	25	25	25	25	25	25	25	25	25	33%
70	Mistral Medium 3.1	$0.0020	3.6s	29%	38	33	33	31	31	30	30	30	30	30	31%
105	Grok 4.3	$0.0031	3.0s	4%	100	100	28	17	17	13	13	10	7	4	31%
130	GPT-5.4 (Reasoning)	$0.027	27.0s	13%	100	25	25	25	25	25	25	17	17	17	30%
74	Mistral Small Creative	$0.0005	4.3s	26%	42	30	29	29	29	29	29	28	28	28	30%
72	Ministral 3 3B	$0.0003	3.0s	28%	32	32	32	31	30	29	28	28	28	28	30%
71	Ministral 3B	$0.0001	2.9s	29%	32	31	30	30	30	29	29	28	28	27	29%
98	Claude Sonnet 4.5	$0.010	3.1s	13%	100	25	25	25	25	25	17	17	17	17	29%
76	GPT-4.1 Mini	$0.0013	5.4s	26%	33	32	32	31	31	31	30	30	30	6	29%
89	Arcee AI: Trinity Large (Preview)	$0.0000	43.4s	27%	29	28	28	28	28	28	28	28	27	26	28%
135	Claude Opus 4	$0.049	4.0s	21%	50	25	25	25	25	25	25	25	25	25	28%
120	GPT-5.4 (Reasoning, Low)	$0.012	10.6s	9%	100	25	25	25	17	17	17	17	17	17	27%
95	Qwen3 235B A22B Instruct 2507	$0.0004	5.9s	11%	100	25	25	25	25	17	17	17	13	13	27%
83	DeepSeek V4 Flash	$0.0003	8.7s	22%	50	30	30	30	28	28	28	28	10	8	27%
109	Grok 4.20 (Beta)	$0.0041	2.6s	8%	100	38	30	30	28	10	7	5	5	3	26%
106	GPT-5.5 (Reasoning, Low)	$0.022	13.1s	25%	25	25	25	25	25	25	25	25	25	25	25%
84	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0049	1.3s	25%	25	25	25	25	25	25	25	25	25	25	25%
122	Rocinante 12B	$0.0011	20.4s	5%	100	50	25	17	13	10	10	8	7	0	24%
145	Qwen 3.5 Plus (2026-02-15)	$0.022	2.2m	20%	42	42	33	30	29	26	10	10	8	3	23%
93	DeepSeek V3 (2024-12-26)	$0.0010	4.1s	19%	50	25	25	25	25	25	25	13	7	6	23%
94	Ministral 3 14B	$0.0010	17.5s	22%	28	28	28	27	27	27	27	25	3	2	22%
100	DeepSeek V3.1	$0.0009	16.3s	17%	38	30	30	25	25	17	17	17	13	6	22%
127	Llama 3.1 70B	$0.0014	14.4s	3%	100	32	31	10	8	6	6	6	5	4	21%
97	DeepSeek V3.2	$0.0007	12.0s	17%	33	30	25	25	25	17	17	17	8	8	20%
103	Gemini 3.1 Flash Lite (Preview)	$0.0010	1.4s	14%	42	35	17	17	17	17	17	17	13	13	20%
101	Llama 3.1 Nemotron 70B	$0.0039	22.2s	22%	31	31	30	30	29	29	8	5	5	3	20%
126	Mistral NeMO	$0.0005	4.7s	2%	100	42	30	6	5	5	4	3	2	1	20%
114	GPT-4o, Aug. 6th (temp=0)	$0.0052	2.6s	15%	33	25	25	17	17	17	17	17	13	13	19%
104	Llama 3.1 8B	$0.0002	25.8s	20%	30	29	29	28	27	26	8	7	3	1	19%
119	Mistral Large 3	$0.0022	8.7s	13%	32	32	32	32	30	5	5	5	4	4	18%
107	Gemini 3.1 Flash Lite	$0.0010	1.4s	14%	38	17	17	17	17	17	17	17	13	10	18%
124	Mistral Large 2	$0.0093	9.4s	14%	33	31	30	29	29	8	4	4	4	3	17%
118	Mistral Small 4	$0.0007	9.2s	13%	31	31	30	30	28	6	5	4	4	3	17%
110	Gemini 3.1 Flash Lite (Reasoning)	$0.0011	2.6s	15%	32	17	17	17	17	17	17	13	13	13	17%
111	Gemini 3 Flash (Preview)	$0.0012	2.0s	16%	25	17	17	17	17	17	17	13	13	13	16%
116	GPT-5.4	$0.0039	2.7s	16%	17	17	17	17	17	17	17	17	17	13	16%
121	Gemini 2.5 Flash	$0.0014	1.5s	11%	29	25	17	17	13	13	13	10	10	8	15%
129	Claude 3 Haiku	$0.0012	3.6s	5%	33	33	32	10	7	7	7	7	6	5	15%
134	GPT-4o, May 13th (temp=1)	$0.016	2.2s	11%	31	17	17	17	13	13	10	10	10	10	15%
123	DeepSeek V3 (2025-03-24)	$0.0006	8.0s	11%	25	25	17	13	13	13	13	10	7	7	14%
132	GPT-4o Mini (temp=0)	$0.0006	8.7s	6%	32	32	26	7	7	7	6	6	6	5	13%
128	GPT-4o Mini (temp=1)	$0.0004	5.3s	7%	38	33	13	10	10	8	8	6	5	4	13%
125	Grok 4.20	$0.0030	5.5s	13%	25	17	17	17	17	13	10	8	3	2	13%
131	DeepSeek-V2 Chat	$0.0012	6.8s	9%	25	17	13	13	10	10	8	6	6	5	11%
143	Mistral Large	$0.0088	11.3s	5%	32	29	7	7	7	6	6	4	4	4	11%
136	Mistral Small 3.2 24B	$0.0004	4.0s	5%	29	7	7	7	6	6	6	6	6	5	9%
147	GPT-4o, May 13th (temp=0)	$0.040	11.9s	8%	26	13	13	10	10	10	2	1	1	1	8%
139	Gemma 3 12B	$0.0003	13.1s	5%	29	7	7	6	6	6	6	5	5	2	8%
141	Gemini 2.5 Flash Lite	$0.0004	1.7s	4%	10	10	5	5	5	4	4	4	4	4	5%
142	Gemma 3 27B	$0.0003	11.4s	5%	6	6	6	6	5	5	5	5	5	5	5%
144	Gemma 3 4B	$0.0002	13.0s	2%	5	5	5	3	3	2	2	1	1	1	3%
57.36%

Median	Evaluator	Top 3	Flop 3
50.0%	Correct "no violations" response	100Claude Opus 4.7 (Reasoning) 100Nemotron 3 Super 100GPT-5.5	0GPT-5.5 (Reasoning, Low) 0Gemma 3 4B 0GPT-5.4
65.0%	No hallucinated violations	100Claude Sonnet 4.6 (Reasoning) 100GPT-5 Mini 100Z.AI GLM 5.1	4Ministral 3 14B 5Arcee AI: Trinity Large (Preview) 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)