Short text (~524 words), big codex (51 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Qwen3.7 Max	100%
GPT-5.4 (Reasoning)	100%
Z.AI GLM 5.2 (Reasoning, High)	100%
Z.AI GLM 5 Turbo	100%
Claude Opus 4.8 (Reasoning)	100%
Claude Opus 4.6	100%
GPT-5 Mini	100%
Grok 4.5 (Reasoning, Low)	100%
Claude Sonnet 4.6	100%
GPT-5.6 Luna (Reasoning)	100%
GPT-5.4 Mini (Reasoning)	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 31B (Reasoning)	100%
Claude Opus 4.5	100%
ByteDance Seed 1.6	100%
GPT-5.2	100%
Gemma 4 26B (Reasoning)	100%
o4 Mini High	100%
DeepSeek V4 Flash (Reasoning)	100%
GPT-4.1	100%

	Score	Cost	Time
GPT-5.4 Nano	74%	$0.0006	1.4s
GPT-5.4 Nano (Reasoning, Low)	93%	$0.0008	3.4s
Ministral 8B	100%	$0.0013	564ms
Gemma 4 26B	88%	$0.0012	5.3s
Arcee AI: Trinity Mini	100%	$0.0007	6.9s
Gemma 4 31B	100%	$0.0018	3.1s
Ministral 3 8B	100%	$0.0020	589ms
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0038	2.7s
ByteDance Seed 1.6 Flash	93%	$0.0013	7.7s
Inception Mercury 2	100%	$0.0031	3.7s
GPT-5.4 Nano (Reasoning)	100%	$0.0020	8.1s
DeepSeek V4 Flash (Reasoning)	100%	$0.0009	19.7s
GPT-5.4 Mini (Reasoning)	100%	$0.0063	5.2s
Gemini 2.5 Flash Lite (Reasoning)	85%	$0.0023	14.2s
Mistral Small 4 (Reasoning)	93%	$0.0028	23.8s
GPT-4.1	100%	$0.0086	1.0s
Mistral Large 3	100%	$0.0066	1.4s
GPT-5.6 Luna (Reasoning)	100%	$0.0075	5.2s
Z.AI GLM 4.6	90%	$0.0053	12.9s
GPT-5.6 Terra (Reasoning)	85%	$0.010	4.3s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
34	Qwen3.7 Max	$0.064	58.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	GPT-5.4 (Reasoning)	$0.032	23.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Z.AI GLM 5.2 (Reasoning, High)	$0.0063	15.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Z.AI GLM 5 Turbo	$0.0077	13.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
38	Claude Opus 4.8 (Reasoning)	$0.122	14.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Claude Opus 4.6	$0.082	9.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	GPT-5 Mini	$0.0049	30.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Grok 4.5 (Reasoning, Low)	$0.013	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Claude Sonnet 4.6	$0.043	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	GPT-5.6 Luna (Reasoning)	$0.0075	5.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	GPT-5.4 Mini (Reasoning)	$0.0063	5.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Qwen 3.5 Plus (2026-04-20)	$0.017	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
32	Gemma 4 31B (Reasoning)	$0.0030	2.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
31	Claude Opus 4.5	$0.072	2.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	ByteDance Seed 1.6	$0.0050	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	GPT-5.2	$0.014	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Gemma 4 26B (Reasoning)	$0.0029	2.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
25	o4 Mini High	$0.020	33.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	DeepSeek V4 Flash (Reasoning)	$0.0009	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-4.1	$0.0086	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	o4 Mini	$0.012	20.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	MiniMax M2.7	$0.0045	17.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Xiaomi MIMO v2.5 Pro	$0.013	49.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Gemma 4 31B	$0.0018	3.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	GPT-OSS 120B	$0.0019	59.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Mistral Large 3	$0.0066	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.4 Mini (Reasoning, Low)	$0.0038	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Inception Mercury 2	$0.0031	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	Nemotron 3 Super	$0.0000	51.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Mistral Large 2	$0.026	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	GPT-5 Nano	$0.0020	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Nano (Reasoning)	$0.0020	8.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0020	589ms	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Arcee AI: Trinity Mini	$0.0007	6.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0013	564ms	100%	100	100	100	100	100	100	100	100	100	100	100%
35	GPT-5.4 Nano (Reasoning, Low)	$0.0008	3.4s	57%	100	100	100	100	100	100	100	100	100	29	93%
63	Qwen3.6 Max Preview	$0.059	2.5m	55%	100	100	100	100	100	100	100	100	100	25	93%
61	Claude Opus 4.7 (Reasoning)	$0.119	10.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
45	Grok 4.3 (Reasoning)	$0.018	30.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
42	GPT-5.4 (Reasoning, Low)	$0.017	6.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
49	Z.AI GLM 5	$0.020	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
56	Gemini 2.5 Pro	$0.067	45.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
41	MiniMax M2.5	$0.0049	21.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
47	Aion 2.0	$0.014	1.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
44	Z.AI GLM 4.5	$0.0079	42.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
39	Gemini 2.5 Flash (Reasoning)	$0.0092	11.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
51	Z.AI GLM 4.7 Flash	$0.0036	2.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
40	Mistral Small 4 (Reasoning)	$0.0028	23.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
36	ByteDance Seed 1.6 Flash	$0.0013	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Z.AI GLM 4.6	$0.0053	12.9s	60%	100	100	100	100	100	100	100	100	50	50	90%
46	Aion 3.0 Mini	$0.013	1.3m	60%	100	100	100	100	100	100	100	100	50	50	90%
43	Gemma 4 26B	$0.0012	5.3s	49%	100	100	100	100	100	100	100	100	50	25	88%
57	Grok 4.5 (Reasoning, High)	$0.025	31.7s	40%	100	100	100	100	100	100	100	100	25	25	85%
70	Z.AI GLM 5.1	$0.031	2.6m	40%	100	100	100	100	100	100	100	100	25	25	85%
50	GPT-5.6 Terra (Reasoning)	$0.010	4.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
76	Claude Opus 4.8 (Reasoning, Low)	$0.121	13.9s	40%	100	100	100	100	100	100	100	100	25	25	85%
53	GPT-5.1	$0.025	23.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
54	MiniMax M3	$0.0037	1.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
144	Qwen 3.5 122B	$0.138	7.0m	40%	100	100	100	100	100	100	100	100	25	25	85%
58	Gemini 3 Flash (Preview, Reasoning)	$0.025	40.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
68	DeepSeek V4 Pro (Reasoning)	$0.023	2.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
67	Qwen 3.6 27B	$0.032	2.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
48	Gemini 2.5 Flash Lite (Reasoning)	$0.0023	14.2s	40%	100	100	100	100	100	100	100	100	25	25	85%
55	Z.AI GLM 4.5 Air	$0.0033	26.9s	32%	100	100	100	100	100	100	100	100	30	3	83%
79	Qwen 3.5 9B	$0.0044	4.4m	45%	100	100	100	100	100	100	100	50	50	25	83%
64	Qwen 3.6 35B	$0.018	1.1m	31%	100	100	100	100	100	100	100	25	25	25	78%
75	Qwen 3.5 35B	$0.032	1.8m	31%	100	100	100	100	100	100	100	25	25	25	78%
62	ByteDance Seed 2.0 Lite	$0.0085	1.0m	31%	100	100	100	100	100	100	100	25	25	25	78%
59	Xiaomi MIMO v2.5	$0.0056	19.2s	31%	100	100	100	100	100	100	100	25	25	25	78%
65	Nemotron 3 Nano	$0.0022	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
88	MoonshotAI: Kimi K2.6	$0.038	2.5m	29%	100	100	100	100	100	100	100	25	25	17	77%
82	Z.AI GLM 4.7	$0.029	2.4m	29%	100	100	100	100	100	100	100	25	25	17	77%
52	GPT-5.4 Nano	$0.0006	1.4s	37%	100	100	100	100	100	100	42	38	38	28	74%
71	Cohere Command R+ (Aug. 2024)	$0.037	14.6s	18%	100	100	100	100	100	100	100	17	17	1	73%
60	Gemini 3.5 Flash (Reasoning, Minimal)	$0.020	1.5s	31%	100	100	100	100	100	100	50	25	25	25	73%
73	Qwen 3.5 Flash	$0.0077	2.2m	31%	100	100	100	100	100	100	50	25	25	25	73%
106	Claude Opus 4.6 (Reasoning)	$0.117	33.8s	27%	100	100	100	100	100	100	25	25	25	25	70%
72	GPT-5	$0.033	46.9s	27%	100	100	100	100	100	100	25	25	25	25	70%
89	MoonshotAI: Kimi K2.5	$0.018	2.7m	27%	100	100	100	100	100	100	25	25	25	25	70%
66	GPT-5.6 Sol	$0.027	1.6s	27%	100	100	100	100	100	100	25	25	25	25	70%
78	Claude Sonnet 5 (Reasoning, Low)	$0.059	24.7s	27%	100	100	100	100	100	100	25	25	25	25	70%
140	Qwen 3.5 27B	$0.060	4.9m	27%	100	100	100	100	100	100	25	25	25	25	70%
125	ByteDance Seed 2.0 Mini	$0.0050	4.5m	22%	100	100	100	100	100	100	25	25	17	17	68%
69	Grok 4.3	$0.016	1.8s	18%	100	100	100	100	100	100	33	13	13	10	67%
139	Aion 3.0	$0.084	2.5m	14%	100	100	100	100	100	25	25	25	25	17	62%
90	Grok 4.20 (Reasoning)	$0.023	59.7s	13%	100	100	100	100	100	25	25	25	17	17	61%
74	GPT-4.1 Mini	$0.0025	3.5s	13%	100	100	100	100	50	31	31	29	25	25	59%
77	Mistral Medium 3.1	$0.0058	3.8s	11%	100	100	100	100	50	30	29	29	17	17	57%
120	Gemini 3.5 Flash (Reasoning)	$0.061	23.1s	7%	100	100	100	100	25	25	25	25	25	25	55%
93	Qwen 3.6 Flash	$0.013	32.4s	7%	100	100	100	100	25	25	25	25	25	25	55%
81	GPT-5.4 Mini	$0.0039	834ms	7%	100	100	100	100	25	25	25	25	25	25	55%
86	Qwen 3 32B	$0.0024	24.4s	7%	100	100	100	100	25	25	25	25	25	25	55%
143	Qwen 3.5 397B A17B	$0.032	5.1m	6%	100	100	100	100	25	25	25	25	25	17	54%
119	Claude Sonnet 5 (Reasoning)	$0.056	22.3s	6%	100	100	100	100	25	25	25	25	25	17	54%
84	Cydonia 24B V4.1	$0.0029	8.3s	7%	100	100	100	100	42	31	28	10	7	3	52%
91	DeepSeek V4 Pro	$0.014	36.5s	13%	100	100	100	50	50	29	28	25	17	17	51%
85	Claude Opus 4.7	$0.100	3.7s	50%	50	50	50	50	50	50	50	50	50	50	50%
136	Gemini 3.1 Pro (Preview)	$0.078	55.6s	8%	100	100	100	25	25	25	25	25	25	25	48%
87	Qwen 2.5 72B	$0.0017	10.6s	13%	100	100	42	38	32	31	29	28	25	17	44%
118	WizardLM 2 8x22b	$0.0087	1.2m	10%	100	100	50	42	28	25	25	25	17	1	41%
142	Claude Sonnet 4.6 (Reasoning)	$0.110	59.4s	10%	100	100	25	25	25	25	25	25	25	17	39%
130	Claude Sonnet 4	$0.046	4.9s	2%	100	100	100	17	13	13	10	10	10	10	38%
80	Claude Haiku 4.5	$0.016	4.7s	32%	50	50	50	42	42	42	38	25	25	17	38%
97	GPT-4.1 Nano	$0.0009	8.8s	9%	100	100	42	27	27	27	26	26	1	1	38%
102	Qwen3 235B A22B Instruct 2507	$0.0018	16.2s	8%	100	100	50	25	25	25	17	17	8	3	37%
101	DeepSeek V3 (2025-03-24)	$0.0032	5.3s	9%	100	100	25	25	25	25	17	17	13	8	35%
110	Gemini 2.5 Flash Lite	$0.0008	2.2s	2%	100	100	100	13	13	13	4	2	2	2	35%
145	Claude Opus 4	$0.225	6.0s	8%	100	50	50	50	17	17	17	17	17	13	35%
121	Ministral 3 14B	$0.0044	1.4m	15%	100	28	28	28	27	26	26	26	25	25	34%
83	DeepSeek V4 Flash	$0.0007	7.4s	27%	50	42	38	38	35	30	27	25	25	25	33%
98	GPT-5.6 Luna	$0.0040	1.3s	14%	100	25	25	25	25	25	25	25	25	25	33%
117	Mistral Small 3.2 24B	$0.0014	3.4s	1%	100	100	38	38	6	6	5	5	5	5	31%
134	Qwen 3.5 Plus (2026-02-15)	$0.023	1.9m	20%	50	50	50	42	33	25	25	17	7	6	31%
109	GPT-5.6 Sol (Reasoning)	$0.029	6.2s	25%	25	25	25	25	25	25	25	25	25	25	25%
114	GPT-5.5 (Reasoning)	$0.033	13.7s	25%	25	25	25	25	25	25	25	25	25	25	25%
111	GPT-5.5 (Reasoning, Low)	$0.028	9.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
99	GPT-5.5	$0.012	3.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
92	Gemini 3.1 Flash Lite (Reasoning)	$0.0023	1.8s	25%	25	25	25	25	25	25	25	25	25	25	25%
95	Gemini 3.1 Flash Lite	$0.0032	2.8s	25%	25	25	25	25	25	25	25	25	25	25	25%
94	Gemini 2.5 Flash	$0.0039	1.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
96	GPT-5.4	$0.0052	2.3s	24%	25	25	25	25	25	25	25	25	25	17	24%
112	Hermes 3 70B	$0.0039	13.9s	16%	50	38	33	32	32	13	13	10	8	8	24%
100	Ministral 3B	$0.0006	3.6s	20%	42	38	32	29	27	27	26	6	5	5	24%
103	Writer: Palmyra X5	$0.010	9.4s	22%	35	28	27	25	25	25	17	17	17	17	23%
129	Llama 3.1 70B	$0.0053	14.3s	3%	100	50	29	17	8	7	6	5	5	4	23%
105	GPT-4o, Aug. 6th (temp=1)	$0.017	1.6s	23%	25	25	25	25	25	25	25	25	17	13	23%
107	GPT-4o, Aug. 6th (temp=0)	$0.019	1.5s	23%	25	25	25	25	25	25	25	17	17	17	23%
108	Gemini 3.1 Flash Lite (Preview)	$0.0030	1.3s	16%	50	25	25	25	25	17	17	13	13	13	22%
133	Claude Sonnet 5	$0.041	6.0s	11%	42	38	35	25	17	13	13	13	13	8	21%
104	GPT-5.6 Terra	$0.011	1.8s	22%	25	25	25	25	25	25	17	17	13	13	21%
115	Ministral 3 3B	$0.0021	33.2s	21%	29	28	28	28	27	27	26	6	1	0	20%
131	Hermes 3 405B	$0.013	7.0s	6%	100	17	13	13	13	13	10	10	7	6	20%
124	GPT-4o Mini (temp=1)	$0.0013	6.0s	9%	38	35	32	29	13	10	10	10	10	6	19%
113	DeepSeek V3 (2024-12-26)	$0.0045	4.4s	18%	25	25	25	25	25	17	17	13	10	8	19%
116	DeepSeek V3.1	$0.0028	11.8s	16%	29	25	25	25	25	13	13	13	13	10	19%
126	DeepSeek V3.2	$0.0031	21.6s	14%	29	25	25	17	17	17	17	10	10	8	17%
135	Claude Sonnet 4.5	$0.047	5.4s	15%	25	25	17	17	17	17	17	13	13	13	17%
123	Mistral Small 4	$0.0022	8.4s	13%	32	30	29	29	28	6	5	4	3	1	17%
122	DeepSeek-V2 Chat	$0.0049	6.9s	14%	25	25	25	17	17	17	13	13	8	5	16%
128	Gemini 3 Flash (Preview)	$0.0074	3.0s	11%	25	17	17	17	13	13	13	13	10	10	15%
132	Grok 4.20	$0.014	4.3s	12%	17	17	17	13	13	13	13	10	8	8	13%
127	GPT-4o Mini (temp=0)	$0.0012	3.9s	12%	13	13	13	13	13	13	13	13	13	10	12%
137	Gemma 3 12B	$0.0010	15.2s	4%	29	7	6	6	5	5	5	4	3	3	7%
138	Gemma 3 27B	$0.0012	11.5s	5%	6	6	5	5	5	5	5	5	5	4	5%
141	Gemma 3 4B	$0.0007	20.4s	1%	2	2	2	1	1	1	1	1	1	1	1%
146	Mistral NeMO	$0.0056	5.0m	1%	3	3	2	2	1	1	1	0	0	0	1%
64.95%

Median	Evaluator	Top 3	Flop 3
70.0%	Correct "no violations" response	100Z.AI GLM 5 Turbo 100Z.AI GLM 5.2 (Reasoning, High) 100MiniMax M2.7	0DeepSeek-V2 Chat 0GPT-5.6 Terra 0GPT-5.5
80.8%	No hallucinated violations	100Claude Opus 4.8 (Reasoning) 100o4 Mini High 100Qwen3.7 Max	2Mistral NeMO 3Gemma 3 4B 5Ministral 3 3B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)