Short text (~524 words), big codex (51 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Z.AI GLM 5 Turbo	100%
GPT-5.4 (Reasoning)	100%
GPT-5 Mini	100%
Claude Opus 4.6	100%
Claude Sonnet 4.6	100%
ByteDance Seed 1.6	100%
GPT-5.4 Mini (Reasoning)	100%
o4 Mini High	100%
GPT-5.2	100%
Claude Opus 4.5	100%
Grok 4.1 Fast	100%
MiniMax M2.7	100%
GPT-4.1	100%
o4 Mini	100%
Grok 4	100%
Grok 4 Fast	100%
GPT-5.4 Mini (Reasoning, Low)	100%
Mistral Large 3	100%
Nemotron 3 Super	100%
Inception Mercury 2	100%

	Score	Cost	Time
GPT-5.4 Nano	74%	$0.0006	1.4s
GPT-5.4 Nano (Reasoning, Low)	93%	$0.0008	3.4s
Ministral 8B	100%	$0.0013	564ms
Arcee AI: Trinity Mini	100%	$0.0007	6.9s
Inception Mercury	100%	$0.0009	8.6s
Ministral 3 8B	100%	$0.0020	589ms
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0038	2.7s
ByteDance Seed 1.6 Flash	93%	$0.0013	7.7s
Inception Mercury 2	100%	$0.0031	3.7s
GPT-5.4 Nano (Reasoning)	100%	$0.0020	8.1s
Grok 4 Fast	100%	$0.0025	16.0s
Grok 4.1 Fast	100%	$0.0028	7.6s
GPT-5.4 Mini (Reasoning)	100%	$0.0063	5.2s
Gemini 2.5 Flash Lite (Reasoning)	85%	$0.0023	14.2s
Mistral Small 4 (Reasoning)	93%	$0.0028	23.8s
GPT-4.1	100%	$0.0086	1.0s
Mistral Large 3	100%	$0.0066	1.4s
Z.AI GLM 4.6	90%	$0.0053	12.9s
MiniMax M2.7	100%	$0.0045	17.4s
ByteDance Seed 1.6	100%	$0.0050	19.7s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
14	Z.AI GLM 5 Turbo	$0.0077	13.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	GPT-5.4 (Reasoning)	$0.032	23.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	GPT-5 Mini	$0.0049	30.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Claude Opus 4.6	$0.082	9.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Claude Sonnet 4.6	$0.043	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	ByteDance Seed 1.6	$0.0050	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5.4 Mini (Reasoning)	$0.0063	5.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	o4 Mini High	$0.020	33.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5.2	$0.014	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	Claude Opus 4.5	$0.072	2.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Grok 4.1 Fast	$0.0028	7.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	MiniMax M2.7	$0.0045	17.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	GPT-4.1	$0.0086	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	o4 Mini	$0.012	20.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Grok 4	$0.063	52.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Grok 4 Fast	$0.0025	16.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-5.4 Mini (Reasoning, Low)	$0.0038	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Mistral Large 3	$0.0066	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Nemotron 3 Super	$0.0000	51.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Inception Mercury 2	$0.0031	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	GPT-5 Nano	$0.0020	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Mistral Large 2	$0.026	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Nano (Reasoning)	$0.0020	8.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Mistral Large	$0.026	1.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury	$0.0009	8.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0020	589ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Arcee AI: Trinity Mini	$0.0007	6.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0013	564ms	100%	100	100	100	100	100	100	100	100	100	100	100%
29	GPT-5.4 Nano (Reasoning, Low)	$0.0008	3.4s	57%	100	100	100	100	100	100	100	100	100	29	93%
35	GPT-5.4 (Reasoning, Low)	$0.017	6.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
40	Z.AI GLM 5	$0.020	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
38	Aion 2.0	$0.014	1.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
34	MiniMax M2.5	$0.0049	21.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
44	Gemini 2.5 Pro	$0.067	45.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
32	Gemini 2.5 Flash (Reasoning)	$0.0092	11.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
36	Z.AI GLM 4.5	$0.0079	42.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
41	Z.AI GLM 4.7 Flash	$0.0036	2.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
33	Mistral Small 4 (Reasoning)	$0.0028	23.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
30	ByteDance Seed 1.6 Flash	$0.0013	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Grok 4.20 (Beta, Reasoning)	$0.025	15.5s	50%	100	100	100	100	100	100	100	100	100	17	92%
31	Z.AI GLM 4.6	$0.0053	12.9s	60%	100	100	100	100	100	100	100	100	50	50	90%
43	GPT-5.1	$0.025	23.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
113	Qwen 3.5 122B	$0.138	7.0m	40%	100	100	100	100	100	100	100	100	25	25	85%
45	Gemini 3 Flash (Preview, Reasoning)	$0.025	40.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
39	Gemini 2.5 Flash Lite (Reasoning)	$0.0023	14.2s	40%	100	100	100	100	100	100	100	100	25	25	85%
56	Qwen 3.5 9B	$0.0044	4.4m	45%	100	100	100	100	100	100	100	50	50	25	83%
67	Gemini 3 Pro (Preview)	$0.109	1.2m	31%	100	100	100	100	100	100	100	25	25	25	78%
54	Qwen 3.5 35B	$0.032	1.8m	31%	100	100	100	100	100	100	100	25	25	25	78%
47	ByteDance Seed 2.0 Lite	$0.0085	1.0m	31%	100	100	100	100	100	100	100	25	25	25	78%
48	Nemotron 3 Nano	$0.0022	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
59	Z.AI GLM 4.7	$0.029	2.4m	29%	100	100	100	100	100	100	100	25	25	17	77%
46	Stealth: Hunter Alpha	$0.0000	56.8s	29%	100	100	100	100	100	100	100	25	25	17	77%
42	GPT-5.4 Nano	$0.0006	1.4s	37%	100	100	100	100	100	100	42	38	38	28	74%
50	Cohere Command R+ (Aug. 2024)	$0.037	14.6s	18%	100	100	100	100	100	100	100	17	17	1	73%
53	Qwen 3.5 Flash	$0.0077	2.2m	31%	100	100	100	100	100	100	50	25	25	25	73%
74	Claude Opus 4.6 (Reasoning)	$0.117	33.8s	27%	100	100	100	100	100	100	25	25	25	25	70%
51	GPT-5	$0.033	46.9s	27%	100	100	100	100	100	100	25	25	25	25	70%
62	MoonshotAI: Kimi K2.5	$0.018	2.7m	27%	100	100	100	100	100	100	25	25	25	25	70%
109	Qwen 3.5 27B	$0.060	4.9m	27%	100	100	100	100	100	100	25	25	25	25	70%
90	ByteDance Seed 2.0 Mini	$0.0050	4.5m	22%	100	100	100	100	100	100	25	25	17	17	68%
49	Stealth: Healer Alpha	$0.0000	22.4s	16%	100	100	100	100	100	25	25	25	25	25	63%
52	GPT-4.1 Mini	$0.0025	3.5s	13%	100	100	100	100	50	31	31	29	25	25	59%
55	Mistral Medium 3.1	$0.0058	3.8s	11%	100	100	100	100	50	30	29	29	17	17	57%
58	GPT-5.4 Mini	$0.0039	834ms	7%	100	100	100	100	25	25	25	25	25	25	55%
60	Qwen 3 32B	$0.0024	24.4s	7%	100	100	100	100	25	25	25	25	25	25	55%
112	Qwen 3.5 397B A17B	$0.032	5.1m	6%	100	100	100	100	25	25	25	25	25	17	54%
72	Claude 3.7 Sonnet	$0.044	2.7s	6%	100	100	100	100	25	25	25	17	17	17	53%
105	Gemini 3.1 Pro (Preview)	$0.078	55.6s	8%	100	100	100	25	25	25	25	25	25	25	48%
61	Qwen 2.5 72B	$0.0017	10.6s	13%	100	100	42	38	32	31	29	28	25	17	44%
83	WizardLM 2 8x22b	$0.0087	1.2m	10%	100	100	50	42	28	25	25	25	17	1	41%
104	Claude 3.5 Sonnet	$0.093	6.9s	10%	100	100	50	50	42	17	17	13	13	10	41%
111	Claude Sonnet 4.6 (Reasoning)	$0.110	59.4s	10%	100	100	25	25	25	25	25	25	25	17	39%
96	Claude Sonnet 4	$0.046	4.9s	2%	100	100	100	17	13	13	10	10	10	10	38%
57	Claude Haiku 4.5	$0.016	4.7s	32%	50	50	50	42	42	42	38	25	25	17	38%
66	GPT-4.1 Nano	$0.0009	8.8s	9%	100	100	42	27	27	27	26	26	1	1	38%
70	Qwen3 235B A22B Instruct 2507	$0.0018	16.2s	8%	100	100	50	25	25	25	17	17	8	3	37%
69	DeepSeek V3 (2025-03-24)	$0.0032	5.3s	9%	100	100	25	25	25	25	17	17	13	8	35%
77	Gemini 2.5 Flash Lite	$0.0008	2.2s	2%	100	100	100	13	13	13	4	2	2	2	35%
115	Claude Opus 4	$0.225	6.0s	8%	100	50	50	50	17	17	17	17	17	13	35%
85	Ministral 3 14B	$0.0044	1.4m	15%	100	28	28	28	27	26	26	26	25	25	34%
82	Mistral Small 3.2 24B	$0.0014	3.4s	1%	100	100	38	38	6	6	5	5	5	5	31%
101	Qwen 3.5 Plus (2026-02-15)	$0.023	1.9m	20%	50	50	50	42	33	25	25	17	7	6	31%
84	Grok 4.20 (Beta)	$0.0086	1.5s	3%	100	100	33	13	10	10	10	10	10	8	30%
64	Mistral Small Creative	$0.0017	7.0s	23%	31	31	29	28	28	28	28	28	27	2	26%
63	Gemini 2.5 Flash	$0.0039	1.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
65	GPT-5.4	$0.0052	2.3s	24%	25	25	25	25	25	25	25	25	25	17	24%
78	Hermes 3 70B	$0.0039	13.9s	16%	50	38	33	32	32	13	13	10	8	8	24%
68	Ministral 3B	$0.0006	3.6s	20%	42	38	32	29	27	27	26	6	5	5	24%
71	Writer: Palmyra X5	$0.010	9.4s	22%	35	28	27	25	25	25	17	17	17	17	23%
95	Llama 3.1 70B	$0.0053	14.3s	3%	100	50	29	17	8	7	6	5	5	4	23%
73	GPT-4o, Aug. 6th (temp=1)	$0.017	1.6s	23%	25	25	25	25	25	25	25	25	17	13	23%
75	GPT-4o, Aug. 6th (temp=0)	$0.019	1.5s	23%	25	25	25	25	25	25	25	17	17	17	23%
76	Gemini 3.1 Flash Lite (Preview)	$0.0030	1.3s	16%	50	25	25	25	25	17	17	13	13	13	22%
108	GPT-4o, May 13th (temp=1)	$0.067	3.0s	13%	42	38	35	17	17	17	17	17	17	4	22%
86	Llama 3.1 8B	$0.0005	54.6s	20%	38	32	31	29	28	26	10	5	3	0	20%
80	Ministral 3 3B	$0.0021	33.2s	21%	29	28	28	28	27	27	26	6	1	0	20%
97	Hermes 3 405B	$0.013	7.0s	6%	100	17	13	13	13	13	10	10	7	6	20%
99	Arcee AI: Trinity Large (Preview)	$0.0000	1.6m	20%	30	28	28	27	27	26	26	4	2	1	20%
103	GPT-4o, May 13th (temp=0)	$0.068	2.8s	21%	33	25	25	25	25	25	13	10	10	7	20%
89	GPT-4o Mini (temp=1)	$0.0013	6.0s	9%	38	35	32	29	13	10	10	10	10	6	19%
79	DeepSeek V3 (2024-12-26)	$0.0045	4.4s	18%	25	25	25	25	25	17	17	13	10	8	19%
81	DeepSeek V3.1	$0.0028	11.8s	16%	29	25	25	25	25	13	13	13	13	10	19%
93	Llama 3.1 Nemotron 70B	$0.016	28.9s	18%	31	30	30	30	30	17	6	5	5	4	18%
91	DeepSeek V3.2	$0.0031	21.6s	14%	29	25	25	17	17	17	17	10	10	8	17%
102	Claude Sonnet 4.5	$0.047	5.4s	15%	25	25	17	17	17	17	17	13	13	13	17%
88	Mistral Small 4	$0.0022	8.4s	13%	32	30	29	29	28	6	5	4	3	1	17%
87	DeepSeek-V2 Chat	$0.0049	6.9s	14%	25	25	25	17	17	17	13	13	8	5	16%
94	Gemini 3 Flash (Preview)	$0.0074	3.0s	11%	25	17	17	17	13	13	13	13	10	10	15%
98	Claude 3 Haiku	$0.0041	4.2s	7%	33	32	17	10	10	8	7	7	7	6	14%
100	Rocinante 12B	$0.0028	11.7s	7%	33	31	17	13	10	8	6	6	6	4	13%
92	GPT-4o Mini (temp=0)	$0.0012	3.9s	12%	13	13	13	13	13	13	13	13	13	10	12%
106	Gemma 3 12B	$0.0010	15.2s	4%	29	7	6	6	5	5	5	4	3	3	7%
107	Gemma 3 27B	$0.0012	11.5s	5%	6	6	5	5	5	5	5	5	5	4	5%
114	LFM2 24B	$0.0013	2.7m	0%	28	3	2	2	0	0	0	0	0	0	4%
110	Gemma 3 4B	$0.0007	20.4s	1%	2	2	2	1	1	1	1	1	1	1	1%
116	Mistral NeMO	$0.0056	5.0m	1%	3	3	2	2	1	1	1	0	0	0	1%
60.48%

Median	Evaluator	Top 3	Flop 3
60.0%	Correct "no violations" response	100Z.AI GLM 5 Turbo 100Claude Opus 4.6 100Claude Opus 4.5	0DeepSeek-V2 Chat 0Gemini 3 Flash (Preview) 0Gemma 3 27B
76.7%	No hallucinated violations	100Claude Opus 4.6 100Arcee AI: Trinity Mini 100GPT-5.2	2LFM2 24B 2Mistral NeMO 3Gemma 3 4B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)