Long text (~1594 words), small codex (11 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
Z.AI GLM 5	100%
o4 Mini High	100%
GPT-4.1	100%
GPT-4o Mini (temp=1)	100%
GPT-4o Mini (temp=0)	100%
ByteDance Seed 1.6 Flash	100%
Gemini 3.1 Pro (Preview)	93%
GPT-5	93%
Gemini 3 Pro (Preview)	93%
o4 Mini	93%
Z.AI GLM 4.7 Flash	93%
GPT-5 Nano	93%
Claude Sonnet 4.6	86%
Grok 4.1 Fast	85%
GPT-4.1 Nano	78%
Gemini 2.5 Flash Lite (Reasoning)	78%

	Score	Cost	Time
GPT-4.1 Nano	78%	$0.0003	1.4s
GPT-4o Mini (temp=1)	100%	$0.0004	597ms
GPT-4o Mini (temp=0)	100%	$0.0005	670ms
GPT-4.1	100%	$0.0045	728ms
ByteDance Seed 1.6 Flash	100%	$0.0008	9.1s
Grok 4.1 Fast	85%	$0.0022	20.0s
Minimax M2.5	73%	$0.0029	22.1s
Gemini 2.5 Flash Lite (Reasoning)	78%	$0.0030	24.9s
Cohere Command R+ (Aug. 2024)	65%	$0.013	2.6s
o4 Mini	93%	$0.0094	17.8s
GPT-5 Mini	100%	$0.0060	40.1s
GPT-5 Nano	93%	$0.0044	1.4m
Z.AI GLM 4.5	72%	$0.0047	42.7s
Z.AI GLM 5	100%	$0.014	1.3m
Claude Sonnet 4.6	86%	$0.029	15.0s
o4 Mini High	100%	$0.022	48.0s
GPT-5.1	100%	$0.028	32.0s
Z.AI GLM 4.7 Flash	93%	$0.0038	2.5m
GPT-5	93%	$0.045	1.3m
Z.AI GLM 4.6	69%	$0.023	1.8m

	Score	Cost	Speed	Stability
GPT-4o Mini (temp=1)	100%	$0.0004	597ms	100%
GPT-4o Mini (temp=0)	100%	$0.0005	670ms	100%
ByteDance Seed 1.6 Flash	100%	$0.0008	9.1s	100%
GPT-4.1	100%	$0.0045	728ms	100%
GPT-5 Mini	100%	$0.0060	40.1s	100%
GPT-5.1	100%	$0.028	32.0s	100%
o4 Mini High	100%	$0.022	48.0s	100%
Z.AI GLM 5	100%	$0.014	1.3m	100%
o4 Mini	93%	$0.0094	17.8s	55%
GPT-5 Nano	93%	$0.0044	1.4m	55%
Grok 4.1 Fast	85%	$0.0022	20.0s	40%
Z.AI GLM 4.7 Flash	93%	$0.0038	2.5m	55%
Claude Sonnet 4.6	86%	$0.029	15.0s	43%
GPT-4.1 Nano	78%	$0.0003	1.4s	29%
GPT-5	93%	$0.045	1.3m	55%
Claude Opus 4.6 (Reasoning)	100%	$0.144	1.3m	100%
Gemini 2.5 Flash Lite (Reasoning)	78%	$0.0030	24.9s	31%
Minimax M2.5	73%	$0.0029	22.1s	31%
Z.AI GLM 4.5	72%	$0.0047	42.7s	28%
Claude Sonnet 4.6 (Reasoning)	100%	$0.150	2.2m	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
16	Claude Opus 4.6 (Reasoning)	$0.144	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Claude Sonnet 4.6 (Reasoning)	$0.150	2.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-5 Mini	$0.0060	40.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.1	$0.028	32.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Z.AI GLM 5	$0.014	1.3m	100%	100	100	100	100	100	100	100	100	100	100	100%
7	o4 Mini High	$0.022	48.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-4.1	$0.0045	728ms	100%	100	100	100	100	100	100	100	100	100	100	100%
1	GPT-4o Mini (temp=1)	$0.0004	597ms	100%	100	100	100	100	100	100	100	100	100	100	100%
2	GPT-4o Mini (temp=0)	$0.0005	670ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	ByteDance Seed 1.6 Flash	$0.0008	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Gemini 3.1 Pro (Preview)	$0.144	1.7m	55%	100	100	100	100	100	100	100	100	100	25	93%
15	GPT-5	$0.045	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
54	Gemini 3 Pro (Preview)	$0.181	2.1m	55%	100	100	100	100	100	100	100	100	100	25	93%
9	o4 Mini	$0.0094	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
12	Z.AI GLM 4.7 Flash	$0.0038	2.5m	55%	100	100	100	100	100	100	100	100	100	25	93%
10	GPT-5 Nano	$0.0044	1.4m	55%	100	100	100	100	100	100	100	100	100	25	93%
13	Claude Sonnet 4.6	$0.029	15.0s	43%	100	100	100	100	100	100	100	100	33	25	86%
11	Grok 4.1 Fast	$0.0022	20.0s	40%	100	100	100	100	100	100	100	100	25	25	85%
14	GPT-4.1 Nano	$0.0003	1.4s	29%	100	100	100	100	100	100	100	50	25	3	78%
17	Gemini 2.5 Flash Lite (Reasoning)	$0.0030	24.9s	31%	100	100	100	100	100	100	100	25	25	25	78%
18	Minimax M2.5	$0.0029	22.1s	31%	100	100	100	100	100	100	50	25	25	25	73%
19	Z.AI GLM 4.5	$0.0047	42.7s	28%	100	100	100	100	100	100	50	33	26	8	72%
29	Z.AI GLM 4.6	$0.023	1.8m	22%	100	100	100	100	100	100	42	25	25	0	69%
22	Cohere Command R+ (Aug. 2024)	$0.013	2.6s	15%	100	100	100	100	100	100	17	13	13	10	65%
44	Z.AI GLM 4.7	$0.029	2.4m	16%	100	100	100	100	100	25	25	25	25	25	63%
24	Grok 4 Fast	$0.0020	24.0s	16%	100	100	100	100	100	25	25	25	25	25	63%
27	GPT-5.2	$0.019	20.2s	14%	100	100	100	100	100	25	25	25	25	17	62%
23	GPT-4o, Aug. 6th (temp=1)	$0.0075	1.3s	15%	100	100	100	100	100	42	38	17	10	10	62%
26	Mistral Medium 3.1	$0.0027	4.4s	11%	100	100	100	100	42	33	33	33	30	6	58%
25	Hermes 3 70B	$0.0016	12.9s	19%	100	100	100	50	50	50	38	33	32	8	56%
42	Aion 2.0	$0.011	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
32	Arcee AI: Trinity Large (Preview)	$0.0000	20.5s	5%	100	100	100	100	29	28	28	10	8	4	51%
35	ByteDance Seed 1.6	$0.0056	46.9s	8%	100	100	100	25	25	25	25	25	25	25	48%
21	Claude Haiku 4.5	$0.0076	5.4s	44%	50	50	50	50	50	50	50	42	38	35	46%
83	Grok 4	$0.108	2.5m	7%	100	100	100	25	25	25	25	17	17	17	45%
48	Gemini 3 Flash (Preview, Reasoning)	$0.025	46.9s	7%	100	100	100	25	25	25	25	17	17	13	45%
70	Gemini 2.5 Pro	$0.060	45.6s	10%	100	100	25	25	25	25	25	25	25	25	40%
37	Gemini 2.5 Flash (Reasoning)	$0.0094	15.1s	10%	100	100	25	25	25	25	25	25	25	25	40%
28	Claude Opus 4.5	$0.033	5.9s	36%	42	42	42	42	38	38	38	38	38	35	39%
36	Claude Opus 4.6	$0.043	14.0s	30%	50	50	50	50	42	42	42	38	13	10	38%
38	Ministral 8B	$0.0007	9.6s	4%	100	100	100	27	26	26	2	1	1	1	38%
34	Qwen 2.5 72B	$0.0007	6.7s	10%	100	100	38	35	31	31	10	10	8	7	37%
39	Llama 3.1 Nemotron 70B	$0.0068	17.7s	9%	100	100	42	35	31	27	10	7	6	6	36%
40	Claude Sonnet 4	$0.020	5.8s	13%	100	50	42	38	38	17	13	13	13	10	33%
30	GPT-4.1 Mini	$0.0023	7.3s	24%	50	50	33	33	33	32	31	29	28	2	32%
75	MoonshotAI: Kimi K2.5	$0.019	2.4m	14%	100	25	25	25	25	25	25	25	25	17	32%
50	Arcee AI: Trinity Mini	$0.0014	45.3s	6%	100	100	25	25	25	17	13	10	2	0	32%
45	Hermes 3 405B	$0.0052	6.8s	4%	100	100	17	13	13	13	13	10	10	10	30%
79	Claude Opus 4	$0.098	8.1s	27%	50	42	42	38	38	38	17	13	13	7	29%
46	Llama 3.1 70B	$0.0024	9.1s	3%	100	100	25	17	13	10	8	6	6	6	29%
73	Qwen 3.5 Plus (2026-02-15)	$0.024	2.2m	20%	50	50	50	32	31	30	28	8	6	6	29%
31	Ministral 3B	$0.0003	7.5s	26%	32	31	29	28	27	27	26	26	26	26	28%
62	Ministral 3 8B	$0.0020	53.4s	0%	100	100	26	26	2	2	1	1	1	0	26%
84	Qwen 3.5 397B A17B	$0.037	7.1m	25%	25	25	25	25	25	25	25	25	25	25	25%
43	WizardLM 2 8x22b	$0.0037	57.8s	21%	50	31	29	29	29	28	28	13	5	4	25%
51	Gemini 2.5 Flash Lite	$0.0007	2.1s	1%	100	100	8	7	7	5	5	4	3	1	24%
56	GPT-4o, May 13th (temp=1)	$0.029	3.2s	14%	38	33	33	33	25	13	8	8	8	7	21%
41	Llama 3.1 8B	$0.0002	16.1s	17%	42	31	30	30	28	17	10	8	6	5	21%
55	Claude 3.7 Sonnet	$0.019	4.1s	11%	50	42	17	17	17	13	13	13	13	13	20%
49	Claude 3 Haiku	$0.0018	3.8s	9%	38	38	33	33	13	13	10	8	6	5	20%
47	DeepSeek V3 (2024-12-26)	$0.0019	6.1s	12%	33	25	25	25	17	13	13	8	8	8	18%
59	Mistral Small 3.2 24B	$0.0010	13.9s	2%	100	32	6	6	6	6	5	4	4	3	17%
53	Mistral Large 3	$0.0043	19.7s	12%	29	29	29	29	28	4	3	3	3	3	16%
66	DeepSeek V3.1	$0.0016	38.9s	6%	50	29	25	10	10	7	7	5	5	1	15%
64	Ministral 3 3B	$0.0017	49.7s	10%	28	28	26	26	25	2	2	1	0	0	14%
76	Claude 3.5 Sonnet	$0.041	8.2s	12%	17	17	17	13	13	13	13	13	13	10	14%
52	DeepSeek V3 (2025-03-24)	$0.0012	12.4s	11%	25	17	13	13	13	13	13	10	8	8	13%
63	DeepSeek V3.2	$0.0015	31.6s	8%	28	26	13	13	10	10	7	7	6	5	12%
58	DeepSeek-V2 Chat	$0.0021	8.9s	8%	35	13	13	13	10	10	8	8	6	6	12%
78	Mistral Large 2	$0.017	17.6s	3%	29	28	27	4	4	4	4	4	4	3	11%
77	Claude Sonnet 4.5	$0.025	8.9s	7%	33	8	8	8	8	7	7	7	7	6	10%
57	Gemini 3 Flash (Preview)	$0.0023	3.7s	10%	13	13	13	10	10	10	8	8	8	7	10%
72	Ministral 3 14B	$0.0015	20.5s	2%	29	28	25	3	3	2	2	1	1	1	9%
69	Mistral Small Creative	$0.0011	10.7s	2%	27	27	27	2	2	2	2	2	2	1	9%
60	Gemma 3 12B	$0.0002	8.5s	8%	17	13	13	10	10	7	6	6	6	6	9%
67	Rocinante 12B	$0.0014	20.1s	6%	30	13	10	7	7	6	4	4	4	3	9%
81	Mistral Large	$0.017	16.5s	3%	29	28	5	4	4	4	4	4	3	3	9%
80	Writer: Palmyra X5	$0.013	22.2s	4%	28	27	5	5	5	4	4	3	2	2	8%
61	Gemini 2.5 Flash	$0.0021	2.0s	8%	10	10	8	8	8	8	8	8	7	5	8%
68	GPT-4o, Aug. 6th (temp=0)	$0.011	2.8s	7%	10	10	10	10	8	7	7	6	5	5	8%
65	Gemma 3 27B	$0.0006	11.4s	7%	10	8	8	8	7	7	7	7	6	5	7%
82	GPT-4o, May 13th (temp=0)	$0.033	4.9s	6%	8	7	7	7	6	6	6	6	5	5	6%
71	Mistral NeMO	$0.0009	7.8s	4%	7	7	6	5	4	4	4	4	3	2	5%
74	Gemma 3 4B	$0.0003	9.9s	3%	4	3	3	3	3	3	3	3	2	2	3%
45.09%

Median	Evaluator	Top 3	Flop 3
32.5%	Correct "no violations" response	100o4 Mini High 100GPT-5 Mini 100GPT-5.1	0DeepSeek V3 (2025-03-24) 0Mistral NeMO 0GPT-4o, Aug. 6th (temp=0)
38.8%	No hallucinated violations	100o4 Mini High 100GPT-5.1 100GPT-4o Mini (temp=0)	3Ministral 3 3B 4Mistral Small Creative 4Ministral 3 14B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)