Long text (~1594 words), small codex (11 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
GPT-5	100%
Z.AI GLM 5	100%
MoonshotAI: Kimi K2.5	100%
ByteDance Seed 1.6	100%
o4 Mini High	100%
GPT-5.2	100%
Grok 4.1 Fast	100%
Z.AI GLM 4.7	100%
GPT-4.1	100%
o4 Mini	100%
Gemini 2.5 Flash (Reasoning)	100%
Grok 4 Fast	100%
GPT-5 Nano	100%
ByteDance Seed 1.6 Flash	100%
Mistral Medium 3.1	95%
Claude Sonnet 4.6	93%

	Score	Cost	Time
GPT-4.1 Nano	93%	$0.0001	962ms
Gemini 2.5 Flash Lite	74%	$0.0003	516ms
Mistral Small 3.2 24B	72%	$0.0003	1.0s
Ministral 3 8B	90%	$0.0004	1.4s
Mistral Medium 3.1	95%	$0.0011	764ms
GPT-4.1	100%	$0.0022	815ms
Grok 4.1 Fast	100%	$0.0009	6.1s
ByteDance Seed 1.6 Flash	100%	$0.0005	7.4s
Grok 4 Fast	100%	$0.0010	13.9s
Hermes 3 70B	69%	$0.0009	6.2s
Gemini 2.5 Flash Lite (Reasoning)	93%	$0.0020	14.3s
GPT-5.2	100%	$0.0083	9.6s
Gemini 2.5 Flash (Reasoning)	100%	$0.0074	11.8s
Minimax M2.5	88%	$0.0020	24.5s
o4 Mini	100%	$0.0085	15.4s
GPT-5.1	100%	$0.012	13.2s
GPT-5 Mini	100%	$0.0044	29.6s
ByteDance Seed 1.6	100%	$0.0033	31.2s
Claude Sonnet 4.6	93%	$0.019	12.6s
Z.AI GLM 5	100%	$0.0087	48.2s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
24	Claude Opus 4.6 (Reasoning)	$0.083	50.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Claude Sonnet 4.6 (Reasoning)	$0.101	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5 Mini	$0.0044	29.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	GPT-5.1	$0.012	13.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	GPT-5	$0.030	57.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Z.AI GLM 5	$0.0087	48.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	MoonshotAI: Kimi K2.5	$0.012	1.6m	100%	100	100	100	100	100	100	100	100	100	100	100%
9	ByteDance Seed 1.6	$0.0033	31.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	o4 Mini High	$0.018	33.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-5.2	$0.0083	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Grok 4.1 Fast	$0.0009	6.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Z.AI GLM 4.7	$0.015	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
1	GPT-4.1	$0.0022	815ms	100%	100	100	100	100	100	100	100	100	100	100	100%
7	o4 Mini	$0.0085	15.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Gemini 2.5 Flash (Reasoning)	$0.0074	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Grok 4 Fast	$0.0010	13.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5 Nano	$0.0033	1.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
3	ByteDance Seed 1.6 Flash	$0.0005	7.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Mistral Medium 3.1	$0.0011	764ms	70%	100	100	100	100	100	100	100	100	100	50	95%
20	Claude Sonnet 4.6	$0.019	12.6s	59%	100	100	100	100	100	100	100	100	100	32	93%
18	GPT-4.1 Nano	$0.0001	962ms	58%	100	100	100	100	100	100	100	100	100	30	93%
23	Aion 2.0	$0.0062	58.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
19	Gemini 2.5 Flash Lite (Reasoning)	$0.0020	14.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
21	Ministral 3 8B	$0.0004	1.4s	41%	100	100	100	100	100	100	100	100	100	2	90%
22	Minimax M2.5	$0.0020	24.5s	49%	100	100	100	100	100	100	100	100	50	25	88%
31	Gemini 2.5 Pro	$0.045	32.3s	37%	100	100	100	100	100	100	100	100	25	17	84%
43	Grok 4	$0.063	1.6m	35%	100	100	100	100	100	100	100	100	25	13	84%
25	Gemini 2.5 Flash Lite	$0.0003	516ms	20%	100	100	100	100	100	100	100	17	17	6	74%
30	Mistral Small 3.2 24B	$0.0003	1.0s	14%	100	100	100	100	100	100	100	6	6	6	72%
27	Hermes 3 70B	$0.0009	6.2s	21%	100	100	100	100	100	100	42	33	7	6	69%
28	Ministral 8B	$0.0004	4.4s	22%	100	100	100	100	100	50	27	27	26	26	66%
32	Z.AI GLM 4.5	$0.0022	16.0s	15%	100	100	100	100	100	32	30	29	25	3	62%
77	Z.AI GLM 4.7 Flash	$0.0040	3.1m	14%	100	100	100	100	100	25	25	25	25	17	62%
69	Z.AI GLM 4.6	$0.033	1.6m	14%	100	100	100	100	50	50	29	25	25	7	59%
82	Gemini 3.1 Pro (Preview)	$0.110	1.3m	7%	100	100	100	100	25	25	25	25	25	25	55%
83	Qwen 3.5 397B A17B	$0.018	3.9m	5%	100	100	100	100	25	25	25	17	13	13	52%
36	Arcee AI: Trinity Mini	$0.0003	8.2s	3%	100	100	100	100	25	17	17	17	13	6	49%
26	Claude Haiku 4.5	$0.0042	3.4s	48%	50	50	50	50	50	50	50	50	50	42	49%
34	Claude Sonnet 4	$0.012	4.5s	30%	50	42	42	42	42	38	35	17	17	13	34%
70	Gemini 3 Flash (Preview, Reasoning)	$0.025	44.3s	14%	100	25	25	25	25	25	25	25	25	17	32%
84	Gemini 3 Pro (Preview)	$0.131	1.5m	14%	100	25	25	25	25	25	25	25	25	17	32%
38	Claude Sonnet 4.5	$0.016	7.6s	29%	35	35	33	33	33	33	33	32	32	10	31%
63	Claude Opus 4	$0.056	6.7s	27%	50	50	42	42	42	38	13	13	13	10	31%
56	WizardLM 2 8x22b	$0.0021	47.3s	8%	100	50	50	50	30	8	6	6	5	4	31%
41	Llama 3.1 Nemotron 70B	$0.0040	22.1s	15%	100	38	35	32	31	31	28	5	5	3	31%
40	GPT-4o Mini (temp=1)	$0.0004	4.8s	15%	100	38	32	32	31	29	13	13	8	6	30%
33	Ministral 3B	$0.0001	3.0s	25%	42	38	31	31	31	31	30	29	29	3	29%
44	GPT-4o, Aug. 6th (temp=1)	$0.0065	2.7s	8%	100	42	42	42	17	17	13	8	7	7	29%
39	Writer: Palmyra X5	$0.0066	13.0s	25%	42	35	32	32	31	30	30	30	29	2	29%
35	GPT-4.1 Mini	$0.0015	7.2s	25%	35	35	35	33	31	30	30	29	28	5	29%
37	Ministral 3 3B	$0.0004	5.0s	24%	38	32	30	30	29	29	29	29	27	1	27%
49	Claude Opus 4.6	$0.030	12.3s	23%	42	42	38	35	32	32	32	6	6	6	27%
42	Qwen 2.5 72B	$0.0006	9.9s	10%	100	35	31	29	25	17	13	6	6	6	27%
45	DeepSeek V3.1	$0.0011	18.9s	9%	100	31	30	29	25	13	13	8	8	7	26%
51	Mistral NeMO	$0.0005	7.8s	1%	100	100	6	4	4	4	4	3	3	1	23%
50	Rocinante 12B	$0.0008	9.3s	4%	100	33	31	17	10	8	7	6	4	3	22%
48	Hermes 3 405B	$0.0029	6.3s	6%	100	25	17	13	13	13	10	10	10	8	22%
81	Qwen 3.5 Plus (2026-02-15)	$0.020	1.9m	14%	50	31	31	31	30	10	10	8	6	2	21%
55	GPT-4o, May 13th (temp=1)	$0.018	3.4s	13%	42	32	32	29	25	10	8	8	7	6	20%
52	DeepSeek V3.2	$0.0009	27.4s	13%	42	30	29	28	17	17	13	5	5	4	19%
71	Cohere Command R+ (Aug. 2024)	$0.012	15.0s	4%	100	17	17	10	10	10	8	7	6	1	19%
65	Claude Opus 4.5	$0.021	5.2s	12%	38	38	17	17	17	13	10	10	10	10	18%
54	DeepSeek V3 (2025-03-24)	$0.0007	15.9s	7%	42	35	33	10	10	10	10	10	8	7	18%
46	DeepSeek-V2 Chat	$0.0011	4.8s	9%	50	25	17	17	13	13	10	10	10	6	17%
66	Claude 3.5 Sonnet	$0.024	6.3s	16%	25	17	17	17	17	17	17	13	13	13	16%
47	DeepSeek V3 (2024-12-26)	$0.0012	5.5s	11%	31	25	17	13	13	13	13	13	13	8	16%
72	Llama 3.1 8B	$0.0001	28.5s	5%	29	28	28	28	7	6	6	5	5	4	15%
74	Arcee AI: Trinity Large (Preview)	$0.0000	31.2s	4%	29	28	28	27	5	5	5	4	3	3	14%
58	Llama 3.1 70B	$0.0014	12.4s	8%	31	25	17	17	10	10	10	10	5	3	14%
60	Claude 3.7 Sonnet	$0.012	4.1s	12%	17	17	13	13	13	13	13	10	10	10	13%
53	Gemini 3 Flash (Preview)	$0.0018	3.3s	8%	35	13	13	13	10	10	8	8	8	8	13%
67	Mistral Small Creative	$0.0008	9.4s	3%	27	26	26	26	4	3	3	2	2	2	12%
59	Claude 3 Haiku	$0.0011	3.0s	7%	33	10	10	8	8	7	7	7	6	6	10%
73	Ministral 3 14B	$0.0009	12.9s	2%	27	26	26	3	3	3	3	2	2	1	10%
57	Gemini 2.5 Flash	$0.0015	1.7s	9%	13	13	13	10	10	10	8	7	6	5	9%
61	Gemma 3 12B	$0.0001	6.8s	8%	17	8	8	8	8	8	7	7	7	7	9%
64	Gemma 3 27B	$0.0003	8.8s	8%	10	10	10	10	8	8	8	8	7	5	9%
68	GPT-4o, Aug. 6th (temp=0)	$0.0071	2.9s	8%	10	10	8	8	8	8	7	7	6	6	8%
62	GPT-4o Mini (temp=0)	$0.0005	5.4s	8%	8	8	8	8	8	8	8	8	6	6	8%
78	Mistral Large 2	$0.011	15.1s	4%	29	6	5	5	5	4	4	3	3	3	6%
75	Mistral Large 3	$0.0026	12.1s	5%	6	6	6	6	6	5	5	5	3	3	5%
80	GPT-4o, May 13th (temp=0)	$0.035	9.4s	4%	8	8	7	7	6	3	1	1	1	1	4%
79	Mistral Large	$0.011	15.0s	4%	6	6	5	5	4	4	4	4	3	3	4%
76	Gemma 3 4B	$0.0002	11.0s	3%	4	3	3	3	3	3	3	2	2	2	3%
49.38%

Median	Evaluator	Top 3	Flop 3
40.0%	Correct "no violations" response	100Claude Sonnet 4.6 (Reasoning) 100GPT-5 Mini 100GPT-5.1	0Gemma 3 12B 0Mistral Large 3 0Mistral Large
33.0%	No hallucinated violations	100GPT-5 Mini 100GPT-5.1 100ByteDance Seed 1.6	4Ministral 3 14B 4Mistral Small Creative 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)