Short text (~524 words), big codex (51 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
GPT-5 Mini	100%
Claude Opus 4.6	100%
Claude Sonnet 4.6	100%
ByteDance Seed 1.6	100%
o4 Mini High	100%
GPT-5.2	100%
Claude Opus 4.5	100%
Grok 4.1 Fast	100%
GPT-4.1	100%
o4 Mini	100%
Grok 4	100%
Grok 4 Fast	100%
Mistral Large 3	100%
GPT-5 Nano	100%
Mistral Large 2	100%
Mistral Large	100%
Ministral 3 8B	100%
Arcee AI: Trinity Mini	100%
Ministral 8B	100%
Z.AI GLM 5	93%

	Score	Cost	Time
Ministral 8B	100%	$0.0013	564ms
Arcee AI: Trinity Mini	100%	$0.0007	6.9s
Ministral 3 8B	100%	$0.0020	589ms
ByteDance Seed 1.6 Flash	93%	$0.0013	7.7s
Grok 4 Fast	100%	$0.0025	16.0s
Grok 4.1 Fast	100%	$0.0028	7.6s
Gemini 2.5 Flash Lite (Reasoning)	85%	$0.0023	14.2s
GPT-4.1	100%	$0.0086	1.0s
Mistral Large 3	100%	$0.0066	1.4s
Z.AI GLM 4.6	90%	$0.0053	12.9s
GPT-5 Nano	100%	$0.0020	36.0s
ByteDance Seed 1.6	100%	$0.0050	19.7s
Minimax M2.5	93%	$0.0049	21.4s
GPT-5 Mini	100%	$0.0049	30.6s
Gemini 2.5 Flash (Reasoning)	93%	$0.0092	11.7s
GPT-5.2	100%	$0.014	11.3s
Z.AI GLM 4.5	93%	$0.0079	42.0s
o4 Mini	100%	$0.012	20.0s
o4 Mini High	100%	$0.020	33.9s
Mistral Large	100%	$0.026	1.6s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
11	GPT-5 Mini	$0.0049	30.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Claude Opus 4.6	$0.082	9.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Claude Sonnet 4.6	$0.043	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	ByteDance Seed 1.6	$0.0050	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	o4 Mini High	$0.020	33.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	GPT-5.2	$0.014	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Claude Opus 4.5	$0.072	2.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Grok 4.1 Fast	$0.0028	7.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-4.1	$0.0086	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	o4 Mini	$0.012	20.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Grok 4	$0.063	52.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Grok 4 Fast	$0.0025	16.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Mistral Large 3	$0.0066	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	GPT-5 Nano	$0.0020	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Mistral Large 2	$0.026	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	Mistral Large	$0.026	1.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0020	589ms	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Arcee AI: Trinity Mini	$0.0007	6.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0013	564ms	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Z.AI GLM 5	$0.020	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
26	Aion 2.0	$0.014	1.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
23	Minimax M2.5	$0.0049	21.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
29	Gemini 2.5 Pro	$0.067	45.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
22	Gemini 2.5 Flash (Reasoning)	$0.0092	11.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
24	Z.AI GLM 4.5	$0.0079	42.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
30	Z.AI GLM 4.7 Flash	$0.0036	2.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
20	ByteDance Seed 1.6 Flash	$0.0013	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
21	Z.AI GLM 4.6	$0.0053	12.9s	60%	100	100	100	100	100	100	100	100	50	50	90%
28	GPT-5.1	$0.025	23.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
31	Gemini 3 Flash (Preview, Reasoning)	$0.025	40.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
25	Gemini 2.5 Flash Lite (Reasoning)	$0.0023	14.2s	40%	100	100	100	100	100	100	100	100	25	25	85%
45	Gemini 3 Pro (Preview)	$0.109	1.2m	31%	100	100	100	100	100	100	100	25	25	25	78%
39	Z.AI GLM 4.7	$0.029	2.4m	29%	100	100	100	100	100	100	100	25	25	17	77%
32	Cohere Command R+ (Aug. 2024)	$0.037	14.6s	18%	100	100	100	100	100	100	100	17	17	1	73%
51	Claude Opus 4.6 (Reasoning)	$0.117	33.8s	27%	100	100	100	100	100	100	25	25	25	25	70%
34	GPT-5	$0.033	46.9s	27%	100	100	100	100	100	100	25	25	25	25	70%
48	MoonshotAI: Kimi K2.5	$0.018	2.7m	27%	100	100	100	100	100	100	25	25	25	25	70%
33	GPT-4.1 Mini	$0.0025	3.5s	13%	100	100	100	100	50	31	31	29	25	25	59%
35	Mistral Medium 3.1	$0.0058	3.8s	11%	100	100	100	100	50	30	29	29	17	17	57%
82	Qwen 3.5 397B A17B	$0.032	5.1m	6%	100	100	100	100	25	25	25	25	25	17	54%
44	Claude 3.7 Sonnet	$0.044	2.7s	6%	100	100	100	100	25	25	25	17	17	17	53%
78	Gemini 3.1 Pro (Preview)	$0.078	55.6s	8%	100	100	100	25	25	25	25	25	25	25	48%
37	Qwen 2.5 72B	$0.0017	10.6s	13%	100	100	42	38	32	31	29	28	25	17	44%
59	WizardLM 2 8x22b	$0.0087	1.2m	10%	100	100	50	42	28	25	25	25	17	1	41%
74	Claude 3.5 Sonnet	$0.093	6.9s	10%	100	100	50	50	42	17	17	13	13	10	41%
81	Claude Sonnet 4.6 (Reasoning)	$0.110	59.4s	10%	100	100	25	25	25	25	25	25	25	17	39%
67	Claude Sonnet 4	$0.046	4.9s	2%	100	100	100	17	13	13	10	10	10	10	38%
36	Claude Haiku 4.5	$0.016	4.7s	32%	50	50	50	42	42	42	38	25	25	17	38%
41	GPT-4.1 Nano	$0.0009	8.8s	9%	100	100	42	27	27	27	26	26	1	1	38%
43	DeepSeek V3 (2025-03-24)	$0.0032	5.3s	9%	100	100	25	25	25	25	17	17	13	8	35%
50	Gemini 2.5 Flash Lite	$0.0008	2.2s	2%	100	100	100	13	13	13	4	2	2	2	35%
83	Claude Opus 4	$0.225	6.0s	8%	100	50	50	50	17	17	17	17	17	13	35%
64	Ministral 3 14B	$0.0044	1.4m	15%	100	28	28	28	27	26	26	26	25	25	34%
54	Mistral Small 3.2 24B	$0.0014	3.4s	1%	100	100	38	38	6	6	5	5	5	5	31%
79	Qwen 3.5 Plus (2026-02-15)	$0.023	1.9m	20%	50	50	50	42	33	25	25	17	7	6	31%
40	Mistral Small Creative	$0.0017	7.0s	23%	31	31	29	28	28	28	28	28	27	2	26%
38	Gemini 2.5 Flash	$0.0039	1.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
53	Hermes 3 70B	$0.0039	13.9s	16%	50	38	33	32	32	13	13	10	8	8	24%
42	Ministral 3B	$0.0006	3.6s	20%	42	38	32	29	27	27	26	6	5	5	24%
47	Writer: Palmyra X5	$0.010	9.4s	22%	35	28	27	25	25	25	17	17	17	17	23%
66	Llama 3.1 70B	$0.0053	14.3s	3%	100	50	29	17	8	7	6	5	5	4	23%
46	GPT-4o, Aug. 6th (temp=1)	$0.017	1.6s	23%	25	25	25	25	25	25	25	25	17	13	23%
49	GPT-4o, Aug. 6th (temp=0)	$0.019	1.5s	23%	25	25	25	25	25	25	25	17	17	17	23%
77	GPT-4o, May 13th (temp=1)	$0.067	3.0s	13%	42	38	35	17	17	17	17	17	17	4	22%
62	Llama 3.1 8B	$0.0005	54.6s	20%	38	32	31	29	28	26	10	5	3	0	20%
56	Ministral 3 3B	$0.0021	33.2s	21%	29	28	28	28	27	27	26	6	1	0	20%
69	Hermes 3 405B	$0.013	7.0s	6%	100	17	13	13	13	13	10	10	7	6	20%
73	Arcee AI: Trinity Large (Preview)	$0.0000	1.6m	20%	30	28	28	27	27	26	26	4	2	1	20%
72	GPT-4o, May 13th (temp=0)	$0.068	2.8s	21%	33	25	25	25	25	25	13	10	10	7	20%
58	GPT-4o Mini (temp=1)	$0.0013	6.0s	9%	38	35	32	29	13	10	10	10	10	6	19%
52	DeepSeek V3 (2024-12-26)	$0.0045	4.4s	18%	25	25	25	25	25	17	17	13	10	8	19%
55	DeepSeek V3.1	$0.0028	11.8s	16%	29	25	25	25	25	13	13	13	13	10	19%
65	Llama 3.1 Nemotron 70B	$0.016	28.9s	18%	31	30	30	30	30	17	6	5	5	4	18%
60	DeepSeek V3.2	$0.0031	21.6s	14%	29	25	25	17	17	17	17	10	10	8	17%
71	Claude Sonnet 4.5	$0.047	5.4s	15%	25	25	17	17	17	17	17	13	13	13	17%
57	DeepSeek-V2 Chat	$0.0049	6.9s	14%	25	25	25	17	17	17	13	13	8	5	16%
63	Gemini 3 Flash (Preview)	$0.0074	3.0s	11%	25	17	17	17	13	13	13	13	10	10	15%
68	Claude 3 Haiku	$0.0041	4.2s	7%	33	32	17	10	10	8	7	7	7	6	14%
70	Rocinante 12B	$0.0028	11.7s	7%	33	31	17	13	10	8	6	6	6	4	13%
61	GPT-4o Mini (temp=0)	$0.0012	3.9s	12%	13	13	13	13	13	13	13	13	13	10	12%
75	Gemma 3 12B	$0.0010	15.2s	4%	29	7	6	6	5	5	5	4	3	3	7%
76	Gemma 3 27B	$0.0012	11.5s	5%	6	6	5	5	5	5	5	5	5	4	5%
80	Gemma 3 4B	$0.0007	20.4s	1%	2	2	2	1	1	1	1	1	1	1	1%
84	Mistral NeMO	$0.0056	5.0m	1%	3	3	2	2	1	1	1	0	0	0	1%
55.69%

Median	Evaluator	Top 3	Flop 3
47.5%	Correct "no violations" response	100Ministral 3 8B 100GPT-5.2 100ByteDance Seed 1.6	0DeepSeek-V2 Chat 0GPT-4o, Aug. 6th (temp=0) 0Claude Sonnet 4.5
56.1%	No hallucinated violations	100Ministral 3 8B 100GPT-4.1 100ByteDance Seed 1.6	2Mistral NeMO 3Gemma 3 4B 5Arcee AI: Trinity Large (Preview)

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)