Short text (~524 words), big codex (51 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Gemini 3.1 Pro (Preview)	100%
Z.AI GLM 5 Turbo	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5 Mini	100%
Claude Opus 4.6	100%
ByteDance Seed 1.6	100%
o4 Mini High	100%
Grok 4.1 Fast	100%
GPT-4.1	100%
o4 Mini	100%
Gemini 2.5 Flash Lite (Reasoning)	100%
Nemotron 3 Super	100%
Inception Mercury 2	100%
GPT-5.4 Nano (Reasoning)	100%
ByteDance Seed 1.6 Flash	100%
LFM2 24B	100%
Ministral 3 8B	93%
GPT-5.1	93%
GPT-5.4 Mini (Reasoning)	93%

	Score	Cost	Time
LFM2 24B	100%	$0.0001	2.4s
Ministral 8B	68%	$0.0003	3.4s
Ministral 3 8B	93%	$0.0004	1.2s
GPT-5.4 Nano	73%	$0.0004	1.4s
Inception Mercury	93%	$0.0002	5.4s
GPT-5.4 Nano (Reasoning, Low)	90%	$0.0006	3.2s
GPT-4.1	100%	$0.0020	737ms
ByteDance Seed 1.6 Flash	100%	$0.0005	6.7s
GPT-5.4 Nano (Reasoning)	100%	$0.0010	5.1s
Arcee AI: Trinity Mini	83%	$0.0003	15.8s
Grok 4.1 Fast	100%	$0.0012	9.6s
Gemini 2.5 Flash Lite (Reasoning)	100%	$0.0011	6.6s
Hermes 3 405B	75%	$0.0026	1.7s
Qwen 3 32B	93%	$0.0004	10.6s
Inception Mercury 2	100%	$0.0024	3.6s
GPT-5.4 Mini (Reasoning, Low)	85%	$0.0023	3.5s
Mistral Small 4 (Reasoning)	93%	$0.0015	12.6s
Z.AI GLM 5 Turbo	100%	$0.0034	7.9s
ByteDance Seed 1.6	100%	$0.0021	16.2s
Gemini 2.5 Flash (Reasoning)	93%	$0.0047	7.7s

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
15	Claude Opus 4.6 (Reasoning)	$0.041	24.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Gemini 3.1 Pro (Preview)	$0.045	32.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Z.AI GLM 5 Turbo	$0.0034	7.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Claude Sonnet 4.6 (Reasoning)	$0.044	35.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5 Mini	$0.0044	28.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Claude Opus 4.6	$0.031	12.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	ByteDance Seed 1.6	$0.0021	16.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	o4 Mini High	$0.020	41.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Grok 4.1 Fast	$0.0012	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	GPT-4.1	$0.0020	737ms	100%	100	100	100	100	100	100	100	100	100	100	100%
11	o4 Mini	$0.010	20.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Gemini 2.5 Flash Lite (Reasoning)	$0.0011	6.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Nemotron 3 Super	$0.0000	51.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury 2	$0.0024	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	GPT-5.4 Nano (Reasoning)	$0.0010	5.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	ByteDance Seed 1.6 Flash	$0.0005	6.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0001	2.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Ministral 3 8B	$0.0004	1.2s	56%	100	100	100	100	100	100	100	100	100	27	93%
25	GPT-5.1	$0.014	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
23	GPT-5.4 Mini (Reasoning)	$0.0056	7.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
27	Aion 2.0	$0.0053	49.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
24	MiniMax M2.7	$0.0022	26.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
22	Gemini 2.5 Flash (Reasoning)	$0.0047	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
21	Mistral Small 4 (Reasoning)	$0.0015	12.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
20	Qwen 3 32B	$0.0004	10.6s	55%	100	100	100	100	100	100	100	100	100	25	93%
19	Inception Mercury	$0.0002	5.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
28	Grok 4.20 (Beta, Reasoning)	$0.021	14.0s	50%	100	100	100	100	100	100	100	100	100	17	92%
30	ByteDance Seed 2.0 Mini	$0.0021	2.0m	50%	100	100	100	100	100	100	100	100	100	17	92%
26	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.2s	42%	100	100	100	100	100	100	100	100	100	4	90%
52	Gemini 2.5 Pro	$0.080	58.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
29	GPT-5.4 Mini (Reasoning, Low)	$0.0023	3.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
40	Z.AI GLM 4.7 Flash	$0.0034	2.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
41	Nemotron 3 Nano	$0.0021	2.3m	37%	100	100	100	100	100	100	100	100	25	17	84%
31	Arcee AI: Trinity Mini	$0.0003	15.8s	31%	100	100	100	100	100	100	100	100	25	5	83%
45	Z.AI GLM 5	$0.015	1.7m	31%	100	100	100	100	100	100	100	25	25	25	78%
112	Gemini 3 Pro (Preview)	$0.140	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
39	GPT-5 Nano	$0.0040	1.3m	31%	100	100	100	100	100	100	100	25	25	25	78%
37	GPT-5.2	$0.0087	9.4s	24%	100	100	100	100	100	100	100	25	17	10	75%
34	Hermes 3 405B	$0.0026	1.7s	24%	100	100	100	100	100	100	100	25	17	10	75%
33	Claude Sonnet 4	$0.0096	2.1s	37%	100	100	100	100	100	100	50	42	42	17	75%
32	GPT-5.4 Nano	$0.0004	1.4s	34%	100	100	100	100	100	100	50	31	28	25	73%
61	Qwen 3.5 9B	$0.0031	3.9m	32%	100	100	100	100	100	50	50	50	50	25	73%
36	MiniMax M2.5	$0.0016	17.5s	27%	100	100	100	100	100	100	25	25	25	25	70%
35	Stealth: Healer Alpha	$0.0000	19.6s	27%	100	100	100	100	100	100	25	25	25	25	70%
38	Ministral 8B	$0.0003	3.4s	21%	100	100	100	100	100	100	28	27	26	3	68%
44	Cohere Command R+ (Aug. 2024)	$0.0080	3.2s	14%	100	100	100	100	100	100	25	13	10	5	65%
43	Hermes 3 70B	$0.0008	7.2s	16%	100	100	100	100	100	42	38	35	8	7	63%
58	Z.AI GLM 4.6	$0.017	48.9s	11%	100	100	100	100	100	50	28	13	4	0	59%
103	Grok 4	$0.056	1.3m	11%	100	100	100	100	100	25	17	17	17	17	59%
104	GPT-5	$0.042	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
114	Qwen 3.5 397B A17B	$0.022	4.5m	7%	100	100	100	100	25	25	25	25	25	25	55%
83	MoonshotAI: Kimi K2.5	$0.013	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
81	Z.AI GLM 4.7	$0.016	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
69	Gemini 3 Flash (Preview, Reasoning)	$0.021	38.5s	6%	100	100	100	100	25	25	25	25	25	17	54%
42	Claude Haiku 4.5	$0.0046	5.1s	35%	100	50	50	50	50	50	50	50	50	42	54%
63	Claude 3.5 Sonnet	$0.020	3.7s	5%	100	100	100	100	25	25	25	17	17	13	52%
115	Qwen 3.5 27B	$0.075	6.6m	11%	100	100	100	42	42	25	25	25	25	17	50%
50	GPT-4o, Aug. 6th (temp=1)	$0.0048	2.0s	11%	100	100	100	42	38	33	25	25	13	10	49%
65	Stealth: Hunter Alpha	$0.0000	55.9s	7%	100	100	100	25	25	25	25	25	25	17	47%
55	Grok 4 Fast	$0.0014	19.2s	7%	100	100	100	25	25	25	25	25	25	17	47%
116	Qwen 3.5 122B	$0.111	6.2m	7%	100	100	100	25	25	25	25	17	17	17	45%
57	Z.AI GLM 4.5	$0.0019	11.5s	7%	100	100	100	38	29	25	25	25	4	2	45%
46	WizardLM 2 8x22b	$0.0018	23.8s	24%	100	50	50	50	42	42	25	25	25	25	43%
64	GPT-4.1 Nano	$0.0005	6.5s	1%	100	100	100	100	25	2	1	1	1	1	43%
106	Qwen 3.5 35B	$0.026	1.5m	10%	100	100	25	25	25	25	25	25	25	17	39%
60	Writer: Palmyra X5	$0.0026	4.4s	9%	100	100	50	29	25	25	17	17	17	13	39%
75	Claude 3.7 Sonnet	$0.011	3.1s	6%	100	100	42	17	17	17	17	17	17	17	36%
56	Qwen 2.5 72B	$0.0005	9.4s	16%	100	42	35	31	31	30	29	29	13	13	35%
54	Claude Sonnet 4.6	$0.014	7.6s	29%	42	38	38	35	35	33	33	33	32	13	33%
72	Claude Opus 4.5	$0.016	2.9s	14%	100	25	25	25	25	25	25	25	25	25	33%
99	Qwen 3.5 Flash	$0.0062	1.8m	14%	100	25	25	25	25	25	25	25	25	25	33%
78	ByteDance Seed 2.0 Lite	$0.0049	49.0s	14%	100	25	25	25	25	25	25	25	25	25	33%
59	GPT-5.4 Mini	$0.0011	683ms	14%	100	25	25	25	25	25	25	25	25	25	33%
47	Mistral Medium 3.1	$0.0020	3.6s	29%	38	33	33	31	31	30	30	30	30	30	31%
96	GPT-5.4 (Reasoning)	$0.027	27.0s	13%	100	25	25	25	25	25	25	17	17	17	30%
51	Mistral Small Creative	$0.0005	4.3s	26%	42	30	29	29	29	29	29	28	28	28	30%
49	Ministral 3 3B	$0.0003	3.0s	28%	32	32	32	31	30	29	28	28	28	28	30%
48	Ministral 3B	$0.0001	2.9s	29%	32	31	30	30	30	29	29	28	28	27	29%
71	Claude Sonnet 4.5	$0.010	3.1s	13%	100	25	25	25	25	25	17	17	17	17	29%
53	GPT-4.1 Mini	$0.0013	5.4s	26%	33	32	32	31	31	31	30	30	30	6	29%
62	Arcee AI: Trinity Large (Preview)	$0.0000	43.4s	27%	29	28	28	28	28	28	28	28	27	26	28%
101	Claude Opus 4	$0.049	4.0s	21%	50	25	25	25	25	25	25	25	25	25	28%
87	GPT-5.4 (Reasoning, Low)	$0.012	10.6s	9%	100	25	25	25	17	17	17	17	17	17	27%
68	Qwen3 235B A22B Instruct 2507	$0.0004	5.9s	11%	100	25	25	25	25	17	17	17	13	13	27%
79	Grok 4.20 (Beta)	$0.0041	2.6s	8%	100	38	30	30	28	10	7	5	5	3	26%
89	Rocinante 12B	$0.0011	20.4s	5%	100	50	25	17	13	10	10	8	7	0	24%
111	Qwen 3.5 Plus (2026-02-15)	$0.022	2.2m	20%	42	42	33	30	29	26	10	10	8	3	23%
66	DeepSeek V3 (2024-12-26)	$0.0010	4.1s	19%	50	25	25	25	25	25	25	13	7	6	23%
67	Ministral 3 14B	$0.0010	17.5s	22%	28	28	28	27	27	27	27	25	3	2	22%
73	DeepSeek V3.1	$0.0009	16.3s	17%	38	30	30	25	25	17	17	17	13	6	22%
93	Llama 3.1 70B	$0.0014	14.4s	3%	100	32	31	10	8	6	6	6	5	4	21%
70	DeepSeek V3.2	$0.0007	12.0s	17%	33	30	25	25	25	17	17	17	8	8	20%
76	Gemini 3.1 Flash Lite (Preview)	$0.0010	1.4s	14%	42	35	17	17	17	17	17	17	13	13	20%
74	Llama 3.1 Nemotron 70B	$0.0039	22.2s	22%	31	31	30	30	29	29	8	5	5	3	20%
92	Mistral NeMO	$0.0005	4.7s	2%	100	42	30	6	5	5	4	3	2	1	20%
82	GPT-4o, Aug. 6th (temp=0)	$0.0052	2.6s	15%	33	25	25	17	17	17	17	17	13	13	19%
77	Llama 3.1 8B	$0.0002	25.8s	20%	30	29	29	28	27	26	8	7	3	1	19%
86	Mistral Large 3	$0.0022	8.7s	13%	32	32	32	32	30	5	5	5	4	4	18%
91	Mistral Large 2	$0.0093	9.4s	14%	33	31	30	29	29	8	4	4	4	3	17%
85	Mistral Small 4	$0.0007	9.2s	13%	31	31	30	30	28	6	5	4	4	3	17%
80	Gemini 3 Flash (Preview)	$0.0012	2.0s	16%	25	17	17	17	17	17	17	13	13	13	16%
84	GPT-5.4	$0.0039	2.7s	16%	17	17	17	17	17	17	17	17	17	13	16%
88	Gemini 2.5 Flash	$0.0014	1.5s	11%	29	25	17	17	13	13	13	10	10	8	15%
95	Claude 3 Haiku	$0.0012	3.6s	5%	33	33	32	10	7	7	7	7	6	5	15%
100	GPT-4o, May 13th (temp=1)	$0.016	2.2s	11%	31	17	17	17	13	13	10	10	10	10	15%
90	DeepSeek V3 (2025-03-24)	$0.0006	8.0s	11%	25	25	17	13	13	13	13	10	7	7	14%
98	GPT-4o Mini (temp=0)	$0.0006	8.7s	6%	32	32	26	7	7	7	6	6	6	5	13%
94	GPT-4o Mini (temp=1)	$0.0004	5.3s	7%	38	33	13	10	10	8	8	6	5	4	13%
97	DeepSeek-V2 Chat	$0.0012	6.8s	9%	25	17	13	13	10	10	8	6	6	5	11%
109	Mistral Large	$0.0088	11.3s	5%	32	29	7	7	7	6	6	4	4	4	11%
102	Mistral Small 3.2 24B	$0.0004	4.0s	5%	29	7	7	7	6	6	6	6	6	5	9%
113	GPT-4o, May 13th (temp=0)	$0.040	11.9s	8%	26	13	13	10	10	10	2	1	1	1	8%
105	Gemma 3 12B	$0.0003	13.1s	5%	29	7	7	6	6	6	6	5	5	2	8%
107	Gemini 2.5 Flash Lite	$0.0004	1.7s	4%	10	10	5	5	5	4	4	4	4	4	5%
108	Gemma 3 27B	$0.0003	11.4s	5%	6	6	6	6	5	5	5	5	5	5	5%
110	Gemma 3 4B	$0.0002	13.0s	2%	5	5	5	3	3	2	2	1	1	1	3%
53.24%

Median	Evaluator	Top 3	Flop 3
45.0%	Correct "no violations" response	100Claude Opus 4.6 100o4 Mini High 100Claude Opus 4.6 (Reasoning)	0GPT-5.4 0Gemma 3 27B 0Gemma 3 4B
59.4%	No hallucinated violations	100Gemini 2.5 Flash Lite (Reasoning) 100Claude Opus 4.6 (Reasoning) 100Nemotron 3 Super	4Ministral 3 14B 5Arcee AI: Trinity Large (Preview) 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)