Long text (~1594 words), small codex (11 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Z.AI GLM 5 Turbo	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5.4 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
GPT-5	100%
Grok 4.20 (Beta, Reasoning)	100%
GPT-5.4 (Reasoning, Low)	100%
Z.AI GLM 5	100%
MoonshotAI: Kimi K2.5	100%
ByteDance Seed 1.6	100%
GPT-5.4 Mini (Reasoning)	100%
o4 Mini High	100%
GPT-5.2	100%
Grok 4.1 Fast	100%
MiniMax M2.7	100%
Z.AI GLM 4.7	100%
GPT-4.1	100%
o4 Mini	100%

	Score	Cost	Time
GPT-4.1 Nano	93%	$0.0001	962ms
LFM2 24B	100%	$0.0001	1.8s
Gemini 2.5 Flash Lite	74%	$0.0003	516ms
Mistral Small 3.2 24B	72%	$0.0003	1.0s
Ministral 3 8B	90%	$0.0004	1.4s
Grok 4.20 (Beta)	77%	$0.0018	984ms
GPT-5.4 Nano	66%	$0.0004	1.6s
Gemini 3.1 Flash Lite (Preview)	70%	$0.0008	882ms
Inception Mercury	100%	$0.0002	3.7s
Mistral Medium 3.1	95%	$0.0011	764ms
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.6s
GPT-4.1	100%	$0.0022	815ms
Inception Mercury 2	100%	$0.0013	2.0s
GPT-5.4 Nano (Reasoning)	100%	$0.0010	3.7s
ByteDance Seed 1.6 Flash	100%	$0.0005	7.4s
Grok 4.1 Fast	100%	$0.0009	6.1s
Hermes 3 70B	69%	$0.0009	6.2s
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0021	3.0s
Grok 4 Fast	100%	$0.0010	13.9s
Stealth: Healer Alpha	100%	$0.0000	15.5s

	Score	Cost	Speed	Stability
LFM2 24B	100%	$0.0001	1.8s	100%
Inception Mercury	100%	$0.0002	3.7s	100%
Inception Mercury 2	100%	$0.0013	2.0s	100%
GPT-5.4 Nano (Reasoning, Low)	100%	$0.0006	3.6s	100%
GPT-4.1	100%	$0.0022	815ms	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0010	3.7s	100%
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0021	3.0s	100%
Grok 4.1 Fast	100%	$0.0009	6.1s	100%
ByteDance Seed 1.6 Flash	100%	$0.0005	7.4s	100%
Stealth: Healer Alpha	100%	$0.0000	15.5s	100%
Grok 4 Fast	100%	$0.0010	13.9s	100%
Z.AI GLM 5 Turbo	100%	$0.0039	9.7s	100%
GPT-5.4 Mini (Reasoning)	100%	$0.0065	7.8s	100%
GPT-5.4 (Reasoning, Low)	100%	$0.0077	5.5s	100%
GPT-5.2	100%	$0.0083	9.6s	100%
Gemini 2.5 Flash (Reasoning)	100%	$0.0074	11.8s	100%
MiniMax M2.7	100%	$0.0020	27.5s	100%
o4 Mini	100%	$0.0085	15.4s	100%
ByteDance Seed 1.6	100%	$0.0033	31.2s	100%
GPT-5 Mini	100%	$0.0044	29.6s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
41	Claude Opus 4.6 (Reasoning)	$0.083	50.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Z.AI GLM 5 Turbo	$0.0039	9.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
46	Claude Sonnet 4.6 (Reasoning)	$0.101	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
23	GPT-5.4 (Reasoning)	$0.015	14.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	GPT-5 Mini	$0.0044	29.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	GPT-5.1	$0.012	13.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	GPT-5	$0.030	57.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	Grok 4.20 (Beta, Reasoning)	$0.015	10.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	GPT-5.4 (Reasoning, Low)	$0.0077	5.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Z.AI GLM 5	$0.0087	48.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
32	MoonshotAI: Kimi K2.5	$0.012	1.6m	100%	100	100	100	100	100	100	100	100	100	100	100%
19	ByteDance Seed 1.6	$0.0033	31.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5.4 Mini (Reasoning)	$0.0065	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	o4 Mini High	$0.018	33.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	GPT-5.2	$0.0083	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Grok 4.1 Fast	$0.0009	6.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	MiniMax M2.7	$0.0020	27.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	Z.AI GLM 4.7	$0.015	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
5	GPT-4.1	$0.0022	815ms	100%	100	100	100	100	100	100	100	100	100	100	100%
18	o4 Mini	$0.0085	15.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	ByteDance Seed 2.0 Mini	$0.0018	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Gemini 2.5 Flash (Reasoning)	$0.0074	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Grok 4 Fast	$0.0010	13.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Stealth: Healer Alpha	$0.0000	15.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Mini (Reasoning, Low)	$0.0021	3.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Nemotron 3 Super	$0.0000	47.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Inception Mercury 2	$0.0013	2.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
26	GPT-5 Nano	$0.0033	1.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5.4 Nano (Reasoning)	$0.0010	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Inception Mercury	$0.0002	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	GPT-5.4 Nano (Reasoning, Low)	$0.0006	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Nemotron 3 Nano	$0.0016	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
9	ByteDance Seed 1.6 Flash	$0.0005	7.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	LFM2 24B	$0.0001	1.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Mistral Medium 3.1	$0.0011	764ms	70%	100	100	100	100	100	100	100	100	100	50	95%
36	Claude Sonnet 4.6	$0.019	12.6s	59%	100	100	100	100	100	100	100	100	100	32	93%
34	GPT-4.1 Nano	$0.0001	962ms	58%	100	100	100	100	100	100	100	100	100	30	93%
37	Stealth: Hunter Alpha	$0.0000	51.9s	57%	100	100	100	100	100	100	100	100	100	28	93%
40	Aion 2.0	$0.0062	58.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
35	Gemini 2.5 Flash Lite (Reasoning)	$0.0020	14.3s	55%	100	100	100	100	100	100	100	100	100	25	93%
38	Ministral 3 8B	$0.0004	1.4s	41%	100	100	100	100	100	100	100	100	100	2	90%
39	MiniMax M2.5	$0.0020	24.5s	49%	100	100	100	100	100	100	100	100	50	25	88%
52	Gemini 2.5 Pro	$0.045	32.3s	37%	100	100	100	100	100	100	100	100	25	17	84%
65	Grok 4	$0.063	1.6m	35%	100	100	100	100	100	100	100	100	25	13	84%
42	Grok 4.20 (Beta)	$0.0018	984ms	29%	100	100	100	100	100	100	100	33	33	6	77%
44	Gemini 2.5 Flash Lite	$0.0003	516ms	20%	100	100	100	100	100	100	100	17	17	6	74%
50	Mistral Small 3.2 24B	$0.0003	1.0s	14%	100	100	100	100	100	100	100	6	6	6	72%
43	Gemini 3.1 Flash Lite (Preview)	$0.0008	882ms	26%	100	100	100	100	100	100	42	25	25	13	70%
48	Mistral Small 4 (Reasoning)	$0.0018	16.7s	24%	100	100	100	100	100	100	25	25	25	17	69%
47	Hermes 3 70B	$0.0009	6.2s	21%	100	100	100	100	100	100	42	33	7	6	69%
54	Qwen 3 32B	$0.0015	1.3m	21%	100	100	100	100	100	100	25	25	17	13	68%
51	GPT-5.4 Nano	$0.0004	1.6s	17%	100	100	100	100	100	100	31	17	8	7	66%
49	Ministral 8B	$0.0004	4.4s	22%	100	100	100	100	100	50	27	27	26	26	66%
114	Qwen 3.5 27B	$0.052	4.7m	18%	100	100	100	100	100	50	25	25	17	13	63%
53	Z.AI GLM 4.5	$0.0022	16.0s	15%	100	100	100	100	100	32	30	29	25	3	62%
98	Z.AI GLM 4.7 Flash	$0.0040	3.1m	14%	100	100	100	100	100	25	25	25	25	17	62%
108	Qwen 3.5 9B	$0.0034	4.5m	22%	100	100	100	50	50	50	50	50	25	25	60%
85	Z.AI GLM 4.6	$0.033	1.6m	14%	100	100	100	100	50	50	29	25	25	7	59%
113	Gemini 3.1 Pro (Preview)	$0.110	1.3m	7%	100	100	100	100	25	25	25	25	25	25	55%
112	Qwen 3.5 397B A17B	$0.018	3.9m	5%	100	100	100	100	25	25	25	17	13	13	52%
58	Arcee AI: Trinity Mini	$0.0003	8.2s	3%	100	100	100	100	25	17	17	17	13	6	49%
45	Claude Haiku 4.5	$0.0042	3.4s	48%	50	50	50	50	50	50	50	50	50	42	49%
116	Qwen 3.5 122B	$0.101	4.5m	7%	100	100	100	25	25	25	25	25	17	17	46%
111	Qwen 3.5 35B	$0.047	2.4m	5%	100	100	100	25	25	17	17	17	13	13	43%
55	Claude Sonnet 4	$0.012	4.5s	30%	50	42	42	42	42	38	35	17	17	13	34%
95	Gemini 3 Flash (Preview, Reasoning)	$0.025	44.3s	14%	100	25	25	25	25	25	25	25	25	17	32%
115	Gemini 3 Pro (Preview)	$0.131	1.5m	14%	100	25	25	25	25	25	25	25	25	17	32%
60	Claude Sonnet 4.5	$0.016	7.6s	29%	35	35	33	33	33	33	33	32	32	10	31%
91	Claude Opus 4	$0.056	6.7s	27%	50	50	42	42	42	38	13	13	13	10	31%
79	WizardLM 2 8x22b	$0.0021	47.3s	8%	100	50	50	50	30	8	6	6	5	4	31%
67	Llama 3.1 Nemotron 70B	$0.0040	22.1s	15%	100	38	35	32	31	31	28	5	5	3	31%
63	GPT-5.4 Mini	$0.0013	787ms	13%	100	25	25	25	25	25	25	25	17	13	30%
62	GPT-4o Mini (temp=1)	$0.0004	4.8s	15%	100	38	32	32	31	29	13	13	8	6	30%
56	Ministral 3B	$0.0001	3.0s	25%	42	38	31	31	31	31	30	29	29	3	29%
70	GPT-4o, Aug. 6th (temp=1)	$0.0065	2.7s	8%	100	42	42	42	17	17	13	8	7	7	29%
61	Writer: Palmyra X5	$0.0066	13.0s	25%	42	35	32	32	31	30	30	30	29	2	29%
57	GPT-4.1 Mini	$0.0015	7.2s	25%	35	35	35	33	31	30	30	29	28	5	29%
59	Ministral 3 3B	$0.0004	5.0s	24%	38	32	30	30	29	29	29	29	27	1	27%
75	Claude Opus 4.6	$0.030	12.3s	23%	42	42	38	35	32	32	32	6	6	6	27%
66	Mistral Small 4	$0.0008	5.6s	12%	100	32	32	30	28	26	6	5	5	4	27%
69	Qwen 2.5 72B	$0.0006	9.9s	10%	100	35	31	29	25	17	13	6	6	6	27%
105	Qwen 3.5 Flash	$0.0061	1.6m	8%	100	25	25	25	17	17	17	17	13	13	27%
71	DeepSeek V3.1	$0.0011	18.9s	9%	100	31	30	29	25	13	13	8	8	7	26%
68	Qwen3 235B A22B Instruct 2507	$0.0009	27.7s	21%	42	35	32	31	30	29	28	5	4	4	24%
78	Mistral NeMO	$0.0005	7.8s	1%	100	100	6	4	4	4	4	3	3	1	23%
87	ByteDance Seed 2.0 Lite	$0.0070	1.2m	23%	25	25	25	25	25	25	25	17	17	17	23%
76	Rocinante 12B	$0.0008	9.3s	4%	100	33	31	17	10	8	7	6	4	3	22%
74	Hermes 3 405B	$0.0029	6.3s	6%	100	25	17	13	13	13	10	10	10	8	22%
64	GPT-5.4	$0.0034	2.2s	22%	25	25	25	25	25	25	25	17	13	7	21%
109	Qwen 3.5 Plus (2026-02-15)	$0.020	1.9m	14%	50	31	31	31	30	10	10	8	6	2	21%
82	GPT-4o, May 13th (temp=1)	$0.018	3.4s	13%	42	32	32	29	25	10	8	8	7	6	20%
77	DeepSeek V3.2	$0.0009	27.4s	13%	42	30	29	28	17	17	13	5	5	4	19%
100	Cohere Command R+ (Aug. 2024)	$0.012	15.0s	4%	100	17	17	10	10	10	8	7	6	1	19%
93	Claude Opus 4.5	$0.021	5.2s	12%	38	38	17	17	17	13	10	10	10	10	18%
80	DeepSeek V3 (2025-03-24)	$0.0007	15.9s	7%	42	35	33	10	10	10	10	10	8	7	18%
72	DeepSeek-V2 Chat	$0.0011	4.8s	9%	50	25	17	17	13	13	10	10	10	6	17%
94	Claude 3.5 Sonnet	$0.024	6.3s	16%	25	17	17	17	17	17	17	13	13	13	16%
73	DeepSeek V3 (2024-12-26)	$0.0012	5.5s	11%	31	25	17	13	13	13	13	13	13	8	16%
99	Llama 3.1 8B	$0.0001	28.5s	5%	29	28	28	28	7	6	6	5	5	4	15%
101	Arcee AI: Trinity Large (Preview)	$0.0000	31.2s	4%	29	28	28	27	5	5	5	4	3	3	14%
84	Llama 3.1 70B	$0.0014	12.4s	8%	31	25	17	17	10	10	10	10	5	3	14%
89	Claude 3.7 Sonnet	$0.012	4.1s	12%	17	17	13	13	13	13	13	10	10	10	13%
81	Gemini 3 Flash (Preview)	$0.0018	3.3s	8%	35	13	13	13	10	10	8	8	8	8	13%
96	Mistral Small Creative	$0.0008	9.4s	3%	27	26	26	26	4	3	3	2	2	2	12%
86	Claude 3 Haiku	$0.0011	3.0s	7%	33	10	10	8	8	7	7	7	6	6	10%
102	Ministral 3 14B	$0.0009	12.9s	2%	27	26	26	3	3	3	3	2	2	1	10%
83	Gemini 2.5 Flash	$0.0015	1.7s	9%	13	13	13	10	10	10	8	7	6	5	9%
88	Gemma 3 12B	$0.0001	6.8s	8%	17	8	8	8	8	8	7	7	7	7	9%
92	Gemma 3 27B	$0.0003	8.8s	8%	10	10	10	10	8	8	8	8	7	5	9%
97	GPT-4o, Aug. 6th (temp=0)	$0.0071	2.9s	8%	10	10	8	8	8	8	7	7	6	6	8%
90	GPT-4o Mini (temp=0)	$0.0005	5.4s	8%	8	8	8	8	8	8	8	8	6	6	8%
106	Mistral Large 2	$0.011	15.1s	4%	29	6	5	5	5	4	4	3	3	3	6%
103	Mistral Large 3	$0.0026	12.1s	5%	6	6	6	6	6	5	5	5	3	3	5%
110	GPT-4o, May 13th (temp=0)	$0.035	9.4s	4%	8	8	7	7	6	3	1	1	1	1	4%
107	Mistral Large	$0.011	15.0s	4%	6	6	5	5	4	4	4	4	3	3	4%
104	Gemma 3 4B	$0.0002	11.0s	3%	4	3	3	3	3	3	3	2	2	2	3%
56.50%

Median	Evaluator	Top 3	Flop 3
47.5%	Correct "no violations" response	100Stealth: Healer Alpha 100Z.AI GLM 5 100ByteDance Seed 2.0 Mini	0GPT-4o Mini (temp=0) 0GPT-4o, May 13th (temp=0) 0Gemini 2.5 Flash
61.9%	No hallucinated violations	100Stealth: Healer Alpha 100Z.AI GLM 5 100Claude Opus 4.6 (Reasoning)	4Ministral 3 14B 4Mistral Small Creative 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Long text (~1594 words), small codex (11 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)