Short text (~524 words), big codex (51 entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Gemini 3.1 Pro (Preview)	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5 Mini	100%
Claude Opus 4.6	100%
ByteDance Seed 1.6	100%
o4 Mini High	100%
Grok 4.1 Fast	100%
GPT-4.1	100%
o4 Mini	100%
Gemini 2.5 Flash Lite (Reasoning)	100%
ByteDance Seed 1.6 Flash	100%
Ministral 3 8B	93%
GPT-5.1	93%
Aion 2.0	93%
Gemini 2.5 Flash (Reasoning)	93%
Gemini 2.5 Pro	85%
Z.AI GLM 4.7 Flash	85%
Arcee AI: Trinity Mini	83%
Z.AI GLM 5	78%

	Score	Cost	Time
Ministral 8B	68%	$0.0003	3.4s
Ministral 3 8B	93%	$0.0004	1.2s
GPT-4.1	100%	$0.0020	737ms
Hermes 3 405B	75%	$0.0026	1.7s
ByteDance Seed 1.6 Flash	100%	$0.0005	6.7s
Gemini 2.5 Flash Lite (Reasoning)	100%	$0.0011	6.6s
Grok 4.1 Fast	100%	$0.0012	9.6s
Arcee AI: Trinity Mini	83%	$0.0003	15.8s
Cohere Command R+ (Aug. 2024)	65%	$0.0080	3.2s
Gemini 2.5 Flash (Reasoning)	93%	$0.0047	7.7s
GPT-5.2	75%	$0.0087	9.4s
Claude Sonnet 4	75%	$0.0096	2.1s
ByteDance Seed 1.6	100%	$0.0021	16.2s
Minimax M2.5	70%	$0.0016	17.5s
GPT-5.1	93%	$0.014	17.8s
GPT-5 Mini	100%	$0.0044	28.8s
o4 Mini	100%	$0.010	20.6s
Aion 2.0	93%	$0.0053	49.2s
Claude Opus 4.6	100%	$0.031	12.1s
o4 Mini High	100%	$0.020	41.1s

	Score	Cost	Speed	Stability
GPT-4.1	100%	$0.0020	737ms	100%
ByteDance Seed 1.6 Flash	100%	$0.0005	6.7s	100%
Gemini 2.5 Flash Lite (Reasoning)	100%	$0.0011	6.6s	100%
Grok 4.1 Fast	100%	$0.0012	9.6s	100%
ByteDance Seed 1.6	100%	$0.0021	16.2s	100%
GPT-5 Mini	100%	$0.0044	28.8s	100%
o4 Mini	100%	$0.010	20.6s	100%
Claude Opus 4.6	100%	$0.031	12.1s	100%
o4 Mini High	100%	$0.020	41.1s	100%
Claude Opus 4.6 (Reasoning)	100%	$0.041	24.0s	100%
Gemini 3.1 Pro (Preview)	100%	$0.045	32.9s	100%
Claude Sonnet 4.6 (Reasoning)	100%	$0.044	35.6s	100%
Ministral 3 8B	93%	$0.0004	1.2s	56%
Gemini 2.5 Flash (Reasoning)	93%	$0.0047	7.7s	55%
GPT-5.1	93%	$0.014	17.8s	55%
Aion 2.0	93%	$0.0053	49.2s	55%
Arcee AI: Trinity Mini	83%	$0.0003	15.8s	31%
Claude Sonnet 4	75%	$0.0096	2.1s	37%
Hermes 3 405B	75%	$0.0026	1.7s	24%
GPT-5.2	75%	$0.0087	9.4s	24%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
10	Claude Opus 4.6 (Reasoning)	$0.041	24.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Gemini 3.1 Pro (Preview)	$0.045	32.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Claude Sonnet 4.6 (Reasoning)	$0.044	35.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	GPT-5 Mini	$0.0044	28.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Claude Opus 4.6	$0.031	12.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	ByteDance Seed 1.6	$0.0021	16.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	o4 Mini High	$0.020	41.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Grok 4.1 Fast	$0.0012	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	GPT-4.1	$0.0020	737ms	100%	100	100	100	100	100	100	100	100	100	100	100%
7	o4 Mini	$0.010	20.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Gemini 2.5 Flash Lite (Reasoning)	$0.0011	6.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	ByteDance Seed 1.6 Flash	$0.0005	6.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	Ministral 3 8B	$0.0004	1.2s	56%	100	100	100	100	100	100	100	100	100	27	93%
15	GPT-5.1	$0.014	17.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
16	Aion 2.0	$0.0053	49.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
14	Gemini 2.5 Flash (Reasoning)	$0.0047	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
37	Gemini 2.5 Pro	$0.080	58.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
27	Z.AI GLM 4.7 Flash	$0.0034	2.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
17	Arcee AI: Trinity Mini	$0.0003	15.8s	31%	100	100	100	100	100	100	100	100	25	5	83%
28	Z.AI GLM 5	$0.015	1.7m	31%	100	100	100	100	100	100	100	25	25	25	78%
83	Gemini 3 Pro (Preview)	$0.140	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
24	GPT-5 Nano	$0.0040	1.3m	31%	100	100	100	100	100	100	100	25	25	25	78%
20	GPT-5.2	$0.0087	9.4s	24%	100	100	100	100	100	100	100	25	17	10	75%
19	Hermes 3 405B	$0.0026	1.7s	24%	100	100	100	100	100	100	100	25	17	10	75%
18	Claude Sonnet 4	$0.0096	2.1s	37%	100	100	100	100	100	100	50	42	42	17	75%
21	Minimax M2.5	$0.0016	17.5s	27%	100	100	100	100	100	100	25	25	25	25	70%
22	Ministral 8B	$0.0003	3.4s	21%	100	100	100	100	100	100	28	27	26	3	68%
26	Cohere Command R+ (Aug. 2024)	$0.0080	3.2s	14%	100	100	100	100	100	100	25	13	10	5	65%
25	Hermes 3 70B	$0.0008	7.2s	16%	100	100	100	100	100	42	38	35	8	7	63%
44	Z.AI GLM 4.6	$0.017	48.9s	11%	100	100	100	100	100	50	28	13	4	0	59%
78	Grok 4	$0.056	1.3m	11%	100	100	100	100	100	25	17	17	17	17	59%
79	GPT-5	$0.042	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
84	Qwen 3.5 397B A17B	$0.022	4.5m	7%	100	100	100	100	25	25	25	25	25	25	55%
67	MoonshotAI: Kimi K2.5	$0.013	1.6m	7%	100	100	100	100	25	25	25	25	25	25	55%
64	Z.AI GLM 4.7	$0.016	1.4m	7%	100	100	100	100	25	25	25	25	25	25	55%
53	Gemini 3 Flash (Preview, Reasoning)	$0.021	38.5s	6%	100	100	100	100	25	25	25	25	25	17	54%
23	Claude Haiku 4.5	$0.0046	5.1s	35%	100	50	50	50	50	50	50	50	50	42	54%
42	Claude 3.5 Sonnet	$0.020	3.7s	5%	100	100	100	100	25	25	25	17	17	13	52%
33	GPT-4o, Aug. 6th (temp=1)	$0.0048	2.0s	11%	100	100	100	42	38	33	25	25	13	10	49%
39	Grok 4 Fast	$0.0014	19.2s	7%	100	100	100	25	25	25	25	25	25	17	47%
40	Z.AI GLM 4.5	$0.0019	11.5s	7%	100	100	100	38	29	25	25	25	4	2	45%
30	WizardLM 2 8x22b	$0.0018	23.8s	24%	100	50	50	50	42	42	25	25	25	25	43%
43	GPT-4.1 Nano	$0.0005	6.5s	1%	100	100	100	100	25	2	1	1	1	1	43%
41	Writer: Palmyra X5	$0.0026	4.4s	9%	100	100	50	29	25	25	17	17	17	13	39%
51	Claude 3.7 Sonnet	$0.011	3.1s	6%	100	100	42	17	17	17	17	17	17	17	36%
38	Qwen 2.5 72B	$0.0005	9.4s	16%	100	42	35	31	31	30	29	29	13	13	35%
36	Claude Sonnet 4.6	$0.014	7.6s	29%	42	38	38	35	35	33	33	33	32	13	33%
49	Claude Opus 4.5	$0.016	2.9s	14%	100	25	25	25	25	25	25	25	25	25	33%
29	Mistral Medium 3.1	$0.0020	3.6s	29%	38	33	33	31	31	30	30	30	30	30	31%
34	Mistral Small Creative	$0.0005	4.3s	26%	42	30	29	29	29	29	29	28	28	28	30%
32	Ministral 3 3B	$0.0003	3.0s	28%	32	32	32	31	30	29	28	28	28	28	30%
31	Ministral 3B	$0.0001	2.9s	29%	32	31	30	30	30	29	29	28	28	27	29%
48	Claude Sonnet 4.5	$0.010	3.1s	13%	100	25	25	25	25	25	17	17	17	17	29%
35	GPT-4.1 Mini	$0.0013	5.4s	26%	33	32	32	31	31	31	30	30	30	6	29%
46	Arcee AI: Trinity Large (Preview)	$0.0000	43.4s	27%	29	28	28	28	28	28	28	28	27	26	28%
72	Claude Opus 4	$0.049	4.0s	21%	50	25	25	25	25	25	25	25	25	25	28%
62	Rocinante 12B	$0.0011	20.4s	5%	100	50	25	17	13	10	10	8	7	0	24%
82	Qwen 3.5 Plus (2026-02-15)	$0.022	2.2m	20%	42	42	33	30	29	26	10	10	8	3	23%
45	DeepSeek V3 (2024-12-26)	$0.0010	4.1s	19%	50	25	25	25	25	25	25	13	7	6	23%
47	Ministral 3 14B	$0.0010	17.5s	22%	28	28	28	27	27	27	27	25	3	2	22%
52	DeepSeek V3.1	$0.0009	16.3s	17%	38	30	30	25	25	17	17	17	13	6	22%
68	Llama 3.1 70B	$0.0014	14.4s	3%	100	32	31	10	8	6	6	6	5	4	21%
50	DeepSeek V3.2	$0.0007	12.0s	17%	33	30	25	25	25	17	17	17	8	8	20%
54	Llama 3.1 Nemotron 70B	$0.0039	22.2s	22%	31	31	30	30	29	29	8	5	5	3	20%
63	Mistral NeMO	$0.0005	4.7s	2%	100	42	30	6	5	5	4	3	2	1	20%
57	GPT-4o, Aug. 6th (temp=0)	$0.0052	2.6s	15%	33	25	25	17	17	17	17	17	13	13	19%
56	Llama 3.1 8B	$0.0002	25.8s	20%	30	29	29	28	27	26	8	7	3	1	19%
58	Mistral Large 3	$0.0022	8.7s	13%	32	32	32	32	30	5	5	5	4	4	18%
61	Mistral Large 2	$0.0093	9.4s	14%	33	31	30	29	29	8	4	4	4	3	17%
55	Gemini 3 Flash (Preview)	$0.0012	2.0s	16%	25	17	17	17	17	17	17	13	13	13	16%
59	Gemini 2.5 Flash	$0.0014	1.5s	11%	29	25	17	17	13	13	13	10	10	8	15%
66	Claude 3 Haiku	$0.0012	3.6s	5%	33	33	32	10	7	7	7	7	6	5	15%
71	GPT-4o, May 13th (temp=1)	$0.016	2.2s	11%	31	17	17	17	13	13	10	10	10	10	15%
60	DeepSeek V3 (2025-03-24)	$0.0006	8.0s	11%	25	25	17	13	13	13	13	10	7	7	14%
70	GPT-4o Mini (temp=0)	$0.0006	8.7s	6%	32	32	26	7	7	7	6	6	6	5	13%
65	GPT-4o Mini (temp=1)	$0.0004	5.3s	7%	38	33	13	10	10	8	8	6	5	4	13%
69	DeepSeek-V2 Chat	$0.0012	6.8s	9%	25	17	13	13	10	10	8	6	6	5	11%
77	Mistral Large	$0.0088	11.3s	5%	32	29	7	7	7	6	6	4	4	4	11%
73	Mistral Small 3.2 24B	$0.0004	4.0s	5%	29	7	7	7	6	6	6	6	6	5	9%
81	GPT-4o, May 13th (temp=0)	$0.040	11.9s	8%	26	13	13	10	10	10	2	1	1	1	8%
75	Gemma 3 12B	$0.0003	13.1s	5%	29	7	7	6	6	6	6	5	5	2	8%
74	Gemini 2.5 Flash Lite	$0.0004	1.7s	4%	10	10	5	5	5	4	4	4	4	4	5%
76	Gemma 3 27B	$0.0003	11.4s	5%	6	6	6	6	5	5	5	5	5	5	5%
80	Gemma 3 4B	$0.0002	13.0s	2%	5	5	5	3	3	2	2	1	1	1	3%
48.95%

Median	Evaluator	Top 3	Flop 3
40.0%	Correct "no violations" response	100ByteDance Seed 1.6 Flash 100ByteDance Seed 1.6 100Claude Opus 4.6 (Reasoning)	0Gemini 2.5 Flash Lite 0DeepSeek V3 (2025-03-24) 0Gemini 3 Flash (Preview)
50.8%	No hallucinated violations	100Claude Sonnet 4.6 (Reasoning) 100Claude Opus 4.6 (Reasoning) 100Claude Opus 4.6	4Ministral 3 14B 5Arcee AI: Trinity Large (Preview) 5Gemma 3 4B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)