Short text (~524 words), big codex (51 detailed entries)

Codex Red Herring (False Positive Detection)

Tests whether models correctly report "no violations" when a codex is fully consistent with the prose passage. Models that hallucinate false violations (false positives) fail. Uses a 2×2 matrix of text length × codex size, with bare and detailed-entry variants.

Hallucination

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Muse Spark 1.2 (Reasoning, Medium)	100%
Qwen 3.7 Max	100%
Muse Spark 1.3 (Reasoning, Medium)	100%
GPT-5.4 (Reasoning)	100%
Muse Spark 1.1 (Reasoning, Medium)	100%
Z.AI GLM 5.2 (Reasoning, High)	100%
Z.AI GLM 5 Turbo	100%
Claude Opus 4.8 (Reasoning)	100%
Claude Opus 4.6	100%
Muse Spark 1.1 (Reasoning, Minimal)	100%
GPT-5 Mini	100%
Grok 4.5 (Reasoning, Low)	100%
Claude Sonnet 4.6	100%
GPT-5.6 Luna (Reasoning)	100%
Claude Opus 5	100%
GPT-5.4 Mini (Reasoning)	100%
Qwen 3.5 Plus (2026-04-20)	100%
Gemma 4 31B (Reasoning)	100%
Claude Opus 4.5	100%
ByteDance Seed 1.6	100%

	Score	Cost	Time
GPT-5.4 Nano	74%	$0.0006	1.4s
Z.AI GLM 5.3 Flash (Reasoning, Low)	100%	$0.0006	3.7s
GPT-5.4 Nano (Reasoning, Low)	93%	$0.0008	3.4s
Ministral 8B	100%	$0.0013	564ms
Gemma 4 26B	88%	$0.0012	5.3s
Arcee AI: Trinity Mini	100%	$0.0007	6.9s
Gemma 4 31B	100%	$0.0018	3.1s
Ministral 3 8B	100%	$0.0020	589ms
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0038	2.7s
Laguna XS 2.1	100%	$0.0009	21.5s
ByteDance Seed 1.6 Flash	93%	$0.0013	7.7s
Inception Mercury 2	100%	$0.0031	3.7s
GPT-5.4 Nano (Reasoning)	100%	$0.0020	8.1s
Laguna S 2.1	95%	$0.0007	31.1s
DeepSeek V4 Flash (Reasoning)	100%	$0.0009	19.7s
GPT-5.4 Mini (Reasoning)	100%	$0.0063	5.2s
Gemini 2.5 Flash Lite (Reasoning)	85%	$0.0023	14.2s
Mistral Small 4 (Reasoning)	93%	$0.0028	23.8s
GPT-4.1	100%	$0.0086	1.0s
Mistral Large 3	100%	$0.0066	1.4s

	Score	Cost	Speed	Stability
Ministral 8B	100%	$0.0013	564ms	100%
Ministral 3 8B	100%	$0.0020	589ms	100%
Z.AI GLM 5.3 Flash (Reasoning, Low)	100%	$0.0006	3.7s	100%
Gemma 4 31B	100%	$0.0018	3.1s	100%
Arcee AI: Trinity Mini	100%	$0.0007	6.9s	100%
Inception Mercury 2	100%	$0.0031	3.7s	100%
GPT-5.4 Mini (Reasoning, Low)	100%	$0.0038	2.7s	100%
GPT-5.4 Nano (Reasoning)	100%	$0.0020	8.1s	100%
Mistral Large 3	100%	$0.0066	1.4s	100%
GPT-5.4 Mini (Reasoning)	100%	$0.0063	5.2s	100%
GPT-4.1	100%	$0.0086	1.0s	100%
Thinking Machines Inkling	100%	$0.0087	1.6s	100%
GPT-5.6 Luna (Reasoning)	100%	$0.0075	5.2s	100%
Gemini 3.6 Flash (Reasoning, Minimal)	100%	$0.0099	862ms	100%
DeepSeek V4 Flash (Reasoning)	100%	$0.0009	19.7s	100%
Laguna XS 2.1	100%	$0.0009	21.5s	100%
MiniMax M2.7	100%	$0.0045	17.4s	100%
Z.AI GLM 5.2 (Reasoning, High)	100%	$0.0063	15.8s	100%
Z.AI GLM 5 Turbo	100%	$0.0077	13.6s	100%
ByteDance Seed 1.6	100%	$0.0050	19.7s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
31	Muse Spark 1.2 (Reasoning, Medium)	$0.027	16.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
42	Qwen 3.7 Max	$0.064	58.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
34	Muse Spark 1.3 (Reasoning, Medium)	$0.026	31.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
36	GPT-5.4 (Reasoning)	$0.032	23.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
27	Muse Spark 1.1 (Reasoning, Medium)	$0.020	11.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Z.AI GLM 5.2 (Reasoning, High)	$0.0063	15.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	Z.AI GLM 5 Turbo	$0.0077	13.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
48	Claude Opus 4.8 (Reasoning)	$0.122	14.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
41	Claude Opus 4.6	$0.082	9.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	Muse Spark 1.1 (Reasoning, Minimal)	$0.016	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	GPT-5 Mini	$0.0049	30.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Grok 4.5 (Reasoning, Low)	$0.013	9.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
35	Claude Sonnet 4.6	$0.043	1.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	GPT-5.6 Luna (Reasoning)	$0.0075	5.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
44	Claude Opus 5	$0.097	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	GPT-5.4 Mini (Reasoning)	$0.0063	5.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
37	Qwen 3.5 Plus (2026-04-20)	$0.017	1.4m	100%	100	100	100	100	100	100	100	100	100	100	100%
40	Gemma 4 31B (Reasoning)	$0.0030	2.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Claude Opus 4.5	$0.072	2.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	ByteDance Seed 1.6	$0.0050	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
22	GPT-5.2	$0.014	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
38	Gemma 4 26B (Reasoning)	$0.0029	2.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
32	o4 Mini High	$0.020	33.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Z.AI GLM 5.3 Flash (Reasoning, Low)	$0.0006	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	DeepSeek V4 Flash (Reasoning)	$0.0009	19.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	GPT-4.1	$0.0086	1.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Gemini 3.6 Flash (Reasoning, Minimal)	$0.0099	862ms	100%	100	100	100	100	100	100	100	100	100	100	100%
26	o4 Mini	$0.012	20.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	MiniMax M2.7	$0.0045	17.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	Xiaomi MIMO v2.5 Pro	$0.013	49.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Gemma 4 31B	$0.0018	3.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	GPT-OSS 120B	$0.0019	59.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Mistral Large 3	$0.0066	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4 Mini (Reasoning, Low)	$0.0038	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Inception Mercury 2	$0.0031	3.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	Nemotron 3 Super	$0.0000	51.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Mistral Large 2	$0.026	1.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	GPT-5 Nano	$0.0020	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	GPT-5.4 Nano (Reasoning)	$0.0020	8.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Laguna XS 2.1	$0.0009	21.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Thinking Machines Inkling	$0.0087	1.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Ministral 3 8B	$0.0020	589ms	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Arcee AI: Trinity Mini	$0.0007	6.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Ministral 8B	$0.0013	564ms	100%	100	100	100	100	100	100	100	100	100	100	100%
43	Laguna S 2.1	$0.0007	31.1s	70%	100	100	100	100	100	100	100	100	100	50	95%
45	GPT-5.4 Nano (Reasoning, Low)	$0.0008	3.4s	57%	100	100	100	100	100	100	100	100	100	29	93%
77	Qwen 3.6 Max Preview	$0.059	2.5m	55%	100	100	100	100	100	100	100	100	100	25	93%
75	Claude Opus 4.7 (Reasoning)	$0.119	10.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
55	Grok 4.3 (Reasoning)	$0.018	30.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
52	GPT-5.4 (Reasoning, Low)	$0.017	6.5s	55%	100	100	100	100	100	100	100	100	100	25	93%
59	Z.AI GLM 5	$0.020	1.3m	55%	100	100	100	100	100	100	100	100	100	25	93%
68	Gemini 2.5 Pro	$0.067	45.2s	55%	100	100	100	100	100	100	100	100	100	25	93%
51	MiniMax M2.5	$0.0049	21.4s	55%	100	100	100	100	100	100	100	100	100	25	93%
57	Aion 2.0	$0.014	1.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
54	Z.AI GLM 4.5	$0.0079	42.0s	55%	100	100	100	100	100	100	100	100	100	25	93%
49	Gemini 2.5 Flash (Reasoning)	$0.0092	11.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
62	Z.AI GLM 4.7 Flash	$0.0036	2.2m	55%	100	100	100	100	100	100	100	100	100	25	93%
50	Mistral Small 4 (Reasoning)	$0.0028	23.8s	55%	100	100	100	100	100	100	100	100	100	25	93%
46	ByteDance Seed 1.6 Flash	$0.0013	7.7s	55%	100	100	100	100	100	100	100	100	100	25	93%
47	Z.AI GLM 4.6	$0.0053	12.9s	60%	100	100	100	100	100	100	100	100	50	50	90%
56	Aion 3.0 Mini	$0.013	1.3m	60%	100	100	100	100	100	100	100	100	50	50	90%
53	Gemma 4 26B	$0.0012	5.3s	49%	100	100	100	100	100	100	100	100	50	25	88%
69	Grok 4.5 (Reasoning, High)	$0.025	31.7s	40%	100	100	100	100	100	100	100	100	25	25	85%
85	Z.AI GLM 5.1	$0.031	2.6m	40%	100	100	100	100	100	100	100	100	25	25	85%
60	GPT-5.6 Terra (Reasoning)	$0.010	4.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
64	Kimi K3 (Reasoning, Low)	$0.024	22.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
92	Claude Opus 4.8 (Reasoning, Low)	$0.121	13.9s	40%	100	100	100	100	100	100	100	100	25	25	85%
110	Thinking Machines Inkling (Reasoning)	$0.029	4.4m	40%	100	100	100	100	100	100	100	100	25	25	85%
65	GPT-5.1	$0.025	23.5s	40%	100	100	100	100	100	100	100	100	25	25	85%
66	MiniMax M3	$0.0037	1.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
176	Qwen 3.5 122B	$0.138	7.0m	40%	100	100	100	100	100	100	100	100	25	25	85%
70	Gemini 3 Flash (Preview, Reasoning)	$0.025	40.3s	40%	100	100	100	100	100	100	100	100	25	25	85%
83	DeepSeek V4 Pro (Reasoning)	$0.023	2.5m	40%	100	100	100	100	100	100	100	100	25	25	85%
61	Muse Glimmer 30B (Reasoning, Medium)	$0.0043	28.6s	40%	100	100	100	100	100	100	100	100	25	25	85%
82	Qwen 3.6 27B	$0.032	2.1m	40%	100	100	100	100	100	100	100	100	25	25	85%
58	Gemini 2.5 Flash Lite (Reasoning)	$0.0023	14.2s	40%	100	100	100	100	100	100	100	100	25	25	85%
67	Z.AI GLM 4.5 Air	$0.0033	26.9s	32%	100	100	100	100	100	100	100	100	30	3	83%
95	Qwen 3.5 9B	$0.0044	4.4m	45%	100	100	100	100	100	100	100	50	50	25	83%
72	Gemini 3.7 Flash (Reasoning, Medium)	$0.015	8.2s	31%	100	100	100	100	100	100	100	25	25	25	78%
79	Qwen 3.6 35B	$0.018	1.1m	31%	100	100	100	100	100	100	100	25	25	25	78%
73	Thinking Machines Inkling Small (Reasoning, High)	$0.0096	41.3s	31%	100	100	100	100	100	100	100	25	25	25	78%
91	Qwen 3.5 35B	$0.032	1.8m	31%	100	100	100	100	100	100	100	25	25	25	78%
76	ByteDance Seed 2.0 Lite	$0.0085	1.0m	31%	100	100	100	100	100	100	100	25	25	25	78%
71	Xiaomi MIMO v2.5	$0.0056	19.2s	31%	100	100	100	100	100	100	100	25	25	25	78%
80	Nemotron 3 Nano	$0.0022	1.6m	31%	100	100	100	100	100	100	100	25	25	25	78%
106	Kimi K2.6	$0.038	2.5m	29%	100	100	100	100	100	100	100	25	25	17	77%
99	Z.AI GLM 4.7	$0.029	2.4m	29%	100	100	100	100	100	100	100	25	25	17	77%
63	GPT-5.4 Nano	$0.0006	1.4s	37%	100	100	100	100	100	100	42	38	38	28	74%
87	Cohere Command R+ (Aug. 2024)	$0.037	14.6s	18%	100	100	100	100	100	100	100	17	17	1	73%
97	DeepSeek V4 Flash 0731 (Reasoning, High)	$0.0025	2.9m	31%	100	100	100	100	100	100	50	25	25	25	73%
74	Gemini 3.5 Flash (Reasoning, Minimal)	$0.020	1.5s	31%	100	100	100	100	100	100	50	25	25	25	73%
89	Qwen 3.5 Flash	$0.0077	2.2m	31%	100	100	100	100	100	100	50	25	25	25	73%
128	Claude Opus 4.6 (Reasoning)	$0.117	33.8s	27%	100	100	100	100	100	100	25	25	25	25	70%
86	Kimi K3 (Reasoning, High)	$0.028	42.1s	27%	100	100	100	100	100	100	25	25	25	25	70%
88	GPT-5	$0.033	46.9s	27%	100	100	100	100	100	100	25	25	25	25	70%
107	Kimi K2.5	$0.018	2.7m	27%	100	100	100	100	100	100	25	25	25	25	70%
81	GPT-5.6 Sol	$0.027	1.6s	27%	100	100	100	100	100	100	25	25	25	25	70%
94	Claude Sonnet 5 (Reasoning, Low)	$0.059	24.7s	27%	100	100	100	100	100	100	25	25	25	25	70%
168	Qwen 3.5 27B	$0.060	4.9m	27%	100	100	100	100	100	100	25	25	25	25	70%
78	Qwen 3.7 Flash (Reasoning)	$0.0013	40.1s	27%	100	100	100	100	100	100	25	25	25	25	70%
149	ByteDance Seed 2.0 Mini	$0.0050	4.5m	22%	100	100	100	100	100	100	25	25	17	17	68%
84	Grok 4.3	$0.016	1.8s	18%	100	100	100	100	100	100	33	13	13	10	67%
104	Gemini 3.6 Flash (Reasoning)	$0.044	18.3s	16%	100	100	100	100	100	25	25	25	25	25	63%
166	Aion 3.0	$0.084	2.5m	14%	100	100	100	100	100	25	25	25	25	17	62%
108	Grok 4.20 (Reasoning)	$0.023	59.7s	13%	100	100	100	100	100	25	25	25	17	17	61%
90	GPT-4.1 Mini	$0.0025	3.5s	13%	100	100	100	100	50	31	31	29	25	25	59%
93	Mistral Medium 3.1	$0.0058	3.8s	11%	100	100	100	100	50	30	29	29	17	17	57%
143	Gemini 3.5 Flash (Reasoning)	$0.061	23.1s	7%	100	100	100	100	25	25	25	25	25	25	55%
112	Qwen 3.6 Flash	$0.013	32.4s	7%	100	100	100	100	25	25	25	25	25	25	55%
98	GPT-5.4 Mini	$0.0039	834ms	7%	100	100	100	100	25	25	25	25	25	25	55%
103	Qwen 3 32B	$0.0024	24.4s	7%	100	100	100	100	25	25	25	25	25	25	55%
175	Qwen 3.5 397B A17B	$0.032	5.1m	6%	100	100	100	100	25	25	25	25	25	17	54%
142	Claude Sonnet 5 (Reasoning)	$0.056	22.3s	6%	100	100	100	100	25	25	25	25	25	17	54%
101	Cydonia 24B V4.1	$0.0029	8.3s	7%	100	100	100	100	42	31	28	10	7	3	52%
109	DeepSeek V4 Pro	$0.014	36.5s	13%	100	100	100	50	50	29	28	25	17	17	51%
102	Claude Opus 4.7	$0.100	3.7s	50%	50	50	50	50	50	50	50	50	50	50	50%
162	Gemini 3.1 Pro (Preview)	$0.078	55.6s	8%	100	100	100	25	25	25	25	25	25	25	48%
164	Grok 4.6 (Reasoning, High)	$0.055	1.7m	8%	100	100	100	25	25	25	25	25	25	25	48%
105	Qwen 2.5 72B	$0.0017	10.6s	13%	100	100	42	38	32	31	29	28	25	17	44%
141	WizardLM 2 8x22b	$0.0087	1.2m	10%	100	100	50	42	28	25	25	25	17	1	41%
131	DeepSeek V4.1 Flash (Reasoning, High)	$0.0070	45.2s	10%	100	100	25	25	25	25	25	25	25	25	40%
144	Z.AI GLM 5.3 Flash (Reasoning, Max)	$0.0017	1.4m	10%	100	100	25	25	25	25	25	25	25	25	40%
173	Claude Opus 5 (Reasoning, Low)	$0.145	24.3s	10%	100	100	25	25	25	25	25	25	25	25	40%
174	Claude Opus 5 (Reasoning)	$0.147	26.1s	10%	100	100	25	25	25	25	25	25	25	25	40%
170	Claude Sonnet 4.6 (Reasoning)	$0.110	59.4s	10%	100	100	25	25	25	25	25	25	25	17	39%
154	Claude Sonnet 4	$0.046	4.9s	2%	100	100	100	17	13	13	10	10	10	10	38%
96	Claude Haiku 4.5	$0.016	4.7s	32%	50	50	50	42	42	42	38	25	25	17	38%
117	GPT-4.1 Nano	$0.0009	8.8s	9%	100	100	42	27	27	27	26	26	1	1	38%
123	Qwen 3 235B A22B Instruct 2507	$0.0018	16.2s	8%	100	100	50	25	25	25	17	17	8	3	37%
122	DeepSeek V3 (2025-03-24)	$0.0032	5.3s	9%	100	100	25	25	25	25	17	17	13	8	35%
133	Gemini 2.5 Flash Lite	$0.0008	2.2s	2%	100	100	100	13	13	13	4	2	2	2	35%
178	Claude Opus 4	$0.225	6.0s	8%	100	50	50	50	17	17	17	17	17	13	35%
145	Ministral 3 14B	$0.0044	1.4m	15%	100	28	28	28	27	26	26	26	25	25	34%
100	DeepSeek V4 Flash	$0.0007	7.4s	27%	50	42	38	38	35	30	27	25	25	25	33%
160	DeepSeek V4 Pro 0813 (Reasoning, High)	$0.037	1.4m	14%	100	25	25	25	25	25	25	25	25	25	33%
118	GPT-5.6 Luna	$0.0040	1.3s	14%	100	25	25	25	25	25	25	25	25	25	33%
140	Mistral Small 3.2 24B	$0.0014	3.4s	1%	100	100	38	38	6	6	5	5	5	5	31%
158	Qwen 3.5 Plus (2026-02-15)	$0.023	1.9m	20%	50	50	50	42	33	25	25	17	7	6	31%
177	Qwen 3.8 Max (Reasoning, XHigh)	$0.080	4.0m	25%	25	25	25	25	25	25	25	25	25	25	25%
127	Gemini 3.8 Flash (Reasoning, Medium)	$0.022	11.4s	25%	25	25	25	25	25	25	25	25	25	25	25%
161	Z.AI GLM 5.3 (Reasoning, Max)	$0.033	1.9m	25%	25	25	25	25	25	25	25	25	25	25	25%
132	GPT-5.6 Sol (Reasoning)	$0.029	6.2s	25%	25	25	25	25	25	25	25	25	25	25	25%
137	GPT-5.5 (Reasoning)	$0.033	13.7s	25%	25	25	25	25	25	25	25	25	25	25	25%
167	Qwen 3.8 27B (Reasoning, XHigh)	$0.029	2.4m	25%	25	25	25	25	25	25	25	25	25	25	25%
134	GPT-5.5 (Reasoning, Low)	$0.028	9.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
119	GPT-5.5	$0.012	3.5s	25%	25	25	25	25	25	25	25	25	25	25	25%
111	Gemini 3.1 Flash Lite (Reasoning)	$0.0023	1.8s	25%	25	25	25	25	25	25	25	25	25	25	25%
115	Gemini 3.1 Flash Lite	$0.0032	2.8s	25%	25	25	25	25	25	25	25	25	25	25	25%
114	Gemini 3.5 Flash Lite (Reasoning)	$0.0042	803ms	25%	25	25	25	25	25	25	25	25	25	25	25%
113	Gemini 2.5 Flash	$0.0039	1.0s	25%	25	25	25	25	25	25	25	25	25	25	25%
171	Hy4 Preview (Reasoning, High)	$0.038	3.4m	24%	25	25	25	25	25	25	25	25	25	17	24%
116	GPT-5.4	$0.0052	2.3s	24%	25	25	25	25	25	25	25	25	25	17	24%
135	Hermes 3 70B	$0.0039	13.9s	16%	50	38	33	32	32	13	13	10	8	8	24%
121	Ministral 3B	$0.0006	3.6s	20%	42	38	32	29	27	27	26	6	5	5	24%
124	Writer: Palmyra X5	$0.010	9.4s	22%	35	28	27	25	25	25	17	17	17	17	23%
153	Llama 3.1 70B	$0.0053	14.3s	3%	100	50	29	17	8	7	6	5	5	4	23%
126	GPT-4o, Aug. 6th (temp=1)	$0.017	1.6s	23%	25	25	25	25	25	25	25	25	17	13	23%
120	Gemini 3.5 Flash Lite (Reasoning, Minimal)	$0.0043	1.1s	23%	25	25	25	25	25	25	25	17	17	17	23%
129	GPT-4o, Aug. 6th (temp=0)	$0.019	1.5s	23%	25	25	25	25	25	25	25	17	17	17	23%
130	Gemini 3.1 Flash Lite (Preview)	$0.0030	1.3s	16%	50	25	25	25	25	17	17	13	13	13	22%
172	Qwen 3.8 Flash (Reasoning)	$0.0087	4.1m	23%	25	25	25	25	25	25	17	17	17	17	22%
157	Claude Sonnet 5	$0.041	6.0s	11%	42	38	35	25	17	13	13	13	13	8	21%
125	GPT-5.6 Terra	$0.011	1.8s	22%	25	25	25	25	25	25	17	17	13	13	21%
138	Ministral 3 3B	$0.0021	33.2s	21%	29	28	28	28	27	27	26	6	1	0	20%
155	Hermes 3 405B	$0.013	7.0s	6%	100	17	13	13	13	13	10	10	7	6	20%
148	GPT-4o Mini (temp=1)	$0.0013	6.0s	9%	38	35	32	29	13	10	10	10	10	6	19%
136	DeepSeek V3 (2024-12-26)	$0.0045	4.4s	18%	25	25	25	25	25	17	17	13	10	8	19%
139	DeepSeek V3.1	$0.0028	11.8s	16%	29	25	25	25	25	13	13	13	13	10	19%
150	DeepSeek V3.2	$0.0031	21.6s	14%	29	25	25	17	17	17	17	10	10	8	17%
159	Claude Sonnet 4.5	$0.047	5.4s	15%	25	25	17	17	17	17	17	13	13	13	17%
147	Mistral Small 4	$0.0022	8.4s	13%	32	30	29	29	28	6	5	4	3	1	17%
146	DeepSeek-V2 Chat	$0.0049	6.9s	14%	25	25	25	17	17	17	13	13	8	5	16%
152	Gemini 3 Flash (Preview)	$0.0074	3.0s	11%	25	17	17	17	13	13	13	13	10	10	15%
156	Grok 4.20	$0.014	4.3s	12%	17	17	17	13	13	13	13	10	8	8	13%
151	GPT-4o Mini (temp=0)	$0.0012	3.9s	12%	13	13	13	13	13	13	13	13	13	10	12%
163	Gemma 3 12B	$0.0010	15.2s	4%	29	7	6	6	5	5	5	4	3	3	7%
165	Gemma 3 27B	$0.0012	11.5s	5%	6	6	5	5	5	5	5	5	5	4	5%
169	Gemma 3 4B	$0.0007	20.4s	1%	2	2	2	1	1	1	1	1	1	1	1%
179	Mistral NeMO	$0.0056	5.0m	1%	3	3	2	2	1	1	1	0	0	0	1%
64.78%

Median	Evaluator	Top 3	Flop 3
65.0%	Correct "no violations" response	100Claude Opus 5 100Muse Spark 1.1 (Reasoning, Medium) 100Muse Spark 1.2 (Reasoning, Medium)	0DeepSeek V3 (2024-12-26) 0GPT-4o Mini (temp=0) 0Grok 4.20
80.0%	No hallucinated violations	100Claude Opus 5 100MiniMax M2.7 100Muse Spark 1.1 (Reasoning, Medium)	2Mistral NeMO 3Gemma 3 4B 5Ministral 3 3B

Codex Red Herring (False Positive Detection)

Short text (~524 words), big codex (51 detailed entries)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)