Correct "no violations" response

Test: Codex Red Herring (False Positive Detection)

Avg. Score

46.1%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Grok 4.1 Fast	96.3%	$0.0019	12.5s	62%
2	o4 Mini	95.0%	$0.014	25.0s	56%
3	ByteDance Seed 1.6 Flash	92.5%	$0.0008	9.1s	47%
4	o4 Mini High	96.3%	$0.027	52.5s	62%
5	GPT-5.1	93.8%	$0.025	26.1s	52%
6	GPT-5 Mini	91.3%	$0.0059	37.8s	43%
7	GPT-4.1	85.6%	$0.0048	1.2s	38%
8	Gemini 2.5 Flash Lite (Reasoning)	88.8%	$0.0023	16.6s	37%
9	GPT-5 Nano	90.0%	$0.0035	1.1m	40%
10	Ministral 8B	74.4%	$0.0007	4.4s	33%
11	Z.AI GLM 5	91.3%	$0.017	1.4m	43%
12	Ministral 3 8B	80.6%	$0.0013	17.9s	30%
13	Gemini 2.5 Flash (Reasoning)	82.5%	$0.0085	14.2s	24%
14	Claude Sonnet 4.6	73.8%	$0.033	17.4s	39%
15	Claude Opus 4.6	81.9%	$0.049	13.2s	38%
16	GPT-4.1 Nano	68.1%	$0.0005	4.6s	24%
17	ByteDance Seed 1.6	81.3%	$0.0043	32.7s	22%
18	GPT-5.2	81.3%	$0.013	14.5s	22%
19	Aion 2.0	83.8%	$0.0096	1.3m	26%
20	Ministral 3B	43.1%	$0.0004	12.4s	33%
21	Minimax M2.5	73.1%	$0.0032	25.9s	12%
22	Claude Haiku 4.5	51.2%	$0.0080	4.2s	24%
23	GPT-4.1 Mini	51.2%	$0.0018	6.9s	18%
24	Mistral Medium 3.1	53.1%	$0.0032	4.7s	17%
25	Mistral Small Creative	37.5%	$0.0011	8.9s	26%
26	Ministral 3 3B	37.5%	$0.0013	31.1s	28%
27	Z.AI GLM 4.5	61.3%	$0.0038	25.0s	12%
28	Arcee AI: Trinity Mini	67.5%	$0.0009	26.3s	6%
29	Claude Opus 4.6 (Reasoning)	92.5%	$0.120	1.0m	47%
30	Grok 4 Fast	66.3%	$0.0019	19.2s	5%
31	Qwen 2.5 72B	45.6%	$0.0009	11.3s	15%
32	Ministral 3 14B	50.6%	$0.0019	29.2s	15%
33	Z.AI GLM 4.6	68.1%	$0.015	59.0s	14%
34	Llama 3.1 8B	26.9%	$0.0003	33.8s	25%
35	Hermes 3 70B	42.5%	$0.0018	12.1s	9%
36	Llama 3.1 Nemotron 70B	35.0%	$0.0077	21.4s	17%
37	Arcee AI: Trinity Large (Preview)	43.1%	$0.0000	1.7m	22%
38	GPT-4o Mini (temp=1)	32.5%	$0.0008	5.6s	7%
39	Cohere Command R+ (Aug. 2024)	50.0%	$0.017	7.4s	0%
40	Z.AI GLM 4.7 Flash	70.0%	$0.0039	2.5m	8%
41	Mistral Large 3	36.9%	$0.0040	11.7s	0%
42	Claude Opus 4.5	40.0%	$0.036	4.4s	9%
43	GPT-4o, Aug. 6th (temp=1)	32.5%	$0.0094	2.4s	0%
44	Hermes 3 405B	30.6%	$0.0059	5.2s	0%
45	Gemini 2.5 Flash Lite	25.0%	$0.0010	4.0s	0%
46	Mistral Small 3.2 24B	26.3%	$0.0010	11.9s	0%
47	Mistral Large 2	35.0%	$0.016	11.4s	0%
48	Claude Sonnet 4	35.0%	$0.022	4.5s	0%
49	GPT-5	67.5%	$0.048	1.5m	6%
50	Mistral Large	33.1%	$0.016	12.3s	0%
51	Claude 3 Haiku	20.6%	$0.0020	3.5s	0%
52	Llama 3.1 70B	25.0%	$0.0031	16.4s	0%
53	Rocinante 12B	22.5%	$0.0015	13.3s	0%
54	Writer: Palmyra X5	22.5%	$0.0085	13.8s	0%
55	GPT-4o Mini (temp=0)	16.9%	$0.0009	14.5s	0%
56	Gemini 3 Flash (Preview, Reasoning)	44.4%	$0.028	49.2s	0%
57	WizardLM 2 8x22b	26.9%	$0.0041	55.3s	0%
58	Claude Sonnet 4.5	20.6%	$0.024	5.9s	0%
59	DeepSeek-V2 Chat	5.0%	$0.0023	8.3s	0%
60	DeepSeek V3 (2024-12-26)	4.4%	$0.0023	6.6s	0%
61	Z.AI GLM 4.7	55.0%	$0.023	2.2m	1%
62	GPT-4o, Aug. 6th (temp=0)	8.8%	$0.011	2.6s	0%
63	DeepSeek V3.1	12.5%	$0.0019	36.6s	0%
64	DeepSeek V3 (2025-03-24)	5.6%	$0.0015	15.7s	0%
65	DeepSeek V3.2	13.8%	$0.0018	42.8s	0%
66	Claude 3.7 Sonnet	15.6%	$0.022	4.3s	0%
67	Gemini 3 Flash (Preview)	2.5%	$0.0034	3.2s	0%
68	Gemini 2.5 Flash	0.6%	$0.0021	1.5s	0%
69	Gemma 3 12B	1.3%	$0.0005	11.7s	0%
70	Gemma 3 27B	0.0%	$0.0007	12.6s	0%
71	Gemma 3 4B	0.0%	$0.0003	15.0s	0%
72	MoonshotAI: Kimi K2.5	52.5%	$0.017	2.6m	0%
73	GPT-4o, May 13th (temp=1)	16.9%	$0.033	3.0s	0%
74	Grok 4	61.3%	$0.074	1.6m	3%
75	Claude 3.5 Sonnet	18.8%	$0.045	6.9s	0%
76	Gemini 2.5 Pro	46.3%	$0.073	52.1s	0%
77	Mistral NeMO	8.8%	$0.0018	1.6m	0%
78	GPT-4o, May 13th (temp=0)	1.9%	$0.043	6.4s	0%
79	Qwen 3.5 Plus (2026-02-15)	18.8%	$0.021	2.0m	0%
80	Claude Sonnet 4.6 (Reasoning)	76.3%	$0.145	2.2m	15%
81	Gemini 3.1 Pro (Preview)	57.5%	$0.120	1.5m	1%
82	Claude Opus 4	18.1%	$0.109	7.1s	0%
83	Gemini 3 Pro (Preview)	53.8%	$0.144	1.7m	0%
84	Qwen 3.5 397B A17B	31.3%	$0.028	5.4m	0%
46.13%

Individual Scenarios

basic entries

▼

Long text (~1594 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini High	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Claude Sonnet 4.6	100	100	100	100	50	50	50	50	0	0	60.0%
Ministral 8B	100	100	100	50	50	50	50	50	0	0	55.0%
Minimax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4.1	100	100	50	50	50	50	50	50	0	0	50.0%
Z.AI GLM 4.6	100	100	100	100	50	0	0	0	0	0	45.0%
ByteDance Seed 1.6	100	100	100	100	0	0	0	0	0	0	40.0%
Writer: Palmyra X5	50	50	50	50	50	50	50	50	0	0	40.0%
Qwen 2.5 72B	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	0	0	40.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Ministral 3B	50	50	50	50	50	50	50	50	0	0	40.0%
Rocinante 12B	100	100	100	50	50	0	0	0	0	0	40.0%
Claude Opus 4.5	50	50	50	50	50	50	50	0	0	0	35.0%
Ministral 3 14B	50	50	50	50	50	50	50	0	0	0	35.0%
Claude 3 Haiku	50	50	50	50	50	50	50	0	0	0	35.0%
GPT-5 Mini	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4.5	50	50	50	50	50	50	0	0	0	0	30.0%
Gemini 2.5 Flash Lite	100	100	100	0	0	0	0	0	0	0	30.0%
WizardLM 2 8x22b	50	50	50	50	50	50	0	0	0	0	30.0%
Ministral 3 3B	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral NeMO	100	100	100	0	0	0	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 3.1 Pro (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Claude Sonnet 4	50	50	50	50	0	0	0	0	0	0	20.0%
Hermes 3 405B	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, Aug. 6th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Medium 3.1	50	50	50	50	0	0	0	0	0	0	20.0%
Claude Opus 4	50	50	50	0	0	0	0	0	0	0	15.0%
Z.AI GLM 4.5	100	50	0	0	0	0	0	0	0	0	15.0%
Claude Haiku 4.5	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
DeepSeek V3.1	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Hermes 3 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Gemini 3 Pro (Preview)	100	0	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.7	100	0	0	0	0	0	0	0	0	0	10.0%
Grok 4	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-4.1 Mini	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Cohere Command R+ (Aug. 2024)	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 3 Flash (Preview, Reasoning)	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3 Flash (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
Claude 3.5 Sonnet	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen 3.5 397B A17B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Pro	0	0	0	0	0	0	0	0	0	0	0.0%
Grok 4 Fast	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Plus (2026-02-15)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.7 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 2	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Long text (~1594 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	0	90.0%
Minimax M2.5	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	100	0	0	80.0%
Ministral 8B	100	100	100	100	100	50	50	50	50	50	75.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 70B	100	100	100	100	100	100	50	50	0	0	70.0%
Z.AI GLM 4.5	100	100	100	100	100	50	50	50	0	0	65.0%
Z.AI GLM 4.6	100	100	100	100	50	50	50	0	0	0	55.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Sonnet 4.5	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Writer: Palmyra X5	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Llama 3.1 Nemotron 70B	100	50	50	50	50	50	50	0	0	0	40.0%
Arcee AI: Trinity Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Opus 4.6	50	50	50	50	50	50	50	0	0	0	35.0%
Claude Sonnet 4	50	50	50	50	50	50	50	0	0	0	35.0%
GPT-4o Mini (temp=1)	100	50	50	50	50	50	0	0	0	0	35.0%
Claude Opus 4	50	50	50	50	50	50	0	0	0	0	30.0%
WizardLM 2 8x22b	100	50	50	50	50	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o, Aug. 6th (temp=1)	100	50	50	50	0	0	0	0	0	0	25.0%
DeepSeek V3.1	100	50	50	50	0	0	0	0	0	0	25.0%
Qwen 2.5 72B	100	50	50	50	0	0	0	0	0	0	25.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
DeepSeek V3.2	50	50	50	50	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Small Creative	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral NeMO	100	100	0	0	0	0	0	0	0	0	20.0%
Llama 3.1 8B	50	50	50	50	0	0	0	0	0	0	20.0%
Rocinante 12B	100	50	50	0	0	0	0	0	0	0	20.0%
DeepSeek V3 (2025-03-24)	50	50	50	0	0	0	0	0	0	0	15.0%
Ministral 3 14B	50	50	50	0	0	0	0	0	0	0	15.0%
Gemini 3 Flash (Preview, Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Opus 4.5	50	50	0	0	0	0	0	0	0	0	10.0%
Gemini 3 Pro (Preview)	100	0	0	0	0	0	0	0	0	0	10.0%
Hermes 3 405B	100	0	0	0	0	0	0	0	0	0	10.0%
Cohere Command R+ (Aug. 2024)	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 3 Flash (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek-V2 Chat	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 2	50	0	0	0	0	0	0	0	0	0	5.0%
Llama 3.1 70B	50	0	0	0	0	0	0	0	0	0	5.0%
Claude 3 Haiku	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.7 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Claude Sonnet 4	100	100	100	100	100	100	50	50	50	0	75.0%
Ministral 8B	100	100	100	100	100	100	50	50	50	0	75.0%
Z.AI GLM 5	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-5.2	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 405B	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-5 Nano	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 70B	100	100	100	100	100	50	50	50	0	0	65.0%
Z.AI GLM 4.6	100	100	100	100	100	50	50	0	0	0	60.0%
Minimax M2.5	100	100	100	100	100	100	0	0	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Haiku 4.5	100	50	50	50	50	50	50	50	50	50	55.0%
Grok 4	100	100	100	100	100	0	0	0	0	0	50.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	50	50	50.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	50	50	50.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Sonnet 4.6	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	50	50	50	0	0	0	0	45.0%
Qwen 2.5 72B	100	50	50	50	50	50	50	50	0	0	45.0%
GPT-5	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
MoonshotAI: Kimi K2.5	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 4.7	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 4.5	100	100	100	50	50	0	0	0	0	0	40.0%
Claude 3.5 Sonnet	100	100	100	100	0	0	0	0	0	0	40.0%
Ministral 3 14B	50	50	50	50	50	50	50	50	0	0	40.0%
GPT-4.1 Nano	100	100	100	100	0	0	0	0	0	0	40.0%
WizardLM 2 8x22b	100	50	50	50	50	50	0	0	0	0	35.0%
Grok 4 Fast	100	100	100	0	0	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	50	0	0	0	0	30.0%
Writer: Palmyra X5	100	100	50	50	0	0	0	0	0	0	30.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Large 3	50	50	50	50	50	0	0	0	0	0	25.0%
Claude 3.7 Sonnet	100	100	50	0	0	0	0	0	0	0	25.0%
Mistral Large 2	50	50	50	50	50	0	0	0	0	0	25.0%
Llama 3.1 70B	100	50	50	0	0	0	0	0	0	0	20.0%
Mistral NeMO	100	50	50	0	0	0	0	0	0	0	20.0%
DeepSeek V3.1	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o Mini (temp=0)	50	50	50	0	0	0	0	0	0	0	15.0%
Claude 3 Haiku	50	50	50	0	0	0	0	0	0	0	15.0%
Rocinante 12B	100	50	0	0	0	0	0	0	0	0	15.0%
Claude Opus 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Sonnet 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Mistral Large	50	50	0	0	0	0	0	0	0	0	10.0%
GPT-4o Mini (temp=1)	50	50	0	0	0	0	0	0	0	0	10.0%
Claude Opus 4	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=1)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, Aug. 6th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 2.5 Flash	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Small 3.2 24B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemma 3 12B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash Lite	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	0	90.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	50	0	0	75.0%
GPT-4.1 Nano	100	100	100	100	100	50	50	50	50	50	75.0%
GPT-5.2	100	100	100	100	100	100	100	0	0	0	70.0%
Aion 2.0	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Grok 4 Fast	100	100	100	100	100	100	100	0	0	0	70.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	50	50	0	0	70.0%
GPT-4.1 Mini	100	100	100	100	50	50	50	50	50	0	65.0%
Claude Haiku 4.5	100	100	50	50	50	50	50	50	50	50	60.0%
Llama 3.1 70B	100	100	100	100	100	50	50	0	0	0	60.0%
Qwen 2.5 72B	100	100	100	100	50	50	50	50	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 397B A17B	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	100	100	0	0	0	0	0	50.0%
Grok 4	100	100	100	100	100	0	0	0	0	0	50.0%
Hermes 3 70B	100	100	100	50	50	50	50	0	0	0	50.0%
Ministral 3 14B	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Sonnet 4.6	50	50	50	50	50	50	50	50	50	0	45.0%
Claude Sonnet 4	100	100	100	100	50	0	0	0	0	0	45.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 8B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Mistral Large 3	50	50	50	50	50	50	50	50	0	0	40.0%
Claude 3.5 Sonnet	100	100	100	50	50	0	0	0	0	0	40.0%
Hermes 3 405B	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Large (Preview)	100	50	50	50	50	50	50	0	0	0	40.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	0	0	40.0%
GPT-4.1	100	100	50	50	50	0	0	0	0	0	35.0%
Ministral 3 8B	50	50	50	50	50	50	50	0	0	0	35.0%
Rocinante 12B	100	100	100	50	0	0	0	0	0	0	35.0%
Z.AI GLM 4.7	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-4o, Aug. 6th (temp=1)	100	50	50	50	50	0	0	0	0	0	30.0%
Mistral Large 2	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Large	50	50	50	50	50	50	0	0	0	0	30.0%
Claude Sonnet 4.5	100	100	50	0	0	0	0	0	0	0	25.0%
DeepSeek V3.2	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o Mini (temp=1)	100	50	50	50	0	0	0	0	0	0	25.0%
GPT-5	100	100	0	0	0	0	0	0	0	0	20.0%
DeepSeek-V2 Chat	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=0)	50	50	50	50	0	0	0	0	0	0	20.0%
Claude 3 Haiku	50	50	50	50	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
DeepSeek V3 (2024-12-26)	100	50	0	0	0	0	0	0	0	0	15.0%
WizardLM 2 8x22b	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 8B	50	50	50	0	0	0	0	0	0	0	15.0%
Qwen 3.5 Plus (2026-02-15)	50	50	0	0	0	0	0	0	0	0	10.0%
Writer: Palmyra X5	50	50	0	0	0	0	0	0	0	0	10.0%
Claude 3.7 Sonnet	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.1	50	0	0	0	0	0	0	0	0	0	5.0%
MoonshotAI: Kimi K2.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash Lite	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

detailed entries

▼

Long text (~1594 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	50	95.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	0	0	80.0%
Minimax M2.5	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	50	0	0	70.0%
Qwen 2.5 72B	100	100	100	100	100	50	50	50	0	0	65.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Sonnet 4.6	100	100	50	50	50	50	50	50	50	50	60.0%
Z.AI GLM 4.7	100	100	100	100	100	100	0	0	0	0	60.0%
Arcee AI: Trinity Large (Preview)	100	100	100	50	50	50	50	50	50	0	60.0%
Z.AI GLM 4.6	100	100	100	100	100	50	0	0	0	0	55.0%
Claude Sonnet 4.5	100	100	50	50	50	50	50	50	0	0	50.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-4.1 Nano	100	100	100	100	50	50	0	0	0	0	50.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	0	45.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 3 Pro (Preview)	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 2.5 Flash Lite	100	100	100	50	50	0	0	0	0	0	40.0%
Ministral 3B	50	50	50	50	50	50	50	0	0	0	35.0%
GPT-4o, Aug. 6th (temp=1)	100	100	50	50	0	0	0	0	0	0	30.0%
Writer: Palmyra X5	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Small 3.2 24B	100	100	100	0	0	0	0	0	0	0	30.0%
WizardLM 2 8x22b	100	100	50	50	0	0	0	0	0	0	30.0%
Ministral 3 3B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o, Aug. 6th (temp=0)	50	50	50	50	50	0	0	0	0	0	25.0%
DeepSeek V3.1	50	50	50	50	50	0	0	0	0	0	25.0%
Claude Opus 4	50	50	50	50	0	0	0	0	0	0	20.0%
DeepSeek V3.2	50	50	50	50	0	0	0	0	0	0	20.0%
Claude Opus 4.5	50	50	50	0	0	0	0	0	0	0	15.0%
Claude 3.7 Sonnet	100	50	0	0	0	0	0	0	0	0	15.0%
Mistral Large	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o Mini (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 Nemotron 70B	50	50	50	0	0	0	0	0	0	0	15.0%
Hermes 3 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Claude 3 Haiku	50	50	50	0	0	0	0	0	0	0	15.0%
Rocinante 12B	50	50	50	0	0	0	0	0	0	0	15.0%
Gemini 3 Flash (Preview)	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek-V2 Chat	50	50	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.7 Flash	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Sonnet 4	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 3	50	0	0	0	0	0	0	0	0	0	5.0%
Hermes 3 405B	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 2	50	0	0	0	0	0	0	0	0	0	5.0%
Llama 3.1 70B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 2.5 Pro	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Plus (2026-02-15)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Long text (~1594 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	50	0	85.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	50	50	0	75.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	50	0	0	75.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.6	100	100	100	100	100	100	50	0	0	0	65.0%
Minimax M2.5	100	100	100	100	100	100	50	0	0	0	65.0%
Mistral Medium 3.1	100	100	100	100	50	50	50	50	50	0	65.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	50	50	0	0	0	60.0%
Hermes 3 70B	100	100	100	50	50	50	50	50	50	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	50	50	50	0	0	0	55.0%
GPT-5.2	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Opus 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Z.AI GLM 4.7	100	100	100	100	100	0	0	0	0	0	50.0%
Grok 4 Fast	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 8B	100	100	100	50	50	50	0	0	0	0	45.0%
Claude Opus 4.6	50	50	50	50	50	50	50	50	0	0	40.0%
Aion 2.0	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 2.5 72B	100	100	50	50	50	50	0	0	0	0	40.0%
Llama 3.1 Nemotron 70B	100	100	50	50	50	50	0	0	0	0	40.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	50	50	0	0	0	35.0%
WizardLM 2 8x22b	50	50	50	50	50	50	50	0	0	0	35.0%
ByteDance Seed 1.6	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4	100	50	50	50	50	0	0	0	0	0	30.0%
Grok 4	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Opus 4	50	50	50	50	50	50	0	0	0	0	30.0%
Ministral 3 8B	100	100	50	50	0	0	0	0	0	0	30.0%
Mistral Large 3	50	50	50	50	50	0	0	0	0	0	25.0%
Ministral 3 3B	50	50	50	50	50	0	0	0	0	0	25.0%
Llama 3.1 8B	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 2.5 Pro	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Flash (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Hermes 3 405B	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Flash Lite	100	100	0	0	0	0	0	0	0	0	20.0%
Llama 3.1 70B	100	100	0	0	0	0	0	0	0	0	20.0%
Claude 3 Haiku	50	50	50	50	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Mini	100	100	0	0	0	0	0	0	0	0	20.0%
Mistral Large 2	50	50	50	0	0	0	0	0	0	0	15.0%
Mistral Small 3.2 24B	100	50	0	0	0	0	0	0	0	0	15.0%
Mistral Small Creative	50	50	50	0	0	0	0	0	0	0	15.0%
Ministral 3 14B	50	50	50	0	0	0	0	0	0	0	15.0%
MoonshotAI: Kimi K2.5	100	0	0	0	0	0	0	0	0	0	10.0%
Claude 3.7 Sonnet	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.1	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Mistral Large	50	50	0	0	0	0	0	0	0	0	10.0%
Writer: Palmyra X5	50	50	0	0	0	0	0	0	0	0	10.0%
Claude Sonnet 4.5	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek-V2 Chat	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Rocinante 12B	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen 3.5 397B A17B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	50	50	90.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.1	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	0	0	0	70.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	0	0	0	70.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5	100	100	100	100	100	100	0	0	0	0	60.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-4.1 Mini	100	100	100	100	50	50	50	50	0	0	60.0%
Mistral Medium 3.1	100	100	100	100	50	50	50	50	0	0	60.0%
Ministral 3 14B	100	50	50	50	50	50	50	50	50	50	55.0%
Qwen 2.5 72B	100	100	50	50	50	50	50	50	0	0	50.0%
GPT-4.1 Nano	100	100	50	50	50	50	50	50	0	0	50.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	0	45.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Claude 3.7 Sonnet	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	0	0	0	35.0%
Claude 3.5 Sonnet	100	100	50	50	50	0	0	0	0	0	35.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	0	0	0	35.0%
WizardLM 2 8x22b	100	100	50	50	50	0	0	0	0	0	35.0%
Ministral 3 3B	50	50	50	50	50	50	50	0	0	0	35.0%
Ministral 3B	50	50	50	50	50	50	50	0	0	0	35.0%
Gemini 3.1 Pro (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 2.5 Flash Lite	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Small 3.2 24B	100	100	50	50	0	0	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Claude Opus 4	100	50	50	50	0	0	0	0	0	0	25.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Hermes 3 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Claude Sonnet 4.6 (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
DeepSeek V3 (2025-03-24)	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Llama 3.1 70B	100	50	50	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
Writer: Palmyra X5	50	50	50	0	0	0	0	0	0	0	15.0%
Hermes 3 405B	100	0	0	0	0	0	0	0	0	0	10.0%
Claude 3 Haiku	50	50	0	0	0	0	0	0	0	0	10.0%
Rocinante 12B	50	50	0	0	0	0	0	0	0	0	10.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.1	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.2	50	0	0	0	0	0	0	0	0	0	5.0%
Gemma 3 12B	50	0	0	0	0	0	0	0	0	0	5.0%
Claude Sonnet 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	50	0	85.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.2	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Minimax M2.5	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 405B	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-4.1 Nano	100	100	100	50	50	50	50	50	50	50	65.0%
Z.AI GLM 4.5	100	100	100	100	100	50	50	0	0	0	60.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 70B	100	100	100	100	50	50	50	0	0	0	55.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	50	50	0	0	0	0	50.0%
Mistral Medium 3.1	100	100	100	50	50	50	0	0	0	0	45.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	50	0	45.0%
Mistral Small Creative	100	100	50	50	50	50	50	0	0	0	45.0%
Claude 3 Haiku	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-5	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Sonnet 4	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 2.5 72B	100	50	50	50	50	50	50	0	0	0	40.0%
Hermes 3 70B	100	100	100	50	50	0	0	0	0	0	40.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	0	0	40.0%
Rocinante 12B	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-4o Mini (temp=1)	50	50	50	50	50	50	50	0	0	0	35.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	50	50	0	0	0	35.0%
Llama 3.1 8B	50	50	50	50	50	50	50	0	0	0	35.0%
Claude 3.5 Sonnet	100	100	100	0	0	0	0	0	0	0	30.0%
Claude 3.7 Sonnet	100	100	50	50	0	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 3.1 Pro (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Claude Opus 4	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3 (2025-03-24)	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash Lite	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
WizardLM 2 8x22b	50	0	0	0	0	0	0	0	0	0	5.0%
Claude Sonnet 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Writer: Palmyra X5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

Correct "no violations" response

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

basic entries

Long text (~1594 words), big codex (51 entries)

Long text (~1594 words), small codex (11 entries)

Short text (~524 words), big codex (51 entries)

Short text (~524 words), small codex (11 entries)

detailed entries

Long text (~1594 words), big codex (51 detailed entries)

Long text (~1594 words), small codex (11 detailed entries)

Short text (~524 words), big codex (51 detailed entries)

Short text (~524 words), small codex (11 detailed entries)