Correct "no violations" response

Test: Codex Red Herring (False Positive Detection)

Avg. Score

50.5%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	GPT-5.4 Nano (Reasoning, Low)	97.5%	$0.0008	3.9s	73%
2	Inception Mercury	97.5%	$0.0004	8.0s	69%
3	Nemotron 3 Super	98.8%	$0.0000	1.3m	78%
4	Grok 4.1 Fast	96.3%	$0.0019	12.5s	62%
5	Inception Mercury 2	95.0%	$0.0027	3.8s	56%
6	Z.AI GLM 5 Turbo	95.0%	$0.0071	16.0s	56%
7	GPT-5.4 Mini (Reasoning, Low)	93.8%	$0.0034	4.0s	52%
8	o4 Mini	95.0%	$0.014	25.0s	56%
9	ByteDance Seed 1.6 Flash	92.5%	$0.0008	9.1s	47%
10	GPT-5.4 Nano (Reasoning)	92.5%	$0.0017	11.4s	47%
11	o4 Mini High	96.3%	$0.027	52.5s	62%
12	GPT-5.1	93.8%	$0.025	26.1s	52%
13	GPT-5 Mini	91.3%	$0.0059	37.8s	43%
14	GPT-4.1	85.6%	$0.0048	1.2s	38%
15	Gemini 2.5 Flash Lite (Reasoning)	88.8%	$0.0023	16.6s	37%
16	GPT-5 Nano	90.0%	$0.0035	1.1m	40%
17	Ministral 8B	74.4%	$0.0007	4.4s	33%
18	MiniMax M2.7	86.3%	$0.0047	34.6s	31%
19	Ministral 3 8B	80.6%	$0.0013	17.9s	30%
20	Z.AI GLM 5	91.3%	$0.017	1.4m	43%
21	GPT-5.4 Mini (Reasoning)	85.0%	$0.0090	10.8s	29%
22	Grok 4.20 (Beta, Reasoning)	86.3%	$0.025	15.9s	31%
23	Gemini 2.5 Flash (Reasoning)	82.5%	$0.0085	14.2s	24%
24	ByteDance Seed 1.6	81.3%	$0.0043	32.7s	22%
25	GPT-4.1 Nano	68.1%	$0.0005	4.6s	24%
26	Stealth: Healer Alpha	78.8%	$0.0000	21.5s	18%
27	GPT-5.2	81.3%	$0.013	14.5s	22%
28	Claude Sonnet 4.6	73.8%	$0.033	17.4s	39%
29	Claude Opus 4.6	81.9%	$0.049	13.2s	38%
30	Aion 2.0	83.8%	$0.0096	1.3m	26%
31	Mistral Small 4 (Reasoning)	77.5%	$0.0025	22.6s	16%
32	GPT-5.4 Nano	66.3%	$0.0005	1.5s	18%
33	Ministral 3B	43.1%	$0.0004	12.4s	33%
34	MiniMax M2.5	73.1%	$0.0032	25.9s	12%
35	Claude Haiku 4.5	51.2%	$0.0080	4.2s	24%
36	LFM2 24B	71.3%	$0.0004	39.0s	11%
37	GPT-5.4 (Reasoning, Low)	71.3%	$0.012	8.8s	9%
38	GPT-4.1 Mini	51.2%	$0.0018	6.9s	18%
39	Mistral Medium 3.1	53.1%	$0.0032	4.7s	17%
40	Nemotron 3 Nano	83.8%	$0.0027	2.9m	26%
41	Arcee AI: Trinity Mini	67.5%	$0.0009	26.3s	6%
42	Z.AI GLM 4.5	61.3%	$0.0038	25.0s	12%
43	Mistral Small Creative	37.5%	$0.0011	8.9s	26%
44	Grok 4 Fast	66.3%	$0.0019	19.2s	5%
45	Ministral 3 3B	37.5%	$0.0013	31.1s	28%
46	Qwen 2.5 72B	45.6%	$0.0009	11.3s	15%
47	Ministral 3 14B	50.6%	$0.0019	29.2s	15%
48	Z.AI GLM 4.6	68.1%	$0.015	59.0s	14%
49	Claude Opus 4.6 (Reasoning)	92.5%	$0.120	1.0m	47%
50	Stealth: Hunter Alpha	64.4%	$0.0000	1.0m	5%
51	Mistral Small 4	28.1%	$0.0012	7.4s	23%
52	Qwen 3 32B	60.6%	$0.0014	37.7s	3%
53	Llama 3.1 8B	26.9%	$0.0003	33.8s	25%
54	Hermes 3 70B	42.5%	$0.0018	12.1s	9%
55	Llama 3.1 Nemotron 70B	35.0%	$0.0077	21.4s	17%
56	Arcee AI: Trinity Large (Preview)	43.1%	$0.0000	1.7m	22%
57	GPT-4o Mini (temp=1)	32.5%	$0.0008	5.6s	7%
58	Z.AI GLM 4.7 Flash	70.0%	$0.0039	2.5m	8%
59	GPT-5.4 (Reasoning)	63.7%	$0.032	31.9s	4%
60	Cohere Command R+ (Aug. 2024)	50.0%	$0.017	7.4s	0%
61	Mistral Large 3	36.9%	$0.0040	11.7s	0%
62	Grok 4.20 (Beta)	34.4%	$0.0061	4.3s	0%
63	ByteDance Seed 2.0 Mini	68.1%	$0.0033	3.1m	7%
64	GPT-4o, Aug. 6th (temp=1)	32.5%	$0.0094	2.4s	0%
65	Hermes 3 405B	30.6%	$0.0059	5.2s	0%
66	Claude Opus 4.5	40.0%	$0.036	4.4s	9%
67	Gemini 2.5 Flash Lite	25.0%	$0.0010	4.0s	0%
68	Mistral Small 3.2 24B	26.3%	$0.0010	11.9s	0%
69	Mistral Large 2	35.0%	$0.016	11.4s	0%
70	GPT-5	67.5%	$0.048	1.5m	6%
71	Mistral Large	33.1%	$0.016	12.3s	0%
72	Claude Sonnet 4	35.0%	$0.022	4.5s	0%
73	Llama 3.1 70B	25.0%	$0.0031	16.4s	0%
74	Claude 3 Haiku	20.6%	$0.0020	3.5s	0%
75	Rocinante 12B	22.5%	$0.0015	13.3s	0%
76	GPT-5.4 Mini	15.0%	$0.0020	1.2s	0%
77	Writer: Palmyra X5	22.5%	$0.0085	13.8s	0%
78	Gemini 3 Flash (Preview, Reasoning)	44.4%	$0.028	49.2s	0%
79	GPT-4o Mini (temp=0)	16.9%	$0.0009	14.5s	0%
80	Qwen3 235B A22B Instruct 2507	21.3%	$0.0011	32.0s	0%
81	Gemini 3.1 Flash Lite (Preview)	11.9%	$0.0017	1.6s	0%
82	WizardLM 2 8x22b	26.9%	$0.0041	55.3s	0%
83	Z.AI GLM 4.7	55.0%	$0.023	2.2m	1%
84	ByteDance Seed 2.0 Lite	25.0%	$0.0067	1.0m	0%
85	Claude Sonnet 4.5	20.6%	$0.024	5.9s	0%
86	DeepSeek V3.1	12.5%	$0.0019	36.6s	0%
87	DeepSeek V3.2	13.8%	$0.0018	42.8s	0%
88	DeepSeek-V2 Chat	5.0%	$0.0023	8.3s	0%
89	DeepSeek V3 (2024-12-26)	4.4%	$0.0023	6.6s	0%
90	DeepSeek V3 (2025-03-24)	5.6%	$0.0015	15.7s	0%
91	GPT-4o, Aug. 6th (temp=0)	8.8%	$0.011	2.6s	0%
92	MoonshotAI: Kimi K2.5	52.5%	$0.017	2.6m	0%
93	Gemini 3 Flash (Preview)	2.5%	$0.0034	3.2s	0%
94	Claude 3.7 Sonnet	15.6%	$0.022	4.3s	0%
95	Qwen 3.5 Flash	34.4%	$0.0071	2.0m	0%
96	Gemini 2.5 Flash	0.6%	$0.0021	1.5s	0%
97	GPT-5.4	2.5%	$0.0048	3.1s	0%
98	Gemma 3 12B	1.3%	$0.0005	11.7s	0%
99	Gemma 3 27B	0.0%	$0.0007	12.6s	0%
100	Gemma 3 4B	0.0%	$0.0003	15.0s	0%
101	GPT-4o, May 13th (temp=1)	16.9%	$0.033	3.0s	0%
102	Grok 4	61.3%	$0.074	1.6m	3%
103	Claude 3.5 Sonnet	18.8%	$0.045	6.9s	0%
104	Mistral NeMO	8.8%	$0.0018	1.6m	0%
105	Gemini 2.5 Pro	46.3%	$0.073	52.1s	0%
106	Qwen 3.5 Plus (2026-02-15)	18.8%	$0.021	2.0m	0%
107	GPT-4o, May 13th (temp=0)	1.9%	$0.043	6.4s	0%
108	Qwen 3.5 35B	35.0%	$0.044	2.4m	0%
109	Qwen 3.5 9B	42.5%	$0.0038	4.7m	0%
110	Claude Sonnet 4.6 (Reasoning)	76.3%	$0.145	2.2m	15%
111	Gemini 3.1 Pro (Preview)	57.5%	$0.120	1.5m	1%
112	Claude Opus 4	18.1%	$0.109	7.1s	0%
113	Qwen 3.5 27B	50.6%	$0.056	4.8m	2%
114	Gemini 3 Pro (Preview)	53.8%	$0.144	1.7m	0%
115	Qwen 3.5 397B A17B	31.3%	$0.028	5.4m	0%
116	Qwen 3.5 122B	49.4%	$0.135	6.4m	0%
50.50%

Individual Scenarios

basic entries

▼

Long text (~1594 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	0	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini High	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	0	0	80.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Sonnet 4.6	100	100	100	100	50	50	50	50	0	0	60.0%
Ministral 8B	100	100	100	50	50	50	50	50	0	0	55.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
MiniMax M2.7	100	100	100	100	100	0	0	0	0	0	50.0%
MiniMax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4.1	100	100	50	50	50	50	50	50	0	0	50.0%
Stealth: Hunter Alpha	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3 32B	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.4 Nano	100	100	100	100	50	50	0	0	0	0	50.0%
Z.AI GLM 4.6	100	100	100	100	50	0	0	0	0	0	45.0%
ByteDance Seed 1.6	100	100	100	100	0	0	0	0	0	0	40.0%
Writer: Palmyra X5	50	50	50	50	50	50	50	50	0	0	40.0%
Qwen 2.5 72B	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	0	0	40.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Ministral 3B	50	50	50	50	50	50	50	50	0	0	40.0%
Rocinante 12B	100	100	100	50	50	0	0	0	0	0	40.0%
Claude Opus 4.5	50	50	50	50	50	50	50	0	0	0	35.0%
Grok 4.20 (Beta)	100	50	50	50	50	50	0	0	0	0	35.0%
Qwen3 235B A22B Instruct 2507	50	50	50	50	50	50	50	0	0	0	35.0%
Mistral Small 4	50	50	50	50	50	50	50	0	0	0	35.0%
Ministral 3 14B	50	50	50	50	50	50	50	0	0	0	35.0%
Claude 3 Haiku	50	50	50	50	50	50	50	0	0	0	35.0%
GPT-5 Mini	100	100	100	0	0	0	0	0	0	0	30.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.4 (Reasoning, Low)	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4.5	50	50	50	50	50	50	0	0	0	0	30.0%
Gemini 2.5 Flash Lite	100	100	100	0	0	0	0	0	0	0	30.0%
WizardLM 2 8x22b	50	50	50	50	50	50	0	0	0	0	30.0%
Ministral 3 3B	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral NeMO	100	100	100	0	0	0	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 3.1 Pro (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Claude Sonnet 4	50	50	50	50	0	0	0	0	0	0	20.0%
Hermes 3 405B	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, Aug. 6th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Medium 3.1	50	50	50	50	0	0	0	0	0	0	20.0%
Claude Opus 4	50	50	50	0	0	0	0	0	0	0	15.0%
Z.AI GLM 4.5	100	50	0	0	0	0	0	0	0	0	15.0%
Claude Haiku 4.5	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
DeepSeek V3.1	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Hermes 3 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Gemini 3 Pro (Preview)	100	0	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.7	100	0	0	0	0	0	0	0	0	0	10.0%
Grok 4	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-4.1 Mini	50	50	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Mini	100	0	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Cohere Command R+ (Aug. 2024)	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 122B	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen 3.5 27B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3 Flash (Preview, Reasoning)	50	0	0	0	0	0	0	0	0	0	5.0%
ByteDance Seed 2.0 Mini	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3 Flash (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
Claude 3.5 Sonnet	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen 3.5 397B A17B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Pro	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 35B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Grok 4 Fast	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 9B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Plus (2026-02-15)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3.1 Flash Lite (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
ByteDance Seed 2.0 Lite	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.7 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 2	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Long text (~1594 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	50	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	50	50	0	80.0%
Ministral 8B	100	100	100	100	100	50	50	50	50	50	75.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 70B	100	100	100	100	100	100	50	50	0	0	70.0%
Z.AI GLM 4.5	100	100	100	100	100	50	50	50	0	0	65.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	50	0	0	0	65.0%
GPT-5.4 Nano	100	100	100	100	100	100	50	0	0	0	65.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3 32B	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 27B	100	100	100	100	100	50	0	0	0	0	55.0%
Z.AI GLM 4.6	100	100	100	100	50	50	50	0	0	0	55.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Sonnet 4.5	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Writer: Palmyra X5	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Llama 3.1 Nemotron 70B	100	50	50	50	50	50	50	0	0	0	40.0%
Arcee AI: Trinity Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Opus 4.6	50	50	50	50	50	50	50	0	0	0	35.0%
Claude Sonnet 4	50	50	50	50	50	50	50	0	0	0	35.0%
Qwen3 235B A22B Instruct 2507	50	50	50	50	50	50	50	0	0	0	35.0%
GPT-4o Mini (temp=1)	100	50	50	50	50	50	0	0	0	0	35.0%
Mistral Small 4	100	50	50	50	50	50	0	0	0	0	35.0%
Qwen 3.5 122B	100	100	100	0	0	0	0	0	0	0	30.0%
Qwen 3.5 35B	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Opus 4	50	50	50	50	50	50	0	0	0	0	30.0%
Qwen 3.5 9B	100	100	100	0	0	0	0	0	0	0	30.0%
WizardLM 2 8x22b	100	50	50	50	50	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o, Aug. 6th (temp=1)	100	50	50	50	0	0	0	0	0	0	25.0%
DeepSeek V3.1	100	50	50	50	0	0	0	0	0	0	25.0%
Qwen 2.5 72B	100	50	50	50	0	0	0	0	0	0	25.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
DeepSeek V3.2	50	50	50	50	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Small Creative	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral NeMO	100	100	0	0	0	0	0	0	0	0	20.0%
Llama 3.1 8B	50	50	50	50	0	0	0	0	0	0	20.0%
Rocinante 12B	100	50	50	0	0	0	0	0	0	0	20.0%
DeepSeek V3 (2025-03-24)	50	50	50	0	0	0	0	0	0	0	15.0%
Ministral 3 14B	50	50	50	0	0	0	0	0	0	0	15.0%
Gemini 3 Flash (Preview, Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Opus 4.5	50	50	0	0	0	0	0	0	0	0	10.0%
Gemini 3 Pro (Preview)	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 Flash	100	0	0	0	0	0	0	0	0	0	10.0%
Hermes 3 405B	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Mini	100	0	0	0	0	0	0	0	0	0	10.0%
Cohere Command R+ (Aug. 2024)	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 3 Flash (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek-V2 Chat	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 2	50	0	0	0	0	0	0	0	0	0	5.0%
Llama 3.1 70B	50	0	0	0	0	0	0	0	0	0	5.0%
Claude 3 Haiku	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
ByteDance Seed 2.0 Lite	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.7 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	100	0	90.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Claude Sonnet 4	100	100	100	100	100	100	50	50	50	0	75.0%
GPT-5.4 Nano	100	100	100	100	100	100	50	50	50	0	75.0%
Ministral 8B	100	100	100	100	100	100	50	50	50	0	75.0%
Z.AI GLM 5	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-5.2	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 405B	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-5 Nano	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 70B	100	100	100	100	100	50	50	50	0	0	65.0%
Z.AI GLM 4.6	100	100	100	100	100	50	50	0	0	0	60.0%
MiniMax M2.5	100	100	100	100	100	100	0	0	0	0	60.0%
Stealth: Healer Alpha	100	100	100	100	100	100	0	0	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Haiku 4.5	100	50	50	50	50	50	50	50	50	50	55.0%
Grok 4	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 9B	100	100	100	100	100	0	0	0	0	0	50.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	50	50	50.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	50	50	50.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Sonnet 4.6	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	50	50	50	0	0	0	0	45.0%
Qwen 2.5 72B	100	50	50	50	50	50	50	50	0	0	45.0%
GPT-5	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
MoonshotAI: Kimi K2.5	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 27B	100	100	100	50	50	0	0	0	0	0	40.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 4.7	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 4.5	100	100	100	50	50	0	0	0	0	0	40.0%
Claude 3.5 Sonnet	100	100	100	100	0	0	0	0	0	0	40.0%
Ministral 3 14B	50	50	50	50	50	50	50	50	0	0	40.0%
GPT-4.1 Nano	100	100	100	100	0	0	0	0	0	0	40.0%
WizardLM 2 8x22b	100	50	50	50	50	50	0	0	0	0	35.0%
Qwen 3.5 122B	100	100	100	0	0	0	0	0	0	0	30.0%
Stealth: Hunter Alpha	100	100	100	0	0	0	0	0	0	0	30.0%
Grok 4 Fast	100	100	100	0	0	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	50	0	0	0	0	30.0%
Grok 4.20 (Beta)	100	50	50	50	50	0	0	0	0	0	30.0%
Writer: Palmyra X5	100	100	50	50	0	0	0	0	0	0	30.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Large 3	50	50	50	50	50	0	0	0	0	0	25.0%
Claude 3.7 Sonnet	100	100	50	0	0	0	0	0	0	0	25.0%
Mistral Large 2	50	50	50	50	50	0	0	0	0	0	25.0%
Mistral Small 4	50	50	50	50	50	0	0	0	0	0	25.0%
Qwen 3.5 35B	100	100	0	0	0	0	0	0	0	0	20.0%
Llama 3.1 70B	100	50	50	0	0	0	0	0	0	0	20.0%
Mistral NeMO	100	50	50	0	0	0	0	0	0	0	20.0%
DeepSeek V3.1	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o Mini (temp=0)	50	50	50	0	0	0	0	0	0	0	15.0%
Claude 3 Haiku	50	50	50	0	0	0	0	0	0	0	15.0%
Rocinante 12B	100	50	0	0	0	0	0	0	0	0	15.0%
GPT-5.4 (Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 (Reasoning, Low)	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Opus 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Sonnet 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 Flash	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 3.1 Flash Lite (Preview)	50	50	0	0	0	0	0	0	0	0	10.0%
ByteDance Seed 2.0 Lite	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Mini	100	0	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Mistral Large	50	50	0	0	0	0	0	0	0	0	10.0%
Qwen3 235B A22B Instruct 2507	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-4o Mini (temp=1)	50	50	0	0	0	0	0	0	0	0	10.0%
Claude Opus 4	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=1)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, Aug. 6th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 2.5 Flash	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Small 3.2 24B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemma 3 12B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash Lite	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	50	0	0	75.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	50	0	0	75.0%
GPT-4.1 Nano	100	100	100	100	100	50	50	50	50	50	75.0%
GPT-5.2	100	100	100	100	100	100	100	0	0	0	70.0%
Aion 2.0	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Grok 4 Fast	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3 32B	100	100	100	100	100	100	100	0	0	0	70.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	50	50	0	0	70.0%
GPT-4.1 Mini	100	100	100	100	50	50	50	50	50	0	65.0%
Qwen 3.5 122B	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Haiku 4.5	100	100	50	50	50	50	50	50	50	50	60.0%
Llama 3.1 70B	100	100	100	100	100	50	50	0	0	0	60.0%
Qwen 2.5 72B	100	100	100	100	50	50	50	50	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 397B A17B	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	100	100	0	0	0	0	0	50.0%
Grok 4	100	100	100	100	100	0	0	0	0	0	50.0%
Hermes 3 70B	100	100	100	50	50	50	50	0	0	0	50.0%
Ministral 3 14B	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Sonnet 4.6	50	50	50	50	50	50	50	50	50	0	45.0%
Claude Sonnet 4	100	100	100	100	50	0	0	0	0	0	45.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 8B	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Qwen 3.5 9B	100	100	100	100	0	0	0	0	0	0	40.0%
Stealth: Healer Alpha	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral Large 3	50	50	50	50	50	50	50	50	0	0	40.0%
Claude 3.5 Sonnet	100	100	100	50	50	0	0	0	0	0	40.0%
Grok 4.20 (Beta)	100	100	100	50	50	0	0	0	0	0	40.0%
Hermes 3 405B	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	0	0	40.0%
Arcee AI: Trinity Large (Preview)	100	50	50	50	50	50	50	0	0	0	40.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	0	0	40.0%
GPT-4.1	100	100	50	50	50	0	0	0	0	0	35.0%
Ministral 3 8B	50	50	50	50	50	50	50	0	0	0	35.0%
Rocinante 12B	100	100	100	50	0	0	0	0	0	0	35.0%
Z.AI GLM 4.7	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-4o, Aug. 6th (temp=1)	100	50	50	50	50	0	0	0	0	0	30.0%
Mistral Large 2	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Large	50	50	50	50	50	50	0	0	0	0	30.0%
Claude Sonnet 4.5	100	100	50	0	0	0	0	0	0	0	25.0%
DeepSeek V3.2	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o Mini (temp=1)	100	50	50	50	0	0	0	0	0	0	25.0%
GPT-5	100	100	0	0	0	0	0	0	0	0	20.0%
Stealth: Hunter Alpha	100	100	0	0	0	0	0	0	0	0	20.0%
DeepSeek-V2 Chat	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-5.4 Mini	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=0)	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Small 4	50	50	50	50	0	0	0	0	0	0	20.0%
Claude 3 Haiku	50	50	50	50	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
DeepSeek V3 (2024-12-26)	100	50	0	0	0	0	0	0	0	0	15.0%
WizardLM 2 8x22b	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 8B	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-5.4 (Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 35B	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 Plus (2026-02-15)	50	50	0	0	0	0	0	0	0	0	10.0%
GPT-5.4	100	0	0	0	0	0	0	0	0	0	10.0%
Writer: Palmyra X5	50	50	0	0	0	0	0	0	0	0	10.0%
Claude 3.7 Sonnet	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.1	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen3 235B A22B Instruct 2507	50	0	0	0	0	0	0	0	0	0	5.0%
MoonshotAI: Kimi K2.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3.1 Flash Lite (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
ByteDance Seed 2.0 Lite	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash Lite	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

detailed entries

▼

Long text (~1594 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	50	95.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	0	90.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	0	0	80.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	0	0	80.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	50	0	0	70.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 2.5 72B	100	100	100	100	100	50	50	50	0	0	65.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Sonnet 4.6	100	100	50	50	50	50	50	50	50	50	60.0%
Z.AI GLM 4.7	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 35B	100	100	100	100	100	100	0	0	0	0	60.0%
Arcee AI: Trinity Large (Preview)	100	100	100	50	50	50	50	50	50	0	60.0%
Z.AI GLM 4.6	100	100	100	100	100	50	0	0	0	0	55.0%
Qwen 3 32B	100	100	100	100	100	50	0	0	0	0	55.0%
Claude Sonnet 4.5	100	100	50	50	50	50	50	50	0	0	50.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Mistral Medium 3.1	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-4.1 Nano	100	100	100	100	50	50	0	0	0	0	50.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	0	45.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 3 Pro (Preview)	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 2.5 Flash Lite	100	100	100	50	50	0	0	0	0	0	40.0%
Qwen3 235B A22B Instruct 2507	100	50	50	50	50	50	50	0	0	0	40.0%
Ministral 3B	50	50	50	50	50	50	50	0	0	0	35.0%
Qwen 3.5 9B	100	100	50	50	0	0	0	0	0	0	30.0%
GPT-4o, Aug. 6th (temp=1)	100	100	50	50	0	0	0	0	0	0	30.0%
Writer: Palmyra X5	50	50	50	50	50	50	0	0	0	0	30.0%
Mistral Small 3.2 24B	100	100	100	0	0	0	0	0	0	0	30.0%
WizardLM 2 8x22b	100	100	50	50	0	0	0	0	0	0	30.0%
Ministral 3 3B	50	50	50	50	50	50	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Grok 4.20 (Beta)	100	50	50	50	0	0	0	0	0	0	25.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	50	0	0	0	0	0	25.0%
GPT-4o, Aug. 6th (temp=0)	50	50	50	50	50	0	0	0	0	0	25.0%
DeepSeek V3.1	50	50	50	50	50	0	0	0	0	0	25.0%
Claude Opus 4	50	50	50	50	0	0	0	0	0	0	20.0%
GPT-5.4 Mini	100	100	0	0	0	0	0	0	0	0	20.0%
DeepSeek V3.2	50	50	50	50	0	0	0	0	0	0	20.0%
Mistral Small 4	50	50	50	50	0	0	0	0	0	0	20.0%
Claude Opus 4.5	50	50	50	0	0	0	0	0	0	0	15.0%
Claude 3.7 Sonnet	100	50	0	0	0	0	0	0	0	0	15.0%
Mistral Large	50	50	50	0	0	0	0	0	0	0	15.0%
GPT-4o Mini (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
Llama 3.1 Nemotron 70B	50	50	50	0	0	0	0	0	0	0	15.0%
Hermes 3 70B	100	50	0	0	0	0	0	0	0	0	15.0%
Claude 3 Haiku	50	50	50	0	0	0	0	0	0	0	15.0%
Rocinante 12B	50	50	50	0	0	0	0	0	0	0	15.0%
Gemini 3 Flash (Preview)	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek-V2 Chat	50	50	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.7 Flash	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 3.5 27B	50	0	0	0	0	0	0	0	0	0	5.0%
Claude Sonnet 4	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3.1 Flash Lite (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 3	50	0	0	0	0	0	0	0	0	0	5.0%
Hermes 3 405B	50	0	0	0	0	0	0	0	0	0	5.0%
Mistral Large 2	50	0	0	0	0	0	0	0	0	0	5.0%
Llama 3.1 70B	50	0	0	0	0	0	0	0	0	0	5.0%
LFM2 24B	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 2.5 Pro	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3.5 Plus (2026-02-15)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Long text (~1594 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	50	0	85.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	0	0	80.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	50	50	0	75.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	50	0	0	75.0%
MiniMax M2.7	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.6	100	100	100	100	100	100	50	0	0	0	65.0%
MiniMax M2.5	100	100	100	100	100	100	50	0	0	0	65.0%
Mistral Medium 3.1	100	100	100	100	50	50	50	50	50	0	65.0%
Qwen 3.5 27B	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 35B	100	100	100	100	100	100	0	0	0	0	60.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	50	50	0	0	0	60.0%
Hermes 3 70B	100	100	100	50	50	50	50	50	50	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	50	50	50	0	0	0	55.0%
GPT-5.2	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Opus 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Z.AI GLM 4.7	100	100	100	100	100	0	0	0	0	0	50.0%
Grok 4 Fast	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-4.1 Mini	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 8B	100	100	100	50	50	50	0	0	0	0	45.0%
Claude Opus 4.6	50	50	50	50	50	50	50	50	0	0	40.0%
Aion 2.0	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral Small 4	100	50	50	50	50	50	50	0	0	0	40.0%
Qwen 2.5 72B	100	100	50	50	50	50	0	0	0	0	40.0%
Llama 3.1 Nemotron 70B	100	100	50	50	50	50	0	0	0	0	40.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	50	50	0	0	0	35.0%
WizardLM 2 8x22b	50	50	50	50	50	50	50	0	0	0	35.0%
Qwen 3.5 122B	100	100	100	0	0	0	0	0	0	0	30.0%
ByteDance Seed 1.6	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4	100	50	50	50	50	0	0	0	0	0	30.0%
Grok 4	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Opus 4	50	50	50	50	50	50	0	0	0	0	30.0%
GPT-5.4 Nano	100	50	50	50	50	0	0	0	0	0	30.0%
Ministral 3 8B	100	100	50	50	0	0	0	0	0	0	30.0%
Mistral Large 3	50	50	50	50	50	0	0	0	0	0	25.0%
Grok 4.20 (Beta)	100	100	50	0	0	0	0	0	0	0	25.0%
Ministral 3 3B	50	50	50	50	50	0	0	0	0	0	25.0%
Llama 3.1 8B	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 2.5 Pro	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Flash (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Hermes 3 405B	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Flash Lite	100	100	0	0	0	0	0	0	0	0	20.0%
Llama 3.1 70B	100	100	0	0	0	0	0	0	0	0	20.0%
Claude 3 Haiku	50	50	50	50	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Mini	100	100	0	0	0	0	0	0	0	0	20.0%
Mistral Large 2	50	50	50	0	0	0	0	0	0	0	15.0%
Mistral Small 3.2 24B	100	50	0	0	0	0	0	0	0	0	15.0%
Mistral Small Creative	50	50	50	0	0	0	0	0	0	0	15.0%
Ministral 3 14B	50	50	50	0	0	0	0	0	0	0	15.0%
MoonshotAI: Kimi K2.5	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4	100	0	0	0	0	0	0	0	0	0	10.0%
Claude 3.7 Sonnet	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.1	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
Mistral Large	50	50	0	0	0	0	0	0	0	0	10.0%
Qwen3 235B A22B Instruct 2507	50	50	0	0	0	0	0	0	0	0	10.0%
Writer: Palmyra X5	50	50	0	0	0	0	0	0	0	0	10.0%
Claude Sonnet 4.5	50	0	0	0	0	0	0	0	0	0	5.0%
Gemini 3.1 Flash Lite (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek-V2 Chat	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3 (2024-12-26)	50	0	0	0	0	0	0	0	0	0	5.0%
Rocinante 12B	50	0	0	0	0	0	0	0	0	0	5.0%
Qwen 3.5 397B A17B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, May 13th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
ByteDance Seed 2.0 Lite	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2025-03-24)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	50	50	90.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.1	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 Nano	100	100	100	100	100	100	50	50	50	50	80.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	0	0	0	70.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	0	0	0	70.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	0	0	0	70.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Flash	100	100	100	100	100	100	50	0	0	0	65.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5	100	100	100	100	100	100	0	0	0	0	60.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 27B	100	100	100	100	100	100	0	0	0	0	60.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-4.1 Mini	100	100	100	100	50	50	50	50	0	0	60.0%
Mistral Medium 3.1	100	100	100	100	50	50	50	50	0	0	60.0%
Ministral 3 14B	100	50	50	50	50	50	50	50	50	50	55.0%
Stealth: Healer Alpha	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 2.5 72B	100	100	50	50	50	50	50	50	0	0	50.0%
GPT-4.1 Nano	100	100	50	50	50	50	50	50	0	0	50.0%
Mistral Small Creative	50	50	50	50	50	50	50	50	50	0	45.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Claude 3.7 Sonnet	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-5.4 Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3 32B	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Haiku 4.5	50	50	50	50	50	50	50	0	0	0	35.0%
Claude 3.5 Sonnet	100	100	50	50	50	0	0	0	0	0	35.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	0	0	0	35.0%
WizardLM 2 8x22b	100	100	50	50	50	0	0	0	0	0	35.0%
Ministral 3 3B	50	50	50	50	50	50	50	0	0	0	35.0%
Ministral 3B	50	50	50	50	50	50	50	0	0	0	35.0%
Gemini 3.1 Pro (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Claude Sonnet 4	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 2.5 Flash Lite	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Small 3.2 24B	100	100	50	50	0	0	0	0	0	0	30.0%
Llama 3.1 8B	50	50	50	50	50	50	0	0	0	0	30.0%
Claude Opus 4	100	50	50	50	0	0	0	0	0	0	25.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
Grok 4.20 (Beta)	100	100	50	0	0	0	0	0	0	0	25.0%
Qwen3 235B A22B Instruct 2507	100	100	50	0	0	0	0	0	0	0	25.0%
Mistral Small 4	50	50	50	50	50	0	0	0	0	0	25.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Hermes 3 70B	50	50	50	50	50	0	0	0	0	0	25.0%
Claude Sonnet 4.6 (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
DeepSeek V3 (2025-03-24)	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o Mini (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Llama 3.1 70B	100	50	50	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	0	0	0	0	0	0	0	15.0%
Writer: Palmyra X5	50	50	50	0	0	0	0	0	0	0	15.0%
Hermes 3 405B	100	0	0	0	0	0	0	0	0	0	10.0%
Claude 3 Haiku	50	50	0	0	0	0	0	0	0	0	10.0%
Rocinante 12B	50	50	0	0	0	0	0	0	0	0	10.0%
Gemini 3.1 Flash Lite (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.1	50	0	0	0	0	0	0	0	0	0	5.0%
DeepSeek V3.2	50	0	0	0	0	0	0	0	0	0	5.0%
Gemma 3 12B	50	0	0	0	0	0	0	0	0	0	5.0%
LFM2 24B	50	0	0	0	0	0	0	0	0	0	5.0%
Claude Sonnet 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Short text (~524 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	0	90.0%
o4 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	50	0	85.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	50	50	50	85.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.2	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	0	0	80.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	0	0	80.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	0	0	80.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
MiniMax M2.5	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Hermes 3 405B	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-4.1 Nano	100	100	100	50	50	50	50	50	50	50	65.0%
Z.AI GLM 4.5	100	100	100	100	100	50	50	0	0	0	60.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	0	0	0	0	60.0%
LFM2 24B	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 70B	100	100	100	100	50	50	50	0	0	0	55.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	50	50	0	0	0	0	50.0%
Mistral Medium 3.1	100	100	100	50	50	50	0	0	0	0	45.0%
Arcee AI: Trinity Large (Preview)	50	50	50	50	50	50	50	50	50	0	45.0%
Mistral Small Creative	100	100	50	50	50	50	50	0	0	0	45.0%
Claude 3 Haiku	50	50	50	50	50	50	50	50	50	0	45.0%
Ministral 3B	50	50	50	50	50	50	50	50	50	0	45.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-5	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 397B A17B	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Sonnet 4	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 Flash	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 9B	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3 32B	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 2.5 72B	100	50	50	50	50	50	50	0	0	0	40.0%
Hermes 3 70B	100	100	100	50	50	0	0	0	0	0	40.0%
Ministral 3 3B	50	50	50	50	50	50	50	50	0	0	40.0%
Rocinante 12B	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-4o Mini (temp=1)	50	50	50	50	50	50	50	0	0	0	35.0%
Llama 3.1 Nemotron 70B	50	50	50	50	50	50	50	0	0	0	35.0%
Llama 3.1 8B	50	50	50	50	50	50	50	0	0	0	35.0%
Qwen 3.5 35B	100	100	100	0	0	0	0	0	0	0	30.0%
Claude 3.5 Sonnet	100	100	100	0	0	0	0	0	0	0	30.0%
Claude 3.7 Sonnet	100	100	50	50	0	0	0	0	0	0	30.0%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	0	0	0	0	0	25.0%
Mistral Small 4	50	50	50	50	50	0	0	0	0	0	25.0%
Gemini 3.1 Pro (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Claude Opus 4	100	100	0	0	0	0	0	0	0	0	20.0%
ByteDance Seed 2.0 Lite	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-4o, May 13th (temp=1)	50	50	50	50	0	0	0	0	0	0	20.0%
Grok 4.20 (Beta)	100	50	0	0	0	0	0	0	0	0	15.0%
ByteDance Seed 2.0 Mini	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Mini	100	0	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	50	50	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3 (2025-03-24)	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash Lite	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen3 235B A22B Instruct 2507	50	50	0	0	0	0	0	0	0	0	10.0%
Gemini 3.1 Flash Lite (Preview)	50	0	0	0	0	0	0	0	0	0	5.0%
GPT-4o, May 13th (temp=0)	50	0	0	0	0	0	0	0	0	0	5.0%
WizardLM 2 8x22b	50	0	0	0	0	0	0	0	0	0	5.0%
Claude Sonnet 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 3 Flash (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3 (2024-12-26)	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash	0	0	0	0	0	0	0	0	0	0	0.0%
Writer: Palmyra X5	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%

Correct "no violations" response

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

basic entries

Long text (~1594 words), big codex (51 entries)

Long text (~1594 words), small codex (11 entries)

Short text (~524 words), big codex (51 entries)

Short text (~524 words), small codex (11 entries)

detailed entries

Long text (~1594 words), big codex (51 detailed entries)

Long text (~1594 words), small codex (11 detailed entries)

Short text (~524 words), big codex (51 detailed entries)

Short text (~524 words), small codex (11 detailed entries)