No hallucinated violations

Test: Codex Red Herring (False Positive Detection)

Avg. Score

61.1%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Inception Mercury	98.8%	$0.0004	8.0s	84%
2	Inception Mercury 2	97.5%	$0.0027	3.8s	78%
3	Grok 4.1 Fast	97.9%	$0.0019	12.5s	79%
4	Nemotron 3 Super	99.4%	$0.0000	1.3m	89%
5	GPT-5.4 Mini (Reasoning, Low)	96.9%	$0.0034	4.0s	76%
6	ByteDance Seed 1.6 Flash	95.7%	$0.0008	9.1s	69%
7	o4 Mini	97.5%	$0.014	25.0s	78%
8	Z.AI GLM 5 Turbo	96.9%	$0.0071	16.0s	73%
9	GPT-5.4 Nano (Reasoning, Low)	96.7%	$0.0008	3.9s	66%
10	GPT-5.4 Nano (Reasoning)	95.2%	$0.0017	11.4s	66%
11	Gemini 2.5 Flash Lite (Reasoning)	94.4%	$0.0023	16.6s	68%
12	GPT-5.1	96.9%	$0.025	26.1s	76%
13	o4 Mini High	98.1%	$0.027	52.5s	81%
14	GPT-5 Mini	95.0%	$0.0059	37.8s	67%
15	MiniMax M2.7	93.1%	$0.0047	34.6s	66%
16	DeepSeek V4 Flash (Reasoning)	92.3%	$0.0009	30.8s	63%
17	GPT-5 Nano	95.0%	$0.0035	1.1m	70%
18	GPT-5.4 Mini (Reasoning)	92.3%	$0.0090	10.8s	63%
19	Gemini 2.5 Flash (Reasoning)	91.3%	$0.0085	14.2s	62%
20	Stealth: Healer Alpha	89.4%	$0.0000	21.5s	59%
21	Grok 4.3 (Reasoning)	94.4%	$0.014	50.4s	66%
22	Xiaomi MIMO v2.5	90.4%	$0.0075	30.3s	60%
23	GPT-OSS 120B	90.6%	$0.0012	56.4s	61%
24	ByteDance Seed 1.6	89.6%	$0.0043	32.7s	56%
25	Mistral Small 4 (Reasoning)	87.5%	$0.0025	22.6s	53%
26	Z.AI GLM 5	95.2%	$0.017	1.4m	69%
27	GPT-5.2	89.3%	$0.013	14.5s	55%
28	Aion 2.0	91.9%	$0.0096	1.3m	63%
29	Grok 4.20 (Beta, Reasoning)	91.5%	$0.025	15.9s	57%
30	MiniMax M2.5	85.5%	$0.0032	25.9s	50%
31	Qwen 3.5 Plus (2026-04-20)	95.0%	$0.020	1.9m	70%
32	GPT-4.1	85.9%	$0.0048	1.2s	43%
33	Grok 4.20 (Reasoning)	90.2%	$0.017	47.9s	55%
34	Xiaomi MIMO v2.5 Pro	89.8%	$0.015	1.1m	57%
35	Gemma 4 26B (Reasoning)	94.4%	$0.0027	2.6m	64%
36	Grok 4 Fast	80.3%	$0.0019	19.2s	44%
37	Qwen 3.6 Flash	84.8%	$0.016	44.4s	53%
38	GPT-5.4 (Reasoning, Low)	82.5%	$0.012	8.8s	44%
39	Z.AI GLM 5.1	94.8%	$0.024	2.1m	66%
40	Qwen 3 32B	77.9%	$0.0014	37.7s	44%
41	Gemma 4 31B	77.3%	$0.0009	11.7s	39%
42	Stealth: Hunter Alpha	80.3%	$0.0000	1.0m	46%
43	Claude Opus 4.7 (Reasoning)	94.4%	$0.075	13.1s	66%
44	Nemotron 3 Nano	91.5%	$0.0027	2.9m	61%
45	Arcee AI: Trinity Mini	79.2%	$0.0009	26.3s	33%
46	Z.AI GLM 4.5 Air	81.8%	$0.0027	43.1s	35%
47	Qwen 3.6 35B	80.8%	$0.017	1.2m	50%
48	GPT-5.4 Nano	69.5%	$0.0005	1.5s	28%
49	Qwen 3.6 27B	90.7%	$0.033	2.3m	57%
50	Gemma 4 31B (Reasoning)	87.7%	$0.0025	3.1m	52%
51	GPT-5.4 (Reasoning)	78.9%	$0.032	31.9s	43%
52	Z.AI GLM 4.7 Flash	83.4%	$0.0039	2.5m	49%
53	Gemma 4 26B	68.8%	$0.0012	39.3s	27%
54	GPT-5	83.5%	$0.048	1.5m	52%
55	GPT-5.4 Mini	55.4%	$0.0020	1.2s	30%
56	Ministral 3 8B	74.5%	$0.0013	17.9s	14%
57	Z.AI GLM 4.6	74.6%	$0.015	59.0s	31%
58	Claude Opus 4.6 (Reasoning)	96.3%	$0.120	1.0m	74%
59	Claude Opus 4.6	79.7%	$0.049	13.2s	34%
60	MoonshotAI: Kimi K2.5	75.6%	$0.017	2.6m	48%
61	Z.AI GLM 4.7	75.9%	$0.023	2.2m	46%
62	DeepSeek V4 Pro (Reasoning)	82.5%	$0.017	3.4m	52%
63	LFM2 24B	71.9%	$0.0004	39.0s	13%
64	ByteDance Seed 2.0 Mini	79.8%	$0.0033	3.1m	41%
65	Claude Haiku 4.5	51.9%	$0.0080	4.2s	28%
66	Z.AI GLM 4.5	63.4%	$0.0038	25.0s	19%
67	GPT-5.5 (Reasoning, Low)	67.7%	$0.025	12.0s	24%
68	GPT-5.4	40.2%	$0.0048	3.1s	34%
69	GPT-5.5	55.9%	$0.013	2.9s	21%
70	Claude Sonnet 4.6	66.2%	$0.033	17.4s	25%
71	GPT-4.1 Nano	58.4%	$0.0005	4.6s	8%
72	Ministral 8B	60.8%	$0.0007	4.4s	6%
73	ByteDance Seed 2.0 Lite	57.9%	$0.0067	1.0m	25%
74	Cohere Command R+ (Aug. 2024)	60.6%	$0.017	7.4s	15%
75	Gemini 3 Flash (Preview, Reasoning)	67.6%	$0.028	49.2s	20%
76	Qwen 3.5 9B	82.8%	$0.0038	4.7m	44%
77	Gemini 3.1 Flash Lite (Reasoning)	40.3%	$0.0015	1.9s	19%
78	Gemini 3.1 Flash Lite	41.2%	$0.0017	3.2s	18%
79	Qwen 3.5 Flash	63.6%	$0.0071	2.0m	23%
80	Gemini 2.5 Flash	31.4%	$0.0021	1.5s	23%
81	DeepSeek V3 (2024-12-26)	34.3%	$0.0023	6.6s	21%
82	GPT-4o, Aug. 6th (temp=1)	45.0%	$0.0094	2.4s	13%
83	GPT-5.5 (Reasoning)	66.7%	$0.051	24.1s	23%
84	Hermes 3 405B	47.5%	$0.0059	5.2s	9%
85	Gemini 3.1 Flash Lite (Preview)	35.7%	$0.0017	1.6s	14%
86	Claude Opus 4.5	52.4%	$0.036	4.4s	20%
87	Hermes 3 70B	43.4%	$0.0018	12.1s	7%
88	Grok 4	75.5%	$0.074	1.6m	37%
89	Claude Sonnet 4	45.5%	$0.022	4.5s	13%
90	DeepSeek-V2 Chat	30.9%	$0.0023	8.3s	15%
91	Gemini 3 Flash (Preview)	25.0%	$0.0034	3.2s	20%
92	DeepSeek V3 (2025-03-24)	31.5%	$0.0015	15.7s	15%
93	Mistral Medium 3.1	39.4%	$0.0032	4.7s	5%
94	DeepSeek V4 Pro	36.1%	$0.0069	42.0s	17%
95	GPT-4o, Aug. 6th (temp=0)	31.4%	$0.011	2.6s	14%
96	Qwen 2.5 72B	35.2%	$0.0009	11.3s	6%
97	GPT-4.1 Mini	35.6%	$0.0018	6.9s	4%
98	Claude Sonnet 4.5	32.8%	$0.024	5.9s	19%
99	Grok 4.3	36.1%	$0.0077	4.4s	6%
100	Grok 4.20 (Beta)	35.3%	$0.0061	4.3s	5%
101	Claude Opus 4.7	52.9%	$0.049	3.9s	14%
102	Llama 3.1 70B	33.5%	$0.0031	16.4s	7%
103	Claude 3 Haiku	19.3%	$0.0020	3.5s	16%
104	Rocinante 12B	32.3%	$0.0015	13.3s	6%
105	Qwen3.6 Max Preview	82.8%	$0.072	3.8m	50%
106	Claude 3.7 Sonnet	35.5%	$0.022	4.3s	13%
107	GPT-4o Mini (temp=1)	28.2%	$0.0008	5.6s	6%
108	Gemini 2.5 Flash Lite	32.1%	$0.0010	4.0s	2%
109	Gemini 2.5 Pro	70.4%	$0.073	52.1s	21%
110	Mistral Small 3.2 24B	31.3%	$0.0010	11.9s	3%
111	Mistral Large 3	32.2%	$0.0040	11.7s	2%
112	WizardLM 2 8x22b	32.6%	$0.0041	55.3s	12%
113	Qwen3 235B A22B Instruct 2507	26.3%	$0.0011	32.0s	10%
114	Writer: Palmyra X5	24.0%	$0.0085	13.8s	12%
115	GPT-4o Mini (temp=0)	24.5%	$0.0009	14.5s	6%
116	DeepSeek V3.1	24.5%	$0.0019	36.6s	11%
117	Qwen 3.5 35B	64.5%	$0.044	2.4m	23%
118	Gemini 3.1 Pro (Preview)	76.9%	$0.120	1.5m	45%
119	Grok 4.20	25.6%	$0.0082	11.7s	8%
120	Claude 3.5 Sonnet	39.1%	$0.045	6.9s	15%
121	Ministral 3 14B	29.3%	$0.0019	29.2s	1%
122	DeepSeek V3.2	21.3%	$0.0018	42.8s	11%
123	Mistral Large 2	32.2%	$0.016	11.4s	2%
124	DeepSeek V4 Flash	17.8%	$0.0005	12.2s	6%
125	Claude Sonnet 4.6 (Reasoning)	87.5%	$0.145	2.2m	55%
126	Mistral Large	31.9%	$0.016	12.3s	2%
127	Gemma 3 12B	13.7%	$0.0005	11.7s	10%
128	MoonshotAI: Kimi K2.6	76.9%	$0.047	5.2m	47%
129	Gemma 3 27B	12.5%	$0.0007	12.6s	10%
130	GPT-4o, May 13th (temp=1)	23.5%	$0.033	3.0s	16%
131	Llama 3.1 Nemotron 70B	22.6%	$0.0077	21.4s	5%
132	Mistral Small 4	12.3%	$0.0012	7.4s	6%
133	Ministral 3B	9.9%	$0.0004	12.4s	6%
134	Mistral Small Creative	9.3%	$0.0011	8.9s	4%
135	Llama 3.1 8B	10.0%	$0.0003	33.8s	8%
136	Gemini 3 Pro (Preview)	75.6%	$0.144	1.7m	47%
137	Gemma 3 4B	5.6%	$0.0003	15.0s	5%
138	Ministral 3 3B	6.7%	$0.0013	31.1s	5%
139	GPT-4o, May 13th (temp=0)	16.7%	$0.043	6.4s	11%
140	Mistral NeMO	14.7%	$0.0018	1.6m	4%
141	Arcee AI: Trinity Large (Preview)	15.7%	$0.0000	1.7m	2%
142	Qwen 3.5 Plus (2026-02-15)	22.0%	$0.021	2.0m	12%
143	Qwen 3.5 397B A17B	61.5%	$0.028	5.4m	23%
144	Claude Opus 4	36.0%	$0.109	7.1s	22%
145	Qwen 3.5 27B	68.8%	$0.056	4.8m	19%
146	Qwen 3.5 122B	70.1%	$0.135	6.4m	20%
61.12%

Individual Scenarios

basic entries

▼

Long text (~1594 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	50	95.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	50	95.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	50	95.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	33	93.3%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	33	93.3%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	33	93.3%
GPT-5.2	100	100	100	100	100	100	100	100	100	33	93.3%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	33	93.3%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	20	92.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	2	90.2%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	1	90.1%
o4 Mini High	100	100	100	100	100	100	100	100	50	50	90.0%
o4 Mini	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-OSS 120B	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	50	50	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	33	33	86.7%
Z.AI GLM 5	100	100	100	100	100	100	100	100	33	33	86.7%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	50	50	50	85.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	50	50	50	85.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	50	50	33	83.3%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	50	50	33	83.3%
Arcee AI: Trinity Mini	100	100	100	100	100	100	50	50	50	50	80.0%
MiniMax M2.7	100	100	100	100	100	50	50	50	50	50	75.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	50	33	33	33	75.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	33	33	33	33	73.3%
Qwen 3 32B	100	100	100	100	100	50	50	50	50	25	72.5%
Stealth: Hunter Alpha	100	100	100	100	100	50	50	33	33	33	70.0%
MiniMax M2.5	100	100	100	100	100	50	50	33	33	20	68.7%
Z.AI GLM 5.1	100	100	100	100	100	50	33	33	33	33	68.3%
Qwen 3.6 27B	100	100	100	100	100	50	33	33	33	33	68.3%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	50	33	33	33	33	68.3%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	33	33	33	33	33	66.7%
GPT-5.4 (Reasoning)	100	100	100	100	100	33	33	33	33	33	66.7%
Z.AI GLM 4.5 Air	100	100	100	100	100	50	50	33	25	7	66.5%
Z.AI GLM 4.6	100	100	100	100	50	50	50	50	33	0	63.3%
Qwen 3.6 35B	100	100	100	50	50	50	50	50	50	33	63.3%
ByteDance Seed 1.6	100	100	100	100	50	33	33	33	33	33	61.7%
Grok 4.20 (Reasoning)	100	100	100	100	33	33	33	33	33	33	60.0%
Claude Opus 4.7 (Reasoning)	100	100	100	50	50	50	50	33	33	33	60.0%
GPT-5 Mini	100	100	100	50	50	50	50	33	33	33	60.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	33	33	33	33	33	33	60.0%
Qwen 3.6 Flash	100	100	50	50	50	50	50	50	50	33	58.3%
Claude Sonnet 4.6	100	100	100	100	50	33	33	33	20	7	57.7%
GPT-5.4 (Reasoning, Low)	100	100	100	50	33	33	33	33	33	33	55.0%
Gemma 4 26B (Reasoning)	100	100	100	50	33	33	33	33	33	33	55.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	50	33	33	33	33	33	33	55.0%
GPT-5.4 Nano	100	100	100	100	50	33	33	20	5	4	54.5%
GPT-5.5 (Reasoning)	100	100	100	33	33	33	33	33	33	33	53.3%
Gemini 3 Pro (Preview)	100	50	50	50	50	50	50	50	50	33	53.3%
Gemini 3.1 Pro (Preview)	100	100	50	50	50	50	33	33	33	25	52.5%
GPT-5.4 Mini	100	50	50	50	50	50	50	50	33	33	51.7%
Qwen3.6 Max Preview	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-5.4 Mini (Reasoning)	50	50	50	50	50	50	50	50	50	33	48.3%
Z.AI GLM 4.7	100	50	50	50	50	33	33	33	33	33	46.7%
Gemma 4 31B (Reasoning)	100	100	33	33	33	33	33	33	33	33	46.7%
Rocinante 12B	100	100	100	50	25	20	20	20	10	6	45.1%
Gemini 2.5 Pro	50	50	50	50	50	50	50	33	33	33	45.0%
Qwen 3.5 35B	50	50	50	50	50	50	33	33	33	25	42.5%
Qwen 3.5 9B	50	50	50	50	50	50	33	33	33	25	42.5%
Grok 4	100	33	33	33	33	33	33	33	33	33	40.0%
Qwen 3.5 Flash	50	50	50	50	33	33	33	33	33	33	40.0%
GPT-4.1	100	100	50	33	33	20	17	17	17	10	39.7%
Hermes 3 405B	100	100	33	33	25	20	20	17	13	13	37.3%
Qwen 3.5 27B	50	50	33	33	33	33	33	33	33	25	35.8%
Qwen 3.5 397B A17B	50	33	33	33	33	33	33	33	33	25	34.2%
ByteDance Seed 2.0 Lite	50	33	33	33	33	33	33	33	25	25	33.3%
Gemini 2.5 Flash Lite	100	100	100	8	8	4	4	3	1	1	32.9%
GPT-4.1 Mini	50	50	50	50	50	50	8	8	8	4	32.8%
Mistral NeMO	100	100	100	8	6	5	4	3	0	0	32.6%
Qwen 3.5 122B	33	33	33	33	33	33	33	33	33	25	32.5%
GPT-5.5	50	50	33	33	33	33	25	25	25	17	32.5%
Ministral 8B	100	100	100	6	3	3	2	2	2	2	32.0%
ByteDance Seed 2.0 Mini	33	33	33	33	33	33	33	33	33	20	32.0%
Claude Opus 4.5	50	33	33	33	33	33	33	25	25	17	31.7%
Grok 4 Fast	33	33	33	33	33	33	33	25	25	25	30.8%
Gemini 3 Flash (Preview, Reasoning)	50	33	33	33	25	25	25	25	25	20	29.5%
Z.AI GLM 4.5	100	50	33	33	25	20	11	8	5	2	28.8%
GPT-5.4	33	33	33	33	25	25	25	25	20	20	27.3%
Claude Haiku 4.5	50	33	25	25	25	25	25	20	20	20	26.8%
Cohere Command R+ (Aug. 2024)	100	33	33	20	17	17	14	14	10	9	26.8%
Claude Opus 4.7	33	33	33	25	25	25	20	20	20	20	25.5%
Hermes 3 70B	100	50	25	14	13	11	10	9	8	8	24.8%
DeepSeek V3 (2024-12-26)	50	50	33	25	25	20	10	9	8	5	23.6%
Llama 3.1 70B	100	33	33	17	13	10	8	7	6	6	23.4%
Claude 3 Haiku	25	25	25	25	25	25	25	25	20	13	23.3%
Claude 3.5 Sonnet	33	33	25	20	20	20	20	20	17	14	22.3%
DeepSeek V4 Pro	50	50	33	33	20	17	8	5	4	2	22.3%
Gemini 2.5 Flash	33	33	25	25	25	20	17	14	10	10	21.3%
GPT-4o, Aug. 6th (temp=1)	50	50	20	17	17	14	14	11	10	7	21.0%
Claude Sonnet 4	33	25	20	20	20	20	17	17	17	17	20.5%
Grok 4.3	100	17	17	14	11	10	9	9	6	3	19.7%
Gemini 3.1 Flash Lite (Preview)	25	20	20	20	20	20	20	17	17	10	18.8%
Claude Opus 4	25	25	25	20	20	17	17	17	11	9	18.5%
Gemma 4 31B	25	25	20	20	20	17	17	17	13	11	18.4%
Gemini 3.1 Flash Lite	20	20	20	20	20	17	17	17	14	14	17.9%
Gemini 3.1 Flash Lite (Reasoning)	20	20	20	20	17	17	17	17	13	13	17.2%
DeepSeek V3 (2025-03-24)	25	25	20	20	17	14	14	13	13	8	16.9%
DeepSeek-V2 Chat	50	33	17	13	13	11	11	9	5	4	16.6%
WizardLM 2 8x22b	50	50	25	20	7	4	3	3	1	1	16.4%
Grok 4.20 (Beta)	100	13	11	11	8	5	5	5	3	3	16.4%
GPT-4o, May 13th (temp=1)	25	25	20	20	17	14	13	11	8	7	15.9%
Gemma 4 26B	33	25	17	14	13	13	11	11	11	11	15.9%
Claude 3.7 Sonnet	20	17	17	17	17	17	14	14	13	9	15.3%
Claude Sonnet 4.5	20	17	17	17	14	14	14	14	14	11	15.3%
Gemini 3 Flash (Preview)	17	17	14	14	14	14	10	9	9	6	12.4%
GPT-4o Mini (temp=1)	25	13	11	11	10	10	8	8	8	7	11.1%
GPT-4o, Aug. 6th (temp=0)	14	14	13	13	11	11	9	8	8	7	10.9%
Qwen 3.5 Plus (2026-02-15)	17	14	13	13	11	9	8	8	7	2	10.1%
Mistral Small 4	50	8	8	7	6	5	5	3	3	3	9.8%
DeepSeek V3.1	17	17	13	13	9	9	7	7	1	1	9.3%
Mistral Large 2	17	11	11	11	10	8	7	6	6	5	9.2%
Llama 3.1 Nemotron 70B	13	11	11	10	10	9	9	7	7	6	9.2%
Gemma 3 27B	11	11	11	9	9	9	8	8	7	5	8.9%
Gemma 3 12B	25	9	8	8	8	7	6	6	5	4	8.6%
Qwen 2.5 72B	20	14	13	8	7	6	6	5	5	2	8.6%
Mistral Large 3	14	11	10	10	8	7	7	7	6	4	8.3%
Llama 3.1 8B	33	11	9	6	5	5	5	4	2	0	8.2%
Ministral 3B	25	14	9	7	6	5	5	3	3	3	8.0%
Writer: Palmyra X5	33	10	8	5	4	4	3	3	3	3	7.7%
Ministral 3 3B	33	8	6	4	4	3	3	2	1	0	6.4%
Mistral Large	14	7	7	6	6	6	5	5	5	4	6.4%
DeepSeek V3.2	13	10	7	7	7	5	4	3	2	2	5.9%
GPT-4o, May 13th (temp=0)	10	9	8	8	5	4	4	3	2	2	5.7%
Mistral Medium 3.1	7	7	6	6	6	6	6	5	5	3	5.6%
Mistral Small 3.2 24B	8	7	6	6	6	5	5	5	4	2	5.5%
Grok 4.20	17	6	5	5	5	4	4	3	3	3	5.4%
GPT-4o Mini (temp=0)	7	7	6	6	6	6	6	5	3	1	5.2%
Qwen3 235B A22B Instruct 2507	17	5	5	5	4	4	4	3	3	2	5.1%
DeepSeek V4 Flash	17	6	5	4	3	3	2	2	1	1	4.4%
Mistral Small Creative	5	4	4	3	3	3	2	2	2	2	2.9%
Arcee AI: Trinity Large (Preview)	13	3	2	2	2	1	1	1	1	0	2.5%
Gemma 3 4B	3	3	3	2	2	2	2	2	2	2	2.3%
Ministral 3 14B	3	3	3	3	2	1	1	1	1	0	1.8%

▼

Long text (~1594 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-OSS 120B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	100	100	50	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	50	95.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	14	91.4%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	9	90.9%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	7	90.7%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	3	90.3%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	50	50	90.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 3.1 Flash Lite	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	50	33	88.3%
Grok 4	100	100	100	100	100	100	100	100	50	25	87.5%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	100	50	50	33	83.3%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	50	50	50	33	78.3%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	33	33	13	77.9%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	50	50	33	25	75.8%
Qwen 3 32B	100	100	100	100	100	100	50	50	33	25	75.8%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	17	17	13	74.6%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	13	11	11	73.5%
Z.AI GLM 4.7 Flash	100	100	100	100	100	50	50	50	50	33	73.3%
Qwen 3.5 27B	100	100	100	100	100	50	50	50	33	25	70.8%
Gemini 3.1 Pro (Preview)	100	100	100	100	50	50	50	50	50	50	70.0%
Hermes 3 70B	100	100	100	100	100	100	33	17	14	13	67.7%
GPT-5.4 Nano	100	100	100	100	100	100	33	17	14	11	67.5%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	33	25	25	63.3%
Z.AI GLM 4.6	100	100	100	100	50	50	50	50	14	8	62.2%
Qwen 3.5 122B	100	100	100	50	50	50	50	50	33	33	61.7%
Gemma 4 26B	100	100	100	50	50	50	50	50	33	33	61.7%
Z.AI GLM 4.5	100	100	100	100	100	50	14	10	8	5	58.8%
Arcee AI: Trinity Mini	100	100	100	100	50	33	33	33	25	11	58.6%
Ministral 8B	100	100	100	100	100	50	4	4	2	2	56.2%
Qwen 3.5 35B	100	100	100	50	50	33	33	33	25	25	55.0%
Gemini 3 Pro (Preview)	100	50	50	50	50	50	50	50	50	33	53.3%
Gemini 3 Flash (Preview, Reasoning)	100	50	50	50	50	50	50	50	50	33	53.3%
GPT-5.4 Mini	100	50	50	50	50	50	50	50	33	25	50.8%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	33	48.3%
ByteDance Seed 2.0 Lite	50	50	50	50	50	50	50	33	33	33	45.0%
Grok 4.20	100	100	100	100	13	13	8	4	4	3	44.4%
Qwen 3.5 Flash	100	50	50	50	33	33	33	33	25	25	43.3%
GPT-5.4	50	50	50	50	50	50	50	33	25	14	42.3%
DeepSeek V4 Pro	50	50	50	50	50	33	33	33	7	6	36.3%
GPT-4o, Aug. 6th (temp=1)	100	33	33	33	33	33	25	17	14	14	33.7%
Hermes 3 405B	100	50	33	25	25	25	20	20	20	17	33.5%
Claude 3.5 Sonnet	50	33	33	33	33	33	33	25	25	25	32.5%
Claude Sonnet 4	50	33	33	33	33	33	33	25	25	20	32.0%
Claude Opus 4	50	50	33	33	33	25	25	25	25	20	32.0%
GPT-5.5	50	50	50	33	33	25	20	20	20	17	31.8%
WizardLM 2 8x22b	100	50	50	50	17	13	11	10	10	8	31.8%
DeepSeek-V2 Chat	50	50	33	33	25	25	20	20	20	11	28.8%
Qwen 2.5 72B	100	50	33	25	20	13	13	13	13	8	28.6%
DeepSeek V3.1	100	50	25	25	17	17	14	11	10	8	27.6%
Cohere Command R+ (Aug. 2024)	100	33	33	20	20	20	17	14	11	2	27.1%
DeepSeek V3 (2024-12-26)	50	33	25	25	25	25	25	25	17	11	26.1%
Mistral NeMO	100	100	11	8	8	7	7	7	7	2	25.6%
Claude Opus 4.5	33	33	33	25	25	25	20	20	20	20	25.5%
GPT-4o Mini (temp=1)	100	25	25	25	17	14	14	13	13	8	25.4%
Claude 3.7 Sonnet	33	33	25	25	25	25	25	20	20	20	25.2%
Claude Opus 4.7	25	25	25	25	25	25	25	25	20	20	24.0%
Rocinante 12B	100	33	20	17	17	14	13	11	8	6	23.9%
Grok 4.3	100	33	25	14	14	11	11	8	8	7	23.2%
Llama 3.1 70B	50	33	33	20	20	20	20	13	9	6	22.4%
Llama 3.1 Nemotron 70B	100	25	20	14	13	13	10	10	7	6	21.7%
DeepSeek V4 Flash	50	50	50	11	10	8	7	7	7	6	20.6%
Gemini 3 Flash (Preview)	25	25	25	20	20	20	17	17	17	17	20.2%
DeepSeek V3 (2025-03-24)	33	20	20	20	20	20	20	17	17	14	20.1%
GPT-4o, May 13th (temp=1)	50	33	20	17	17	14	14	14	11	8	19.9%
Claude Opus 4.6	33	33	25	20	14	14	14	13	11	11	18.9%
Gemini 2.5 Flash	25	25	25	20	20	20	17	14	13	10	18.8%
Mistral Small 4	100	14	14	13	10	10	9	8	6	3	18.7%
Gemma 3 12B	33	17	17	17	17	17	14	14	14	14	17.4%
DeepSeek V3.2	33	33	33	25	9	9	9	8	8	5	17.4%
Claude Sonnet 4.5	20	20	20	17	17	17	17	17	14	14	17.2%
Qwen 3.5 Plus (2026-02-15)	50	20	20	17	13	13	13	13	10	4	17.1%
Gemma 3 27B	20	20	20	20	17	17	17	17	14	10	17.1%
GPT-4o, Aug. 6th (temp=0)	20	20	17	17	17	17	14	14	13	13	16.0%
GPT-4o Mini (temp=0)	17	17	17	17	17	17	17	17	13	13	15.8%
Claude 3 Haiku	20	20	17	17	17	14	14	14	13	11	15.6%
Ministral 3B	33	25	13	13	11	11	10	8	8	6	13.8%
Writer: Palmyra X5	33	20	14	14	13	10	9	9	8	4	13.5%
GPT-4.1 Mini	20	20	20	17	13	9	9	9	8	6	13.0%
Qwen3 235B A22B Instruct 2507	33	20	14	11	10	9	8	8	7	5	12.6%
Mistral Large 3	11	11	11	11	11	10	10	10	7	7	9.9%
Ministral 3 3B	25	14	10	9	8	8	8	7	4	1	9.5%
Llama 3.1 8B	14	11	11	10	10	8	8	6	6	5	9.1%
GPT-4o, May 13th (temp=0)	17	17	14	14	13	6	3	3	1	1	8.8%
Mistral Large	11	11	9	9	8	8	8	8	6	6	8.5%
Mistral Large 2	11	9	9	9	8	7	7	7	6	5	7.9%
Arcee AI: Trinity Large (Preview)	10	10	10	8	8	7	6	6	6	5	7.4%
Gemma 3 4B	7	6	6	6	6	5	5	5	4	4	5.4%
Mistral Small Creative	8	6	6	5	4	4	4	3	2	2	4.4%
Ministral 3 14B	6	6	6	5	5	4	3	3	3	3	4.3%

▼

Short text (~524 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	50	95.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	100	50	95.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	100	50	95.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	33	93.3%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	33	93.3%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	33	93.3%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	8	90.8%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	3	90.3%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	50	50	90.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	50	50	90.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	50	50	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	50	33	88.3%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	50	9	85.9%
Z.AI GLM 5	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-5 Nano	100	100	100	100	100	100	100	50	50	50	85.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	100	10	9	81.9%
GPT-5.2	100	100	100	100	100	100	100	50	33	20	80.3%
Hermes 3 405B	100	100	100	100	100	100	100	50	33	20	80.3%
MiniMax M2.5	100	100	100	100	100	100	50	50	50	50	80.0%
GPT-OSS 120B	100	100	100	100	100	100	50	50	50	50	80.0%
Stealth: Healer Alpha	100	100	100	100	100	100	50	50	50	50	80.0%
Claude Opus 4.7	100	100	100	100	100	50	50	50	50	50	75.0%
Claude Sonnet 4	100	100	100	100	100	100	50	33	33	33	75.0%
GPT-5.4 Nano	100	100	100	100	100	100	50	50	11	6	71.7%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	50	25	20	10	70.5%
MoonshotAI: Kimi K2.6	100	100	100	100	50	50	50	50	50	50	70.0%
GPT-5	100	100	100	100	50	50	50	50	50	50	70.0%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	50	50	70.0%
MoonshotAI: Kimi K2.5	100	100	100	100	50	50	50	50	50	50	70.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	50	50	50	50	50	50	70.0%
Z.AI GLM 4.7	100	100	100	100	50	50	50	50	50	50	70.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	50	50	50	50	50	33	68.3%
Grok 4	100	100	100	100	100	50	33	33	33	33	68.3%
GPT-5.5 (Reasoning)	100	100	100	50	50	50	50	50	50	50	65.0%
Claude 3.5 Sonnet	100	100	100	100	50	50	50	33	33	25	64.2%
Stealth: Hunter Alpha	100	100	100	50	50	50	50	50	50	33	63.3%
Grok 4 Fast	100	100	100	50	50	50	50	50	50	33	63.3%
Ministral 8B	100	100	100	100	100	100	6	5	4	1	61.6%
Hermes 3 70B	100	100	100	100	100	33	25	20	17	14	60.9%
Qwen 3.5 122B	100	100	100	50	50	50	50	33	33	33	60.0%
Qwen 3.5 27B	100	100	100	50	50	50	50	33	33	33	60.0%
Z.AI GLM 4.6	100	100	100	100	100	50	25	8	5	0	58.9%
Qwen 3.5 35B	100	100	50	50	50	50	50	50	50	33	58.3%
Claude Opus 4.5	100	50	50	50	50	50	50	50	50	50	55.0%
Qwen 3.5 Flash	100	50	50	50	50	50	50	50	50	50	55.0%
ByteDance Seed 2.0 Lite	100	50	50	50	50	50	50	50	50	50	55.0%
GPT-5.4 Mini	100	50	50	50	50	50	50	50	50	50	55.0%
Claude Haiku 4.5	100	50	50	50	50	50	50	50	50	33	53.3%
GPT-4o, Aug. 6th (temp=1)	100	100	100	50	50	33	25	25	20	17	52.0%
WizardLM 2 8x22b	100	50	50	50	50	50	50	50	33	33	51.7%
GPT-5.5 (Reasoning, Low)	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Opus 4	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-5.4 (Reasoning)	100	50	50	50	50	50	50	33	33	33	50.0%
Z.AI GLM 4.5	100	100	100	50	50	50	25	8	7	5	49.4%
Claude Sonnet 4.5	100	50	50	50	50	50	33	33	33	33	48.3%
Writer: Palmyra X5	100	100	50	50	50	33	33	33	25	7	48.2%
DeepSeek V4 Pro	100	100	50	50	50	50	33	20	13	8	47.4%
Claude 3.7 Sonnet	100	100	33	33	33	33	33	33	33	33	46.7%
GPT-4.1 Nano	100	100	100	100	50	3	2	2	2	2	46.0%
GPT-5.4 (Reasoning, Low)	100	50	50	50	33	33	33	33	33	33	45.0%
Qwen3 235B A22B Instruct 2507	100	50	50	50	50	33	33	33	25	25	45.0%
DeepSeek V3 (2024-12-26)	50	50	50	50	50	50	50	25	14	11	40.0%
Grok 4.3	100	100	33	33	25	25	20	14	7	6	36.4%
GPT-4o, Aug. 6th (temp=0)	50	50	33	33	33	33	33	25	25	17	33.3%
Rocinante 12B	100	50	50	33	25	20	20	17	14	1	33.0%
GPT-5.4	33	33	33	33	33	33	33	33	33	25	32.5%
Gemini 3 Flash (Preview)	50	33	33	33	33	33	33	25	25	25	32.5%
DeepSeek V3.2	50	50	50	33	33	33	17	17	17	9	30.9%
Gemini 3.1 Flash Lite (Preview)	33	33	33	33	33	33	33	25	25	20	30.3%
Gemini 3.1 Flash Lite	33	33	33	33	33	33	33	25	25	20	30.3%
Gemini 3.1 Flash Lite (Reasoning)	33	33	33	33	33	33	25	25	25	14	28.9%
DeepSeek V3 (2025-03-24)	50	50	33	25	25	25	25	20	14	14	28.2%
DeepSeek V3.1	50	50	33	33	33	25	25	13	9	9	28.1%
Gemini 2.5 Flash	50	33	33	25	25	25	20	20	17	7	25.5%
Grok 4.20	50	33	33	33	33	25	20	17	6	4	25.5%
Qwen 2.5 72B	100	33	25	25	20	11	11	10	8	8	25.2%
GPT-4o, May 13th (temp=1)	33	33	33	25	25	20	20	20	20	11	24.1%
DeepSeek-V2 Chat	50	33	25	25	20	20	17	13	13	10	22.5%
Llama 3.1 70B	100	20	17	14	13	13	11	11	10	7	21.5%
Claude Sonnet 4.6	33	25	25	25	20	20	17	17	17	14	21.3%
Grok 4.20 (Beta)	100	25	20	14	10	10	10	10	6	6	21.1%
Mistral NeMO	100	33	13	10	10	10	7	6	4	3	19.5%
GPT-4o Mini (temp=1)	25	25	20	20	17	17	17	11	9	7	16.7%
Qwen 3.5 Plus (2026-02-15)	33	33	20	20	17	17	10	8	7	2	16.7%
Claude 3 Haiku	20	17	17	14	14	14	14	14	11	9	14.5%
DeepSeek V4 Flash	50	20	17	9	9	9	7	6	6	5	13.8%
Mistral Medium 3.1	25	17	17	11	11	10	10	9	9	9	12.8%
GPT-4.1 Mini	17	14	14	13	13	13	11	10	10	9	12.3%
Mistral Small 3.2 24B	14	14	14	13	13	13	11	11	10	8	12.1%
GPT-4o, May 13th (temp=0)	25	25	20	20	20	4	2	1	1	1	12.0%
GPT-4o Mini (temp=0)	14	14	14	14	14	13	13	11	10	2	11.9%
Mistral Large 3	14	14	14	14	10	10	10	9	8	8	11.2%
Mistral Large	14	14	14	14	11	11	8	8	8	8	11.1%
Gemini 2.5 Flash Lite	20	20	10	10	10	8	8	8	7	7	10.9%
Gemma 3 12B	14	14	13	11	11	11	10	10	8	4	10.6%
Gemma 3 27B	11	11	11	11	10	10	10	10	10	9	10.4%
Mistral Small Creative	33	9	8	8	7	7	7	7	7	6	10.0%
Llama 3.1 Nemotron 70B	17	11	11	10	10	9	9	8	7	7	9.9%
Mistral Large 2	17	17	11	9	8	8	8	7	7	7	9.8%
Ministral 3 3B	14	14	14	11	9	8	7	7	6	6	9.6%
Mistral Small 4	13	13	11	10	9	9	8	8	7	7	9.4%
Ministral 3B	14	11	10	10	9	8	7	7	7	5	8.8%
Llama 3.1 8B	17	14	9	8	7	6	5	4	3	2	7.4%
Gemma 3 4B	10	9	9	6	6	5	3	3	2	2	5.4%
Arcee AI: Trinity Large (Preview)	8	6	6	6	6	5	5	5	4	3	5.4%
Ministral 3 14B	6	6	6	5	5	5	4	3	3	1	4.3%

▼

Short text (~524 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	50	95.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	100	50	95.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	100	33	93.3%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	5	90.5%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-OSS 120B	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	50	50	90.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	50	33	88.3%
Aion 2.0	100	100	100	100	100	100	100	50	50	50	85.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
Qwen 3 32B	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	50	50	50	85.0%
Grok 4 Fast	100	100	100	100	100	100	100	50	50	33	83.3%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	50	50	33	83.3%
GPT-5.2	100	100	100	100	100	100	100	50	50	25	82.5%
Qwen 3.6 27B	100	100	100	100	100	100	100	50	50	25	82.5%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	11	0	81.1%
Z.AI GLM 4.5	100	100	100	100	100	100	100	50	50	7	80.7%
Qwen 3.5 122B	100	100	100	100	100	100	50	50	50	50	80.0%
Gemini 2.5 Pro	100	100	100	100	100	100	50	50	50	50	80.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	50	50	50	50	80.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	50	50	50	50	50	75.0%
Xiaomi MIMO v2.5	100	100	100	100	100	50	50	50	50	50	75.0%
Qwen 3.5 397B A17B	100	100	100	100	100	50	50	50	50	33	73.3%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	50	50	50	50	33	73.3%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	50	50	50	33	33	71.7%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	50	20	17	17	70.3%
Stealth: Healer Alpha	100	100	100	100	50	50	50	50	50	50	70.0%
Claude Sonnet 4	100	100	100	100	50	50	50	50	50	33	68.3%
Grok 4	100	100	100	100	100	50	33	33	33	25	67.5%
GPT-5.5 (Reasoning, Low)	100	100	100	50	50	50	50	50	50	50	65.0%
Z.AI GLM 4.7	100	100	100	50	50	50	50	50	50	50	65.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	14	13	8	7	64.2%
GPT-5	100	100	50	50	50	50	50	50	50	50	60.0%
Stealth: Hunter Alpha	100	100	50	50	50	50	50	50	50	50	60.0%
Claude Haiku 4.5	100	100	50	50	50	50	50	50	50	50	60.0%
DeepSeek-V2 Chat	100	100	50	50	50	50	50	50	50	33	58.3%
GPT-5.4 Mini	100	100	50	50	50	50	50	50	50	33	58.3%
Llama 3.1 70B	100	100	100	100	100	33	20	9	9	8	57.9%
Hermes 3 405B	100	100	100	100	33	33	33	33	25	17	57.5%
Grok 4.20 (Beta)	100	100	100	50	50	50	33	33	33	20	57.0%
Qwen 2.5 72B	100	100	100	100	50	33	33	17	14	8	55.6%
GPT-5.5 (Reasoning)	100	50	50	50	50	50	50	50	50	50	55.0%
DeepSeek V4 Pro (Reasoning)	100	50	50	50	50	50	50	50	50	50	55.0%
Claude Sonnet 4.5	100	100	50	50	50	50	50	33	33	33	55.0%
Qwen 3.5 35B	100	50	50	50	50	50	50	50	50	50	55.0%
GPT-5.4	100	50	50	50	50	50	50	50	50	33	53.3%
GPT-4.1 Nano	100	100	100	100	100	13	8	6	4	1	53.2%
Grok 4.20	100	50	50	50	50	50	50	50	50	25	52.5%
Claude 3.5 Sonnet	100	100	100	50	33	33	25	25	25	20	51.2%
MoonshotAI: Kimi K2.5	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Opus 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
Qwen 3.5 Flash	50	50	50	50	50	50	50	50	50	50	50.0%
ByteDance Seed 2.0 Lite	50	50	50	50	50	50	50	50	50	50	50.0%
DeepSeek V3 (2024-12-26)	100	50	50	50	50	50	50	50	33	7	49.0%
GPT-5.4 (Reasoning)	100	50	50	50	50	50	33	33	33	33	48.3%
GPT-4.1 Mini	100	100	100	100	20	17	14	11	11	9	48.2%
GPT-4.1	100	100	50	33	33	33	33	33	33	25	47.5%
Hermes 3 70B	100	100	100	50	33	25	25	17	14	9	47.3%
DeepSeek V3 (2025-03-24)	50	50	50	50	50	50	50	33	33	33	45.0%
Gemini 2.5 Flash	50	50	50	50	50	50	50	33	33	33	45.0%
GPT-4o, Aug. 6th (temp=0)	50	50	50	50	50	50	50	33	33	25	44.2%
Rocinante 12B	100	100	100	50	20	20	20	11	10	8	43.9%
Grok 4.3	100	100	33	33	33	33	25	25	25	11	41.9%
Gemini 3.1 Flash Lite (Reasoning)	50	50	50	50	50	33	33	25	25	25	39.2%
Claude Opus 4	50	50	50	33	33	33	33	33	25	25	36.7%
Qwen3 235B A22B Instruct 2507	50	50	50	33	33	33	33	33	33	9	35.9%
Writer: Palmyra X5	50	50	50	50	50	33	33	25	8	6	35.5%
WizardLM 2 8x22b	50	50	50	50	50	33	25	17	17	11	35.3%
Gemini 3.1 Flash Lite	50	50	50	50	33	25	25	25	20	20	34.8%
Gemini 3.1 Flash Lite (Preview)	50	50	33	33	33	33	25	25	25	25	33.3%
Gemini 3 Flash (Preview)	50	33	33	33	33	33	33	33	25	25	33.3%
DeepSeek V4 Pro	50	50	50	50	33	25	25	20	13	10	32.6%
GPT-4o, Aug. 6th (temp=1)	100	33	33	25	25	25	20	20	20	20	32.2%
Claude 3.7 Sonnet	50	33	33	33	33	25	25	25	25	25	30.8%
DeepSeek V3.1	50	50	50	50	33	20	17	14	11	7	30.3%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	20	20	20	17	14	9	30.0%
GPT-4o, May 13th (temp=0)	33	33	33	33	33	25	25	25	25	20	28.7%
GPT-4o, May 13th (temp=1)	33	33	33	33	33	25	25	20	17	11	26.4%
DeepSeek V3.2	50	50	50	33	33	14	8	8	7	6	26.0%
GPT-4o Mini (temp=1)	100	25	25	20	17	17	14	14	14	13	25.9%
Mistral Medium 3.1	50	33	33	33	33	20	14	13	10	9	24.9%
Claude Sonnet 4.6	33	25	25	25	25	20	20	20	20	17	23.0%
DeepSeek V4 Flash	50	50	20	17	14	14	13	10	7	6	20.0%
Gemini 2.5 Flash Lite	33	33	25	20	20	20	11	11	8	7	18.9%
Claude 3 Haiku	25	25	25	20	20	17	17	14	14	11	18.8%
GPT-4o Mini (temp=0)	20	20	20	20	20	20	17	17	17	13	18.3%
Arcee AI: Trinity Large (Preview)	100	20	11	8	8	8	7	6	6	0	17.4%
Gemma 3 27B	25	20	20	17	17	17	14	13	13	11	16.5%
Ministral 3 14B	50	50	10	9	9	8	8	8	7	6	16.5%
Mistral Large 2	20	20	20	17	17	14	13	13	13	9	15.4%
Mistral Large	20	20	17	17	14	14	13	11	11	9	14.6%
Mistral Large 3	14	14	14	13	13	13	13	13	13	9	12.7%
Ministral 3B	50	17	10	9	8	7	7	7	6	4	12.4%
Mistral Small 3.2 24B	20	20	14	11	11	11	10	8	8	8	12.1%
Gemma 3 4B	14	14	14	13	11	11	11	11	11	8	11.9%
Mistral NeMO	17	14	14	14	13	11	10	10	8	6	11.7%
Mistral Small 4	17	14	11	11	10	9	9	8	7	6	10.3%
Gemma 3 12B	14	11	11	10	10	10	10	10	8	7	10.2%
Llama 3.1 8B	14	11	10	10	10	9	8	8	7	5	9.2%
Mistral Small Creative	14	11	9	9	8	8	8	7	6	5	8.5%
Ministral 3 3B	13	11	8	8	8	8	8	6	5	4	7.9%
Ministral 8B	6	5	5	5	5	4	4	4	4	1	4.4%
Ministral 3 8B	6	6	5	4	4	4	3	1	0	0	3.3%

detailed entries

▼

Long text (~1594 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	50	95.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	33	93.3%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	33	93.3%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	20	92.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	5	90.5%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	50	50	90.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	50	50	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	50	50	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	50	33	88.3%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	50	33	88.3%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	50	25	87.5%
MiniMax M2.5	100	100	100	100	100	100	100	100	50	4	85.4%
GPT-5.4 Nano	100	100	100	100	100	100	100	50	50	50	85.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	50	50	33	83.3%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	50	33	33	81.7%
Grok 4	100	100	100	100	100	100	100	50	33	20	80.3%
Z.AI GLM 4.7	100	100	100	100	100	100	50	50	50	25	77.5%
Qwen 3.5 35B	100	100	100	100	100	100	50	50	33	33	76.7%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	50	50	33	25	75.8%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	33	33	33	25	72.5%
Mistral Small 4 (Reasoning)	100	100	100	100	100	50	50	50	33	33	71.7%
Z.AI GLM 4.5	100	100	100	100	100	100	50	33	11	8	70.3%
GPT-OSS 120B	100	100	100	100	50	50	50	50	50	50	70.0%
Z.AI GLM 4.6	100	100	100	100	100	50	50	50	33	1	68.4%
Qwen 3 32B	100	100	100	100	100	50	50	33	25	8	66.7%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	33	25	65.8%
Gemini 3 Pro (Preview)	100	100	100	100	50	50	50	33	25	25	63.3%
Qwen 2.5 72B	100	100	100	100	100	50	25	17	13	5	60.9%
GPT-5.5 (Reasoning)	100	100	100	100	33	33	33	33	33	33	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	33	25	25	6	2	59.2%
GPT-5.4 Mini	100	100	50	50	50	50	50	50	33	25	55.8%
Qwen 3.5 9B	100	100	100	50	33	33	33	33	25	6	51.5%
Z.AI GLM 4.5 Air	100	100	100	100	33	33	20	17	7	2	51.2%
Gemma 4 31B	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Sonnet 4.6	100	100	50	50	50	50	50	20	14	13	49.7%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	33	48.3%
GPT-5.4	50	50	50	50	50	50	50	50	50	25	47.5%
GPT-4.1 Nano	100	100	100	100	50	7	3	2	2	1	46.5%
Z.AI GLM 4.7 Flash	100	50	50	50	50	33	33	33	33	25	45.8%
GPT-4o, Aug. 6th (temp=1)	100	100	50	50	33	33	33	20	20	17	45.7%
Claude Sonnet 4.5	100	100	33	33	33	33	25	25	25	17	42.5%
GPT-5.5	100	50	50	33	33	33	33	25	25	20	40.3%
Gemini 2.5 Flash	50	50	50	50	33	33	33	33	33	33	40.0%
DeepSeek V4 Pro	50	50	50	50	50	50	25	25	2	1	35.4%
Qwen 3.5 27B	50	50	50	33	33	33	25	25	20	20	34.0%
Rocinante 12B	100	100	50	14	13	13	13	11	10	8	33.1%
Mistral Small 3.2 24B	100	100	100	4	4	4	3	3	2	0	31.9%
Gemini 2.5 Flash Lite	100	100	100	7	5	3	2	1	1	0	31.8%
WizardLM 2 8x22b	100	100	33	25	20	14	7	6	6	3	31.4%
Arcee AI: Trinity Large (Preview)	100	100	100	2	2	2	2	1	1	1	31.2%
Grok 4.3	100	100	33	20	17	11	10	4	3	2	30.0%
Gemini 2.5 Pro	50	33	33	33	33	25	25	25	20	20	29.8%
Claude Opus 4.5	50	50	33	33	33	20	20	17	17	17	29.0%
Gemma 4 26B	33	33	33	33	33	33	25	25	17	17	28.3%
GPT-4o, May 13th (temp=1)	50	50	33	33	25	20	17	17	14	13	27.2%
Claude 3.7 Sonnet	100	25	20	20	20	20	20	17	14	14	27.0%
Gemini 3.1 Flash Lite (Reasoning)	33	33	33	25	25	25	25	25	25	20	27.0%
Gemini 3.1 Flash Lite	33	33	33	33	25	25	25	20	20	20	26.8%
Claude Opus 4	50	33	25	25	25	25	20	20	20	14	25.8%
Claude Sonnet 4	33	25	25	25	25	25	25	25	25	20	25.3%
Hermes 3 405B	33	33	33	25	25	25	20	20	14	13	24.2%
DeepSeek-V2 Chat	50	50	50	20	14	14	11	10	8	1	22.9%
Hermes 3 70B	100	20	17	17	14	11	10	10	9	8	21.6%
Qwen3 235B A22B Instruct 2507	100	25	20	20	17	9	7	7	6	3	21.3%
Gemini 3 Flash (Preview)	25	25	25	25	25	20	20	20	11	10	20.6%
Claude Opus 4.7	25	25	25	20	20	20	17	17	17	13	19.8%
Claude 3 Haiku	33	25	25	25	25	17	13	13	11	11	19.7%
Gemini 3.1 Flash Lite (Preview)	25	20	20	20	20	20	20	17	14	7	18.3%
Grok 4.20 (Beta)	100	33	17	10	4	4	3	3	3	2	17.9%
GPT-4o, Aug. 6th (temp=0)	33	33	33	20	14	13	11	9	6	4	17.7%
DeepSeek V3 (2024-12-26)	50	33	20	14	13	13	10	10	8	7	17.7%
DeepSeek V4 Flash	50	33	25	20	20	8	8	4	3	2	17.3%
GPT-4o Mini (temp=1)	20	20	20	20	17	17	17	14	13	7	16.4%
Claude 3.5 Sonnet	20	17	17	17	17	17	14	14	14	14	16.0%
DeepSeek V3.2	50	50	20	14	6	3	3	2	1	1	15.0%
Mistral Medium 3.1	33	25	17	17	13	13	9	8	8	3	14.4%
Llama 3.1 70B	33	20	20	17	13	10	10	9	6	6	14.3%
Qwen 3.5 Plus (2026-02-15)	25	25	20	20	17	14	9	6	3	2	14.2%
Writer: Palmyra X5	33	33	25	11	10	7	5	4	3	2	13.4%
Mistral Small 4	33	20	20	14	13	11	5	4	4	3	12.7%
GPT-4o Mini (temp=0)	17	17	14	14	14	11	11	9	8	8	12.3%
Gemma 3 12B	17	17	17	13	11	10	8	8	8	3	11.0%
Llama 3.1 Nemotron 70B	17	13	13	11	11	11	11	8	6	6	10.6%
LFM2 24B	50	25	11	4	3	3	2	1	0	0	9.8%
DeepSeek V3 (2025-03-24)	17	17	13	10	10	9	9	9	3	1	9.7%
GPT-4.1 Mini	50	9	6	6	6	5	5	3	3	3	9.6%
Llama 3.1 8B	17	17	17	13	13	7	5	4	4	1	9.6%
Mistral Large 3	17	11	11	10	8	7	6	6	4	3	8.3%
Mistral Large 2	17	11	10	8	8	7	7	6	5	3	8.2%
Mistral Large	11	10	9	9	8	7	5	4	4	3	7.0%
Gemma 3 27B	14	8	8	7	6	5	5	5	4	4	6.6%
Mistral NeMO	20	14	10	8	7	2	2	2	0	0	6.5%
GPT-4o, May 13th (temp=0)	9	9	7	7	7	6	6	5	5	4	6.5%
Mistral Small Creative	50	2	2	2	2	2	1	1	1	1	6.5%
Grok 4.20	8	8	6	6	6	5	3	3	3	3	5.0%
DeepSeek V3.1	7	6	5	4	4	3	2	2	1	1	3.4%
Ministral 3B	5	5	4	3	2	2	1	1	1	0	2.3%
Gemma 3 4B	4	3	2	2	2	2	2	2	1	1	2.1%
Ministral 3 3B	3	3	2	1	1	1	0	0	0	0	1.1%

▼

Long text (~1594 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
LFM2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	50	95.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	50	95.0%
o4 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-OSS 120B	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	50	50	90.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	100	50	50	90.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	50	50	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	50	33	88.3%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	50	17	86.7%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	50	50	50	85.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	50	50	50	85.0%
MiniMax M2.7	100	100	100	100	100	100	100	50	50	50	85.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	50	50	50	85.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	50	50	50	85.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	50	50	6	80.6%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	50	50	50	50	80.0%
Qwen 3.5 27B	100	100	100	100	100	100	50	50	50	50	80.0%
MiniMax M2.5	100	100	100	100	100	100	50	50	50	50	80.0%
Qwen 3.5 35B	100	100	100	100	100	100	50	50	50	50	80.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	50	50	50	50	80.0%
Z.AI GLM 4.7	100	100	100	100	100	50	50	50	50	50	75.0%
Grok 4 Fast	100	100	100	100	100	50	50	50	50	50	75.0%
Z.AI GLM 4.6	100	100	100	100	100	100	50	50	33	0	73.3%
GPT-5.2	100	100	100	100	100	50	50	50	50	33	73.3%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	33	25	25	20	70.3%
Gemma 4 31B (Reasoning)	100	100	100	100	50	50	50	50	50	50	70.0%
Aion 2.0	100	100	100	100	50	50	50	50	50	50	70.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	17	17	1	68.4%
ByteDance Seed 1.6	100	100	100	50	50	50	50	50	50	50	65.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	33	33	25	20	20	63.2%
Gemini 2.5 Pro	100	100	50	50	50	50	50	50	50	50	60.0%
Grok 4	100	100	100	50	50	50	50	33	33	33	60.0%
Gemini 2.5 Flash (Reasoning)	100	100	50	50	50	50	50	50	50	50	60.0%
Qwen3.6 Max Preview	100	100	100	50	50	50	50	33	33	25	59.2%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	50	50	50	50	33	33	25	59.2%
Qwen 3.5 122B	100	100	100	50	50	50	50	33	25	25	58.3%
MoonshotAI: Kimi K2.6	100	50	50	50	50	50	50	50	50	50	55.0%
Gemma 4 26B	100	100	100	100	33	33	25	25	14	14	54.5%
MoonshotAI: Kimi K2.5	100	50	50	50	50	50	50	50	50	33	53.3%
Hermes 3 70B	100	100	100	50	50	50	25	17	17	14	52.3%
Mistral Medium 3.1	100	100	100	100	33	17	17	17	11	9	50.4%
Qwen 3.5 397B A17B	50	50	50	50	50	50	50	50	50	50	50.0%
Gemma 4 31B	50	50	50	50	50	50	50	50	50	50	50.0%
GPT-5.4 Mini	50	50	50	50	50	50	50	50	33	33	46.7%
Arcee AI: Trinity Large (Preview)	100	100	100	100	20	17	8	7	7	7	46.5%
Arcee AI: Trinity Mini	100	100	50	50	50	33	25	20	4	0	43.2%
Claude Haiku 4.5	50	50	50	50	50	50	50	33	25	20	42.8%
GPT-5.5	100	50	50	50	33	33	33	25	25	25	42.5%
GPT-5.4 Nano	100	50	50	50	50	50	33	20	11	8	42.2%
Hermes 3 405B	100	100	33	25	25	25	25	20	20	20	39.3%
Llama 3.1 70B	100	100	50	33	25	20	17	13	13	13	38.3%
Claude Opus 4.6	50	50	50	50	33	33	33	25	25	20	37.0%
Claude Sonnet 4	100	50	33	33	25	25	25	25	25	20	36.2%
ByteDance Seed 2.0 Lite	50	33	33	33	33	33	33	33	33	33	35.0%
Qwen 2.5 72B	100	100	25	20	20	20	17	14	13	11	34.0%
Llama 3.1 Nemotron 70B	100	100	33	20	20	14	13	11	11	5	32.7%
Ministral 8B	100	100	100	4	3	3	3	3	2	1	31.9%
Claude 3.7 Sonnet	50	33	33	33	33	25	25	25	25	25	30.8%
DeepSeek V4 Pro	50	50	50	50	33	20	20	17	8	6	30.4%
DeepSeek V3 (2024-12-26)	50	50	50	33	25	25	17	17	17	17	30.0%
Gemini 3.1 Flash Lite	33	33	33	33	33	33	33	25	20	17	29.5%
Claude Opus 4	50	33	33	33	25	25	25	25	25	14	28.9%
Claude Opus 4.7	33	33	33	33	33	25	25	25	25	20	28.7%
Gemini 2.5 Flash Lite	100	100	17	14	14	10	9	8	7	2	28.1%
Claude Opus 4.5	33	33	33	33	25	25	25	25	25	20	27.8%
Gemini 3.1 Flash Lite (Reasoning)	33	33	33	25	25	25	25	25	25	20	27.0%
Claude 3.5 Sonnet	33	33	33	25	25	25	25	25	25	20	27.0%
Grok 4.20 (Beta)	100	100	9	9	9	9	8	8	7	7	26.7%
DeepSeek V3 (2025-03-24)	50	33	25	25	25	25	25	20	17	17	26.2%
GPT-5.4	100	33	33	20	17	17	10	7	7	7	25.1%
Qwen 3.5 Plus (2026-02-15)	50	50	50	17	14	11	11	11	9	7	23.0%
Ministral 3 8B	100	100	3	3	3	2	2	2	1	0	21.6%
GPT-4o, May 13th (temp=1)	50	25	25	17	17	17	17	17	17	14	21.4%
Gemini 3.1 Flash Lite (Preview)	25	25	25	25	20	20	20	20	17	17	21.3%
Grok 4.3	100	25	17	13	13	11	9	8	8	8	21.1%
Gemini 3 Flash (Preview)	25	25	25	20	20	20	17	17	17	14	19.9%
DeepSeek V3.1	50	50	20	20	14	14	10	10	8	2	19.8%
GPT-4.1 Mini	50	50	17	17	17	14	11	8	7	3	19.4%
DeepSeek-V2 Chat	25	25	25	20	20	20	17	17	13	11	19.2%
Mistral Small 3.2 24B	100	14	13	11	11	11	9	8	8	5	19.1%
Claude 3 Haiku	25	25	25	25	20	17	17	17	11	10	19.1%
Gemma 3 12B	33	25	25	20	20	14	13	13	13	11	18.6%
Gemini 2.5 Flash	20	20	17	17	17	17	17	17	14	10	16.4%
Llama 3.1 8B

No hallucinated violations

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

basic entries

Long text (~1594 words), big codex (51 entries)

Long text (~1594 words), small codex (11 entries)

Short text (~524 words), big codex (51 entries)

Short text (~524 words), small codex (11 entries)

detailed entries

Long text (~1594 words), big codex (51 detailed entries)

Long text (~1594 words), small codex (11 detailed entries)