Accuracy (recall)

Test: Codex Violation Detection

Avg. Score

73.7%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Gemini 2.5 Flash (Reasoning)	94.0%	$0.0079	12.5s	86%
2	Grok 4.1 Fast	93.8%	$0.0021	21.1s	84%
3	GPT-5.4	95.9%	$0.013	8.8s	85%
4	Z.AI GLM 5 Turbo	94.8%	$0.0072	17.1s	85%
5	Gemini 3 Flash (Preview, Reasoning)	94.7%	$0.011	18.0s	86%
6	DeepSeek V4 Flash (Reasoning)	94.4%	$0.0010	40.3s	82%
7	GPT-5.5	96.7%	$0.030	7.7s	90%
8	Grok 4 Fast	92.4%	$0.0018	12.8s	72%
9	Grok 4.20 (Beta, Reasoning)	96.1%	$0.026	16.8s	89%
10	Gemma 4 31B	92.5%	$0.0009	37.9s	78%
11	Grok 4.20 (Reasoning)	95.8%	$0.015	45.4s	90%
12	GPT-5.2	95.7%	$0.024	24.2s	87%
13	Gemini 3.5 Flash (Reasoning, Minimal)	90.7%	$0.011	3.6s	73%
14	Gemini 3 Flash (Preview)	88.8%	$0.0031	4.5s	69%
15	Claude Opus 4.5	98.0%	$0.041	9.7s	91%
16	Gemini 2.5 Pro	97.4%	$0.035	23.2s	91%
17	GPT-5.4 (Reasoning, Low)	92.4%	$0.020	13.5s	79%
18	Claude Sonnet 4.5	93.6%	$0.024	8.9s	80%
19	Xiaomi MIMO v2.5 Pro	92.5%	$0.0091	37.0s	80%
20	Gemini 2.5 Flash	82.5%	$0.0025	2.8s	69%
21	Claude Opus 4.6	97.3%	$0.040	10.2s	88%
22	Stealth: Healer Alpha	89.0%	$0.0000	21.8s	67%
23	GPT-5.5 (Reasoning, Low)	95.5%	$0.040	14.2s	88%
24	Qwen 3.6 Flash	92.4%	$0.011	29.9s	72%
25	Stealth: Hunter Alpha	89.5%	$0.0000	44.0s	71%
26	Qwen 3.5 Flash	92.8%	$0.0038	1.0m	77%
27	Xiaomi MIMO v2.5	89.2%	$0.0058	21.8s	65%
28	Inception Mercury 2	82.2%	$0.0030	4.6s	62%
29	Claude Sonnet 4	90.3%	$0.023	9.0s	72%
30	Qwen 3.6 35B	92.1%	$0.013	51.0s	78%
31	Qwen 3.5 35B	93.3%	$0.017	54.8s	81%
32	ByteDance Seed 1.6	91.9%	$0.0067	1.0m	77%
33	Z.AI GLM 5	92.1%	$0.013	56.4s	79%
34	Gemini 3.1 Flash Lite	81.1%	$0.0018	3.2s	59%
35	GPT-5 Mini	91.2%	$0.0092	51.7s	74%
36	Gemini 3.5 Flash (Reasoning)	94.8%	$0.045	18.1s	86%
37	GPT-5.4 Mini (Reasoning, Low)	83.0%	$0.0055	6.7s	61%
38	Gemini 3.1 Flash Lite (Preview)	81.9%	$0.0019	2.2s	57%
39	o4 Mini	88.9%	$0.019	28.1s	74%
40	Z.AI GLM 4.7	91.3%	$0.0091	1.0m	77%
41	Gemini 2.5 Flash Lite (Reasoning)	81.1%	$0.0020	17.6s	62%
42	Z.AI GLM 5.1	95.5%	$0.017	1.6m	88%
43	ByteDance Seed 2.0 Lite	89.8%	$0.0067	1.1m	73%
44	Qwen 3.5 Plus (2026-04-20)	93.4%	$0.015	1.4m	83%
45	Claude Sonnet 4.6	82.9%	$0.024	9.9s	70%
46	Gemini 3.1 Flash Lite (Reasoning)	79.3%	$0.0018	3.7s	52%
47	Grok 4.3 (Reasoning)	94.3%	$0.019	1.4m	81%
48	GPT-5.4 Mini (Reasoning)	87.9%	$0.020	28.6s	68%
49	Qwen 3.5 27B	95.6%	$0.021	1.7m	89%
50	DeepSeek V4 Flash	78.1%	$0.0003	9.3s	53%
51	Mistral Large 3	75.5%	$0.0030	10.2s	58%
52	Qwen 3.5 122B	92.7%	$0.026	1.2m	81%
53	Mistral Large	79.3%	$0.012	9.1s	59%
54	MiniMax M2.7	79.1%	$0.0022	29.4s	59%
55	MoonshotAI: Kimi K2.5	92.9%	$0.015	1.5m	79%
56	Qwen 3.5 Plus (2026-02-15)	85.0%	$0.0041	34.2s	55%
57	DeepSeek V4 Pro	81.7%	$0.0031	25.6s	53%
58	GPT-5.1	92.7%	$0.037	52.8s	79%
59	Mistral Large 2	75.9%	$0.012	8.8s	57%
60	Qwen3.7 Max	96.3%	$0.045	1.1m	85%
61	o4 Mini High	90.7%	$0.033	51.3s	76%
62	Grok 4.20	77.0%	$0.0053	6.4s	50%
63	Aion 2.0	91.7%	$0.0084	1.3m	64%
64	Claude Opus 4.7	88.3%	$0.052	7.6s	76%
65	Claude Opus 4.7 (Reasoning)	92.8%	$0.067	11.4s	82%
66	Mistral Medium 3.1	75.1%	$0.0029	7.7s	46%
67	Grok 4.20 (Beta)	74.8%	$0.0059	2.5s	47%
68	Gemini 3.1 Pro (Preview)	98.3%	$0.068	52.3s	93%
69	GPT-5.4 (Reasoning)	91.7%	$0.042	43.1s	75%
70	MiniMax M2.5	74.6%	$0.0020	25.5s	52%
71	Claude Opus 4.6 (Reasoning)	97.3%	$0.076	32.0s	90%
72	GPT-5.5 (Reasoning)	95.6%	$0.071	30.3s	88%
73	DeepSeek V3.2	75.0%	$0.0013	17.2s	45%
74	Gemma 4 31B (Reasoning)	94.4%	$0.0017	2.8m	85%
75	Z.AI GLM 4.5	75.6%	$0.0026	18.5s	45%
76	GPT-4.1	77.7%	$0.0081	10.0s	44%
77	Z.AI GLM 4.6	86.1%	$0.0049	1.3m	59%
78	Mistral Small 4 (Reasoning)	70.9%	$0.0020	16.3s	47%
79	ByteDance Seed 1.6 Flash	67.9%	$0.0009	12.0s	46%
80	Gemma 4 26B (Reasoning)	90.0%	$0.0022	2.0m	65%
81	Gemini 3 Pro (Preview)	91.0%	$0.050	34.0s	70%
82	Z.AI GLM 4.5 Air	77.1%	$0.0025	39.8s	46%
83	DeepSeek-V2 Chat	71.1%	$0.0020	13.1s	41%
84	Grok 4	94.1%	$0.051	1.2m	80%
85	GPT-4.1 Mini	63.7%	$0.0015	5.8s	43%
86	Claude Haiku 4.5	66.8%	$0.0078	5.8s	45%
87	DeepSeek V3 (2024-12-26)	69.9%	$0.0019	15.6s	41%
88	GPT-OSS 120B	81.0%	$0.0020	1.5m	59%
89	Qwen 3.6 27B	89.5%	$0.022	1.4m	63%
90	Qwen3 235B A22B Instruct 2507	67.3%	$0.0007	21.1s	43%
91	Writer: Palmyra X5	67.2%	$0.0062	12.1s	44%
92	Gemma 4 26B	75.0%	$0.0006	25.0s	35%
93	Stealth: Aurora Alpha	77.5%	—	6.2s	46%
94	ByteDance Seed 2.0 Mini	90.8%	$0.0029	2.7m	73%
95	Claude Sonnet 4.6 (Reasoning)	94.9%	$0.076	51.3s	84%
96	Claude 3.7 Sonnet	77.4%	$0.023	10.2s	42%
97	Grok 4.3	64.3%	$0.0061	6.1s	39%
98	Inception Mercury	63.1%	$0.0005	9.5s	34%
99	DeepSeek V3 (2025-03-24)	68.9%	$0.0015	22.9s	34%
100	GPT-4o, Aug. 6th (temp=1)	65.4%	$0.011	3.6s	36%
101	Mistral Small Creative	56.8%	$0.0006	4.3s	36%
102	GPT-4o, Aug. 6th (temp=0)	67.9%	$0.015	5.0s	36%
103	Claude 3.5 Sonnet	79.2%	$0.042	10.5s	47%
104	Qwen3.6 Max Preview	96.4%	$0.050	2.5m	90%
105	GPT-5	93.3%	$0.061	1.6m	81%
106	Z.AI GLM 4.7 Flash	68.2%	$0.0018	1.1m	45%
107	DeepSeek V3.1	66.1%	$0.0014	31.1s	33%
108	Nemotron 3 Super	83.1%	$0.0000	2.3m	55%
109	Qwen 3 32B	64.1%	$0.0010	27.8s	31%
110	Qwen 3.5 9B	84.7%	$0.0020	2.4m	55%
111	Mistral Small 3.2 24B	53.8%	$0.0006	10.4s	32%
112	GPT-5.4 Nano (Reasoning)	67.5%	$0.0035	16.6s	23%
113	GPT-4o, May 13th (temp=0)	71.6%	$0.030	5.4s	36%
114	Gemma 3 27B	53.2%	$0.0005	13.3s	33%
115	Qwen 3.5 397B A17B	93.7%	$0.026	2.9m	75%
116	GPT-5.4 Mini	54.5%	$0.0033	3.1s	29%
117	DeepSeek V4 Pro (Reasoning)	91.5%	$0.014	2.8m	60%
118	Gemini 2.5 Flash Lite	47.9%	$0.0005	2.3s	24%
119	GPT-4o, May 13th (temp=1)	58.1%	$0.026	3.7s	34%
120	Ministral 3 14B	46.4%	$0.0010	6.3s	25%
121	Hermes 3 405B	56.2%	$0.0044	17.2s	20%
122	Llama 3.1 Nemotron 70B	55.8%	$0.0055	18.5s	19%
123	GPT-5 Nano	67.2%	$0.0049	1.9m	42%
124	Ministral 3 8B	39.9%	$0.0007	4.8s	20%
125	Llama 3.1 70B	51.2%	$0.0021	24.3s	18%
126	Qwen 2.5 72B	42.3%	$0.0008	13.9s	18%
127	Ministral 8B	34.7%	$0.0005	5.6s	18%
128	Arcee AI: Trinity Mini	33.3%	$0.0004	10.3s	18%
129	Claude Opus 4	84.4%	$0.116	15.8s	65%
130	Mistral Small 4	30.7%	$0.0008	4.3s	18%
131	Ministral 3 3B	28.6%	$0.0005	3.3s	16%
132	Gemma 3 12B	35.9%	$0.0003	12.0s	11%
133	MoonshotAI: Kimi K2.6	94.7%	$0.038	4.1m	75%
134	GPT-4o Mini (temp=0)	31.3%	$0.0006	25.0s	19%
135	GPT-4o Mini (temp=1)	29.4%	$0.0006	7.4s	13%
136	Ministral 3B	25.0%	$0.0002	2.9s	14%
137	Cohere Command R+ (Aug. 2024)	32.3%	$0.014	10.0s	15%
138	GPT-5.4 Nano (Reasoning, Low)	32.9%	$0.0013	6.1s	2%
139	Hermes 3 70B	28.5%	$0.0013	29.2s	14%
140	Claude 3 Haiku	19.4%	$0.0015	3.6s	12%
141	GPT-5.4 Nano	21.3%	$0.0008	2.9s	4%
142	Mistral NeMO	20.6%	$0.0007	13.3s	7%
143	Nemotron 3 Nano	64.5%	$0.0031	3.5m	37%
144	Llama 3.1 8B	16.1%	$0.0002	16.1s	0%
145	WizardLM 2 8x22b	17.1%	$0.0036	15.7s	0%
146	Rocinante 12B	6.4%	$0.0009	6.0s	0%
147	GPT-4.1 Nano	2.6%	$0.0004	3.9s	0%
148	Gemma 3 4B	3.2%	$0.0002	12.5s	0%
149	Arcee AI: Trinity Large (Preview)	46.2%	$0.0000	3.1m	24%
150	LFM2 24B	0.0%	$0.0008	1.9m	0%
73.69%

Individual Scenarios

matrix

▼

Large codex (40 entries), long passage (1,019 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.7 Max	98	98	98	98	97	95	95	95	92	92	96.2%
Grok 4	100	98	98	97	97	95	95	94	94	91	96.1%
Qwen3.6 Max Preview	97	97	97	94	94	94	94	94	91	91	94.2%
Claude Sonnet 4.6 (Reasoning)	97	95	95	94	94	94	94	92	92	88	93.6%
GPT-5.5	97	97	95	94	94	94	92	92	89	89	93.5%
Grok 4.20 (Reasoning)	100	97	97	94	94	92	92	91	91	86	93.5%
Qwen 3.5 397B A17B	100	100	97	95	94	94	89	89	88	85	93.2%
GPT-5.5 (Reasoning)	97	97	94	94	94	92	92	89	89	89	92.9%
GPT-5.5 (Reasoning, Low)	95	95	95	95	92	92	92	92	91	83	92.6%
GPT-5.4	97	94	94	92	92	92	91	91	91	91	92.6%
MoonshotAI: Kimi K2.6	100	98	95	94	94	89	88	88	86	79	91.2%
GPT-5.2	97	97	94	92	92	91	89	88	86	85	91.2%
Gemini 2.5 Pro	100	95	92	92	92	89	88	88	88	86	91.2%
Qwen 3.5 27B	94	92	92	92	91	91	91	91	88	86	90.9%
DeepSeek V4 Flash (Reasoning)	95	95	94	92	92	92	91	89	85	82	90.9%
Grok 4 Fast	100	100	97	92	91	89	86	86	83	82	90.8%
GPT-5	94	94	92	92	91	91	88	88	88	88	90.6%
Grok 4.1 Fast	100	97	94	91	91	91	89	89	82	82	90.6%
Grok 4.20 (Beta, Reasoning)	97	97	94	92	89	89	88	86	86	82	90.2%
Gemini 3.1 Pro (Preview)	94	94	91	91	91	89	88	88	88	86	90.0%
Claude Opus 4.6 (Reasoning)	94	92	91	91	91	91	91	88	86	85	90.0%
GPT-5.4 (Reasoning, Low)	94	94	92	92	91	89	88	88	86	83	89.8%
Z.AI GLM 5.1	95	95	92	91	91	91	89	85	85	82	89.7%
GPT-5.4 (Reasoning)	94	94	92	91	91	91	86	85	85	82	89.1%
Claude Opus 4.5	92	92	91	91	89	89	88	88	85	83	88.9%
Grok 4.3 (Reasoning)	100	91	88	88	88	88	88	88	85	80	88.3%
Claude Opus 4.7 (Reasoning)	91	91	91	89	88	88	88	86	85	83	88.0%
Z.AI GLM 5 Turbo	92	92	91	89	89	86	86	83	83	83	87.7%
Nemotron 3 Super	94	91	89	89	89	88	86	83	83	80	87.4%
Claude Opus 4.7	92	91	91	89	89	86	85	83	83	80	87.1%
GPT-5 Mini	91	91	88	88	86	86	85	85	85	85	87.0%
Gemini 3.5 Flash (Reasoning)	92	91	91	88	86	86	85	85	82	79	86.5%
Gemini 2.5 Flash (Reasoning)	91	89	89	89	88	85	85	85	85	77	86.4%
ByteDance Seed 1.6	94	91	91	89	88	85	83	82	80	79	86.2%
Z.AI GLM 4.6	94	92	92	88	88	86	83	82	77	77	86.1%
Qwen 3.6 35B	97	97	91	91	85	85	82	80	77	74	85.9%
Qwen 3.5 Plus (2026-04-20)	97	91	89	89	88	85	85	82	77	74	85.8%
o4 Mini High	94	91	91	88	85	83	82	82	82	79	85.6%
GPT-5.1	94	91	88	85	85	85	82	82	80	80	85.2%
Qwen 3.6 Flash	89	88	88	88	88	85	85	80	79	76	84.5%
Gemma 4 31B (Reasoning)	92	91	89	89	88	88	83	79	74	68	84.2%
Qwen 3.5 122B	89	88	86	86	83	82	82	80	79	79	83.5%
Qwen 3.5 35B	92	91	89	85	85	83	79	79	76	71	83.0%
Z.AI GLM 5	97	85	85	83	82	82	82	79	79	77	83.0%
Gemini 3 Flash (Preview, Reasoning)	89	88	85	83	83	82	82	82	82	74	83.0%
Claude Opus 4.6	85	85	85	85	83	83	80	80	80	80	82.7%
Aion 2.0	100	97	94	92	91	89	88	88	88	0	82.7%
Xiaomi MIMO v2.5 Pro	92	88	86	85	83	83	80	79	79	70	82.6%
Z.AI GLM 4.7	94	88	88	83	83	82	79	79	74	73	82.3%
Qwen 3.6 27B	92	91	85	85	85	83	77	76	76	73	82.3%
Gemma 4 31B	86	86	85	83	82	82	80	80	79	79	82.3%
Claude Opus 4	92	89	85	83	82	82	79	77	77	70	81.7%
Qwen 3.5 Flash	94	91	86	85	82	82	79	77	76	62	81.4%
MoonshotAI: Kimi K2.5	91	86	86	86	82	82	80	80	73	65	81.2%
Claude Sonnet 4.6	86	82	82	82	80	77	77	76	76	74	79.2%
Gemma 4 26B (Reasoning)	88	85	80	80	79	79	77	76	74	74	79.2%
Claude Sonnet 4	83	82	80	80	80	79	79	77	73	71	78.5%
Gemini 3 Flash (Preview)	85	82	82	80	79	76	76	76	74	74	78.3%
Claude Sonnet 4.5	85	83	82	80	79	77	77	77	71	70	78.2%
Gemini 3.5 Flash (Reasoning, Minimal)	85	82	82	80	79	79	77	74	71	70	77.9%
o4 Mini	88	86	82	82	79	76	76	73	70	67	77.7%
Stealth: Hunter Alpha	88	83	83	82	80	77	76	73	68	59	77.0%
GPT-5.4 Mini (Reasoning)	83	82	82	80	80	74	73	73	71	70	76.8%
ByteDance Seed 2.0 Lite	85	85	83	80	79	77	76	68	65	62	76.1%
Stealth: Healer Alpha	94	91	91	82	79	76	74	62	58	42	74.8%
ByteDance Seed 2.0 Mini	83	82	76	74	74	74	74	73	73	64	74.7%
Qwen 3.5 9B	85	82	79	77	77	76	73	71	62	62	74.4%
Xiaomi MIMO v2.5	89	85	85	82	79	79	76	70	70	0	71.4%
DeepSeek V4 Pro (Reasoning)	97	95	94	92	88	85	82	76	0	0	70.9%
Gemini 2.5 Flash	79	77	76	76	74	73	70	64	61	59	70.8%
Gemini 3 Pro (Preview)	83	82	80	76	76	76	74	73	67	15	70.2%
Inception Mercury 2	79	77	74	71	71	70	70	70	61	58	70.0%
Gemini 2.5 Flash Lite (Reasoning)	79	77	76	73	73	68	67	67	62	55	69.5%
GPT-5.4 Mini (Reasoning, Low)	77	77	71	70	70	68	67	64	62	59	68.5%
DeepSeek V4 Pro	77	76	71	71	68	68	67	64	56	53	67.1%
GPT-OSS 120B	76	76	73	70	67	67	62	61	56	53	65.9%
Qwen 3.5 Plus (2026-02-15)	92	85	83	82	79	77	76	39	38	0	65.2%
Mistral Large	79	70	67	67	65	65	62	61	59	58	65.2%
Mistral Large 3	73	73	70	68	68	64	61	61	56	52	64.4%
GPT-4.1	77	74	70	67	65	64	62	58	56	45	63.8%
Gemma 4 26B	70	67	67	65	65	64	64	59	58	56	63.3%
MiniMax M2.7	77	71	68	64	64	64	59	56	55	48	62.6%
Gemini 3.1 Flash Lite	70	67	62	62	62	58	58	58	56	48	60.0%
DeepSeek V4 Flash	79	73	67	67	64	64	59	55	38	32	59.5%
Stealth: Aurora Alpha	68	67	65	65	59	58	58	58	55	42	59.4%
Gemini 3.1 Flash Lite (Preview)	67	65	65	62	59	59	56	56	53	52	59.4%
Mistral Large 2	67	67	67	61	59	58	56	56	53	47	58.9%
Gemini 3.1 Flash Lite (Reasoning)	65	65	62	61	61	61	61	59	53	32	57.9%
GPT-5.4 Nano (Reasoning)	82	80	76	74	70	68	64	59	0	0	57.3%
DeepSeek V3.2	94	65	59	59	52	47	47	47	44	39	55.3%
MiniMax M2.5	73	71	64	61	53	53	50	42	41	36	54.4%
Z.AI GLM 4.5	76	67	55	53	53	50	45	45	45	36	52.6%
Z.AI GLM 4.5 Air	80	71	68	68	67	56	52	32	23	3	52.0%
Grok 4.20	73	61	55	53	52	50	47	47	39	35	51.1%
Claude Haiku 4.5	67	58	50	50	50	50	48	47	47	42	50.9%
DeepSeek-V2 Chat	61	58	56	53	50	50	47	44	39	33	49.1%
Mistral Medium 3.1	62	56	53	52	52	48	48	48	47	21	48.8%
DeepSeek V3 (2024-12-26)	67	64	59	58	58	56	48	35	24	17	48.5%
Grok 4.20 (Beta)	61	58	56	55	50	48	39	39	38	36	48.0%
GPT-5 Nano	58	58	56	52	50	48	45	44	35	35	48.0%
Qwen3 235B A22B Instruct 2507	58	55	55	55	55	44	44	42	36	33	47.6%
DeepSeek V3.1	71	67	64	62	58	48	41	36	24	0	47.1%
Mistral Small 4 (Reasoning)	53	50	50	47	47	44	44	41	39	29	44.4%
Claude 3.7 Sonnet	61	59	48	44	39	39	38	38	35	30	43.2%
Writer: Palmyra X5	56	56	53	47	44	44	41	39	38	3	42.1%
Grok 4.3	73	61	61	55	50	47	42	21	0	0	40.9%
Z.AI GLM 4.7 Flash	53	47	45	42	41	38	33	33	32	29	39.4%
DeepSeek V3 (2025-03-24)	68	62	48	47	45	33	32	30	23	0	38.9%
ByteDance Seed 1.6 Flash	59	41	39	39	38	38	35	32	29	23	37.3%
Mistral Small Creative	47	41	41	41	38	35	33	33	29	5	34.2%
Claude 3.5 Sonnet	38	36	35	33	33	32	32	32	30	30	33.2%
Qwen 3 32B	44	41	39	39	38	32	26	18	17	12	30.6%
GPT-4.1 Mini	53	44	33	33	30	29	26	21	20	14	30.3%
Nemotron 3 Nano	39	38	36	32	30	29	29	27	14	3	27.7%
Inception Mercury	52	38	35	35	29	23	20	20	17	2	26.8%
GPT-5.4 Nano (Reasoning, Low)	59	48	47	41	39	32	0	0	0	0	26.7%
GPT-5.4 Mini	38	32	32	29	29	29	23	18	15	14	25.8%
Mistral Small 3.2 24B	44	39	39	35	24	21	18	18	9	0	24.8%
GPT-4o, May 13th (temp=1)	36	30	29	29	29	26	24	18	17	6	24.4%
GPT-4o, Aug. 6th (temp=0)	44	35	29	29	29	29	27	23	0	0	24.4%
GPT-4o, Aug. 6th (temp=1)	35	32	29	27	24	21	20	15	14	14	23.0%
Ministral 3 8B	35	30	29	24	21	21	20	18	14	3	21.5%
Ministral 3 14B	32	32	27	24	24	21	14	6	0	0	18.0%
GPT-4o, May 13th (temp=0)	39	35	33	33	32	0	0	0	0	0	17.3%
Qwen 2.5 72B	27	26	20	17	15	15	15	12	9	8	16.4%
Ministral 8B	33	32	24	23	23	12	2	2	2	2	15.3%
Gemma 3 27B	36	26	15	15	14	12	11	9	9	0	14.7%
Mistral Small 4	29	20	18	12	12	11	9	2	0	0	11.2%
Gemini 2.5 Flash Lite	32	30	26	18	5	0	0	0	0	0	11.1%
Hermes 3 405B	27	18	11	11	8	8	6	6	5	2	10.0%
Hermes 3 70B	14	11	11	8	8	8	5	2	0	0	6.4%
Llama 3.1 Nemotron 70B	12	12	11	8	6	5	3	0	0	0	5.6%
Arcee AI: Trinity Mini	15	9	6	6	5	3	3	2	0	0	4.8%
Cohere Command R+ (Aug. 2024)	12	12	9	6	5	3	0	0	0	0	4.7%
Gemma 3 12B	11	9	8	6	5	0	0	0	0	0	3.8%
GPT-5.4 Nano	23	9	3	3	0	0	0	0	0	0	3.8%
GPT-4o Mini (temp=1)	9	6	6	5	2	0	0	0	0	0	2.7%
Llama 3.1 70B	12	6	5	3	2	0	0	0	0	0	2.7%
Claude 3 Haiku	11	8	3	2	2	2	0	0	0	0	2.6%
WizardLM 2 8x22b	9	6	6	2	2	0	0	0	0	0	2.4%
Mistral NeMO	11	0	0	0	0	0	0	0	0	0	1.1%
GPT-4o Mini (temp=0)	8	0	0	0	0	0	0	0	0	0	0.8%
Llama 3.1 8B	5	3	0	0	0	0	0	0	0	0	0.8%
Ministral 3 3B	5	3	0	0	0	0	0	0	0	0	0.8%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Ministral 3B	3	0	0	0	0	0	0	0	0	0	0.3%
Gemma 3 4B	2	0	0	0	0	0	0	0	0	0	0.2%
Arcee AI: Trinity Large (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Large codex (40 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	97	97	97	97	98.9%
Claude Opus 4.6	100	100	100	100	97	97	97	97	97	92	97.9%
GPT-5.4	100	100	100	100	97	97	97	97	97	92	97.9%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	95	95	95	95	92	97.1%
Grok 4	100	100	100	100	95	95	95	95	95	95	96.8%
Z.AI GLM 5 Turbo	100	100	100	100	100	97	97	92	92	87	96.6%
MoonshotAI: Kimi K2.6	100	100	97	97	97	97	95	95	95	92	96.6%
Z.AI GLM 5.1	100	100	100	97	95	95	95	95	95	92	96.3%
Claude Sonnet 4.6 (Reasoning)	100	97	97	97	97	97	97	92	92	92	96.1%
MoonshotAI: Kimi K2.5	100	100	100	97	97	95	95	92	92	92	96.1%
Claude Sonnet 4.6	100	100	100	100	95	95	95	95	92	87	95.8%
Qwen3.7 Max	100	100	100	95	95	95	95	95	92	92	95.8%
Gemini 3.5 Flash (Reasoning)	100	95	95	95	95	95	95	95	92	92	94.7%
DeepSeek V4 Pro (Reasoning)	100	97	97	95	95	95	95	95	92	87	94.7%
GPT-5.5 (Reasoning)	97	97	95	95	95	95	95	92	92	92	94.5%
DeepSeek V4 Flash (Reasoning)	100	100	100	97	97	95	92	89	84	84	93.9%
Claude Sonnet 4	100	100	100	100	92	92	92	92	87	84	93.9%
Qwen3.6 Max Preview	100	95	95	95	95	92	92	92	92	89	93.7%
Qwen 3.6 35B	100	100	100	95	95	95	95	89	89	76	93.4%
Grok 4.3 (Reasoning)	100	95	95	95	95	95	95	89	89	84	93.2%
Qwen 3.5 35B	100	97	97	95	92	92	89	89	87	87	92.6%
GPT-5.2	97	97	95	95	95	92	89	89	89	87	92.6%
Grok 4.20 (Beta)	97	97	97	97	97	92	89	89	87	79	92.4%
Claude Opus 4.6 (Reasoning)	100	100	95	95	95	92	87	87	87	87	92.4%
Qwen 3.6 Flash	95	95	95	95	95	95	89	89	87	87	92.1%
ByteDance Seed 1.6	100	95	95	95	92	89	89	89	89	84	91.8%
Gemini 3 Flash (Preview, Reasoning)	100	97	95	95	92	92	89	87	87	84	91.8%
Xiaomi MIMO v2.5 Pro	100	100	95	92	92	92	92	89	84	82	91.8%
GPT-5.4 (Reasoning)	95	95	95	92	92	92	92	89	87	87	91.6%
Grok 4.20 (Beta, Reasoning)	95	95	95	95	95	89	89	89	89	84	91.6%
Qwen 3.5 Flash	100	95	92	92	89	89	89	89	87	87	91.1%
Grok 4.20 (Reasoning)	95	95	95	95	89	89	89	89	89	84	91.1%
Gemma 4 31B (Reasoning)	100	97	95	95	95	95	92	84	79	76	90.8%
Claude Opus 4.7	92	92	92	92	92	92	92	92	92	79	90.8%
Grok 4.20	97	97	97	97	97	89	89	89	82	71	90.8%
Qwen 3.5 122B	100	95	92	92	92	89	89	87	87	82	90.5%
Gemini 2.5 Flash (Reasoning)	100	95	92	89	89	89	89	89	87	84	90.5%
Qwen 3.5 27B	92	92	92	92	89	89	89	89	89	87	90.3%
Qwen 3.5 Plus (2026-04-20)	100	95	89	89	89	89	89	87	87	84	90.0%
Grok 4 Fast	95	95	95	89	89	89	89	89	89	79	90.0%
GPT-5	95	95	92	89	89	89	89	89	87	82	89.7%
Grok 4.1 Fast	95	95	95	92	92	89	89	87	84	79	89.7%
Qwen 3.5 397B A17B	92	92	92	92	92	92	92	84	84	84	89.7%
Gemma 4 31B	92	92	92	92	92	92	92	84	84	84	89.7%
GPT-5.4 (Reasoning, Low)	95	95	89	89	89	89	89	89	84	84	89.5%
Claude Opus 4.7 (Reasoning)	95	95	95	89	89	89	89	89	82	82	89.5%
Qwen 3.6 27B	100	95	95	95	92	89	89	89	84	63	89.2%
Gemini 3.5 Flash (Reasoning, Minimal)	92	92	92	92	92	92	87	84	84	84	89.2%
GPT-5.4 Mini (Reasoning)	95	95	89	89	89	89	84	84	84	84	88.4%
Aion 2.0	95	95	89	89	89	89	89	87	84	76	88.4%
Z.AI GLM 4.7	100	92	92	92	89	87	84	84	84	74	87.9%
Z.AI GLM 5	100	100	95	92	89	82	82	79	79	79	87.6%
Stealth: Hunter Alpha	100	95	95	92	89	87	87	84	82	66	87.6%
Stealth: Healer Alpha	100	92	89	89	89	87	84	84	82	79	87.6%
Xiaomi MIMO v2.5	95	95	89	89	89	87	84	84	82	82	87.6%
GPT-5.1	92	92	92	89	89	89	84	82	82	82	87.4%
Gemini 3 Flash (Preview)	92	92	92	92	92	84	84	84	84	76	87.4%
o4 Mini High	95	95	87	87	87	87	84	84	82	82	86.8%
Gemini 3 Pro (Preview)	100	95	92	92	89	89	82	79	76	74	86.8%
Z.AI GLM 4.6	97	92	92	89	89	89	87	84	74	71	86.6%
ByteDance Seed 2.0 Lite	95	89	89	89	89	84	82	82	82	82	86.3%
GPT-4.1	100	100	95	92	92	84	84	76	71	66	86.1%
o4 Mini	95	89	89	87	87	87	82	82	79	74	85.0%
Nemotron 3 Super	89	89	89	84	84	84	84	84	82	79	85.0%
Qwen 3.5 9B	95	95	89	89	84	84	82	76	74	74	84.2%
DeepSeek V4 Flash	95	92	92	92	87	87	79	79	74	66	84.2%
Gemma 4 26B (Reasoning)	95	92	87	84	84	84	79	79	79	76	83.9%
Gemini 3.1 Flash Lite (Preview)	97	89	82	82	82	82	82	82	82	79	83.7%
ByteDance Seed 2.0 Mini	95	95	92	84	82	82	82	79	76	68	83.4%
Gemini 2.5 Flash	92	92	89	89	87	82	82	71	71	68	82.4%
Gemini 3.1 Flash Lite (Reasoning)	100	92	84	84	84	79	76	76	74	71	82.1%
DeepSeek V4 Pro	100	87	87	84	84	79	79	74	74	74	82.1%
Claude Opus 4	95	92	87	84	79	79	79	79	71	68	81.3%
GPT-5.4 Mini (Reasoning, Low)	89	89	87	84	82	79	79	76	74	74	81.3%
Qwen 3.5 Plus (2026-02-15)	92	92	84	84	84	84	79	79	68	63	81.1%
Inception Mercury 2	84	84	84	82	79	79	79	79	76	68	79.5%
Writer: Palmyra X5	89	89	87	84	82	79	74	71	68	66	78.9%
Stealth: Aurora Alpha	89	89	84	76	76	76	76	76	76	66	78.7%
GPT-5 Mini	89	89	87	87	87	87	84	79	53	45	78.7%
GPT-OSS 120B	84	82	79	79	79	79	79	76	76	68	78.2%
Gemini 3.1 Flash Lite	89	84	82	79	79	76	74	74	74	68	77.9%
Z.AI GLM 4.5	87	87	82	82	82	79	76	74	63	63	77.4%
MiniMax M2.5	89	89	82	82	79	76	74	71	68	61	77.1%
Gemini 2.5 Flash Lite (Reasoning)	84	84	82	82	79	76	76	74	68	63	76.8%
GPT-5.4 Nano (Reasoning)	84	84	82	79	76	76	74	71	71	63	76.1%
Claude 3.5 Sonnet	82	82	82	82	74	74	74	66	66	61	73.9%
MiniMax M2.7	89	89	84	79	76	71	66	63	61	58	73.7%
Claude 3.7 Sonnet	76	74	74	74	74	74	74	71	71	71	73.2%
Mistral Medium 3.1	76	76	74	74	74	74	74	68	66	63	71.8%
Mistral Large	87	84	79	71	68	68	66	66	61	61	71.1%
GPT-4o, May 13th (temp=0)	79	79	74	74	74	68	68	68	63	61	70.8%
DeepSeek V3 (2024-12-26)	84	82	79	71	68	68	66	63	63	61	70.5%
Mistral Large 2	82	79	79	71	66	66	66	66	63	63	70.0%
DeepSeek-V2 Chat	89	76	76	74	68	68	63	63	61	58	69.7%
Qwen3 235B A22B Instruct 2507	84	82	79	76	74	68	66	63	58	45	69.5%
DeepSeek V3.2	95	84	82	76	74	71	71	63	61	18	69.5%
Grok 4.3	82	76	74	68	68	68	66	63	61	58	68.4%
Z.AI GLM 4.7 Flash	79	74	74	71	68	68	63	63	61	55	67.6%
Mistral Large 3	74	71	71	68	68	68	66	63	63	61	67.4%
Mistral Small 4 (Reasoning)	79	76	74	68	63	63	63	58	55	53	65.3%
GPT-4.1 Mini	76	74	74	71	66	66	63	61	50	34	63.4%
Z.AI GLM 4.5 Air	87	82	79	79	74	66	66	53	32	0	61.6%
DeepSeek V3 (2025-03-24)	89	89	74	74	68	66	63	58	29	0	61.1%
Nemotron 3 Nano	79	74	66	63	63	61	58	47	45	37	59.2%
Claude Haiku 4.5	87	79	79	79	71	71	63	61	0	0	58.9%
GPT-4o, Aug. 6th (temp=0)	74	61	61	58	58	58	55	55	55	53	58.7%
Mistral Small 3.2 24B	68	63	63	61	58	55	55	55	55	53	58.7%
ByteDance Seed 1.6 Flash	74	68	66	63	61	61	61	55	50	29	58.7%
GPT-5 Nano	82	82	76	74	71	68	66	61	0	0	57.9%
DeepSeek V3.1	82	82	82	79	71	71	55	42	0	0	56.3%
Gemini 2.5 Flash Lite	76	68	61	61	58	50	47	47	45	39	55.3%
Qwen 3 32B	76	74	63	61	58	53	53	50	37	26	55.0%
GPT-4o, Aug. 6th (temp=1)	68	68	66	63	61	58	55	47	45	0	53.2%
Inception Mercury	84	66	66	53	53	50	45	42	42	29	52.9%
GPT-4o, May 13th (temp=1)	74	71	68	53	53	53	45	45	32	32	52.4%
GPT-5.4 Mini	87	82	79	74	71	66	50	0	0	0	50.8%
Mistral Small Creative	63	53	53	50	47	47	45	39	39	21	45.8%
Arcee AI: Trinity Large (Preview)	76	61	58	50	50	42	39	37	32	0	44.5%
Ministral 3 8B	53	50	47	47	45	45	42	39	39	0	40.8%
Ministral 8B	50	45	42	42	39	37	37	37	34	34	39.7%
Gemma 3 27B	47	45	42	39	39	37	37	37	37	37	39.7%
Ministral 3 14B	50	47	45	42	39	37	37	34	32	24	38.7%
Hermes 3 405B	50	50	47	45	42	37	34	29	29	21	38.4%
Ministral 3B	53	50	45	39	39	26	24	18	13	8	31.6%
GPT-5.4 Nano (Reasoning, Low)	84	79	79	71	0	0	0	0	0	0	31.3%
Mistral Small 4	58	47	34	32	32	29	29	24	13	0	29.7%
GPT-4o Mini (temp=0)	32	32	32	32	32	32	32	32	32	11	29.5%
Gemma 3 12B	37	34	34	29	29	26	26	24	21	11	27.1%
Arcee AI: Trinity Mini	37	37	37	37	34	26	24	16	11	8	26.6%
Qwen 2.5 72B	42	39	39	37	34	32	21	8	3	0	25.5%
Ministral 3 3B	50	39	39	32	29	26	16	16	5	0	25.3%
Gemma 4 26B	79	79	79	0	0	0	0	0	0	0	23.7%
Llama 3.1 Nemotron 70B	39	32	29	26	26	24	21	18	16	0	23.2%
Llama 3.1 70B	47	34	32	26	21	16	16	13	3	0	20.8%
GPT-5.4 Nano	53	45	37	32	29	0	0	0	0	0	19.5%
Hermes 3 70B	34	34	26	26	18	16	16	11	3	0	18.4%
Cohere Command R+ (Aug. 2024)	39	34	32	26	24	8	5	5	0	0	17.4%
GPT-4o Mini (temp=1)	24	21	18	16	13	11	11	5	0	0	11.8%
Claude 3 Haiku	26	18	11	8	5	5	5	3	0	0	8.2%
Mistral NeMO	18	11	0	0	0	0	0	0	0	0	2.9%
Llama 3.1 8B	5	0	0	0	0	0	0	0	0	0	0.5%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Rocinante 12B	3	0	0	0	0	0	0	0	0	0	0.3%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), long passage (734 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	95	90	98.5%
Claude Opus 4.6	100	100	100	100	100	100	100	100	90	85	97.5%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	85	85	97.0%
Grok 4.20 (Reasoning)	100	100	100	100	95	95	95	95	95	95	97.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	90	90	80	96.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	90	90	90	85	95.5%
Qwen 3.5 397B A17B	100	100	100	100	100	100	90	90	90	85	95.5%
GPT-5.2	100	95	95	95	95	95	95	95	95	95	95.5%
GPT-5 Mini	100	100	95	95	95	95	95	95	95	85	95.0%
Qwen 3.5 27B	100	100	100	100	100	90	90	90	90	90	95.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	95	90	90	85	85	94.5%
Qwen3.6 Max Preview	100	100	100	100	90	90	90	90	90	90	94.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	95	95	90	90	70	94.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	90	90	90	90	85	85	93.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	90	90	85	85	80	93.0%
Qwen 3.5 Flash	100	100	100	100	90	90	90	90	90	80	93.0%
Z.AI GLM 5.1	100	100	100	100	100	90	90	85	85	75	92.5%
GPT-5.1	100	100	100	100	90	90	90	90	85	70	91.5%
Qwen 3.5 122B	100	100	100	100	100	90	90	90	80	65	91.5%
Gemini 2.5 Flash (Reasoning)	100	100	100	95	90	90	90	90	80	80	91.5%
Aion 2.0	100	100	100	90	90	90	90	90	85	75	91.0%
Stealth: Hunter Alpha	100	100	100	100	90	90	85	85	85	75	91.0%
Gemma 4 31B	100	100	90	90	90	90	90	90	90	80	91.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	85	85	85	85	85	85	91.0%
DeepSeek V4 Pro (Reasoning)	100	100	95	90	90	90	90	90	90	70	90.5%
GPT-5.5 (Reasoning)	95	95	95	95	95	95	90	80	80	80	90.0%
Claude Opus 4.7	90	90	90	90	90	90	90	90	90	90	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	90	70	70	70	90.0%
GPT-5	100	100	95	95	90	90	90	85	85	65	89.5%
Qwen 3.5 Plus (2026-04-20)	100	90	90	90	90	90	90	90	90	75	89.5%
GPT-5.5	100	100	100	85	85	85	85	85	85	85	89.5%
DeepSeek V4 Flash (Reasoning)	100	100	95	90	90	90	85	85	80	75	89.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	90	90	90	90	90	90	75	75	89.0%
GPT-5.5 (Reasoning, Low)	95	95	95	95	95	95	90	80	75	70	88.5%
Z.AI GLM 5	100	100	90	90	90	90	90	90	75	70	88.5%
Qwen 3.6 Flash	100	90	90	90	90	90	90	90	75	75	88.0%
Xiaomi MIMO v2.5 Pro	100	100	95	95	90	90	85	85	70	70	88.0%
Qwen3.7 Max	100	100	100	100	100	90	90	80	60	60	88.0%
Grok 4 Fast	100	100	95	95	85	85	85	80	80	75	88.0%
Gemini 3 Pro (Preview)	100	90	90	90	90	90	90	80	80	75	87.5%
o4 Mini	90	90	90	90	90	90	85	85	85	80	87.5%
Qwen 3.6 27B	90	90	90	90	90	90	90	85	80	75	87.0%
Grok 4.1 Fast	100	100	95	95	90	85	85	80	75	65	87.0%
Claude Sonnet 4.6	100	90	85	85	85	85	85	85	85	85	87.0%
Z.AI GLM 5 Turbo	100	100	100	90	90	90	75	75	75	70	86.5%
MoonshotAI: Kimi K2.6	100	100	95	95	90	85	85	70	70	70	86.0%
ByteDance Seed 1.6	90	90	90	90	90	90	90	75	75	75	85.5%
o4 Mini High	95	90	90	90	90	85	85	80	80	70	85.5%
ByteDance Seed 2.0 Mini	100	90	90	90	90	90	80	80	75	65	85.0%
Xiaomi MIMO v2.5	90	90	90	90	90	85	85	75	75	75	84.5%
Qwen 3.5 35B	100	100	90	90	90	90	75	75	70	65	84.5%
Qwen 3.6 35B	90	90	90	90	90	90	80	75	75	70	84.0%
GPT-5.4	100	100	100	100	85	70	70	70	70	70	83.5%
Gemma 4 26B (Reasoning)	100	90	90	90	90	90	90	90	85	0	81.5%
Grok 4	100	95	90	90	90	85	85	65	60	55	81.5%
Claude Sonnet 4.5	100	90	85	85	85	75	75	70	70	70	80.5%
ByteDance Seed 2.0 Lite	90	90	90	85	80	80	75	75	75	65	80.5%
MoonshotAI: Kimi K2.5	90	90	90	90	90	85	80	65	60	60	80.0%
Z.AI GLM 4.7	100	90	85	80	80	75	75	75	70	70	80.0%
Z.AI GLM 4.6	90	90	85	85	85	80	75	75	75	60	80.0%
Claude Haiku 4.5	80	80	80	80	80	80	80	80	80	80	80.0%
GPT-OSS 120B	90	90	90	80	80	75	75	75	70	65	79.0%
Z.AI GLM 4.5 Air	90	80	80	80	80	80	80	80	75	60	78.5%
Stealth: Healer Alpha	90	90	90	90	85	85	75	65	65	45	78.0%
DeepSeek V4 Flash	90	90	85	80	75	75	75	70	70	70	78.0%
Claude Sonnet 4	100	85	85	85	70	70	70	70	70	70	77.5%
Gemma 4 26B	80	80	80	80	75	75	75	75	75	75	77.0%
MiniMax M2.7	90	80	80	80	80	80	75	75	75	55	77.0%
Claude 3.7 Sonnet	90	90	80	75	75	75	75	75	65	65	76.5%
Gemini 3 Flash (Preview)	90	75	75	75	75	75	75	75	75	70	76.0%
Gemini 2.5 Flash	90	90	80	80	75	75	70	70	70	55	75.5%
Inception Mercury 2	75	75	75	75	75	75	75	75	75	70	74.5%
GPT-5.4 (Reasoning)	85	85	80	70	65	65	65	65	65	65	71.0%
GPT-5.4 (Reasoning, Low)	100	75	70	70	70	65	65	65	65	65	71.0%
Gemini 3.1 Flash Lite (Preview)	75	70	70	70	70	70	70	70	70	70	70.5%
Gemini 3.1 Flash Lite	75	70	70	70	70	70	70	70	70	70	70.5%
Mistral Large	90	80	75	75	75	65	65	60	60	60	70.5%
Claude 3.5 Sonnet	80	75	70	70	70	70	70	65	65	65	70.0%
MiniMax M2.5	80	80	75	75	70	70	65	65	60	60	70.0%
Claude Opus 4	90	75	75	75	75	65	65	60	60	60	70.0%
Grok 4.20	80	80	75	70	70	70	70	65	60	60	70.0%
Gemini 3.1 Flash Lite (Reasoning)	70	70	70	70	70	70	70	70	70	65	69.5%
Z.AI GLM 4.5	75	75	75	75	75	70	70	65	60	55	69.5%
Nemotron 3 Super	100	95	90	70	70	70	60	55	45	35	69.0%
GPT-5.4 Mini (Reasoning)	75	75	75	70	70	70	70	70	65	50	69.0%
ByteDance Seed 1.6 Flash	90	80	75	70	70	65	65	60	55	55	68.5%
Qwen 3.5 9B	90	80	80	80	75	75	70	65	65	0	68.0%
Grok 4.20 (Beta)	80	80	75	75	70	70	70	55	55	45	67.5%
GPT-5.4 Mini (Reasoning, Low)	80	75	75	75	70	65	65	55	55	55	67.0%
Mistral Medium 3.1	75	75	75	75	75	75	75	70	70	0	66.5%
GPT-4.1 Mini	70	70	70	70	70	70	65	65	65	50	66.5%
Gemini 2.5 Flash Lite (Reasoning)	75	75	75	70	70	65	65	60	55	55	66.5%
Mistral Large 3	85	75	65	65	65	60	60	60	60	60	65.5%
GPT-4o, May 13th (temp=0)	70	70	70	65	65	65	65	65	65	50	65.0%
GPT-5.4 Nano (Reasoning)	85	80	75	75	75	70	65	65	60	0	65.0%
Llama 3.1 Nemotron 70B	75	65	65	65	65	65	60	60	60	60	64.0%
Mistral Large 2	80	75	65	65	65	65	65	60	50	50	64.0%
Stealth: Aurora Alpha	85	80	75	75	75	75	75	75	10	10	63.5%
Mistral Small 3.2 24B	75	75	75	70	65	65	55	55	50	50	63.5%
GPT-4o, Aug. 6th (temp=0)	65	65	65	65	65	65	65	60	60	55	63.0%
GPT-4.1	75	65	65	65	65	65	65	60	55	50	63.0%
Z.AI GLM 4.7 Flash	70	70	70	70	65	65	60	55	55	45	62.5%
Mistral Small 4 (Reasoning)	80	75	70	65	65	65	55	55	50	40	62.0%
DeepSeek-V2 Chat	80	70	70	70	70	60	55	50	45	45	61.5%
DeepSeek V4 Pro	85	85	70	70	70	60	55	55	55	0	60.5%
GPT-4o, Aug. 6th (temp=1)	70	70	65	65	60	55	55	55	55	50	60.0%
Grok 4.3	80	70	70	70	65	65	55	55	45	20	59.5%
Qwen 3 32B	80	80	65	65	65	55	55	45	45	35	59.0%
Mistral Small Creative	75	70	70	60	55	55	55	50	45	40	57.5%
Writer: Palmyra X5	70	70	60	60	60	55	55	55	45	45	57.5%
Llama 3.1 70B	70	65	60	55	55	55	55	55	55	50	57.5%
Ministral 3 14B	80	80	75	75	65	65	65	60	5	5	57.5%
GPT-4o, May 13th (temp=1)	70	65	60	60	60	60	55	50	50	40	57.0%
GPT-5 Nano	75	65	65	60	60	55	55	45	45	45	57.0%
Qwen3 235B A22B Instruct 2507	65	65	65	60	60	55	50	50	45	40	55.5%
DeepSeek V3 (2025-03-24)	70	70	60	55	55	55	50	45	45	45	55.0%
DeepSeek V3 (2024-12-26)	70	65	60	55	55	55	55	45	45	45	55.0%
Nemotron 3 Nano	65	60	60	60	55	55	55	45	45	40	54.0%
Qwen 2.5 72B	70	60	60	60	55	55	50	50	45	35	54.0%
Gemini 2.5 Flash Lite	75	60	60	55	55	55	55	50	40	35	54.0%
Gemma 3 27B	70	65	55	55	55	50	50	45	45	45	53.5%
DeepSeek V3.2	65	65	65	55	55	55	45	40	35	30	51.0%
Inception Mercury	65	65	60	55	55	50	50	50	30	15	49.5%
Arcee AI: Trinity Large (Preview)	70	55	55	55	55	50	50	45	35	20	49.0%
GPT-5.4 Mini	65	60	55	55	55	50	45	45	45	5	48.0%
Hermes 3 405B	65	55	50	45	45	45	45	40	40	40	47.0%
DeepSeek V3.1	70	55	55	55	50	45	40	40	40	0	45.0%
Ministral 3 3B	50	45	45	40	40	40	35	35	30	30	39.0%
Ministral 3 8B	55	50	45	45	40	35	30	25	20	5	35.0%
Ministral 8B	65	45	45	45	35	30	30	25	0	0	32.0%
Gemma 3 12B	50	35	35	35	25	25	25	20	20	15	28.5%
Ministral 3B	55	40	30	30	30	30	30	15	15	5	28.0%
GPT-5.4 Nano (Reasoning, Low)	75	75	65	60	0	0	0	0	0	0	27.5%
GPT-4o Mini (temp=1)	35	35	35	35	35	25	25	20	10	10	26.5%
Hermes 3 70B	45	40	35	35	30	30	25	20	5	0	26.5%
GPT-4o Mini (temp=0)	35	25	25	25	25	25	25	25	25	25	26.0%
Arcee AI: Trinity Mini	45	35	30	30	25	20	20	20	20	10	25.5%
Mistral Small 4	40	35	35	30	30	25	20	20	15	0	25.0%
Cohere Command R+ (Aug. 2024)	50	45	40	40	35	30	0	0	0	0	24.0%
Mistral NeMO	50	40	40	35	25	25	0	0	0	0	21.5%
Claude 3 Haiku	30	30	20	20	20	20	20	20	15	5	20.0%
Llama 3.1 8B	35	30	30	30	30	20	15	10	0	0	20.0%
GPT-5.4 Nano	35	20	20	15	15	15	0	0	0	0	12.0%
Gemma 3 4B	30	25	20	10	10	5	0	0	0	0	10.0%
Rocinante 12B	50	45	0	0	0	0	0	0	0	0	9.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	95	99.5%
Gemini 3.1 Flash Lite	100	100	100	100	100	100	100	100	100	95	99.5%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	90	99.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	90	99.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	90	99.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	95	90	98.5%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	95	90	98.5%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	90	90	98.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	90	90	98.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	80	98.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	95	95	95	95	95	97.5%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	90	90	90	97.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	90	90	90	97.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	90	90	90	97.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	90	90	90	97.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	85	85	97.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	85	85	97.0%
Z.AI GLM 4.6	100	100	100	100	100	100	95	90	90	90	96.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	95	95	95	95	95	95	95	96.5%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	90	90	85	96.5%
DeepSeek V3.2	100	100	100	100	100	100	100	90	85	80	95.5%
o4 Mini High	100	100	100	100	100	90	90	90	90	90	95.0%
Z.AI GLM 4.5	100	100	100	100	100	90	90	90	90	90	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	85	85	85	85	94.0%
o4 Mini	100	100	100	90	90	90	90	90	90	90	93.0%
Grok 4.20 (Beta)	100	100	100	100	100	85	85	85	85	85	92.5%
Grok 4.20	100	100	100	100	100	85	85	85	85	85	92.5%
Grok 4	100	100	90	90	90	90	90	90	90	90	92.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	90	90	85	85	85	85	92.0%
DeepSeek V3.1	100	100	100	100	90	90	85	85	85	85	92.0%
Gemma 4 31B	100	90	90	90	90	90	90	90	90	90	91.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	100	100	100	55	55	91.0%
GPT-OSS 120B	100	100	90	90	90	90	90	90	90	80	91.0%
Llama 3.1 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Llama 3.1 Nemotron 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Claude Opus 4	100	100	100	100	85	85	85	85	85	85	91.0%
Gemma 4 26B	90	90	90	90	90	90	90	90	90	90	90.0%
ByteDance Seed 2.0 Lite	100	100	90	90	90	90	90	90	80	80	90.0%
Inception Mercury 2	90	90	90	90	90	90	90	90	90	90	90.0%
Gemini 2.5 Flash	90	90	90	90	90	90	90	90	90	90	90.0%
Z.AI GLM 4.5 Air	100	100	90	90	90	90	90	90	80	75	89.5%
Mistral Large	100	90	90	90	90	90	90	85	85	85	89.5%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	95	0	89.5%
Mistral Large 2	90	90	90	90	90	90	90	90	85	85	89.0%
Claude Opus 4.7	100	100	90	90	90	90	90	90	75	75	89.0%
Aion 2.0	100	100	100	100	100	100	100	100	90	0	89.0%
DeepSeek V4 Flash	90	90	90	90	90	90	90	90	85	85	89.0%
Hermes 3 405B	100	100	100	90	90	90	90	85	80	60	88.5%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	90	90	0	88.0%
Inception Mercury	100	90	90	90	90	90	90	90	75	75	88.0%
Mistral Large 3	90	90	90	90	90	85	85	85	85	85	87.5%
Gemini 2.5 Flash Lite (Reasoning)	90	90	90	90	90	90	90	90	80	65	86.5%
MiniMax M2.7	100	100	90	90	90	85	85	75	75	75	86.5%
GPT-4o, May 13th (temp=1)	100	100	100	90	85	85	85	85	65	65	86.0%
Nemotron 3 Nano	100	90	90	90	90	80	80	80	80	80	86.0%
Gemma 3 12B	90	90	90	90	90	90	80	80	80	75	85.5%
Qwen 3.6 27B	100	100	100	100	90	90	90	90	80	10	85.0%
MiniMax M2.5	100	100	90	90	90	85	75	75	75	70	85.0%
Claude Sonnet 4.6	85	85	85	85	85	85	85	85	85	85	85.0%
Grok 4.3	100	95	90	90	90	85	80	75	75	65	84.5%
DeepSeek V4 Pro	100	100	95	95	95	90	90	90	85	0	84.0%
GPT-4.1 Mini	90	90	90	90	85	80	80	75	75	75	83.0%
ByteDance Seed 1.6 Flash	90	90	90	90	80	80	80	80	75	65	82.0%
Stealth: Aurora Alpha	90	90	90	90	90	90	90	90	90	0	81.0%
Qwen 3 32B	90	90	90	90	90	80	80	75	70	50	80.5%
Mistral Small 3.2 24B	90	90	90	80	75	75	75	75	75	75	80.0%
Qwen3 235B A22B Instruct 2507	85	85	85	85	80	75	75	75	75	65	78.5%
GPT-5 Nano	90	90	80	80	80	80	70	70	70	70	78.0%
Mistral Small 4 (Reasoning)	90	90	80	80	80	75	75	75	70	65	78.0%
Qwen 2.5 72B	90	90	90	80	80	80	80	80	70	35	77.5%
Z.AI GLM 4.7 Flash	100	90	90	80	75	75	75	65	60	55	76.5%
Arcee AI: Trinity Large (Preview)	90	85	85	80	75	70	70	70	70	65	76.0%
Writer: Palmyra X5	85	85	85	85	75	75	70	65	65	65	75.5%
Gemini 2.5 Flash Lite	75	75	75	75	75	75	75	75	75	70	74.5%
GPT-5.4 Mini	90	85	80	75	75	75	75	65	65	40	72.5%
Mistral Small Creative	80	80	70	70	70	70	70	70	70	65	71.5%
Cohere Command R+ (Aug. 2024)	90	90	80	70	70	70	65	60	60	55	71.0%
Claude Haiku 4.5	75	75	75	75	75	75	75	60	60	60	70.5%
Gemma 3 27B	75	75	70	65	65	65	65	55	55	55	64.5%
Ministral 3 8B	70	70	65	65	60	60	60	50	50	35	58.5%
Ministral 3 14B	85	75	65	60	60	60	60	55	45	5	57.0%
Ministral 8B	75	75	65	65	60	60	55	55	35	20	56.5%
GPT-4o Mini (temp=0)	60	60	60	60	60	60	50	50	50	50	56.0%
Ministral 3 3B	65	65	65	65	60	60	55	55	40	30	56.0%
GPT-5.4 Nano	90	80	75	75	70	60	55	40	0	0	54.5%
Hermes 3 70B	80	75	65	60	60	60	50	45	40	0	53.5%
GPT-4o Mini (temp=1)	60	60	60	50	50	50	50	50	50	35	51.5%
Ministral 3B	65	60	55	50	50	50	50	45	40	35	50.0%
Llama 3.1 8B	75	70	65	60	60	40	40	35	30	20	49.5%
Mistral NeMO	65	55	55	45	45	45	45	45	45	45	49.0%
Mistral Small 4	70	65	65	60	55	50	50	45	30	0	49.0%
Arcee AI: Trinity Mini	60	60	55	50	45	45	40	35	35	35	46.0%
GPT-5.4 Nano (Reasoning, Low)	100	90	80	80	75	0	0	0	0	0	42.5%
Claude 3 Haiku	45	45	45	40	35	35	35	35	20	15	35.0%
Rocinante 12B	55	40	35	15	15	15	5	5	5	0	19.0%
GPT-4.1 Nano	20	20	15	10	10	10	10	10	0	0	10.5%
WizardLM 2 8x22b	10	10	0	0	0	0	0	0	0	0	2.0%
Gemma 3 4B	5	0	0	0	0	0	0	0	0	0	0.5%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

tiers

▼

5 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	90	99.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4	100	100	100	100	100	100	100	90	90	90	97.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	70	97.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	90	80	97.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
GPT-5	100	100	100	100	100	100	90	90	90	90	96.0%
Claude Opus 4.7	100	100	100	100	100	100	100	90	90	80	96.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	90	70	96.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	80	80	96.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	90	90	90	90	96.0%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	90	90	90	90	96.0%
Gemini 3.1 Flash Lite	100	100	100	100	100	90	90	90	90	90	95.0%
DeepSeek V4 Pro	100	100	100	100	100	100	100	90	90	70	95.0%
DeepSeek V3.2	100	100	100	100	100	90	90	90	90	90	95.0%
DeepSeek V4 Flash	100	100	100	100	100	100	100	80	80	80	94.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	80	80	80	94.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	30	93.0%
Mistral Large	100	100	100	90	90	90	90	90	90	90	93.0%
GPT-OSS 120B	100	100	100	100	100	100	80	80	80	80	92.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	90	80	80	70	92.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	90	30	92.0%
Llama 3.1 70B	100	100	100	100	100	100	90	80	70	70	91.0%
Mistral Large 3	90	90	90	90	90	90	90	90	90	90	90.0%
Mistral Large 2	90	90	90	90	90	90	90	90	90	90	90.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	80	80	80	60	90.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	80	70	50	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	80	80	80	60	90.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	80	0	88.0%
Inception Mercury 2	100	100	100	80	80	80	80	80	80	80	86.0%
Mistral Small 4 (Reasoning)	100	100	90	90	90	80	80	80	80	70	86.0%
Claude Opus 4	100	100	100	100	100	70	70	70	70	70	85.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	80	80	70	70	50	85.0%
Stealth: Aurora Alpha	100	100	100	100	100	80	80	80	80	30	85.0%
MiniMax M2.7	100	100	100	80	80	80	80	80	70	70	84.0%
DeepSeek V3.1	100	100	100	100	100	100	90	80	70	0	84.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	90	90	70	60	60	60	83.0%
ByteDance Seed 1.6 Flash	100	80	80	80	80	80	80	80	80	70	81.0%
Gemini 2.5 Flash	100	80	80	80	80	80	80	80	70	70	80.0%
Inception Mercury	100	80	80	80	80	80	80	80	60	60	78.0%
MiniMax M2.5	100	100	90	80	70	70	70	50	50	50	73.0%
GPT-5 Nano	100	100	100	80	80	80	80	50	50	0	72.0%
Mistral Medium 3.1	100	70	70	70	70	70	70	70	60	60	71.0%
Claude Sonnet 4.6	70	70	70	70	70	70	70	70	70	70	70.0%
Gemma 3 27B	70	70	70	70	70	70	70	70	70	70	70.0%
GPT-4o, Aug. 6th (temp=0)	90	90	90	60	60	60	60	60	60	60	69.0%
GPT-4o, May 13th (temp=1)	90	90	70	70	60	60	60	60	60	60	68.0%
GPT-4.1 Mini	80	80	70	70	70	70	70	70	70	30	68.0%
Qwen 2.5 72B	80	80	80	80	80	80	50	50	50	50	68.0%
GPT-4o, Aug. 6th (temp=1)	90	90	70	60	60	60	60	60	60	60	67.0%
Grok 4.20	100	90	80	70	60	60	60	50	50	40	66.0%
Grok 4.20 (Beta)	80	70	70	60	60	60	60	60	60	50	63.0%
Grok 4.3	70	70	70	70	70	70	60	60	50	40	63.0%
Arcee AI: Trinity Mini	100	80	80	80	60	60	50	50	30	30	62.0%
Writer: Palmyra X5	70	70	70	70	70	70	60	60	60	0	60.0%
GPT-4o Mini (temp=1)	80	80	80	80	50	50	50	50	40	40	60.0%
Qwen3 235B A22B Instruct 2507	70	70	70	70	70	60	60	60	60	0	59.0%
Arcee AI: Trinity Large (Preview)	70	70	70	70	60	60	50	50	40	40	58.0%
Claude Haiku 4.5	70	70	70	70	60	60	50	40	40	40	57.0%
GPT-5.4 Mini	90	70	70	60	60	60	60	40	40	10	56.0%
GPT-4o Mini (temp=0)	80	80	50	50	50	50	50	50	50	50	56.0%
Gemma 3 12B	80	50	50	50	50	50	50	50	50	50	53.0%
Mistral Small Creative	60	60	60	60	60	60	50	50	40	30	53.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	80	30	20	0	0	0	0	43.0%
Llama 3.1 8B	70	70	70	70	40	30	30	20	20	0	42.0%
Gemini 2.5 Flash Lite	50	50	40	40	40	40	40	40	40	40	42.0%
Cohere Command R+ (Aug. 2024)	80	60	60	60	50	40	20	20	20	0	41.0%
Ministral 3 3B	70	70	60	50	40	40	40	30	0	0	40.0%
Mistral Small 3.2 24B	60	50	50	40	40	40	30	30	30	10	38.0%
Mistral NeMO	70	50	50	50	40	40	30	30	10	0	37.0%
Claude 3 Haiku	60	60	40	40	40	30	20	20	10	10	33.0%
GPT-5.4 Nano	60	60	50	50	40	30	30	0	0	0	32.0%
Ministral 3 14B	50	50	40	40	40	20	20	20	20	10	31.0%
Ministral 3B	70	70	60	40	20	20	10	10	0	0	30.0%
Mistral Small 4	50	50	40	40	40	30	30	20	0	0	30.0%
Hermes 3 70B	60	50	50	30	30	20	20	10	0	0	27.0%
GPT-4.1 Nano	20	20	20	20	0	0	0	0	0	0	8.0%
Ministral 8B	50	10	10	0	0	0	0	0	0	0	7.0%
Rocinante 12B	40	20	10	0	0	0	0	0	0	0	7.0%
Gemma 3 4B	20	20	20	0	0	0	0	0	0	0	6.0%
Ministral 3 8B	10	0	0	0	0	0	0	0	0	0	1.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

10 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	92	99.2%
Aion 2.0	100	100	100	100	100	100	100	100	100	92	99.2%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	92	99.2%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	92	99.2%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	92	99.2%
GPT-5.2	100	100	100	100	100	100	100	100	100	92	99.2%
GPT-5.5	100	100	100	100	100	100	100	100	100	92	99.2%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
Grok 4	100	100	100	100	100	100	100	100	100	92	99.2%
Claude Opus 4	100	100	100	100	100	100	100	100	100	92	99.2%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	92	92	98.3%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	92	92	98.3%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	92	92	98.3%
Inception Mercury 2	100	100	100	100	100	100	100	100	92	92	98.3%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	75	97.5%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	92	92	92	97.5%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	75	97.5%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	92	83	97.5%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	75	97.5%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	92	92	92	92	96.7%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	83	83	96.7%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	83	83	96.7%
DeepSeek V4 Pro	100	100	100	100	100	100	100	100	92	75	96.7%
GPT-5.1	100	100	100	100	92	92	92	92	92	92	95.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	75	75	95.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	83	83	83	95.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	92	92	92	75	95.0%
GPT-5	100	100	100	100	92	92	92	92	92	92	95.0%
MiniMax M2.7	100	100	100	100	100	100	100	83	83	67	93.3%
MiniMax M2.5	100	100	100	100	100	100	100	83	75	75	93.3%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	33	93.3%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	92	42	93.3%
o4 Mini	100	100	100	100	100	100	92	92	75	75	93.3%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	75	75	75	92.5%
GPT-5 Mini	100	92	92	92	92	92	92	92	92	92	92.5%
Llama 3.1 Nemotron 70B	100	100	100	100	100	92	83	83	83	83	92.5%
o4 Mini High	100	100	100	100	100	92	92	92	75	75	92.5%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	83	83	83	67	91.7%
Qwen 3.6 35B	100	100	100	100	100	100	92	75	75	75	91.7%
Qwen 3.5 Flash	100	100	100	100	100	100	100	92	75	50	91.7%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	75	75	75	75	90.0%
Writer: Palmyra X5	100	100	100	100	100	100	75	75	75	75	90.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	75	67	50	89.2%
GPT-OSS 120B	100	100	100	100	100	92	92	75	75	58	89.2%
Z.AI GLM 4.5	100	100	100	100	100	92	75	75	75	75	89.2%
Gemini 2.5 Flash Lite (Reasoning)	100	100	92	92	92	83	83	83	83	83	89.2%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	75	75	75	75	75	87.5%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	75	75	75	75	75	87.5%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	83	75	75	42	87.5%
DeepSeek V3 (2025-03-24)	100	100	100	92	83	83	83	75	75	75	86.7%
Mistral Small Creative	100	100	100	92	92	83	83	83	75	58	86.7%
Z.AI GLM 4.6	100	100	100	100	100	100	92	92	75	8	86.7%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	92	75	58	42	86.7%
Stealth: Hunter Alpha	100	100	100	100	83	75	75	75	75	75	85.8%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	83	83	83	67	67	67	85.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	92	92	83	75	0	84.2%
Inception Mercury	100	100	100	100	83	75	75	75	75	58	84.2%
ByteDance Seed 1.6 Flash	100	92	83	83	83	83	83	83	83	67	84.2%
Gemini 3.1 Flash Lite	100	100	100	83	83	83	75	75	75	50	82.5%
GPT-4.1 Mini	92	92	92	92	92	75	75	75	75	67	82.5%
Qwen 3 32B	100	100	100	83	83	83	83	83	67	42	82.5%
Mistral Large	100	100	100	92	75	75	75	75	67	67	82.5%
GPT-4o, Aug. 6th (temp=1)	100	83	83	83	83	83	83	83	67	67	81.7%
Gemini 2.5 Flash	92	92	92	92	83	75	75	75	75	67	81.7%
GPT-4o, Aug. 6th (temp=0)	83	83	83	83	83	83	83	83	83	67	81.7%
GPT-5 Nano	100	100	83	75	75	75	75	75	75	75	80.8%
GPT-4.1	100	100	75	75	75	75	75	75	75	75	80.0%
DeepSeek V3.2	100	100	100	92	92	92	92	58	50	25	80.0%
Gemma 3 27B	100	100	92	75	75	75	75	75	75	50	79.2%
DeepSeek V4 Flash	100	92	92	92	75	75	75	75	58	58	79.2%
Claude Opus 4.7	92	92	75	75	75	75	75	75	75	75	78.3%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	83	83	83	58	58	58	42	76.7%
Ministral 3 8B	83	83	83	75	75	75	75	75	67	67	75.8%
Nemotron 3 Super	83	75	75	75	75	75	75	75	75	75	75.8%
Claude Sonnet 4.6	75	75	75	75	75	75	75	75	75	75	75.0%
Claude Haiku 4.5	75	75	75	75	75	75	75	75	75	75	75.0%
DeepSeek-V2 Chat	100	100	83	75	75	75	67	58	58	58	75.0%
Grok 4.20	92	92	92	75	75	75	75	75	50	42	74.2%
Llama 3.1 70B	83	83	83	83	83	83	83	83	75	0	74.2%
Mistral Large 3	92	75	75	75	75	75	67	67	67	67	73.3%
Mistral Small 3.2 24B	100	92	83	83	75	75	75	58	50	42	73.3%
Grok 4.20 (Beta)	92	83	75	75	75	75	75	67	50	50	71.7%
Z.AI GLM 4.7 Flash	100	83	67	67	67	67	67	67	67	58	70.8%
Mistral Large 2	100	75	75	75	75	67	67	67	50	50	70.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	92	0	0	0	69.2%
DeepSeek V3.1	92	92	83	75	75	75	67	58	42	33	69.2%
Hermes 3 405B	100	83	75	75	75	75	75	67	58	0	68.3%
Gemma 3 12B	92	75	75	75	75	75	67	67	50	33	68.3%
Ministral 8B	100	83	75	75	75	67	58	50	50	42	67.5%
DeepSeek V3 (2024-12-26)	92	83	75	75	75	67	58	58	42	42	66.7%
Nemotron 3 Nano	83	83	75	75	75	75	67	58	33	25	65.0%
WizardLM 2 8x22b	100	92	83	75	75	67	58	50	50	0	65.0%
Grok 4.3	100	92	83	75	75	67	58	50	50	0	65.0%
Gemini 2.5 Flash Lite	83	83	75	67	67	58	58	58	50	42	64.2%
GPT-4o, May 13th (temp=1)	100	83	83	83	83	67	67	17	17	17	61.7%
Arcee AI: Trinity Large (Preview)	75	75	67	58	50	50	50	50	50	42	56.7%
GPT-4o Mini (temp=1)	67	67	67	67	50	50	50	50	50	42	55.8%
Arcee AI: Trinity Mini	83	83	67	50	50	50	50	50	33	33	55.0%
Qwen 2.5 72B	83	67	67	67	67	67	50	25	25	25	54.2%
GPT-5.4 Mini	75	67	58	50	50	50	50	50	50	25	52.5%
Hermes 3 70B	75	75	67	67	67	50	50	42	25	0	51.7%
GPT-4o Mini (temp=0)	50	50	50	50	50	50	50	50	50	50	50.0%
Cohere Command R+ (Aug. 2024)	75	67	67	58	58	42	33	33	8	0	44.2%
Claude 3 Haiku	58	58	58	58	42	33	25	25	25	25	40.8%
Ministral 3 14B	50	50	50	50	42	42	33	33	25	17	39.2%
Mistral NeMO	67	67	42	42	33	33	33	25	25	17	38.3%
Mistral Small 4	75	50	50	42	33	17	8	0	0	0	27.5%
Ministral 3B	42	42	33	33	8	8	0	0	0	0	16.7%
GPT-5.4 Nano (Reasoning, Low)	83	33	25	17	0	0	0	0	0	0	15.8%
Ministral 3 3B	75	25	17	17	17	8	0	0	0	0	15.8%
Rocinante 12B	42	42	25	25	17	0	0	0	0	0	15.0%
Gemma 3 4B	25	25	17	8	8	0	0	0	0	0	8.3%
GPT-5.4 Nano	42	17	8	0	0	0	0	0	0	0	6.7%
GPT-4.1 Nano	17	0	0	0	0	0	0	0	0	0	1.7%
Llama 3.1 8B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

20 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	95	95	99.1%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	100	100	86	98.6%
Grok 4.20	100	100	100	100	100	100	100	100	91	91	98.2%
Gemini 2.5 Pro	100	100	100	95	95	95	95	95	95	95	96.8%
MoonshotAI: Kimi K2.6	100	100	100	100	100	95	95	95	86	86	95.9%
ByteDance Seed 2.0 Lite	100	100	100	100	95	95	95	95	91	86	95.9%
GPT-5.4	95	95	95	95	95	95	95	95	91	91	94.5%
Claude Sonnet 4.5	100	100	100	100	100	100	86	86	86	86	94.5%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	95	95	95	91	86	82	82	92.7%
Claude Sonnet 4.6 (Reasoning)	95	95	95	95	95	91	91	91	91	82	92.3%
GPT-5.5	95	91	91	91	91	91	91	91	91	91	91.4%
Xiaomi MIMO v2.5 Pro	100	100	95	95	91	91	91	86	82	82	91.4%
Gemini 3 Pro (Preview)	100	95	95	91	91	91	86	86	86	86	90.9%
Qwen 3.5 Flash	100	95	95	95	95	91	91	86	82	77	90.9%
GPT-5.2	95	95	95	95	91	91	86	86	86	86	90.9%
Grok 4	100	100	91	91	91	91	91	91	82	77	90.5%
Qwen 3.5 35B	100	95	95	95	91	91	91	86	86	73	90.5%
Z.AI GLM 5 Turbo	100	100	95	95	95	91	86	86	77	77	90.5%
GPT-5	91	91	91	91	91	91	91	91	91	86	90.5%
Gemma 4 31B (Reasoning)	100	95	95	95	91	86	86	86	86	82	90.5%
DeepSeek V4 Pro (Reasoning)	100	100	95	95	91	91	91	86	82	73	90.5%
Qwen3.7 Max	100	95	91	91	91	91	86	86	86	86	90.5%
Qwen 3.5 27B	100	100	100	86	86	86	86	86	86	82	90.0%
MoonshotAI: Kimi K2.5	95	95	95	91	91	86	86	86	86	82	89.5%
Qwen3.6 Max Preview	100	100	95	95	86	86	86	82	82	77	89.1%
GPT-5.5 (Reasoning, Low)	95	95	95	86	86	86	86	86	86	86	89.1%
DeepSeek V4 Flash (Reasoning)	100	100	95	95	95	86	86	82	77	73	89.1%
GPT-5.1	91	91	91	91	91	91	91	86	86	82	89.1%
GPT-5.4 (Reasoning, Low)	95	91	91	91	91	86	86	86	86	86	89.1%
Grok 4.20 (Reasoning)	100	91	91	91	91	91	91	82	82	82	89.1%
Claude 3.5 Sonnet	95	95	95	95	95	91	86	82	77	73	88.6%
Z.AI GLM 5.1	95	91	91	91	91	91	86	86	82	82	88.6%
Grok 4.20 (Beta, Reasoning)	100	91	91	91	91	91	91	82	82	77	88.6%
Aion 2.0	100	100	95	91	91	86	82	82	82	77	88.6%
ByteDance Seed 2.0 Mini	100	91	91	91	86	86	86	86	86	82	88.6%
Grok 4.3 (Reasoning)	100	100	91	91	91	91	82	82	82	68	87.7%
Gemini 2.5 Flash	100	86	86	86	86	86	86	86	86	86	87.7%
GPT-5.5 (Reasoning)	95	86	86	86	86	86	86	86	86	86	87.3%
Qwen 3.6 27B	100	91	91	91	91	86	82	82	82	77	87.3%
Gemini 2.5 Flash (Reasoning)	100	95	91	91	91	86	82	82	77	77	87.3%
GPT-5.4 (Reasoning)	91	91	91	86	86	86	86	86	82	82	86.8%
Mistral Medium 3.1	95	95	86	86	86	86	86	86	82	77	86.8%
Qwen 3.6 Flash	100	95	95	91	86	86	82	82	73	73	86.4%
Z.AI GLM 4.7	95	95	86	86	86	86	82	82	82	82	86.4%
Gemma 4 31B	86	86	86	86	86	86	86	86	86	86	86.4%
Gemini 3 Flash (Preview)	86	86	86	86	86	86	86	86	86	86	86.4%
Z.AI GLM 5	100	91	91	91	91	86	82	82	73	73	85.9%
Stealth: Hunter Alpha	100	91	91	91	86	86	82	82	77	73	85.9%
Gemini 3.5 Flash (Reasoning, Minimal)	86	86	86	86	86	86	86	86	86	82	85.9%
Gemini 3.1 Flash Lite (Reasoning)	95	95	91	82	82	82	82	82	82	82	85.5%
Gemini 3.5 Flash (Reasoning)	91	91	91	91	86	86	82	82	82	73	85.5%
Claude Sonnet 4	86	86	86	86	86	86	86	86	82	82	85.5%
Grok 4.1 Fast	91	91	91	91	91	86	82	82	73	73	85.0%
Claude Sonnet 4.6	100	100	91	91	86	82	82	77	68	68	84.5%
o4 Mini High	91	91	91	91	91	82	77	77	77	73	84.1%
Qwen 3.5 122B	91	91	91	86	86	82	82	82	77	68	83.6%
Gemini 3.1 Flash Lite	95	95	82	82	82	82	82	82	82	73	83.6%
Claude Opus 4	95	95	95	82	82	82	82	73	73	73	83.2%
Qwen 3.5 Plus (2026-04-20)	95	91	91	91	86	86	77	73	68	68	82.7%
GPT-4o, Aug. 6th (temp=0)	86	86	86	86	86	86	77	77	77	77	82.7%
Claude Opus 4.7	91	82	82	82	82	82	82	82	77	77	81.8%
Qwen 3.6 35B	100	95	95	91	86	77	73	68	68	64	81.8%
Stealth: Healer Alpha	91	91	91	91	82	82	77	73	73	68	81.8%
Gemini 3.1 Flash Lite (Preview)	82	82	82	82	82	82	82	82	82	82	81.8%
GPT-5.4 Mini (Reasoning)	86	86	86	86	86	86	82	77	68	68	81.4%
DeepSeek V4 Pro	95	95	86	82	82	82	82	77	68	64	81.4%
Gemma 4 26B (Reasoning)	91	91	86	86	82	82	77	77	77	64	81.4%
Xiaomi MIMO v2.5	95	95	91	86	82	77	77	77	68	64	81.4%
Mistral Large	95	95	95	86	82	82	77	68	64	64	80.9%
Grok 4 Fast	100	95	91	91	91	91	82	82	82	0	80.5%
o4 Mini	91	82	82	82	82	82	77	77	77	73	80.5%
Claude Opus 4.7 (Reasoning)	100	82	82	77	77	77	77	77	77	68	79.5%
Grok 4.3	91	86	86	86	82	82	82	77	64	59	79.5%
Qwen 3.5 Plus (2026-02-15)	91	91	86	82	82	77	77	77	77	50	79.1%
ByteDance Seed 1.6	86	86	77	77	77	77	77	77	77	77	79.1%
GPT-5.4 Mini (Reasoning, Low)	86	86	82	82	82	82	77	77	68	64	78.6%
GPT-4o, Aug. 6th (temp=1)	91	91	91	91	86	77	73	68	59	59	78.6%
DeepSeek V4 Flash	100	91	91	91	86	77	73	73	50	50	78.2%
DeepSeek V3.2	100	91	91	91	77	73	73	68	64	55	78.2%
Mistral Large 2	86	86	86	82	82	82	77	73	64	59	77.7%
GPT-5 Mini	86	82	77	77	77	77	77	77	73	68	77.3%
Z.AI GLM 4.6	100	95	86	86	82	82	73	59	59	45	76.8%
Qwen 3.5 397B A17B	95	86	86	86	86	86	82	82	73	0	76.4%
Gemini 2.5 Flash Lite (Reasoning)	86	82	82	82	82	73	73	68	68	64	75.9%
GPT-5 Nano	91	82	82	82	77	77	73	68	64	55	75.0%
MiniMax M2.7	91	82	82	82	77	77	73	68	68	50	75.0%
Qwen 3.5 9B	95	91	91	86	77	77	77	68	68	9	74.1%
GPT-5.4 Mini	91	91	86	82	73	73	68	64	64	45	73.6%
Claude Haiku 4.5	91	91	91	82	82	77	73	68	68	0	72.3%
Gemma 4 26B	82	73	73	73	73	73	73	68	64	64	71.4%
Z.AI GLM 4.7 Flash	91	77	73	73	73	68	68	68	64	59	71.4%
Z.AI GLM 4.5 Air	91	91	73	68	68	68	68	68	59	59	71.4%
Writer: Palmyra X5	86	82	82	77	68	68	68	68	59	55	71.4%
Inception Mercury	95	82	82	77	73	73	68	64	55	45	71.4%
Mistral Large 3	91	77	77	77	68	64	64	64	64	64	70.9%
Inception Mercury 2	77	77	77	77	73	73	68	68	59	59	70.9%
Stealth: Aurora Alpha	73	73	73	73	73	73	73	68	68	64	70.9%
DeepSeek V3.1	95	95	86	82	82	77	64	45	41	41	70.9%
MiniMax M2.5	82	82	77	73	68	68	64	64	64	59	70.0%
Claude 3.7 Sonnet	100	100	86	86	77	64	59	50	45	32	70.0%
GPT-OSS 120B	82	77	77	73	73	68	68	59	59	55	69.1%
Z.AI GLM 4.5	91	73	68	68	68	68	64	64	59	59	68.2%
Qwen3 235B A22B Instruct 2507	91	86	86	82	77	68	59	59	50	23	68.2%
ByteDance Seed 1.6 Flash	82	73	73	73	73	73	64	59	59	45	67.3%
GPT-4o, May 13th (temp=0)	77	77	77	73	68	68	59	59	59	50	66.8%
Nemotron 3 Nano	82	82	77	73	73	59	59	59	59	45	66.8%
GPT-4.1	95	82	82	82	82	82	77	64	0	0	64.5%
Gemini 2.5 Flash Lite	100	95	91	77	73	73	64	55	0	0	62.7%
Mistral Small 4 (Reasoning)	82	73	68	68	64	59	59	59	50	45	62.7%
DeepSeek V3 (2024-12-26)	82	73	73	68	68	68	59	45	45	36	61.8%
Ministral 3 14B	82	82	77	68	68	45	45	45	41	41	59.5%
GPT-4.1 Mini	77	68	68	68	59	59	55	41	41	36	57.3%
Arcee AI: Trinity Large (Preview)	77	73	73	64	55	55	50	50	45	27	56.8%
Nemotron 3 Super	77	77	73	73	73	68	68	59	0	0	56.8%
Qwen 3 32B	77	73	73	68	64	55	50	50	45	9	56.4%
GPT-4o, May 13th (temp=1)	73	68	59	59	59	55	50	45	45	41	55.5%
Mistral Small Creative	64	64	64	59	55	55	50	50	50	41	55.0%
DeepSeek-V2 Chat	73	68	68	68	59	55	50	45	41	0	52.7%
Mistral Small 3.2 24B	64	59	59	59	55	50	50	45	45	41	52.7%
DeepSeek V3 (2025-03-24)	68	68	59	59	59	55	45	45	45	9	51.4%
Hermes 3 405B	82	73	68	64	59	50	36	27	23	23	50.5%
Gemma 3 27B	59	59	55	45	45	41	41	41	41	41	46.8%
Llama 3.1 Nemotron 70B	59	59	59	45	41	41	36	36	36	27	44.1%
GPT-5.4 Nano (Reasoning)	82	73	73	73	73	64	0	0	0	0	43.6%
Llama 3.1 70B	59	59	55	55	45	45	45	27	27	9	42.7%
Cohere Command R+ (Aug. 2024)	64	59	55	55	45	45	41	27	23	14	42.7%
Ministral 3 8B	73	64	45	41	41	32	32	23	18	14	38.2%
Ministral 8B	68	64	55	50	41	41	32	23	0	0	37.3%
Mistral Small 4	50	50	45	45	45	32	32	27	23	18	36.8%
WizardLM 2 8x22b	59	45	41	41	41	41	36	27	23	9	36.4%
GPT-5.4 Nano (Reasoning, Low)	77	73	73	64	50	0	0	0	0	0	33.6%
Arcee AI: Trinity Mini	45	45	45	45	41	36	36	18	14	9	33.6%
Hermes 3 70B	50	41	36	36	32	23	18	0	0	0	23.6%
GPT-5.4 Nano	64	45	45	45	14	0	0	0	0	0	21.4%
GPT-4o Mini (temp=0)	32	23	23	18	18	18	18	18	18	18	20.5%
GPT-4o Mini (temp=1)	50	32	32	23	23	18	14	5	0	0	19.5%
Gemma 3 12B	32	27	27	27	23	14	14	9	5	0	17.7%
Ministral 3B	32	32	27	23	23	14	14	9	5	0	17.7%
Claude 3 Haiku	41	27	23	18	14	14	9	9	5	0	15.9%
Ministral 3 3B	32	23	18	18	18	14	5	0	0	0	12.7%
Mistral NeMO	27	27	27	14	9	5	5	0	0	0	11.4%
Llama 3.1 8B	41	32	27	14	0	0	0	0	0	0	11.4%
Qwen 2.5 72B	73	9	9	0	0	0	0	0	0	0	9.1%
Gemma 3 4B	5	0	0	0	0	0	0	0	0	0	0.5%
Rocinante 12B	5	0	0	0	0	0	0	0	0	0	0.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

40 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	95	99.5%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	95	99.5%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	90	99.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	95	95	99.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5.4	100	100	100	100	100	100	100	100	95	95	99.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	85	98.5%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	85	98.5%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	85	98.5%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	85	98.5%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	85	98.5%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	85	98.5%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	90	90	98.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	95	85	98.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	95	90	90	97.5%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	90	85	97.5%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	95	85	85	96.5%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	95	95	90	85	96.5%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	90	90	90	90	96.0%
o4 Mini High	100	100	100	100	100	95	95	90	90	90	96.0%
GPT-5.2	100	100	100	95	95	95	95	95	95	90	96.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	85	85	85	95.5%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	85	85	85	95.5%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	90	90	90	85	95.5%
Qwen 3.5 Flash	100	100	100	100	100	100	100	85	85	85	95.5%
GPT-5	100	100	100	100	100	100	95	90	85	85	95.5%
Claude Opus 4.5	100	100	100	100	100	90	90	90	90	90	95.0%
Aion 2.0	100	100	100	100	100	90	90	90	90	90	95.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	90	85	75	95.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	90	80	80	95.0%
o4 Mini	100	100	100	100	100	90	90	90	90	85	94.5%
Qwen 3.5 122B	100	100	100	100	100	100	90	85	85	85	94.5%
Xiaomi MIMO v2.5	100	100	100	100	100	100	90	85	85	85	94.5%
Z.AI GLM 5	100	100	100	100	100	90	90	90	85	85	94.0%
GPT-5.1	100	100	100	100	95	90	90	90	90	80	93.5%
ByteDance Seed 1.6	100	100	100	100	100	100	90	85	85	75	93.5%
Claude Opus 4.7	100	100	100	100	100	100	85	85	85	80	93.5%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	85	85	85	85	85	92.5%
Stealth: Hunter Alpha	100	100	100	100	100	90	85	85	85	80	92.5%
Gemini 2.5 Flash	100	100	95	95	90	90	90	90	90	80	92.0%
Nemotron 3 Super	100	100	90	90	90	90	90	90	90	80	91.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	90	90	90	90	90	80	70	90.0%
ByteDance Seed 2.0 Lite	90	90	90	90	90	90	90	90	90	90	90.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	95	0	89.5%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	95	95	95	0	88.5%
Inception Mercury 2	100	95	90	90	90	90	90	85	75	75	88.0%
Mistral Large 2	100	90	90	90	90	90	85	80	80	80	87.5%
Claude Sonnet 4.6	90	90	90	90	90	90	90	90	75	75	87.0%
Claude Sonnet 4	90	90	90	90	90	90	90	80	80	80	87.0%
Claude 3.7 Sonnet	90	90	90	90	90	90	90	90	75	75	87.0%
GPT-5.4 Mini (Reasoning, Low)	90	90	90	90	90	90	90	85	75	75	86.5%
Qwen 3.6 27B	100	100	100	100	100	100	100	90	60	15	86.5%
Z.AI GLM 4.5 Air	100	100	90	90	85	85	85	80	80	70	86.5%
DeepSeek V4 Pro	100	95	85	85	85	85	85	85	85	75	86.5%
Qwen 3.5 9B	100	100	100	100	90	90	90	90	85	10	85.5%
Mistral Large 3	100	90	90	90	90	85	80	75	75	75	85.0%
Qwen 3.5 Plus (2026-02-15)	100	100	85	85	85	85	85	80	70	70	84.5%
Claude Opus 4	100	100	90	85	85	85	85	75	70	65	84.0%
GPT-OSS 120B	100	100	90	90	90	80	80	75	75	60	84.0%
Mistral Small 4 (Reasoning)	100	90	90	90	90	80	80	80	80	45	82.5%
Gemini 3 Flash (Preview)	100	85	85	85	85	85	85	70	70	70	82.0%
Mistral Large	90	90	90	90	90	90	80	65	65	65	81.5%
Stealth: Aurora Alpha	90	90	90	90	90	85	80	75	75	50	81.5%
Z.AI GLM 4.6	100	100	100	100	90	85	85	75	55	15	80.5%
MiniMax M2.7	100	90	85	80	80	80	80	75	70	65	80.5%
Gemini 3.1 Flash Lite	85	85	85	85	85	85	80	70	70	70	80.0%
GPT-4o, May 13th (temp=0)	80	80	80	80	80	80	80	80	70	70	78.0%
Gemini 3.1 Flash Lite (Reasoning)	95	85	85	85	85	75	70	70	65	60	77.5%
DeepSeek V3.2	95	85	85	80	80	75	70	65	60	60	75.5%
Gemma 4 26B	85	85	85	85	85	85	85	85	70	0	75.0%
MiniMax M2.5	90	90	80	80	80	70	70	65	60	55	74.0%
Gemini 3.1 Flash Lite (Preview)	90	90	75	75	75	75	75	70	65	50	74.0%
Claude 3.5 Sonnet	90	80	75	75	75	75	75	65	65	65	74.0%
Grok 4.20	95	90	85	85	80	70	65	60	60	45	73.5%
DeepSeek-V2 Chat	90	90	85	85	85	80	70	55	40	40	72.0%
Z.AI GLM 4.7 Flash	90	90	85	85	80	70	70	65	45	40	72.0%
Ministral 3 14B	85	80	80	75	75	70	70	70	55	45	70.5%
Claude Haiku 4.5	90	90	85	75	75	75	75	70	60	0	69.5%
GPT-5 Nano	95	90	80	65	65	65	65	60	55	50	69.0%
GPT-4o, Aug. 6th (temp=1)	80	80	80	80	75	70	65	55	55	40	68.0%
DeepSeek V3 (2024-12-26)	90	85	85	70	65	65	60	55	55	45	67.5%
Nemotron 3 Nano	90	80	75	75	70	65	60	60	50	45	67.0%
GPT-4o, Aug. 6th (temp=0)	80	80	70	70	70	65	65	65	55	40	66.0%
Qwen3 235B A22B Instruct 2507	85	80	75	70	70	65	60	50	50	50	65.5%
Grok 4.20 (Beta)	80	80	80	75	65	60	55	55	50	50	65.0%
GPT-4.1	100	75	70	70	65	65	65	60	50	30	65.0%
Mistral Medium 3.1	90	85	75	70	70	65	55	55	45	40	65.0%
DeepSeek V3.1	90	85	75	70	70	65	65	60	35	30	64.5%
ByteDance Seed 1.6 Flash	80	80	80	70	70	65	60	60	45	35	64.5%
DeepSeek V3 (2025-03-24)	85	80	75	70	65	60	55	55	50	45	64.0%
DeepSeek V4 Flash	80	80	75	70	70	65	65	65	55	5	63.0%
Writer: Palmyra X5	75	75	70	65	65	65	60	60	50	40	62.5%
GPT-4o, May 13th (temp=1)	75	70	70	65	60	55	55	55	50	45	60.0%
GPT-4.1 Mini	70	65	65	65	60	60	55	55	50	45	59.0%
GPT-5.4 Mini	95	95	95	90	80	70	30	5	5	5	57.0%
Gemma 3 27B	65	60	60	60	55	55	55	55	55	50	57.0%
Z.AI GLM 4.5	80	75	75	75	60	55	50	45	15	10	54.0%
Inception Mercury	85	65	65	60	55	55	50	45	40	20	54.0%
Grok 4.3	75	70	70	60	55	55	55	50	45	0	53.5%
Mistral Small Creative	70	70	65	65	65	65	50	25	15	15	50.5%
Qwen 3 32B	90	70	60	60	60	55	50	25	20	0	49.0%
Ministral 3 8B	70	70	65	65	65	60	35	25	25	0	48.0%
Hermes 3 405B	55	50	50	50	50	50	45	40	40	35	46.5%
GPT-5.4 Nano (Reasoning, Low)	85	85	80	75	75	25	0	0	0	0	42.5%
Mistral Small 3.2 24B	55	50	50	50	50	35	35	35	35	0	39.5%
Ministral 3 3B	55	50	50	50	50	45	35	25	20	10	39.0%
GPT-5.4 Nano (Reasoning)	90	90	90	70	40	0	0	0	0	0	38.0%
Mistral Small 4	60	60	50	45	35	35	30	25	20	0	36.0%
Llama 3.1 Nemotron 70B	65	50	50	50	35	35	30	15	10	0	34.0%
Qwen 2.5 72B	50	45	45	40	35	35	35	25	15	10	33.5%
WizardLM 2 8x22b	50	50	45	35	30	30	30	15	15	10	31.0%
Llama 3.1 70B	60	50	40	40	35	30	20	15	10	0	30.0%
Arcee AI: Trinity Large (Preview)	65	60	50	50	35	30	0	0	0	0	29.0%
Ministral 3B	50	40	35	30	20	20	20	15	15	15	26.0%
Ministral 8B	55	55	45	40	25	0	0	0	0	0	22.0%
Hermes 3 70B	40	40	30	25	20	15	15	10	10	5	21.0%
GPT-5.4 Nano	70	55	35	20	20	5	0	0	0	0	20.5%
Gemini 2.5 Flash Lite	45	40	30	25	25	20	10	0	0	0	19.5%
Cohere Command R+ (Aug. 2024)	35	35	20	20	20	5	0	0	0	0	13.5%
Arcee AI: Trinity Mini	30	25	20	15	15	10	10	0	0	0	12.5%
GPT-4o Mini (temp=0)	30	25	20	20	5	5	5	5	5	0	12.0%
GPT-4o Mini (temp=1)	20	15	15	5	5	5	5	0	0	0	7.0%
Llama 3.1 8B	30	20	0	0	0	0	0	0	0	0	5.0%
Mistral NeMO	35	5	0	0	0	0	0	0	0	0	4.0%
Gemma 3 12B	15	5	5	5	5	0	0	0	0	0	3.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

Accuracy (recall)

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

matrix

Large codex (40 entries), long passage (1,019 words)

Large codex (40 entries), short passage (165 words)

Small codex (7 entries), long passage (734 words)

Small codex (7 entries), short passage (165 words)

tiers

5 codex entries

10 codex entries

20 codex entries

40 codex entries