Accuracy (recall)

Test: Codex Violation Detection

Avg. Score

73.3%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Gemini 2.5 Flash (Reasoning)	94.0%	$0.0079	12.5s	86%
2	Grok 4.1 Fast	93.8%	$0.0021	21.1s	84%
3	GPT-5.4	95.9%	$0.013	8.8s	85%
4	Z.AI GLM 5 Turbo	94.8%	$0.0072	17.1s	85%
5	Gemini 3 Flash (Preview, Reasoning)	94.7%	$0.011	18.0s	86%
6	DeepSeek V4 Flash (Reasoning)	94.4%	$0.0010	40.3s	82%
7	GPT-5.5	96.7%	$0.030	7.7s	90%
8	Grok 4 Fast	92.4%	$0.0018	12.8s	72%
9	Grok 4.20 (Beta, Reasoning)	96.1%	$0.026	16.8s	89%
10	Gemma 4 31B	92.5%	$0.0009	37.9s	78%
11	Grok 4.20 (Reasoning)	95.8%	$0.015	45.4s	90%
12	GPT-5.2	95.7%	$0.024	24.2s	87%
13	Gemini 3 Flash (Preview)	88.8%	$0.0031	4.5s	69%
14	Claude Opus 4.5	98.0%	$0.041	9.7s	91%
15	Gemini 2.5 Pro	97.4%	$0.035	23.2s	91%
16	GPT-5.4 (Reasoning, Low)	92.4%	$0.020	13.5s	79%
17	Claude Sonnet 4.5	93.6%	$0.024	8.9s	80%
18	Xiaomi MIMO v2.5 Pro	92.5%	$0.0091	37.0s	80%
19	Gemini 2.5 Flash	82.5%	$0.0025	2.8s	69%
20	Claude Opus 4.6	97.3%	$0.040	10.2s	88%
21	Stealth: Healer Alpha	89.0%	$0.0000	21.8s	67%
22	GPT-5.5 (Reasoning, Low)	95.5%	$0.040	14.2s	88%
23	Qwen 3.6 Flash	92.4%	$0.011	29.9s	72%
24	Stealth: Hunter Alpha	89.5%	$0.0000	44.0s	71%
25	Qwen 3.5 Flash	92.8%	$0.0038	1.0m	77%
26	Xiaomi MIMO v2.5	89.2%	$0.0058	21.8s	65%
27	Inception Mercury 2	82.2%	$0.0030	4.6s	62%
28	Claude Sonnet 4	90.3%	$0.023	9.0s	72%
29	Qwen 3.6 35B	92.1%	$0.013	51.0s	78%
30	Qwen 3.5 35B	93.3%	$0.017	54.8s	81%
31	ByteDance Seed 1.6	91.9%	$0.0067	1.0m	77%
32	Z.AI GLM 5	92.1%	$0.013	56.4s	79%
33	Gemini 3.1 Flash Lite	81.1%	$0.0018	3.2s	59%
34	GPT-5 Mini	91.2%	$0.0092	51.7s	74%
35	GPT-5.4 Mini (Reasoning, Low)	83.0%	$0.0055	6.7s	61%
36	Gemini 3.1 Flash Lite (Preview)	81.9%	$0.0019	2.2s	57%
37	o4 Mini	88.9%	$0.019	28.1s	74%
38	Z.AI GLM 4.7	91.3%	$0.0091	1.0m	77%
39	Gemini 2.5 Flash Lite (Reasoning)	81.1%	$0.0020	17.6s	62%
40	Z.AI GLM 5.1	95.5%	$0.017	1.6m	88%
41	ByteDance Seed 2.0 Lite	89.8%	$0.0067	1.1m	73%
42	Qwen 3.5 Plus (2026-04-20)	93.4%	$0.015	1.4m	83%
43	Claude Sonnet 4.6	82.9%	$0.024	9.9s	70%
44	Gemini 3.1 Flash Lite (Reasoning)	79.3%	$0.0018	3.7s	52%
45	Grok 4.3 (Reasoning)	94.3%	$0.019	1.4m	81%
46	GPT-5.4 Mini (Reasoning)	87.9%	$0.020	28.6s	68%
47	Qwen 3.5 27B	95.6%	$0.021	1.7m	89%
48	DeepSeek V4 Flash	78.1%	$0.0003	9.3s	53%
49	Mistral Large 3	75.5%	$0.0030	10.2s	58%
50	Qwen 3.5 122B	92.7%	$0.026	1.2m	81%
51	Mistral Large	79.3%	$0.012	9.1s	59%
52	MiniMax M2.7	79.1%	$0.0022	29.4s	59%
53	MoonshotAI: Kimi K2.5	92.9%	$0.015	1.5m	79%
54	Qwen 3.5 Plus (2026-02-15)	85.0%	$0.0041	34.2s	55%
55	DeepSeek V4 Pro	81.7%	$0.0031	25.6s	53%
56	GPT-5.1	92.7%	$0.037	52.8s	79%
57	Mistral Large 2	75.9%	$0.012	8.8s	57%
58	o4 Mini High	90.7%	$0.033	51.3s	76%
59	Grok 4.20	77.0%	$0.0053	6.4s	50%
60	Aion 2.0	91.7%	$0.0084	1.3m	64%
61	Claude Opus 4.7	88.3%	$0.052	7.6s	76%
62	Claude Opus 4.7 (Reasoning)	92.8%	$0.067	11.4s	82%
63	Mistral Medium 3.1	75.1%	$0.0029	7.7s	46%
64	Grok 4.20 (Beta)	74.8%	$0.0059	2.5s	47%
65	Gemini 3.1 Pro (Preview)	98.3%	$0.068	52.3s	93%
66	GPT-5.4 (Reasoning)	91.7%	$0.042	43.1s	75%
67	MiniMax M2.5	74.6%	$0.0020	25.5s	52%
68	Claude Opus 4.6 (Reasoning)	97.3%	$0.076	32.0s	90%
69	GPT-5.5 (Reasoning)	95.6%	$0.071	30.3s	88%
70	DeepSeek V3.2	75.0%	$0.0013	17.2s	45%
71	Gemma 4 31B (Reasoning)	94.4%	$0.0017	2.8m	85%
72	Z.AI GLM 4.5	75.6%	$0.0026	18.5s	45%
73	GPT-4.1	77.7%	$0.0081	10.0s	44%
74	Z.AI GLM 4.6	86.1%	$0.0049	1.3m	59%
75	Mistral Small 4 (Reasoning)	70.9%	$0.0020	16.3s	47%
76	ByteDance Seed 1.6 Flash	67.9%	$0.0009	12.0s	46%
77	Gemma 4 26B (Reasoning)	90.0%	$0.0022	2.0m	65%
78	Gemini 3 Pro (Preview)	91.0%	$0.050	34.0s	70%
79	Z.AI GLM 4.5 Air	77.1%	$0.0025	39.8s	46%
80	DeepSeek-V2 Chat	71.1%	$0.0020	13.1s	41%
81	Grok 4	94.1%	$0.051	1.2m	80%
82	GPT-4.1 Mini	63.7%	$0.0015	5.8s	43%
83	Claude Haiku 4.5	66.8%	$0.0078	5.8s	45%
84	DeepSeek V3 (2024-12-26)	69.9%	$0.0019	15.6s	41%
85	GPT-OSS 120B	81.0%	$0.0020	1.5m	59%
86	Qwen 3.6 27B	89.5%	$0.022	1.4m	63%
87	Qwen3 235B A22B Instruct 2507	67.3%	$0.0007	21.1s	43%
88	Writer: Palmyra X5	67.2%	$0.0062	12.1s	44%
89	Gemma 4 26B	75.0%	$0.0006	25.0s	35%
90	Stealth: Aurora Alpha	77.5%	—	6.2s	46%
91	ByteDance Seed 2.0 Mini	90.8%	$0.0029	2.7m	73%
92	Claude Sonnet 4.6 (Reasoning)	94.9%	$0.076	51.3s	84%
93	Claude 3.7 Sonnet	77.4%	$0.023	10.2s	42%
94	Grok 4.3	64.3%	$0.0061	6.1s	39%
95	Inception Mercury	63.1%	$0.0005	9.5s	34%
96	DeepSeek V3 (2025-03-24)	68.9%	$0.0015	22.9s	34%
97	GPT-4o, Aug. 6th (temp=1)	65.4%	$0.011	3.6s	36%
98	Mistral Small Creative	56.8%	$0.0006	4.3s	36%
99	GPT-4o, Aug. 6th (temp=0)	67.9%	$0.015	5.0s	36%
100	Claude 3.5 Sonnet	79.2%	$0.042	10.5s	47%
101	Qwen3.6 Max Preview	96.4%	$0.050	2.5m	90%
102	GPT-5	93.3%	$0.061	1.6m	81%
103	Z.AI GLM 4.7 Flash	68.2%	$0.0018	1.1m	45%
104	DeepSeek V3.1	66.1%	$0.0014	31.1s	33%
105	Nemotron 3 Super	83.1%	$0.0000	2.3m	55%
106	Qwen 3 32B	64.1%	$0.0010	27.8s	31%
107	Qwen 3.5 9B	84.7%	$0.0020	2.4m	55%
108	Mistral Small 3.2 24B	53.8%	$0.0006	10.4s	32%
109	GPT-5.4 Nano (Reasoning)	67.5%	$0.0035	16.6s	23%
110	GPT-4o, May 13th (temp=0)	71.6%	$0.030	5.4s	36%
111	Gemma 3 27B	53.2%	$0.0005	13.3s	33%
112	Qwen 3.5 397B A17B	93.7%	$0.026	2.9m	75%
113	GPT-5.4 Mini	54.5%	$0.0033	3.1s	29%
114	DeepSeek V4 Pro (Reasoning)	91.5%	$0.014	2.8m	60%
115	Gemini 2.5 Flash Lite	47.9%	$0.0005	2.3s	24%
116	GPT-4o, May 13th (temp=1)	58.1%	$0.026	3.7s	34%
117	Ministral 3 14B	46.4%	$0.0010	6.3s	25%
118	Hermes 3 405B	56.2%	$0.0044	17.2s	20%
119	Llama 3.1 Nemotron 70B	55.8%	$0.0055	18.5s	19%
120	GPT-5 Nano	67.2%	$0.0049	1.9m	42%
121	Ministral 3 8B	39.9%	$0.0007	4.8s	20%
122	Llama 3.1 70B	51.2%	$0.0021	24.3s	18%
123	Qwen 2.5 72B	42.3%	$0.0008	13.9s	18%
124	Ministral 8B	34.7%	$0.0005	5.6s	18%
125	Arcee AI: Trinity Mini	33.3%	$0.0004	10.3s	18%
126	Claude Opus 4	84.4%	$0.116	15.8s	65%
127	Mistral Small 4	30.7%	$0.0008	4.3s	18%
128	Ministral 3 3B	28.6%	$0.0005	3.3s	16%
129	Gemma 3 12B	35.9%	$0.0003	12.0s	11%
130	MoonshotAI: Kimi K2.6	94.7%	$0.038	4.1m	75%
131	GPT-4o Mini (temp=0)	31.3%	$0.0006	25.0s	19%
132	GPT-4o Mini (temp=1)	29.4%	$0.0006	7.4s	13%
133	Ministral 3B	25.0%	$0.0002	2.9s	14%
134	Cohere Command R+ (Aug. 2024)	32.3%	$0.014	10.0s	15%
135	GPT-5.4 Nano (Reasoning, Low)	32.9%	$0.0013	6.1s	2%
136	Hermes 3 70B	28.5%	$0.0013	29.2s	14%
137	Claude 3 Haiku	19.4%	$0.0015	3.6s	12%
138	GPT-5.4 Nano	21.3%	$0.0008	2.9s	4%
139	Mistral NeMO	20.6%	$0.0007	13.3s	7%
140	Nemotron 3 Nano	64.5%	$0.0031	3.5m	37%
141	Llama 3.1 8B	16.1%	$0.0002	16.1s	0%
142	WizardLM 2 8x22b	17.1%	$0.0036	15.7s	0%
143	Rocinante 12B	6.4%	$0.0009	6.0s	0%
144	GPT-4.1 Nano	2.6%	$0.0004	3.9s	0%
145	Gemma 3 4B	3.2%	$0.0002	12.5s	0%
146	Arcee AI: Trinity Large (Preview)	46.2%	$0.0000	3.1m	24%
147	LFM2 24B	0.0%	$0.0008	1.9m	0%
73.28%

Individual Scenarios

matrix

▼

Large codex (40 entries), long passage (1,019 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Grok 4	100	98	98	97	97	95	95	94	94	91	96.1%
Qwen3.6 Max Preview	97	97	97	94	94	94	94	94	91	91	94.2%
Claude Sonnet 4.6 (Reasoning)	97	95	95	94	94	94	94	92	92	88	93.6%
GPT-5.5	97	97	95	94	94	94	92	92	89	89	93.5%
Grok 4.20 (Reasoning)	100	97	97	94	94	92	92	91	91	86	93.5%
Qwen 3.5 397B A17B	100	100	97	95	94	94	89	89	88	85	93.2%
GPT-5.5 (Reasoning)	97	97	94	94	94	92	92	89	89	89	92.9%
GPT-5.5 (Reasoning, Low)	95	95	95	95	92	92	92	92	91	83	92.6%
GPT-5.4	97	94	94	92	92	92	91	91	91	91	92.6%
Gemini 2.5 Pro	100	95	92	92	92	89	88	88	88	86	91.2%
MoonshotAI: Kimi K2.6	100	98	95	94	94	89	88	88	86	79	91.2%
GPT-5.2	97	97	94	92	92	91	89	88	86	85	91.2%
DeepSeek V4 Flash (Reasoning)	95	95	94	92	92	92	91	89	85	82	90.9%
Qwen 3.5 27B	94	92	92	92	91	91	91	91	88	86	90.9%
Grok 4 Fast	100	100	97	92	91	89	86	86	83	82	90.8%
GPT-5	94	94	92	92	91	91	88	88	88	88	90.6%
Grok 4.1 Fast	100	97	94	91	91	91	89	89	82	82	90.6%
Grok 4.20 (Beta, Reasoning)	97	97	94	92	89	89	88	86	86	82	90.2%
Claude Opus 4.6 (Reasoning)	94	92	91	91	91	91	91	88	86	85	90.0%
Gemini 3.1 Pro (Preview)	94	94	91	91	91	89	88	88	88	86	90.0%
GPT-5.4 (Reasoning, Low)	94	94	92	92	91	89	88	88	86	83	89.8%
Z.AI GLM 5.1	95	95	92	91	91	91	89	85	85	82	89.7%
GPT-5.4 (Reasoning)	94	94	92	91	91	91	86	85	85	82	89.1%
Claude Opus 4.5	92	92	91	91	89	89	88	88	85	83	88.9%
Grok 4.3 (Reasoning)	100	91	88	88	88	88	88	88	85	80	88.3%
Claude Opus 4.7 (Reasoning)	91	91	91	89	88	88	88	86	85	83	88.0%
Z.AI GLM 5 Turbo	92	92	91	89	89	86	86	83	83	83	87.7%
Nemotron 3 Super	94	91	89	89	89	88	86	83	83	80	87.4%
Claude Opus 4.7	92	91	91	89	89	86	85	83	83	80	87.1%
GPT-5 Mini	91	91	88	88	86	86	85	85	85	85	87.0%
Gemini 2.5 Flash (Reasoning)	91	89	89	89	88	85	85	85	85	77	86.4%
ByteDance Seed 1.6	94	91	91	89	88	85	83	82	80	79	86.2%
Z.AI GLM 4.6	94	92	92	88	88	86	83	82	77	77	86.1%
Qwen 3.6 35B	97	97	91	91	85	85	82	80	77	74	85.9%
Qwen 3.5 Plus (2026-04-20)	97	91	89	89	88	85	85	82	77	74	85.8%
o4 Mini High	94	91	91	88	85	83	82	82	82	79	85.6%
GPT-5.1	94	91	88	85	85	85	82	82	80	80	85.2%
Qwen 3.6 Flash	89	88	88	88	88	85	85	80	79	76	84.5%
Gemma 4 31B (Reasoning)	92	91	89	89	88	88	83	79	74	68	84.2%
Qwen 3.5 122B	89	88	86	86	83	82	82	80	79	79	83.5%
Z.AI GLM 5	97	85	85	83	82	82	82	79	79	77	83.0%
Gemini 3 Flash (Preview, Reasoning)	89	88	85	83	83	82	82	82	82	74	83.0%
Qwen 3.5 35B	92	91	89	85	85	83	79	79	76	71	83.0%
Claude Opus 4.6	85	85	85	85	83	83	80	80	80	80	82.7%
Aion 2.0	100	97	94	92	91	89	88	88	88	0	82.7%
Xiaomi MIMO v2.5 Pro	92	88	86	85	83	83	80	79	79	70	82.6%
Qwen 3.6 27B	92	91	85	85	85	83	77	76	76	73	82.3%
Z.AI GLM 4.7	94	88	88	83	83	82	79	79	74	73	82.3%
Gemma 4 31B	86	86	85	83	82	82	80	80	79	79	82.3%
Claude Opus 4	92	89	85	83	82	82	79	77	77	70	81.7%
Qwen 3.5 Flash	94	91	86	85	82	82	79	77	76	62	81.4%
MoonshotAI: Kimi K2.5	91	86	86	86	82	82	80	80	73	65	81.2%
Gemma 4 26B (Reasoning)	88	85	80	80	79	79	77	76	74	74	79.2%
Claude Sonnet 4.6	86	82	82	82	80	77	77	76	76	74	79.2%
Claude Sonnet 4	83	82	80	80	80	79	79	77	73	71	78.5%
Gemini 3 Flash (Preview)	85	82	82	80	79	76	76	76	74	74	78.3%
Claude Sonnet 4.5	85	83	82	80	79	77	77	77	71	70	78.2%
o4 Mini	88	86	82	82	79	76	76	73	70	67	77.7%
Stealth: Hunter Alpha	88	83	83	82	80	77	76	73	68	59	77.0%
GPT-5.4 Mini (Reasoning)	83	82	82	80	80	74	73	73	71	70	76.8%
ByteDance Seed 2.0 Lite	85	85	83	80	79	77	76	68	65	62	76.1%
Stealth: Healer Alpha	94	91	91	82	79	76	74	62	58	42	74.8%
ByteDance Seed 2.0 Mini	83	82	76	74	74	74	74	73	73	64	74.7%
Qwen 3.5 9B	85	82	79	77	77	76	73	71	62	62	74.4%
Xiaomi MIMO v2.5	89	85	85	82	79	79	76	70	70	0	71.4%
DeepSeek V4 Pro (Reasoning)	97	95	94	92	88	85	82	76	0	0	70.9%
Gemini 2.5 Flash	79	77	76	76	74	73	70	64	61	59	70.8%
Gemini 3 Pro (Preview)	83	82	80	76	76	76	74	73	67	15	70.2%
Inception Mercury 2	79	77	74	71	71	70	70	70	61	58	70.0%
Gemini 2.5 Flash Lite (Reasoning)	79	77	76	73	73	68	67	67	62	55	69.5%
GPT-5.4 Mini (Reasoning, Low)	77	77	71	70	70	68	67	64	62	59	68.5%
DeepSeek V4 Pro	77	76	71	71	68	68	67	64	56	53	67.1%
GPT-OSS 120B	76	76	73	70	67	67	62	61	56	53	65.9%
Qwen 3.5 Plus (2026-02-15)	92	85	83	82	79	77	76	39	38	0	65.2%
Mistral Large	79	70	67	67	65	65	62	61	59	58	65.2%
Mistral Large 3	73	73	70	68	68	64	61	61	56	52	64.4%
GPT-4.1	77	74	70	67	65	64	62	58	56	45	63.8%
Gemma 4 26B	70	67	67	65	65	64	64	59	58	56	63.3%
MiniMax M2.7	77	71	68	64	64	64	59	56	55	48	62.6%
Gemini 3.1 Flash Lite	70	67	62	62	62	58	58	58	56	48	60.0%
DeepSeek V4 Flash	79	73	67	67	64	64	59	55	38	32	59.5%
Gemini 3.1 Flash Lite (Preview)	67	65	65	62	59	59	56	56	53	52	59.4%
Stealth: Aurora Alpha	68	67	65	65	59	58	58	58	55	42	59.4%
Mistral Large 2	67	67	67	61	59	58	56	56	53	47	58.9%
Gemini 3.1 Flash Lite (Reasoning)	65	65	62	61	61	61	61	59	53	32	57.9%
GPT-5.4 Nano (Reasoning)	82	80	76	74	70	68	64	59	0	0	57.3%
DeepSeek V3.2	94	65	59	59	52	47	47	47	44	39	55.3%
MiniMax M2.5	73	71	64	61	53	53	50	42	41	36	54.4%
Z.AI GLM 4.5	76	67	55	53	53	50	45	45	45	36	52.6%
Z.AI GLM 4.5 Air	80	71	68	68	67	56	52	32	23	3	52.0%
Grok 4.20	73	61	55	53	52	50	47	47	39	35	51.1%
Claude Haiku 4.5	67	58	50	50	50	50	48	47	47	42	50.9%
DeepSeek-V2 Chat	61	58	56	53	50	50	47	44	39	33	49.1%
Mistral Medium 3.1	62	56	53	52	52	48	48	48	47	21	48.8%
DeepSeek V3 (2024-12-26)	67	64	59	58	58	56	48	35	24	17	48.5%
GPT-5 Nano	58	58	56	52	50	48	45	44	35	35	48.0%
Grok 4.20 (Beta)	61	58	56	55	50	48	39	39	38	36	48.0%
Qwen3 235B A22B Instruct 2507	58	55	55	55	55	44	44	42	36	33	47.6%
DeepSeek V3.1	71	67	64	62	58	48	41	36	24	0	47.1%
Mistral Small 4 (Reasoning)	53	50	50	47	47	44	44	41	39	29	44.4%
Claude 3.7 Sonnet	61	59	48	44	39	39	38	38	35	30	43.2%
Writer: Palmyra X5	56	56	53	47	44	44	41	39	38	3	42.1%
Grok 4.3	73	61	61	55	50	47	42	21	0	0	40.9%
Z.AI GLM 4.7 Flash	53	47	45	42	41	38	33	33	32	29	39.4%
DeepSeek V3 (2025-03-24)	68	62	48	47	45	33	32	30	23	0	38.9%
ByteDance Seed 1.6 Flash	59	41	39	39	38	38	35	32	29	23	37.3%
Mistral Small Creative	47	41	41	41	38	35	33	33	29	5	34.2%
Claude 3.5 Sonnet	38	36	35	33	33	32	32	32	30	30	33.2%
Qwen 3 32B	44	41	39	39	38	32	26	18	17	12	30.6%
GPT-4.1 Mini	53	44	33	33	30	29	26	21	20	14	30.3%
Nemotron 3 Nano	39	38	36	32	30	29	29	27	14	3	27.7%
Inception Mercury	52	38	35	35	29	23	20	20	17	2	26.8%
GPT-5.4 Nano (Reasoning, Low)	59	48	47	41	39	32	0	0	0	0	26.7%
GPT-5.4 Mini	38	32	32	29	29	29	23	18	15	14	25.8%
Mistral Small 3.2 24B	44	39	39	35	24	21	18	18	9	0	24.8%
GPT-4o, Aug. 6th (temp=0)	44	35	29	29	29	29	27	23	0	0	24.4%
GPT-4o, May 13th (temp=1)	36	30	29	29	29	26	24	18	17	6	24.4%
GPT-4o, Aug. 6th (temp=1)	35	32	29	27	24	21	20	15	14	14	23.0%
Ministral 3 8B	35	30	29	24	21	21	20	18	14	3	21.5%
Ministral 3 14B	32	32	27	24	24	21	14	6	0	0	18.0%
GPT-4o, May 13th (temp=0)	39	35	33	33	32	0	0	0	0	0	17.3%
Qwen 2.5 72B	27	26	20	17	15	15	15	12	9	8	16.4%
Ministral 8B	33	32	24	23	23	12	2	2	2	2	15.3%
Gemma 3 27B	36	26	15	15	14	12	11	9	9	0	14.7%
Mistral Small 4	29	20	18	12	12	11	9	2	0	0	11.2%
Gemini 2.5 Flash Lite	32	30	26	18	5	0	0	0	0	0	11.1%
Hermes 3 405B	27	18	11	11	8	8	6	6	5	2	10.0%
Hermes 3 70B	14	11	11	8	8	8	5	2	0	0	6.4%
Llama 3.1 Nemotron 70B	12	12	11	8	6	5	3	0	0	0	5.6%
Arcee AI: Trinity Mini	15	9	6	6	5	3	3	2	0	0	4.8%
Cohere Command R+ (Aug. 2024)	12	12	9	6	5	3	0	0	0	0	4.7%
Gemma 3 12B	11	9	8	6	5	0	0	0	0	0	3.8%
GPT-5.4 Nano	23	9	3	3	0	0	0	0	0	0	3.8%
GPT-4o Mini (temp=1)	9	6	6	5	2	0	0	0	0	0	2.7%
Llama 3.1 70B	12	6	5	3	2	0	0	0	0	0	2.7%
Claude 3 Haiku	11	8	3	2	2	2	0	0	0	0	2.6%
WizardLM 2 8x22b	9	6	6	2	2	0	0	0	0	0	2.4%
Mistral NeMO	11	0	0	0	0	0	0	0	0	0	1.1%
GPT-4o Mini (temp=0)	8	0	0	0	0	0	0	0	0	0	0.8%
Llama 3.1 8B	5	3	0	0	0	0	0	0	0	0	0.8%
Ministral 3 3B	5	3	0	0	0	0	0	0	0	0	0.8%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Ministral 3B	3	0	0	0	0	0	0	0	0	0	0.3%
Gemma 3 4B	2	0	0	0	0	0	0	0	0	0	0.2%
Arcee AI: Trinity Large (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Large codex (40 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	97	97	97	97	98.9%
GPT-5.4	100	100	100	100	97	97	97	97	97	92	97.9%
Claude Opus 4.6	100	100	100	100	97	97	97	97	97	92	97.9%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	95	95	95	95	92	97.1%
Grok 4	100	100	100	100	95	95	95	95	95	95	96.8%
MoonshotAI: Kimi K2.6	100	100	97	97	97	97	95	95	95	92	96.6%
Z.AI GLM 5 Turbo	100	100	100	100	100	97	97	92	92	87	96.6%
Z.AI GLM 5.1	100	100	100	97	95	95	95	95	95	92	96.3%
MoonshotAI: Kimi K2.5	100	100	100	97	97	95	95	92	92	92	96.1%
Claude Sonnet 4.6 (Reasoning)	100	97	97	97	97	97	97	92	92	92	96.1%
Claude Sonnet 4.6	100	100	100	100	95	95	95	95	92	87	95.8%
DeepSeek V4 Pro (Reasoning)	100	97	97	95	95	95	95	95	92	87	94.7%
GPT-5.5 (Reasoning)	97	97	95	95	95	95	95	92	92	92	94.5%
DeepSeek V4 Flash (Reasoning)	100	100	100	97	97	95	92	89	84	84	93.9%
Claude Sonnet 4	100	100	100	100	92	92	92	92	87	84	93.9%
Qwen3.6 Max Preview	100	95	95	95	95	92	92	92	92	89	93.7%
Qwen 3.6 35B	100	100	100	95	95	95	95	89	89	76	93.4%
Grok 4.3 (Reasoning)	100	95	95	95	95	95	95	89	89	84	93.2%
GPT-5.2	97	97	95	95	95	92	89	89	89	87	92.6%
Qwen 3.5 35B	100	97	97	95	92	92	89	89	87	87	92.6%
Claude Opus 4.6 (Reasoning)	100	100	95	95	95	92	87	87	87	87	92.4%
Grok 4.20 (Beta)	97	97	97	97	97	92	89	89	87	79	92.4%
Qwen 3.6 Flash	95	95	95	95	95	95	89	89	87	87	92.1%
ByteDance Seed 1.6	100	95	95	95	92	89	89	89	89	84	91.8%
Xiaomi MIMO v2.5 Pro	100	100	95	92	92	92	92	89	84	82	91.8%
Gemini 3 Flash (Preview, Reasoning)	100	97	95	95	92	92	89	87	87	84	91.8%
GPT-5.4 (Reasoning)	95	95	95	92	92	92	92	89	87	87	91.6%
Grok 4.20 (Beta, Reasoning)	95	95	95	95	95	89	89	89	89	84	91.6%
Grok 4.20 (Reasoning)	95	95	95	95	89	89	89	89	89	84	91.1%
Qwen 3.5 Flash	100	95	92	92	89	89	89	89	87	87	91.1%
Gemma 4 31B (Reasoning)	100	97	95	95	95	95	92	84	79	76	90.8%
Claude Opus 4.7	92	92	92	92	92	92	92	92	92	79	90.8%
Grok 4.20	97	97	97	97	97	89	89	89	82	71	90.8%
Qwen 3.5 122B	100	95	92	92	92	89	89	87	87	82	90.5%
Gemini 2.5 Flash (Reasoning)	100	95	92	89	89	89	89	89	87	84	90.5%
Qwen 3.5 27B	92	92	92	92	89	89	89	89	89	87	90.3%
Qwen 3.5 Plus (2026-04-20)	100	95	89	89	89	89	89	87	87	84	90.0%
Grok 4 Fast	95	95	95	89	89	89	89	89	89	79	90.0%
GPT-5	95	95	92	89	89	89	89	89	87	82	89.7%
Qwen 3.5 397B A17B	92	92	92	92	92	92	92	84	84	84	89.7%
Grok 4.1 Fast	95	95	95	92	92	89	89	87	84	79	89.7%
Gemma 4 31B	92	92	92	92	92	92	92	84	84	84	89.7%
Claude Opus 4.7 (Reasoning)	95	95	95	89	89	89	89	89	82	82	89.5%
GPT-5.4 (Reasoning, Low)	95	95	89	89	89	89	89	89	84	84	89.5%
Qwen 3.6 27B	100	95	95	95	92	89	89	89	84	63	89.2%
Aion 2.0	95	95	89	89	89	89	89	87	84	76	88.4%
GPT-5.4 Mini (Reasoning)	95	95	89	89	89	89	84	84	84	84	88.4%
Z.AI GLM 4.7	100	92	92	92	89	87	84	84	84	74	87.9%
Z.AI GLM 5	100	100	95	92	89	82	82	79	79	79	87.6%
Stealth: Healer Alpha	100	92	89	89	89	87	84	84	82	79	87.6%
Stealth: Hunter Alpha	100	95	95	92	89	87	87	84	82	66	87.6%
Xiaomi MIMO v2.5	95	95	89	89	89	87	84	84	82	82	87.6%
GPT-5.1	92	92	92	89	89	89	84	82	82	82	87.4%
Gemini 3 Flash (Preview)	92	92	92	92	92	84	84	84	84	76	87.4%
o4 Mini High	95	95	87	87	87	87	84	84	82	82	86.8%
Gemini 3 Pro (Preview)	100	95	92	92	89	89	82	79	76	74	86.8%
Z.AI GLM 4.6	97	92	92	89	89	89	87	84	74	71	86.6%
ByteDance Seed 2.0 Lite	95	89	89	89	89	84	82	82	82	82	86.3%
GPT-4.1	100	100	95	92	92	84	84	76	71	66	86.1%
Nemotron 3 Super	89	89	89	84	84	84	84	84	82	79	85.0%
o4 Mini	95	89	89	87	87	87	82	82	79	74	85.0%
Qwen 3.5 9B	95	95	89	89	84	84	82	76	74	74	84.2%
DeepSeek V4 Flash	95	92	92	92	87	87	79	79	74	66	84.2%
Gemma 4 26B (Reasoning)	95	92	87	84	84	84	79	79	79	76	83.9%
Gemini 3.1 Flash Lite (Preview)	97	89	82	82	82	82	82	82	82	79	83.7%
ByteDance Seed 2.0 Mini	95	95	92	84	82	82	82	79	76	68	83.4%
Gemini 2.5 Flash	92	92	89	89	87	82	82	71	71	68	82.4%
Gemini 3.1 Flash Lite (Reasoning)	100	92	84	84	84	79	76	76	74	71	82.1%
DeepSeek V4 Pro	100	87	87	84	84	79	79	74	74	74	82.1%
Claude Opus 4	95	92	87	84	79	79	79	79	71	68	81.3%
GPT-5.4 Mini (Reasoning, Low)	89	89	87	84	82	79	79	76	74	74	81.3%
Qwen 3.5 Plus (2026-02-15)	92	92	84	84	84	84	79	79	68	63	81.1%
Inception Mercury 2	84	84	84	82	79	79	79	79	76	68	79.5%
Writer: Palmyra X5	89	89	87	84	82	79	74	71	68	66	78.9%
Stealth: Aurora Alpha	89	89	84	76	76	76	76	76	76	66	78.7%
GPT-5 Mini	89	89	87	87	87	87	84	79	53	45	78.7%
GPT-OSS 120B	84	82	79	79	79	79	79	76	76	68	78.2%
Gemini 3.1 Flash Lite	89	84	82	79	79	76	74	74	74	68	77.9%
Z.AI GLM 4.5	87	87	82	82	82	79	76	74	63	63	77.4%
MiniMax M2.5	89	89	82	82	79	76	74	71	68	61	77.1%
Gemini 2.5 Flash Lite (Reasoning)	84	84	82	82	79	76	76	74	68	63	76.8%
GPT-5.4 Nano (Reasoning)	84	84	82	79	76	76	74	71	71	63	76.1%
Claude 3.5 Sonnet	82	82	82	82	74	74	74	66	66	61	73.9%
MiniMax M2.7	89	89	84	79	76	71	66	63	61	58	73.7%
Claude 3.7 Sonnet	76	74	74	74	74	74	74	71	71	71	73.2%
Mistral Medium 3.1	76	76	74	74	74	74	74	68	66	63	71.8%
Mistral Large	87	84	79	71	68	68	66	66	61	61	71.1%
GPT-4o, May 13th (temp=0)	79	79	74	74	74	68	68	68	63	61	70.8%
DeepSeek V3 (2024-12-26)	84	82	79	71	68	68	66	63	63	61	70.5%
Mistral Large 2	82	79	79	71	66	66	66	66	63	63	70.0%
DeepSeek-V2 Chat	89	76	76	74	68	68	63	63	61	58	69.7%
DeepSeek V3.2	95	84	82	76	74	71	71	63	61	18	69.5%
Qwen3 235B A22B Instruct 2507	84	82	79	76	74	68	66	63	58	45	69.5%
Grok 4.3	82	76	74	68	68	68	66	63	61	58	68.4%
Z.AI GLM 4.7 Flash	79	74	74	71	68	68	63	63	61	55	67.6%
Mistral Large 3	74	71	71	68	68	68	66	63	63	61	67.4%
Mistral Small 4 (Reasoning)	79	76	74	68	63	63	63	58	55	53	65.3%
GPT-4.1 Mini	76	74	74	71	66	66	63	61	50	34	63.4%
Z.AI GLM 4.5 Air	87	82	79	79	74	66	66	53	32	0	61.6%
DeepSeek V3 (2025-03-24)	89	89	74	74	68	66	63	58	29	0	61.1%
Nemotron 3 Nano	79	74	66	63	63	61	58	47	45	37	59.2%
Claude Haiku 4.5	87	79	79	79	71	71	63	61	0	0	58.9%
ByteDance Seed 1.6 Flash	74	68	66	63	61	61	61	55	50	29	58.7%
GPT-4o, Aug. 6th (temp=0)	74	61	61	58	58	58	55	55	55	53	58.7%
Mistral Small 3.2 24B	68	63	63	61	58	55	55	55	55	53	58.7%
GPT-5 Nano	82	82	76	74	71	68	66	61	0	0	57.9%
DeepSeek V3.1	82	82	82	79	71	71	55	42	0	0	56.3%
Gemini 2.5 Flash Lite	76	68	61	61	58	50	47	47	45	39	55.3%
Qwen 3 32B	76	74	63	61	58	53	53	50	37	26	55.0%
GPT-4o, Aug. 6th (temp=1)	68	68	66	63	61	58	55	47	45	0	53.2%
Inception Mercury	84	66	66	53	53	50	45	42	42	29	52.9%
GPT-4o, May 13th (temp=1)	74	71	68	53	53	53	45	45	32	32	52.4%
GPT-5.4 Mini	87	82	79	74	71	66	50	0	0	0	50.8%
Mistral Small Creative	63	53	53	50	47	47	45	39	39	21	45.8%
Arcee AI: Trinity Large (Preview)	76	61	58	50	50	42	39	37	32	0	44.5%
Ministral 3 8B	53	50	47	47	45	45	42	39	39	0	40.8%
Gemma 3 27B	47	45	42	39	39	37	37	37	37	37	39.7%
Ministral 8B	50	45	42	42	39	37	37	37	34	34	39.7%
Ministral 3 14B	50	47	45	42	39	37	37	34	32	24	38.7%
Hermes 3 405B	50	50	47	45	42	37	34	29	29	21	38.4%
Ministral 3B	53	50	45	39	39	26	24	18	13	8	31.6%
GPT-5.4 Nano (Reasoning, Low)	84	79	79	71	0	0	0	0	0	0	31.3%
Mistral Small 4	58	47	34	32	32	29	29	24	13	0	29.7%
GPT-4o Mini (temp=0)	32	32	32	32	32	32	32	32	32	11	29.5%
Gemma 3 12B	37	34	34	29	29	26	26	24	21	11	27.1%
Arcee AI: Trinity Mini	37	37	37	37	34	26	24	16	11	8	26.6%
Qwen 2.5 72B	42	39	39	37	34	32	21	8	3	0	25.5%
Ministral 3 3B	50	39	39	32	29	26	16	16	5	0	25.3%
Gemma 4 26B	79	79	79	0	0	0	0	0	0	0	23.7%
Llama 3.1 Nemotron 70B	39	32	29	26	26	24	21	18	16	0	23.2%
Llama 3.1 70B	47	34	32	26	21	16	16	13	3	0	20.8%
GPT-5.4 Nano	53	45	37	32	29	0	0	0	0	0	19.5%
Hermes 3 70B	34	34	26	26	18	16	16	11	3	0	18.4%
Cohere Command R+ (Aug. 2024)	39	34	32	26	24	8	5	5	0	0	17.4%
GPT-4o Mini (temp=1)	24	21	18	16	13	11	11	5	0	0	11.8%
Claude 3 Haiku	26	18	11	8	5	5	5	3	0	0	8.2%
Mistral NeMO	18	11	0	0	0	0	0	0	0	0	2.9%
Llama 3.1 8B	5	0	0	0	0	0	0	0	0	0	0.5%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Rocinante 12B	3	0	0	0	0	0	0	0	0	0	0.3%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), long passage (734 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	95	90	98.5%
Claude Opus 4.6	100	100	100	100	100	100	100	100	90	85	97.5%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	85	85	97.0%
Grok 4.20 (Reasoning)	100	100	100	100	95	95	95	95	95	95	97.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	90	90	80	96.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	90	90	90	85	95.5%
GPT-5.2	100	95	95	95	95	95	95	95	95	95	95.5%
GPT-5 Mini	100	100	95	95	95	95	95	95	95	85	95.0%
Qwen 3.5 27B	100	100	100	100	100	90	90	90	90	90	95.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	95	90	90	85	85	94.5%
Grok 4.3 (Reasoning)	100	100	100	100	100	95	95	90	90	70	94.0%
Qwen3.6 Max Preview	100	100	100	100	90	90	90	90	90	90	94.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	90	90	90	90	85	85	93.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	90	90	85	85	80	93.0%
Qwen 3.5 Flash	100	100	100	100	90	90	90	90	90	80	93.0%
Z.AI GLM 5.1	100	100	100	100	100	90	90	85	85	75	92.5%
GPT-5.1	100	100	100	100	90	90	90	90	85	70	91.5%
Qwen 3.5 122B	100	100	100	100	100	90	90	90	80	65	91.5%
Gemini 2.5 Flash (Reasoning)	100	100	100	95	90	90	90	90	80	80	91.5%
Aion 2.0	100	100	100	90	90	90	90	90	85	75	91.0%
Gemma 4 31B	100	100	90	90	90	90	90	90	90	80	91.0%
Stealth: Hunter Alpha	100	100	100	100	90	90	85	85	85	75	91.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	85	85	85	85	85	85	91.0%
DeepSeek V4 Pro (Reasoning)	100	100	95	90	90	90	90	90	90	70	90.5%
GPT-5.5 (Reasoning)	95	95	95	95	95	95	90	80	80	80	90.0%
Claude Opus 4.7	90	90	90	90	90	90	90	90	90	90	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	90	70	70	70	90.0%
GPT-5	100	100	95	95	90	90	90	85	85	65	89.5%
Qwen 3.5 Plus (2026-04-20)	100	90	90	90	90	90	90	90	90	75	89.5%
GPT-5.5	100	100	100	85	85	85	85	85	85	85	89.5%
DeepSeek V4 Flash (Reasoning)	100	100	95	90	90	90	85	85	80	75	89.0%
GPT-5.5 (Reasoning, Low)	95	95	95	95	95	95	90	80	75	70	88.5%
Z.AI GLM 5	100	100	90	90	90	90	90	90	75	70	88.5%
Qwen 3.6 Flash	100	90	90	90	90	90	90	90	75	75	88.0%
Xiaomi MIMO v2.5 Pro	100	100	95	95	90	90	85	85	70	70	88.0%
Grok 4 Fast	100	100	95	95	85	85	85	80	80	75	88.0%
o4 Mini	90	90	90	90	90	90	85	85	85	80	87.5%
Gemini 3 Pro (Preview)	100	90	90	90	90	90	90	80	80	75	87.5%
Qwen 3.6 27B	90	90	90	90	90	90	90	85	80	75	87.0%
Grok 4.1 Fast	100	100	95	95	90	85	85	80	75	65	87.0%
Claude Sonnet 4.6	100	90	85	85	85	85	85	85	85	85	87.0%
Z.AI GLM 5 Turbo	100	100	100	90	90	90	75	75	75	70	86.5%
MoonshotAI: Kimi K2.6	100	100	95	95	90	85	85	70	70	70	86.0%
ByteDance Seed 1.6	90	90	90	90	90	90	90	75	75	75	85.5%
o4 Mini High	95	90	90	90	90	85	85	80	80	70	85.5%
ByteDance Seed 2.0 Mini	100	90	90	90	90	90	80	80	75	65	85.0%
Qwen 3.5 35B	100	100	90	90	90	90	75	75	70	65	84.5%
Xiaomi MIMO v2.5	90	90	90	90	90	85	85	75	75	75	84.5%
Qwen 3.6 35B	90	90	90	90	90	90	80	75	75	70	84.0%
GPT-5.4	100	100	100	100	85	70	70	70	70	70	83.5%
Gemma 4 26B (Reasoning)	100	90	90	90	90	90	90	90	85	0	81.5%
Grok 4	100	95	90	90	90	85	85	65	60	55	81.5%
ByteDance Seed 2.0 Lite	90	90	90	85	80	80	75	75	75	65	80.5%
Claude Sonnet 4.5	100	90	85	85	85	75	75	70	70	70	80.5%
Z.AI GLM 4.6	90	90	85	85	85	80	75	75	75	60	80.0%
Z.AI GLM 4.7	100	90	85	80	80	75	75	75	70	70	80.0%
MoonshotAI: Kimi K2.5	90	90	90	90	90	85	80	65	60	60	80.0%
Claude Haiku 4.5	80	80	80	80	80	80	80	80	80	80	80.0%
GPT-OSS 120B	90	90	90	80	80	75	75	75	70	65	79.0%
Z.AI GLM 4.5 Air	90	80	80	80	80	80	80	80	75	60	78.5%
Stealth: Healer Alpha	90	90	90	90	85	85	75	65	65	45	78.0%
DeepSeek V4 Flash	90	90	85	80	75	75	75	70	70	70	78.0%
Claude Sonnet 4	100	85	85	85	70	70	70	70	70	70	77.5%
MiniMax M2.7	90	80	80	80	80	80	75	75	75	55	77.0%
Gemma 4 26B	80	80	80	80	75	75	75	75	75	75	77.0%
Claude 3.7 Sonnet	90	90	80	75	75	75	75	75	65	65	76.5%
Gemini 3 Flash (Preview)	90	75	75	75	75	75	75	75	75	70	76.0%
Gemini 2.5 Flash	90	90	80	80	75	75	70	70	70	55	75.5%
Inception Mercury 2	75	75	75	75	75	75	75	75	75	70	74.5%
GPT-5.4 (Reasoning, Low)	100	75	70	70	70	65	65	65	65	65	71.0%
GPT-5.4 (Reasoning)	85	85	80	70	65	65	65	65	65	65	71.0%
Gemini 3.1 Flash Lite (Preview)	75	70	70	70	70	70	70	70	70	70	70.5%
Gemini 3.1 Flash Lite	75	70	70	70	70	70	70	70	70	70	70.5%
Mistral Large	90	80	75	75	75	65	65	60	60	60	70.5%
Claude 3.5 Sonnet	80	75	70	70	70	70	70	65	65	65	70.0%
Grok 4.20	80	80	75	70	70	70	70	65	60	60	70.0%
MiniMax M2.5	80	80	75	75	70	70	65	65	60	60	70.0%
Claude Opus 4	90	75	75	75	75	65	65	60	60	60	70.0%
Gemini 3.1 Flash Lite (Reasoning)	70	70	70	70	70	70	70	70	70	65	69.5%
Z.AI GLM 4.5	75	75	75	75	75	70	70	65	60	55	69.5%
GPT-5.4 Mini (Reasoning)	75	75	75	70	70	70	70	70	65	50	69.0%
Nemotron 3 Super	100	95	90	70	70	70	60	55	45	35	69.0%
ByteDance Seed 1.6 Flash	90	80	75	70	70	65	65	60	55	55	68.5%
Qwen 3.5 9B	90	80	80	80	75	75	70	65	65	0	68.0%
Grok 4.20 (Beta)	80	80	75	75	70	70	70	55	55	45	67.5%
GPT-5.4 Mini (Reasoning, Low)	80	75	75	75	70	65	65	55	55	55	67.0%
Gemini 2.5 Flash Lite (Reasoning)	75	75	75	70	70	65	65	60	55	55	66.5%
Mistral Medium 3.1	75	75	75	75	75	75	75	70	70	0	66.5%
GPT-4.1 Mini	70	70	70	70	70	70	65	65	65	50	66.5%
Mistral Large 3	85	75	65	65	65	60	60	60	60	60	65.5%
GPT-4o, May 13th (temp=0)	70	70	70	65	65	65	65	65	65	50	65.0%
GPT-5.4 Nano (Reasoning)	85	80	75	75	75	70	65	65	60	0	65.0%
Mistral Large 2	80	75	65	65	65	65	65	60	50	50	64.0%
Llama 3.1 Nemotron 70B	75	65	65	65	65	65	60	60	60	60	64.0%
Stealth: Aurora Alpha	85	80	75	75	75	75	75	75	10	10	63.5%
Mistral Small 3.2 24B	75	75	75	70	65	65	55	55	50	50	63.5%
GPT-4.1	75	65	65	65	65	65	65	60	55	50	63.0%
GPT-4o, Aug. 6th (temp=0)	65	65	65	65	65	65	65	60	60	55	63.0%
Z.AI GLM 4.7 Flash	70	70	70	70	65	65	60	55	55	45	62.5%
Mistral Small 4 (Reasoning)	80	75	70	65	65	65	55	55	50	40	62.0%
DeepSeek-V2 Chat	80	70	70	70	70	60	55	50	45	45	61.5%
DeepSeek V4 Pro	85	85	70	70	70	60	55	55	55	0	60.5%
GPT-4o, Aug. 6th (temp=1)	70	70	65	65	60	55	55	55	55	50	60.0%
Grok 4.3	80	70	70	70	65	65	55	55	45	20	59.5%
Qwen 3 32B	80	80	65	65	65	55	55	45	45	35	59.0%
Writer: Palmyra X5	70	70	60	60	60	55	55	55	45	45	57.5%
Llama 3.1 70B	70	65	60	55	55	55	55	55	55	50	57.5%
Mistral Small Creative	75	70	70	60	55	55	55	50	45	40	57.5%
Ministral 3 14B	80	80	75	75	65	65	65	60	5	5	57.5%
GPT-4o, May 13th (temp=1)	70	65	60	60	60	60	55	50	50	40	57.0%
GPT-5 Nano	75	65	65	60	60	55	55	45	45	45	57.0%
Qwen3 235B A22B Instruct 2507	65	65	65	60	60	55	50	50	45	40	55.5%
DeepSeek V3 (2024-12-26)	70	65	60	55	55	55	55	45	45	45	55.0%
DeepSeek V3 (2025-03-24)	70	70	60	55	55	55	50	45	45	45	55.0%
Gemini 2.5 Flash Lite	75	60	60	55	55	55	55	50	40	35	54.0%
Qwen 2.5 72B	70	60	60	60	55	55	50	50	45	35	54.0%
Nemotron 3 Nano	65	60	60	60	55	55	55	45	45	40	54.0%
Gemma 3 27B	70	65	55	55	55	50	50	45	45	45	53.5%
DeepSeek V3.2	65	65	65	55	55	55	45	40	35	30	51.0%
Inception Mercury	65	65	60	55	55	50	50	50	30	15	49.5%
Arcee AI: Trinity Large (Preview)	70	55	55	55	55	50	50	45	35	20	49.0%
GPT-5.4 Mini	65	60	55	55	55	50	45	45	45	5	48.0%
Hermes 3 405B	65	55	50	45	45	45	45	40	40	40	47.0%
DeepSeek V3.1	70	55	55	55	50	45	40	40	40	0	45.0%
Ministral 3 3B	50	45	45	40	40	40	35	35	30	30	39.0%
Ministral 3 8B	55	50	45	45	40	35	30	25	20	5	35.0%
Ministral 8B	65	45	45	45	35	30	30	25	0	0	32.0%
Gemma 3 12B	50	35	35	35	25	25	25	20	20	15	28.5%
Ministral 3B	55	40	30	30	30	30	30	15	15	5	28.0%
GPT-5.4 Nano (Reasoning, Low)	75	75	65	60	0	0	0	0	0	0	27.5%
GPT-4o Mini (temp=1)	35	35	35	35	35	25	25	20	10	10	26.5%
Hermes 3 70B	45	40	35	35	30	30	25	20	5	0	26.5%
GPT-4o Mini (temp=0)	35	25	25	25	25	25	25	25	25	25	26.0%
Arcee AI: Trinity Mini	45	35	30	30	25	20	20	20	20	10	25.5%
Mistral Small 4	40	35	35	30	30	25	20	20	15	0	25.0%
Cohere Command R+ (Aug. 2024)	50	45	40	40	35	30	0	0	0	0	24.0%
Mistral NeMO	50	40	40	35	25	25	0	0	0	0	21.5%
Claude 3 Haiku	30	30	20	20	20	20	20	20	15	5	20.0%
Llama 3.1 8B	35	30	30	30	30	20	15	10	0	0	20.0%
GPT-5.4 Nano	35	20	20	15	15	15	0	0	0	0	12.0%
Gemma 3 4B	30	25	20	10	10	5	0	0	0	0	10.0%
Rocinante 12B	50	45	0	0	0	0	0	0	0	0	9.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	95	99.5%
Gemini 3.1 Flash Lite	100	100	100	100	100	100	100	100	100	95	99.5%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	90	99.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	90	99.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	90	99.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	95	90	98.5%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	95	90	98.5%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	90	90	98.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	90	90	98.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	80	98.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	95	95	95	95	95	97.5%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	90	90	90	97.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	90	90	90	97.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	90	90	90	97.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	90	90	90	97.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	85	85	97.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	85	85	97.0%
Z.AI GLM 4.6	100	100	100	100	100	100	95	90	90	90	96.5%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	90	90	85	96.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	95	95	95	95	95	95	95	96.5%
DeepSeek V3.2	100	100	100	100	100	100	100	90	85	80	95.5%
Z.AI GLM 4.5	100	100	100	100	100	90	90	90	90	90	95.0%
o4 Mini High	100	100	100	100	100	90	90	90	90	90	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	85	85	85	85	94.0%
o4 Mini	100	100	100	90	90	90	90	90	90	90	93.0%
Grok 4.20 (Beta)	100	100	100	100	100	85	85	85	85	85	92.5%
Grok 4.20	100	100	100	100	100	85	85	85	85	85	92.5%
Grok 4	100	100	90	90	90	90	90	90	90	90	92.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	90	90	85	85	85	85	92.0%
DeepSeek V3.1	100	100	100	100	90	90	85	85	85	85	92.0%
Gemma 4 31B	100	90	90	90	90	90	90	90	90	90	91.0%
Llama 3.1 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Llama 3.1 Nemotron 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Claude Opus 4	100	100	100	100	85	85	85	85	85	85	91.0%
GPT-OSS 120B	100	100	90	90	90	90	90	90	90	80	91.0%
Gemma 4 26B	90	90	90	90	90	90	90	90	90	90	90.0%
ByteDance Seed 2.0 Lite	100	100	90	90	90	90	90	90	80	80	90.0%
Inception Mercury 2	90	90	90	90	90	90	90	90	90	90	90.0%
Gemini 2.5 Flash	90	90	90	90	90	90	90	90	90	90	90.0%
Z.AI GLM 4.5 Air	100	100	90	90	90	90	90	90	80	75	89.5%
Mistral Large	100	90	90	90	90	90	90	85	85	85	89.5%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	95	0	89.5%
Claude Opus 4.7	100	100	90	90	90	90	90	90	75	75	89.0%
Aion 2.0	100	100	100	100	100	100	100	100	90	0	89.0%
Mistral Large 2	90	90	90	90	90	90	90	90	85	85	89.0%
DeepSeek V4 Flash	90	90	90	90	90	90	90	90	85	85	89.0%
Hermes 3 405B	100	100	100	90	90	90	90	85	80	60	88.5%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	90	90	0	88.0%
Inception Mercury	100	90	90	90	90	90	90	90	75	75	88.0%
Mistral Large 3	90	90	90	90	90	85	85	85	85	85	87.5%
Gemini 2.5 Flash Lite (Reasoning)	90	90	90	90	90	90	90	90	80	65	86.5%
MiniMax M2.7	100	100	90	90	90	85	85	75	75	75	86.5%
Nemotron 3 Nano	100	90	90	90	90	80	80	80	80	80	86.0%
GPT-4o, May 13th (temp=1)	100	100	100	90	85	85	85	85	65	65	86.0%
Gemma 3 12B	90	90	90	90	90	90	80	80	80	75	85.5%
Qwen 3.6 27B	100	100	100	100	90	90	90	90	80	10	85.0%
MiniMax M2.5	100	100	90	90	90	85	75	75	75	70	85.0%
Claude Sonnet 4.6	85	85	85	85	85	85	85	85	85	85	85.0%
Grok 4.3	100	95	90	90	90	85	80	75	75	65	84.5%
DeepSeek V4 Pro	100	100	95	95	95	90	90	90	85	0	84.0%
GPT-4.1 Mini	90	90	90	90	85	80	80	75	75	75	83.0%
ByteDance Seed 1.6 Flash	90	90	90	90	80	80	80	80	75	65	82.0%
Stealth: Aurora Alpha	90	90	90	90	90	90	90	90	90	0	81.0%
Qwen 3 32B	90	90	90	90	90	80	80	75	70	50	80.5%
Mistral Small 3.2 24B	90	90	90	80	75	75	75	75	75	75	80.0%
Qwen3 235B A22B Instruct 2507	85	85	85	85	80	75	75	75	75	65	78.5%
GPT-5 Nano	90	90	80	80	80	80	70	70	70	70	78.0%
Mistral Small 4 (Reasoning)	90	90	80	80	80	75	75	75	70	65	78.0%
Qwen 2.5 72B	90	90	90	80	80	80	80	80	70	35	77.5%
Z.AI GLM 4.7 Flash	100	90	90	80	75	75	75	65	60	55	76.5%
Arcee AI: Trinity Large (Preview)	90	85	85	80	75	70	70	70	70	65	76.0%
Writer: Palmyra X5	85	85	85	85	75	75	70	65	65	65	75.5%
Gemini 2.5 Flash Lite	75	75	75	75	75	75	75	75	75	70	74.5%
GPT-5.4 Mini	90	85	80	75	75	75	75	65	65	40	72.5%
Mistral Small Creative	80	80	70	70	70	70	70	70	70	65	71.5%
Cohere Command R+ (Aug. 2024)	90	90	80	70	70	70	65	60	60	55	71.0%
Claude Haiku 4.5	75	75	75	75	75	75	75	60	60	60	70.5%
Gemma 3 27B	75	75	70	65	65	65	65	55	55	55	64.5%
Ministral 3 8B	70	70	65	65	60	60	60	50	50	35	58.5%
Ministral 3 14B	85	75	65	60	60	60	60	55	45	5	57.0%
Ministral 8B	75	75	65	65	60	60	55	55	35	20	56.5%
GPT-4o Mini (temp=0)	60	60	60	60	60	60	50	50	50	50	56.0%
Ministral 3 3B	65	65	65	65	60	60	55	55	40	30	56.0%
GPT-5.4 Nano	90	80	75	75	70	60	55	40	0	0	54.5%
Hermes 3 70B	80	75	65	60	60	60	50	45	40	0	53.5%
GPT-4o Mini (temp=1)	60	60	60	50	50	50	50	50	50	35	51.5%
Ministral 3B	65	60	55	50	50	50	50	45	40	35	50.0%
Llama 3.1 8B	75	70	65	60	60	40	40	35	30	20	49.5%
Mistral Small 4	70	65	65	60	55	50	50	45	30	0	49.0%
Mistral NeMO	65	55	55	45	45	45	45	45	45	45	49.0%
Arcee AI: Trinity Mini	60	60	55	50	45	45	40	35	35	35	46.0%
GPT-5.4 Nano (Reasoning, Low)	100	90	80	80	75	0	0	0	0	0	42.5%
Claude 3 Haiku	45	45	45	40	35	35	35	35	20	15	35.0%
Rocinante 12B	55	40	35	15	15	15	5	5	5	0	19.0%
GPT-4.1 Nano	20	20	15	10	10	10	10	10	0	0	10.5%
WizardLM 2 8x22b	10	10	0	0	0	0	0	0	0	0	2.0%
Gemma 3 4B	5	0	0	0	0	0	0	0	0	0	0.5%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

tiers

▼

5 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	90	99.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	90	99.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	90	90	98.0%
Grok 4	100	100	100	100	100	100	100	90	90	90	97.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	70	97.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	90	80	97.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
Claude Opus 4.7	100	100	100	100	100	100	100	90	90	80	96.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	90	90	90	90	96.0%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	90	90	90	90	96.0%
GPT-5	100	100	100	100	100	100	90	90	90	90	96.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	90	70	96.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	80	80	96.0%
DeepSeek V3.2	100	100	100	100	100	90	90	90	90	90	95.0%
Gemini 3.1 Flash Lite	100	100	100	100	100	90	90	90	90	90	95.0%
DeepSeek V4 Pro	100	100	100	100	100	100	100	90	90	70	95.0%
DeepSeek V4 Flash	100	100	100	100	100	100	100	80	80	80	94.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	80	80	80	94.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	30	93.0%
Mistral Large	100	100	100	90	90	90	90	90	90	90	93.0%
GPT-OSS 120B	100	100	100	100	100	100	80	80	80	80	92.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	90	80	80	70	92.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	90	30	92.0%
Llama 3.1 70B	100	100	100	100	100	100	90	80	70	70	91.0%
Mistral Large 3	90	90	90	90	90	90	90	90	90	90	90.0%
Mistral Large 2	90	90	90	90	90	90	90	90	90	90	90.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	80	80	80	60	90.0%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	100	80	70	50	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	80	80	80	60	90.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	80	0	88.0%
Mistral Small 4 (Reasoning)	100	100	90	90	90	80	80	80	80	70	86.0%
Inception Mercury 2	100	100	100	80	80	80	80	80	80	80	86.0%
Claude Opus 4	100	100	100	100	100	70	70	70	70	70	85.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	80	80	70	70	50	85.0%
Stealth: Aurora Alpha	100	100	100	100	100	80	80	80	80	30	85.0%
DeepSeek V3.1	100	100	100	100	100	100	90	80	70	0	84.0%
MiniMax M2.7	100	100	100	80	80	80	80	80	70	70	84.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	90	90	70	60	60	60	83.0%
ByteDance Seed 1.6 Flash	100	80	80	80	80	80	80	80	80	70	81.0%
Gemini 2.5 Flash	100	80	80	80	80	80	80	80	70	70	80.0%
Inception Mercury	100	80	80	80	80	80	80	80	60	60	78.0%
MiniMax M2.5	100	100	90	80	70	70	70	50	50	50	73.0%
GPT-5 Nano	100	100	100	80	80	80	80	50	50	0	72.0%
Mistral Medium 3.1	100	70	70	70	70	70	70	70	60	60	71.0%
Claude Sonnet 4.6	70	70	70	70	70	70	70	70	70	70	70.0%
Gemma 3 27B	70	70	70	70	70	70	70	70	70	70	70.0%
GPT-4o, Aug. 6th (temp=0)	90	90	90	60	60	60	60	60	60	60	69.0%
GPT-4o, May 13th (temp=1)	90	90	70	70	60	60	60	60	60	60	68.0%
GPT-4.1 Mini	80	80	70	70	70	70	70	70	70	30	68.0%
Qwen 2.5 72B	80	80	80	80	80	80	50	50	50	50	68.0%
GPT-4o, Aug. 6th (temp=1)	90	90	70	60	60	60	60	60	60	60	67.0%
Grok 4.20	100	90	80	70	60	60	60	50	50	40	66.0%
Grok 4.20 (Beta)	80	70	70	60	60	60	60	60	60	50	63.0%
Grok 4.3	70	70	70	70	70	70	60	60	50	40	63.0%
Arcee AI: Trinity Mini	100	80	80	80	60	60	50	50	30	30	62.0%
Writer: Palmyra X5	70	70	70	70	70	70	60	60	60	0	60.0%
GPT-4o Mini (temp=1)	80	80	80	80	50	50	50	50	40	40	60.0%
Qwen3 235B A22B Instruct 2507	70	70	70	70	70	60	60	60	60	0	59.0%
Arcee AI: Trinity Large (Preview)	70	70	70	70	60	60	50	50	40	40	58.0%
Claude Haiku 4.5	70	70	70	70	60	60	50	40	40	40	57.0%
GPT-5.4 Mini	90	70	70	60	60	60	60	40	40	10	56.0%
GPT-4o Mini (temp=0)	80	80	50	50	50	50	50	50	50	50	56.0%
Gemma 3 12B	80	50	50	50	50	50	50	50	50	50	53.0%
Mistral Small Creative	60	60	60	60	60	60	50	50	40	30	53.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	80	30	20	0	0	0	0	43.0%
Llama 3.1 8B	70	70	70	70	40	30	30	20	20	0	42.0%
Gemini 2.5 Flash Lite	50	50	40	40	40	40	40	40	40	40	42.0%
Cohere Command R+ (Aug. 2024)	80	60	60	60	50	40	20	20	20	0	41.0%
Ministral 3 3B	70	70	60	50	40	40	40	30	0	0	40.0%
Mistral Small 3.2 24B	60	50	50	40	40	40	30	30	30	10	38.0%
Mistral NeMO	70	50	50	50	40	40	30	30	10	0	37.0%
Claude 3 Haiku	60	60	40	40	40	30	20	20	10	10	33.0%
GPT-5.4 Nano	60	60	50	50	40	30	30	0	0	0	32.0%
Ministral 3 14B	50	50	40	40	40	20	20	20	20	10	31.0%
Ministral 3B	70	70	60	40	20	20	10	10	0	0	30.0%
Mistral Small 4	50	50	40	40	40	30	30	20	0	0	30.0%
Hermes 3 70B	60	50	50	30	30	20	20	10	0	0	27.0%
GPT-4.1 Nano	20	20	20	20	0	0	0	0	0	0	8.0%
Ministral 8B	50	10	10	0	0	0	0	0	0	0	7.0%
Rocinante 12B	40	20	10	0	0	0	0	0	0	0	7.0%
Gemma 3 4B	20	20	20	0	0	0	0	0	0	0	6.0%
Ministral 3 8B	10	0	0	0	0	0	0	0	0	0	1.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

10 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	92	99.2%
GPT-5.2	100	100	100	100	100	100	100	100	100	92	99.2%
GPT-5.5	100	100	100	100	100	100	100	100	100	92	99.2%
Claude Opus 4	100	100	100	100	100	100	100	100	100	92	99.2%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100	100	92	99.2%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100	100	92	99.2%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100	100	92	99.2%
Aion 2.0	100	100	100	100	100	100	100	100	100	92	99.2%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
Grok 4	100	100	100	100	100	100	100	100	100	92	99.2%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	92	92	98.3%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	92	92	98.3%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100	92	92	98.3%
Inception Mercury 2	100	100	100	100	100	100	100	100	92	92	98.3%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	92	92	92	97.5%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	75	97.5%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	92	83	97.5%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100	100	75	97.5%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	83	83	96.7%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	83	83	96.7%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	92	92	92	92	96.7%
DeepSeek V4 Pro	100	100	100	100	100	100	100	100	92	75	96.7%
GPT-5.1	100	100	100	100	92	92	92	92	92	92	95.0%
GPT-5	100	100	100	100	92	92	92	92	92	92	95.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	75	75	95.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	83	83	83	95.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	92	92	92	75	95.0%
MiniMax M2.7	100	100	100	100	100	100	100	83	83	67	93.3%
MiniMax M2.5	100	100	100	100	100	100	100	83	75	75	93.3%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100	92	42	93.3%
o4 Mini	100	100	100	100	100	100	92	92	75	75	93.3%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	33	93.3%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	75	75	75	92.5%
GPT-5 Mini	100	92	92	92	92	92	92	92	92	92	92.5%
o4 Mini High	100	100	100	100	100	92	92	92	75	75	92.5%
Llama 3.1 Nemotron 70B	100	100	100	100	100	92	83	83	83	83	92.5%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	83	83	83	67	91.7%
Qwen 3.6 35B	100	100	100	100	100	100	92	75	75	75	91.7%
Qwen 3.5 Flash	100	100	100	100	100	100	100	92	75	50	91.7%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	75	75	75	75	90.0%
Writer: Palmyra X5	100	100	100	100	100	100	75	75	75	75	90.0%
Z.AI GLM 4.5	100	100	100	100	100	92	75	75	75	75	89.2%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	75	67	50	89.2%
Gemini 2.5 Flash Lite (Reasoning)	100	100	92	92	92	83	83	83	83	83	89.2%
GPT-OSS 120B	100	100	100	100	100	92	92	75	75	58	89.2%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	75	75	75	75	75	87.5%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	75	75	75	75	75	87.5%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	83	75	75	42	87.5%
Z.AI GLM 4.6	100	100	100	100	100	100	92	92	75	8	86.7%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	92	75	58	42	86.7%
DeepSeek V3 (2025-03-24)	100	100	100	92	83	83	83	75	75	75	86.7%
Mistral Small Creative	100	100	100	92	92	83	83	83	75	58	86.7%
Stealth: Hunter Alpha	100	100	100	100	83	75	75	75	75	75	85.8%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	83	83	83	67	67	67	85.0%
Inception Mercury	100	100	100	100	83	75	75	75	75	58	84.2%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	92	92	83	75	0	84.2%
ByteDance Seed 1.6 Flash	100	92	83	83	83	83	83	83	83	67	84.2%
Gemini 3.1 Flash Lite	100	100	100	83	83	83	75	75	75	50	82.5%
GPT-4.1 Mini	92	92	92	92	92	75

Accuracy (recall)

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

matrix

Large codex (40 entries), long passage (1,019 words)

Large codex (40 entries), short passage (165 words)

Small codex (7 entries), long passage (734 words)

Small codex (7 entries), short passage (165 words)

tiers

5 codex entries

10 codex entries