Accuracy (recall)

Test: Codex Violation Detection

Avg. Score

65.2%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Grok 4.1 Fast	93.8%	$0.0021	21.1s	84%
2	Grok 4 Fast	92.4%	$0.0018	12.8s	72%
3	Gemini 3 Flash (Preview)	88.8%	$0.0031	4.5s	69%
4	Claude Opus 4.5	98.0%	$0.041	9.7s	91%
5	GPT-5.2	95.7%	$0.024	24.2s	87%
6	Claude Sonnet 4.5	93.6%	$0.024	8.9s	80%
7	Gemini 2.5 Flash	82.5%	$0.0025	2.8s	69%
8	Gemini 2.5 Pro	97.4%	$0.035	23.2s	91%
9	Claude Opus 4.6	97.3%	$0.040	10.2s	88%
10	Claude Sonnet 4	90.3%	$0.023	9.0s	72%
11	o4 Mini	88.9%	$0.019	28.1s	74%
12	Z.AI GLM 5	92.1%	$0.013	56.4s	79%
13	ByteDance Seed 1.6	91.9%	$0.0067	1.0m	77%
14	GPT-5 Mini	91.2%	$0.0092	51.7s	74%
15	Z.AI GLM 4.7	91.3%	$0.0091	1.0m	77%
16	Claude Sonnet 4.6	82.9%	$0.024	9.9s	70%
17	Mistral Large 3	75.5%	$0.0030	10.2s	58%
18	Mistral Large	79.3%	$0.012	9.1s	59%
19	Qwen 3.5 Plus (2026-02-15)	85.0%	$0.0041	34.2s	55%
20	Mistral Large 2	75.9%	$0.012	8.8s	57%
21	Mistral Medium 3.1	75.1%	$0.0029	7.7s	46%
22	GPT-5.1	92.7%	$0.037	52.8s	79%
23	o4 Mini High	90.7%	$0.033	51.3s	76%
24	MoonshotAI: Kimi K2.5	92.9%	$0.015	1.5m	79%
25	Minimax M2.5	74.6%	$0.0020	25.5s	52%
26	DeepSeek V3.2	75.0%	$0.0013	17.2s	45%
27	GPT-4.1	77.7%	$0.0081	10.0s	44%
28	Gemini 3.1 Pro (Preview)	98.3%	$0.068	52.3s	93%
29	Z.AI GLM 4.5	75.6%	$0.0026	18.5s	45%
30	ByteDance Seed 1.6 Flash	67.9%	$0.0009	12.0s	46%
31	DeepSeek-V2 Chat	71.1%	$0.0020	13.1s	41%
32	GPT-4.1 Mini	63.7%	$0.0015	5.8s	43%
33	Claude Haiku 4.5	66.8%	$0.0078	5.8s	45%
34	Gemini 3 Pro (Preview)	91.0%	$0.050	34.0s	70%
35	DeepSeek V3 (2024-12-26)	69.9%	$0.0019	15.6s	41%
36	Z.AI GLM 4.6	86.1%	$0.0049	1.3m	59%
37	Writer: Palmyra X5	67.2%	$0.0062	12.1s	44%
38	Stealth: Aurora Alpha	77.5%	—	6.2s	46%
39	Grok 4	94.1%	$0.051	1.2m	80%
40	Claude 3.7 Sonnet	77.4%	$0.023	10.2s	42%
41	GPT-4o, Aug. 6th (temp=1)	65.4%	$0.011	3.6s	36%
42	Mistral Small Creative	56.8%	$0.0006	4.3s	36%
43	DeepSeek V3 (2025-03-24)	68.9%	$0.0015	22.9s	34%
44	GPT-4o, Aug. 6th (temp=0)	67.9%	$0.015	5.0s	36%
45	Claude 3.5 Sonnet	79.2%	$0.042	10.5s	47%
46	DeepSeek V3.1	66.1%	$0.0014	31.1s	33%
47	GPT-4o, May 13th (temp=0)	71.6%	$0.030	5.4s	36%
48	Mistral Small 3.2 24B	53.8%	$0.0006	10.4s	32%
49	Gemma 3 27B	53.2%	$0.0005	13.3s	33%
50	Z.AI GLM 4.7 Flash	68.2%	$0.0018	1.1m	45%
51	GPT-5	93.3%	$0.061	1.6m	81%
52	Gemini 2.5 Flash Lite	47.9%	$0.0005	2.3s	24%
53	Claude 3.5 Haiku	51.2%	$0.0049	6.2s	25%
54	GPT-4o, May 13th (temp=1)	58.1%	$0.026	3.7s	34%
55	Ministral 3 14B	46.4%	$0.0010	6.3s	25%
56	Hermes 3 405B	56.2%	$0.0044	17.2s	20%
57	Llama 3.1 Nemotron 70B	55.8%	$0.0055	18.5s	19%
58	Qwen 3.5 397B A17B	93.7%	$0.026	2.9m	75%
59	Ministral 3 8B	39.9%	$0.0007	4.8s	20%
60	Llama 3.1 70B	51.2%	$0.0021	24.3s	18%
61	Qwen 2.5 72B	42.3%	$0.0008	13.9s	18%
62	Ministral 8B	34.7%	$0.0005	5.6s	18%
63	Claude Opus 4	84.4%	$0.116	15.8s	65%
64	Arcee AI: Trinity Mini	33.3%	$0.0004	10.3s	18%
65	GPT-5 Nano	67.2%	$0.0049	1.9m	42%
66	Ministral 3 3B	28.6%	$0.0005	3.3s	16%
67	Gemma 3 12B	35.9%	$0.0003	12.0s	11%
68	GPT-4o Mini (temp=1)	29.4%	$0.0006	7.4s	13%
69	Ministral 3B	25.0%	$0.0002	2.9s	14%
70	GPT-4o Mini (temp=0)	31.3%	$0.0006	25.0s	19%
71	Cohere Command R+ (Aug. 2024)	32.3%	$0.014	10.0s	15%
72	Claude 3 Haiku	19.4%	$0.0015	3.6s	12%
73	Hermes 3 70B	28.5%	$0.0013	29.2s	14%
74	Mistral NeMO	20.6%	$0.0007	13.3s	7%
75	Llama 3.1 8B	16.1%	$0.0002	16.1s	0%
76	WizardLM 2 8x22b	17.1%	$0.0036	15.7s	0%
77	Rocinante 12B	6.4%	$0.0009	6.0s	0%
78	GPT-4.1 Nano	2.6%	$0.0004	3.9s	0%
79	Gemma 3 4B	3.2%	$0.0002	12.5s	0%
80	Arcee AI: Trinity Large (Preview)	46.2%	$0.0000	3.1m	24%
65.23%

Individual Scenarios

matrix

▼

Large codex (40 entries), long passage (1,019 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Grok 4	100	98	98	97	97	95	95	94	94	91	96.1%
Qwen 3.5 397B A17B	100	100	97	95	94	94	89	89	88	85	93.2%
Gemini 2.5 Pro	100	95	92	92	92	89	88	88	88	86	91.2%
GPT-5.2	97	97	94	92	92	91	89	88	86	85	91.2%
Grok 4 Fast	100	100	97	92	91	89	86	86	83	82	90.8%
Grok 4.1 Fast	100	97	94	91	91	91	89	89	82	82	90.6%
GPT-5	94	94	92	92	91	91	88	88	88	88	90.6%
Gemini 3.1 Pro (Preview)	94	94	91	91	91	89	88	88	88	86	90.0%
Claude Opus 4.5	92	92	91	91	89	89	88	88	85	83	88.9%
GPT-5 Mini	91	91	88	88	86	86	85	85	85	85	87.0%
ByteDance Seed 1.6	94	91	91	89	88	85	83	82	80	79	86.2%
Z.AI GLM 4.6	94	92	92	88	88	86	83	82	77	77	86.1%
o4 Mini High	94	91	91	88	85	83	82	82	82	79	85.6%
GPT-5.1	94	91	88	85	85	85	82	82	80	80	85.2%
Z.AI GLM 5	97	85	85	83	82	82	82	79	79	77	83.0%
Claude Opus 4.6	85	85	85	85	83	83	80	80	80	80	82.7%
Z.AI GLM 4.7	94	88	88	83	83	82	79	79	74	73	82.3%
Claude Opus 4	92	89	85	83	82	82	79	77	77	70	81.7%
MoonshotAI: Kimi K2.5	91	86	86	86	82	82	80	80	73	65	81.2%
Claude Sonnet 4.6	86	82	82	82	80	77	77	76	76	74	79.2%
Claude Sonnet 4	83	82	80	80	80	79	79	77	73	71	78.5%
Gemini 3 Flash (Preview)	85	82	82	80	79	76	76	76	74	74	78.3%
Claude Sonnet 4.5	85	83	82	80	79	77	77	77	71	70	78.2%
o4 Mini	88	86	82	82	79	76	76	73	70	67	77.7%
Gemini 2.5 Flash	79	77	76	76	74	73	70	64	61	59	70.8%
Gemini 3 Pro (Preview)	83	82	80	76	76	76	74	73	67	15	70.2%
Qwen 3.5 Plus (2026-02-15)	92	85	83	82	79	77	76	39	38	0	65.2%
Mistral Large	79	70	67	67	65	65	62	61	59	58	65.2%
Mistral Large 3	73	73	70	68	68	64	61	61	56	52	64.4%
GPT-4.1	77	74	70	67	65	64	62	58	56	45	63.8%
Stealth: Aurora Alpha	68	67	65	65	59	58	58	58	55	42	59.4%
Mistral Large 2	67	67	67	61	59	58	56	56	53	47	58.9%
DeepSeek V3.2	94	65	59	59	52	47	47	47	44	39	55.3%
Minimax M2.5	73	71	64	61	53	53	50	42	41	36	54.4%
Z.AI GLM 4.5	76	67	55	53	53	50	45	45	45	36	52.6%
Claude Haiku 4.5	67	58	50	50	50	50	48	47	47	42	50.9%
DeepSeek-V2 Chat	61	58	56	53	50	50	47	44	39	33	49.1%
Mistral Medium 3.1	62	56	53	52	52	48	48	48	47	21	48.8%
DeepSeek V3 (2024-12-26)	67	64	59	58	58	56	48	35	24	17	48.5%
GPT-5 Nano	58	58	56	52	50	48	45	44	35	35	48.0%
DeepSeek V3.1	71	67	64	62	58	48	41	36	24	0	47.1%
Claude 3.7 Sonnet	61	59	48	44	39	39	38	38	35	30	43.2%
Writer: Palmyra X5	56	56	53	47	44	44	41	39	38	3	42.1%
Z.AI GLM 4.7 Flash	53	47	45	42	41	38	33	33	32	29	39.4%
DeepSeek V3 (2025-03-24)	68	62	48	47	45	33	32	30	23	0	38.9%
ByteDance Seed 1.6 Flash	59	41	39	39	38	38	35	32	29	23	37.3%
Mistral Small Creative	47	41	41	41	38	35	33	33	29	5	34.2%
Claude 3.5 Sonnet	38	36	35	33	33	32	32	32	30	30	33.2%
GPT-4.1 Mini	53	44	33	33	30	29	26	21	20	14	30.3%
Mistral Small 3.2 24B	44	39	39	35	24	21	18	18	9	0	24.8%
GPT-4o, May 13th (temp=1)	36	30	29	29	29	26	24	18	17	6	24.4%
GPT-4o, Aug. 6th (temp=0)	44	35	29	29	29	29	27	23	0	0	24.4%
GPT-4o, Aug. 6th (temp=1)	35	32	29	27	24	21	20	15	14	14	23.0%
Ministral 3 8B	35	30	29	24	21	21	20	18	14	3	21.5%
Ministral 3 14B	32	32	27	24	24	21	14	6	0	0	18.0%
GPT-4o, May 13th (temp=0)	39	35	33	33	32	0	0	0	0	0	17.3%
Qwen 2.5 72B	27	26	20	17	15	15	15	12	9	8	16.4%
Ministral 8B	33	32	24	23	23	12	2	2	2	2	15.3%
Gemma 3 27B	36	26	15	15	14	12	11	9	9	0	14.7%
Claude 3.5 Haiku	24	18	15	15	14	12	11	9	9	0	12.7%
Gemini 2.5 Flash Lite	32	30	26	18	5	0	0	0	0	0	11.1%
Hermes 3 405B	27	18	11	11	8	8	6	6	5	2	10.0%
Hermes 3 70B	14	11	11	8	8	8	5	2	0	0	6.4%
Llama 3.1 Nemotron 70B	12	12	11	8	6	5	3	0	0	0	5.6%
Arcee AI: Trinity Mini	15	9	6	6	5	3	3	2	0	0	4.8%
Cohere Command R+ (Aug. 2024)	12	12	9	6	5	3	0	0	0	0	4.7%
Gemma 3 12B	11	9	8	6	5	0	0	0	0	0	3.8%
GPT-4o Mini (temp=1)	9	6	6	5	2	0	0	0	0	0	2.7%
Llama 3.1 70B	12	6	5	3	2	0	0	0	0	0	2.7%
Claude 3 Haiku	11	8	3	2	2	2	0	0	0	0	2.6%
WizardLM 2 8x22b	9	6	6	2	2	0	0	0	0	0	2.4%
Mistral NeMO	11	0	0	0	0	0	0	0	0	0	1.1%
GPT-4o Mini (temp=0)	8	0	0	0	0	0	0	0	0	0	0.8%
Llama 3.1 8B	5	3	0	0	0	0	0	0	0	0	0.8%
Ministral 3 3B	5	3	0	0	0	0	0	0	0	0	0.8%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Ministral 3B	3	0	0	0	0	0	0	0	0	0	0.3%
Gemma 3 4B	2	0	0	0	0	0	0	0	0	0	0.2%
Arcee AI: Trinity Large (Preview)	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Large codex (40 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	97	97	97	97	98.9%
Claude Opus 4.6	100	100	100	100	97	97	97	97	97	92	97.9%
Grok 4	100	100	100	100	95	95	95	95	95	95	96.8%
MoonshotAI: Kimi K2.5	100	100	100	97	97	95	95	92	92	92	96.1%
Claude Sonnet 4.6	100	100	100	100	95	95	95	95	92	87	95.8%
Claude Sonnet 4	100	100	100	100	92	92	92	92	87	84	93.9%
GPT-5.2	97	97	95	95	95	92	89	89	89	87	92.6%
ByteDance Seed 1.6	100	95	95	95	92	89	89	89	89	84	91.8%
Grok 4 Fast	95	95	95	89	89	89	89	89	89	79	90.0%
Grok 4.1 Fast	95	95	95	92	92	89	89	87	84	79	89.7%
GPT-5	95	95	92	89	89	89	89	89	87	82	89.7%
Qwen 3.5 397B A17B	92	92	92	92	92	92	92	84	84	84	89.7%
Z.AI GLM 4.7	100	92	92	92	89	87	84	84	84	74	87.9%
Z.AI GLM 5	100	100	95	92	89	82	82	79	79	79	87.6%
Gemini 3 Flash (Preview)	92	92	92	92	92	84	84	84	84	76	87.4%
GPT-5.1	92	92	92	89	89	89	84	82	82	82	87.4%
o4 Mini High	95	95	87	87	87	87	84	84	82	82	86.8%
Gemini 3 Pro (Preview)	100	95	92	92	89	89	82	79	76	74	86.8%
Z.AI GLM 4.6	97	92	92	89	89	89	87	84	74	71	86.6%
GPT-4.1	100	100	95	92	92	84	84	76	71	66	86.1%
o4 Mini	95	89	89	87	87	87	82	82	79	74	85.0%
Gemini 2.5 Flash	92	92	89	89	87	82	82	71	71	68	82.4%
Claude Opus 4	95	92	87	84	79	79	79	79	71	68	81.3%
Qwen 3.5 Plus (2026-02-15)	92	92	84	84	84	84	79	79	68	63	81.1%
Writer: Palmyra X5	89	89	87	84	82	79	74	71	68	66	78.9%
Stealth: Aurora Alpha	89	89	84	76	76	76	76	76	76	66	78.7%
GPT-5 Mini	89	89	87	87	87	87	84	79	53	45	78.7%
Z.AI GLM 4.5	87	87	82	82	82	79	76	74	63	63	77.4%
Minimax M2.5	89	89	82	82	79	76	74	71	68	61	77.1%
Claude 3.5 Sonnet	82	82	82	82	74	74	74	66	66	61	73.9%
Claude 3.7 Sonnet	76	74	74	74	74	74	74	71	71	71	73.2%
Mistral Medium 3.1	76	76	74	74	74	74	74	68	66	63	71.8%
Mistral Large	87	84	79	71	68	68	66	66	61	61	71.1%
GPT-4o, May 13th (temp=0)	79	79	74	74	74	68	68	68	63	61	70.8%
DeepSeek V3 (2024-12-26)	84	82	79	71	68	68	66	63	63	61	70.5%
Mistral Large 2	82	79	79	71	66	66	66	66	63	63	70.0%
DeepSeek-V2 Chat	89	76	76	74	68	68	63	63	61	58	69.7%
DeepSeek V3.2	95	84	82	76	74	71	71	63	61	18	69.5%
Z.AI GLM 4.7 Flash	79	74	74	71	68	68	63	63	61	55	67.6%
Mistral Large 3	74	71	71	68	68	68	66	63	63	61	67.4%
GPT-4.1 Mini	76	74	74	71	66	66	63	61	50	34	63.4%
DeepSeek V3 (2025-03-24)	89	89	74	74	68	66	63	58	29	0	61.1%
Claude Haiku 4.5	87	79	79	79	71	71	63	61	0	0	58.9%
ByteDance Seed 1.6 Flash	74	68	66	63	61	61	61	55	50	29	58.7%
GPT-4o, Aug. 6th (temp=0)	74	61	61	58	58	58	55	55	55	53	58.7%
Mistral Small 3.2 24B	68	63	63	61	58	55	55	55	55	53	58.7%
GPT-5 Nano	82	82	76	74	71	68	66	61	0	0	57.9%
DeepSeek V3.1	82	82	82	79	71	71	55	42	0	0	56.3%
Gemini 2.5 Flash Lite	76	68	61	61	58	50	47	47	45	39	55.3%
GPT-4o, Aug. 6th (temp=1)	68	68	66	63	61	58	55	47	45	0	53.2%
GPT-4o, May 13th (temp=1)	74	71	68	53	53	53	45	45	32	32	52.4%
Mistral Small Creative	63	53	53	50	47	47	45	39	39	21	45.8%
Arcee AI: Trinity Large (Preview)	76	61	58	50	50	42	39	37	32	0	44.5%
Ministral 3 8B	53	50	47	47	45	45	42	39	39	0	40.8%
Gemma 3 27B	47	45	42	39	39	37	37	37	37	37	39.7%
Ministral 8B	50	45	42	42	39	37	37	37	34	34	39.7%
Ministral 3 14B	50	47	45	42	39	37	37	34	32	24	38.7%
Hermes 3 405B	50	50	47	45	42	37	34	29	29	21	38.4%
Claude 3.5 Haiku	39	37	34	34	34	34	34	34	32	32	34.5%
Ministral 3B	53	50	45	39	39	26	24	18	13	8	31.6%
GPT-4o Mini (temp=0)	32	32	32	32	32	32	32	32	32	11	29.5%
Gemma 3 12B	37	34	34	29	29	26	26	24	21	11	27.1%
Arcee AI: Trinity Mini	37	37	37	37	34	26	24	16	11	8	26.6%
Qwen 2.5 72B	42	39	39	37	34	32	21	8	3	0	25.5%
Ministral 3 3B	50	39	39	32	29	26	16	16	5	0	25.3%
Llama 3.1 Nemotron 70B	39	32	29	26	26	24	21	18	16	0	23.2%
Llama 3.1 70B	47	34	32	26	21	16	16	13	3	0	20.8%
Hermes 3 70B	34	34	26	26	18	16	16	11	3	0	18.4%
Cohere Command R+ (Aug. 2024)	39	34	32	26	24	8	5	5	0	0	17.4%
GPT-4o Mini (temp=1)	24	21	18	16	13	11	11	5	0	0	11.8%
Claude 3 Haiku	26	18	11	8	5	5	5	3	0	0	8.2%
Mistral NeMO	18	11	0	0	0	0	0	0	0	0	2.9%
Llama 3.1 8B	5	0	0	0	0	0	0	0	0	0	0.5%
GPT-4.1 Nano	3	0	0	0	0	0	0	0	0	0	0.3%
Rocinante 12B	3	0	0	0	0	0	0	0	0	0	0.3%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), long passage (734 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	95	90	98.5%
Claude Opus 4.6	100	100	100	100	100	100	100	100	90	85	97.5%
Gemini 2.5 Pro	100	100	100	100	100	100	100	90	90	80	96.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	90	90	90	85	95.5%
GPT-5.2	100	95	95	95	95	95	95	95	95	95	95.5%
GPT-5 Mini	100	100	95	95	95	95	95	95	95	85	95.0%
GPT-5.1	100	100	100	100	90	90	90	90	85	70	91.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	85	85	85	85	85	85	91.0%
GPT-5	100	100	95	95	90	90	90	85	85	65	89.5%
Z.AI GLM 5	100	100	90	90	90	90	90	90	75	70	88.5%
Grok 4 Fast	100	100	95	95	85	85	85	80	80	75	88.0%
o4 Mini	90	90	90	90	90	90	85	85	85	80	87.5%
Gemini 3 Pro (Preview)	100	90	90	90	90	90	90	80	80	75	87.5%
Claude Sonnet 4.6	100	90	85	85	85	85	85	85	85	85	87.0%
Grok 4.1 Fast	100	100	95	95	90	85	85	80	75	65	87.0%
o4 Mini High	95	90	90	90	90	85	85	80	80	70	85.5%
ByteDance Seed 1.6	90	90	90	90	90	90	90	75	75	75	85.5%
Grok 4	100	95	90	90	90	85	85	65	60	55	81.5%
Claude Sonnet 4.5	100	90	85	85	85	75	75	70	70	70	80.5%
MoonshotAI: Kimi K2.5	90	90	90	90	90	85	80	65	60	60	80.0%
Z.AI GLM 4.7	100	90	85	80	80	75	75	75	70	70	80.0%
Z.AI GLM 4.6	90	90	85	85	85	80	75	75	75	60	80.0%
Claude Haiku 4.5	80	80	80	80	80	80	80	80	80	80	80.0%
Claude Sonnet 4	100	85	85	85	70	70	70	70	70	70	77.5%
Claude 3.7 Sonnet	90	90	80	75	75	75	75	75	65	65	76.5%
Gemini 3 Flash (Preview)	90	75	75	75	75	75	75	75	75	70	76.0%
Gemini 2.5 Flash	90	90	80	80	75	75	70	70	70	55	75.5%
Mistral Large	90	80	75	75	75	65	65	60	60	60	70.5%
Claude 3.5 Sonnet	80	75	70	70	70	70	70	65	65	65	70.0%
Claude Opus 4	90	75	75	75	75	65	65	60	60	60	70.0%
Minimax M2.5	80	80	75	75	70	70	65	65	60	60	70.0%
Z.AI GLM 4.5	75	75	75	75	75	70	70	65	60	55	69.5%
ByteDance Seed 1.6 Flash	90	80	75	70	70	65	65	60	55	55	68.5%
Mistral Medium 3.1	75	75	75	75	75	75	75	70	70	0	66.5%
GPT-4.1 Mini	70	70	70	70	70	70	65	65	65	50	66.5%
Mistral Large 3	85	75	65	65	65	60	60	60	60	60	65.5%
GPT-4o, May 13th (temp=0)	70	70	70	65	65	65	65	65	65	50	65.0%
Mistral Large 2	80	75	65	65	65	65	65	60	50	50	64.0%
Llama 3.1 Nemotron 70B	75	65	65	65	65	65	60	60	60	60	64.0%
Stealth: Aurora Alpha	85	80	75	75	75	75	75	75	10	10	63.5%
Mistral Small 3.2 24B	75	75	75	70	65	65	55	55	50	50	63.5%
GPT-4.1	75	65	65	65	65	65	65	60	55	50	63.0%
GPT-4o, Aug. 6th (temp=0)	65	65	65	65	65	65	65	60	60	55	63.0%
Z.AI GLM 4.7 Flash	70	70	70	70	65	65	60	55	55	45	62.5%
DeepSeek-V2 Chat	80	70	70	70	70	60	55	50	45	45	61.5%
GPT-4o, Aug. 6th (temp=1)	70	70	65	65	60	55	55	55	55	50	60.0%
Ministral 3 14B	80	80	75	75	65	65	65	60	5	5	57.5%
Writer: Palmyra X5	70	70	60	60	60	55	55	55	45	45	57.5%
Llama 3.1 70B	70	65	60	55	55	55	55	55	55	50	57.5%
Mistral Small Creative	75	70	70	60	55	55	55	50	45	40	57.5%
GPT-4o, May 13th (temp=1)	70	65	60	60	60	60	55	50	50	40	57.0%
GPT-5 Nano	75	65	65	60	60	55	55	45	45	45	57.0%
DeepSeek V3 (2025-03-24)	70	70	60	55	55	55	50	45	45	45	55.0%
DeepSeek V3 (2024-12-26)	70	65	60	55	55	55	55	45	45	45	55.0%
Gemini 2.5 Flash Lite	75	60	60	55	55	55	55	50	40	35	54.0%
Qwen 2.5 72B	70	60	60	60	55	55	50	50	45	35	54.0%
Gemma 3 27B	70	65	55	55	55	50	50	45	45	45	53.5%
Claude 3.5 Haiku	60	60	60	55	50	50	50	45	45	40	51.5%
DeepSeek V3.2	65	65	65	55	55	55	45	40	35	30	51.0%
Arcee AI: Trinity Large (Preview)	70	55	55	55	55	50	50	45	35	20	49.0%
Hermes 3 405B	65	55	50	45	45	45	45	40	40	40	47.0%
DeepSeek V3.1	70	55	55	55	50	45	40	40	40	0	45.0%
Ministral 3 3B	50	45	45	40	40	40	35	35	30	30	39.0%
Ministral 3 8B	55	50	45	45	40	35	30	25	20	5	35.0%
Ministral 8B	65	45	45	45	35	30	30	25	0	0	32.0%
Gemma 3 12B	50	35	35	35	25	25	25	20	20	15	28.5%
Ministral 3B	55	40	30	30	30	30	30	15	15	5	28.0%
GPT-4o Mini (temp=1)	35	35	35	35	35	25	25	20	10	10	26.5%
Hermes 3 70B	45	40	35	35	30	30	25	20	5	0	26.5%
GPT-4o Mini (temp=0)	35	25	25	25	25	25	25	25	25	25	26.0%
Arcee AI: Trinity Mini	45	35	30	30	25	20	20	20	20	10	25.5%
Cohere Command R+ (Aug. 2024)	50	45	40	40	35	30	0	0	0	0	24.0%
Mistral NeMO	50	40	40	35	25	25	0	0	0	0	21.5%
Llama 3.1 8B	35	30	30	30	30	20	15	10	0	0	20.0%
Claude 3 Haiku	30	30	20	20	20	20	20	20	15	5	20.0%
Gemma 3 4B	30	25	20	10	10	5	0	0	0	0	10.0%
Rocinante 12B	50	45	0	0	0	0	0	0	0	0	9.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	95	99.5%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	90	99.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	95	90	98.5%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	95	90	98.5%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	80	98.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	90	90	98.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	90	90	98.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	85	85	97.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	85	85	97.0%
Z.AI GLM 4.6	100	100	100	100	100	100	95	90	90	90	96.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	95	95	95	95	95	95	95	96.5%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	90	90	85	96.5%
DeepSeek V3.2	100	100	100	100	100	100	100	90	85	80	95.5%
o4 Mini High	100	100	100	100	100	90	90	90	90	90	95.0%
Z.AI GLM 4.5	100	100	100	100	100	90	90	90	90	90	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	85	85	85	85	94.0%
o4 Mini	100	100	100	90	90	90	90	90	90	90	93.0%
Grok 4	100	100	90	90	90	90	90	90	90	90	92.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	90	90	85	85	85	85	92.0%
DeepSeek V3.1	100	100	100	100	90	90	85	85	85	85	92.0%
Llama 3.1 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Llama 3.1 Nemotron 70B	100	90	90	90	90	90	90	90	90	90	91.0%
Claude Opus 4	100	100	100	100	85	85	85	85	85	85	91.0%
Gemini 2.5 Flash	90	90	90	90	90	90	90	90	90	90	90.0%
Mistral Large	100	90	90	90	90	90	90	85	85	85	89.5%
Mistral Large 2	90	90	90	90	90	90	90	90	85	85	89.0%
Hermes 3 405B	100	100	100	90	90	90	90	85	80	60	88.5%
Mistral Large 3	90	90	90	90	90	85	85	85	85	85	87.5%
GPT-4o, May 13th (temp=1)	100	100	100	90	85	85	85	85	65	65	86.0%
Gemma 3 12B	90	90	90	90	90	90	80	80	80	75	85.5%
Minimax M2.5	100	100	90	90	90	85	75	75	75	70	85.0%
Claude Sonnet 4.6	85	85	85	85	85	85	85	85	85	85	85.0%
GPT-4.1 Mini	90	90	90	90	85	80	80	75	75	75	83.0%
ByteDance Seed 1.6 Flash	90	90	90	90	80	80	80	80	75	65	82.0%
Stealth: Aurora Alpha	90	90	90	90	90	90	90	90	90	0	81.0%
Mistral Small 3.2 24B	90	90	90	80	75	75	75	75	75	75	80.0%
GPT-5 Nano	90	90	80	80	80	80	70	70	70	70	78.0%
Qwen 2.5 72B	90	90	90	80	80	80	80	80	70	35	77.5%
Z.AI GLM 4.7 Flash	100	90	90	80	75	75	75	65	60	55	76.5%
Arcee AI: Trinity Large (Preview)	90	85	85	80	75	70	70	70	70	65	76.0%
Writer: Palmyra X5	85	85	85	85	75	75	70	65	65	65	75.5%
Gemini 2.5 Flash Lite	75	75	75	75	75	75	75	75	75	70	74.5%
Mistral Small Creative	80	80	70	70	70	70	70	70	70	65	71.5%
Cohere Command R+ (Aug. 2024)	90	90	80	70	70	70	65	60	60	55	71.0%
Claude Haiku 4.5	75	75	75	75	75	75	75	60	60	60	70.5%
Gemma 3 27B	75	75	70	65	65	65	65	55	55	55	64.5%
Ministral 3 8B	70	70	65	65	60	60	60	50	50	35	58.5%
Ministral 3 14B	85	75	65	60	60	60	60	55	45	5	57.0%
Ministral 8B	75	75	65	65	60	60	55	55	35	20	56.5%
GPT-4o Mini (temp=0)	60	60	60	60	60	60	50	50	50	50	56.0%
Ministral 3 3B	65	65	65	65	60	60	55	55	40	30	56.0%
Claude 3.5 Haiku	65	65	65	55	55	55	55	55	40	25	53.5%
Hermes 3 70B	80	75	65	60	60	60	50	45	40	0	53.5%
GPT-4o Mini (temp=1)	60	60	60	50	50	50	50	50	50	35	51.5%
Ministral 3B	65	60	55	50	50	50	50	45	40	35	50.0%
Llama 3.1 8B	75	70	65	60	60	40	40	35	30	20	49.5%
Mistral NeMO	65	55	55	45	45	45	45	45	45	45	49.0%
Arcee AI: Trinity Mini	60	60	55	50	45	45	40	35	35	35	46.0%
Claude 3 Haiku	45	45	45	40	35	35	35	35	20	15	35.0%
Rocinante 12B	55	40	35	15	15	15	5	5	5	0	19.0%
GPT-4.1 Nano	20	20	15	10	10	10	10	10	0	0	10.5%
WizardLM 2 8x22b	10	10	0	0	0	0	0	0	0	0	2.0%
Gemma 3 4B	5	0	0	0	0	0	0	0	0	0	0.5%

tiers

▼

5 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	90	99.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	90	99.0%
Grok 4	100	100	100	100	100	100	100	90	90	90	97.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	70	97.0%
GPT-5	100	100	100	100	100	100	90	90	90	90	96.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	90	70	96.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	80	80	96.0%
DeepSeek V3.2	100	100	100	100	100	90	90	90	90	90	95.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	70	70	94.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	30	93.0%
Mistral Large	100	100	100	90	90	90	90	90	90	90	93.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	90	80	80	70	92.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	90	30	92.0%
Llama 3.1 70B	100	100	100	100	100	100	90	80	70	70	91.0%
Mistral Large 3	90	90	90	90	90	90	90	90	90	90	90.0%
Mistral Large 2	90	90	90	90	90	90	90	90	90	90	90.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	80	80	80	60	90.0%
Claude Opus 4	100	100	100	100	100	70	70	70	70	70	85.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	80	80	70	70	50	85.0%
Stealth: Aurora Alpha	100	100	100	100	100	80	80	80	80	30	85.0%
DeepSeek V3.1	100	100	100	100	100	100	90	80	70	0	84.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	90	90	70	60	60	60	83.0%
ByteDance Seed 1.6 Flash	100	80	80	80	80	80	80	80	80	70	81.0%
Gemini 2.5 Flash	100	80	80	80	80	80	80	80	70	70	80.0%
Minimax M2.5	100	100	90	80	70	70	70	50	50	50	73.0%
GPT-5 Nano	100	100	100	80	80	80	80	50	50	0	72.0%
Mistral Medium 3.1	100	70	70	70	70	70	70	70	60	60	71.0%
Claude Sonnet 4.6	70	70	70	70	70	70	70	70	70	70	70.0%
Gemma 3 27B	70	70	70	70	70	70	70	70	70	70	70.0%
GPT-4o, Aug. 6th (temp=0)	90	90	90	60	60	60	60	60	60	60	69.0%
GPT-4.1 Mini	80	80	70	70	70	70	70	70	70	30	68.0%
GPT-4o, May 13th (temp=1)	90	90	70	70	60	60	60	60	60	60	68.0%
Qwen 2.5 72B	80	80	80	80	80	80	50	50	50	50	68.0%
GPT-4o, Aug. 6th (temp=1)	90	90	70	60	60	60	60	60	60	60	67.0%
Arcee AI: Trinity Mini	100	80	80	80	60	60	50	50	30	30	62.0%
Writer: Palmyra X5	70	70	70	70	70	70	60	60	60	0	60.0%
GPT-4o Mini (temp=1)	80	80	80	80	50	50	50	50	40	40	60.0%
Arcee AI: Trinity Large (Preview)	70	70	70	70	60	60	50	50	40	40	58.0%
Claude Haiku 4.5	70	70	70	70	60	60	50	40	40	40	57.0%
GPT-4o Mini (temp=0)	80	80	50	50	50	50	50	50	50	50	56.0%
Gemma 3 12B	80	50	50	50	50	50	50	50	50	50	53.0%
Mistral Small Creative	60	60	60	60	60	60	50	50	40	30	53.0%
Gemini 2.5 Flash Lite	50	50	40	40	40	40	40	40	40	40	42.0%
Llama 3.1 8B	70	70	70	70	40	30	30	20	20	0	42.0%
Cohere Command R+ (Aug. 2024)	80	60	60	60	50	40	20	20	20	0	41.0%
Ministral 3 3B	70	70	60	50	40	40	40	30	0	0	40.0%
Mistral Small 3.2 24B	60	50	50	40	40	40	30	30	30	10	38.0%
Mistral NeMO	70	50	50	50	40	40	30	30	10	0	37.0%
Claude 3 Haiku	60	60	40	40	40	30	20	20	10	10	33.0%
Ministral 3 14B	50	50	40	40	40	20	20	20	20	10	31.0%
Ministral 3B	70	70	60	40	20	20	10	10	0	0	30.0%
Hermes 3 70B	60	50	50	30	30	20	20	10	0	0	27.0%
GPT-4.1 Nano	20	20	20	20	0	0	0	0	0	0	8.0%
Ministral 8B	50	10	10	0	0	0	0	0	0	0	7.0%
Rocinante 12B	40	20	10	0	0	0	0	0	0	0	7.0%
Gemma 3 4B	20	20	20	0	0	0	0	0	0	0	6.0%
Ministral 3 8B	10	0	0	0	0	0	0	0	0	0	1.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%

▼

10 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	92	99.2%
GPT-5.2	100	100	100	100	100	100	100	100	100	92	99.2%
Grok 4	100	100	100	100	100	100	100	100	100	92	99.2%
Claude Opus 4	100	100	100	100	100	100	100	100	100	92	99.2%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	92	99.2%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	75	97.5%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	83	83	96.7%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	83	83	96.7%
GPT-5.1	100	100	100	100	92	92	92	92	92	92	95.0%
GPT-5	100	100	100	100	92	92	92	92	92	92	95.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	83	83	83	95.0%
o4 Mini	100	100	100	100	100	100	92	92	75	75	93.3%
Minimax M2.5	100	100	100	100	100	100	100	83	75	75	93.3%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	33	93.3%
GPT-5 Mini	100	92	92	92	92	92	92	92	92	92	92.5%
o4 Mini High	100	100	100	100	100	92	92	92	75	75	92.5%
Llama 3.1 Nemotron 70B	100	100	100	100	100	92	83	83	83	83	92.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	75	75	75	75	90.0%
Writer: Palmyra X5	100	100	100	100	100	100	75	75	75	75	90.0%
Z.AI GLM 4.5	100	100	100	100	100	92	75	75	75	75	89.2%
Z.AI GLM 4.6	100	100	100	100	100	100	92	92	75	8	86.7%
DeepSeek V3 (2025-03-24)	100	100	100	92	83	83	83	75	75	75	86.7%
Mistral Small Creative	100	100	100	92	92	83	83	83	75	58	86.7%
ByteDance Seed 1.6 Flash	100	92	83	83	83	83	83	83	83	67	84.2%
GPT-4.1 Mini	92	92	92	92	92	75	75	75	75	67	82.5%
Mistral Large	100	100	100	92	75	75	75	75	67	67	82.5%
Gemini 2.5 Flash	92	92	92	92	83	75	75	75	75	67	81.7%
GPT-4o, Aug. 6th (temp=1)	100	83	83	83	83	83	83	83	67	67	81.7%
GPT-4o, Aug. 6th (temp=0)	83	83	83	83	83	83	83	83	83	67	81.7%
GPT-5 Nano	100	100	83	75	75	75	75	75	75	75	80.8%
GPT-4.1	100	100	75	75	75	75	75	75	75	75	80.0%
DeepSeek V3.2	100	100	100	92	92	92	92	58	50	25	80.0%
Gemma 3 27B	100	100	92	75	75	75	75	75	75	50	79.2%
Ministral 3 8B	83	83	83	75	75	75	75	75	67	67	75.8%
Claude Sonnet 4.6	75	75	75	75	75	75	75	75	75	75	75.0%
Claude Haiku 4.5	75	75	75	75	75	75	75	75	75	75	75.0%
DeepSeek-V2 Chat	100	100	83	75	75	75	67	58	58	58	75.0%
Llama 3.1 70B	83	83	83	83	83	83	83	83	75	0	74.2%
Mistral Large 3	92	75	75	75	75	75	67	67	67	67	73.3%
Mistral Small 3.2 24B	100	92	83	83	75	75	75	58	50	42	73.3%
Z.AI GLM 4.7 Flash	100	83	67	67	67	67	67	67	67	58	70.8%
Mistral Large 2	100	75	75	75	75	67	67	67	50	50	70.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	92	0	0	0	69.2%
DeepSeek V3.1	92	92	83	75	75	75	67	58	42	33	69.2%
Hermes 3 405B	100	83	75	75	75	75	75	67	58	0	68.3%
Gemma 3 12B	92	75	75	75	75	75	67	67	50	33	68.3%
Claude 3.5 Haiku	83	83	83	83	67	67	67	67	50	25	67.5%
Ministral 8B	100	83	75	75	75	67	58	50	50	42	67.5%
DeepSeek V3 (2024-12-26)	92	83	75	75	75	67	58	58	42	42	66.7%
WizardLM 2 8x22b	100	92	83	75	75	67	58	50	50	0	65.0%
Gemini 2.5 Flash Lite	83	83	75	67	67	58	58	58	50	42	64.2%
GPT-4o, May 13th (temp=1)	100	83	83	83	83	67	67	17	17	17	61.7%
Arcee AI: Trinity Large (Preview)	75	75	67	58	50	50	50	50	50	42	56.7%
GPT-4o Mini (temp=1)	67	67	67	67	50	50	50	50	50	42	55.8%
Arcee AI: Trinity Mini	83	83	67	50	50	50	50	50	33	33	55.0%
Qwen 2.5 72B	83	67	67	67	67	67	50	25	25	25	54.2%
Hermes 3 70B	75	75	67	67	67	50	50	42	25	0	51.7%
GPT-4o Mini (temp=0)	50	50	50	50	50	50	50	50	50	50	50.0%
Cohere Command R+ (Aug. 2024)	75	67	67	58	58	42	33	33	8	0	44.2%
Claude 3 Haiku	58	58	58	58	42	33	25	25	25	25	40.8%
Ministral 3 14B	50	50	50	50	42	42	33	33	25	17	39.2%
Mistral NeMO	67	67	42	42	33	33	33	25	25	17	38.3%
Ministral 3B	42	42	33	33	8	8	0	0	0	0	16.7%
Ministral 3 3B	75	25	17	17	17	8	0	0	0	0	15.8%
Rocinante 12B	42	42	25	25	17	0	0	0	0	0	15.0%
Gemma 3 4B	25	25	17	8	8	0	0	0	0	0	8.3%
GPT-4.1 Nano	17	0	0	0	0	0	0	0	0	0	1.7%
Llama 3.1 8B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

20 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	95	95	99.1%
Gemini 2.5 Pro	100	100	100	95	95	95	95	95	95	95	96.8%
Claude Sonnet 4.5	100	100	100	100	100	100	86	86	86	86	94.5%
GPT-5.2	95	95	95	95	91	91	86	86	86	86	90.9%
Gemini 3 Pro (Preview)	100	95	95	91	91	91	86	86	86	86	90.9%
Grok 4	100	100	91	91	91	91	91	91	82	77	90.5%
GPT-5	91	91	91	91	91	91	91	91	91	86	90.5%
MoonshotAI: Kimi K2.5	95	95	95	91	91	86	86	86	86	82	89.5%
GPT-5.1	91	91	91	91	91	91	91	86	86	82	89.1%
Claude 3.5 Sonnet	95	95	95	95	95	91	86	82	77	73	88.6%
Gemini 2.5 Flash	100	86	86	86	86	86	86	86	86	86	87.7%
Mistral Medium 3.1	95	95	86	86	86	86	86	86	82	77	86.8%
Z.AI GLM 4.7	95	95	86	86	86	86	82	82	82	82	86.4%
Gemini 3 Flash (Preview)	86	86	86	86	86	86	86	86	86	86	86.4%
Z.AI GLM 5	100	91	91	91	91	86	82	82	73	73	85.9%
Claude Sonnet 4	86	86	86	86	86	86	86	86	82	82	85.5%
Grok 4.1 Fast	91	91	91	91	91	86	82	82	73	73	85.0%
Claude Sonnet 4.6	100	100	91	91	86	82	82	77	68	68	84.5%
o4 Mini High	91	91	91	91	91	82	77	77	77	73	84.1%
Claude Opus 4	95	95	95	82	82	82	82	73	73	73	83.2%
GPT-4o, Aug. 6th (temp=0)	86	86	86	86	86	86	77	77	77	77	82.7%
Mistral Large	95	95	95	86	82	82	77	68	64	64	80.9%
Grok 4 Fast	100	95	91	91	91	91	82	82	82	0	80.5%
o4 Mini	91	82	82	82	82	82	77	77	77	73	80.5%
ByteDance Seed 1.6	86	86	77	77	77	77	77	77	77	77	79.1%
Qwen 3.5 Plus (2026-02-15)	91	91	86	82	82	77	77	77	77	50	79.1%
GPT-4o, Aug. 6th (temp=1)	91	91	91	91	86	77	73	68	59	59	78.6%
DeepSeek V3.2	100	91	91	91	77	73	73	68	64	55	78.2%
Mistral Large 2	86	86	86	82	82	82	77	73	64	59	77.7%
GPT-5 Mini	86	82	77	77	77	77	77	77	73	68	77.3%
Z.AI GLM 4.6	100	95	86	86	82	82	73	59	59	45	76.8%
Qwen 3.5 397B A17B	95	86	86	86	86	86	82	82	73	0	76.4%
GPT-5 Nano	91	82	82	82	77	77	73	68	64	55	75.0%
Claude Haiku 4.5	91	91	91	82	82	77	73	68	68	0	72.3%
Z.AI GLM 4.7 Flash	91	77	73	73	73	68	68	68	64	59	71.4%
Writer: Palmyra X5	86	82	82	77	68	68	68	68	59	55	71.4%
Stealth: Aurora Alpha	73	73	73	73	73	73	73	68	68	64	70.9%
Mistral Large 3	91	77	77	77	68	64	64	64	64	64	70.9%
DeepSeek V3.1	95	95	86	82	82	77	64	45	41	41	70.9%
Minimax M2.5	82	82	77	73	68	68	64	64	64	59	70.0%
Claude 3.7 Sonnet	100	100	86	86	77	64	59	50	45	32	70.0%
Z.AI GLM 4.5	91	73	68	68	68	68	64	64	59	59	68.2%
ByteDance Seed 1.6 Flash	82	73	73	73	73	73	64	59	59	45	67.3%
GPT-4o, May 13th (temp=0)	77	77	77	73	68	68	59	59	59	50	66.8%
GPT-4.1	95	82	82	82	82	82	77	64	0	0	64.5%
Gemini 2.5 Flash Lite	100	95	91	77	73	73	64	55	0	0	62.7%
DeepSeek V3 (2024-12-26)	82	73	73	68	68	68	59	45	45	36	61.8%
Ministral 3 14B	82	82	77	68	68	45	45	45	41	41	59.5%
GPT-4.1 Mini	77	68	68	68	59	59	55	41	41	36	57.3%
Arcee AI: Trinity Large (Preview)	77	73	73	64	55	55	50	50	45	27	56.8%
GPT-4o, May 13th (temp=1)	73	68	59	59	59	55	50	45	45	41	55.5%
Mistral Small Creative	64	64	64	59	55	55	50	50	50	41	55.0%
DeepSeek-V2 Chat	73	68	68	68	59	55	50	45	41	0	52.7%
Mistral Small 3.2 24B	64	59	59	59	55	50	50	45	45	41	52.7%
DeepSeek V3 (2025-03-24)	68	68	59	59	59	55	45	45	45	9	51.4%
Claude 3.5 Haiku	55	55	55	55	55	55	55	45	41	41	50.9%
Hermes 3 405B	82	73	68	64	59	50	36	27	23	23	50.5%
Gemma 3 27B	59	59	55	45	45	41	41	41	41	41	46.8%
Llama 3.1 Nemotron 70B	59	59	59	45	41	41	36	36	36	27	44.1%
Llama 3.1 70B	59	59	55	55	45	45	45	27	27	9	42.7%
Cohere Command R+ (Aug. 2024)	64	59	55	55	45	45	41	27	23	14	42.7%
Ministral 3 8B	73	64	45	41	41	32	32	23	18	14	38.2%
Ministral 8B	68	64	55	50	41	41	32	23	0	0	37.3%
WizardLM 2 8x22b	59	45	41	41	41	41	36	27	23	9	36.4%
Arcee AI: Trinity Mini	45	45	45	45	41	36	36	18	14	9	33.6%
Hermes 3 70B	50	41	36	36	32	23	18	0	0	0	23.6%
GPT-4o Mini (temp=0)	32	23	23	18	18	18	18	18	18	18	20.5%
GPT-4o Mini (temp=1)	50	32	32	23	23	18	14	5	0	0	19.5%
Gemma 3 12B	32	27	27	27	23	14	14	9	5	0	17.7%
Ministral 3B	32	32	27	23	23	14	14	9	5	0	17.7%
Claude 3 Haiku	41	27	23	18	14	14	9	9	5	0	15.9%
Ministral 3 3B	32	23	18	18	18	14	5	0	0	0	12.7%
Llama 3.1 8B	41	32	27	14	0	0	0	0	0	0	11.4%
Mistral NeMO	27	27	27	14	9	5	5	0	0	0	11.4%
Qwen 2.5 72B	73	9	9	0	0	0	0	0	0	0	9.1%
Gemma 3 4B	5	0	0	0	0	0	0	0	0	0	0.5%
Rocinante 12B	5	0	0	0	0	0	0	0	0	0	0.5%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%

▼

40 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	95	95	99.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	85	98.5%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	85	98.5%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	95	85	98.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	95	85	85	96.5%
o4 Mini High	100	100	100	100	100	95	95	90	90	90	96.0%
GPT-5.2	100	100	100	95	95	95	95	95	95	90	96.0%
GPT-5	100	100	100	100	100	100	95	90	85	85	95.5%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	85	85	85	95.5%
Claude Opus 4.5	100	100	100	100	100	90	90	90	90	90	95.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	90	85	75	95.0%
o4 Mini	100	100	100	100	100	90	90	90	90	85	94.5%
Z.AI GLM 5	100	100	100	100	100	90	90	90	85	85	94.0%
GPT-5.1	100	100	100	100	95	90	90	90	90	80	93.5%
ByteDance Seed 1.6	100	100	100	100	100	100	90	85	85	75	93.5%
Gemini 2.5 Flash	100	100	95	95	90	90	90	90	90	80	92.0%
Mistral Large 2	100	90	90	90	90	90	85	80	80	80	87.5%
Claude Sonnet 4	90	90	90	90	90	90	90	80	80	80	87.0%
Claude Sonnet 4.6	90	90	90	90	90	90	90	90	75	75	87.0%
Claude 3.7 Sonnet	90	90	90	90	90	90	90	90	75	75	87.0%
Mistral Large 3	100	90	90	90	90	85	80	75	75	75	85.0%
Qwen 3.5 Plus (2026-02-15)	100	100	85	85	85	85	85	80	70	70	84.5%
Claude Opus 4	100	100	90	85	85	85	85	75	70	65	84.0%
Gemini 3 Flash (Preview)	100	85	85	85	85	85	85	70	70	70	82.0%
Stealth: Aurora Alpha	90	90	90	90	90	85	80	75	75	50	81.5%
Mistral Large	90	90	90	90	90	90	80	65	65	65	81.5%
Z.AI GLM 4.6	100	100	100	100	90	85	85	75	55	15	80.5%
GPT-4o, May 13th (temp=0)	80	80	80	80	80	80	80	80	70	70	78.0%
DeepSeek V3.2	95	85	85	80	80	75	70	65	60	60	75.5%
Minimax M2.5	90	90	80	80	80	70	70	65	60	55	74.0%
Claude 3.5 Sonnet	90	80	75	75	75	75	75	65	65	65	74.0%
Z.AI GLM 4.7 Flash	90	90	85	85	80	70	70	65	45	40	72.0%
DeepSeek-V2 Chat	90	90	85	85	85	80	70	55	40	40	72.0%
Ministral 3 14B	85	80	80	75	75	70	70	70	55	45	70.5%
Claude Haiku 4.5	90	90	85	75	75	75	75	70	60	0	69.5%
GPT-5 Nano	95	90	80	65	65	65	65	60	55	50	69.0%
GPT-4o, Aug. 6th (temp=1)	80	80	80	80	75	70	65	55	55	40	68.0%
DeepSeek V3 (2024-12-26)	90	85	85	70	65	65	60	55	55	45	67.5%
GPT-4o, Aug. 6th (temp=0)	80	80	70	70	70	65	65	65	55	40	66.0%
Mistral Medium 3.1	90	85	75	70	70	65	55	55	45	40	65.0%
GPT-4.1	100	75	70	70	65	65	65	60	50	30	65.0%
DeepSeek V3.1	90	85	75	70	70	65	65	60	35	30	64.5%
ByteDance Seed 1.6 Flash	80	80	80	70	70	65	60	60	45	35	64.5%
DeepSeek V3 (2025-03-24)	85	80	75	70	65	60	55	55	50	45	64.0%
Writer: Palmyra X5	75	75	70	65	65	65	60	60	50	40	62.5%
GPT-4o, May 13th (temp=1)	75	70	70	65	60	55	55	55	50	45	60.0%
GPT-4.1 Mini	70	65	65	65	60	60	55	55	50	45	59.0%
Gemma 3 27B	65	60	60	60	55	55	55	55	55	50	57.0%
Z.AI GLM 4.5	80	75	75	75	60	55	50	45	15	10	54.0%
Mistral Small Creative	70	70	65	65	65	65	50	25	15	15	50.5%
Ministral 3 8B	70	70	65	65	65	60	35	25	25	0	48.0%
Hermes 3 405B	55	50	50	50	50	50	45	40	40	35	46.5%
Claude 3.5 Haiku	60	50	50	50	50	50	50	50	40	0	45.0%
Mistral Small 3.2 24B	55	50	50	50	50	35	35	35	35	0	39.5%
Ministral 3 3B	55	50	50	50	50	45	35	25	20	10	39.0%
Llama 3.1 Nemotron 70B	65	50	50	50	35	35	30	15	10	0	34.0%
Qwen 2.5 72B	50	45	45	40	35	35	35	25	15	10	33.5%
WizardLM 2 8x22b	50	50	45	35	30	30	30	15	15	10	31.0%
Llama 3.1 70B	60	50	40	40	35	30	20	15	10	0	30.0%
Arcee AI: Trinity Large (Preview)	65	60	50	50	35	30	0	0	0	0	29.0%
Ministral 3B	50	40	35	30	20	20	20	15	15	15	26.0%
Ministral 8B	55	55	45	40	25	0	0	0	0	0	22.0%
Hermes 3 70B	40	40	30	25	20	15	15	10	10	5	21.0%
Gemini 2.5 Flash Lite	45	40	30	25	25	20	10	0	0	0	19.5%
Cohere Command R+ (Aug. 2024)	35	35	20	20	20	5	0	0	0	0	13.5%
Arcee AI: Trinity Mini	30	25	20	15	15	10	10	0	0	0	12.5%
GPT-4o Mini (temp=0)	30	25	20	20	5	5	5	5	5	0	12.0%
GPT-4o Mini (temp=1)	20	15	15	5	5	5	5	0	0	0	7.0%
Llama 3.1 8B	30	20	0	0	0	0	0	0	0	0	5.0%
Mistral NeMO	35	5	0	0	0	0	0	0	0	0	4.0%
Gemma 3 12B	15	5	5	5	5	0	0	0	0	0	3.5%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

Accuracy (recall)

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

matrix

Large codex (40 entries), long passage (1,019 words)

Large codex (40 entries), short passage (165 words)

Small codex (7 entries), long passage (734 words)

Small codex (7 entries), short passage (165 words)

tiers

5 codex entries

10 codex entries

20 codex entries

40 codex entries