Precision

Test: Codex Violation Detection

Avg. Score

80.7%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Grok 4.1 Fast	98.0%	$0.0021	21.1s	90%
2	ByteDance Seed 1.6 Flash	94.1%	$0.0009	12.0s	85%
3	Gemini 3 Flash (Preview)	93.9%	$0.0031	4.5s	83%
4	Grok 4 Fast	96.8%	$0.0018	12.8s	76%
5	Gemini 2.5 Flash	91.4%	$0.0025	2.8s	77%
6	Minimax M2.5	93.5%	$0.0020	25.5s	84%
7	Claude Opus 4.5	99.8%	$0.041	9.7s	98%
8	Claude Sonnet 4.5	96.3%	$0.024	8.9s	88%
9	Claude Sonnet 4	95.6%	$0.023	9.0s	88%
10	o4 Mini	97.4%	$0.019	28.1s	90%
11	Claude Opus 4.6	98.5%	$0.040	10.2s	94%
12	Mistral Large 3	87.5%	$0.0030	10.2s	74%
13	Stealth: Aurora Alpha	92.9%	—	6.2s	74%
14	GPT-5.2	95.7%	$0.024	24.2s	89%
15	Gemini 2.5 Pro	98.4%	$0.035	23.2s	93%
16	Claude 3.5 Haiku	89.8%	$0.0049	6.2s	68%
17	ByteDance Seed 1.6	96.8%	$0.0067	1.0m	91%
18	DeepSeek V3 (2024-12-26)	89.6%	$0.0019	15.6s	70%
19	Mistral Large	89.9%	$0.012	9.1s	75%
20	Z.AI GLM 5	97.9%	$0.013	56.4s	90%
21	DeepSeek-V2 Chat	89.5%	$0.0020	13.1s	66%
22	GPT-5 Mini	94.8%	$0.0092	51.7s	86%
23	Z.AI GLM 4.7	96.2%	$0.0091	1.0m	89%
24	GPT-4.1 Mini	84.4%	$0.0015	5.8s	66%
25	Mistral Large 2	87.9%	$0.012	8.8s	72%
26	Z.AI GLM 4.5	88.7%	$0.0026	18.5s	68%
27	Z.AI GLM 4.7 Flash	94.0%	$0.0018	1.1m	86%
28	Claude Sonnet 4.6	91.5%	$0.024	9.9s	77%
29	Mistral Medium 3.1	85.4%	$0.0029	7.7s	64%
30	DeepSeek V3.2	86.6%	$0.0013	17.2s	66%
31	Claude 3.5 Sonnet	94.4%	$0.042	10.5s	85%
32	GPT-4o, Aug. 6th (temp=1)	87.6%	$0.011	3.6s	62%
33	GPT-4.1	89.1%	$0.0081	10.0s	61%
34	Writer: Palmyra X5	84.3%	$0.0062	12.1s	66%
35	Qwen 3.5 Plus (2026-02-15)	89.3%	$0.0041	34.2s	66%
36	o4 Mini High	97.1%	$0.033	51.3s	90%
37	GPT-4o, Aug. 6th (temp=0)	87.3%	$0.015	5.0s	60%
38	GPT-5.1	97.3%	$0.037	52.8s	90%
39	DeepSeek V3 (2025-03-24)	86.8%	$0.0015	22.9s	56%
40	Gemini 3 Pro (Preview)	97.1%	$0.050	34.0s	91%
41	Gemma 3 27B	78.6%	$0.0005	13.3s	60%
42	Mistral Small Creative	74.7%	$0.0006	4.3s	56%
43	MoonshotAI: Kimi K2.5	95.9%	$0.015	1.5m	88%
44	GPT-4o, May 13th (temp=1)	85.5%	$0.026	3.7s	64%
45	Hermes 3 405B	82.2%	$0.0044	17.2s	53%
46	Z.AI GLM 4.6	91.9%	$0.0049	1.3m	76%
47	Arcee AI: Trinity Mini	80.9%	$0.0004	10.3s	47%
48	Claude Haiku 4.5	81.8%	$0.0078	5.8s	49%
49	Claude 3.7 Sonnet	88.5%	$0.023	10.2s	56%
50	GPT-4o, May 13th (temp=0)	88.6%	$0.030	5.4s	54%
51	Gemini 3.1 Pro (Preview)	99.3%	$0.068	52.3s	96%
52	Mistral Small 3.2 24B	72.3%	$0.0006	10.4s	49%
53	Qwen 2.5 72B	78.0%	$0.0008	13.9s	41%
54	Llama 3.1 Nemotron 70B	78.2%	$0.0055	18.5s	45%
55	Grok 4	97.4%	$0.051	1.2m	87%
56	DeepSeek V3.1	79.4%	$0.0014	31.1s	44%
57	Ministral 3 14B	67.1%	$0.0010	6.3s	41%
58	GPT-4o Mini (temp=1)	67.1%	$0.0006	7.4s	34%
59	Gemini 2.5 Flash Lite	64.3%	$0.0005	2.3s	31%
60	Llama 3.1 70B	72.2%	$0.0021	24.3s	32%
61	Ministral 3 8B	58.4%	$0.0007	4.8s	36%
62	Gemma 3 12B	62.3%	$0.0003	12.0s	34%
63	GPT-4o Mini (temp=0)	69.1%	$0.0006	25.0s	32%
64	GPT-5	96.2%	$0.061	1.6m	87%
65	GPT-5 Nano	88.5%	$0.0049	1.9m	57%
66	Ministral 3 3B	56.1%	$0.0005	3.3s	29%
67	Ministral 8B	52.4%	$0.0005	5.6s	31%
68	Ministral 3B	53.5%	$0.0002	2.9s	28%
69	Claude 3 Haiku	55.9%	$0.0015	3.6s	26%
70	Hermes 3 70B	60.7%	$0.0013	29.2s	31%
71	Claude Opus 4	90.4%	$0.116	15.8s	77%
72	Cohere Command R+ (Aug. 2024)	60.3%	$0.014	10.0s	24%
73	Qwen 3.5 397B A17B	95.5%	$0.026	2.9m	76%
74	Mistral NeMO	44.0%	$0.0007	13.3s	21%
75	Llama 3.1 8B	34.1%	$0.0002	16.1s	13%
76	WizardLM 2 8x22b	32.2%	$0.0036	15.7s	6%
77	Gemma 3 4B	20.8%	$0.0002	12.5s	14%
78	Rocinante 12B	21.8%	$0.0009	6.0s	3%
79	GPT-4.1 Nano	20.4%	$0.0004	3.9s	0%
80	Arcee AI: Trinity Large (Preview)	60.2%	$0.0000	3.1m	29%
80.75%

Individual Scenarios

matrix

▼

Large codex (40 entries), long passage (1,019 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
o4 Mini	100	100	100	100	100	100	100	100	100	97	99.7%
o4 Mini High	100	100	100	100	100	100	100	100	100	97	99.7%
GPT-5	100	100	100	100	100	100	100	100	97	97	99.4%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	97	97	99.3%
Z.AI GLM 5	100	100	100	100	100	100	100	100	97	96	99.3%
Claude Opus 4.5	100	100	100	100	100	97	97	97	97	97	98.4%
GPT-5.2	100	100	100	100	100	97	97	97	97	94	98.1%
GPT-5 Mini	100	100	100	100	100	100	97	97	94	94	98.1%
GPT-5.1	100	100	100	100	100	100	100	96	94	90	98.0%
Grok 4	100	100	100	100	97	97	97	97	94	94	97.6%
Z.AI GLM 4.7	100	100	100	100	100	97	97	96	93	93	97.5%
Qwen 3.5 397B A17B	100	100	100	100	97	97	97	94	94	94	97.2%
Gemini 2.5 Pro	100	100	100	97	97	97	97	97	94	91	96.9%
Grok 4 Fast	100	100	100	100	97	97	97	91	91	88	96.0%
Grok 4.1 Fast	100	100	100	100	100	97	94	91	88	88	95.8%
Z.AI GLM 4.6	100	97	97	97	96	96	94	94	93	91	95.5%
Claude Sonnet 4.6	100	97	96	96	93	93	93	93	90	90	94.2%
Claude Sonnet 4	97	96	96	96	93	93	93	93	90	89	93.7%
ByteDance Seed 1.6	100	97	94	94	94	94	93	91	88	85	92.9%
Gemini 3 Pro (Preview)	100	97	96	93	93	93	93	90	87	86	92.7%
Stealth: Aurora Alpha	100	96	96	96	91	91	91	90	88	88	92.6%
Claude 3.5 Sonnet	100	100	100	93	92	92	92	92	85	75	92.0%
MoonshotAI: Kimi K2.5	100	97	97	96	93	91	90	88	87	78	91.7%
Minimax M2.5	100	100	100	96	95	94	88	85	83	75	91.7%
Claude Opus 4.6	94	94	94	94	91	91	90	90	90	90	91.7%
Gemini 3 Flash (Preview)	94	93	93	93	93	93	90	90	90	87	91.6%
Mistral Large	100	100	100	88	88	88	88	86	85	84	90.7%
GPT-5 Nano	100	100	100	95	91	89	86	84	81	81	90.7%
Mistral Large 3	100	95	93	92	92	90	89	89	85	82	90.6%
Z.AI GLM 4.7 Flash	100	95	94	93	93	92	88	86	86	79	90.5%
GPT-4.1	96	95	95	92	92	91	90	90	86	75	90.1%
Mistral Large 2	100	95	95	92	91	87	87	86	84	78	89.5%
Gemini 2.5 Flash	96	93	93	93	90	88	87	86	85	79	89.0%
Claude Sonnet 4.5	94	93	91	90	90	88	86	85	85	84	88.5%
DeepSeek V3 (2024-12-26)	100	100	100	95	95	91	89	81	71	56	88.0%
Claude Opus 4	97	93	91	88	88	86	85	83	81	81	87.3%
ByteDance Seed 1.6 Flash	100	92	92	89	88	88	83	82	79	75	86.7%
DeepSeek-V2 Chat	100	95	95	94	86	85	79	79	77	76	86.6%
Claude 3.7 Sonnet	94	91	89	88	88	82	82	79	68	68	83.0%
Mistral Medium 3.1	89	87	83	82	82	80	80	77	76	73	80.9%
DeepSeek V3.2	100	85	83	81	81	79	78	72	69	68	79.7%
Claude Haiku 4.5	86	86	86	79	79	78	78	76	74	74	79.6%
GPT-4o, May 13th (temp=1)	100	100	90	82	79	79	78	71	62	50	78.9%
Z.AI GLM 4.5	100	90	81	80	76	75	75	74	70	63	78.5%
GPT-4.1 Mini	100	95	94	79	77	76	73	69	62	58	78.2%
Qwen 3.5 Plus (2026-02-15)	97	94	93	91	88	88	85	82	47	0	76.3%
Qwen 2.5 72B	100	90	86	75	75	75	73	71	67	50	76.2%
DeepSeek V3.1	100	88	86	85	84	83	82	79	71	0	75.8%
DeepSeek V3 (2025-03-24)	96	95	86	85	82	81	80	75	68	0	74.8%
Writer: Palmyra X5	87	85	84	80	76	75	72	71	68	50	74.8%
GPT-4o, Aug. 6th (temp=1)	91	83	81	80	80	75	73	71	58	53	74.6%
Claude 3.5 Haiku	100	100	83	80	75	75	71	67	67	0	71.8%
Mistral Small Creative	76	71	69	67	65	65	63	61	59	53	64.9%
GPT-4o, Aug. 6th (temp=0)	92	84	79	79	79	79	75	73	2	2	64.3%
Hermes 3 405B	100	83	80	63	60	57	57	45	44	40	63.0%
Arcee AI: Trinity Mini	100	100	100	60	50	50	50	50	30	28	61.8%
Gemma 3 27B	78	77	75	57	54	53	50	50	45	33	57.2%
Hermes 3 70B	80	80	75	67	57	50	50	44	14	0	51.8%
Ministral 3 8B	71	58	57	55	52	50	48	47	42	35	51.5%
Ministral 8B	63	57	56	50	50	50	50	47	46	45	51.5%
Mistral Small 3.2 24B	68	67	64	57	55	52	46	44	39	21	51.2%
Llama 3.1 Nemotron 70B	100	67	63	54	54	50	43	33	33	0	49.6%
GPT-4o, May 13th (temp=0)	100	92	92	88	70	16	14	0	0	0	47.2%
Claude 3 Haiku	75	63	50	43	40	38	33	33	33	33	44.1%
Ministral 3 14B	57	57	57	57	53	50	44	38	26	0	43.9%
Gemma 3 12B	75	67	60	45	40	33	33	29	29	25	43.5%
GPT-4o Mini (temp=1)	67	60	56	50	38	33	33	30	20	0	38.6%
Cohere Command R+ (Aug. 2024)	56	54	50	45	43	42	27	25	0	0	34.1%
WizardLM 2 8x22b	56	41	39	36	35	30	29	25	21	17	32.9%
Ministral 3B	36	33	33	30	30	30	28	27	21	18	28.8%
Gemini 2.5 Flash Lite	56	52	51	47	38	27	0	0	0	0	27.1%
Llama 3.1 70B	54	50	50	40	38	29	2	0	0	0	26.1%
Ministral 3 3B	37	35	31	26	24	22	22	20	13	13	24.3%
Llama 3.1 8B	67	50	25	20	14	8	0	0	0	0	18.4%
Mistral NeMO	55	25	24	17	14	11	9	9	5	5	17.4%
Gemma 3 4B	35	29	20	20	13	13	11	0	0	0	14.1%
GPT-4.1 Nano	100	21	12	4	0	0	0	0	0	0	13.7%
GPT-4o Mini (temp=0)	50	33	22	20	6	0	0	0	0	0	13.2%
Rocinante 12B	33	33	25	0	0	0	0	0	0	0	9.2%
Arcee AI: Trinity Large (Preview)	17	13	12	3	1	1	1	1	0	0	4.9%

▼

Large codex (40 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	95	99.5%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	95	99.5%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	92	99.2%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	95	94	98.9%
Grok 4.1 Fast	100	100	100	100	100	100	100	95	95	94	98.4%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	95	95	95	95	98.0%
GPT-5	100	100	100	100	100	100	95	94	94	90	97.3%
Claude Opus 4.6	100	100	100	100	95	95	95	95	95	95	97.0%
o4 Mini High	100	100	100	100	94	94	94	94	94	94	96.6%
MoonshotAI: Kimi K2.5	100	100	100	100	95	95	95	95	95	90	96.5%
Z.AI GLM 5	100	100	100	100	100	100	95	94	90	85	96.4%
Claude Sonnet 4	100	100	100	100	95	95	95	95	94	89	96.3%
GPT-4.1	100	100	100	100	95	95	94	93	93	89	95.9%
Minimax M2.5	100	100	100	100	94	94	94	93	92	88	95.5%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	93	92	88	80	95.3%
o4 Mini	100	100	100	94	94	94	94	94	89	88	94.9%
Z.AI GLM 4.7	100	100	100	95	95	95	94	90	89	88	94.6%
GPT-5.1	100	100	95	95	95	94	94	94	90	89	94.6%
Stealth: Aurora Alpha	100	100	100	94	94	94	94	94	94	82	94.5%
GPT-5 Mini	100	100	94	94	94	94	90	90	89	89	93.6%
Gemini 3 Pro (Preview)	100	100	100	100	95	95	89	88	85	84	93.6%
Qwen 3.5 397B A17B	95	95	95	95	95	95	95	89	89	89	93.2%
ByteDance Seed 1.6 Flash	100	100	100	93	92	92	92	91	86	81	92.7%
Z.AI GLM 4.6	100	95	95	95	94	90	90	89	88	83	92.0%
Gemini 3 Flash (Preview)	95	95	95	95	95	89	89	89	89	84	91.6%
Z.AI GLM 4.5	94	94	94	94	94	94	89	88	87	87	91.5%
GPT-5.2	95	95	95	90	90	90	90	90	90	86	91.2%
GPT-4o, Aug. 6th (temp=0)	100	92	92	92	92	92	86	86	86	85	90.1%
Claude Opus 4	100	95	94	89	89	89	89	89	83	79	89.6%
DeepSeek-V2 Chat	100	100	94	94	88	88	87	87	81	76	89.4%
Claude 3.5 Sonnet	94	94	94	94	88	88	88	82	82	81	88.7%
Qwen 3.5 Plus (2026-02-15)	95	95	89	89	89	89	89	89	79	78	88.2%
DeepSeek V3 (2024-12-26)	100	100	94	88	88	87	83	82	81	78	88.0%
Gemini 2.5 Flash	95	95	94	90	90	85	85	83	83	79	88.0%
Writer: Palmyra X5	100	93	90	89	89	86	85	82	80	79	87.4%
Claude 3.5 Haiku	100	89	88	88	88	88	88	88	78	78	86.9%
Claude 3.7 Sonnet	94	88	88	88	88	88	83	83	83	80	86.5%
Arcee AI: Trinity Mini	100	100	100	100	100	88	83	67	67	60	86.4%
GPT-4o, May 13th (temp=1)	100	100	93	90	85	85	85	78	77	67	85.9%
Mistral Large	94	89	89	88	88	83	82	82	81	81	85.8%
Mistral Medium 3.1	94	88	88	88	88	88	84	82	79	78	85.8%
Mistral Large 3	88	88	88	88	87	87	83	83	82	81	85.4%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	93	92	92	90	86	0	85.3%
Mistral Large 2	94	89	89	87	83	82	82	82	82	78	84.9%
DeepSeek V3.2	100	94	89	88	84	83	83	81	78	50	83.2%
Mistral Small Creative	100	91	85	85	83	83	77	75	75	71	82.5%
Gemma 3 27B	90	89	83	82	80	80	80	80	80	75	81.9%
Hermes 3 405B	100	91	90	88	82	79	70	70	69	60	79.8%
DeepSeek V3 (2025-03-24)	100	100	88	88	88	87	86	82	76	0	79.5%
GPT-4o Mini (temp=0)	78	78	78	78	78	78	78	78	78	60	76.0%
Mistral Small 3.2 24B	88	81	80	78	78	76	72	68	67	67	75.5%
GPT-5 Nano	100	94	94	94	93	93	92	88	0	0	74.9%
GPT-4.1 Mini	80	80	78	77	76	75	75	74	65	64	74.4%
Ministral 3B	91	90	89	85	83	75	75	63	42	42	73.4%
Arcee AI: Trinity Large (Preview)	94	89	82	80	78	76	71	71	68	12	72.0%
Qwen 2.5 72B	100	89	88	80	78	75	71	50	38	33	70.1%
Claude Haiku 4.5	94	89	89	89	87	83	83	81	0	0	69.6%
Gemini 2.5 Flash Lite	84	79	74	71	70	68	67	62	59	57	69.0%
Gemma 3 12B	80	75	73	73	71	70	70	67	64	45	68.8%
Ministral 3 8B	85	77	77	75	75	73	73	71	71	0	67.7%
Ministral 3 3B	100	91	89	89	86	75	56	47	38	6	67.6%
Ministral 8B	79	77	71	67	65	64	63	63	63	59	66.9%
DeepSeek V3.1	94	94	89	85	83	83	72	65	0	0	66.6%
Ministral 3 14B	71	71	69	67	64	63	62	60	53	50	62.9%
Llama 3.1 Nemotron 70B	89	86	78	75	58	57	56	54	50	0	60.2%
Cohere Command R+ (Aug. 2024)	100	88	83	78	64	57	50	40	33	0	59.3%
Hermes 3 70B	100	75	73	73	67	56	55	50	36	0	58.4%
Llama 3.1 70B	88	83	78	75	56	54	50	47	38	0	56.7%
GPT-4o Mini (temp=1)	83	71	63	60	57	56	50	43	29	0	51.1%
Claude 3 Haiku	75	63	50	44	43	43	43	38	17	0	41.5%
Mistral NeMO	55	50	31	30	22	18	17	15	13	7	25.7%
Llama 3.1 8B	40	33	14	11	9	8	0	0	0	0	11.6%
GPT-4.1 Nano	50	20	15	11	8	0	0	0	0	0	10.5%
Rocinante 12B	50	33	11	0	0	0	0	0	0	0	9.4%
Gemma 3 4B	20	13	10	10	8	7	0	0	0	0	6.8%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Small codex (7 entries), long passage (734 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	91	99.1%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	90	99.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	90	99.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	89	98.9%
GPT-5.1	100	100	100	100	100	100	100	100	90	80	97.0%
o4 Mini	100	100	100	100	100	100	100	90	90	90	97.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	89	80	96.9%
ByteDance Seed 1.6	100	100	100	100	100	100	100	89	89	89	96.7%
Minimax M2.5	100	100	100	100	100	100	89	89	88	88	95.3%
Claude 3.5 Haiku	100	100	100	100	100	100	100	86	83	83	95.2%
Claude 3.5 Sonnet	100	100	100	100	100	100	89	88	88	88	95.1%
o4 Mini High	100	100	100	100	100	100	91	90	90	80	95.1%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	90	90	90	90	90	90	94.0%
GPT-4.1 Mini	100	100	100	100	100	100	88	88	88	75	93.8%
Qwen 2.5 72B	100	100	100	100	100	100	86	86	83	80	93.5%
Z.AI GLM 4.7 Flash	100	100	100	100	100	88	88	86	86	83	93.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	86	83	83	75	92.7%
GPT-5 Mini	100	100	91	91	91	91	91	91	91	90	92.6%
Writer: Palmyra X5	100	100	100	100	100	86	86	86	83	83	92.4%
Claude Sonnet 4.6	100	100	90	90	90	90	90	90	90	90	92.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	88	86	75	71	92.0%
Claude 3.7 Sonnet	100	100	100	89	89	89	89	89	88	88	91.9%
GPT-5.2	100	91	91	91	91	91	91	91	91	91	91.8%
GPT-5	100	100	100	100	91	91	90	90	83	73	91.8%
Z.AI GLM 4.7	100	100	100	100	90	89	89	89	80	80	91.7%
Claude 3 Haiku	100	100	100	100	100	100	100	100	67	50	91.7%
MoonshotAI: Kimi K2.5	100	100	100	100	100	90	88	82	78	78	91.5%
Grok 4 Fast	100	100	100	91	91	90	90	90	82	75	90.9%
Grok 4.1 Fast	100	100	100	91	91	90	90	89	82	73	90.5%
Stealth: Aurora Alpha	100	100	90	89	89	89	89	89	89	82	90.5%
GPT-4o, May 13th (temp=0)	100	100	100	88	88	88	88	88	88	75	90.0%
GPT-4o, Aug. 6th (temp=0)	100	100	88	88	88	88	88	88	88	86	89.8%
ByteDance Seed 1.6 Flash	100	100	100	89	88	88	86	86	80	78	89.3%
GPT-4o, Aug. 6th (temp=1)	100	100	100	88	88	86	86	86	86	75	89.3%
Grok 4	100	100	100	100	91	90	90	78	73	70	89.1%
Gemini 3 Flash (Preview)	100	89	89	89	89	89	89	89	89	80	89.1%
Claude Sonnet 4.5	100	100	90	90	90	89	89	80	80	80	88.8%
Z.AI GLM 4.6	100	100	90	90	90	89	89	89	82	67	88.5%
DeepSeek V3 (2025-03-24)	100	100	100	86	86	86	83	83	83	75	88.2%
DeepSeek V3 (2024-12-26)	100	100	88	86	86	86	86	83	83	83	88.0%
Mistral Large	100	100	89	89	89	88	88	78	78	78	87.5%
Gemini 2.5 Flash	100	100	100	89	89	82	80	80	80	70	87.0%
Claude Opus 4	100	89	89	89	89	88	88	78	78	78	86.4%
Z.AI GLM 4.5	100	89	89	89	89	89	88	80	78	70	86.0%
Mistral Large 2	100	89	88	88	88	88	88	78	75	75	85.4%
GPT-4.1	89	88	88	88	88	88	88	86	78	75	85.2%
Claude Sonnet 4	100	90	90	90	80	80	80	80	80	80	85.0%
Llama 3.1 70B	88	86	86	86	86	86	86	80	78	75	83.5%
Mistral Large 3	90	89	88	88	88	78	78	78	78	78	83.0%
GPT-5 Nano	100	89	88	88	86	86	83	78	67	67	83.0%
Mistral Small 3.2 24B	89	89	89	88	86	86	80	75	75	73	82.8%
Llama 3.1 Nemotron 70B	89	88	88	88	88	78	78	78	78	73	82.3%
Gemini 2.5 Flash Lite	100	89	86	86	86	86	78	75	71	63	81.8%
Ministral 3 14B	100	100	89	89	88	88	88	78	50	50	81.8%
DeepSeek V3.2	88	88	88	86	86	86	83	71	67	63	80.4%
Gemma 3 27B	100	88	86	86	80	75	70	67	67	67	78.4%
Mistral Medium 3.1	89	89	89	89	89	89	89	80	80	0	78.2%
GPT-4o Mini (temp=0)	80	75	75	75	75	75	75	75	75	75	75.5%
Hermes 3 405B	88	86	83	75	71	71	71	67	67	67	74.6%
DeepSeek V3.1	100	86	86	86	83	75	71	71	71	0	73.0%
Arcee AI: Trinity Mini	100	100	83	80	67	60	60	60	57	50	71.7%
GPT-4o Mini (temp=1)	80	80	80	80	80	75	75	60	50	50	71.0%
Mistral Small Creative	89	80	80	70	70	67	64	62	58	54	69.3%
Arcee AI: Trinity Large (Preview)	80	75	70	70	70	67	64	62	55	41	65.3%
Hermes 3 70B	100	83	80	75	71	67	63	56	36	0	63.1%
Ministral 3 3B	75	67	67	63	60	60	60	56	56	50	61.2%
Gemma 3 12B	75	63	63	63	57	57	57	50	50	44	57.8%
Ministral 3B	86	56	56	56	56	54	50	44	42	38	53.5%
Ministral 3 8B	64	62	58	58	54	50	47	46	40	35	51.4%
Cohere Command R+ (Aug. 2024)	100	83	80	75	71	67	0	0	0	0	47.6%
Ministral 8B	73	58	53	53	50	50	47	46	19	0	45.0%
Llama 3.1 8B	67	63	56	56	50	50	50	44	0	0	43.5%
Mistral NeMO	75	55	54	54	50	50	33	33	0	0	40.4%
Gemma 3 4B	57	47	43	38	37	35	33	29	27	26	37.2%
Rocinante 12B	67	64	33	14	0	0	0	0	0	0	17.8%
GPT-4.1 Nano	25	23	18	12	0	0	0	0	0	0	7.7%
WizardLM 2 8x22b	20	0	0	0	0	0	0	0	0	0	2.0%

▼

Small codex (7 entries), short passage (165 words)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	91	99.1%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	91	99.1%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	91	99.1%
GPT-4.1	100	100	100	100	100	100	100	100	100	91	99.1%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	90	99.0%
DeepSeek V3.2	100	100	100	100	100	100	100	100	100	90	99.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	90	99.0%
Gemma 3 12B	100	100	100	100	100	100	100	100	100	89	98.9%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	90	90	98.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	90	90	98.0%
Mistral Large 2	100	100	100	100	100	100	100	100	90	90	98.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	80	98.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	89	88	97.6%
Mistral Large	100	100	100	100	100	100	100	90	90	90	97.0%
Qwen 2.5 72B	100	100	100	100	100	100	100	100	100	63	96.3%
Mistral Medium 3.1	100	100	100	100	100	100	90	90	90	90	96.0%
DeepSeek V3.1	100	100	100	100	100	100	90	90	90	90	96.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	90	90	90	90	96.0%
Minimax M2.5	100	100	100	100	100	100	90	89	89	89	95.7%
GPT-4.1 Mini	100	100	100	100	100	100	90	89	89	89	95.7%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	88	86	78	95.1%
Mistral Large 3	100	100	100	100	100	90	90	90	90	90	95.0%
Claude Opus 4	100	100	100	100	90	90	90	90	90	90	94.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	89	89	89	88	86	94.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	91	91	91	91	91	91	91	93.6%
GPT-4o, May 13th (temp=1)	100	100	100	100	90	90	90	90	88	88	93.5%
Mistral Small 3.2 24B	100	100	100	100	89	89	89	89	89	89	93.3%
Claude Sonnet 4.6	90	90	90	90	90	90	90	90	90	90	90.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	0	90.0%
Arcee AI: Trinity Mini	100	100	100	100	86	83	83	80	80	80	89.2%
Gemma 3 27B	100	89	89	88	88	88	88	86	86	86	88.5%
Writer: Palmyra X5	90	90	90	90	89	89	88	88	88	80	88.0%
Gemini 2.5 Flash Lite	89	89	89	89	89	89	89	89	89	80	88.0%
Claude Haiku 4.5	89	89	89	89	89	89	89	78	78	78	85.6%
Mistral NeMO	88	86	86	83	83	83	83	83	83	83	84.2%
Hermes 3 70B	100	100	100	100	100	89	88	83	71	0	83.1%
Claude 3.5 Haiku	88	88	88	86	86	86	86	86	71	57	82.0%
Arcee AI: Trinity Large (Preview)	90	90	83	82	80	80	80	75	73	69	80.2%
Mistral Small Creative	82	82	80	80	80	80	80	80	80	73	79.6%
Ministral 3 3B	88	88	88	88	86	86	78	71	67	56	79.3%
Llama 3.1 8B	100	100	89	88	80	78	71	63	56	50	77.4%
Ministral 3 14B	90	89	88	78	78	78	78	70	67	50	76.4%
Ministral 3B	88	86	83	78	75	75	75	71	64	63	75.7%
Claude 3 Haiku	83	83	83	80	80	80	80	71	60	50	75.1%
GPT-4.1 Nano	100	100	100	100	100	100	67	50	33	0	75.0%
Ministral 8B	89	89	88	78	73	70	70	67	60	55	73.7%
Ministral 3 8B	88	80	80	75	75	73	67	67	60	55	71.8%
Rocinante 12B	100	86	80	67	67	50	50	50	50	25	62.4%
Gemma 3 4B	40	33	20	20	20	20	17	17	17	14	21.8%
WizardLM 2 8x22b	100	100	17	0	0	0	0	0	0	0	21.7%

tiers

▼

5 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	83	98.3%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	83	98.3%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	83	98.3%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	80	98.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	80	98.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	67	96.7%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	67	96.7%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	67	96.7%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	83	80	96.3%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	83	80	96.3%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	80	80	96.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	80	80	96.0%
Grok 4	100	100	100	100	100	100	100	83	83	83	95.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	83	67	95.0%
Llama 3.1 70B	100	100	100	100	100	100	100	83	80	80	94.3%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	80	80	75	93.5%
GPT-5	100	100	100	100	100	100	83	83	83	83	93.3%
DeepSeek V3.2	100	100	100	100	100	83	83	83	83	83	91.7%
Claude Opus 4	100	100	100	100	100	80	80	80	80	80	90.0%
Qwen 2.5 72B	100	100	100	100	100	100	75	75	75	75	90.0%
Mistral Large	100	100	100	83	83	83	83	83	83	83	88.3%
Arcee AI: Trinity Mini	100	100	100	100	100	100	75	75	67	67	88.3%
GPT-5 Nano	100	100	100	100	100	100	100	75	75	0	85.0%
Minimax M2.5	100	100	100	83	80	80	80	75	75	75	84.8%
GPT-4o, May 13th (temp=0)	100	100	100	100	83	83	80	67	67	67	84.7%
DeepSeek V3.1	100	100	100	100	100	100	83	80	71	0	83.5%
Mistral Large 3	83	83	83	83	83	83	83	83	83	83	83.3%
Mistral Large 2	83	83	83	83	83	83	83	83	83	83	83.3%
GPT-4o Mini (temp=1)	100	100	100	100	75	75	75	75	60	60	82.0%
GPT-4.1 Mini	100	100	80	80	80	80	80	80	80	50	81.0%
GPT-4o Mini (temp=0)	100	100	75	75	75	75	75	75	75	75	80.0%
Claude Sonnet 4.6	80	80	80	80	80	80	80	80	80	80	80.0%
Gemma 3 27B	80	80	80	80	80	80	80	80	80	80	80.0%
Mistral Medium 3.1	100	80	80	80	80	80	80	80	67	67	79.3%
Gemma 3 12B	100	75	75	75	75	75	75	75	75	75	77.5%
GPT-4o, May 13th (temp=1)	83	83	80	80	67	67	67	67	67	67	72.7%
GPT-4o, Aug. 6th (temp=0)	83	83	83	67	67	67	67	67	67	67	71.7%
GPT-4o, Aug. 6th (temp=1)	83	83	80	67	67	67	67	67	67	67	71.3%
Writer: Palmyra X5	80	80	80	80	80	80	67	67	67	33	71.3%
Claude Haiku 4.5	80	80	80	80	75	67	67	60	60	60	70.8%
Cohere Command R+ (Aug. 2024)	100	100	100	75	67	60	50	50	50	33	68.5%
Arcee AI: Trinity Large (Preview)	80	80	80	80	67	67	60	57	57	50	67.8%
Gemini 2.5 Flash Lite	75	75	60	60	60	60	60	60	60	60	63.0%
Mistral NeMO	80	75	75	75	67	60	60	50	40	33	61.5%
Mistral Small Creative	67	67	67	67	67	67	57	57	50	50	61.4%
Ministral 3 3B	80	80	67	60	60	57	50	50	31	25	56.0%
Mistral Small 3.2 24B	67	60	60	60	57	57	50	50	50	40	55.1%
Llama 3.1 8B	80	80	63	63	50	50	50	50	43	17	54.5%
Claude 3 Haiku	67	67	60	60	60	50	50	50	40	40	54.3%
Hermes 3 70B	75	75	67	67	50	50	50	40	33	0	50.7%
Ministral 3B	80	67	63	60	50	43	40	40	33	25	50.0%
Ministral 3 14B	57	57	50	50	50	50	43	43	43	43	48.6%
GPT-4.1 Nano	100	100	100	100	33	0	0	0	0	0	43.3%
Ministral 8B	57	38	38	33	33	30	30	30	30	27	34.6%
Gemma 3 4B	50	50	50	33	25	25	25	25	25	25	33.3%
Ministral 3 8B	38	33	33	33	30	27	20	20	20	18	27.3%
Rocinante 12B	60	50	50	20	14	11	0	0	0	0	20.5%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%

▼

10 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	86	98.6%
GPT-5.2	100	100	100	100	100	100	100	100	100	86	98.6%
Claude Opus 4	100	100	100	100	100	100	100	100	100	86	98.6%
Grok 4	100	100	100	100	100	100	100	100	100	86	98.6%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	86	98.6%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	86	98.6%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	100	86	98.6%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	83	98.3%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	80	98.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	75	97.5%
Minimax M2.5	100	100	100	100	100	100	100	100	83	83	96.7%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	67	96.7%
o4 Mini	100	100	100	100	100	100	86	86	83	83	93.8%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	86	83	83	83	93.6%
Mistral Small Creative	100	100	100	100	100	100	86	86	83	80	93.5%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	83	83	83	83	93.3%
Writer: Palmyra X5	100	100	100	100	100	100	83	83	83	83	93.3%
o4 Mini High	100	100	100	100	100	86	86	86	83	83	92.4%
Z.AI GLM 4.5	100	100	100	100	100	86	83	83	83	83	91.9%
GPT-5.1	100	100	100	100	86	86	86	86	86	86	91.4%
GPT-5	100	100	100	100	86	86	86	86	86	86	91.4%
Z.AI GLM 4.6	100	100	100	100	100	100	86	86	83	50	90.5%
Qwen 2.5 72B	100	100	100	100	100	100	100	67	67	67	90.0%
DeepSeek-V2 Chat	100	100	100	100	83	83	83	80	80	80	89.0%
GPT-5 Nano	100	100	100	83	83	83	83	83	83	83	88.3%
Llama 3.1 70B	100	100	100	100	100	100	100	100	83	0	88.3%
GPT-5 Mini	100	86	86	86	86	86	86	86	86	86	87.1%
GPT-4.1	100	100	83	83	83	83	83	83	83	83	86.7%
Mistral Large	100	100	100	86	83	83	83	83	71	71	86.2%
Gemma 3 27B	100	100	86	83	83	83	83	83	83	67	85.2%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 2.5 Flash	100	86	86	86	86	83	83	83	83	71	84.8%
DeepSeek V3 (2024-12-26)	100	100	86	83	83	83	80	80	75	75	84.6%
DeepSeek V3.2	100	100	100	86	86	86	86	80	67	50	84.0%
Claude Sonnet 4.6	83	83	83	83	83	83	83	83	83	83	83.3%
Claude Haiku 4.5	83	83	83	83	83	83	83	83	83	83	83.3%
GPT-4.1 Mini	86	86	86	86	86	83	83	83	83	71	83.3%
Mistral Small 3.2 24B	100	100	100	86	83	83	83	67	63	57	82.2%
Gemini 2.5 Flash Lite	100	100	100	83	80	80	80	71	67	57	81.9%
Hermes 3 405B	100	100	100	83	83	83	83	83	80	20	81.7%
DeepSeek V3.1	100	86	86	83	83	83	80	71	60	57	79.0%
Mistral Large 3	86	83	83	83	83	83	71	71	71	71	78.8%
Mistral Large 2	100	83	83	83	83	71	71	71	67	67	78.1%
Gemma 3 12B	86	83	83	83	83	83	71	71	67	60	77.2%
Arcee AI: Trinity Large (Preview)	100	100	83	83	80	67	67	67	67	57	77.0%
WizardLM 2 8x22b	100	100	86	83	83	80	71	67	67	0	73.7%
Claude 3.7 Sonnet	100	100	100	100	100	100	86	0	0	0	68.6%
Ministral 3 8B	75	75	75	67	67	67	67	67	60	60	67.8%
Ministral 8B	100	75	67	67	67	60	56	56	55	50	65.1%
Cohere Command R+ (Aug. 2024)	100	100	83	80	80	60	57	50	40	0	65.0%
Claude 3 Haiku	80	80	80	80	60	57	50	50	50	50	63.7%
Hermes 3 70B	83	75	71	71	71	67	67	67	50	0	62.3%
Mistral NeMO	75	75	71	71	60	60	60	50	50	43	61.6%
Ministral 3 14B	67	67	67	67	60	57	57	50	50	43	58.4%
Ministral 3 3B	83	44	43	40	40	38	22	22	18	15	36.6%
Ministral 3B	75	57	50	45	40	38	19	13	0	0	33.6%
Rocinante 12B	75	75	67	67	50	0	0	0	0	0	33.3%
Gemma 3 4B	67	50	50	40	40	33	25	17	0	0	32.2%
Llama 3.1 8B	33	33	30	25	17	13	0	0	0	0	15.1%
GPT-4.1 Nano	100	0	0	0	0	0	0	0	0	0	10.0%

▼

20 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	91	99.1%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	91	99.1%
Grok 4	100	100	100	100	100	100	100	100	100	90	99.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	92	92	98.3%
GPT-5.1	100	100	100	100	100	100	100	100	91	91	98.2%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	83	83	96.7%
Claude Sonnet 4.5	100	100	100	100	100	100	91	91	91	91	96.4%
Claude Sonnet 4.6	100	100	100	100	100	100	91	90	89	89	95.9%
o4 Mini	100	100	100	100	100	100	90	90	90	82	95.2%
o4 Mini High	100	100	100	100	100	100	90	90	90	82	95.2%
GPT-5 Nano	100	100	100	100	100	100	90	90	89	82	95.1%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	88	88	75	95.0%
Gemini 3 Pro (Preview)	100	100	100	100	92	92	91	91	91	91	94.7%
Z.AI GLM 5	100	100	100	100	100	100	91	85	85	82	94.2%
Gemini 2.5 Pro	100	100	100	92	92	92	92	92	92	92	94.2%
Minimax M2.5	100	100	100	100	100	90	89	89	88	80	93.5%
GPT-5.2	100	100	92	92	92	92	91	91	91	91	93.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	91	90	89	88	88	82	92.7%
Z.AI GLM 4.7 Flash	100	100	100	100	90	89	89	89	88	82	92.6%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	83	71	67	92.1%
Gemini 2.5 Flash	100	91	91	91	91	91	91	91	91	91	91.8%
Gemini 3 Flash (Preview)	91	91	91	91	91	91	91	91	91	91	90.9%
MoonshotAI: Kimi K2.5	100	92	92	92	91	91	91	91	85	83	90.7%
GPT-4o, Aug. 6th (temp=0)	91	91	91	91	91	91	90	90	90	90	90.5%
Claude 3.5 Sonnet	100	92	92	92	92	92	91	90	83	82	90.4%
Mistral Medium 3.1	92	92	91	91	91	91	91	91	90	83	90.2%
ByteDance Seed 1.6	91	91	90	90	90	90	90	90	90	90	90.2%
GPT-4o, May 13th (temp=0)	100	90	90	90	89	89	88	88	88	86	89.6%
Claude Sonnet 4	91	91	91	91	91	91	91	91	83	83	89.4%
Grok 4 Fast	100	100	100	100	100	100	100	100	92	0	89.2%
Writer: Palmyra X5	100	91	90	89	89	89	89	88	83	78	88.5%
GPT-5 Mini	91	90	90	90	90	90	90	89	83	82	88.5%
Z.AI GLM 4.7	92	92	91	91	91	91	83	83	83	83	88.0%
Mistral Large 3	100	100	90	90	90	89	80	80	80	80	87.9%
DeepSeek V3.2	100	100	100	100	82	82	80	78	77	75	87.3%
Mistral Large	92	92	92	91	90	89	83	83	80	80	87.1%
GPT-4o, May 13th (temp=1)	100	100	89	88	88	88	86	83	75	75	87.0%
DeepSeek V3 (2024-12-26)	100	100	100	89	89	89	88	75	75	60	86.4%
Z.AI GLM 4.5	100	89	89	89	88	88	82	80	80	75	85.8%
Claude Opus 4	92	92	92	83	83	83	83	82	82	82	85.4%
Claude Haiku 4.5	100	100	100	100	100	90	89	89	82	0	85.0%
Stealth: Aurora Alpha	100	89	89	82	82	82	82	82	82	80	84.9%
Mistral Large 2	91	91	91	90	83	83	83	82	80	73	84.7%
Z.AI GLM 4.6	100	92	91	91	83	83	82	75	73	73	84.2%
GPT-4.1 Mini	100	90	89	89	89	88	88	71	67	67	83.6%
Qwen 3.5 Plus (2026-02-15)	91	90	86	85	85	83	83	77	77	77	83.3%
DeepSeek V3.1	92	92	91	90	83	83	80	75	67	67	81.9%
Ministral 3 14B	100	100	90	89	75	75	75	75	67	67	81.2%
Claude 3.7 Sonnet	100	100	91	91	90	73	70	69	64	55	80.2%
Qwen 3.5 397B A17B	92	91	91	91	91	91	83	83	82	0	79.5%
GPT-4.1	100	100	100	100	100	100	92	90	0	0	78.2%
Llama 3.1 Nemotron 70B	100	88	88	88	75	71	71	67	67	67	78.0%
DeepSeek V3 (2025-03-24)	89	89	88	88	78	75	75	75	73	43	77.1%
Llama 3.1 70B	100	88	88	78	78	75	75	75	56	43	75.4%
Mistral Small 3.2 24B	88	86	80	75	75	73	73	70	67	67	75.2%
Hermes 3 405B	100	100	89	80	73	71	70	57	56	50	74.6%
Mistral Small Creative	80	80	80	78	78	73	70	70	70	67	74.5%
DeepSeek-V2 Chat	100	89	89	89	88	78	75	70	67	0	74.4%
Arcee AI: Trinity Large (Preview)	90	82	82	80	78	70	70	67	64	56	73.7%
Gemma 3 27B	88	78	75	75	73	67	67	67	67	67	72.1%
Cohere Command R+ (Aug. 2024)	100	88	78	78	75	67	64	58	57	50	71.4%
Gemini 2.5 Flash Lite	100	100	92	90	82	82	80	78	0	0	70.3%
WizardLM 2 8x22b	88	75	67	67	67	67	60	58	57	43	64.8%
GPT-4o Mini (temp=0)	80	75	60	60	60	60	60	60	60	57	63.2%
GPT-4o Mini (temp=1)	86	80	80	75	75	60	50	40	33	33	61.2%
Ministral 3 8B	80	71	69	64	58	58	55	46	45	42	58.9%
Hermes 3 70B	86	80	71	71	67	60	57	33	0	0	52.6%
Claude 3 Haiku	83	67	60	57	50	50	43	43	40	20	51.3%
Ministral 8B	69	67	67	62	58	58	55	46	0	0	48.1%
Gemma 3 12B	63	57	56	56	56	44	43	42	38	0	45.3%
Ministral 3B	63	55	50	50	50	44	44	40	36	18	45.0%
Ministral 3 3B	60	60	57	55	50	50	36	29	25	22	44.4%
Mistral NeMO	67	56	56	44	40	40	38	33	33	27	43.3%
Llama 3.1 8B	80	67	56	50	29	27	17	9	0	0	33.4%
Qwen 2.5 72B	100	38	38	33	29	24	19	16	14	9	31.8%
Rocinante 12B	50	25	20	20	14	13	0	0	0	0	14.2%
Gemma 3 4B	40	20	14	13	12	9	9	7	5	0	12.8%
GPT-4.1 Nano	22	0	0	0	0	0	0	0	0	0	2.2%

▼

40 codex entries

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	91	99.1%
o4 Mini	100	100	100	100	100	100	100	100	100	90	99.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	90	99.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	90	99.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	90	99.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	91	91	98.2%
o4 Mini High	100	100	100	100	100	100	100	100	91	91	98.2%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	91	90	98.1%
Z.AI GLM 5	100	100	100	100	100	100	100	100	90	90	98.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	90	89	97.9%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	89	89	97.8%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	89	89	97.8%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	91	90	90	97.1%
GPT-5	100	100	100	100	100	100	100	91	90	90	97.1%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	90	90	90	97.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	90	90	89	96.9%
Mistral Large	100	100	100	100	100	100	100	88	88	88	96.3%
Mistral Large 3	100	100	100	100	100	100	90	89	89	89	95.7%
Minimax M2.5	100	100	100	100	100	100	100	88	86	78	95.1%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	90	90	88	83	95.1%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	88	83	80	95.1%
Gemini 2.5 Flash	100	100	100	100	100	100	91	91	83	82	94.7%
Stealth: Aurora Alpha	100	100	100	100	100	100	90	89	89	75	94.3%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	33	93.3%
GPT-5.2	100	100	100	91	91	91	91	91	91	83	92.9%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	88	88	88	86	71	92.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	89	88	86	86	71	91.9%
Claude Opus 4	100	100	100	90	90	90	90	89	88	80	91.6%
GPT-5 Nano	100	100	100	100	91	88	88	88	88	70	91.1%
Claude 3.5 Sonnet	100	100	89	89	89	89	89	88	88	88	90.7%
GPT-4o, May 13th (temp=1)	100	100	100	89	88	86	86	86	83	75	89.2%
Qwen 3.5 Plus (2026-02-15)	100	100	90	90	90	90	90	82	80	80	89.2%
Z.AI GLM 4.6	100	100	100	100	90	90	89	83	70	67	88.9%
DeepSeek-V2 Chat	100	100	100	100	90	90	90	71	71	70	88.3%
Gemini 3 Flash (Preview)	100	90	90	90	90	90	90	80	80	80	88.0%
DeepSeek V3.2	100	100	91	90	90	89	88	78	78	69	87.2%
Gemma 3 27B	100	100	88	86	86	86	86	78	75	70	85.3%
DeepSeek V3 (2024-12-26)	100	90	90	88	88	86	83	80	78	70	85.2%
Hermes 3 405B	100	100	100	100	86	80	75	71	71	67	85.0%
GPT-4.1 Mini	100	88	88	88	86	86	83	78	78	75	84.8%
Ministral 3 14B	100	100	90	89	89	82	80	80	67	62	83.8%
DeepSeek V3 (2025-03-24)	100	100	90	89	80	75	73	70	70	67	81.3%
Claude Haiku 4.5	100	100	90	89	89	89	89	80	78	0	80.3%
Ministral 3 3B	100	100	100	86	80	75	75	67	60	50	79.2%
DeepSeek V3.1	100	100	90	88	80	80	75	73	56	50	79.1%
Writer: Palmyra X5	100	89	89	88	78	75	73	73	67	60	79.0%
GPT-4.1	100	89	80	80	78	75	73	73	73	56	77.5%
Z.AI GLM 4.5	100	89	89	89	86	83	78	75	44	40	77.3%
Qwen 2.5 72B	100	100	83	83	80	80	67	63	57	50	76.3%
Mistral Medium 3.1	90	83	80	80	75	70	70	64	60	58	73.1%
Mistral Small Creative	88	80	80	75	75	73	73	73	67	40	72.2%
Ministral 3 8B	88	88	88	80	80	78	75	75	55	0	70.5%
Ministral 3B	100	100	80	67	60	60	60	50	50	50	67.7%
Hermes 3 70B	100	75	71	67	67	60	56	50	50	43	63.8%
Mistral Small 3.2 24B	86	75	75	75	64	63	55	55	55	31	63.1%
WizardLM 2 8x22b	83	75	75	67	67	63	56	50	44	43	62.2%
Llama 3.1 Nemotron 70B	100	88	75	75	63	63	56	44	38	20	62.1%
Arcee AI: Trinity Mini	100	100	75	67	67	50	50	33	33	0	57.5%
Llama 3.1 70B	100	64	63	60	60	56	50	44	40	0	53.6%
GPT-4o Mini (temp=0)	67	57	50	50	40	40	40	40	40	25	44.9%
Cohere Command R+ (Aug. 2024)	63	63	60	60	60	38	31	22	20	0	41.5%
Arcee AI: Trinity Large (Preview)	100	75	73	64	47	44	1	1	0	0	40.5%
GPT-4o Mini (temp=1)	60	50	50	40	40	40	40	33	20	0	37.3%
Ministral 8B	75	70	70	67	60	0	0	0	0	0	34.2%
Gemini 2.5 Flash Lite	54	53	50	47	44	43	38	7	0	0	33.5%
Gemma 3 12B	50	40	40	40	40	20	20	20	20	0	29.0%
Claude 3 Haiku	33	33	33	25	25	25	20	20	20	17	25.2%
Llama 3.1 8B	67	50	31	25	14	0	0	0	0	0	18.7%
Mistral NeMO	80	40	22	15	10	9	6	0	0	0	18.3%
Gemma 3 4B	25	20	17	13	11	0	0	0	0	0	8.5%
Rocinante 12B	33	25	14	0	0	0	0	0	0	0	7.3%
GPT-4.1 Nano	5	0	0	0	0	0	0	0	0	0	0.5%

Precision

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

matrix

Large codex (40 entries), long passage (1,019 words)

Large codex (40 entries), short passage (165 words)

Small codex (7 entries), long passage (734 words)

Small codex (7 entries), short passage (165 words)

tiers

5 codex entries

10 codex entries

20 codex entries

40 codex entries