Accuracy

Test: Codex Extraction

Avg. Score

80.3%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Gemini 3 Flash (Preview)	90.9%	$0.0027	3.9s	85%
2	Z.AI GLM 5 Turbo	92.2%	$0.0068	16.0s	87%
3	Grok 4 Fast	89.1%	$0.0012	8.7s	80%
4	Qwen 3.5 Plus (2026-02-15)	90.4%	$0.0030	10.6s	81%
5	Gemini 3 Flash (Preview, Reasoning)	93.5%	$0.0096	22.2s	86%
6	Mistral Small Creative	87.4%	$0.0006	3.9s	77%
7	Stealth: Healer Alpha	89.3%	$0.0000	24.6s	83%
8	Grok 4.20 (Beta)	87.2%	$0.0049	2.0s	78%
9	Stealth: Hunter Alpha	90.8%	$0.0000	36.7s	84%
10	Mistral Medium 3.1	87.8%	$0.0026	5.8s	76%
11	Claude Haiku 4.5	86.8%	$0.0073	4.5s	80%
12	Grok 4.1 Fast	89.5%	$0.0017	22.1s	79%
13	Mistral Large 3	87.8%	$0.0027	8.2s	76%
14	Claude Sonnet 4.6	91.4%	$0.022	7.4s	86%
15	DeepSeek-V2 Chat	86.5%	$0.0019	14.8s	76%
16	Gemini 2.5 Flash	82.8%	$0.0023	2.5s	75%
17	Z.AI GLM 4.5	86.9%	$0.0028	16.8s	77%
18	Gemini 2.5 Flash (Reasoning)	86.6%	$0.0082	11.8s	79%
19	Mistral Large 2	88.0%	$0.011	8.3s	78%
20	Claude Opus 4.6	95.1%	$0.037	8.1s	90%
21	DeepSeek V3 (2024-12-26)	84.0%	$0.0017	13.5s	76%
22	Claude Opus 4.5	94.9%	$0.037	7.7s	89%
23	Grok 4.20 (Beta, Reasoning)	90.1%	$0.020	12.7s	80%
24	Gemini 2.5 Flash Lite (Reasoning)	84.3%	$0.0021	15.6s	72%
25	MiniMax M2.7	85.8%	$0.0022	21.7s	73%
26	DeepSeek V3.2	86.4%	$0.0011	36.7s	76%
27	Writer: Palmyra X5	80.9%	$0.0052	7.5s	74%
28	Qwen 3.5 Flash	88.5%	$0.0031	49.5s	81%
29	Z.AI GLM 4.6	88.3%	$0.0057	38.8s	78%
30	Mistral Small 3.2 24B	78.9%	$0.0005	4.4s	70%
31	Inception Mercury 2	80.2%	$0.0022	3.5s	70%
32	Z.AI GLM 5	91.1%	$0.0091	51.4s	83%
33	Ministral 3 14B	80.7%	$0.0009	6.0s	68%
34	Gemini 3.1 Flash Lite (Preview)	78.5%	$0.0017	2.0s	69%
35	GPT-5.4 Mini (Reasoning)	88.3%	$0.017	25.6s	80%
36	MiniMax M2.5	84.9%	$0.0023	30.1s	73%
37	Claude Sonnet 4.5	87.9%	$0.022	6.6s	75%
38	Grok 4	94.3%	$0.031	37.2s	88%
39	Ministral 3 8B	79.7%	$0.0006	3.3s	65%
40	Nemotron 3 Super	89.5%	$0.0000	1.0m	78%
41	Claude 3.7 Sonnet	86.3%	$0.021	7.8s	76%
42	Aion 2.0	92.6%	$0.0080	1.2m	86%
43	GPT-5 Mini	86.9%	$0.0070	45.5s	80%
44	DeepSeek V3 (2025-03-24)	84.7%	$0.0014	27.0s	70%
45	Qwen3 235B A22B Instruct 2507	80.2%	$0.0007	19.3s	71%
46	Hermes 3 405B	83.4%	$0.0040	17.8s	69%
47	Gemini 2.5 Pro	92.4%	$0.034	22.8s	85%
48	Ministral 8B	77.5%	$0.0004	3.8s	67%
49	GPT-4o, Aug. 6th (temp=0)	77.9%	$0.0100	4.0s	74%
50	Gemini 2.5 Flash Lite	77.6%	$0.0005	1.9s	65%
51	Claude Sonnet 4	86.7%	$0.022	7.9s	74%
52	GPT-5.4 Nano (Reasoning)	79.8%	$0.0023	12.2s	68%
53	Qwen 2.5 72B	77.2%	$0.0008	11.0s	67%
54	GPT-5.4 (Reasoning, Low)	83.1%	$0.017	10.4s	73%
55	WizardLM 2 8x22b	82.5%	$0.0026	31.3s	70%
56	GPT-5.4 Mini (Reasoning, Low)	76.5%	$0.0042	3.7s	67%
57	Qwen 3.5 35B	87.5%	$0.015	47.3s	80%
58	GPT-5.4 Mini	75.4%	$0.0031	2.2s	65%
59	GPT-5.2	83.0%	$0.018	16.8s	74%
60	GPT-5.4	79.7%	$0.011	6.4s	67%
61	o4 Mini	83.8%	$0.014	21.5s	72%
62	o4 Mini High	89.0%	$0.025	40.5s	81%
63	GPT-4.1	75.6%	$0.0073	4.7s	66%
64	GPT-5.1	92.1%	$0.035	43.6s	86%
65	Claude Opus 4.6 (Reasoning)	94.5%	$0.055	21.4s	89%
66	GPT-4o, Aug. 6th (temp=1)	74.4%	$0.0092	3.8s	66%
67	GPT-4o, May 13th (temp=0)	79.8%	$0.025	4.3s	72%
68	GPT-4.1 Mini	73.5%	$0.0015	6.6s	60%
69	Gemma 3 4B	70.1%	$0.0002	6.7s	63%
70	DeepSeek V3.1	85.4%	$0.0012	26.1s	54%
71	Claude 3 Haiku	70.8%	$0.0017	5.5s	62%
72	Hermes 3 70B	76.6%	$0.0012	15.8s	59%
73	Arcee AI: Trinity Mini	69.4%	$0.0003	5.8s	61%
74	Qwen 3 32B	75.9%	$0.0010	37.9s	68%
75	Mistral Small 4	69.0%	$0.0008	3.2s	60%
76	ByteDance Seed 2.0 Lite	87.1%	$0.0078	1.4m	77%
77	Mistral Large	83.3%	$0.011	7.6s	52%
78	Claude Sonnet 4.6 (Reasoning)	91.8%	$0.052	36.0s	86%
79	Mistral Small 4 (Reasoning)	72.6%	$0.0019	15.6s	59%
80	Z.AI GLM 4.7	90.0%	$0.0098	1.7m	81%
81	Ministral 3B	65.4%	$0.0002	1.8s	59%
82	GPT-4o, May 13th (temp=1)	76.2%	$0.025	4.0s	67%
83	Inception Mercury	66.9%	$0.0006	3.9s	57%
84	Claude 3.5 Sonnet	83.1%	$0.043	13.9s	76%
85	Qwen 3.5 397B A17B	91.3%	$0.012	1.8m	82%
86	Gemma 3 27B	67.8%	$0.0005	14.5s	58%
87	Ministral 3 3B	63.6%	$0.0004	1.8s	57%
88	GPT-4o Mini (temp=0)	66.1%	$0.0005	7.9s	57%
89	GPT-4o Mini (temp=1)	66.7%	$0.0006	8.0s	56%
90	GPT-5.4 Nano	64.7%	$0.0011	3.8s	56%
91	ByteDance Seed 1.6 Flash	72.1%	$0.0011	39.3s	61%
92	Llama 3.1 70B	71.9%	$0.0016	16.0s	51%
93	GPT-5.4 Nano (Reasoning, Low)	65.0%	$0.0011	3.7s	53%
94	ByteDance Seed 1.6	81.1%	$0.0073	1.3m	70%
95	Nemotron 3 Nano	82.0%	$0.0013	1.6m	72%
96	GPT-5	94.0%	$0.049	1.3m	89%
97	Gemini 3 Pro (Preview)	90.1%	$0.055	35.6s	80%
98	Qwen 3.5 9B	82.0%	$0.0013	1.5m	68%
99	GPT-5.4 (Reasoning)	87.0%	$0.044	53.6s	77%
100	Llama 3.1 Nemotron 70B	69.9%	$0.0050	17.3s	49%
101	Arcee AI: Trinity Large (Preview)	72.5%	$0.0000	20.5s	39%
102	Gemma 3 12B	63.0%	$0.0003	14.1s	46%
103	MoonshotAI: Kimi K2.5	88.9%	$0.013	2.4m	81%
104	Cohere Command R+ (Aug. 2024)	66.1%	$0.014	17.9s	51%
105	GPT-5 Nano	71.8%	$0.0043	1.3m	59%
106	GPT-4.1 Nano	51.8%	$0.0003	2.8s	42%
107	Z.AI GLM 4.7 Flash	69.1%	$0.0019	1.2m	52%
108	Gemini 3.1 Pro (Preview)	88.3%	$0.065	1.0m	78%
109	Llama 3.1 8B	56.2%	$0.0001	10.0s	37%
110	Claude Opus 4	91.4%	$0.110	13.5s	84%
111	ByteDance Seed 2.0 Mini	84.4%	$0.0034	3.4m	77%
112	Qwen 3.5 27B	81.4%	$0.021	1.6m	41%
113	Rocinante 12B	39.8%	$0.0013	25.6s	20%
114	Qwen 3.5 122B	90.4%	$0.079	3.6m	83%
115	Mistral NeMO	22.4%	$0.0006	1.4s	0%
116	LFM2 24B	17.5%	$0.0002	12.5s	0%
80.30%

Individual Scenarios

▼

Long: The Spire of Echoes (Dense)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Gemini 3 Pro (Preview)	96	96	95	95	95	95.2%
Gemini 3 Flash (Preview, Reasoning)	97	96	95	93	92	94.3%
Aion 2.0	98	94	93	92	90	93.6%
Claude Opus 4.5	95	93	93	93	93	93.1%
Claude Opus 4.6	93	93	92	92	92	92.2%
Claude Opus 4	92	92	92	92	91	91.7%
Claude Opus 4.6 (Reasoning)	96	91	91	91	89	91.5%
Nemotron 3 Super	96	92	91	89	89	91.4%
Gemini 3 Flash (Preview)	94	93	91	90	89	91.3%
Stealth: Hunter Alpha	96	95	90	89	86	91.3%
o4 Mini High	94	94	90	89	88	91.2%
GPT-5	93	93	91	90	88	91.1%
GPT-5.4 (Reasoning)	94	92	90	90	89	90.9%
Z.AI GLM 5	93	93	91	88	88	90.5%
Grok 4	92	91	90	90	86	89.9%
Claude Sonnet 4.6 (Reasoning)	92	90	89	89	88	89.7%
Gemini 2.5 Pro	92	91	89	88	86	89.4%
Gemini 3.1 Pro (Preview)	92	91	90	89	82	88.9%
Stealth: Healer Alpha	93	88	88	88	85	88.2%
Z.AI GLM 5 Turbo	89	89	88	87	87	87.9%
Qwen 3.5 27B	92	89	88	86	84	87.8%
GPT-5.4 Mini (Reasoning)	89	88	88	88	86	87.6%
Qwen 3.5 Plus (2026-02-15)	89	88	88	88	86	87.4%
GPT-5.1	89	88	87	87	85	87.3%
Claude Sonnet 4.6	90	89	87	85	85	87.2%
DeepSeek-V2 Chat	91	90	87	86	81	86.9%
Mistral Medium 3.1	87	87	87	86	85	86.7%
Qwen 3.5 122B	94	91	87	81	80	86.7%
Qwen 3.5 35B	92	92	85	83	82	86.7%
Gemini 2.5 Flash (Reasoning)	90	89	87	85	83	86.6%
GPT-5.4 (Reasoning, Low)	88	87	87	87	81	86.1%
Grok 4.20 (Beta, Reasoning)	93	88	87	84	78	86.0%
MoonshotAI: Kimi K2.5	91	88	87	86	78	86.0%
Nemotron 3 Nano	94	91	87	84	73	85.9%
Z.AI GLM 4.7	91	91	85	83	80	85.8%
Mistral Small Creative	89	87	85	84	83	85.6%
Qwen 3.5 397B A17B	89	86	85	84	83	85.4%
Z.AI GLM 4.5	88	86	85	85	82	85.4%
Qwen 3.5 Flash	89	89	86	82	80	85.3%
Grok 4 Fast	88	86	86	85	81	85.3%
GPT-5.2	88	87	85	84	81	85.1%
Gemini 2.5 Flash	88	88	85	83	82	85.1%
Claude Haiku 4.5	88	85	84	84	83	84.8%
Claude 3.7 Sonnet	89	84	84	83	83	84.6%
Gemini 2.5 Flash Lite (Reasoning)	92	88	81	80	79	84.2%
Mistral Large 2	86	86	85	84	79	84.1%
o4 Mini	93	89	87	82	68	84.1%
Z.AI GLM 4.6	87	87	86	81	80	84.0%
Inception Mercury 2	89	87	85	81	77	83.9%
GPT-5 Mini	89	83	83	83	80	83.7%
Grok 4.20 (Beta)	85	84	84	83	83	83.7%
Claude Sonnet 4	91	88	85	78	76	83.6%
DeepSeek V3.2	89	84	83	81	79	83.2%
Mistral Large 3	87	85	80	80	80	82.7%
GPT-5.4	88	83	82	82	79	82.6%
Hermes 3 405B	88	85	83	80	77	82.6%
GPT-5.4 Mini (Reasoning, Low)	86	83	82	80	79	81.8%
WizardLM 2 8x22b	88	85	84	80	71	81.6%
Mistral Small 3.2 24B	85	84	83	81	74	81.3%
Grok 4.1 Fast	88	88	80	76	73	80.9%
GPT-5.4 Nano (Reasoning)	88	84	81	76	75	80.8%
Claude 3.5 Sonnet	85	84	82	77	75	80.7%
Claude Sonnet 4.5	82	81	81	80	78	80.5%
ByteDance Seed 2.0 Lite	85	84	83	77	73	80.2%
MiniMax M2.7	84	82	80	78	75	80.0%
DeepSeek V3 (2024-12-26)	84	82	78	77	77	79.8%
GPT-4o, Aug. 6th (temp=0)	81	81	81	81	76	79.6%
Hermes 3 70B	85	83	82	79	68	79.2%
ByteDance Seed 2.0 Mini	83	82	78	78	75	79.1%
Qwen3 235B A22B Instruct 2507	78	78	78	77	76	77.4%
Writer: Palmyra X5	84	77	76	75	75	77.3%
GPT-4o, May 13th (temp=0)	80	78	78	77	73	77.2%
Qwen 3.5 9B	86	85	83	80	52	77.2%
MiniMax M2.5	84	82	80	75	65	77.1%
Ministral 3 14B	79	77	77	76	74	76.7%
Llama 3.1 Nemotron 70B	81	80	78	76	66	76.1%
ByteDance Seed 1.6	83	77	74	74	72	76.0%
Qwen 3 32B	82	82	79	69	69	76.0%
Gemini 3.1 Flash Lite (Preview)	78	78	77	74	74	76.0%
DeepSeek V3 (2025-03-24)	83	83	74	70	67	75.3%
Llama 3.1 70B	79	78	74	73	70	74.6%
GPT-5.4 Mini	80	77	73	71	71	74.3%
GPT-4.1 Mini	79	77	77	70	68	74.1%
GPT-4.1	76	74	74	74	69	73.3%
GPT-4o, Aug. 6th (temp=1)	77	76	74	71	69	73.3%
Gemini 2.5 Flash Lite	78	76	70	69	65	71.7%
DeepSeek V3.1	93	93	86	84	0	71.3%
GPT-4o Mini (temp=1)	75	74	74	70	63	71.2%
Mistral Small 4 (Reasoning)	79	78	70	69	60	71.0%
Ministral 8B	73	72	72	69	68	71.0%
GPT-4o, May 13th (temp=1)	76	74	71	69	65	70.9%
Ministral 3 8B	73	73	71	69	68	70.8%
GPT-5 Nano	79	78	77	72	47	70.3%
Mistral Small 4	76	72	71	70	61	70.1%
GPT-4o Mini (temp=0)	75	72	70	69	64	70.1%
Qwen 2.5 72B	76	72	72	66	65	70.0%
Cohere Command R+ (Aug. 2024)	85	72	67	64	62	69.9%
LFM2 24B	74	74	73	73	54	69.9%
Gemma 3 4B	73	72	71	67	65	69.6%
Arcee AI: Trinity Mini	75	71	69	67	65	69.3%
Claude 3 Haiku	71	69	66	65	64	67.0%
ByteDance Seed 1.6 Flash	78	77	70	63	45	67.0%
Mistral Large	87	83	82	80	0	66.4%
GPT-5.4 Nano (Reasoning, Low)	75	70	64	62	57	65.8%
GPT-5.4 Nano	73	71	70	69	44	65.4%
Inception Mercury	71	65	64	61	60	64.4%
Ministral 3B	69	65	63	62	59	63.7%
Gemma 3 12B	68	67	67	64	53	63.7%
Ministral 3 3B	66	63	61	61	61	62.2%
Z.AI GLM 4.7 Flash	80	61	60	59	42	60.5%
Gemma 3 27B	68	67	58	58	47	59.4%
Arcee AI: Trinity Large (Preview)	74	72	68	62	0	55.4%
Llama 3.1 8B	76	72	66	56	0	54.1%
GPT-4.1 Nano	58	48	46	39	32	44.6%
Rocinante 12B	77	71	68	0	0	43.1%
Mistral NeMO	0	0	0	0	0	0.0%

▼

Medium: The Hollow (Inferred)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Claude Opus 4.6	99	99	99	99	99	98.6%
Claude Opus 4.6 (Reasoning)	99	99	99	97	93	97.3%
Claude Opus 4.5	99	99	96	96	96	97.1%
GPT-5	96	95	95	95	95	94.9%
Grok 4	97	94	94	94	91	94.2%
GPT-5.1	96	95	95	91	91	93.5%
Gemini 3.1 Pro (Preview)	96	96	95	91	89	93.3%
Gemini 3 Pro (Preview)	95	94	93	91	88	92.3%
Z.AI GLM 5 Turbo	96	92	92	91	90	92.1%
Claude Sonnet 4.6	92	92	92	92	91	92.1%
Gemini 3 Flash (Preview, Reasoning)	96	93	92	91	86	91.7%
Aion 2.0	95	92	92	90	89	91.5%
Z.AI GLM 4.7	98	94	90	88	86	91.3%
Qwen 3.5 Flash	96	93	92	89	86	91.3%
Claude Sonnet 4.6 (Reasoning)	92	92	91	91	90	91.2%
Mistral Large 2	94	91	91	90	88	90.9%
Mistral Large 3	92	92	92	92	86	90.8%
Qwen 3.5 27B	95	94	90	89	85	90.6%
GPT-5.4 (Reasoning)	92	91	90	90	89	90.4%
Qwen 3.5 122B	93	91	91	90	87	90.3%
Gemini 2.5 Pro	93	92	90	90	85	90.1%
Z.AI GLM 4.6	96	94	92	84	83	89.8%
Qwen 3.5 397B A17B	96	92	91	87	82	89.5%
Grok 4.20 (Beta, Reasoning)	96	89	88	88	86	89.4%
DeepSeek V3.1	90	90	89	89	89	89.4%
Mistral Large	92	90	90	88	86	89.1%
Claude Opus 4	95	91	90	86	84	89.0%
Stealth: Healer Alpha	91	91	90	87	85	88.8%
Stealth: Hunter Alpha	93	90	89	87	83	88.6%
DeepSeek-V2 Chat	91	91	91	88	81	88.4%
DeepSeek V3.2	91	91	89	87	84	88.3%
GPT-5 Mini	92	91	88	85	84	87.9%
ByteDance Seed 2.0 Mini	89	89	88	88	83	87.5%
Qwen 3.5 Plus (2026-02-15)	91	90	88	85	84	87.5%
Grok 4.1 Fast	90	90	88	86	83	87.4%
Grok 4 Fast	93	91	89	85	80	87.4%
Gemini 3 Flash (Preview)	91	89	86	86	85	87.3%
Claude 3.5 Sonnet	90	89	88	86	83	87.2%
Qwen 3.5 35B	91	88	87	86	85	87.2%
ByteDance Seed 2.0 Lite	91	89	88	83	83	86.8%
DeepSeek V3 (2025-03-24)	88	87	87	86	85	86.6%
MiniMax M2.7	89	87	87	84	83	85.9%
MoonshotAI: Kimi K2.5	89	89	89	83	79	85.8%
Gemini 2.5 Flash (Reasoning)	89	87	85	85	82	85.6%
DeepSeek V3 (2024-12-26)	91	89	88	82	78	85.6%
MiniMax M2.5	89	87	86	84	80	85.3%
GPT-5.4 Mini (Reasoning)	90	88	88	81	80	85.3%
o4 Mini High	90	88	87	82	79	85.3%
Claude Haiku 4.5	86	86	84	84	84	85.2%
Claude Sonnet 4.5	86	86	85	85	83	85.1%
Qwen 3.5 9B	93	88	88	86	70	85.1%
Gemini 3.1 Flash Lite (Preview)	88	87	86	84	80	84.8%
Grok 4.20 (Beta)	87	86	85	84	82	84.8%
Z.AI GLM 4.5	93	85	83	82	79	84.5%
Gemini 2.5 Flash Lite (Reasoning)	90	88	84	82	76	84.0%
Mistral Small Creative	85	85	84	83	83	84.0%
Z.AI GLM 5	87	85	84	82	81	83.7%
GPT-5.2	89	85	82	81	80	83.6%
WizardLM 2 8x22b	88	88	83	81	76	83.2%
GPT-5.4	85	84	84	81	81	83.1%
Nemotron 3 Super	86	86	82	81	80	83.0%
GPT-5.4 (Reasoning, Low)	85	85	84	83	78	82.9%
GPT-5 Nano	88	86	82	81	78	82.9%
Nemotron 3 Nano	90	85	81	81	77	82.7%
Mistral Small 3.2 24B	84	84	84	81	77	82.2%
Claude 3.7 Sonnet	85	82	82	81	81	82.1%
ByteDance Seed 1.6	86	83	81	80	79	82.0%
Mistral Medium 3.1	84	84	82	80	80	81.9%
Writer: Palmyra X5	85	84	82	79	78	81.7%
Ministral 3 8B	82	82	82	82	79	81.6%
Claude Sonnet 4	86	81	81	80	78	81.2%
GPT-4o, May 13th (temp=0)	83	83	82	80	78	81.1%
Ministral 3 14B	84	82	82	81	77	81.1%
Gemini 2.5 Flash	84	84	82	79	76	80.8%
Qwen 2.5 72B	82	82	81	80	76	80.3%
Qwen3 235B A22B Instruct 2507	90	84	82	80	64	80.2%
Ministral 8B	82	81	81	79	77	80.1%
GPT-4o, Aug. 6th (temp=0)	83	80	80	80	77	80.0%
o4 Mini	87	85	83	75	70	79.9%
Inception Mercury 2	91	79	77	76	75	79.6%
Gemini 2.5 Flash Lite	87	82	81	80	67	79.3%
GPT-4.1 Mini	86	79	79	77	76	79.2%
GPT-4o, May 13th (temp=1)	84	82	81	80	68	78.7%
GPT-4o, Aug. 6th (temp=1)	84	82	80	79	65	78.0%
Hermes 3 405B	82	79	77	77	74	78.0%
GPT-4.1	81	80	80	76	72	77.9%
GPT-5.4 Mini (Reasoning, Low)	82	80	76	76	73	77.7%
Llama 3.1 70B	80	80	80	79	66	77.0%
GPT-5.4 Mini	79	79	79	75	70	76.6%
Qwen 3 32B	85	79	76	75	68	76.6%
Hermes 3 70B	81	77	74	74	72	75.4%
Claude 3 Haiku	81	79	74	70	68	74.5%
Z.AI GLM 4.7 Flash	82	81	70	68	66	73.4%
Mistral Small 4 (Reasoning)	77	76	73	71	68	73.1%
GPT-5.4 Nano (Reasoning)	83	82	72	63	63	72.5%
GPT-4o Mini (temp=1)	83	76	69	67	65	71.8%
ByteDance Seed 1.6 Flash	82	78	67	67	62	71.1%
Gemma 3 4B	73	72	72	70	68	71.1%
Arcee AI: Trinity Mini	78	74	69	68	61	69.8%
Mistral Small 4	81	70	68	68	57	68.8%
Inception Mercury	75	74	65	63	58	67.0%
Arcee AI: Trinity Large (Preview)	86	86	82	78	0	66.5%
GPT-4o Mini (temp=0)	73	70	67	62	60	66.3%
Cohere Command R+ (Aug. 2024)	72	66	65	63	58	64.8%
GPT-5.4 Nano	72	64	61	61	58	63.1%
Llama 3.1 8B	70	68	61	60	56	62.8%
Ministral 3B	66	63	63	62	60	62.8%
Gemma 3 27B	72	70	62	57	53	62.7%
Ministral 3 3B	63	63	62	61	61	61.6%
GPT-5.4 Nano (Reasoning, Low)	71	66	64	55	48	60.9%
GPT-4.1 Nano	66	57	56	54	53	57.3%
Llama 3.1 Nemotron 70B	73	71	69	66	0	55.9%
Rocinante 12B	76	71	70	60	0	55.4%
Gemma 3 12B	71	71	67	65	0	54.9%
Mistral NeMO	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0.0%

▼

Medium: Through the Thornveil (Scattered)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Grok 4	99	96	95	93	92	95.2%
Claude Opus 4.6 (Reasoning)	96	95	95	95	95	94.9%
Z.AI GLM 5 Turbo	96	95	94	94	93	94.5%
Claude Opus 4.6	95	95	95	95	93	94.3%
Gemini 3.1 Pro (Preview)	96	95	94	94	92	94.3%
Gemini 3 Flash (Preview, Reasoning)	98	95	94	93	92	94.3%
Z.AI GLM 5	96	95	95	92	91	93.9%
Grok 4.1 Fast	96	94	94	93	92	93.7%
Gemini 3 Flash (Preview)	95	95	93	93	91	93.6%
Qwen 3.5 397B A17B	95	95	93	92	89	92.9%
Claude Opus 4.5	94	94	94	94	88	92.9%
GPT-5	94	93	93	92	92	92.7%
Gemini 2.5 Pro	94	93	93	91	90	92.2%
GPT-5.1	95	92	92	92	88	92.0%
GPT-5.4 (Reasoning)	96	92	92	90	89	91.8%
Gemini 3 Pro (Preview)	94	94	92	91	88	91.6%
Claude Sonnet 4.6	93	91	91	91	91	91.4%
MoonshotAI: Kimi K2.5	95	93	90	89	89	91.2%
GPT-5.4 (Reasoning, Low)	95	91	90	90	89	90.9%
Claude Sonnet 4.6 (Reasoning)	92	91	91	90	90	90.7%
GPT-5 Mini	93	90	90	90	89	90.6%
Stealth: Hunter Alpha	94	91	90	89	88	90.3%
Aion 2.0	92	91	91	91	86	90.3%
Grok 4 Fast	94	92	90	88	87	90.1%
Qwen 3.5 122B	93	92	90	90	86	89.9%
Claude Haiku 4.5	92	90	90	90	88	89.7%
Grok 4.20 (Beta, Reasoning)	94	92	88	88	85	89.5%
Qwen 3.5 27B	92	91	90	89	86	89.5%
Z.AI GLM 4.6	94	92	90	87	83	89.3%
ByteDance Seed 2.0 Lite	93	92	90	86	85	89.2%
o4 Mini High	92	90	88	88	87	89.0%
Qwen 3.5 Plus (2026-02-15)	91	89	89	88	87	88.9%
Z.AI GLM 4.7	95	91	89	85	84	88.8%
GPT-5.4	92	88	88	88	86	88.5%
Stealth: Healer Alpha	93	91	89	85	84	88.5%
Claude Sonnet 4.5	93	90	87	87	85	88.4%
Claude Opus 4	92	90	88	86	86	88.4%
GPT-5.2	92	92	88	87	80	87.8%
Gemini 2.5 Flash	90	89	89	88	83	87.7%
DeepSeek V3.1	89	88	88	85	84	87.0%
GPT-5.4 Mini (Reasoning)	90	87	87	87	84	87.0%
MiniMax M2.5	92	87	86	86	85	87.0%
Claude 3.5 Sonnet	88	88	87	86	85	87.0%
Grok 4.20 (Beta)	89	88	87	85	85	86.9%
o4 Mini	90	88	88	83	82	86.2%
Claude Sonnet 4	91	86	86	85	83	86.2%
Nemotron 3 Super	91	89	84	84	83	86.1%
Qwen 3.5 Flash	93	87	87	82	81	85.8%
Gemini 2.5 Flash (Reasoning)	89	88	84	84	83	85.7%
Writer: Palmyra X5	89	89	87	81	80	85.4%
Claude 3.7 Sonnet	90	88	85	84	79	85.2%
Qwen 3.5 35B	87	87	84	84	83	85.1%
DeepSeek V3.2	87	87	85	84	81	85.1%
Mistral Medium 3.1	87	86	84	84	84	85.0%
Mistral Small Creative	88	88	84	83	81	84.7%
Z.AI GLM 4.5	87	85	84	83	79	83.4%
Qwen3 235B A22B Instruct 2507	88	87	87	80	73	83.0%
Qwen 3.5 9B	88	87	85	78	75	82.7%
MiniMax M2.7	92	83	81	80	77	82.6%
ByteDance Seed 2.0 Mini	85	85	83	81	79	82.5%
Mistral Large 2	87	82	81	81	81	82.4%
Mistral Large	83	83	82	81	81	82.0%
DeepSeek V3 (2024-12-26)	89	88	78	77	76	81.7%
Mistral Large 3	83	81	81	81	81	81.6%
GPT-4.1	86	84	83	81	71	81.1%
Arcee AI: Trinity Large (Preview)	88	83	79	77	76	80.6%
DeepSeek V3 (2025-03-24)	84	82	79	79	78	80.5%
DeepSeek-V2 Chat	89	88	83	71	68	79.9%
GPT-4o, May 13th (temp=0)	81	80	79	79	79	79.9%
Gemini 3.1 Flash Lite (Preview)	85	81	78	77	76	79.5%
ByteDance Seed 1.6 Flash	83	83	81	76	74	79.3%
GPT-4o, May 13th (temp=1)	83	82	80	76	74	79.1%
ByteDance Seed 1.6	88	80	77	76	72	78.6%
Hermes 3 405B	82	79	79	77	77	78.5%
Nemotron 3 Nano	80	80	79	77	75	78.2%
GPT-4o, Aug. 6th (temp=0)	78	78	78	78	78	78.1%
GPT-4.1 Mini	86	81	78	72	71	77.5%
GPT-5.4 Nano (Reasoning)	86	81	78	75	67	77.3%
GPT-4o, Aug. 6th (temp=1)	81	78	77	76	75	77.3%
Gemini 2.5 Flash Lite (Reasoning)	80	79	79	76	69	76.9%
WizardLM 2 8x22b	86	79	75	73	66	75.8%
Mistral Small 3.2 24B	77	77	77	75	70	75.3%
Inception Mercury 2	82	80	77	74	63	75.3%
GPT-5.4 Mini (Reasoning, Low)	78	76	76	75	71	75.2%
GPT-4o Mini (temp=0)	76	76	76	76	71	75.0%
GPT-5.4 Mini	86	74	74	70	68	74.3%
Ministral 3 8B	79	76	75	72	70	74.3%
Qwen 2.5 72B	77	75	74	74	71	74.2%
Gemma 3 27B	79	79	71	71	70	74.1%
Ministral 3 14B	77	75	75	72	72	74.1%
Ministral 8B	77	73	72	71	70	72.8%
Claude 3 Haiku	82	75	75	66	66	72.6%
GPT-5 Nano	81	79	73	67	63	72.4%
Qwen 3 32B	76	73	72	71	67	71.9%
Gemini 2.5 Flash Lite	75	74	71	70	69	71.8%
Mistral Small 4 (Reasoning)	80	78	72	66	63	71.7%
Hermes 3 70B	81	73	73	68	62	71.6%
Mistral Small 4	73	72	71	67	63	69.4%
GPT-4o Mini (temp=1)	76	71	69	67	64	69.4%
Z.AI GLM 4.7 Flash	84	75	73	69	42	68.5%
Llama 3.1 Nemotron 70B	76	68	67	65	65	68.3%
Ministral 3 3B	74	73	71	64	59	68.1%
Ministral 3B	74	70	70	65	56	67.3%
Arcee AI: Trinity Mini	77	71	69	61	58	67.0%
Gemma 3 12B	67	67	62	61	60	63.6%
Gemma 3 4B	75	63	61	59	55	62.4%
Inception Mercury	66	63	62	61	60	62.4%
GPT-5.4 Nano (Reasoning, Low)	69	69	64	58	52	62.3%
GPT-5.4 Nano	69	68	65	59	48	61.8%
Cohere Command R+ (Aug. 2024)	69	65	60	58	53	61.0%
Llama 3.1 70B	80	79	69	63	0	58.1%
GPT-4.1 Nano	53	53	51	48	45	50.1%
Llama 3.1 8B	66	66	56	55	0	48.3%
Mistral NeMO	82	72	72	0	0	45.3%
Rocinante 12B	72	60	56	16	0	40.9%
LFM2 24B	0	0	0	0	0	0.0%

▼

Short: The Rusty Lantern (Explicit)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Gemini 2.5 Pro	98	98	98	98	96	97.9%
Qwen 3.5 Plus (2026-02-15)	98	98	98	98	96	97.9%
Grok 4	98	98	98	97	96	97.7%
Claude Sonnet 4.5	98	98	98	96	96	97.5%
Mistral Medium 3.1	98	98	97	97	96	97.5%
GPT-5	98	97	97	97	97	97.3%
Qwen 3.5 397B A17B	98	98	98	97	94	97.3%
Nemotron 3 Super	98	98	97	97	96	97.2%
Claude Opus 4.5	98	98	98	94	94	96.7%
DeepSeek V3 (2025-03-24)	99	97	97	95	94	96.5%
Claude Opus 4	98	98	98	94	93	96.5%
Mistral Large 3	98	98	95	95	95	96.3%
Z.AI GLM 5	97	97	97	95	94	96.2%
Grok 4.1 Fast	98	96	96	96	94	96.0%
Claude Sonnet 4	96	96	96	96	96	96.0%
Claude Sonnet 4.6 (Reasoning)	98	96	96	94	94	95.7%
GPT-5.1	98	96	96	95	92	95.6%
Mistral Large	98	98	95	95	91	95.6%
Claude Opus 4.6	95	95	95	95	95	95.3%
Grok 4.20 (Beta, Reasoning)	98	97	96	94	90	95.2%
Mistral Small Creative	95	95	95	95	95	95.2%
MiniMax M2.7	98	98	96	94	88	94.9%
Aion 2.0	98	98	97	91	90	94.9%
Claude Sonnet 4.6	96	96	96	93	93	94.8%
Qwen 3.5 122B	98	96	94	92	92	94.7%
Hermes 3 405B	95	95	95	94	93	94.6%
Mistral Large 2	98	95	95	92	92	94.6%
Z.AI GLM 4.5	95	95	95	95	92	94.3%
Z.AI GLM 5 Turbo	96	96	95	93	92	94.2%
Claude Opus 4.6 (Reasoning)	94	94	94	94	94	94.1%
DeepSeek V3.1	97	97	96	91	89	93.9%
Gemini 3 Flash (Preview, Reasoning)	98	97	97	97	79	93.9%
Z.AI GLM 4.7	98	94	94	93	90	93.9%
Grok 4 Fast	97	95	94	94	88	93.7%
Grok 4.20 (Beta)	96	95	94	92	90	93.7%
GPT-5.4 Mini (Reasoning)	98	95	94	93	87	93.5%
Claude 3.7 Sonnet	94	94	94	92	92	93.2%
Stealth: Hunter Alpha	97	95	94	92	87	93.1%
MoonshotAI: Kimi K2.5	96	94	92	90	90	92.4%
Ministral 3 8B	94	94	94	91	90	92.3%
Gemini 2.5 Flash Lite (Reasoning)	96	94	93	92	87	92.3%
ByteDance Seed 2.0 Lite	96	94	94	90	86	92.0%
Stealth: Healer Alpha	96	95	92	88	87	91.9%
Gemini 3 Flash (Preview)	94	94	92	92	85	91.5%
Qwen 3.5 Flash	97	92	90	90	88	91.5%
Qwen 3.5 35B	95	92	92	92	86	91.2%
Ministral 3 14B	95	92	92	88	88	91.0%
DeepSeek-V2 Chat	96	92	92	90	84	90.9%
o4 Mini High	97	95	94	90	78	90.7%
MiniMax M2.5	96	96	95	89	75	90.2%
Z.AI GLM 4.6	96	96	89	85	84	90.1%
WizardLM 2 8x22b	97	95	93	87	75	89.4%
DeepSeek V3.2	98	95	92	89	70	88.9%
DeepSeek V3 (2024-12-26)	92	90	90	87	86	88.9%
GPT-5.4 Nano (Reasoning)	91	91	88	88	86	88.7%
Gemini 2.5 Flash (Reasoning)	98	91	90	88	75	88.4%
ByteDance Seed 2.0 Mini	94	89	88	87	85	88.4%
ByteDance Seed 1.6	98	92	86	81	81	87.7%
Claude Haiku 4.5	94	90	90	90	74	87.7%
Gemini 2.5 Flash Lite	92	88	87	86	85	87.5%
Arcee AI: Trinity Large (Preview)	95	92	87	85	78	87.5%
Ministral 8B	95	87	83	83	82	86.1%
GPT-5 Mini	93	92	92	79	73	85.6%
o4 Mini	96	94	93	72	69	84.8%
Qwen 2.5 72B	90	87	83	81	81	84.4%
Qwen 3.5 9B	93	93	85	80	64	82.8%
Inception Mercury 2	88	86	81	78	76	82.0%
Gemini 3 Pro (Preview)	99	78	78	77	75	81.4%
Nemotron 3 Nano	89	82	80	79	77	81.3%
GPT-4o, May 13th (temp=0)	96	79	77	77	76	81.1%
Hermes 3 70B	100	91	88	73	48	80.0%
Qwen3 235B A22B Instruct 2507	81	81	81	79	77	79.9%
Llama 3.1 Nemotron 70B	82	82	79	79	75	79.3%
Writer: Palmyra X5	81	81	79	79	75	79.3%
Qwen 3 32B	84	81	80	76	74	78.9%
Llama 3.1 70B	82	81	78	77	70	77.9%
Gemini 2.5 Flash	78	78	78	77	77	77.7%
Claude 3.5 Sonnet	79	77	77	77	77	77.5%
Gemma 3 4B	81	78	76	76	75	77.3%
Gemini 3.1 Pro (Preview)	78	78	78	75	75	76.9%
Mistral Small 3.2 24B	89	81	74	72	68	76.8%
GPT-5.4 Mini	92	81	71	70	68	76.6%
GPT-4o, May 13th (temp=1)	89	78	77	75	62	76.3%
GPT-5.2	77	76	75	75	73	75.3%
Gemma 3 27B	79	77	76	72	72	75.0%
GPT-5.4 (Reasoning)	76	76	75	74	74	74.9%
Mistral Small 4 (Reasoning)	90	88	81	66	47	74.3%
Inception Mercury	84	77	73	69	67	73.8%
GPT-4o, Aug. 6th (temp=0)	74	74	74	74	72	73.8%
Z.AI GLM 4.7 Flash	92	78	70	68	61	73.7%
Gemini 3.1 Flash Lite (Preview)	77	77	76	71	67	73.5%
GPT-5.4 (Reasoning, Low)	75	74	73	72	70	72.6%
Arcee AI: Trinity Mini	79	74	71	70	63	71.5%
GPT-5.4 Mini (Reasoning, Low)	85	73	73	68	57	71.4%
ByteDance Seed 1.6 Flash	81	76	70	67	62	71.2%
GPT-5.4 Nano (Reasoning, Low)	81	80	73	63	59	71.2%
GPT-4.1	77	74	72	65	62	70.0%
Gemma 3 12B	79	75	71	71	54	69.9%
GPT-4o, Aug. 6th (temp=1)	81	70	65	65	64	68.9%
Claude 3 Haiku	75	71	69	68	62	68.9%
Cohere Command R+ (Aug. 2024)	88	81	71	67	36	68.6%
GPT-5.4 Nano	77	73	70	63	60	68.5%
Ministral 3B	72	70	68	67	63	68.0%
Mistral Small 4	84	70	65	63	55	67.6%
GPT-5.4	67	65	64	64	62	64.5%
GPT-4.1 Mini	77	73	70	63	34	63.1%
Ministral 3 3B	65	64	63	62	58	62.6%
GPT-5 Nano	78	70	58	56	47	61.7%
Llama 3.1 8B	70	63	63	60	42	59.6%
Qwen 3.5 27B	98	96	94	0	0	57.7%
GPT-4.1 Nano	79	66	53	41	37	55.1%
GPT-4o Mini (temp=1)	64	54	54	50	50	54.5%
GPT-4o Mini (temp=0)	54	54	54	54	50	52.9%
Mistral NeMO	81	80	61	0	0	44.3%
Rocinante 12B	69	29	0	0	0	19.7%
LFM2 24B	0	0	0	0	0	0.0%

Accuracy

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

Long: The Spire of Echoes (Dense)

Medium: The Hollow (Inferred)

Medium: Through the Thornveil (Scattered)

Short: The Rusty Lantern (Explicit)