Medium: The Hollow (Inferred)

Codex Extraction

Evaluates a model's ability to extract structured codex entries (characters, locations, objects, lore) from prose passages and return them as well-formed XML.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6	99%
Claude Opus 4.6 (Reasoning)	99%
Claude Opus 4.7	99%
GPT-5	99%
Claude Opus 4.7 (Reasoning)	99%
Claude Opus 4.5	99%
GPT-5.1	98%
Grok 4	98%
Claude Sonnet 4.6	98%
Gemini 3.1 Pro (Preview)	98%
Gemini 3 Flash (Preview, Reasoning)	98%
Grok 4.20 (Reasoning)	98%
GPT-5.4 (Reasoning)	98%
Gemini 3.5 Flash (Reasoning)	98%
Qwen 3.5 27B	97%
Claude Sonnet 4.6 (Reasoning)	97%
Z.AI GLM 4.7	97%
Claude Opus 4	97%
Qwen 3.5 397B A17B	97%
Gemini 3 Pro (Preview)	97%

	Score	Cost	Time
Gemini 3.1 Flash Lite (Reasoning)	96%	$0.0016	1.7s
Gemini 3.1 Flash Lite	96%	$0.0016	1.8s
Gemini 3.1 Flash Lite (Preview)	96%	$0.0016	2.2s
DeepSeek V4 Flash	96%	$0.0003	7.0s
Gemini 2.5 Flash Lite	92%	$0.0004	1.9s
Grok 4 Fast	96%	$0.0012	7.5s
Mistral Small Creative	94%	$0.0005	3.4s
Gemini 2.5 Flash	95%	$0.0019	2.2s
Ministral 3 8B	94%	$0.0006	2.6s
Mistral Large 3	97%	$0.0025	6.6s
Ministral 8B	93%	$0.0004	3.0s
Gemma 4 31B	97%	$0.0007	14.8s
Qwen 3.5 Plus (2026-02-15)	97%	$0.0027	8.7s
Z.AI GLM 4.5	96%	$0.0024	14.6s
Mistral Small 3.2 24B	93%	$0.0005	4.1s
Gemini 3 Flash (Preview)	95%	$0.0025	3.3s
Inception Mercury 2	95%	$0.0020	3.2s
Gemma 4 26B	94%	$0.0005	16.6s
Gemini 2.5 Flash Lite (Reasoning)	95%	$0.0016	11.4s
Cydonia 24B V4.1	93%	$0.0009	9.0s

	Score	Cost	Speed	Stability
Gemini 3.1 Flash Lite (Reasoning)	96%	$0.0016	1.7s	95%
Gemini 3.1 Flash Lite (Preview)	96%	$0.0016	2.2s	95%
Gemini 3.1 Flash Lite	96%	$0.0016	1.8s	95%
Mistral Large 3	97%	$0.0025	6.6s	96%
DeepSeek V4 Flash	96%	$0.0003	7.0s	93%
Qwen 3.5 Plus (2026-02-15)	97%	$0.0027	8.7s	96%
Grok 4 Fast	96%	$0.0012	7.5s	94%
Ministral 3 8B	94%	$0.0006	2.6s	92%
Gemma 4 31B	97%	$0.0007	14.8s	95%
Mistral Small Creative	94%	$0.0005	3.4s	92%
Gemini 2.5 Flash	95%	$0.0019	2.2s	93%
Ministral 8B	93%	$0.0004	3.0s	92%
Gemini 3 Flash (Preview)	95%	$0.0025	3.3s	92%
Inception Mercury 2	95%	$0.0020	3.2s	91%
Mistral Small 3.2 24B	93%	$0.0005	4.1s	92%
Claude Haiku 4.5	95%	$0.0066	3.8s	95%
GPT-5.4 Mini (Reasoning, Low)	94%	$0.0039	3.5s	93%
GPT-5.4 Mini	94%	$0.0029	2.0s	92%
Gemini 2.5 Flash Lite (Reasoning)	95%	$0.0016	11.4s	93%
Gemini 3.5 Flash (Reasoning, Minimal)	97%	$0.010	2.8s	95%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	Total
94	Claude Opus 4.6	$0.034	7.1s	98%	100	100	99	99	99	99%
132	Claude Opus 4.6 (Reasoning)	$0.056	25.0s	97%	100	99	99	99	97	99%
107	Claude Opus 4.7	$0.043	5.0s	97%	100	99	99	98	98	99%
137	GPT-5	$0.044	1.1m	98%	99	99	99	99	99	99%
104	Claude Opus 4.7 (Reasoning)	$0.042	4.8s	98%	99	99	99	99	98	99%
96	Claude Opus 4.5	$0.035	6.8s	97%	100	99	98	98	98	99%
117	GPT-5.1	$0.031	40.6s	98%	99	99	99	98	98	98%
122	Grok 4	$0.032	40.2s	96%	99	99	98	98	97	98%
54	Claude Sonnet 4.6	$0.021	6.9s	98%	98	98	98	98	98	98%
141	Gemini 3.1 Pro (Preview)	$0.064	57.7s	96%	99	99	98	98	96	98%
25	Gemini 3 Flash (Preview, Reasoning)	$0.0086	14.7s	96%	99	98	98	98	97	98%
87	Grok 4.20 (Reasoning)	$0.012	45.1s	97%	99	98	98	98	97	98%
130	GPT-5.4 (Reasoning)	$0.042	45.9s	97%	98	98	98	98	97	98%
109	Gemini 3.5 Flash (Reasoning)	$0.035	14.2s	95%	99	98	97	97	96	98%
110	Qwen 3.5 27B	$0.014	1.1m	95%	99	98	97	97	96	97%
136	Claude Sonnet 4.6 (Reasoning)	$0.053	39.3s	97%	98	98	98	97	97	97%
125	Z.AI GLM 4.7	$0.010	1.6m	96%	99	97	97	97	96	97%
146	Claude Opus 4	$0.099	12.1s	96%	99	98	97	96	96	97%
124	Qwen 3.5 397B A17B	$0.012	1.4m	95%	99	98	98	97	95	97%
134	Gemini 3 Pro (Preview)	$0.052	34.3s	95%	99	98	97	97	95	97%
69	Grok 4.20 (Beta, Reasoning)	$0.021	12.5s	96%	98	97	97	97	97	97%
102	Aion 2.0	$0.0075	1.1m	95%	99	98	98	97	94	97%
9	Gemma 4 31B	$0.0007	14.8s	95%	98	98	97	96	96	97%
133	Qwen3.7 Max	$0.040	1.1m	97%	97	97	97	97	96	97%
106	Gemini 2.5 Pro	$0.031	21.7s	95%	98	97	97	96	96	97%
135	Gemma 4 31B (Reasoning)	$0.0017	2.4m	94%	99	97	96	96	96	97%
6	Qwen 3.5 Plus (2026-02-15)	$0.0027	8.7s	96%	98	98	97	96	96	97%
20	Gemini 3.5 Flash (Reasoning, Minimal)	$0.010	2.8s	95%	97	97	97	97	95	97%
98	GPT-5.5 (Reasoning, Low)	$0.031	10.2s	95%	98	97	97	97	95	97%
144	MoonshotAI: Kimi K2.6	$0.028	2.7m	95%	97	97	97	97	95	97%
118	Z.AI GLM 5.1	$0.018	1.0m	96%	98	97	97	96	96	97%
95	Qwen 3.5 35B	$0.013	44.1s	96%	98	97	97	96	96	97%
138	Qwen 3.5 122B	$0.032	1.4m	95%	98	98	98	97	94	97%
50	DeepSeek V4 Flash (Reasoning)	$0.0009	37.8s	95%	98	97	97	97	95	97%
52	Z.AI GLM 4.6	$0.0060	24.6s	93%	99	98	98	96	92	97%
4	Mistral Large 3	$0.0025	6.6s	96%	97	97	97	97	96	97%
29	Mistral Large 2	$0.010	6.4s	94%	98	98	97	95	94	96%
114	o4 Mini High	$0.025	41.2s	95%	98	97	97	96	95	96%
44	Z.AI GLM 5 Turbo	$0.0066	19.6s	94%	98	97	97	95	94	96%
62	Claude Sonnet 4.5	$0.020	6.0s	96%	97	97	96	96	96	96%
111	ByteDance Seed 2.0 Lite	$0.0076	1.3m	95%	97	97	96	96	95	96%
5	DeepSeek V4 Flash	$0.0003	7.0s	93%	98	97	95	95	95	96%
149	Qwen3.6 Max Preview	$0.045	2.5m	93%	98	97	96	95	95	96%
1	Gemini 3.1 Flash Lite (Reasoning)	$0.0016	1.7s	95%	97	96	96	96	95	96%
116	Claude 3.5 Sonnet	$0.039	12.2s	96%	96	96	96	96	96	96%
127	Qwen 3.5 Plus (2026-04-20)	$0.015	1.4m	94%	97	97	97	96	94	96%
23	Grok 4.1 Fast	$0.0015	22.0s	95%	97	96	96	96	96	96%
28	DeepSeek V3.1	$0.0011	22.6s	94%	98	97	96	95	94	96%
101	Z.AI GLM 5	$0.0095	57.4s	95%	97	96	96	95	95	96%
92	Qwen 3.5 Flash	$0.0031	51.8s	91%	99	97	97	96	90	96%
2	Gemini 3.1 Flash Lite (Preview)	$0.0016	2.2s	95%	96	96	96	96	95	96%
22	Xiaomi MIMO v2.5	$0.0033	14.2s	94%	97	97	96	94	94	96%
63	GPT-5.4 Mini (Reasoning)	$0.012	16.7s	93%	97	97	95	95	95	96%
76	GPT-5 Mini	$0.0060	39.1s	94%	97	97	96	95	94	96%
145	MoonshotAI: Kimi K2.5	$0.014	3.1m	93%	97	97	97	96	91	96%
3	Gemini 3.1 Flash Lite	$0.0016	1.8s	95%	96	96	96	96	94	96%
66	DeepSeek V3.2	$0.0011	40.3s	93%	98	97	95	95	94	96%
88	GPT-5.5	$0.023	5.0s	92%	98	97	97	96	91	96%
7	Grok 4 Fast	$0.0012	7.5s	94%	96	96	96	95	94	96%
24	Z.AI GLM 4.5	$0.0024	14.6s	93%	98	96	96	95	94	96%
112	DeepSeek V4 Pro (Reasoning)	$0.0032	1.4m	94%	96	96	96	94	94	96%
67	Claude 3.7 Sonnet	$0.020	6.7s	95%	96	96	96	95	95	96%
19	Gemini 2.5 Flash Lite (Reasoning)	$0.0016	11.4s	93%	97	96	95	95	94	95%
16	Claude Haiku 4.5	$0.0066	3.8s	95%	96	96	95	95	95	95%
147	ByteDance Seed 2.0 Mini	$0.0034	3.6m	93%	97	96	95	95	94	95%
129	Grok 4.3 (Reasoning)	$0.019	1.4m	94%	97	97	96	94	93	95%
83	GPT-5.2	$0.017	17.2s	93%	97	96	95	95	93	95%
126	Qwen 3.5 9B	$0.0015	1.7m	91%	97	97	97	96	90	95%
70	Claude Sonnet 4	$0.020	6.8s	94%	96	95	95	95	95	95%
77	GPT-4o, May 13th (temp=0)	$0.024	3.4s	94%	96	96	95	95	94	95%
73	Qwen 3.6 Flash	$0.0098	29.9s	94%	96	96	95	95	95	95%
26	MiniMax M2.7	$0.0033	16.6s	94%	96	96	95	95	94	95%
41	Xiaomi MIMO v2.5 Pro	$0.0045	17.8s	93%	96	96	95	95	94	95%
79	Qwen 3.6 35B	$0.0076	36.4s	93%	96	96	95	95	94	95%
48	Mistral Large	$0.010	6.5s	92%	97	97	94	94	93	95%
131	GPT-5.5 (Reasoning)	$0.046	22.4s	92%	98	95	95	94	93	95%
42	DeepSeek V4 Pro	$0.0041	20.0s	93%	97	95	95	95	94	95%
57	Stealth: Hunter Alpha	$0.0000	37.1s	93%	96	96	95	94	94	95%
80	Nemotron 3 Super	$0.0000	51.6s	92%	96	96	95	95	93	95%
13	Gemini 3 Flash (Preview)	$0.0025	3.3s	92%	98	95	95	94	94	95%
103	GPT-5 Nano	$0.0038	1.1m	93%	96	96	95	94	94	95%
78	o4 Mini	$0.014	20.7s	93%	96	96	96	94	92	95%
11	Gemini 2.5 Flash	$0.0019	2.2s	93%	96	96	95	94	93	95%
14	Inception Mercury 2	$0.0020	3.2s	91%	98	95	94	94	93	95%
84	GPT-OSS 120B	$0.0012	48.7s	92%	97	97	96	92	91	95%
46	Stealth: Healer Alpha	$0.0000	22.8s	91%	98	95	94	94	91	95%
30	GPT-5.4	$0.0080	5.0s	94%	95	95	95	94	93	95%
140	Gemma 4 26B (Reasoning)	$0.0023	2.8m	89%	97	97	96	94	87	95%
71	GPT-5.4 (Reasoning, Low)	$0.016	9.7s	92%	96	96	95	93	92	95%
17	GPT-5.4 Mini (Reasoning, Low)	$0.0039	3.5s	93%	96	95	94	94	93	94%
53	WizardLM 2 8x22b	$0.0025	24.9s	92%	96	96	94	93	93	94%
8	Ministral 3 8B	$0.0006	2.6s	92%	96	96	94	94	93	94%
35	Grok 4.20	$0.0047	4.1s	89%	97	96	93	93	93	94%
33	MiniMax M2.5	$0.0020	13.9s	92%	96	94	94	94	93	94%
10	Mistral Small Creative	$0.0005	3.4s	92%	95	94	94	93	93	94%
113	ByteDance Seed 1.6	$0.0066	1.1m	92%	96	95	95	92	92	94%
39	DeepSeek-V2 Chat	$0.0019	13.3s	90%	95	95	95	94	88	94%
18	GPT-5.4 Mini	$0.0029	2.0s	92%	95	95	94	93	92	94%
93	GPT-4o, May 13th (temp=1)	$0.024	3.2s	91%	95	95	93	93	92	94%
34	Gemma 4 26B	$0.0005	16.6s	92%	95	94	94	93	92	94%
58	DeepSeek V3 (2025-03-24)	$0.0010	25.4s	90%	96	94	93	92	91	94%
37	GPT-4.1	$0.0059	3.8s	90%	95	95	94	93	90	93%
38	GPT-4o, Aug. 6th (temp=0)	$0.0086	3.0s	92%	94	94	94	93	92	93%
51	GPT-4o, Aug. 6th (temp=1)	$0.0086	2.7s	90%	95	95	94	93	89	93%
128	Qwen 3.6 27B	$0.017	1.2m	89%	97	94	94	92	89	93%
12	Ministral 8B	$0.0004	3.0s	92%	94	94	93	93	92	93%
21	Mistral Medium 3.1	$0.0024	4.1s	92%	94	94	93	93	93	93%
32	Grok 4.20 (Beta)	$0.0031	1.6s	89%	96	94	92	92	91	93%
36	Cydonia 24B V4.1	$0.0009	9.0s	89%	96	94	94	93	88	93%
15	Mistral Small 3.2 24B	$0.0005	4.1s	92%	94	93	93	93	92	93%
60	Gemini 2.5 Flash (Reasoning)	$0.0080	11.2s	90%	94	94	93	91	90	92%
27	Gemini 2.5 Flash Lite	$0.0004	1.9s	87%	96	95	94	90	87	92%
74	Z.AI GLM 4.5 Air	$0.0020	27.7s	88%	95	93	93	91	88	92%
45	GPT-5.4 Nano (Reasoning)	$0.0018	9.4s	89%	95	93	92	91	90	92%
85	Qwen 3 32B	$0.0009	36.4s	88%	95	93	92	90	90	92%
55	DeepSeek V3 (2024-12-26)	$0.0016	10.9s	87%	95	94	91	90	88	92%
99	Z.AI GLM 4.7 Flash	$0.0018	51.0s	89%	93	93	93	92	88	92%
65	Mistral Small 4 (Reasoning)	$0.0017	16.6s	87%	94	93	90	90	90	91%
31	Ministral 3 14B	$0.0008	4.5s	89%	93	92	92	91	89	91%
61	Hermes 3 405B	$0.0039	14.8s	89%	94	92	92	89	89	91%
120	Nemotron 3 Nano	$0.0011	1.2m	86%	94	93	90	90	88	91%
49	Claude 3 Haiku	$0.0016	4.6s	87%	94	92	90	89	89	91%
43	GPT-4.1 Mini	$0.0015	5.4s	88%	92	92	90	90	89	91%
40	Qwen 2.5 72B	$0.0008	8.8s	90%	91	91	91	90	90	91%
56	Gemma 3 27B	$0.0005	12.3s	87%	93	92	91	88	88	91%
59	Writer: Palmyra X5	$0.0046	6.7s	88%	92	92	91	90	88	91%
115	ByteDance Seed 1.6 Flash	$0.0012	48.8s	82%	94	91	87	87	85	89%
64	Mistral Small 4	$0.0007	2.8s	83%	94	92	89	86	83	89%
47	GPT-5.4 Nano	$0.0008	2.6s	87%	89	89	89	88	87	89%
72	Llama 3.1 70B	$0.0016	14.2s	86%	91	90	89	86	86	88%
75	Hermes 3 70B	$0.0012	14.0s	85%	90	89	88	87	86	88%
108	Qwen3 235B A22B Instruct 2507	$0.0006	13.8s	71%	93	92	91	90	65	86%
81	GPT-5.4 Nano (Reasoning, Low)	$0.0009	2.7s	80%	90	87	85	83	82	86%
90	Inception Mercury	$0.0006	2.8s	78%	90	90	84	83	80	85%
119	Grok 4.3	$0.0047	3.3s	67%	93	93	92	90	58	85%
68	Ministral 3B	$0.0002	1.4s	84%	86	85	85	85	84	85%
82	GPT-4o Mini (temp=1)	$0.0005	6.4s	82%	87	86	84	84	84	85%
89	GPT-4o Mini (temp=0)	$0.0005	5.2s	80%	87	86	84	82	81	84%
91	Arcee AI: Trinity Mini	$0.0003	4.9s	80%	87	85	83	83	80	84%
86	Ministral 3 3B	$0.0004	1.4s	80%	85	84	84	83	79	83%
123	Cohere Command R+ (Aug. 2024)	$0.013	9.0s	77%	87	85	82	81	79	83%
105	Skyfall 36B V2	$0.0016	6.6s	73%	92	81	81	80	80	83%
97	Gemma 3 4B	$0.0002	6.2s	79%	83	82	81	80	79	81%
100	GPT-4.1 Nano	$0.0003	2.3s	77%	83	80	79	79	79	80%
121	Llama 3.1 8B	$0.0001	9.2s	70%	84	81	77	76	71	78%
139	Llama 3.1 Nemotron 70B	$0.0049	14.5s	55%	86	84	83	83	41	76%
142	Arcee AI: Trinity Large (Preview)	$0.0000	15.7s	26%	94	92	92	91	3	74%
143	Gemma 3 12B	$0.0002	12.7s	28%	88	87	86	85	3	70%
148	Rocinante 12B	$0.0009	9.5s	29%	83	74	68	66	3	59%
150	LFM2 24B	$0.0002	12.2s	36%	36	36	36	36	36	36%
151	Mistral NeMO	$0.0005	519ms	0%	0	0	0	0	0	0%
92.22%

Median	Evaluator	Top 3	Flop 3
84.9%	Accuracy	99Claude Opus 4.6 97Claude Opus 4.6 (Reasoning) 97Claude Opus 4.5	0LFM2 24B 0Mistral NeMO 55Gemma 3 12B
100.0%	Precision	100Claude Opus 4.5 100Qwen 2.5 72B 100DeepSeek V3 (2024-12-26)	0Mistral NeMO 69Rocinante 12B 74Gemma 3 12B
96.6%	Recall	100GPT-5.1 100Claude Opus 4.7 100Gemini 3 Flash (Preview, Reasoning)	0Mistral NeMO 21LFM2 24B 45Rocinante 12B
100.0%	Structural validity	100Qwen 3.6 35B 100Claude Opus 4.6 100GPT-5.4 Mini (Reasoning)	0Mistral NeMO 25LFM2 24B 65Rocinante 12B

Codex Extraction

Medium: The Hollow (Inferred)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

	Score	Consistency	Stability
GPT-5	99%	100%	98%
Claude Opus 4.7 (Reasoning)	99%	99%	98%
Claude Opus 4.6	99%	99%	98%
Claude Sonnet 4.6	98%	100%	98%
GPT-5.1	98%	99%	98%
Claude Opus 4.7	99%	99%	97%
Claude Opus 4.6 (Reasoning)	99%	98%	97%
GPT-5.4 (Reasoning)	98%	99%	97%
Claude Sonnet 4.6 (Reasoning)	97%	99%	97%
Claude Opus 4.5	99%	99%	97%
Grok 4.20 (Reasoning)	98%	99%	97%
Qwen3.7 Max	97%	99%	97%
Gemini 3 Flash (Preview, Reasoning)	98%	98%	96%
Gemini 3.1 Pro (Preview)	98%	98%	96%
Mistral Large 3	97%	100%	96%
Grok 4	98%	98%	96%
Grok 4.20 (Beta, Reasoning)	97%	99%	96%
Z.AI GLM 4.7	97%	99%	96%
Claude Sonnet 4.5	96%	99%	96%
Claude 3.5 Sonnet	96%	99%	96%