Small codex (7 entries), short passage (165 words)

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Qwen3.6 Max Preview	100%
Gemini 3.1 Pro (Preview)	100%
Z.AI GLM 5.1	100%
Claude Sonnet 4.6 (Reasoning)	100%
GPT-5.4 (Reasoning)	100%
Claude Opus 4.7 (Reasoning)	100%
GPT-5.5 (Reasoning)	100%
GPT-5 Mini	100%
GPT-5.1	100%
Claude Opus 4.6	100%
GPT-5	100%
Qwen 3.5 122B	100%
Qwen 3.5 Plus (2026-04-20)	100%
Grok 4.20 (Beta, Reasoning)	100%
GPT-5.4 (Reasoning, Low)	100%
Z.AI GLM 5	100%
Qwen 3.5 27B	100%
ByteDance Seed 1.6	100%
Qwen 3.6 Flash	100%
GPT-5.4 Mini (Reasoning)	100%

	Score	Cost	Time
Stealth: Healer Alpha	100%	$0.0000	18.6s
Gemini 3.1 Flash Lite	100%	$0.0012	2.0s
Gemini 3.1 Flash Lite (Preview)	100%	$0.0012	1.9s
Gemini 3.1 Flash Lite (Reasoning)	90%	$0.0011	6.5s
DeepSeek V3.2	98%	$0.0006	16.2s
Grok 4 Fast	99%	$0.0012	9.5s
Stealth: Hunter Alpha	100%	$0.0000	28.3s
DeepSeek V3 (2024-12-26)	100%	$0.0011	14.8s
Mistral Medium 3.1	97%	$0.0018	5.6s
DeepSeek-V2 Chat	99%	$0.0011	12.7s
Grok 4.1 Fast	99%	$0.0011	13.4s
Gemini 3 Flash (Preview)	100%	$0.0025	3.6s
DeepSeek V4 Flash	96%	$0.0002	6.6s
Inception Mercury	95%	$0.0004	5.9s
GPT-5.4 Nano (Reasoning)	89%	$0.0022	9.4s
Stealth: Aurora Alpha	87%	—	5.8s
Xiaomi MIMO v2.5	99%	$0.0031	11.9s
Gemini 2.5 Flash Lite (Reasoning)	95%	$0.0012	6.9s
Llama 3.1 70B	97%	$0.0009	14.9s
Gemma 4 26B	97%	$0.0003	24.6s

	Score	Cost	Speed	Stability
Gemini 3.1 Flash Lite (Preview)	100%	$0.0012	1.9s	100%
Gemini 3 Flash (Preview)	100%	$0.0025	3.6s	100%
Gemini 3.1 Flash Lite	100%	$0.0012	2.0s	97%
Stealth: Healer Alpha	100%	$0.0000	18.6s	100%
Grok 4 Fast	99%	$0.0012	9.5s	97%
DeepSeek V3 (2024-12-26)	100%	$0.0011	14.8s	98%
Gemini 2.5 Flash	97%	$0.0017	2.1s	97%
Grok 4.1 Fast	99%	$0.0011	13.4s	97%
Inception Mercury 2	97%	$0.0018	2.6s	97%
Gemini 2.5 Flash (Reasoning)	100%	$0.0052	7.8s	100%
Stealth: Hunter Alpha	100%	$0.0000	28.3s	100%
GPT-5.4 Mini (Reasoning, Low)	99%	$0.0041	4.4s	97%
Xiaomi MIMO v2.5	99%	$0.0031	11.9s	97%
DeepSeek-V2 Chat	99%	$0.0011	12.7s	95%
Z.AI GLM 5 Turbo	100%	$0.0047	10.5s	98%
DeepSeek V4 Flash	96%	$0.0002	6.6s	93%
DeepSeek V3.2	98%	$0.0006	16.2s	94%
Z.AI GLM 4.5	98%	$0.0019	15.2s	95%
Llama 3.1 70B	97%	$0.0009	14.9s	95%
Xiaomi MIMO v2.5 Pro	100%	$0.0051	20.3s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
139	Qwen3.6 Max Preview	$0.038	2.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
124	Gemini 3.1 Pro (Preview)	$0.040	30.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
49	Z.AI GLM 5.1	$0.0068	37.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
97	Claude Sonnet 4.6 (Reasoning)	$0.026	16.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
85	GPT-5.4 (Reasoning)	$0.022	19.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
107	Claude Opus 4.7 (Reasoning)	$0.037	6.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
119	GPT-5.5 (Reasoning)	$0.040	14.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
38	GPT-5 Mini	$0.0063	29.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
82	GPT-5.1	$0.019	24.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
81	Claude Opus 4.6	$0.024	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
129	GPT-5	$0.039	53.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
95	Qwen 3.5 122B	$0.018	45.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
90	Qwen 3.5 Plus (2026-04-20)	$0.011	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
68	Grok 4.20 (Beta, Reasoning)	$0.019	10.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
54	GPT-5.4 (Reasoning, Low)	$0.015	9.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
35	Z.AI GLM 5	$0.0062	29.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
108	Qwen 3.5 27B	$0.015	1.5m	100%	100	100	100	100	100	100	100	100	100	100	100%
32	ByteDance Seed 1.6	$0.0036	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Qwen 3.6 Flash	$0.0073	20.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
29	GPT-5.4 Mini (Reasoning)	$0.0091	13.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
36	Gemini 3 Flash (Preview, Reasoning)	$0.0098	15.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
56	GPT-5.2	$0.015	13.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
77	Claude Opus 4.5	$0.023	7.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
76	GPT-5.5	$0.023	6.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
60	Qwen 3.6 35B	$0.0095	36.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Claude Sonnet 4	$0.013	6.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
42	Claude Sonnet 4.5	$0.013	6.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Xiaomi MIMO v2.5 Pro	$0.0051	20.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Stealth: Hunter Alpha	$0.0000	28.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Gemini 2.5 Flash (Reasoning)	$0.0052	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Stealth: Healer Alpha	$0.0000	18.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3.1 Flash Lite (Preview)	$0.0012	1.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Gemini 3 Flash (Preview)	$0.0025	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
58	Nemotron 3 Super	$0.0000	1.2m	100%	100	100	100	100	100	100	100	100	100	100	100%
25	GPT-5.4	$0.010	6.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
51	Claude 3.7 Sonnet	$0.015	8.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
121	Claude Opus 4.6 (Reasoning)	$0.040	19.2s	98%	100	100	100	100	100	100	100	100	100	97	100%
15	Z.AI GLM 5 Turbo	$0.0047	10.5s	98%	100	100	100	100	100	100	100	100	100	97	100%
87	Grok 4.3 (Reasoning)	$0.012	52.1s	98%	100	100	100	100	100	100	100	100	100	97	100%
99	GPT-5.5 (Reasoning, Low)	$0.028	10.1s	98%	100	100	100	100	100	100	100	100	100	97	100%
134	MoonshotAI: Kimi K2.6	$0.026	2.3m	98%	100	100	100	100	100	100	100	100	100	97	100%
138	Qwen 3.5 397B A17B	$0.023	2.8m	98%	100	100	100	100	100	100	100	100	100	97	100%
61	Grok 4.20 (Reasoning)	$0.0094	30.9s	98%	100	100	100	100	100	100	100	100	100	97	100%
44	Z.AI GLM 4.7	$0.0061	29.6s	98%	100	100	100	100	100	100	100	100	100	97	100%
102	ByteDance Seed 2.0 Mini	$0.0021	2.0m	98%	100	100	100	100	100	100	100	100	100	97	100%
47	Qwen 3.5 Flash	$0.0027	45.4s	98%	100	100	100	100	100	100	100	100	100	97	100%
6	DeepSeek V3 (2024-12-26)	$0.0011	14.8s	98%	100	100	100	100	100	100	100	100	100	97	100%
21	GPT-4.1	$0.0065	7.7s	97%	100	100	100	100	100	100	100	100	100	95	100%
3	Gemini 3.1 Flash Lite	$0.0012	2.0s	97%	100	100	100	100	100	100	100	100	100	95	100%
140	Gemma 4 31B (Reasoning)	$0.0011	4.2m	97%	100	100	100	100	100	100	100	100	97	97	99%
8	Grok 4.1 Fast	$0.0011	13.4s	97%	100	100	100	100	100	100	100	100	97	97	99%
5	Grok 4 Fast	$0.0012	9.5s	97%	100	100	100	100	100	100	100	100	97	97	99%
13	Xiaomi MIMO v2.5	$0.0031	11.9s	97%	100	100	100	100	100	100	100	100	97	97	99%
23	GPT-4o, Aug. 6th (temp=0)	$0.0080	3.4s	97%	100	100	100	100	100	100	100	100	97	97	99%
34	DeepSeek V3 (2025-03-24)	$0.0007	36.3s	96%	100	100	100	100	100	100	100	100	100	93	99%
78	MoonshotAI: Kimi K2.5	$0.010	46.2s	97%	100	100	100	100	100	100	100	100	97	95	99%
112	Gemini 2.5 Pro	$0.031	21.3s	97%	100	100	100	100	100	100	100	100	97	95	99%
75	Gemma 4 26B (Reasoning)	$0.0018	1.2m	97%	100	100	100	100	100	100	100	97	97	97	99%
133	DeepSeek V4 Pro (Reasoning)	$0.0090	3.2m	97%	100	100	100	100	100	100	100	97	97	97	99%
131	Gemini 3 Pro (Preview)	$0.042	49.1s	97%	100	100	100	100	100	100	100	97	97	97	99%
93	Qwen 3.5 35B	$0.014	47.7s	97%	100	100	100	100	100	100	100	97	97	97	99%
103	Qwen 3.5 9B	$0.0015	1.9m	97%	100	100	100	100	100	100	100	97	97	97	99%
12	GPT-5.4 Mini (Reasoning, Low)	$0.0041	4.4s	97%	100	100	100	100	100	100	100	97	97	97	99%
73	Z.AI GLM 4.6	$0.0037	1.0m	96%	100	100	100	100	100	100	97	97	97	95	99%
14	DeepSeek-V2 Chat	$0.0011	12.7s	95%	100	100	100	100	100	100	100	97	97	92	99%
120	o4 Mini High	$0.028	46.6s	95%	100	100	100	100	100	97	97	97	97	97	98%
18	Z.AI GLM 4.5	$0.0019	15.2s	95%	100	100	100	100	100	97	97	97	97	97	98%
70	GPT-4o, May 13th (temp=0)	$0.016	3.7s	93%	100	100	100	100	100	100	100	100	92	92	98%
105	Claude 3.5 Sonnet	$0.027	10.5s	93%	100	100	100	100	100	100	100	100	92	92	98%
17	DeepSeek V3.2	$0.0006	16.2s	94%	100	100	100	100	100	100	100	97	93	92	98%
83	o4 Mini	$0.015	20.9s	94%	100	100	100	97	97	97	97	97	97	97	98%
37	DeepSeek V4 Flash (Reasoning)	$0.0006	27.3s	93%	100	100	100	100	100	95	95	95	95	95	98%
127	Grok 4	$0.032	47.5s	94%	100	100	97	97	97	97	97	97	97	97	97%
28	Gemma 4 31B	$0.0004	23.8s	95%	100	97	97	97	97	97	97	97	97	97	97%
63	GPT-OSS 120B	$0.0012	45.6s	93%	100	100	97	97	97	97	97	97	97	93	97%
19	Llama 3.1 70B	$0.0009	14.9s	95%	100	97	97	97	97	97	97	97	97	97	97%
26	Llama 3.1 Nemotron 70B	$0.0027	15.3s	95%	100	97	97	97	97	97	97	97	97	97	97%
45	Qwen 3.5 Plus (2026-02-15)	$0.0022	21.4s	91%	100	100	100	95	95	95	95	95	95	95	97%
84	ByteDance Seed 2.0 Lite	$0.0050	52.7s	93%	100	100	97	97	97	97	97	97	93	93	97%
24	Gemma 4 26B	$0.0003	24.6s	97%	97	97	97	97	97	97	97	97	97	97	97%
9	Inception Mercury 2	$0.0018	2.6s	97%	97	97	97	97	97	97	97	97	97	97	97%
7	Gemini 2.5 Flash	$0.0017	2.1s	97%	97	97	97	97	97	97	97	97	97	97	97%
22	Mistral Medium 3.1	$0.0018	5.6s	92%	100	100	100	100	100	100	92	92	92	92	97%
62	Z.AI GLM 4.5 Air	$0.0018	32.1s	90%	100	100	97	97	97	97	97	97	93	88	96%
57	GPT-4o, Aug. 6th (temp=1)	$0.0081	3.7s	89%	100	100	100	100	97	97	92	92	92	92	96%
50	DeepSeek V3.1	$0.0009	23.3s	89%	100	100	100	100	97	97	92	92	92	92	96%
43	Grok 4.20 (Beta)	$0.0040	1.9s	88%	100	100	100	100	100	92	92	92	92	92	96%
55	Hermes 3 405B	$0.0022	21.4s	89%	100	100	100	97	97	97	97	93	92	87	96%
40	Grok 4.20	$0.0026	4.0s	88%	100	100	100	100	100	92	92	92	92	92	96%
46	Mistral Large 2	$0.0065	7.1s	93%	97	97	97	97	97	97	97	97	92	92	96%
16	DeepSeek V4 Flash	$0.0002	6.6s	93%	97	97	97	97	97	97	97	97	92	92	96%
110	Claude Opus 4.7	$0.028	4.9s	89%	100	100	97	97	97	97	97	97	88	88	96%
53	Mistral Large	$0.0066	7.4s	91%	100	97	97	97	97	97	97	92	92	92	96%
117	Nemotron 3 Nano	$0.0015	2.0m	91%	100	97	97	97	97	93	93	93	93	93	95%
27	Inception Mercury	$0.0004	5.9s	89%	100	97	97	97	97	97	97	97	88	88	95%
33	Gemini 2.5 Flash Lite (Reasoning)	$0.0012	6.9s	89%	97	97	97	97	97	97	97	97	93	84	95%
125	Qwen 3.6 27B	$0.016	1.1m	80%	100	100	100	100	97	97	97	97	93	70	95%
144	Claude Opus 4	$0.067	12.3s	84%	100	100	100	100	92	92	92	92	92	92	95%
31	Gemma 3 12B	$0.0001	14.1s	91%	97	97	97	97	97	97	93	93	93	88	95%
41	Mistral Large 3	$0.0017	8.7s	89%	97	97	97	97	97	92	92	92	92	92	94%
67	MiniMax M2.7	$0.0015	20.7s	85%	100	100	97	97	97	92	92	88	88	88	94%
65	MiniMax M2.5	$0.0014	17.0s	85%	100	100	97	97	97	92	90	88	88	88	94%
52	ByteDance Seed 1.6 Flash	$0.0006	9.0s	86%	97	97	97	97	93	93	93	93	88	84	93%
100	GPT-4o, May 13th (temp=1)	$0.015	3.7s	81%	100	100	100	97	92	92	92	92	84	84	93%
64	Qwen 3 32B	$0.0006	21.5s	87%	97	97	97	97	97	93	93	90	88	83	93%
48	GPT-4.1 Mini	$0.0014	5.1s	87%	97	97	97	97	93	93	92	88	88	88	93%
115	GPT-5 Nano	$0.0036	1.6m	89%	97	97	93	93	93	93	90	90	90	90	93%
69	Grok 4.3	$0.0033	4.9s	83%	100	97	97	95	92	91	88	88	87	84	92%
88	Claude Sonnet 4.6	$0.014	7.3s	92%	92	92	92	92	92	92	92	92	92	92	92%
79	Qwen 2.5 72B	$0.0005	13.6s	77%	97	97	97	93	93	93	93	93	90	66	91%
66	Mistral Small 3.2 24B	$0.0003	7.0s	81%	97	97	97	93	88	88	88	88	88	88	91%
72	Mistral Small 4 (Reasoning)	$0.0015	13.2s	82%	97	97	93	93	93	88	88	88	84	83	90%
104	Z.AI GLM 4.7 Flash	$0.0011	46.8s	78%	100	97	97	93	88	88	88	87	84	80	90%
123	Gemini 3.1 Flash Lite (Reasoning)	$0.0011	6.5s	42%	100	100	100	100	100	100	100	100	95	3	90%
135	Aion 2.0	$0.0048	49.3s	40%	100	100	100	100	100	100	100	100	97	0	90%
128	GPT-5.4 Nano (Reasoning)	$0.0022	9.4s	40%	100	100	100	100	100	100	100	97	97	0	89%
74	Qwen3 235B A22B Instruct 2507	$0.0005	17.6s	84%	92	92	92	92	88	88	88	88	87	84	89%
98	Cohere Command R+ (Aug. 2024)	$0.0082	9.5s	80%	97	97	93	90	90	90	87	84	80	79	89%
86	Writer: Palmyra X5	$0.0041	10.6s	82%	92	92	92	92	88	88	84	84	84	83	88%
59	Gemini 2.5 Flash Lite	$0.0004	1.6s	86%	88	88	88	88	88	88	88	88	88	83	88%
89	GPT-5.4 Mini	$0.0028	2.0s	76%	97	93	92	88	88	88	88	84	84	70	87%
130	DeepSeek V4 Pro	$0.0025	18.3s	42%	100	100	97	97	97	95	95	95	92	3	87%
137	Stealth: Aurora Alpha	—	5.8s	41%	97	97	97	97	97	97	97	97	97	0	87%
96	Arcee AI: Trinity Large (Preview)	$0.0000	16.2s	76%	92	92	91	87	83	83	83	83	80	79	85%
92	Claude Haiku 4.5	$0.0046	4.5s	81%	88	88	88	88	88	88	88	79	79	79	85%
71	GPT-4o Mini (temp=0)	$0.0004	7.3s	84%	87	87	87	87	87	87	83	83	83	83	85%
91	Gemma 3 27B	$0.0003	11.6s	79%	90	88	88	84	84	84	84	80	80	80	84%
80	Mistral Small Creative	$0.0003	3.3s	80%	87	87	83	83	83	83	83	83	83	79	84%
94	GPT-4o Mini (temp=1)	$0.0004	6.7s	76%	87	87	87	83	83	83	83	83	83	72	83%
106	Ministral 3 3B	$0.0002	2.0s	69%	84	84	84	84	80	80	79	76	70	62	78%
118	Arcee AI: Trinity Mini	$0.0003	33.1s	69%	87	87	83	80	80	76	76	72	72	72	78%
114	Ministral 3 14B	$0.0004	5.1s	63%	92	88	84	79	79	79	79	75	71	52	78%
101	Mistral NeMO	$0.0003	2.9s	72%	84	80	80	76	76	76	76	76	76	76	78%
111	Ministral 3 8B	$0.0003	3.6s	67%	84	83	83	79	76	76	75	75	73	63	77%
113	Ministral 8B	$0.0002	3.3s	64%	88	88	84	79	79	76	75	75	63	60	77%
116	Llama 3.1 8B	$0.0001	7.0s	63%	88	87	84	83	80	79	70	66	62	57	76%
132	Hermes 3 70B	$0.0006	15.0s	41%	93	88	87	87	87	84	83	76	70	0	76%
109	Ministral 3B	$0.0001	1.9s	68%	84	80	79	76	75	75	75	71	70	66	75%
126	Mistral Small 4	$0.0004	3.4s	55%	84	83	79	79	75	75	75	71	62	37	72%
122	Claude 3 Haiku	$0.0009	3.0s	62%	76	76	76	72	72	72	72	70	60	55	70%
141	GPT-5.4 Nano	$0.0008	2.6s	25%	97	93	90	88	88	80	79	67	0	0	68%
136	Rocinante 12B	$0.0004	4.9s	44%	80	80	72	61	61	55	52	52	52	42	60%
142	GPT-4.1 Nano	$0.0002	2.0s	35%	73	70	65	61	53	53	53	37	28	17	51%
145	GPT-5.4 Nano (Reasoning, Low)	$0.0010	4.4s	2%	100	97	93	93	88	0	0	0	0	0	47%
143	Gemma 3 4B	$0.0001	3.6s	38%	48	44	40	40	40	40	39	39	39	38	41%
146	WizardLM 2 8x22b	$0.0019	16.8s	2%	70	53	22	17	3	3	3	3	0	0	18%
147	LFM2 24B	$0.0001	1.5s	0%	0	0	0	0	0	0	0	0	0	0	0%
92.28%

Median	Evaluator	Top 3	Flop 3
93.0%	Accuracy (recall)	100Qwen3.6 Max Preview 100Qwen 3.5 Plus (2026-04-20) 100Claude Sonnet 4.5	0LFM2 24B 0Gemma 3 4B 2WizardLM 2 8x22b
100.0%	Precision	100Llama 3.1 Nemotron 70B 100GPT-5.4 Mini (Reasoning, Low) 100Grok 4.1 Fast	0LFM2 24B 22WizardLM 2 8x22b 22Gemma 3 4B
100.0%	Structural validity	100Arcee AI: Trinity Mini 100GPT-5.4 (Reasoning, Low) 100Claude Opus 4.7	0LFM2 24B 29WizardLM 2 8x22b 50GPT-5.4 Nano (Reasoning, Low)

Codex Violation Detection

Small codex (7 entries), short passage (165 words)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)