Small codex (7 entries), long passage (734 words)

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Grok 4.20 (Beta, Reasoning)	100%
Claude Opus 4.5	100%
Gemini 3.1 Pro (Preview)	99%
Claude Opus 4.6	99%
Gemini 2.5 Pro	99%
Qwen 3.5 27B	98%
Claude Opus 4.6 (Reasoning)	98%
Qwen 3.5 397B A17B	98%
Qwen 3.5 Flash	98%
Gemini 3 Flash (Preview, Reasoning)	97%
Qwen 3.5 122B	97%
Aion 2.0	96%
GPT-5.1	96%
GPT-5 Mini	96%
GPT-5.2	96%
Gemini 3 Pro (Preview)	95%
Z.AI GLM 5	95%
Stealth: Hunter Alpha	95%
Qwen 3.5 Plus (2026-02-15)	95%
o4 Mini	95%

	Score	Cost	Time
Claude Haiku 4.5	93%	$0.0051	4.6s
GPT-4.1 Mini	87%	$0.0011	4.4s
Grok 4 Fast	93%	$0.0019	16.5s
Stealth: Healer Alpha	90%	$0.0000	25.0s
Qwen 3.5 Plus (2026-02-15)	95%	$0.0025	21.6s
MiniMax M2.7	91%	$0.0021	28.0s
DeepSeek-V2 Chat	85%	$0.0013	9.8s
Gemini 2.5 Flash	87%	$0.0020	2.4s
Gemini 3 Flash (Preview)	88%	$0.0022	3.6s
Grok 4.1 Fast	93%	$0.0025	31.2s
Mistral Medium 3.1	79%	$0.0021	5.0s
Z.AI GLM 5 Turbo	94%	$0.0077	19.5s
Inception Mercury 2	88%	$0.0032	4.9s
Gemini 2.5 Flash (Reasoning)	95%	$0.0094	16.9s
Gemini 3 Flash (Preview, Reasoning)	97%	$0.012	20.1s
Stealth: Hunter Alpha	95%	$0.0000	1.2m
Z.AI GLM 4.5	85%	$0.0018	13.7s
Ministral 3 14B	80%	$0.0006	4.7s
Gemini 2.5 Flash Lite (Reasoning)	86%	$0.0021	16.2s
MiniMax M2.5	88%	$0.0018	23.0s

	Score	Cost	Speed	Stability
Claude Haiku 4.5	93%	$0.0051	4.6s	93%
Gemini 3 Flash (Preview, Reasoning)	97%	$0.012	20.1s	92%
Qwen 3.5 Plus (2026-02-15)	95%	$0.0025	21.6s	84%
Grok 4 Fast	93%	$0.0019	16.5s	83%
Inception Mercury 2	88%	$0.0032	4.9s	86%
Gemini 3 Flash (Preview)	88%	$0.0022	3.6s	83%
Claude Opus 4.5	100%	$0.030	9.1s	100%
Gemini 2.5 Flash (Reasoning)	95%	$0.0094	16.9s	87%
GPT-4.1 Mini	87%	$0.0011	4.4s	82%
Z.AI GLM 5 Turbo	94%	$0.0077	19.5s	85%
MiniMax M2.7	91%	$0.0021	28.0s	85%
Gemini 3.1 Flash Lite (Preview)	84%	$0.0014	1.9s	81%
Grok 4.20 (Beta, Reasoning)	100%	$0.032	17.9s	100%
Grok 4.1 Fast	93%	$0.0025	31.2s	82%
MiniMax M2.5	88%	$0.0018	23.0s	83%
Stealth: Healer Alpha	90%	$0.0000	25.0s	79%
Gemini 2.5 Flash	87%	$0.0020	2.4s	76%
Qwen 3.5 Flash	98%	$0.0044	1.3m	93%
Gemini 2.5 Flash Lite (Reasoning)	86%	$0.0021	16.2s	81%
Claude Opus 4.6	99%	$0.030	10.2s	95%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
13	Grok 4.20 (Beta, Reasoning)	$0.032	17.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Claude Opus 4.5	$0.030	9.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
109	Gemini 3.1 Pro (Preview)	$0.099	1.4m	97%	100	100	100	100	100	100	100	100	97	95	99%
20	Claude Opus 4.6	$0.030	10.2s	95%	100	100	100	100	100	100	100	100	97	92	99%
52	Gemini 2.5 Pro	$0.039	26.5s	96%	100	100	100	100	100	100	100	97	97	93	99%
69	Qwen 3.5 27B	$0.020	1.9m	95%	100	100	100	100	100	97	97	97	97	97	98%
95	Claude Opus 4.6 (Reasoning)	$0.074	37.2s	93%	100	100	100	100	100	100	100	100	92	92	98%
101	Qwen 3.5 397B A17B	$0.030	3.8m	95%	100	100	100	100	100	100	97	97	97	92	98%
18	Qwen 3.5 Flash	$0.0044	1.3m	93%	100	100	100	100	97	97	97	97	97	93	98%
2	Gemini 3 Flash (Preview, Reasoning)	$0.012	20.1s	92%	100	100	100	100	100	97	97	95	92	92	97%
84	Qwen 3.5 122B	$0.033	1.5m	89%	100	100	100	100	100	97	97	97	93	84	97%
53	Aion 2.0	$0.0089	1.6m	90%	100	100	100	97	97	97	97	97	92	88	96%
86	GPT-5.1	$0.044	1.0m	87%	100	100	100	100	97	97	97	97	92	83	96%
23	GPT-5 Mini	$0.011	52.1s	91%	100	100	95	95	95	95	95	95	95	92	96%
28	GPT-5.2	$0.024	24.2s	93%	100	95	95	95	95	95	95	95	95	95	96%
71	Gemini 3 Pro (Preview)	$0.045	29.8s	91%	100	97	97	97	97	97	97	93	93	88	95%
56	Z.AI GLM 5	$0.014	1.2m	87%	100	100	97	97	97	97	97	97	88	83	95%
25	Stealth: Hunter Alpha	$0.0000	1.2m	86%	100	100	100	100	97	92	92	92	91	88	95%
3	Qwen 3.5 Plus (2026-02-15)	$0.0025	21.6s	84%	100	100	100	100	92	92	92	92	92	92	95%
34	o4 Mini	$0.023	33.7s	92%	97	97	97	97	97	97	93	92	92	92	95%
98	Claude Sonnet 4.6 (Reasoning)	$0.067	43.4s	85%	100	100	100	100	100	100	97	83	83	83	95%
8	Gemini 2.5 Flash (Reasoning)	$0.0094	16.9s	87%	100	100	100	97	97	95	91	91	87	87	95%
81	ByteDance Seed 2.0 Mini	$0.0028	2.7m	88%	100	97	97	97	97	97	93	93	88	84	94%
24	ByteDance Seed 1.6	$0.0058	57.6s	89%	97	97	97	97	97	97	97	88	88	88	94%
108	GPT-5	$0.072	1.8m	84%	100	100	97	97	95	95	92	92	91	79	94%
10	Z.AI GLM 5 Turbo	$0.0077	19.5s	85%	100	100	100	97	97	97	88	88	88	83	94%
76	o4 Mini High	$0.034	52.9s	87%	97	97	97	97	95	93	93	92	92	83	94%
1	Claude Haiku 4.5	$0.0051	4.6s	93%	93	93	93	93	93	93	93	93	93	93	93%
66	Qwen 3.5 35B	$0.019	59.9s	85%	100	100	97	97	97	97	88	88	84	83	93%
21	Claude Sonnet 4.6	$0.018	7.7s	87%	100	97	92	92	92	92	92	92	92	92	93%
4	Grok 4 Fast	$0.0019	16.5s	83%	100	100	95	95	93	92	92	92	87	83	93%
14	Grok 4.1 Fast	$0.0025	31.2s	82%	100	100	97	95	95	92	92	88	87	79	93%
46	ByteDance Seed 2.0 Lite	$0.0063	1.1m	85%	97	97	97	93	93	92	88	88	88	84	92%
36	GPT-5.4	$0.010	7.1s	74%	100	100	100	100	92	83	83	83	83	83	91%
11	MiniMax M2.7	$0.0021	28.0s	85%	97	93	93	93	93	93	88	88	88	80	91%
74	Z.AI GLM 4.7	$0.0096	1.6m	81%	100	97	93	93	92	88	88	88	83	83	91%
94	MoonshotAI: Kimi K2.5	$0.019	2.4m	81%	97	97	97	97	97	92	87	84	79	79	90%
16	Stealth: Healer Alpha	$0.0000	25.0s	79%	97	97	97	97	92	92	88	84	84	76	90%
110	Grok 4	$0.065	1.7m	78%	100	97	97	97	95	92	92	79	79	75	90%
39	Claude Sonnet 4.5	$0.017	7.4s	80%	100	97	92	92	92	88	88	83	83	83	90%
82	Z.AI GLM 4.6	$0.0059	1.9m	80%	97	97	92	92	92	88	88	88	87	76	90%
42	Claude 3.7 Sonnet	$0.017	9.8s	80%	97	97	93	88	88	88	88	88	84	84	89%
15	MiniMax M2.5	$0.0018	23.0s	83%	93	93	90	90	88	88	87	87	84	84	88%
63	Claude 3.5 Sonnet	$0.030	10.0s	85%	93	90	90	90	90	90	88	84	84	84	88%
6	Gemini 3 Flash (Preview)	$0.0022	3.6s	83%	97	88	88	88	88	88	88	88	88	83	88%
5	Inception Mercury 2	$0.0032	4.9s	86%	88	88	88	88	88	88	88	88	88	83	88%
57	Claude Sonnet 4	$0.016	7.3s	74%	100	92	92	92	83	83	83	83	83	83	87%
17	Gemini 2.5 Flash	$0.0020	2.4s	76%	97	97	93	88	88	87	83	83	83	75	87%
9	GPT-4.1 Mini	$0.0011	4.4s	82%	90	90	90	90	90	90	84	84	84	75	87%
37	Mistral Large	$0.0082	7.4s	76%	97	93	88	88	88	84	84	79	79	79	86%
19	Gemini 2.5 Flash Lite (Reasoning)	$0.0021	16.2s	81%	90	90	88	88	88	87	84	84	80	80	86%
29	ByteDance Seed 1.6 Flash	$0.0009	13.8s	75%	97	93	90	88	84	84	83	80	80	79	86%
106	Claude Opus 4	$0.079	13.6s	77%	97	88	88	88	88	84	84	79	79	79	85%
68	Z.AI GLM 4.7 Flash	$0.0020	1.2m	77%	90	90	90	90	87	84	84	80	80	76	85%
22	Z.AI GLM 4.5	$0.0018	13.7s	80%	90	88	88	88	88	88	84	83	79	75	85%
58	GPT-4o, May 13th (temp=0)	$0.018	3.3s	77%	90	90	90	84	84	84	84	84	84	75	85%
26	DeepSeek-V2 Chat	$0.0013	9.8s	76%	93	90	90	90	90	87	80	76	76	75	85%
55	Stealth: Aurora Alpha	—	5.0s	75%	92	88	88	88	88	88	88	87	70	70	85%
77	GPT-5.4 Mini (Reasoning)	$0.022	31.3s	77%	90	88	88	88	84	83	83	83	83	75	85%
27	GPT-4o, Aug. 6th (temp=0)	$0.0081	2.7s	81%	87	87	84	84	84	84	84	84	84	80	84%
105	Qwen 3.5 9B	$0.0023	3.0m	58%	97	93	93	93	90	88	88	84	79	33	84%
41	GPT-5.4 Mini (Reasoning, Low)	$0.0054	10.2s	75%	93	88	88	88	84	84	83	80	75	75	84%
12	Gemini 3.1 Flash Lite (Preview)	$0.0014	1.9s	81%	88	83	83	83	83	83	83	83	83	83	84%
72	GPT-5.4 (Reasoning, Low)	$0.020	14.4s	71%	100	88	83	83	83	79	79	79	79	79	83%
38	Writer: Palmyra X5	$0.0039	8.6s	75%	90	90	87	87	87	80	80	80	76	76	83%
48	Mistral Large 2	$0.0080	7.0s	75%	93	88	84	84	84	84	84	79	75	75	83%
44	GPT-4o, Aug. 6th (temp=1)	$0.0078	3.0s	75%	90	90	87	84	84	80	80	80	80	75	83%
62	GPT-4o, May 13th (temp=1)	$0.017	3.2s	76%	90	87	87	87	87	84	83	80	75	70	83%
97	GPT-5.4 (Reasoning)	$0.042	42.1s	71%	92	92	87	83	79	79	79	79	79	79	83%
33	Mistral Large 3	$0.0021	7.9s	75%	92	88	84	84	84	79	79	79	79	79	83%
35	GPT-4.1	$0.0062	7.7s	78%	88	84	84	84	84	84	84	80	79	75	83%
32	Qwen 2.5 72B	$0.0006	10.1s	75%	90	87	87	87	83	83	80	80	76	72	82%
51	Mistral Small 4 (Reasoning)	$0.0024	21.0s	74%	93	88	84	84	84	83	80	80	75	71	82%
30	Claude 3.5 Haiku	$0.0036	6.1s	77%	87	87	87	83	83	83	80	80	76	76	82%
31	Mistral Small 3.2 24B	$0.0004	7.7s	74%	88	88	88	84	83	80	80	79	75	75	82%
43	Llama 3.1 Nemotron 70B	$0.0037	18.2s	77%	88	84	84	84	84	79	79	79	79	79	82%
104	Nemotron 3 Super	$0.0000	3.3m	66%	100	95	91	83	83	83	76	72	68	68	82%
60	Qwen 3 32B	$0.0008	24.8s	69%	93	93	84	84	84	80	80	76	76	66	82%
114	Nemotron 3 Nano	$0.0047	5.0m	74%	87	87	87	84	80	80	80	80	76	76	82%
40	Qwen3 235B A22B Instruct 2507	$0.0005	16.6s	75%	87	87	84	84	84	83	80	76	75	70	81%
54	DeepSeek V3 (2025-03-24)	$0.0009	20.7s	71%	90	90	87	80	80	80	76	76	76	75	81%
45	DeepSeek V3 (2024-12-26)	$0.0012	12.3s	73%	90	87	84	80	80	80	80	76	76	76	81%
47	Llama 3.1 70B	$0.0013	21.0s	76%	84	83	80	80	80	80	80	80	77	75	80%
93	GPT-5 Nano	$0.0051	2.0m	71%	88	87	84	84	80	80	79	76	71	71	80%
50	Grok 4.20 (Beta)	$0.0048	2.3s	73%	87	87	83	83	83	83	80	72	72	68	80%
67	Ministral 3 14B	$0.0006	4.7s	60%	93	93	88	88	84	84	84	79	52	52	80%
49	Gemini 2.5 Flash Lite	$0.0003	1.7s	70%	88	87	80	80	80	80	79	75	70	66	79%
85	Mistral Medium 3.1	$0.0021	5.0s	44%	88	88	88	88	88	88	88	83	83	3	79%
91	GPT-5.4 Nano (Reasoning)	$0.0036	22.3s	41%	93	92	88	88	88	84	84	83	79	0	78%
61	Gemma 3 27B	$0.0003	15.1s	69%	84	83	83	80	80	75	75	71	71	71	77%
59	DeepSeek V3.2	$0.0007	9.8s	69%	84	84	84	80	80	80	76	70	66	66	77%
64	Inception Mercury	$0.0004	8.5s	66%	84	84	80	80	79	77	75	75	75	55	76%
65	Mistral Small Creative	$0.0005	3.8s	65%	88	83	83	76	75	75	72	71	68	65	76%
78	Hermes 3 405B	$0.0029	19.0s	64%	84	80	76	75	71	71	71	70	70	70	74%
79	GPT-5.4 Mini	$0.0030	2.2s	59%	84	79	76	76	75	75	75	75	68	43	73%
75	Arcee AI: Trinity Large (Preview)	$0.0000	15.1s	62%	83	75	75	75	75	72	71	71	63	54	71%
70	Claude 3 Haiku	$0.0009	2.3s	62%	77	77	73	73	73	73	73	73	61	52	71%
96	DeepSeek V3.1	$0.0007	23.3s	41%	90	80	80	80	76	75	70	70	70	3	70%
73	GPT-4o Mini (temp=0)	$0.0004	5.4s	65%	72	67	67	67	67	67	67	67	67	67	67%
80	Ministral 3 3B	$0.0003	3.0s	61%	75	71	71	67	67	67	66	62	62	61	67%
83	GPT-4o Mini (temp=1)	$0.0004	6.7s	59%	72	72	72	72	72	67	67	60	53	53	66%
87	Arcee AI: Trinity Mini	$0.0004	8.3s	53%	77	76	73	72	66	61	60	60	60	53	66%
88	Ministral 3 8B	$0.0005	5.6s	53%	72	71	68	68	65	62	59	57	53	47	62%
89	Gemma 3 12B	$0.0003	11.0s	53%	75	66	66	66	61	61	61	57	57	53	62%
90	Ministral 3B	$0.0001	3.1s	51%	80	65	62	62	62	62	60	53	52	48	61%
100	Hermes 3 70B	$0.0008	18.6s	38%	76	73	72	70	67	66	66	62	47	0	60%
92	Mistral Small 4	$0.0006	4.1s	50%	70	66	66	62	60	58	55	55	53	43	59%
99	Ministral 8B	$0.0004	6.4s	41%	79	68	66	66	62	60	59	57	40	17	57%
103	Llama 3.1 8B	$0.0001	12.1s	37%	66	66	62	62	60	57	53	53	33	3	51%
102	Gemma 3 4B	$0.0002	10.3s	42%	61	59	54	49	49	47	44	43	42	42	49%
107	Mistral NeMO	$0.0005	5.6s	29%	75	65	65	63	58	58	44	44	0	0	47%
112	Cohere Command R+ (Aug. 2024)	$0.0083	5.7s	19%	80	76	75	72	70	66	0	0	0	0	44%
111	GPT-5.4 Nano	$0.0008	2.8s	26%	63	57	54	53	53	52	33	0	0	0	37%
113	Rocinante 12B	$0.0007	6.9s	19%	70	65	44	38	33	33	33	30	0	0	35%
116	GPT-5.4 Nano (Reasoning, Low)	$0.0013	8.0s	0%	88	88	84	79	0	0	0	0	0	0	34%
115	WizardLM 2 8x22b	$0.0023	12.7s	17%	33	33	33	33	23	21	17	17	3	3	22%
117	GPT-4.1 Nano	$0.0003	4.0s	15%	33	33	25	24	23	20	0	0	0	0	16%
118	LFM2 24B	$0.0001	1.3s	0%	0	0	0	0	0	0	0	0	0	0	0%
80.66%

Median	Evaluator	Top 3	Flop 3
67.3%	Accuracy (recall)	100Grok 4.20 (Beta, Reasoning) 100Claude Opus 4.5 99Gemini 3.1 Pro (Preview)	0LFM2 24B 0GPT-4.1 Nano 0WizardLM 2 8x22b
88.6%	Precision	100Gemini 2.5 Pro 100Claude Haiku 4.5 100Qwen 3.5 Flash	0LFM2 24B 2WizardLM 2 8x22b 8GPT-4.1 Nano
100.0%	Structural validity	100Grok 4.20 (Beta, Reasoning) 100Z.AI GLM 5 Turbo 100Claude Sonnet 4	0LFM2 24B 40GPT-4.1 Nano 40GPT-5.4 Nano (Reasoning, Low)

Codex Violation Detection

Small codex (7 entries), long passage (734 words)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)