Large codex (40 entries), short passage (165 words)

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.5	100%
Claude Sonnet 4.5	100%
Gemini 2.5 Pro	100%
Gemini 3.1 Pro (Preview)	99%
Grok 4	99%
Claude Opus 4.6	98%
Claude Sonnet 4.6	98%
GPT-5.4	98%
Z.AI GLM 5 Turbo	98%
MoonshotAI: Kimi K2.5	98%
Grok 4.20 (Beta, Reasoning)	97%
Claude Sonnet 4.6 (Reasoning)	97%
ByteDance Seed 1.6	97%
Claude Sonnet 4	97%
Grok 4 Fast	97%
Claude Opus 4.6 (Reasoning)	97%
Qwen 3.5 35B	96%
GPT-5.4 Mini (Reasoning)	96%
Grok 4.1 Fast	96%
Qwen 3.5 Flash	96%

	Score	Cost	Time
Grok 4 Fast	97%	$0.0024	15.1s
Stealth: Healer Alpha	95%	$0.0000	25.4s
Z.AI GLM 5 Turbo	98%	$0.0097	19.4s
Grok 4.1 Fast	96%	$0.0032	24.9s
Grok 4.20 (Beta)	95%	$0.0090	3.5s
Gemini 3 Flash (Preview)	93%	$0.0039	6.2s
Inception Mercury 2	92%	$0.0040	5.5s
Stealth: Hunter Alpha	95%	$0.0000	44.1s
Gemini 2.5 Flash	90%	$0.0038	4.0s
GPT-5.4	98%	$0.018	11.4s
GPT-5.4 Mini (Reasoning, Low)	92%	$0.0070	6.8s
GPT-4.1	94%	$0.011	15.1s
MiniMax M2.5	91%	$0.0023	30.1s
Gemini 2.5 Flash (Reasoning)	95%	$0.0098	14.4s
Gemini 3.1 Flash Lite (Preview)	90%	$0.0031	3.0s
Gemini 3 Flash (Preview, Reasoning)	96%	$0.012	19.4s
Gemini 2.5 Flash Lite (Reasoning)	90%	$0.0030	31.0s
GPT-5.4 Nano (Reasoning)	90%	$0.0045	18.9s
Mistral Small 4 (Reasoning)	87%	$0.0025	16.2s
Qwen 3.5 Flash	96%	$0.0045	1.3m

	Score	Cost	Speed	Stability
Grok 4 Fast	97%	$0.0024	15.1s	94%
Z.AI GLM 5 Turbo	98%	$0.0097	19.4s	94%
Grok 4.1 Fast	96%	$0.0032	24.9s	93%
Grok 4.20 (Beta)	95%	$0.0090	3.5s	90%
Stealth: Healer Alpha	95%	$0.0000	25.4s	91%
GPT-5.4	98%	$0.018	11.4s	93%
Gemini 3 Flash (Preview)	93%	$0.0039	6.2s	88%
Claude Sonnet 4.5	100%	$0.036	11.3s	100%
Inception Mercury 2	92%	$0.0040	5.5s	88%
Gemini 2.5 Flash (Reasoning)	95%	$0.0098	14.4s	90%
Gemini 3 Flash (Preview, Reasoning)	96%	$0.012	19.4s	91%
GPT-5.4 Mini (Reasoning, Low)	92%	$0.0070	6.8s	86%
Gemini 3.1 Flash Lite (Preview)	90%	$0.0031	3.0s	84%
Grok 4.20 (Beta, Reasoning)	97%	$0.031	17.2s	95%
Stealth: Hunter Alpha	95%	$0.0000	44.1s	88%
GPT-4.1	94%	$0.011	15.1s	86%
Gemini 2.5 Flash	90%	$0.0038	4.0s	83%
Gemini 2.5 Pro	100%	$0.038	23.8s	97%
Claude Sonnet 4.6	98%	$0.038	13.8s	94%
GPT-5.4 Nano (Reasoning)	90%	$0.0045	18.9s	84%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
30	Claude Opus 4.5	$0.064	13.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Claude Sonnet 4.5	$0.036	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Gemini 2.5 Pro	$0.038	23.8s	97%	100	100	100	100	100	100	100	100	100	96	100%
78	Gemini 3.1 Pro (Preview)	$0.088	1.1m	98%	100	100	100	100	100	100	97	97	97	97	99%
71	Grok 4	$0.064	1.4m	97%	100	100	100	100	98	98	98	98	98	98	99%
47	Claude Opus 4.6	$0.065	15.3s	95%	100	100	100	100	97	97	97	97	97	96	98%
19	Claude Sonnet 4.6	$0.038	13.8s	94%	100	100	100	100	98	98	98	98	96	94	98%
6	GPT-5.4	$0.018	11.4s	93%	100	100	100	100	97	97	97	97	97	93	98%
2	Z.AI GLM 5 Turbo	$0.0097	19.4s	94%	100	100	100	100	100	97	97	96	96	94	98%
58	MoonshotAI: Kimi K2.5	$0.017	2.0m	94%	100	100	100	98	97	97	96	96	96	95	98%
14	Grok 4.20 (Beta, Reasoning)	$0.031	17.2s	95%	98	98	98	98	98	96	96	96	96	95	97%
97	Claude Sonnet 4.6 (Reasoning)	$0.115	1.3m	95%	100	97	97	97	97	97	97	96	96	96	97%
35	ByteDance Seed 1.6	$0.0098	1.5m	94%	100	98	98	98	96	96	96	96	96	95	97%
24	Claude Sonnet 4	$0.036	13.3s	90%	100	100	100	100	96	96	96	96	94	91	97%
1	Grok 4 Fast	$0.0024	15.1s	94%	98	98	98	96	96	96	96	96	96	93	97%
82	Claude Opus 4.6 (Reasoning)	$0.097	33.1s	92%	100	100	98	98	98	96	94	94	94	94	97%
32	Qwen 3.5 35B	$0.020	1.1m	93%	100	98	97	97	96	96	96	96	94	94	96%
28	GPT-5.4 Mini (Reasoning)	$0.025	46.1s	94%	98	98	96	96	96	96	95	95	95	95	96%
3	Grok 4.1 Fast	$0.0032	24.9s	93%	98	98	98	96	96	96	96	95	94	93	96%
23	Qwen 3.5 Flash	$0.0045	1.3m	92%	100	98	96	96	96	96	96	94	94	93	96%
76	GPT-5.4 (Reasoning)	$0.067	1.2m	92%	98	98	98	96	96	96	96	94	94	93	96%
11	Gemini 3 Flash (Preview, Reasoning)	$0.012	19.4s	91%	100	98	98	97	96	96	94	94	93	91	96%
89	GPT-5	$0.075	1.9m	93%	98	98	96	96	96	96	96	94	93	92	96%
40	Aion 2.0	$0.011	1.4m	92%	98	98	96	96	96	96	95	94	93	90	95%
5	Stealth: Healer Alpha	$0.0000	25.4s	91%	100	96	96	96	96	95	95	94	93	92	95%
10	Gemini 2.5 Flash (Reasoning)	$0.0098	14.4s	90%	100	98	96	96	96	94	93	93	93	91	95%
4	Grok 4.20 (Beta)	$0.0090	3.5s	90%	97	97	97	97	97	96	94	93	93	87	95%
15	Stealth: Hunter Alpha	$0.0000	44.1s	88%	100	98	98	96	96	95	94	94	92	86	95%
54	Nemotron 3 Super	$0.0000	2.2m	92%	96	96	96	95	95	95	95	95	93	92	95%
46	Z.AI GLM 5	$0.017	1.1m	87%	100	100	98	96	93	93	93	93	92	89	95%
41	ByteDance Seed 2.0 Lite	$0.0091	1.4m	91%	98	96	96	96	96	95	92	92	92	92	95%
39	GPT-5.2	$0.036	35.1s	91%	97	97	96	95	95	95	93	93	93	91	95%
64	Qwen 3.5 122B	$0.029	1.3m	89%	100	98	96	96	96	93	93	92	91	91	95%
63	o4 Mini High	$0.040	1.0m	90%	98	98	95	95	94	94	94	94	92	92	94%
84	Qwen 3.5 397B A17B	$0.027	2.8m	92%	96	96	96	96	96	96	96	91	91	91	94%
43	Z.AI GLM 4.7	$0.012	1.2m	89%	100	96	96	96	95	95	94	93	91	87	94%
26	GPT-5.4 (Reasoning, Low)	$0.026	18.2s	89%	98	98	96	93	93	93	93	93	91	91	94%
70	GPT-5.1	$0.050	1.2m	91%	96	96	96	96	96	93	92	92	92	91	94%
16	GPT-4.1	$0.011	15.1s	86%	100	100	98	96	96	95	91	90	88	86	94%
72	Qwen 3.5 9B	$0.0023	2.6m	88%	98	98	96	96	95	95	92	91	90	87	94%
96	Qwen 3.5 27B	$0.039	3.1m	90%	96	96	96	96	93	93	93	93	93	91	94%
73	Gemini 3 Pro (Preview)	$0.068	39.7s	87%	100	98	96	96	96	96	89	89	87	87	93%
38	o4 Mini	$0.023	41.0s	88%	98	96	96	94	94	94	92	92	89	87	93%
86	ByteDance Seed 2.0 Mini	$0.0036	3.1m	86%	98	98	96	95	93	92	92	92	90	85	93%
7	Gemini 3 Flash (Preview)	$0.0039	6.2s	88%	96	96	96	96	96	91	91	91	91	87	93%
51	Z.AI GLM 4.6	$0.0054	1.5m	86%	97	96	96	96	94	93	93	91	87	85	93%
9	Inception Mercury 2	$0.0040	5.5s	88%	95	95	95	93	93	93	93	92	90	85	92%
12	GPT-5.4 Mini (Reasoning, Low)	$0.0070	6.8s	86%	96	96	95	94	93	92	91	89	87	87	92%
27	Stealth: Aurora Alpha	—	7.5s	83%	96	96	95	90	90	90	90	90	90	83	91%
21	MiniMax M2.5	$0.0023	30.1s	84%	96	96	93	92	92	91	90	88	85	84	91%
53	GPT-5 Mini	$0.010	1.1m	83%	96	94	94	94	94	93	91	89	83	78	91%
108	Claude Opus 4	$0.181	21.6s	81%	98	96	94	91	89	89	89	89	85	82	90%
22	Gemini 2.5 Flash Lite (Reasoning)	$0.0030	31.0s	84%	95	95	92	92	90	90	89	89	87	83	90%
20	GPT-5.4 Nano (Reasoning)	$0.0045	18.9s	84%	95	95	93	92	90	90	88	88	87	83	90%
17	Gemini 2.5 Flash	$0.0038	4.0s	83%	96	96	94	93	93	89	89	85	85	82	90%
48	GPT-4o, May 13th (temp=0)	$0.043	5.5s	86%	93	93	91	91	91	89	89	89	88	84	90%
13	Gemini 3.1 Flash Lite (Preview)	$0.0031	3.0s	84%	97	93	89	89	89	89	89	89	89	87	90%
36	Qwen 3.5 Plus (2026-02-15)	$0.0054	38.7s	83%	96	96	91	91	91	91	89	89	82	80	90%
25	Z.AI GLM 4.5	$0.0043	29.1s	84%	94	94	92	92	92	90	89	87	83	83	90%
37	MiniMax M2.7	$0.0028	33.9s	80%	96	96	95	93	90	88	86	85	84	81	89%
31	Writer: Palmyra X5	$0.010	16.4s	81%	96	93	91	91	89	89	88	85	83	82	89%
50	Z.AI GLM 4.7 Flash	$0.0022	1.0m	81%	93	91	91	89	88	88	88	85	84	78	88%
79	Claude 3.5 Sonnet	$0.067	15.2s	80%	92	92	92	92	87	87	87	83	83	81	88%
33	Mistral Small 4 (Reasoning)	$0.0025	16.2s	79%	93	91	90	89	88	88	88	81	79	78	87%
60	Claude 3.7 Sonnet	$0.038	15.8s	84%	90	87	87	87	87	87	85	85	85	85	87%
42	DeepSeek-V2 Chat	$0.0033	20.2s	76%	96	91	90	90	85	85	83	83	81	78	86%
45	DeepSeek V3 (2024-12-26)	$0.0031	26.1s	77%	95	93	92	85	85	85	83	83	81	80	86%
29	Mistral Medium 3.1	$0.0046	11.2s	82%	90	87	87	87	87	87	87	83	82	80	86%
52	Mistral Large	$0.020	14.0s	78%	94	91	89	85	85	85	83	83	81	81	86%
109	Nemotron 3 Nano	$0.0054	5.6m	78%	93	91	88	88	86	86	84	82	79	78	86%
56	Mistral Large 2	$0.020	13.7s	77%	92	89	89	85	83	83	83	83	83	80	85%
34	Mistral Large 3	$0.0049	13.6s	82%	87	85	85	85	85	85	83	83	83	81	84%
61	DeepSeek V3.2	$0.0020	19.2s	68%	98	92	91	87	87	85	85	81	80	56	84%
55	Qwen3 235B A22B Instruct 2507	$0.0010	26.4s	73%	92	91	89	87	85	83	83	82	78	68	84%
44	ByteDance Seed 1.6 Flash	$0.0012	14.8s	75%	91	89	88	86	84	84	82	81	80	72	84%
49	GPT-4o, Aug. 6th (temp=0)	$0.016	4.2s	77%	91	84	84	82	82	82	81	81	81	79	83%
62	Qwen 3 32B	$0.0012	25.7s	71%	91	90	86	84	84	84	82	80	69	67	82%
85	GPT-4o, May 13th (temp=1)	$0.042	5.3s	67%	91	89	88	79	79	79	78	74	70	66	79%
59	GPT-4.1 Mini	$0.0025	11.0s	71%	85	85	85	82	80	80	80	78	72	66	79%
57	Mistral Small 3.2 24B	$0.0009	12.4s	73%	85	81	80	80	78	78	76	74	74	74	78%
95	DeepSeek V3 (2025-03-24)	$0.0025	31.1s	41%	96	96	87	87	85	83	83	78	72	3	77%
69	Inception Mercury	$0.0008	9.7s	62%	95	86	83	80	76	76	71	71	69	62	77%
94	GPT-4o, Aug. 6th (temp=1)	$0.016	4.5s	42%	89	89	88	86	84	82	82	81	78	3	76%
65	Mistral Small Creative	$0.0009	4.0s	68%	88	80	79	79	77	77	74	71	71	64	76%
66	Gemini 2.5 Flash Lite	$0.0008	3.2s	66%	85	82	78	76	76	74	71	68	68	66	74%
67	Gemma 3 27B	$0.0007	14.1s	69%	78	77	76	75	72	72	72	72	72	71	74%
68	Claude 3.5 Haiku	$0.0076	7.8s	70%	79	76	74	74	74	74	74	74	70	70	74%
80	Hermes 3 405B	$0.0070	24.4s	64%	82	80	78	76	75	74	69	66	66	60	73%
88	Arcee AI: Trinity Large (Preview)	$0.0000	27.9s	54%	90	78	76	76	75	74	74	72	70	34	72%
112	GPT-5 Nano	$0.0063	2.6m	27%	92	92	90	89	88	87	86	84	3	0	71%
74	Arcee AI: Trinity Mini	$0.0005	7.9s	62%	79	79	79	79	75	74	69	61	58	57	71%
93	GPT-5.4 Mini	$0.0040	2.8s	42%	94	89	89	87	85	83	80	33	33	33	71%
99	Claude Haiku 4.5	$0.013	7.9s	28%	94	89	89	89	85	85	83	81	3	3	70%
75	Ministral 8B	$0.0008	6.3s	63%	76	74	71	69	69	66	66	66	66	64	69%
77	GPT-4o Mini (temp=0)	$0.0008	11.1s	64%	70	70	70	70	70	70	70	70	70	57	68%
83	Ministral 3B	$0.0003	2.2s	55%	80	79	78	76	71	69	67	60	52	50	68%
102	DeepSeek V3.1	$0.0024	37.1s	28%	92	92	89	89	85	85	76	69	3	3	68%
81	Ministral 3 14B	$0.0015	8.7s	62%	74	71	71	70	69	66	66	64	62	59	67%
98	Ministral 3 8B	$0.0011	4.2s	41%	79	74	74	74	74	74	71	71	71	0	66%
87	Gemma 3 12B	$0.0006	15.5s	59%	72	69	69	67	66	66	64	63	63	52	65%
90	Qwen 2.5 72B	$0.0011	13.4s	53%	81	76	74	72	70	69	64	53	47	44	65%
91	Ministral 3 3B	$0.0008	3.3s	50%	80	77	76	76	72	67	57	54	48	35	64%
92	Mistral Small 4	$0.0015	6.4s	50%	81	74	71	64	63	62	61	59	57	42	63%
104	Llama 3.1 Nemotron 70B	$0.0091	34.0s	37%	76	72	70	67	61	61	58	57	55	3	58%
105	Cohere Command R+ (Aug. 2024)	$0.021	11.9s	38%	74	70	69	68	66	63	52	49	44	0	56%
103	Llama 3.1 70B	$0.0035	13.2s	34%	76	74	70	67	58	57	54	52	47	0	56%
107	Hermes 3 70B	$0.0021	57.3s	35%	75	69	69	67	61	57	54	50	46	0	55%
101	GPT-4o Mini (temp=1)	$0.0009	9.8s	37%	69	64	60	57	57	57	54	49	43	3	51%
100	Claude 3 Haiku	$0.0025	5.2s	40%	67	60	54	51	49	49	49	47	39	33	50%
106	Mistral NeMO	$0.0011	7.4s	35%	58	54	44	43	41	39	39	38	38	36	43%
115	GPT-5.4 Nano (Reasoning, Low)	$0.0016	7.1s	0%	95	93	93	88	0	0	0	0	0	0	37%
110	Gemma 3 4B	$0.0004	20.8s	29%	40	38	37	36	35	33	33	33	20	17	32%
111	Llama 3.1 8B	$0.0002	19.9s	25%	48	44	38	36	36	35	33	33	3	3	31%
114	GPT-5.4 Nano	$0.0009	3.2s	9%	79	74	56	53	43	3	0	0	0	0	31%
117	LFM2 24B	$0.0015	3.5m	25%	38	35	34	34	34	33	33	33	3	3	28%
113	Rocinante 12B	$0.0014	6.6s	22%	51	44	35	33	33	33	33	3	0	0	27%
116	GPT-4.1 Nano	$0.0006	6.9s	12%	51	22	20	19	17	17	13	3	3	0	17%
118	WizardLM 2 8x22b	$0.0060	28.7s	0%	3	3	3	3	0	0	0	0	0	0	1%
81.00%

Median	Evaluator	Top 3	Flop 3
75.0%	Accuracy (recall)	100Claude Sonnet 4.5 100Claude Opus 4.5 99Gemini 2.5 Pro	0Gemma 3 4B 0WizardLM 2 8x22b 0LFM2 24B
91.3%	Precision	100Claude Opus 4.5 100Grok 4 100Grok 4 Fast	0WizardLM 2 8x22b 3LFM2 24B 7Gemma 3 4B
100.0%	Structural validity	100Gemini 3 Pro (Preview) 100Gemini 3.1 Flash Lite (Preview) 100Claude 3.5 Sonnet	4WizardLM 2 8x22b 36GPT-5.4 Nano 39GPT-4.1 Nano

Codex Violation Detection

Large codex (40 entries), short passage (165 words)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)