Large codex (40 entries), long passage (1,019 words)

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Grok 4	98%
Claude Sonnet 4.6 (Reasoning)	97%
Qwen 3.5 397B A17B	97%
GPT-5	97%
Gemini 3.1 Pro (Preview)	96%
GPT-5.2	96%
Claude Opus 4.6 (Reasoning)	96%
Gemini 2.5 Pro	96%
Grok 4.20 (Beta, Reasoning)	96%
Claude Opus 4.5	96%
Grok 4 Fast	96%
Qwen 3.5 27B	96%
GPT-5.4 (Reasoning)	95%
Grok 4.1 Fast	95%
o4 Mini High	95%
GPT-5 Mini	95%
GPT-5.4	95%
Nemotron 3 Super	95%
Gemini 2.5 Flash (Reasoning)	95%
GPT-5.4 (Reasoning, Low)	95%

	Score	Cost	Time
Grok 4 Fast	96%	$0.0032	21.1s
Grok 4.1 Fast	95%	$0.0040	40.0s
Gemini 2.5 Flash (Reasoning)	95%	$0.016	24.6s
Stealth: Healer Alpha	91%	$0.0000	33.9s
Inception Mercury 2	89%	$0.0066	10.4s
Gemini 3 Flash (Preview)	90%	$0.0068	9.3s
Stealth: Hunter Alpha	91%	$0.0000	1.1m
Z.AI GLM 5 Turbo	94%	$0.016	36.7s
Gemini 2.5 Flash	87%	$0.0051	5.7s
Gemini 2.5 Flash Lite (Reasoning)	89%	$0.0041	42.5s
GPT-5.4	95%	$0.028	20.6s
Qwen 3.5 Flash	93%	$0.0058	1.4m
GPT-5 Mini	95%	$0.017	1.3m
GPT-5.4 Mini (Reasoning, Low)	87%	$0.010	9.6s
Gemini 3 Flash (Preview, Reasoning)	92%	$0.021	33.3s
Z.AI GLM 4.7	93%	$0.018	2.3m
Aion 2.0	87%	$0.015	2.3m
Gemini 3.1 Flash Lite (Preview)	83%	$0.0038	3.7s
Mistral Large 3	85%	$0.0060	25.8s
GPT-5.4 Nano (Reasoning)	70%	$0.0096	40.4s

	Score	Cost	Speed	Stability
Grok 4 Fast	96%	$0.0032	21.1s	89%
Grok 4.1 Fast	95%	$0.0040	40.0s	90%
Gemini 2.5 Flash (Reasoning)	95%	$0.016	24.6s	92%
Z.AI GLM 5 Turbo	94%	$0.016	36.7s	92%
GPT-5.4	95%	$0.028	20.6s	93%
GPT-5 Mini	95%	$0.017	1.3m	93%
Gemini 3 Flash (Preview)	90%	$0.0068	9.3s	87%
Stealth: Hunter Alpha	91%	$0.0000	1.1m	86%
Inception Mercury 2	89%	$0.0066	10.4s	85%
Gemini 3 Flash (Preview, Reasoning)	92%	$0.021	33.3s	88%
GPT-5.4 (Reasoning, Low)	95%	$0.042	30.5s	92%
Qwen 3.5 Flash	93%	$0.0058	1.4m	86%
Gemini 2.5 Flash Lite (Reasoning)	89%	$0.0041	42.5s	85%
Grok 4.20 (Beta, Reasoning)	96%	$0.049	27.1s	92%
Stealth: Healer Alpha	91%	$0.0000	33.9s	81%
Qwen 3.5 27B	96%	$0.027	2.2m	94%
GPT-5.4 Mini (Reasoning, Low)	87%	$0.010	9.6s	84%
o4 Mini	92%	$0.031	42.1s	89%
GPT-5.2	96%	$0.052	55.4s	93%
Gemini 2.5 Flash	87%	$0.0051	5.7s	81%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
44	Grok 4	$0.095	2.0m	95%	100	99	99	99	99	98	97	97	96	95	98%
89	Claude Sonnet 4.6 (Reasoning)	$0.229	2.9m	95%	99	98	98	98	98	97	97	96	96	94	97%
51	Qwen 3.5 397B A17B	$0.044	4.6m	92%	100	100	99	97	96	96	96	95	95	93	97%
63	GPT-5	$0.115	3.2m	95%	98	98	97	97	96	96	96	96	96	96	97%
66	Gemini 3.1 Pro (Preview)	$0.153	2.0m	94%	98	98	97	97	97	96	96	96	95	94	96%
19	GPT-5.2	$0.052	55.4s	93%	99	99	98	97	96	96	96	95	94	93	96%
74	Claude Opus 4.6 (Reasoning)	$0.189	1.4m	94%	98	97	97	97	97	97	96	95	94	94	96%
31	Gemini 2.5 Pro	$0.073	49.3s	93%	100	97	96	96	96	96	96	95	94	92	96%
14	Grok 4.20 (Beta, Reasoning)	$0.049	27.1s	92%	99	99	98	96	95	95	94	94	94	94	96%
32	Claude Opus 4.5	$0.085	20.6s	94%	97	97	96	96	96	96	95	95	95	93	96%
1	Grok 4 Fast	$0.0032	21.1s	89%	100	100	99	97	96	95	94	92	91	90	96%
16	Qwen 3.5 27B	$0.027	2.2m	94%	96	96	96	96	96	95	95	95	95	94	96%
49	GPT-5.4 (Reasoning)	$0.095	1.5m	92%	98	98	97	97	97	96	94	93	93	92	95%
2	Grok 4.1 Fast	$0.0040	40.0s	90%	100	99	98	97	97	95	95	94	90	90	95%
34	o4 Mini High	$0.059	1.5m	91%	98	97	97	96	95	94	94	94	93	93	95%
6	GPT-5 Mini	$0.017	1.3m	93%	97	97	96	95	95	95	94	94	94	93	95%
5	GPT-5.4	$0.028	20.6s	93%	97	96	95	95	95	95	95	95	94	93	95%
36	Nemotron 3 Super	$0.0000	5.1m	92%	98	96	95	95	95	95	94	93	93	92	95%
3	Gemini 2.5 Flash (Reasoning)	$0.016	24.6s	92%	97	96	95	95	95	95	95	95	93	91	95%
11	GPT-5.4 (Reasoning, Low)	$0.042	30.5s	92%	96	96	96	96	96	95	94	94	92	91	95%
54	GPT-5.1	$0.088	2.0m	90%	98	97	96	95	95	94	94	93	92	90	94%
4	Z.AI GLM 5 Turbo	$0.016	36.7s	92%	96	96	95	95	95	94	93	93	92	91	94%
24	Z.AI GLM 5	$0.030	2.1m	90%	99	95	95	94	94	94	93	93	93	91	94%
23	Z.AI GLM 4.6	$0.012	3.0m	90%	96	96	96	96	94	93	92	92	91	91	94%
26	Z.AI GLM 4.7	$0.018	2.3m	88%	98	96	96	93	93	93	93	92	90	88	93%
21	Qwen 3.5 35B	$0.027	1.4m	88%	97	96	95	95	95	93	92	91	91	87	93%
22	ByteDance Seed 1.6	$0.015	2.4m	88%	97	96	95	94	94	93	93	92	89	89	93%
12	Qwen 3.5 Flash	$0.0058	1.4m	86%	98	97	95	94	94	93	92	92	91	83	93%
33	Qwen 3.5 122B	$0.039	1.9m	90%	95	94	94	94	93	92	92	92	91	91	93%
18	o4 Mini	$0.031	42.1s	89%	96	94	94	94	93	92	92	91	90	89	92%
10	Gemini 3 Flash (Preview, Reasoning)	$0.021	33.3s	88%	95	94	93	93	92	92	92	91	90	88	92%
42	Claude Opus 4.6	$0.082	21.5s	89%	93	93	93	93	91	91	90	90	90	90	91%
30	Claude Sonnet 4.6	$0.051	24.9s	88%	94	94	92	92	91	91	90	90	90	88	91%
8	Stealth: Hunter Alpha	$0.0000	1.1m	86%	96	94	93	93	92	91	91	90	86	85	91%
29	ByteDance Seed 2.0 Lite	$0.013	2.1m	86%	95	95	93	92	92	91	91	88	87	86	91%
41	MoonshotAI: Kimi K2.5	$0.028	2.4m	84%	97	94	94	92	92	92	90	90	86	81	91%
15	Stealth: Healer Alpha	$0.0000	33.9s	81%	98	97	97	94	92	91	90	86	83	81	91%
28	Claude Sonnet 4	$0.047	20.4s	88%	93	92	92	92	92	91	91	89	88	87	91%
50	ByteDance Seed 2.0 Mini	$0.0051	4.9m	87%	94	93	92	91	90	90	90	90	88	88	91%
7	Gemini 3 Flash (Preview)	$0.0068	9.3s	87%	93	92	92	91	90	90	90	88	88	88	90%
97	Claude Opus 4	$0.248	37.7s	83%	96	94	92	91	90	90	87	87	86	84	90%
48	GPT-5.4 Mini (Reasoning)	$0.056	1.2m	84%	94	93	92	92	90	88	88	87	86	85	90%
40	Qwen 3.5 9B	$0.0031	3.4m	84%	95	93	92	91	91	90	87	87	86	83	89%
13	Gemini 2.5 Flash Lite (Reasoning)	$0.0041	42.5s	85%	93	92	91	91	91	89	88	86	86	85	89%
38	Claude Sonnet 4.5	$0.051	20.8s	84%	93	92	91	90	89	89	87	87	85	85	89%
9	Inception Mercury 2	$0.0066	10.4s	85%	93	91	90	90	90	90	89	89	84	83	89%
78	Aion 2.0	$0.015	2.3m	42%	100	99	98	97	96	96	96	95	94	3	87%
17	GPT-5.4 Mini (Reasoning, Low)	$0.010	9.6s	84%	91	90	89	89	88	87	86	85	85	84	87%
67	Gemini 3 Pro (Preview)	$0.096	1.0m	77%	93	92	92	90	90	90	88	86	84	68	87%
20	Gemini 2.5 Flash	$0.0051	5.7s	81%	91	91	90	88	88	88	85	83	82	80	87%
35	MiniMax M2.7	$0.0048	1.2m	80%	91	89	88	86	85	85	85	85	84	79	86%
39	Mistral Large	$0.024	19.0s	78%	93	89	89	85	85	85	83	82	82	81	85%
25	Mistral Large 3	$0.0060	25.8s	81%	88	87	87	86	86	85	85	84	82	81	85%
37	GPT-4.1	$0.017	22.4s	78%	89	88	87	87	86	85	84	84	83	73	85%
47	Stealth: Aurora Alpha	—	12.5s	77%	89	88	87	87	83	83	83	82	82	77	84%
27	Gemini 3.1 Flash Lite (Preview)	$0.0038	3.7s	80%	86	85	85	84	83	83	82	82	81	79	83%
43	Mistral Large 2	$0.024	17.4s	77%	89	86	85	84	84	83	81	81	81	75	83%
45	MiniMax M2.5	$0.0039	52.7s	74%	90	89	87	83	83	81	79	77	76	75	82%
64	GPT-5 Nano	$0.0079	3.0m	73%	86	84	84	83	83	79	78	76	72	72	80%
56	DeepSeek V3 (2024-12-26)	$0.0038	27.4s	67%	89	88	86	85	84	83	79	72	65	58	79%
52	DeepSeek-V2 Chat	$0.0038	24.8s	71%	87	84	83	81	79	79	77	74	73	70	79%
46	Mistral Small 4 (Reasoning)	$0.0038	30.0s	75%	81	81	80	80	79	79	78	76	75	73	78%
58	DeepSeek V3.2	$0.0028	41.9s	65%	98	83	80	80	77	75	75	73	72	71	78%
82	Qwen 3.5 Plus (2026-02-15)	$0.011	1.9m	41%	96	93	92	91	89	87	86	76	62	3	77%
59	Z.AI GLM 4.5	$0.0053	35.0s	66%	92	82	81	78	76	75	75	73	72	66	77%
55	Claude Haiku 4.5	$0.016	12.8s	72%	84	79	79	77	76	76	75	75	75	72	77%
61	Z.AI GLM 4.7 Flash	$0.0030	1.7m	71%	83	82	80	78	77	77	74	73	73	69	77%
53	Mistral Medium 3.1	$0.0057	16.6s	71%	81	81	79	78	78	77	76	75	73	67	77%
60	Qwen3 235B A22B Instruct 2507	$0.0014	43.6s	66%	83	82	82	79	76	74	74	72	68	65	75%
69	Claude 3.7 Sonnet	$0.043	18.3s	66%	84	82	79	79	76	73	73	73	68	66	75%
72	Claude 3.5 Sonnet	$0.071	15.9s	71%	79	78	78	77	76	74	74	74	72	68	75%
57	ByteDance Seed 1.6 Flash	$0.0016	22.7s	69%	82	80	76	76	75	74	73	71	71	69	75%
62	Grok 4.20 (Beta)	$0.011	5.2s	66%	82	81	77	77	77	72	69	68	66	65	73%
86	Nemotron 3 Nano	$0.0056	5.7m	66%	80	77	77	76	74	74	73	68	65	60	72%
65	Writer: Palmyra X5	$0.013	20.0s	62%	81	78	77	77	76	72	71	71	69	51	72%
81	DeepSeek V3.1	$0.0023	1.2m	42%	85	84	83	83	79	77	76	75	65	3	71%
71	Qwen 3 32B	$0.0019	1.2m	61%	79	78	77	76	74	71	68	65	62	53	70%
94	GPT-5.4 Nano (Reasoning)	$0.0096	40.4s	26%	92	92	90	88	88	85	85	82	0	0	70%
68	GPT-4.1 Mini	$0.0024	9.0s	58%	83	79	77	70	69	68	65	64	63	57	70%
83	DeepSeek V3 (2025-03-24)	$0.0030	35.9s	41%	88	86	77	77	75	73	71	66	66	3	68%
79	GPT-4o, May 13th (temp=1)	$0.047	5.3s	59%	79	77	72	69	69	69	66	65	59	52	68%
70	Mistral Small Creative	$0.0012	8.2s	60%	72	72	71	68	66	66	65	64	61	57	66%
76	GPT-4o, Aug. 6th (temp=1)	$0.019	5.8s	59%	73	72	71	70	67	65	64	63	57	56	66%
77	Inception Mercury	$0.0010	19.8s	52%	81	73	72	70	66	62	61	61	57	46	65%
75	Qwen 2.5 72B	$0.0015	12.7s	56%	76	72	67	64	63	63	63	61	59	53	64%
73	GPT-5.4 Mini	$0.0060	3.8s	59%	71	66	66	66	66	64	63	58	57	57	64%
93	GPT-4o, Aug. 6th (temp=0)	$0.052	18.5s	49%	76	76	69	69	69	69	67	66	34	34	63%
84	Mistral Small 3.2 24B	$0.0013	31.5s	49%	70	66	65	65	62	58	57	54	49	40	59%
92	Claude 3.5 Haiku	$0.0082	7.4s	38%	75	73	66	64	63	63	61	59	59	0	58%
80	Ministral 3 8B	$0.0014	10.2s	51%	66	64	62	58	58	57	57	56	52	46	58%
88	Hermes 3 405B	$0.0079	17.3s	46%	73	70	64	58	55	55	55	51	50	47	58%
87	Gemma 3 27B	$0.0009	24.2s	47%	71	68	63	57	56	55	53	53	52	44	57%
85	Ministral 8B	$0.0010	15.3s	49%	65	63	60	57	56	53	51	51	51	51	56%
90	Arcee AI: Trinity Mini	$0.0006	10.2s	41%	72	70	69	55	52	51	51	51	43	43	56%
91	Mistral Small 4	$0.0017	7.8s	43%	73	58	57	54	52	52	50	46	44	44	53%
96	Ministral 3 14B	$0.0019	14.1s	39%	63	63	60	60	60	57	53	48	42	3	51%
100	Hermes 3 70B	$0.0024	23.5s	34%	64	64	61	60	55	53	51	50	38	0	49%
95	Claude 3 Haiku	$0.0026	4.0s	42%	61	58	51	49	47	46	44	44	44	44	49%
98	Gemma 3 12B	$0.0007	23.4s	40%	61	59	55	52	48	44	44	43	40	40	49%
104	Llama 3.1 Nemotron 70B	$0.010	26.3s	36%	68	58	57	55	55	52	50	44	44	3	49%
112	GPT-4o, May 13th (temp=0)	$0.074	20.0s	20%	78	76	76	74	68	39	38	3	3	3	46%
102	GPT-4o Mini (temp=1)	$0.0010	10.0s	35%	57	55	55	52	46	44	44	43	40	17	45%
101	WizardLM 2 8x22b	$0.0069	30.7s	40%	55	49	48	46	46	43	43	42	39	39	45%
106	GPT-5.4 Nano (Reasoning, Low)	$0.0030	13.8s	18%	82	77	75	75	73	64	0	0	0	0	45%
99	Ministral 3B	$0.0004	8.2s	41%	46	44	44	43	43	43	42	41	40	39	43%
103	Ministral 3 3B	$0.0011	10.1s	39%	47	46	44	42	41	41	41	40	38	38	42%
107	Cohere Command R+ (Aug. 2024)	$0.027	18.8s	30%	55	55	54	51	49	49	42	42	3	3	40%
105	Mistral NeMO	$0.0014	1.3m	33%	55	42	41	39	38	37	36	36	35	35	39%
111	Llama 3.1 70B	$0.0044	1.5m	26%	55	52	52	46	46	43	34	3	3	0	33%
108	Llama 3.1 8B	$0.0004	30.0s	23%	57	51	42	38	36	33	33	32	3	3	33%
109	Gemini 2.5 Flash Lite	$0.0011	4.7s	21%	61	61	56	49	48	42	0	0	0	0	32%
110	Rocinante 12B	$0.0018	7.3s	23%	44	44	42	33	33	33	33	33	3	0	30%
118	Arcee AI: Trinity Large (Preview)	$0.0000	14.1m	25%	39	38	37	34	34	34	34	34	3	3	29%
117	LFM2 24B	$0.0019	4.9m	24%	36	35	34	33	33	33	33	3	3	3	25%
113	Gemma 3 4B	$0.0005	29.8s	18%	43	40	38	37	29	23	21	3	3	3	24%
116	GPT-4o Mini (temp=0)	$0.0015	2.5m	12%	53	44	41	40	35	3	3	3	3	3	23%
114	GPT-4.1 Nano	$0.0008	9.4s	11%	51	35	33	17	17	16	3	3	0	0	18%
115	GPT-5.4 Nano	$0.0015	5.5s	10%	41	36	34	30	26	3	0	0	0	0	17%
73.97%

Median	Evaluator	Top 3	Flop 3
56.3%	Accuracy (recall)	96Grok 4 94Claude Sonnet 4.6 (Reasoning) 93Qwen 3.5 397B A17B	0Rocinante 12B 0LFM2 24B 0Arcee AI: Trinity Large (Preview)
89.1%	Precision	100o4 Mini 100o4 Mini High 99GPT-5	2LFM2 24B 5Arcee AI: Trinity Large (Preview) 9Rocinante 12B
100.0%	Structural validity	100Gemini 3 Flash (Preview) 100Grok 4.1 Fast 100Hermes 3 405B	26GPT-5.4 Nano 39GPT-4.1 Nano 55GPT-4o Mini (temp=0)

Codex Violation Detection

Large codex (40 entries), long passage (1,019 words)

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)