20 codex entries

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Claude Opus 4.6	100%
Claude Opus 4.5	100%
Grok 4.20 (Beta)	99%
Gemini 3.1 Pro (Preview)	99%
ByteDance Seed 2.0 Lite	97%
Gemini 2.5 Pro	97%
Claude Sonnet 4.5	97%
GPT-5	97%
Grok 4	96%
Grok 4.20 (Beta, Reasoning)	96%
GPT-5.1	96%
Gemini 3 Pro (Preview)	95%
GPT-5.4	95%
Gemini 3 Flash (Preview, Reasoning)	95%
Z.AI GLM 5 Turbo	95%
Grok 4.1 Fast	95%
GPT-5.2	95%
Qwen 3.5 27B	94%
Qwen 3.5 35B	94%

	Score	Cost	Time
Grok 4.20 (Beta)	99%	$0.0047	2.0s
Grok 4 Fast	87%	$0.0018	13.2s
Gemini 2.5 Flash	93%	$0.0023	2.5s
Stealth: Healer Alpha	92%	$0.0000	23.5s
Gemini 3 Flash (Preview)	92%	$0.0031	4.0s
Grok 4.1 Fast	95%	$0.0022	21.9s
Mistral Medium 3.1	92%	$0.0027	8.1s
ByteDance Seed 1.6 Flash	87%	$0.0009	11.8s
Gemini 3.1 Flash Lite (Preview)	88%	$0.0019	2.2s
Gemini 2.5 Flash Lite (Reasoning)	91%	$0.0022	17.1s
GPT-4.1	75%	$0.0076	8.2s
GPT-5.4 Mini (Reasoning, Low)	91%	$0.0054	8.4s
Z.AI GLM 5 Turbo	95%	$0.0077	20.7s
GPT-5.4	95%	$0.011	8.4s
Claude Haiku 4.5	83%	$0.0070	5.0s
Inception Mercury 2	86%	$0.0026	4.0s
Gemini 2.5 Flash (Reasoning)	93%	$0.0076	12.3s
Stealth: Hunter Alpha	93%	$0.0000	52.6s
GPT-5.4 Nano (Reasoning)	55%	$0.0019	14.4s
Gemini 2.5 Flash Lite	71%	$0.0004	1.5s

	Score	Cost	Speed	Stability
Grok 4.20 (Beta)	99%	$0.0047	2.0s	95%
Gemini 3 Flash (Preview)	92%	$0.0031	4.0s	92%
Gemini 2.5 Flash	93%	$0.0023	2.5s	88%
Mistral Medium 3.1	92%	$0.0027	8.1s	88%
GPT-5.4	95%	$0.011	8.4s	93%
Gemini 3.1 Flash Lite (Preview)	88%	$0.0019	2.2s	88%
Grok 4.1 Fast	95%	$0.0022	21.9s	91%
GPT-4o, Aug. 6th (temp=0)	91%	$0.011	3.8s	89%
Gemini 2.5 Flash (Reasoning)	93%	$0.0076	12.3s	86%
GPT-5.4 Mini (Reasoning, Low)	91%	$0.0054	8.4s	85%
Z.AI GLM 5 Turbo	95%	$0.0077	20.7s	89%
Claude Sonnet 4.5	97%	$0.022	8.9s	93%
Gemini 2.5 Flash Lite (Reasoning)	91%	$0.0022	17.1s	86%
Claude Opus 4.6	100%	$0.035	8.0s	100%
Claude Opus 4.5	100%	$0.037	8.6s	100%
Stealth: Healer Alpha	92%	$0.0000	23.5s	84%
Gemini 3 Flash (Preview, Reasoning)	95%	$0.013	20.4s	88%
GPT-5.4 (Reasoning, Low)	94%	$0.018	11.7s	88%
Inception Mercury 2	86%	$0.0026	4.0s	80%
ByteDance Seed 1.6 Flash	87%	$0.0009	11.8s	80%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
64	Claude Opus 4.6 (Reasoning)	$0.067	25.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Claude Opus 4.6	$0.035	8.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	Claude Opus 4.5	$0.037	8.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Grok 4.20 (Beta)	$0.0047	2.0s	95%	100	100	100	100	100	100	100	100	100	92	99%
72	Gemini 3.1 Pro (Preview)	$0.059	44.7s	97%	100	100	100	100	100	100	100	100	96	96	99%
32	ByteDance Seed 2.0 Lite	$0.0065	1.0m	91%	100	100	100	100	97	96	96	96	96	92	97%
41	Gemini 2.5 Pro	$0.035	21.2s	92%	100	100	100	96	96	96	96	96	96	96	97%
12	Claude Sonnet 4.5	$0.022	8.9s	93%	100	100	100	100	100	100	92	92	92	92	97%
91	GPT-5	$0.051	1.3m	94%	97	97	97	97	97	97	97	97	97	92	97%
88	Grok 4	$0.048	1.2m	91%	100	100	97	97	97	97	97	97	94	89	96%
28	Grok 4.20 (Beta, Reasoning)	$0.024	26.7s	92%	100	97	97	97	97	97	97	94	94	89	96%
47	GPT-5.1	$0.029	42.6s	93%	97	97	97	97	97	97	97	94	92	92	96%
70	Gemini 3 Pro (Preview)	$0.051	33.2s	91%	100	97	97	97	96	96	92	92	92	92	95%
5	GPT-5.4	$0.011	8.4s	93%	96	96	96	96	96	96	96	96	92	92	95%
17	Gemini 3 Flash (Preview, Reasoning)	$0.013	20.4s	88%	100	100	100	96	96	96	92	92	88	88	95%
11	Z.AI GLM 5 Turbo	$0.0077	20.7s	89%	100	100	97	96	96	96	92	92	89	89	95%
7	Grok 4.1 Fast	$0.0022	21.9s	91%	97	97	97	97	97	94	94	92	91	91	95%
24	GPT-5.2	$0.023	21.2s	92%	97	97	96	96	96	96	92	92	92	92	95%
77	Qwen 3.5 27B	$0.019	1.5m	85%	100	100	100	92	92	92	92	92	92	88	94%
53	Qwen 3.5 35B	$0.018	55.5s	88%	100	97	97	96	96	96	92	92	92	85	94%
61	Aion 2.0	$0.0086	1.3m	87%	100	100	97	97	96	94	92	89	88	88	94%
103	ByteDance Seed 2.0 Mini	$0.0033	3.3m	86%	100	97	97	97	92	92	92	92	92	88	94%
18	GPT-5.4 (Reasoning, Low)	$0.018	11.7s	88%	97	97	97	96	92	92	92	92	92	92	94%
27	Claude Sonnet 4.6	$0.022	8.3s	85%	100	100	97	97	94	94	92	89	86	86	93%
93	Claude Sonnet 4.6 (Reasoning)	$0.067	43.0s	89%	96	96	96	96	96	92	92	92	92	88	93%
87	MoonshotAI: Kimi K2.5	$0.015	1.9m	88%	97	96	96	96	92	92	92	92	92	88	93%
9	Gemini 2.5 Flash (Reasoning)	$0.0076	12.3s	86%	100	97	97	97	96	92	89	89	88	88	93%
59	Z.AI GLM 5	$0.014	1.1m	86%	100	97	97	94	94	92	92	92	91	85	93%
31	Stealth: Hunter Alpha	$0.0000	52.6s	86%	100	97	97	94	92	92	92	91	89	88	93%
3	Gemini 2.5 Flash	$0.0023	2.5s	88%	100	92	92	92	92	92	92	92	92	92	93%
67	o4 Mini High	$0.031	50.7s	87%	97	97	97	97	97	94	89	89	89	85	93%
52	Claude 3.5 Sonnet	$0.041	10.2s	88%	97	96	96	96	96	96	92	89	88	85	93%
62	Qwen 3.5 Flash	$0.0041	1.3m	82%	100	97	97	96	96	96	96	88	85	76	93%
66	GPT-5.4 (Reasoning)	$0.038	37.1s	87%	97	97	92	92	92	92	92	92	88	88	92%
2	Gemini 3 Flash (Preview)	$0.0031	4.0s	92%	92	92	92	92	92	92	92	92	92	92	92%
4	Mistral Medium 3.1	$0.0027	8.1s	88%	96	96	92	92	92	92	92	92	89	88	92%
16	Stealth: Healer Alpha	$0.0000	23.5s	84%	97	97	97	97	94	91	89	88	86	85	92%
37	o4 Mini	$0.017	28.6s	87%	97	94	94	94	94	94	89	89	89	85	92%
22	Claude Sonnet 4	$0.022	9.0s	89%	92	92	92	92	92	92	92	92	88	88	92%
55	Z.AI GLM 4.7	$0.010	1.0m	87%	96	96	92	92	92	92	88	88	88	88	91%
8	GPT-4o, Aug. 6th (temp=0)	$0.011	3.8s	89%	92	92	92	92	92	92	89	89	89	89	91%
13	Gemini 2.5 Flash Lite (Reasoning)	$0.0022	17.1s	86%	94	94	94	94	92	91	91	88	86	86	91%
45	GPT-5.4 Mini (Reasoning)	$0.020	29.3s	87%	94	92	92	92	92	92	92	89	86	86	91%
10	GPT-5.4 Mini (Reasoning, Low)	$0.0054	8.4s	85%	94	94	94	94	92	92	89	89	88	81	91%
79	Qwen 3.5 122B	$0.025	1.1m	81%	97	97	97	92	92	88	88	88	85	81	91%
21	GPT-4o, Aug. 6th (temp=1)	$0.010	4.3s	80%	97	97	97	97	92	89	86	85	82	82	90%
86	GPT-5 Nano	$0.0052	1.8m	82%	97	94	94	94	89	89	88	86	85	85	90%
50	ByteDance Seed 1.6	$0.0066	1.0m	87%	92	92	89	89	89	89	89	89	89	89	90%
108	Claude Opus 4	$0.109	14.4s	81%	96	96	96	88	88	88	88	85	85	85	90%
30	Mistral Large	$0.011	8.7s	79%	96	96	96	92	89	88	88	86	81	81	89%
26	MiniMax M2.7	$0.0019	23.0s	80%	97	94	94	94	89	89	86	86	85	79	89%
48	GPT-5 Mini	$0.0094	45.9s	86%	92	89	89	89	89	89	89	88	86	85	89%
35	DeepSeek V3.2	$0.0011	12.7s	71%	100	97	97	97	85	85	85	81	81	77	89%
6	Gemini 3.1 Flash Lite (Preview)	$0.0019	2.2s	88%	88	88	88	88	88	88	88	88	88	88	88%
73	Z.AI GLM 4.7 Flash	$0.0023	1.5m	80%	97	91	91	89	88	86	86	86	85	82	88%
25	MiniMax M2.5	$0.0018	21.3s	81%	94	94	91	89	88	88	86	86	82	81	88%
33	Mistral Large 2	$0.012	8.8s	80%	92	92	92	89	88	88	88	85	81	77	87%
42	Qwen 3.5 Plus (2026-02-15)	$0.0039	34.5s	81%	92	92	92	89	88	88	85	85	85	79	87%
20	ByteDance Seed 1.6 Flash	$0.0009	11.8s	80%	94	91	91	91	91	91	88	82	82	73	87%
102	Qwen 3.5 9B	$0.0020	2.4m	67%	97	97	96	92	89	89	89	86	86	53	87%
76	Grok 4 Fast	$0.0018	13.2s	43%	100	97	97	97	97	96	94	94	94	3	87%
83	Z.AI GLM 4.6	$0.0053	1.4m	74%	100	96	92	91	88	88	85	77	77	73	87%
29	Writer: Palmyra X5	$0.0053	10.4s	78%	94	92	89	88	86	86	86	86	82	77	87%
23	Mistral Large 3	$0.0028	8.0s	78%	97	89	89	89	88	86	81	81	81	81	86%
106	Nemotron 3 Nano	$0.0023	3.1m	75%	94	94	91	91	89	82	82	82	82	73	86%
19	Inception Mercury 2	$0.0026	4.0s	80%	91	89	89	89	89	86	85	82	81	77	86%
39	GPT-5.4 Mini	$0.0033	8.8s	71%	97	97	92	88	85	85	81	81	81	70	86%
54	GPT-4o, May 13th (temp=0)	$0.025	3.5s	79%	91	89	89	89	86	86	82	82	82	79	85%
38	Stealth: Aurora Alpha	—	6.2s	81%	91	86	86	85	85	85	85	85	85	81	85%
43	Z.AI GLM 4.5	$0.0026	20.2s	76%	97	86	86	86	85	82	82	81	81	81	85%
51	Qwen3 235B A22B Instruct 2507	$0.0007	20.9s	68%	97	92	92	89	88	86	82	82	79	55	84%
58	DeepSeek V3.1	$0.0014	35.2s	71%	96	96	92	89	88	88	81	73	69	69	84%
40	Mistral Small 4 (Reasoning)	$0.0018	14.8s	75%	94	91	88	86	86	82	82	82	79	73	84%
36	Inception Mercury	$0.0006	7.7s	72%	96	94	89	88	85	85	81	78	77	70	84%
78	Claude 3.7 Sonnet	$0.023	10.0s	63%	100	100	92	92	89	78	77	73	70	62	83%
46	DeepSeek V3 (2024-12-26)	$0.0017	13.9s	71%	94	91	91	86	86	86	82	73	73	65	83%
82	Claude Haiku 4.5	$0.0070	5.0s	42%	97	97	97	94	94	89	86	86	85	3	83%
34	Claude 3.5 Haiku	$0.0047	6.1s	78%	85	85	85	85	85	85	85	82	75	75	83%
118	Qwen 3.5 397B A17B	$0.027	3.0m	43%	96	92	92	92	92	92	88	88	85	3	82%
65	GPT-4o, May 13th (temp=1)	$0.024	3.4s	73%	91	86	85	82	82	82	79	75	73	73	81%
69	Qwen 3 32B	$0.0011	32.4s	64%	91	91	89	88	86	79	79	77	73	53	81%
44	GPT-4.1 Mini	$0.0013	5.3s	71%	89	86	86	86	85	82	82	69	69	69	80%
57	Ministral 3 14B	$0.0009	4.5s	63%	94	94	89	86	81	73	73	73	69	69	80%
63	Arcee AI: Trinity Large (Preview)	$0.0000	13.2s	64%	89	85	85	81	77	74	73	73	70	61	77%
49	Mistral Small Creative	$0.0005	6.8s	71%	81	81	81	77	77	77	73	73	73	69	76%
71	DeepSeek V3 (2025-03-24)	$0.0013	23.4s	62%	86	86	82	82	77	77	73	73	73	51	76%
56	Mistral Small 3.2 24B	$0.0005	8.6s	70%	82	81	79	77	77	74	73	73	73	69	76%
60	Arcee AI: Trinity Mini	$0.0003	7.0s	66%	82	82	82	82	79	75	73	70	69	60	75%
81	Hermes 3 405B	$0.0041	19.8s	57%	94	91	86	81	77	73	69	61	60	58	75%
99	GPT-4.1	$0.0076	8.2s	27%	96	94	94	94	94	94	89	85	3	3	75%
74	Llama 3.1 Nemotron 70B	$0.0051	16.2s	62%	82	82	82	79	73	69	69	69	69	65	74%
68	Gemma 3 27B	$0.0005	14.1s	65%	82	77	77	73	73	69	69	69	69	69	73%
75	Llama 3.1 70B	$0.0018	15.9s	61%	82	82	77	77	76	73	73	73	61	51	73%
92	DeepSeek-V2 Chat	$0.0017	12.2s	41%	91	86	86	86	82	77	73	73	69	3	73%
89	Cohere Command R+ (Aug. 2024)	$0.013	11.8s	58%	88	82	77	77	73	70	66	65	60	55	71%
98	Gemini 2.5 Flash Lite	$0.0004	1.5s	24%	100	97	96	89	85	85	81	77	0	0	71%
113	Nemotron 3 Super	$0.0000	1.9m	29%	89	89	86	86	85	85	85	77	3	3	69%
84	WizardLM 2 8x22b	$0.0031	8.4s	57%	82	73	69	69	69	66	65	65	60	51	67%
80	Mistral Small 4	$0.0008	4.1s	57%	73	73	73	70	68	62	62	61	60	56	66%
85	Ministral 3 8B	$0.0007	4.7s	54%	81	78	71	70	66	66	62	56	55	52	66%
90	GPT-4o Mini (temp=0)	$0.0005	7.0s	55%	71	66	60	59	59	59	59	59	59	59	61%
94	GPT-4o Mini (temp=1)	$0.0005	8.2s	48%	79	71	71	66	66	59	55	48	44	44	60%
100	Ministral 8B	$0.0004	4.0s	36%	78	78	74	71	66	66	62	56	33	0	58%
97	Claude 3 Haiku	$0.0016	4.3s	45%	75	65	60	59	55	55	51	51	48	40	56%
105	Hermes 3 70B	$0.0012	29.1s	36%	79	71	69	69	69	60	59	44	33	0	55%
111	GPT-5.4 Nano (Reasoning)	$0.0019	14.4s	10%	94	91	91	91	91	88	0	0	0	0	55%
95	Ministral 3B	$0.0002	2.5s	47%	65	62	59	58	58	53	53	50	47	39	54%
96	Ministral 3 3B	$0.0004	2.2s	46%	62	60	59	59	56	55	47	43	42	41	52%
101	Mistral NeMO	$0.0006	4.2s	42%	65	61	61	53	50	48	47	44	44	42	52%
104	Gemma 3 12B	$0.0005	10.4s	37%	65	61	61	61	60	53	52	51	47	3	51%
107	Llama 3.1 8B	$0.0001	8.1s	30%	71	69	61	55	43	42	39	36	33	27	48%
112	Qwen 2.5 72B	$0.0012	41.3s	29%	91	49	49	44	43	41	40	39	38	36	47%
114	GPT-5.4 Nano (Reasoning, Low)	$0.0010	4.8s	5%	91	91	89	88	79	0	0	0	0	0	44%
110	Gemma 3 4B	$0.0003	24.5s	34%	48	40	38	38	37	36	36	36	35	32	38%
109	Rocinante 12B	$0.0009	8.3s	34%	43	42	40	38	38	38	33	33	29	28	36%
115	GPT-5.4 Nano	$0.0007	2.3s	7%	81	73	73	70	51	0	0	0	0	0	35%
116	LFM2 24B	$0.0006	1.4m	31%	40	39	39	36	36	34	33	28	25	25	33%
117	GPT-4.1 Nano	$0.0004	4.1s	13%	33	33	24	17	17	17	3	3	0	0	15%
81.27%

Median	Evaluator	Top 3	Flop 3
74.5%	Accuracy (recall)	100Claude Opus 4.5 100Claude Opus 4.6 (Reasoning) 100Claude Opus 4.6	0LFM2 24B 0GPT-4.1 Nano 0Rocinante 12B
88.4%	Precision	100Claude Opus 4.6 (Reasoning) 100Claude Opus 4.5 100Claude Opus 4.6	2GPT-4.1 Nano 9LFM2 24B 13Gemma 3 4B
100.0%	Structural validity	100Claude Sonnet 4.6 (Reasoning) 100GPT-4o Mini (temp=0) 100Qwen 3.5 27B	42GPT-4.1 Nano 50GPT-5.4 Nano (Reasoning, Low) 50GPT-5.4 Nano

Codex Violation Detection

20 codex entries

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)