10 codex entries

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Tooling Reasoning

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

	Score
Claude Opus 4.6 (Reasoning)	100%
Qwen3.7 Max	100%
Gemini 3.1 Pro (Preview)	100%
Qwen3.6 Max Preview	100%
GPT-5.5 (Reasoning)	100%
Z.AI GLM 5.2 (Reasoning, High)	100%
Z.AI GLM 5 Turbo	100%
GPT-5.5 (Reasoning, Low)	100%
Claude Opus 4.8 (Reasoning)	100%
Claude Opus 4.6	100%
Claude Opus 4.8 (Reasoning, Low)	100%
Qwen 3.5 397B A17B	100%
GPT-5.4 (Reasoning, Low)	100%
MoonshotAI: Kimi K2.5	100%
Claude Sonnet 5 (Reasoning)	100%
Claude Sonnet 5 (Reasoning, Low)	100%
Qwen 3.5 27B	100%
Gemini 3 Flash (Preview, Reasoning)	100%
Gemma 4 31B (Reasoning)	100%
Claude Opus 4.5	100%

	Score	Cost	Time
Gemini 3.1 Flash Lite (Preview)	94%	$0.0011	1.5s
Gemma 4 26B	100%	$0.0003	10.1s
Gemini 3 Flash (Preview)	100%	$0.0015	2.7s
Mistral Medium 3.1	99%	$0.0015	3.7s
DeepSeek V4 Pro	98%	$0.0027	11.8s
Inception Mercury 2	98%	$0.0020	3.2s
Gemma 4 31B	100%	$0.0005	32.7s
Qwen3 235B A22B Instruct 2507	96%	$0.0004	10.3s
Qwen 3.5 Plus (2026-02-15)	94%	$0.0018	12.4s
DeepSeek V4 Flash (Reasoning)	98%	$0.0005	17.7s
Mistral Small 4 (Reasoning)	93%	$0.0014	11.4s
MiniMax M2.5	97%	$0.0014	17.9s
Z.AI GLM 5 Turbo	100%	$0.0035	9.0s
Gemini 3.1 Flash Lite (Reasoning)	89%	$0.0010	2.2s
Writer: Palmyra X5	94%	$0.0034	6.6s
GPT-5.4 Nano (Reasoning)	87%	$0.0018	7.4s
ByteDance Seed 1.6 Flash	94%	$0.0005	6.5s
MiniMax M2.7	98%	$0.0014	19.7s
Xiaomi MIMO v2.5	99%	$0.0031	12.0s
GPT-5.4	100%	$0.0061	4.2s

	Score	Cost	Speed	Stability
Gemini 3 Flash (Preview)	100%	$0.0015	2.7s	100%
Gemma 4 26B	100%	$0.0003	10.1s	100%
Mistral Medium 3.1	99%	$0.0015	3.7s	96%
Z.AI GLM 5 Turbo	100%	$0.0035	9.0s	100%
Gemini 3.5 Flash (Reasoning, Minimal)	100%	$0.0065	2.2s	100%
Gemini 2.5 Flash (Reasoning)	100%	$0.0049	7.5s	100%
GPT-5.4	100%	$0.0061	4.2s	100%
Inception Mercury 2	98%	$0.0020	3.2s	94%
Z.AI GLM 5.2 (Reasoning, High)	100%	$0.0044	11.2s	100%
Gemini 3 Flash (Preview, Reasoning)	100%	$0.0067	11.1s	100%
DeepSeek V4 Flash (Reasoning)	98%	$0.0005	17.7s	94%
GPT-5.4 (Reasoning, Low)	100%	$0.0096	6.3s	100%
Gemma 4 31B	100%	$0.0005	32.7s	100%
Qwen3 235B A22B Instruct 2507	96%	$0.0004	10.3s	90%
ByteDance Seed 1.6 Flash	94%	$0.0005	6.5s	90%
Xiaomi MIMO v2.5	99%	$0.0031	12.0s	92%
Claude Sonnet 4	100%	$0.012	4.6s	100%
Llama 3.1 Nemotron 70B	97%	$0.0030	9.7s	91%
DeepSeek V4 Pro	98%	$0.0027	11.8s	91%
Z.AI GLM 4.7	100%	$0.0047	26.8s	100%

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
100	Claude Opus 4.6 (Reasoning)	$0.034	15.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
81	Qwen3.7 Max	$0.023	30.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
102	Gemini 3.1 Pro (Preview)	$0.031	24.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
130	Qwen3.6 Max Preview	$0.034	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
84	GPT-5.5 (Reasoning)	$0.030	12.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Z.AI GLM 5.2 (Reasoning, High)	$0.0044	11.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Z.AI GLM 5 Turbo	$0.0035	9.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
58	GPT-5.5 (Reasoning, Low)	$0.025	9.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
103	Claude Opus 4.8 (Reasoning)	$0.037	9.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
44	Claude Opus 4.6	$0.021	6.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
96	Claude Opus 4.8 (Reasoning, Low)	$0.036	9.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
124	Qwen 3.5 397B A17B	$0.018	2.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
12	GPT-5.4 (Reasoning, Low)	$0.0096	6.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
54	MoonshotAI: Kimi K2.5	$0.0093	48.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
61	Claude Sonnet 5 (Reasoning)	$0.023	16.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
62	Claude Sonnet 5 (Reasoning, Low)	$0.023	15.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
95	Qwen 3.5 27B	$0.014	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Gemini 3 Flash (Preview, Reasoning)	$0.0067	11.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
92	Gemma 4 31B (Reasoning)	$0.0010	1.7m	100%	100	100	100	100	100	100	100	100	100	100	100%
43	Claude Opus 4.5	$0.021	5.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
48	DeepSeek V4 Pro (Reasoning)	$0.0054	52.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
49	Gemma 4 26B (Reasoning)	$0.0018	1.1m	100%	100	100	100	100	100	100	100	100	100	100	100%
98	Gemini 3 Pro (Preview)	$0.032	21.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
85	Qwen 3.6 27B	$0.015	57.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Z.AI GLM 4.7	$0.0047	26.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
17	Claude Sonnet 4	$0.012	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Claude Sonnet 4.5	$0.012	5.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
75	Qwen 3.5 35B	$0.014	51.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Gemini 3.5 Flash (Reasoning, Minimal)	$0.0065	2.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
101	ByteDance Seed 2.0 Mini	$0.0020	1.8m	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3 Flash (Preview)	$0.0015	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
13	Gemma 4 31B	$0.0005	32.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Gemma 4 26B	$0.0003	10.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	GPT-5.4	$0.0061	4.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
37	ByteDance Seed 2.0 Lite	$0.0044	45.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Gemini 2.5 Flash (Reasoning)	$0.0049	7.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
55	Z.AI GLM 5.1	$0.0091	42.4s	95%	100	100	100	100	100	100	100	100	100	92	99%
135	MoonshotAI: Kimi K2.6	$0.030	2.9m	95%	100	100	100	100	100	100	100	100	100	92	99%
94	Qwen 3.5 Plus (2026-04-20)	$0.011	1.1m	95%	100	100	100	100	100	100	100	100	100	92	99%
27	ByteDance Seed 1.6	$0.0034	31.9s	95%	100	100	100	100	100	100	100	100	100	92	99%
36	GPT-5.2	$0.013	12.3s	95%	100	100	100	100	100	100	100	100	100	92	99%
35	GPT-5.5	$0.015	4.7s	95%	100	100	100	100	100	100	100	100	100	92	99%
69	Gemini 2.5 Pro	$0.022	14.8s	95%	100	100	100	100	100	100	100	100	100	92	99%
126	Claude Opus 4	$0.062	8.6s	95%	100	100	100	100	100	100	100	100	100	92	99%
51	Aion 2.0	$0.0052	48.7s	95%	100	100	100	100	100	100	100	100	100	92	99%
3	Mistral Medium 3.1	$0.0015	3.7s	96%	100	100	100	100	100	100	100	100	94	94	99%
116	Qwen 3.5 9B	$0.0015	2.1m	95%	100	100	100	100	100	100	100	100	94	92	99%
87	Gemini 3.5 Flash (Reasoning)	$0.026	10.1s	92%	100	100	100	100	100	100	100	100	100	86	99%
67	Z.AI GLM 5	$0.0079	41.9s	92%	100	100	100	100	100	100	100	100	100	86	99%
16	Xiaomi MIMO v2.5	$0.0031	12.0s	92%	100	100	100	100	100	100	100	100	100	86	99%
33	MiniMax M3	$0.0020	35.9s	94%	100	100	100	100	100	100	100	100	92	92	98%
112	Qwen 3.5 122B	$0.021	1.1m	94%	100	100	100	100	100	100	100	100	92	92	98%
11	DeepSeek V4 Flash (Reasoning)	$0.0005	17.7s	94%	100	100	100	100	100	100	100	100	92	92	98%
8	Inception Mercury 2	$0.0020	3.2s	94%	100	100	100	100	100	100	100	100	92	92	98%
19	DeepSeek V4 Pro	$0.0027	11.8s	91%	100	100	100	100	100	100	100	100	92	86	98%
22	MiniMax M2.7	$0.0014	19.7s	93%	100	100	100	100	100	100	100	94	94	89	98%
91	GPT-5.4 (Reasoning)	$0.021	30.3s	93%	100	100	100	100	100	100	100	92	92	92	98%
31	GPT-5.4 Mini (Reasoning)	$0.0082	10.2s	93%	100	100	100	100	100	100	94	94	94	89	97%
52	Claude Sonnet 5	$0.011	6.7s	83%	100	100	100	100	100	100	100	100	100	72	97%
18	Llama 3.1 Nemotron 70B	$0.0030	9.7s	91%	100	100	100	100	100	94	94	94	94	92	97%
82	Grok 4.20 (Reasoning)	$0.012	42.3s	93%	100	100	100	100	100	100	92	92	92	92	97%
26	MiniMax M2.5	$0.0014	17.9s	89%	100	100	100	100	100	100	100	94	86	86	97%
14	Qwen3 235B A22B Instruct 2507	$0.0004	10.3s	90%	100	100	100	100	100	100	92	92	92	86	96%
111	Grok 4.3 (Reasoning)	$0.013	1.0m	81%	100	100	100	100	100	100	100	100	92	68	96%
108	Claude Opus 4.7 (Reasoning)	$0.035	6.4s	87%	100	100	100	100	100	100	100	86	86	86	96%
70	o4 Mini	$0.014	20.7s	89%	100	100	100	100	100	100	92	92	86	86	96%
133	GPT-5	$0.042	1.0m	86%	100	100	100	100	92	92	92	92	92	92	95%
106	GPT-5.1	$0.020	29.2s	86%	100	100	100	100	92	92	92	92	92	92	95%
88	Qwen 3.6 35B	$0.0100	40.3s	87%	100	100	100	100	100	100	92	86	86	86	95%
90	Qwen 3.5 Flash	$0.0031	55.2s	82%	100	100	100	100	100	100	100	92	86	72	95%
24	GPT-5.4 Mini (Reasoning, Low)	$0.0029	4.0s	86%	100	100	100	100	94	94	94	89	89	89	95%
110	o4 Mini High	$0.024	36.5s	86%	100	100	100	100	100	92	92	92	86	86	95%
23	Gemini 2.5 Flash Lite (Reasoning)	$0.0014	10.7s	89%	100	100	94	94	94	94	94	92	92	92	95%
32	Qwen 3.5 Plus (2026-02-15)	$0.0018	12.4s	86%	100	100	100	100	100	100	86	86	86	86	94%
28	Writer: Palmyra X5	$0.0034	6.6s	86%	100	100	100	100	100	100	86	86	86	86	94%
15	ByteDance Seed 1.6 Flash	$0.0005	6.5s	90%	100	94	94	94	94	94	94	94	92	89	94%
29	GPT-4o, Aug. 6th (temp=1)	$0.0062	2.3s	89%	100	94	94	94	94	94	94	94	89	89	94%
25	GPT-4o, Aug. 6th (temp=0)	$0.0063	2.2s	91%	94	94	94	94	94	94	94	94	94	89	94%
63	Z.AI GLM 4.5 Air	$0.0018	32.0s	82%	100	100	100	100	100	100	94	86	86	72	94%
30	Gemini 3.1 Flash Lite (Preview)	$0.0011	1.5s	80%	100	100	100	100	100	100	100	86	79	72	94%
34	Z.AI GLM 4.5	$0.0014	10.9s	84%	100	100	100	100	100	92	86	86	86	86	94%
89	GPT-OSS 120B	$0.0013	56.1s	82%	100	100	100	100	100	92	92	86	86	79	94%
38	DeepSeek V3 (2025-03-24)	$0.0008	16.3s	84%	100	100	100	94	94	94	92	86	86	86	93%
40	Qwen 3 32B	$0.0006	11.1s	80%	100	100	100	94	94	94	94	94	89	72	93%
76	GPT-5 Mini	$0.0072	38.4s	88%	100	92	92	92	92	92	92	92	92	92	93%
114	Claude Sonnet 4.6 (Reasoning)	$0.027	19.3s	80%	100	100	100	100	100	86	86	86	86	86	93%
56	Xiaomi MIMO v2.5 Pro	$0.0040	17.9s	80%	100	100	100	100	100	86	86	86	86	86	93%
42	Mistral Small 4 (Reasoning)	$0.0014	11.4s	80%	100	100	100	100	100	100	92	86	79	72	93%
99	Z.AI GLM 4.6	$0.0025	40.3s	72%	100	100	100	100	100	100	92	92	86	53	92%
46	Gemini 3.1 Flash Lite	$0.0011	1.9s	75%	100	100	100	94	94	86	86	86	86	72	91%
120	Qwen 3.6 Flash	$0.0066	20.1s	42%	100	100	100	100	100	100	100	100	99	3	90%
119	GPT-5 Nano	$0.0037	1.4m	76%	100	100	94	86	86	86	86	86	86	86	90%
80	Z.AI GLM 4.7 Flash	$0.0010	36.3s	80%	100	94	89	89	89	89	89	89	89	79	90%
65	GPT-4.1	$0.0044	5.8s	77%	100	100	86	86	86	86	86	86	86	86	89%
39	Gemini 2.5 Flash	$0.0013	1.6s	81%	94	92	92	92	92	86	86	86	86	79	89%
53	Gemini 3.1 Flash Lite (Reasoning)	$0.0010	2.2s	73%	100	100	100	94	94	94	79	79	79	66	89%
41	GPT-4.1 Mini	$0.0010	4.0s	82%	92	92	92	92	92	86	86	86	86	79	89%
59	Gemma 3 27B	$0.0003	8.1s	73%	100	100	92	86	86	86	86	86	86	72	88%
50	DeepSeek V4 Flash	$0.0002	4.8s	76%	100	92	92	92	86	86	86	86	79	79	88%
66	DeepSeek-V2 Chat	$0.0010	7.9s	73%	100	100	94	89	86	86	86	79	79	79	88%
79	DeepSeek V3.2	$0.0008	12.8s	68%	100	100	100	92	92	92	92	79	72	58	88%
109	Claude Opus 4.7	$0.027	4.3s	82%	92	92	86	86	86	86	86	86	86	86	87%
105	Nemotron 3 Super	$0.0000	1.0m	82%	94	86	86	86	86	86	86	86	86	86	87%
113	GPT-5.4 Nano (Reasoning)	$0.0018	7.4s	40%	100	100	100	100	100	94	92	92	86	0	87%
78	Claude Sonnet 4.6	$0.013	5.4s	86%	86	86	86	86	86	86	86	86	86	86	86%
47	Claude Haiku 4.5	$0.0042	3.6s	86%	86	86	86	86	86	86	86	86	86	86	86%
74	Mistral Small 3.2 24B	$0.0003	5.2s	68%	100	94	94	92	86	86	86	74	72	66	85%
64	Arcee AI: Trinity Mini	$0.0002	5.2s	74%	94	94	89	83	83	83	83	83	78	78	85%
73	Grok 4.20	$0.0027	3.4s	72%	92	92	92	86	86	86	86	86	72	66	85%
60	GPT-4o Mini (temp=1)	$0.0003	4.9s	75%	89	89	89	89	83	83	83	83	83	72	84%
115	Llama 3.1 70B	$0.0012	6.9s	41%	94	94	94	94	94	94	94	94	86	0	84%
57	Mistral Large 3	$0.0017	5.3s	79%	92	86	86	86	86	86	79	79	79	79	84%
129	Nemotron 3 Nano	$0.0016	1.8m	69%	94	94	89	86	86	86	86	79	78	58	84%
72	DeepSeek V3 (2024-12-26)	$0.0011	8.7s	73%	94	92	89	86	86	86	79	79	72	72	84%
107	Hermes 3 405B	$0.0025	12.1s	60%	100	94	89	86	86	86	86	86	79	40	83%
45	GPT-4o Mini (temp=0)	$0.0003	3.4s	83%	83	83	83	83	83	83	83	83	83	83	83%
93	DeepSeek V3.1	$0.0008	14.6s	69%	94	92	92	86	86	86	79	79	66	64	83%
97	Mistral Large 2	$0.0067	5.3s	70%	100	86	86	86	86	79	79	79	72	72	83%
83	Gemini 2.5 Flash Lite	$0.0002	1.2s	66%	94	94	89	86	79	79	79	79	72	66	82%
77	Gemma 3 12B	$0.0001	9.3s	73%	92	86	86	86	86	86	79	79	72	64	82%
71	GPT-5.4 Mini	$0.0015	1.1s	73%	89	86	83	83	83	83	83	83	79	64	82%
86	Qwen 2.5 72B	$0.0005	5.9s	68%	94	89	89	89	89	89	83	64	64	64	81%
68	Ministral 3 8B	$0.0004	3.0s	75%	86	86	86	81	81	81	81	81	76	76	81%
104	Ministral 8B	$0.0003	3.4s	62%	100	86	81	81	81	76	71	69	69	64	78%
122	WizardLM 2 8x22b	$0.0018	5.9s	38%	100	94	92	86	86	79	79	72	72	0	76%
121	Grok 4.3	$0.0034	3.4s	40%	100	94	92	86	86	79	79	72	69	3	76%
127	Hermes 3 70B	$0.0008	53.3s	54%	86	81	79	79	79	72	72	72	58	33	71%
125	Cohere Command R+ (Aug. 2024)	$0.0081	6.4s	47%	89	89	86	79	79	66	64	61	49	33	70%
118	Mistral NeMO	$0.0004	2.7s	54%	79	79	72	72	64	64	64	58	58	53	67%
117	Ministral 3 14B	$0.0005	3.3s	58%	72	72	72	72	66	66	64	61	58	53	66%
123	Cydonia 24B V4.1	$0.0006	9.3s	51%	79	72	72	72	72	72	66	64	40	33	64%
128	Mistral Small 4	$0.0005	3.4s	39%	86	72	69	66	61	52	49	44	43	33	58%
131	Ministral 3 3B	$0.0003	2.8s	37%	86	56	53	52	52	49	41	41	39	37	51%
132	Ministral 3B	$0.0001	2.4s	36%	72	66	61	60	49	49	40	38	33	33	50%
134	Gemma 3 4B	$0.0001	3.0s	38%	64	58	56	49	49	44	42	39	33	33	47%
136	GPT-5.4 Nano (Reasoning, Low)	$0.0009	3.2s	6%	86	60	56	52	28	3	3	0	0	0	29%
137	GPT-5.4 Nano	$0.0005	1.8s	1%	66	53	49	27	3	0	0	0	0	0	20%
138	GPT-4.1 Nano	$0.0002	1.9s	0%	56	33	17	17	0	0	0	0	0	0	12%
90.26%

Median	Evaluator	Top 3	Flop 3
91.7%	Accuracy (recall)	100Claude Opus 4.6 100GPT-5.5 (Reasoning, Low) 100Gemini 3 Flash (Preview, Reasoning)	2GPT-4.1 Nano 7GPT-5.4 Nano 8Gemma 3 4B
96.4%	Precision	100GPT-5.5 (Reasoning) 100Gemini 3 Flash (Preview) 100GPT-5.4 (Reasoning, Low)	10GPT-4.1 Nano 17GPT-5.4 Nano 23GPT-5.4 Nano (Reasoning, Low)
100.0%	Structural validity	100Hermes 3 405B 100Hermes 3 70B 100GPT-5.4 (Reasoning, Low)	25GPT-4.1 Nano 36GPT-5.4 Nano 47GPT-5.4 Nano (Reasoning, Low)

Codex Violation Detection

10 codex entries

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)