Small codex, long passage

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
1	Gemini 3 Flash (Preview)	$0.0022	3.6s	97%	100	100	100	100	100	100	100	100	100	95	99%
4	Claude 3.7 Sonnet	$0.017	9.8s	97%	100	100	100	100	100	100	100	96	96	96	99%
39	Gemini 3 Pro (Preview)	$0.045	29.8s	96%	100	100	100	100	100	100	100	96	96	95	99%
27	Z.AI GLM 5	$0.014	1.2m	95%	100	100	100	100	100	100	100	95	95	95	98%
26	o4 Mini	$0.023	33.7s	91%	100	100	100	100	100	100	96	95	95	86	97%
38	Z.AI GLM 4.7	$0.0096	1.6m	92%	100	100	100	100	96	96	95	95	95	95	97%
46	GPT-5.1	$0.044	1.0m	90%	100	100	100	100	95	95	95	95	95	95	97%
14	Claude Sonnet 4.5	$0.017	7.4s	90%	100	100	100	95	95	95	95	95	95	95	96%
56	Qwen 3.5 397B A17B	$0.030	3.8m	90%	100	100	100	95	95	95	95	95	95	95	96%
2	Claude Haiku 4.5	$0.0051	4.6s	96%	96	96	96	96	96	96	96	96	96	96	96%
40	Gemini 2.5 Pro	$0.039	26.5s	91%	100	100	96	95	95	95	95	95	95	95	96%
44	o4 Mini High	$0.034	52.9s	89%	100	100	100	96	96	95	95	95	91	90	96%
50	MoonshotAI: Kimi K2.5	$0.019	2.4m	88%	100	100	100	100	96	95	91	91	91	90	95%
28	Claude Opus 4.6	$0.030	10.2s	92%	100	95	95	95	95	95	95	95	95	95	95%
41	Z.AI GLM 4.6	$0.0059	1.9m	87%	100	100	100	100	100	95	95	90	86	86	95%
11	Claude Sonnet 4	$0.016	7.3s	95%	95	95	95	95	95	95	95	95	95	95	95%
25	Claude Opus 4.5	$0.030	9.1s	95%	95	95	95	95	95	95	95	95	95	95	95%
3	Qwen 3.5 Plus (2026-02-15)	$0.0025	21.6s	95%	95	95	95	95	95	95	95	95	95	95	95%
53	Claude Opus 4	$0.079	13.6s	86%	100	100	100	96	96	91	91	91	91	91	95%
13	Mistral Large	$0.0082	7.4s	87%	100	100	96	96	96	91	91	91	91	91	94%
7	Z.AI GLM 4.5	$0.0018	13.7s	85%	100	100	100	96	95	93	91	91	91	86	94%
8	Mistral Large 2	$0.0080	7.0s	88%	100	96	96	96	96	96	96	91	87	87	94%
57	GPT-5	$0.072	1.8m	87%	100	100	95	95	95	95	90	90	90	86	94%
10	Llama 3.1 70B	$0.0013	21.0s	88%	96	95	93	93	93	93	93	93	89	87	92%
17	Llama 3.1 Nemotron 70B	$0.0037	18.2s	83%	96	96	96	96	91	91	91	91	90	81	92%
37	Claude 3.5 Sonnet	$0.030	10.0s	87%	96	96	93	93	93	93	93	91	87	87	92%
5	DeepSeek V3 (2024-12-26)	$0.0012	12.3s	88%	96	93	93	93	93	93	89	89	89	89	91%
6	DeepSeek-V2 Chat	$0.0013	9.8s	88%	96	93	93	93	93	93	89	89	89	87	91%
33	GPT-5 Mini	$0.011	52.1s	86%	95	95	90	90	90	90	90	90	90	86	91%
36	GPT-5.2	$0.024	24.2s	88%	95	90	90	90	90	90	90	90	90	90	91%
29	Z.AI GLM 4.7 Flash	$0.0020	1.2m	88%	93	93	93	93	93	93	89	89	87	87	91%
15	Writer: Palmyra X5	$0.0039	8.6s	83%	93	93	93	93	93	89	89	89	89	78	90%
16	DeepSeek V3 (2025-03-24)	$0.0009	20.7s	83%	93	93	93	93	89	89	89	89	87	82	89%
9	Gemini 2.5 Flash	$0.0020	2.4s	85%	96	91	91	91	91	90	86	86	86	86	89%
12	GPT-4.1 Mini	$0.0011	4.4s	83%	93	93	93	93	93	93	87	87	87	77	89%
34	GPT-4o, May 13th (temp=0)	$0.018	3.3s	79%	93	93	93	87	87	87	87	87	87	77	87%
32	GPT-4o, May 13th (temp=1)	$0.017	3.2s	79%	93	93	89	89	89	89	87	87	85	72	87%
31	Stealth: Aurora Alpha	—	5.0s	75%	95	91	91	91	91	91	91	90	70	70	87%
18	GPT-4o, Aug. 6th (temp=0)	$0.0081	2.7s	84%	89	89	87	87	87	87	87	87	87	82	87%
20	Mistral Small Creative	$0.0005	3.8s	73%	100	95	95	86	86	86	82	82	78	74	86%
22	GPT-4o, Aug. 6th (temp=1)	$0.0078	3.0s	80%	93	93	89	87	87	87	82	82	82	82	86%
19	Qwen 2.5 72B	$0.0006	10.1s	78%	93	89	89	89	85	85	85	82	82	78	86%
24	GPT-4.1	$0.0062	7.7s	80%	91	87	87	87	87	87	87	82	81	77	85%
47	GPT-5 Nano	$0.0051	2.0m	77%	91	91	89	87	87	82	82	81	81	78	85%
23	Claude 3.5 Haiku	$0.0036	6.1s	78%	89	89	89	85	85	85	82	81	78	78	84%
30	DeepSeek V3.2	$0.0007	9.8s	71%	96	93	87	87	82	82	78	78	77	72	83%
35	Hermes 3 405B	$0.0029	19.0s	73%	93	89	87	82	82	82	81	81	77	72	83%
21	Gemini 2.5 Flash Lite	$0.0003	1.7s	75%	91	89	82	82	82	82	82	81	77	77	83%
49	DeepSeek V3.1	$0.0007	23.3s	42%	93	93	93	93	89	87	82	82	82	3	80%
42	Claude 3 Haiku	$0.0009	2.3s	62%	78	78	74	74	74	74	74	74	61	52	71%
43	GPT-4o Mini (temp=0)	$0.0004	5.4s	65%	73	68	68	68	68	68	68	68	68	68	68%
45	GPT-4o Mini (temp=1)	$0.0004	6.7s	60%	73	73	73	73	73	68	68	61	54	54	67%
51	Ministral 8B	$0.0004	6.4s	43%	82	78	76	71	69	68	68	66	41	17	63%
48	Ministral 3B	$0.0001	3.1s	51%	82	74	69	63	63	63	63	54	53	48	63%
52	Hermes 3 70B	$0.0008	18.6s	38%	82	78	74	73	72	68	67	67	47	0	63%
54	Llama 3.1 8B	$0.0001	12.1s	39%	69	68	67	67	67	63	63	54	33	3	55%
55	Mistral NeMO	$0.0005	5.6s	29%	77	68	66	66	64	59	52	44	0	0	50%
58	Cohere Command R+ (Aug. 2024)	$0.0083	5.7s	18%	81	78	77	73	72	67	0	0	0	0	45%
59	Rocinante 12B	$0.0007	6.9s	18%	72	67	56	38	33	33	33	30	0	0	36%
60	WizardLM 2 8x22b	$0.0023	12.7s	17%	33	33	33	33	23	21	17	17	3	3	22%
61	GPT-4.1 Nano	$0.0003	4.0s	15%	33	33	25	24	23	20	0	0	0	0	16%
83.69%

Codex Violation Detection

Small codex, long passage

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)