5 codex entries

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
29	Claude 3.5 Sonnet	$0.017	5.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Claude 3.7 Sonnet	$0.0087	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Hermes 3 405B	$0.0016	10.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	GPT-4.1	$0.0035	3.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	o4 Mini	$0.010	13.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
35	o4 Mini High	$0.016	29.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Claude Sonnet 4	$0.0081	4.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Gemini 2.5 Pro	$0.018	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Gemini 3 Pro (Preview)	$0.023	15.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3 Flash (Preview)	$0.0015	2.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Claude Opus 4.5	$0.014	4.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Claude Opus 4.6	$0.014	4.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Z.AI GLM 4.7	$0.0053	39.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Z.AI GLM 5	$0.0047	18.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	MoonshotAI: Kimi K2.5	$0.0078	45.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5 Mini	$0.0051	25.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	GPT-5.1	$0.014	21.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	GPT-5.2	$0.0090	9.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Z.AI GLM 4.5	$0.0013	11.9s	95%	100	100	100	100	100	100	100	100	100	91	99%
56	Qwen 3.5 397B A17B	$0.017	2.0m	95%	100	100	100	100	100	100	100	100	100	91	99%
4	DeepSeek V3 (2025-03-24)	$0.0006	9.6s	95%	100	100	100	100	100	100	100	100	93	93	99%
18	Claude Sonnet 4.5	$0.0081	4.2s	90%	100	100	100	100	100	100	100	100	100	83	98%
20	Z.AI GLM 4.6	$0.0014	23.7s	89%	100	100	100	100	100	100	100	100	91	83	97%
11	Claude 3.5 Haiku	$0.0022	5.3s	87%	100	100	100	100	100	100	100	100	83	83	97%
23	Qwen 3.5 Plus (2026-02-15)	$0.0014	12.5s	79%	100	100	100	100	100	100	100	100	100	66	97%
60	GPT-5	$0.040	1.1m	91%	100	100	100	100	100	100	91	91	91	91	96%
6	DeepSeek-V2 Chat	$0.0007	6.2s	89%	100	100	100	100	100	100	93	93	91	83	96%
21	Llama 3.1 Nemotron 70B	$0.0020	8.2s	79%	100	100	100	100	100	100	100	100	91	66	96%
12	DeepSeek V3 (2024-12-26)	$0.0007	7.3s	85%	100	100	100	100	100	100	93	93	93	76	96%
8	DeepSeek V3.2	$0.0004	7.6s	87%	100	100	100	100	100	91	91	91	91	91	96%
14	Llama 3.1 70B	$0.0008	10.8s	87%	100	100	100	100	100	100	93	90	83	83	95%
26	Stealth: Aurora Alpha	—	2.7s	77%	100	100	100	100	100	93	93	93	93	66	94%
22	Mistral Large	$0.0046	4.7s	84%	100	100	100	91	91	91	91	91	91	91	94%
45	Z.AI GLM 4.7 Flash	$0.0010	52.6s	80%	100	100	100	100	100	93	93	83	83	75	93%
13	Gemini 2.5 Flash	$0.0009	1.2s	84%	100	93	93	93	93	93	93	93	83	83	92%
55	Claude Opus 4	$0.040	6.7s	76%	100	100	100	100	100	83	83	83	83	83	92%
17	Mistral Large 2	$0.0047	5.1s	91%	91	91	91	91	91	91	91	91	91	91	91%
38	GPT-4o, May 13th (temp=0)	$0.011	2.0s	72%	100	100	100	100	91	91	83	76	76	76	89%
47	DeepSeek V3.1	$0.0005	14.7s	43%	100	100	100	100	100	100	91	84	83	3	86%
27	Qwen 2.5 72B	$0.0002	5.8s	77%	93	93	93	93	93	93	75	75	75	75	86%
32	GPT-4.1 Mini	$0.0010	3.2s	69%	93	93	83	83	83	83	83	83	83	60	83%
57	GPT-5 Nano	$0.0033	1.3m	41%	100	100	100	93	93	93	93	75	75	3	83%
37	GPT-4o Mini (temp=1)	$0.0003	5.4s	59%	93	93	93	93	75	75	75	75	67	67	81%
46	GPT-4o, May 13th (temp=1)	$0.011	2.7s	66%	91	91	83	83	76	76	76	76	76	76	80%
42	GPT-4o, Aug. 6th (temp=0)	$0.0054	2.2s	65%	91	91	91	76	76	76	76	76	76	76	80%
41	GPT-4o, Aug. 6th (temp=1)	$0.0052	2.4s	66%	91	91	83	76	76	76	76	76	76	76	79%
34	GPT-4o Mini (temp=0)	$0.0003	5.0s	64%	93	93	75	75	75	75	75	75	75	75	79%
44	Writer: Palmyra X5	$0.0028	7.2s	64%	83	83	83	83	83	83	76	76	76	44	77%
40	Claude Haiku 4.5	$0.0029	3.1s	65%	83	83	83	83	76	76	75	67	67	67	76%
36	Mistral Small Creative	$0.0002	2.1s	67%	76	76	76	76	76	76	69	69	63	60	71%
51	Cohere Command R+ (Aug. 2024)	$0.0057	5.8s	49%	93	87	87	76	75	67	57	57	57	44	70%
43	Gemini 2.5 Flash Lite	$0.0002	947ms	62%	75	75	67	67	67	67	67	67	67	67	68%
48	Mistral NeMO	$0.0002	2.0s	51%	83	75	75	75	67	67	60	60	50	44	66%
50	Llama 3.1 8B	$0.0001	4.6s	45%	83	83	78	78	63	60	60	57	54	39	65%
49	Claude 3 Haiku	$0.0007	3.3s	52%	76	76	67	67	67	60	57	57	50	50	62%
52	Ministral 3B	$0.0001	1.2s	40%	83	78	76	67	57	54	50	50	44	42	60%
54	Hermes 3 70B	$0.0005	13.2s	34%	76	75	75	66	60	57	57	50	44	0	56%
53	Ministral 8B	$0.0002	3.6s	37%	69	49	49	44	44	43	43	43	43	42	47%
59	Rocinante 12B	$0.0004	5.1s	21%	67	57	53	40	38	37	33	3	3	3	34%
58	GPT-4.1 Nano	$0.0001	951ms	20%	57	57	57	57	44	33	17	0	0	0	32%
61	WizardLM 2 8x22b	$0.0016	14.5s	3%	3	3	3	3	3	3	0	0	0	0	2%
85.72%

Codex Violation Detection

5 codex entries

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)