40 codex entries

Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
10	Claude Opus 4.6	$0.050	8.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
6	Gemini 2.5 Pro	$0.027	16.8s	95%	100	100	100	100	100	100	100	100	100	92	99%
3	Claude Sonnet 4.5	$0.030	7.3s	95%	100	100	100	100	100	100	100	100	100	92	99%
28	GPT-5 Mini	$0.0075	1.3m	96%	100	100	100	100	100	100	100	100	95	95	99%
23	MoonshotAI: Kimi K2.5	$0.012	54.2s	95%	100	100	100	100	100	100	100	100	95	92	99%
16	Claude Opus 4.5	$0.050	8.2s	95%	100	100	100	100	100	97	97	97	97	97	98%
26	o4 Mini High	$0.031	45.4s	94%	100	100	100	100	100	97	97	97	95	95	98%
49	Qwen 3.5 397B A17B	$0.023	2.2m	93%	100	100	100	100	100	100	100	95	92	92	98%
9	o4 Mini	$0.016	25.1s	93%	100	100	100	100	100	97	97	97	97	92	98%
7	Z.AI GLM 4.7	$0.0070	28.9s	92%	100	100	100	100	100	100	100	97	92	88	98%
29	GPT-5.1	$0.034	49.6s	92%	100	100	100	100	97	97	97	97	95	93	98%
45	GPT-5	$0.056	1.4m	93%	100	100	100	100	100	100	97	95	92	92	98%
24	Gemini 3 Pro (Preview)	$0.045	22.7s	92%	100	100	100	100	100	100	100	92	92	92	98%
11	Z.AI GLM 5	$0.0082	33.0s	92%	100	100	100	100	100	97	97	97	92	92	97%
15	GPT-5.2	$0.023	22.0s	90%	100	100	100	95	95	95	95	95	95	91	96%
5	Claude Sonnet 4	$0.029	7.4s	94%	97	97	97	97	97	97	97	93	93	93	96%
1	Gemini 2.5 Flash	$0.0027	3.3s	90%	100	100	97	97	97	97	95	95	91	87	96%
2	Mistral Large 2	$0.016	6.3s	92%	100	97	97	97	97	97	93	93	93	92	96%
12	Claude 3.7 Sonnet	$0.030	8.2s	90%	97	97	97	97	97	97	97	97	88	88	95%
14	GPT-4o, May 13th (temp=0)	$0.038	3.4s	91%	93	93	93	93	93	93	93	93	90	90	93%
8	Mistral Large	$0.016	6.2s	86%	97	97	97	97	97	97	93	84	84	84	93%
13	Stealth: Aurora Alpha	—	5.8s	83%	97	97	97	97	97	93	92	88	88	75	92%
52	Claude Opus 4	$0.144	11.2s	82%	100	100	97	92	92	92	92	88	84	83	92%
17	Qwen 3.5 Plus (2026-02-15)	$0.0041	20.6s	82%	100	100	92	92	92	92	92	87	83	83	91%
4	Gemini 3 Flash (Preview)	$0.0032	4.1s	83%	100	92	92	92	92	92	92	83	83	83	90%
37	Z.AI GLM 4.6	$0.0035	46.7s	69%	100	100	100	100	92	92	91	88	75	61	90%
40	Z.AI GLM 4.7 Flash	$0.0017	1.0m	79%	97	97	93	92	92	90	90	84	80	76	89%
36	Claude 3.5 Sonnet	$0.056	9.4s	81%	97	93	88	88	88	88	88	84	84	84	88%
18	DeepSeek V3.2	$0.0019	17.7s	79%	95	93	93	92	92	88	84	80	79	79	88%
21	DeepSeek-V2 Chat	$0.0027	10.9s	73%	97	97	93	92	92	92	90	75	70	70	87%
48	GPT-5 Nano	$0.0044	1.6m	74%	97	95	93	87	84	84	84	84	83	75	87%
19	GPT-4o, Aug. 6th (temp=1)	$0.013	3.1s	76%	93	93	93	93	90	88	84	80	80	70	87%
20	GPT-4o, Aug. 6th (temp=0)	$0.012	2.8s	76%	93	93	90	90	90	84	84	84	80	70	86%
25	DeepSeek V3 (2024-12-26)	$0.0028	13.9s	72%	97	92	92	84	84	83	80	79	76	75	84%
35	GPT-4o, May 13th (temp=1)	$0.038	3.7s	74%	90	90	88	87	84	80	80	80	76	75	83%
27	DeepSeek V3 (2025-03-24)	$0.0020	9.9s	69%	93	92	88	87	83	79	75	75	74	71	82%
22	GPT-4.1 Mini	$0.0015	4.4s	73%	90	84	84	84	80	80	79	79	76	75	81%
38	DeepSeek V3.1	$0.0018	30.1s	65%	97	92	87	84	83	83	83	79	62	62	81%
32	GPT-4.1	$0.0092	9.9s	65%	100	88	83	83	79	79	79	79	75	62	81%
34	Writer: Palmyra X5	$0.0072	17.3s	68%	90	88	88	84	79	79	79	76	75	67	81%
42	Claude Haiku 4.5	$0.010	5.0s	42%	97	97	92	88	88	88	88	83	79	3	80%
30	Claude 3.5 Haiku	$0.0070	5.8s	64%	87	83	83	83	83	83	83	83	80	44	79%
33	Hermes 3 405B	$0.0067	12.8s	69%	83	83	83	83	80	75	72	71	70	70	77%
39	Z.AI GLM 4.5	$0.0025	12.4s	58%	93	88	88	88	80	79	76	75	53	50	77%
31	Mistral Small Creative	$0.0008	3.6s	63%	84	83	83	79	79	79	75	67	61	52	74%
41	Qwen 2.5 72B	$0.0011	8.6s	59%	83	80	76	76	72	72	66	61	61	53	70%
47	Llama 3.1 Nemotron 70B	$0.0085	20.0s	48%	84	83	75	75	66	66	62	53	49	40	65%
43	Ministral 3B	$0.0003	1.3s	48%	83	80	72	66	60	60	60	55	55	55	65%
44	WizardLM 2 8x22b	$0.0051	8.0s	54%	76	75	75	66	66	66	62	55	53	51	64%
46	Hermes 3 70B	$0.0020	23.5s	50%	80	70	67	62	61	61	60	53	52	51	62%
54	Llama 3.1 70B	$0.0031	23.7s	35%	87	71	67	67	63	62	57	53	50	0	58%
50	GPT-4o Mini (temp=0)	$0.0008	8.7s	42%	66	61	57	57	48	48	48	48	48	42	52%
55	Cohere Command R+ (Aug. 2024)	$0.020	10.4s	42%	66	66	60	60	60	48	44	41	40	33	52%
51	GPT-4o Mini (temp=1)	$0.0007	7.8s	41%	60	55	55	48	48	48	48	44	40	33	48%
56	Gemini 2.5 Flash Lite	$0.0008	3.3s	28%	66	65	59	58	56	54	49	18	0	0	43%
53	Claude 3 Haiku	$0.0021	3.2s	40%	44	44	44	42	42	42	40	40	40	39	42%
57	Mistral NeMO	$0.0011	6.2s	24%	72	48	41	38	37	36	35	33	33	0	37%
59	Ministral 8B	$0.0007	2.2s	10%	75	75	71	67	67	0	0	0	0	0	35%
60	Llama 3.1 8B	$0.0002	36.7s	23%	60	57	44	42	38	33	32	25	3	3	34%
58	Rocinante 12B	$0.0015	5.4s	23%	44	42	38	33	33	33	33	17	3	0	28%
61	GPT-4.1 Nano	$0.0003	2.2s	14%	33	18	17	17	17	17	17	17	0	0	15%
79.26%

Codex Violation Detection

40 codex entries

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)