Codex Violation Detection

Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.

Model	Total ▼	Small codex, short passage	Large codex, short passage	Small codex, long passage	Large codex, long passage	5 codex entries	10 codex entries	20 codex entries	40 codex entries
Claude Opus 4.5	99%	100%	100%	95%	95%	100%	100%	100%	98%
Claude Opus 4.6	98%	100%	98%	95%	94%	100%	100%	100%	100%
Gemini 2.5 Pro	98%	99%	100%	96%	95%	100%	99%	97%	99%
Claude Sonnet 4.5	98%	100%	100%	96%	90%	98%	100%	97%	99%
MoonshotAI: Kimi K2.5	97%	99%	98%	95%	91%	100%	100%	93%	99%
Z.AI GLM 5	97%	100%	95%	98%	91%	100%	99%	93%	97%
Z.AI GLM 4.7	97%	100%	94%	97%	93%	100%	100%	91%	98%
Gemini 3 Pro (Preview)	97%	99%	93%	99%	89%	100%	100%	95%	98%
Claude Sonnet 4	96%	100%	97%	95%	93%	100%	100%	92%	96%
GPT-5.1	96%	100%	94%	97%	91%	100%	95%	96%	98%
o4 Mini High	96%	98%	94%	96%	93%	100%	95%	93%	98%
GPT-5.2	96%	100%	95%	91%	92%	100%	99%	95%	96%
GPT-5	96%	100%	96%	94%	92%	96%	95%	97%	98%
Gemini 3 Flash (Preview)	96%	100%	93%	99%	92%	100%	100%	92%	90%
o4 Mini	96%	98%	93%	97%	92%	100%	96%	92%	98%

1–15 of 61

Page 1 / 5

Rows:

Cost vs Performance

Compares total cost for this test against the test score. Quadrant lines are drawn at the median values. Only models with available cost data are shown.

Small codex, short passage

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
15	Claude 3.7 Sonnet	$0.015	8.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
11	Claude Sonnet 4	$0.013	6.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3 Flash (Preview)	$0.0025	3.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
12	Claude Sonnet 4.5	$0.013	6.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Claude Opus 4.5	$0.023	7.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Claude Opus 4.6	$0.024	7.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
14	Z.AI GLM 5	$0.0062	29.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5 Mini	$0.0063	29.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
53	GPT-5	$0.039	53.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
34	GPT-5.1	$0.019	24.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
23	GPT-5.2	$0.015	13.9s	100%	100	100	100	100	100	100	100	100	100	100	100%
21	Z.AI GLM 4.7	$0.0061	29.6s	98%	100	100	100	100	100	100	100	100	100	97	100%
60	Qwen 3.5 397B A17B	$0.023	2.8m	98%	100	100	100	100	100	100	100	100	100	97	100%
3	DeepSeek V3 (2024-12-26)	$0.0011	14.8s	98%	100	100	100	100	100	100	100	100	100	97	100%
6	GPT-4.1	$0.0065	7.7s	97%	100	100	100	100	100	100	100	100	100	95	100%
5	GPT-4o, Aug. 6th (temp=0)	$0.0080	3.4s	97%	100	100	100	100	100	100	100	100	97	97	99%
18	DeepSeek V3 (2025-03-24)	$0.0007	36.3s	96%	100	100	100	100	100	100	100	100	100	93	99%
44	Gemini 2.5 Pro	$0.031	21.3s	97%	100	100	100	100	100	100	100	100	97	95	99%
37	MoonshotAI: Kimi K2.5	$0.010	46.2s	97%	100	100	100	100	100	100	100	100	97	95	99%
54	Gemini 3 Pro (Preview)	$0.042	49.1s	97%	100	100	100	100	100	100	100	97	97	97	99%
38	Z.AI GLM 4.6	$0.0037	1.0m	96%	100	100	100	100	100	100	97	97	97	95	99%
4	DeepSeek-V2 Chat	$0.0011	12.7s	95%	100	100	100	100	100	100	100	97	97	92	99%
50	o4 Mini High	$0.028	46.6s	95%	100	100	100	100	100	97	97	97	97	97	98%
7	Z.AI GLM 4.5	$0.0019	15.2s	95%	100	100	100	100	100	97	97	97	97	97	98%
42	Claude 3.5 Sonnet	$0.027	10.5s	93%	100	100	100	100	100	100	100	100	92	92	98%
27	GPT-4o, May 13th (temp=0)	$0.016	3.7s	93%	100	100	100	100	100	100	100	100	92	92	98%
8	DeepSeek V3.2	$0.0006	16.2s	94%	100	100	100	100	100	100	100	97	93	92	98%
33	o4 Mini	$0.015	20.9s	94%	100	100	100	97	97	97	97	97	97	97	98%
9	Llama 3.1 70B	$0.0009	14.9s	95%	100	97	97	97	97	97	97	97	97	97	97%
10	Llama 3.1 Nemotron 70B	$0.0027	15.3s	95%	100	97	97	97	97	97	97	97	97	97	97%
20	Qwen 3.5 Plus (2026-02-15)	$0.0022	21.4s	91%	100	100	100	95	95	95	95	95	95	95	97%
2	Gemini 2.5 Flash	$0.0017	2.1s	97%	97	97	97	97	97	97	97	97	97	97	97%
22	GPT-4o, Aug. 6th (temp=1)	$0.0081	3.7s	89%	100	100	100	100	97	97	92	92	92	92	96%
24	DeepSeek V3.1	$0.0009	23.3s	89%	100	100	100	100	97	97	92	92	92	92	96%
26	Hermes 3 405B	$0.0022	21.4s	89%	100	100	100	97	97	97	97	93	92	87	96%
13	Mistral Large 2	$0.0065	7.1s	93%	97	97	97	97	97	97	97	97	92	92	96%
19	Mistral Large	$0.0066	7.4s	91%	100	97	97	97	97	97	97	92	92	92	96%
59	Claude Opus 4	$0.067	12.3s	84%	100	100	100	100	92	92	92	92	92	92	95%
41	GPT-4o, May 13th (temp=1)	$0.015	3.7s	81%	100	100	100	97	92	92	92	92	84	84	93%
17	GPT-4.1 Mini	$0.0014	5.1s	87%	97	97	97	97	93	93	92	88	88	88	93%
52	GPT-5 Nano	$0.0036	1.6m	89%	97	97	93	93	93	93	90	90	90	90	93%
32	Qwen 2.5 72B	$0.0005	13.6s	77%	97	97	97	93	93	93	93	93	90	66	91%
46	Z.AI GLM 4.7 Flash	$0.0011	46.8s	78%	100	97	97	93	88	88	88	87	84	80	90%
40	Cohere Command R+ (Aug. 2024)	$0.0082	9.5s	80%	97	97	93	90	90	90	87	84	80	79	89%
35	Writer: Palmyra X5	$0.0041	10.6s	82%	92	92	92	92	88	88	84	84	84	83	88%
25	Gemini 2.5 Flash Lite	$0.0004	1.6s	86%	88	88	88	88	88	88	88	88	88	83	88%
56	Stealth: Aurora Alpha	—	5.8s	41%	97	97	97	97	97	97	97	97	97	0	87%
36	Claude Haiku 4.5	$0.0046	4.5s	81%	88	88	88	88	88	88	88	79	79	79	85%
29	GPT-4o Mini (temp=0)	$0.0004	7.3s	84%	87	87	87	87	87	87	83	83	83	83	85%
31	Mistral Small Creative	$0.0003	3.3s	80%	87	87	83	83	83	83	83	83	83	79	84%
39	GPT-4o Mini (temp=1)	$0.0004	6.7s	76%	87	87	87	83	83	83	83	83	83	72	83%
47	Claude 3.5 Haiku	$0.0030	6.3s	69%	84	84	84	80	80	80	80	80	70	61	78%
43	Mistral NeMO	$0.0003	2.9s	72%	84	80	80	76	76	76	76	76	76	76	78%
48	Ministral 8B	$0.0002	3.3s	64%	88	88	84	79	79	76	75	75	63	60	77%
49	Llama 3.1 8B	$0.0001	7.0s	63%	88	87	84	83	80	79	70	66	62	57	76%
55	Hermes 3 70B	$0.0006	15.0s	41%	93	88	87	87	87	84	83	76	70	0	76%
45	Ministral 3B	$0.0001	1.9s	68%	84	80	79	76	75	75	75	71	70	66	75%
51	Claude 3 Haiku	$0.0009	3.0s	62%	76	76	76	72	72	72	72	70	60	55	70%
57	Rocinante 12B	$0.0004	4.9s	44%	80	80	72	61	61	55	52	52	52	42	60%
58	GPT-4.1 Nano	$0.0002	2.0s	35%	73	70	65	61	53	53	53	37	28	17	51%
61	WizardLM 2 8x22b	$0.0019	16.8s	2%	70	53	22	17	3	3	3	3	0	0	18%
90.97%

Large codex, short passage

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
2	Claude Sonnet 4.5	$0.036	11.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Claude Opus 4.5	$0.064	13.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Gemini 2.5 Pro	$0.038	23.8s	97%	100	100	100	100	100	100	100	100	100	96	100%
14	Claude Opus 4.6	$0.065	15.3s	95%	100	100	100	100	97	97	97	97	97	96	98%
42	MoonshotAI: Kimi K2.5	$0.017	2.0m	94%	100	100	100	98	97	97	96	96	96	95	98%
7	Claude Sonnet 4	$0.036	13.3s	90%	100	100	100	100	96	96	96	96	94	91	97%
49	GPT-5	$0.075	1.9m	93%	98	98	96	96	96	96	96	94	93	92	96%
27	Z.AI GLM 5	$0.017	1.1m	87%	100	100	98	96	93	93	93	93	92	89	95%
17	GPT-5.2	$0.036	35.1s	91%	97	97	96	95	95	95	93	93	93	91	95%
33	o4 Mini High	$0.040	1.0m	90%	98	98	95	95	94	94	94	94	92	92	94%
53	Qwen 3.5 397B A17B	$0.027	2.8m	92%	96	96	96	96	96	96	96	91	91	91	94%
28	Z.AI GLM 4.7	$0.012	1.2m	89%	100	96	96	96	95	95	94	93	91	87	94%
40	GPT-5.1	$0.050	1.2m	91%	96	96	96	96	96	93	92	92	92	91	94%
4	GPT-4.1	$0.011	15.1s	86%	100	100	98	96	96	95	91	90	88	86	94%
38	Gemini 3 Pro (Preview)	$0.068	39.7s	87%	100	98	96	96	96	96	89	89	87	87	93%
19	o4 Mini	$0.023	41.0s	88%	98	96	96	94	94	94	92	92	89	87	93%
1	Gemini 3 Flash (Preview)	$0.0039	6.2s	88%	96	96	96	96	96	91	91	91	91	87	93%
36	Z.AI GLM 4.6	$0.0054	1.5m	86%	97	96	96	96	94	93	93	91	87	85	93%
6	Stealth: Aurora Alpha	—	7.5s	83%	96	96	95	90	90	90	90	90	90	83	91%
30	GPT-5 Mini	$0.010	1.1m	83%	96	94	94	94	94	93	91	89	83	78	91%
55	Claude Opus 4	$0.181	21.6s	81%	98	96	94	91	89	89	89	89	85	82	90%
3	Gemini 2.5 Flash	$0.0038	4.0s	83%	96	96	94	93	93	89	89	85	85	82	90%
12	GPT-4o, May 13th (temp=0)	$0.043	5.5s	86%	93	93	91	91	91	89	89	89	88	84	90%
15	Qwen 3.5 Plus (2026-02-15)	$0.0054	38.7s	83%	96	96	91	91	91	91	89	89	82	80	90%
10	Z.AI GLM 4.5	$0.0043	29.1s	84%	94	94	92	92	92	90	89	87	83	83	90%
9	Writer: Palmyra X5	$0.010	16.4s	81%	96	93	91	91	89	89	88	85	83	82	89%
29	Z.AI GLM 4.7 Flash	$0.0022	1.0m	81%	93	91	91	89	88	88	88	85	84	78	88%
35	Claude 3.5 Sonnet	$0.067	15.2s	80%	92	92	92	92	87	87	87	83	83	81	88%
23	Claude 3.7 Sonnet	$0.038	15.8s	84%	90	87	87	87	87	87	85	85	85	85	87%
13	DeepSeek-V2 Chat	$0.0033	20.2s	76%	96	91	90	90	85	85	83	83	81	78	86%
16	DeepSeek V3 (2024-12-26)	$0.0031	26.1s	77%	95	93	92	85	85	85	83	83	81	80	86%
18	Mistral Large	$0.020	14.0s	78%	94	91	89	85	85	85	83	83	81	81	86%
20	Mistral Large 2	$0.020	13.7s	77%	92	89	89	85	83	83	83	83	83	80	85%
24	DeepSeek V3.2	$0.0020	19.2s	68%	98	92	91	87	87	85	85	81	80	56	84%
11	GPT-4o, Aug. 6th (temp=0)	$0.016	4.2s	77%	91	84	84	82	82	82	81	81	81	79	83%
37	GPT-4o, May 13th (temp=1)	$0.042	5.3s	67%	91	89	88	79	79	79	78	74	70	66	79%
21	GPT-4.1 Mini	$0.0025	11.0s	71%	85	85	85	82	80	80	80	78	72	66	79%
44	DeepSeek V3 (2025-03-24)	$0.0025	31.1s	41%	96	96	87	87	85	83	83	78	72	3	77%
43	GPT-4o, Aug. 6th (temp=1)	$0.016	4.5s	42%	89	89	88	86	84	82	82	81	78	3	76%
22	Mistral Small Creative	$0.0009	4.0s	68%	88	80	79	79	77	77	74	71	71	64	76%
25	Gemini 2.5 Flash Lite	$0.0008	3.2s	66%	85	82	78	76	76	74	71	68	68	66	74%
26	Claude 3.5 Haiku	$0.0076	7.8s	70%	79	76	74	74	74	74	74	74	70	70	74%
39	Hermes 3 405B	$0.0070	24.4s	64%	82	80	78	76	75	74	69	66	66	60	73%
60	GPT-5 Nano	$0.0063	2.6m	27%	92	92	90	89	88	87	86	84	3	0	71%
45	Claude Haiku 4.5	$0.013	7.9s	28%	94	89	89	89	85	85	83	81	3	3	70%
31	Ministral 8B	$0.0008	6.3s	63%	76	74	71	69	69	66	66	66	66	64	69%
32	GPT-4o Mini (temp=0)	$0.0008	11.1s	64%	70	70	70	70	70	70	70	70	70	57	68%
34	Ministral 3B	$0.0003	2.2s	55%	80	79	78	76	71	69	67	60	52	50	68%
51	DeepSeek V3.1	$0.0024	37.1s	28%	92	92	89	89	85	85	76	69	3	3	68%
41	Qwen 2.5 72B	$0.0011	13.4s	53%	81	76	74	72	70	69	64	53	47	44	65%
54	Llama 3.1 Nemotron 70B	$0.0091	34.0s	37%	76	72	70	67	61	61	58	57	55	3	58%
50	Cohere Command R+ (Aug. 2024)	$0.021	11.9s	38%	74	70	69	68	66	63	52	49	44	0	56%
48	Llama 3.1 70B	$0.0035	13.2s	34%	76	74	70	67	58	57	54	52	47	0	56%
56	Hermes 3 70B	$0.0021	57.3s	35%	75	69	69	67	61	57	54	50	46	0	55%
47	GPT-4o Mini (temp=1)	$0.0009	9.8s	37%	69	64	60	57	57	57	54	49	43	3	51%
46	Claude 3 Haiku	$0.0025	5.2s	40%	67	60	54	51	49	49	49	47	39	33	50%
52	Mistral NeMO	$0.0011	7.4s	35%	58	54	44	43	41	39	39	38	38	36	43%
57	Llama 3.1 8B	$0.0002	19.9s	25%	48	44	38	36	36	35	33	33	3	3	31%
58	Rocinante 12B	$0.0014	6.6s	22%	51	44	35	33	33	33	33	3	0	0	27%
59	GPT-4.1 Nano	$0.0006	6.9s	12%	51	22	20	19	17	17	13	3	3	0	17%
61	WizardLM 2 8x22b	$0.0060	28.7s	0%	3	3	3	3	0	0	0	0	0	0	1%
77.81%

Small codex, long passage

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
1	Gemini 3 Flash (Preview)	$0.0022	3.6s	97%	100	100	100	100	100	100	100	100	100	95	99%
4	Claude 3.7 Sonnet	$0.017	9.8s	97%	100	100	100	100	100	100	100	96	96	96	99%
39	Gemini 3 Pro (Preview)	$0.045	29.8s	96%	100	100	100	100	100	100	100	96	96	95	99%
27	Z.AI GLM 5	$0.014	1.2m	95%	100	100	100	100	100	100	100	95	95	95	98%
26	o4 Mini	$0.023	33.7s	91%	100	100	100	100	100	100	96	95	95	86	97%
38	Z.AI GLM 4.7	$0.0096	1.6m	92%	100	100	100	100	96	96	95	95	95	95	97%
46	GPT-5.1	$0.044	1.0m	90%	100	100	100	100	95	95	95	95	95	95	97%
14	Claude Sonnet 4.5	$0.017	7.4s	90%	100	100	100	95	95	95	95	95	95	95	96%
56	Qwen 3.5 397B A17B	$0.030	3.8m	90%	100	100	100	95	95	95	95	95	95	95	96%
2	Claude Haiku 4.5	$0.0051	4.6s	96%	96	96	96	96	96	96	96	96	96	96	96%
40	Gemini 2.5 Pro	$0.039	26.5s	91%	100	100	96	95	95	95	95	95	95	95	96%
44	o4 Mini High	$0.034	52.9s	89%	100	100	100	96	96	95	95	95	91	90	96%
50	MoonshotAI: Kimi K2.5	$0.019	2.4m	88%	100	100	100	100	96	95	91	91	91	90	95%
28	Claude Opus 4.6	$0.030	10.2s	92%	100	95	95	95	95	95	95	95	95	95	95%
41	Z.AI GLM 4.6	$0.0059	1.9m	87%	100	100	100	100	100	95	95	90	86	86	95%
11	Claude Sonnet 4	$0.016	7.3s	95%	95	95	95	95	95	95	95	95	95	95	95%
25	Claude Opus 4.5	$0.030	9.1s	95%	95	95	95	95	95	95	95	95	95	95	95%
3	Qwen 3.5 Plus (2026-02-15)	$0.0025	21.6s	95%	95	95	95	95	95	95	95	95	95	95	95%
53	Claude Opus 4	$0.079	13.6s	86%	100	100	100	96	96	91	91	91	91	91	95%
13	Mistral Large	$0.0082	7.4s	87%	100	100	96	96	96	91	91	91	91	91	94%
7	Z.AI GLM 4.5	$0.0018	13.7s	85%	100	100	100	96	95	93	91	91	91	86	94%
8	Mistral Large 2	$0.0080	7.0s	88%	100	96	96	96	96	96	96	91	87	87	94%
57	GPT-5	$0.072	1.8m	87%	100	100	95	95	95	95	90	90	90	86	94%
10	Llama 3.1 70B	$0.0013	21.0s	88%	96	95	93	93	93	93	93	93	89	87	92%
17	Llama 3.1 Nemotron 70B	$0.0037	18.2s	83%	96	96	96	96	91	91	91	91	90	81	92%
37	Claude 3.5 Sonnet	$0.030	10.0s	87%	96	96	93	93	93	93	93	91	87	87	92%
5	DeepSeek V3 (2024-12-26)	$0.0012	12.3s	88%	96	93	93	93	93	93	89	89	89	89	91%
6	DeepSeek-V2 Chat	$0.0013	9.8s	88%	96	93	93	93	93	93	89	89	89	87	91%
33	GPT-5 Mini	$0.011	52.1s	86%	95	95	90	90	90	90	90	90	90	86	91%
36	GPT-5.2	$0.024	24.2s	88%	95	90	90	90	90	90	90	90	90	90	91%
29	Z.AI GLM 4.7 Flash	$0.0020	1.2m	88%	93	93	93	93	93	93	89	89	87	87	91%
15	Writer: Palmyra X5	$0.0039	8.6s	83%	93	93	93	93	93	89	89	89	89	78	90%
16	DeepSeek V3 (2025-03-24)	$0.0009	20.7s	83%	93	93	93	93	89	89	89	89	87	82	89%
9	Gemini 2.5 Flash	$0.0020	2.4s	85%	96	91	91	91	91	90	86	86	86	86	89%
12	GPT-4.1 Mini	$0.0011	4.4s	83%	93	93	93	93	93	93	87	87	87	77	89%
34	GPT-4o, May 13th (temp=0)	$0.018	3.3s	79%	93	93	93	87	87	87	87	87	87	77	87%
32	GPT-4o, May 13th (temp=1)	$0.017	3.2s	79%	93	93	89	89	89	89	87	87	85	72	87%
31	Stealth: Aurora Alpha	—	5.0s	75%	95	91	91	91	91	91	91	90	70	70	87%
18	GPT-4o, Aug. 6th (temp=0)	$0.0081	2.7s	84%	89	89	87	87	87	87	87	87	87	82	87%
20	Mistral Small Creative	$0.0005	3.8s	73%	100	95	95	86	86	86	82	82	78	74	86%
22	GPT-4o, Aug. 6th (temp=1)	$0.0078	3.0s	80%	93	93	89	87	87	87	82	82	82	82	86%
19	Qwen 2.5 72B	$0.0006	10.1s	78%	93	89	89	89	85	85	85	82	82	78	86%
24	GPT-4.1	$0.0062	7.7s	80%	91	87	87	87	87	87	87	82	81	77	85%
47	GPT-5 Nano	$0.0051	2.0m	77%	91	91	89	87	87	82	82	81	81	78	85%
23	Claude 3.5 Haiku	$0.0036	6.1s	78%	89	89	89	85	85	85	82	81	78	78	84%
30	DeepSeek V3.2	$0.0007	9.8s	71%	96	93	87	87	82	82	78	78	77	72	83%
35	Hermes 3 405B	$0.0029	19.0s	73%	93	89	87	82	82	82	81	81	77	72	83%
21	Gemini 2.5 Flash Lite	$0.0003	1.7s	75%	91	89	82	82	82	82	82	81	77	77	83%
49	DeepSeek V3.1	$0.0007	23.3s	42%	93	93	93	93	89	87	82	82	82	3	80%
42	Claude 3 Haiku	$0.0009	2.3s	62%	78	78	74	74	74	74	74	74	61	52	71%
43	GPT-4o Mini (temp=0)	$0.0004	5.4s	65%	73	68	68	68	68	68	68	68	68	68	68%
45	GPT-4o Mini (temp=1)	$0.0004	6.7s	60%	73	73	73	73	73	68	68	61	54	54	67%
51	Ministral 8B	$0.0004	6.4s	43%	82	78	76	71	69	68	68	66	41	17	63%
48	Ministral 3B	$0.0001	3.1s	51%	82	74	69	63	63	63	63	54	53	48	63%
52	Hermes 3 70B	$0.0008	18.6s	38%	82	78	74	73	72	68	67	67	47	0	63%
54	Llama 3.1 8B	$0.0001	12.1s	39%	69	68	67	67	67	63	63	54	33	3	55%
55	Mistral NeMO	$0.0005	5.6s	29%	77	68	66	66	64	59	52	44	0	0	50%
58	Cohere Command R+ (Aug. 2024)	$0.0083	5.7s	18%	81	78	77	73	72	67	0	0	0	0	45%
59	Rocinante 12B	$0.0007	6.9s	18%	72	67	56	38	33	33	33	30	0	0	36%
60	WizardLM 2 8x22b	$0.0023	12.7s	17%	33	33	33	33	23	21	17	17	3	3	22%
61	GPT-4.1 Nano	$0.0003	4.0s	15%	33	33	25	24	23	20	0	0	0	0	16%
83.69%

Large codex, long passage

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
7	Claude Opus 4.5	$0.085	20.6s	92%	98	97	96	96	96	94	94	94	93	93	95%
13	Gemini 2.5 Pro	$0.073	49.3s	91%	99	97	96	96	95	95	94	94	94	90	95%
24	Z.AI GLM 4.6	$0.012	3.0m	91%	96	96	94	94	94	94	94	94	91	90	94%
11	Claude Opus 4.6	$0.082	21.5s	90%	96	96	96	96	94	93	93	93	91	90	94%
44	Qwen 3.5 397B A17B	$0.044	4.6m	89%	97	96	95	95	93	93	93	91	91	90	93%
18	o4 Mini High	$0.059	1.5m	91%	96	95	94	94	94	94	93	93	92	90	93%
5	GPT-5 Mini	$0.017	1.3m	90%	95	94	94	93	93	92	92	92	92	91	93%
21	Z.AI GLM 4.7	$0.018	2.3m	89%	96	96	93	93	93	93	93	91	90	89	93%
3	Claude Sonnet 4	$0.047	20.4s	89%	95	95	95	93	93	93	92	91	89	89	93%
4	o4 Mini	$0.031	42.1s	89%	96	94	94	93	93	91	91	91	89	89	92%
12	GPT-5.2	$0.052	55.4s	89%	94	94	93	93	92	92	91	91	90	90	92%
1	Gemini 3 Flash (Preview)	$0.0068	9.3s	90%	93	93	93	92	92	92	92	90	90	90	92%
45	GPT-5	$0.115	3.2m	90%	93	93	93	92	92	91	91	90	90	90	92%
34	GPT-5.1	$0.088	2.0m	86%	95	95	93	93	92	91	91	90	87	86	91%
23	Z.AI GLM 5	$0.030	2.1m	88%	94	94	93	91	91	91	90	90	89	89	91%
28	MoonshotAI: Kimi K2.5	$0.028	2.4m	85%	95	94	94	93	92	92	90	90	86	83	91%
9	Claude Sonnet 4.5	$0.051	20.8s	85%	96	94	93	91	91	90	89	88	87	85	90%
31	Gemini 3 Pro (Preview)	$0.096	1.0m	79%	94	93	92	92	92	92	90	86	86	69	89%
49	Claude Opus 4	$0.248	37.7s	82%	94	92	91	90	89	87	87	87	83	83	88%
6	Mistral Large	$0.024	19.0s	81%	93	91	88	88	87	87	85	84	82	81	87%
8	GPT-4.1	$0.017	22.4s	79%	92	90	90	89	88	87	85	83	82	75	86%
2	Gemini 2.5 Flash	$0.0051	5.7s	81%	91	90	89	88	88	88	85	82	81	79	86%
10	Mistral Large 2	$0.024	17.4s	79%	91	88	87	86	86	84	83	83	82	77	85%
15	Stealth: Aurora Alpha	—	12.5s	76%	88	84	83	83	82	82	81	80	78	75	82%
17	DeepSeek V3 (2024-12-26)	$0.0038	27.4s	68%	91	90	88	87	86	85	81	77	66	58	81%
14	DeepSeek-V2 Chat	$0.0038	24.8s	72%	89	86	84	83	83	80	79	78	71	71	80%
19	DeepSeek V3.2	$0.0028	41.9s	70%	93	85	82	82	79	78	77	76	76	74	80%
20	Z.AI GLM 4.5	$0.0053	35.0s	69%	91	83	83	81	79	78	77	76	73	68	79%
30	Z.AI GLM 4.7 Flash	$0.0030	1.7m	72%	86	83	82	81	79	78	78	76	74	70	79%
39	GPT-5 Nano	$0.0079	3.0m	71%	84	82	82	82	81	78	77	76	69	69	78%
46	Qwen 3.5 Plus (2026-02-15)	$0.011	1.9m	41%	97	93	92	91	89	89	87	77	61	3	78%
25	Claude 3.7 Sonnet	$0.043	18.3s	69%	87	86	81	81	78	77	74	72	71	71	78%
16	Claude Haiku 4.5	$0.016	12.8s	74%	81	80	80	79	78	78	77	76	74	74	78%
29	Claude 3.5 Sonnet	$0.071	15.9s	74%	80	80	79	79	78	77	76	76	73	73	77%
22	Writer: Palmyra X5	$0.013	20.0s	67%	83	82	82	81	79	76	72	72	70	61	76%
40	DeepSeek V3.1	$0.0023	1.2m	41%	89	88	85	85	81	79	78	76	62	3	73%
26	GPT-4.1 Mini	$0.0024	9.0s	59%	84	78	77	71	70	68	66	65	64	58	70%
38	DeepSeek V3 (2025-03-24)	$0.0030	35.9s	40%	90	88	79	78	77	74	72	67	67	3	70%
35	GPT-4o, May 13th (temp=1)	$0.047	5.3s	59%	80	78	73	70	70	69	67	66	60	52	69%
27	Mistral Small Creative	$0.0012	8.2s	61%	74	74	72	69	68	67	66	65	62	57	67%
32	GPT-4o, Aug. 6th (temp=1)	$0.019	5.8s	59%	74	73	72	71	68	66	65	64	58	56	67%
33	Qwen 2.5 72B	$0.0015	12.7s	57%	77	73	68	66	65	64	64	64	62	53	65%
42	GPT-4o, Aug. 6th (temp=0)	$0.052	18.5s	49%	78	77	70	70	70	70	68	67	34	34	64%
36	Hermes 3 405B	$0.0079	17.3s	54%	73	71	64	64	62	62	58	56	55	55	62%
41	Claude 3.5 Haiku	$0.0082	7.4s	38%	76	73	66	64	64	64	62	59	59	0	59%
37	Ministral 8B	$0.0010	15.3s	48%	66	64	64	58	57	53	51	51	51	51	56%
48	Llama 3.1 Nemotron 70B	$0.010	26.3s	37%	70	68	64	60	60	57	57	50	49	3	54%
50	Hermes 3 70B	$0.0024	23.5s	35%	71	64	64	62	58	55	55	51	40	0	52%
43	Claude 3 Haiku	$0.0026	4.0s	41%	61	58	51	49	47	46	44	44	44	44	49%
58	GPT-4o, May 13th (temp=0)	$0.074	20.0s	20%	79	77	77	76	69	39	38	3	3	3	46%
52	WizardLM 2 8x22b	$0.0069	30.7s	40%	55	52	49	46	46	43	42	42	42	40	46%
51	GPT-4o Mini (temp=1)	$0.0010	10.0s	35%	57	56	55	52	46	44	44	43	40	17	46%
47	Ministral 3B	$0.0004	8.2s	41%	49	44	44	44	44	43	42	41	40	39	43%
53	Cohere Command R+ (Aug. 2024)	$0.027	18.8s	30%	56	55	55	54	49	49	44	42	3	3	41%
56	Mistral NeMO	$0.0014	1.3m	33%	55	42	40	39	38	37	36	36	35	35	39%
59	Llama 3.1 70B	$0.0044	1.5m	26%	60	58	52	52	51	46	34	3	3	0	36%
57	Llama 3.1 8B	$0.0004	30.0s	23%	57	51	42	38	36	33	33	32	3	3	33%
54	Gemini 2.5 Flash Lite	$0.0011	4.7s	21%	62	61	57	50	48	42	0	0	0	0	32%
55	Rocinante 12B	$0.0018	7.3s	23%	44	44	42	33	33	33	33	33	3	0	30%
61	GPT-4o Mini (temp=0)	$0.0015	2.5m	12%	53	44	41	40	35	3	3	3	3	3	23%
60	GPT-4.1 Nano	$0.0008	9.4s	11%	51	34	33	17	17	16	3	3	0	0	17%
71.28%

5 codex entries

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
29	Claude 3.5 Sonnet	$0.017	5.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
9	Claude 3.7 Sonnet	$0.0087	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
3	Hermes 3 405B	$0.0016	10.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
2	GPT-4.1	$0.0035	3.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
19	o4 Mini	$0.010	13.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
35	o4 Mini High	$0.016	29.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
7	Claude Sonnet 4	$0.0081	4.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	Gemini 2.5 Pro	$0.018	11.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
39	Gemini 3 Pro (Preview)	$0.023	15.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3 Flash (Preview)	$0.0015	2.3s	100%	100	100	100	100	100	100	100	100	100	100	100%
24	Claude Opus 4.5	$0.014	4.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
25	Claude Opus 4.6	$0.014	4.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
28	Z.AI GLM 4.7	$0.0053	39.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
10	Z.AI GLM 5	$0.0047	18.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
33	MoonshotAI: Kimi K2.5	$0.0078	45.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	GPT-5 Mini	$0.0051	25.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
31	GPT-5.1	$0.014	21.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
15	GPT-5.2	$0.0090	9.2s	100%	100	100	100	100	100	100	100	100	100	100	100%
5	Z.AI GLM 4.5	$0.0013	11.9s	95%	100	100	100	100	100	100	100	100	100	91	99%
56	Qwen 3.5 397B A17B	$0.017	2.0m	95%	100	100	100	100	100	100	100	100	100	91	99%
4	DeepSeek V3 (2025-03-24)	$0.0006	9.6s	95%	100	100	100	100	100	100	100	100	93	93	99%
18	Claude Sonnet 4.5	$0.0081	4.2s	90%	100	100	100	100	100	100	100	100	100	83	98%
20	Z.AI GLM 4.6	$0.0014	23.7s	89%	100	100	100	100	100	100	100	100	91	83	97%
11	Claude 3.5 Haiku	$0.0022	5.3s	87%	100	100	100	100	100	100	100	100	83	83	97%
23	Qwen 3.5 Plus (2026-02-15)	$0.0014	12.5s	79%	100	100	100	100	100	100	100	100	100	66	97%
60	GPT-5	$0.040	1.1m	91%	100	100	100	100	100	100	91	91	91	91	96%
6	DeepSeek-V2 Chat	$0.0007	6.2s	89%	100	100	100	100	100	100	93	93	91	83	96%
21	Llama 3.1 Nemotron 70B	$0.0020	8.2s	79%	100	100	100	100	100	100	100	100	91	66	96%
12	DeepSeek V3 (2024-12-26)	$0.0007	7.3s	85%	100	100	100	100	100	100	93	93	93	76	96%
8	DeepSeek V3.2	$0.0004	7.6s	87%	100	100	100	100	100	91	91	91	91	91	96%
14	Llama 3.1 70B	$0.0008	10.8s	87%	100	100	100	100	100	100	93	90	83	83	95%
26	Stealth: Aurora Alpha	—	2.7s	77%	100	100	100	100	100	93	93	93	93	66	94%
22	Mistral Large	$0.0046	4.7s	84%	100	100	100	91	91	91	91	91	91	91	94%
45	Z.AI GLM 4.7 Flash	$0.0010	52.6s	80%	100	100	100	100	100	93	93	83	83	75	93%
13	Gemini 2.5 Flash	$0.0009	1.2s	84%	100	93	93	93	93	93	93	93	83	83	92%
55	Claude Opus 4	$0.040	6.7s	76%	100	100	100	100	100	83	83	83	83	83	92%
17	Mistral Large 2	$0.0047	5.1s	91%	91	91	91	91	91	91	91	91	91	91	91%
38	GPT-4o, May 13th (temp=0)	$0.011	2.0s	72%	100	100	100	100	91	91	83	76	76	76	89%
47	DeepSeek V3.1	$0.0005	14.7s	43%	100	100	100	100	100	100	91	84	83	3	86%
27	Qwen 2.5 72B	$0.0002	5.8s	77%	93	93	93	93	93	93	75	75	75	75	86%
32	GPT-4.1 Mini	$0.0010	3.2s	69%	93	93	83	83	83	83	83	83	83	60	83%
57	GPT-5 Nano	$0.0033	1.3m	41%	100	100	100	93	93	93	93	75	75	3	83%
37	GPT-4o Mini (temp=1)	$0.0003	5.4s	59%	93	93	93	93	75	75	75	75	67	67	81%
46	GPT-4o, May 13th (temp=1)	$0.011	2.7s	66%	91	91	83	83	76	76	76	76	76	76	80%
42	GPT-4o, Aug. 6th (temp=0)	$0.0054	2.2s	65%	91	91	91	76	76	76	76	76	76	76	80%
41	GPT-4o, Aug. 6th (temp=1)	$0.0052	2.4s	66%	91	91	83	76	76	76	76	76	76	76	79%
34	GPT-4o Mini (temp=0)	$0.0003	5.0s	64%	93	93	75	75	75	75	75	75	75	75	79%
44	Writer: Palmyra X5	$0.0028	7.2s	64%	83	83	83	83	83	83	76	76	76	44	77%
40	Claude Haiku 4.5	$0.0029	3.1s	65%	83	83	83	83	76	76	75	67	67	67	76%
36	Mistral Small Creative	$0.0002	2.1s	67%	76	76	76	76	76	76	69	69	63	60	71%
51	Cohere Command R+ (Aug. 2024)	$0.0057	5.8s	49%	93	87	87	76	75	67	57	57	57	44	70%
43	Gemini 2.5 Flash Lite	$0.0002	947ms	62%	75	75	67	67	67	67	67	67	67	67	68%
48	Mistral NeMO	$0.0002	2.0s	51%	83	75	75	75	67	67	60	60	50	44	66%
50	Llama 3.1 8B	$0.0001	4.6s	45%	83	83	78	78	63	60	60	57	54	39	65%
49	Claude 3 Haiku	$0.0007	3.3s	52%	76	76	67	67	67	60	57	57	50	50	62%
52	Ministral 3B	$0.0001	1.2s	40%	83	78	76	67	57	54	50	50	44	42	60%
54	Hermes 3 70B	$0.0005	13.2s	34%	76	75	75	66	60	57	57	50	44	0	56%
53	Ministral 8B	$0.0002	3.6s	37%	69	49	49	44	44	43	43	43	43	42	47%
59	Rocinante 12B	$0.0004	5.1s	21%	67	57	53	40	38	37	33	3	3	3	34%
58	GPT-4.1 Nano	$0.0001	951ms	20%	57	57	57	57	44	33	17	0	0	0	32%
61	WizardLM 2 8x22b	$0.0016	14.5s	3%	3	3	3	3	3	3	0	0	0	0	2%
85.72%

10 codex entries

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
3	Claude Sonnet 4	$0.012	4.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
41	Gemini 3 Pro (Preview)	$0.032	21.4s	100%	100	100	100	100	100	100	100	100	100	100	100%
1	Gemini 3 Flash (Preview)	$0.0015	2.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Claude Sonnet 4.5	$0.012	5.1s	100%	100	100	100	100	100	100	100	100	100	100	100%
18	Claude Opus 4.5	$0.021	5.7s	100%	100	100	100	100	100	100	100	100	100	100	100%
20	Claude Opus 4.6	$0.021	6.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
8	Z.AI GLM 4.7	$0.0047	26.8s	100%	100	100	100	100	100	100	100	100	100	100	100%
30	MoonshotAI: Kimi K2.5	$0.0093	48.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
56	Qwen 3.5 397B A17B	$0.018	2.0m	100%	100	100	100	100	100	100	100	100	100	100	100%
2	Stealth: Aurora Alpha	—	4.5s	100%	100	100	100	100	100	100	100	100	100	100	100%
53	Claude Opus 4	$0.062	8.6s	95%	100	100	100	100	100	100	100	100	100	92	99%
27	Gemini 2.5 Pro	$0.022	14.8s	95%	100	100	100	100	100	100	100	100	100	92	99%
16	GPT-5.2	$0.013	12.3s	95%	100	100	100	100	100	100	100	100	100	92	99%
26	Claude 3.5 Sonnet	$0.024	6.9s	96%	100	100	100	100	100	100	100	100	94	94	99%
34	Z.AI GLM 5	$0.0079	41.9s	92%	100	100	100	100	100	100	100	100	100	86	99%
12	GPT-4o, May 13th (temp=0)	$0.015	2.2s	95%	100	100	100	100	100	100	100	94	94	94	98%
5	Llama 3.1 Nemotron 70B	$0.0030	9.7s	91%	100	100	100	100	100	94	94	94	94	92	97%
31	o4 Mini	$0.014	20.7s	89%	100	100	100	100	100	100	92	92	86	86	96%
57	GPT-5	$0.042	1.0m	86%	100	100	100	100	92	92	92	92	92	92	95%
44	GPT-5.1	$0.020	29.2s	86%	100	100	100	100	92	92	92	92	92	92	95%
49	o4 Mini High	$0.024	36.5s	86%	100	100	100	100	100	92	92	92	86	86	95%
10	Writer: Palmyra X5	$0.0034	6.6s	86%	100	100	100	100	100	100	86	86	86	86	94%
11	Qwen 3.5 Plus (2026-02-15)	$0.0018	12.4s	86%	100	100	100	100	100	100	86	86	86	86	94%
9	GPT-4o, Aug. 6th (temp=1)	$0.0062	2.3s	89%	100	94	94	94	94	94	94	94	89	89	94%
7	GPT-4o, Aug. 6th (temp=0)	$0.0063	2.2s	91%	94	94	94	94	94	94	94	94	94	89	94%
14	Z.AI GLM 4.5	$0.0014	10.9s	84%	100	100	100	100	100	92	86	86	86	86	94%
17	DeepSeek V3 (2025-03-24)	$0.0008	16.3s	84%	100	100	100	94	94	94	92	86	86	86	93%
6	Mistral Small Creative	$0.0003	2.3s	83%	100	100	100	94	94	94	92	92	86	79	93%
36	GPT-5 Mini	$0.0072	38.4s	88%	100	92	92	92	92	92	92	92	92	92	93%
43	Z.AI GLM 4.6	$0.0025	40.3s	72%	100	100	100	100	100	100	92	92	86	53	92%
52	GPT-5 Nano	$0.0037	1.4m	76%	100	100	94	86	86	86	86	86	86	86	90%
37	Z.AI GLM 4.7 Flash	$0.0010	36.3s	80%	100	94	89	89	89	89	89	89	89	79	90%
29	Mistral Large	$0.0067	5.1s	73%	100	100	100	92	86	86	86	86	79	79	90%
23	GPT-4.1	$0.0044	5.8s	77%	100	100	86	86	86	86	86	86	86	86	89%
13	Gemini 2.5 Flash	$0.0013	1.6s	81%	94	92	92	92	92	86	86	86	86	79	89%
15	GPT-4.1 Mini	$0.0010	4.0s	82%	92	92	92	92	92	86	86	86	86	79	89%
24	Claude 3.5 Haiku	$0.0028	4.4s	73%	94	94	94	94	89	89	89	89	83	64	88%
25	DeepSeek-V2 Chat	$0.0010	7.9s	73%	100	100	94	89	86	86	86	79	79	79	88%
33	DeepSeek V3.2	$0.0008	12.8s	68%	100	100	100	92	92	92	92	79	72	58	88%
21	Claude Haiku 4.5	$0.0042	3.6s	86%	86	86	86	86	86	86	86	86	86	86	86%
22	GPT-4o Mini (temp=1)	$0.0003	4.9s	75%	89	89	89	89	83	83	83	83	83	72	84%
45	Llama 3.1 70B	$0.0012	6.9s	41%	94	94	94	94	94	94	94	94	86	0	84%
28	DeepSeek V3 (2024-12-26)	$0.0011	8.7s	73%	94	92	89	86	86	86	79	79	72	72	84%
42	Hermes 3 405B	$0.0025	12.1s	60%	100	94	89	86	86	86	86	86	79	40	83%
19	GPT-4o Mini (temp=0)	$0.0003	3.4s	83%	83	83	83	83	83	83	83	83	83	83	83%
38	DeepSeek V3.1	$0.0008	14.6s	69%	94	92	92	86	86	86	79	79	66	64	83%
39	Mistral Large 2	$0.0067	5.3s	70%	100	86	86	86	86	79	79	79	72	72	83%
46	GPT-4o, May 13th (temp=1)	$0.014	2.1s	59%	100	94	94	94	94	89	89	56	56	56	82%
32	Gemini 2.5 Flash Lite	$0.0002	1.2s	66%	94	94	89	86	79	79	79	79	72	66	82%
35	Qwen 2.5 72B	$0.0005	5.9s	68%	94	89	89	89	89	89	83	64	64	64	81%
40	Ministral 8B	$0.0003	3.4s	62%	100	86	81	81	81	76	71	69	69	64	78%
50	WizardLM 2 8x22b	$0.0018	5.9s	38%	100	94	92	86	86	79	79	72	72	0	76%
55	Hermes 3 70B	$0.0008	53.3s	54%	86	81	79	79	79	72	72	72	58	33	71%
58	Claude 3.7 Sonnet	$0.013	6.0s	12%	100	100	100	100	100	100	92	3	3	3	70%
51	Cohere Command R+ (Aug. 2024)	$0.0081	6.4s	47%	89	89	86	79	79	66	64	61	49	33	70%
48	Claude 3 Haiku	$0.0010	3.5s	53%	79	79	79	79	66	64	58	58	58	58	68%
47	Mistral NeMO	$0.0004	2.7s	54%	79	79	72	72	64	64	64	58	58	53	67%
54	Ministral 3B	$0.0001	2.4s	36%	72	66	61	60	49	49	40	38	33	33	50%
59	Rocinante 12B	$0.0005	3.7s	18%	72	72	64	64	51	33	33	3	0	0	39%
60	Llama 3.1 8B	$0.0001	10.6s	27%	44	44	42	40	39	38	33	33	3	3	32%
61	GPT-4.1 Nano	$0.0002	1.9s	0%	56	33	17	17	0	0	0	0	0	0	12%
86.29%

20 codex entries

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank	Model	Avg. Cost	Avg. Time	Stability	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Total
6	Claude Opus 4.5	$0.037	8.6s	100%	100	100	100	100	100	100	100	100	100	100	100%
4	Claude Opus 4.6	$0.035	8.0s	100%	100	100	100	100	100	100	100	100	100	100	100%
16	Gemini 2.5 Pro	$0.035	21.2s	92%	100	100	100	96	96	96	96	96	96	96	97%
5	Claude Sonnet 4.5	$0.022	8.9s	93%	100	100	100	100	100	100	92	92	92	92	97%
46	GPT-5	$0.051	1.3m	94%	97	97	97	97	97	97	97	97	97	92	97%
21	GPT-5.1	$0.029	42.6s	93%	97	97	97	97	97	97	97	94	92	92	96%
32	Gemini 3 Pro (Preview)	$0.051	33.2s	91%	100	97	97	97	96	96	92	92	92	92	95%
9	GPT-5.2	$0.023	21.2s	92%	97	97	96	96	96	96	92	92	92	92	95%
43	MoonshotAI: Kimi K2.5	$0.015	1.9m	88%	97	96	96	96	92	92	92	92	92	88	93%
29	Z.AI GLM 5	$0.014	1.1m	86%	100	97	97	94	94	92	92	92	91	85	93%
2	Gemini 2.5 Flash	$0.0023	2.5s	88%	100	92	92	92	92	92	92	92	92	92	93%
31	o4 Mini High	$0.031	50.7s	87%	97	97	97	97	97	94	89	89	89	85	93%
25	Claude 3.5 Sonnet	$0.041	10.2s	88%	97	96	96	96	96	96	92	89	88	85	93%
1	Gemini 3 Flash (Preview)	$0.0031	4.0s	92%	92	92	92	92	92	92	92	92	92	92	92%
15	o4 Mini	$0.017	28.6s	87%	97	94	94	94	94	94	89	89	89	85	92%
8	Claude Sonnet 4	$0.022	9.0s	89%	92	92	92	92	92	92	92	92	88	88	92%
27	Z.AI GLM 4.7	$0.010	1.0m	87%	96	96	92	92	92	92	88	88	88	88	91%
3	GPT-4o, Aug. 6th (temp=0)	$0.011	3.8s	89%	92	92	92	92	92	92	89	89	89	89	91%
7	GPT-4o, Aug. 6th (temp=1)	$0.010	4.3s	80%	97	97	97	97	92	89	86	85	82	82	90%
42	GPT-5 Nano	$0.0052	1.8m	82%	97	94	94	94	89	89	88	86	85	85	90%
55	Claude Opus 4	$0.109	14.4s	81%	96	96	96	88	88	88	88	85	85	85	90%
10	Mistral Large	$0.011	8.7s	79%	96	96	96	92	89	88	88	86	81	81	89%
23	GPT-5 Mini	$0.0094	45.9s	86%	92	89	89	89	89	89	89	88	86	85	89%
14	DeepSeek V3.2	$0.0011	12.7s	71%	100	97	97	97	85	85	85	81	81	77	89%
36	Z.AI GLM 4.7 Flash	$0.0023	1.5m	80%	97	91	91	89	88	86	86	86	85	82	88%
12	Mistral Large 2	$0.012	8.8s	80%	92	92	92	89	88	88	88	85	81	77	87%
19	Qwen 3.5 Plus (2026-02-15)	$0.0039	34.5s	81%	92	92	92	89	88	88	85	85	85	79	87%
41	Z.AI GLM 4.6	$0.0053	1.4m	74%	100	96	92	91	88	88	85	77	77	73	87%
11	Writer: Palmyra X5	$0.0053	10.4s	78%	94	92	89	88	86	86	86	86	82	77	87%
26	GPT-4o, May 13th (temp=0)	$0.025	3.5s	79%	91	89	89	89	86	86	82	82	82	79	85%
17	Stealth: Aurora Alpha	—	6.2s	81%	91	86	86	85	85	85	85	85	85	81	85%
18	Z.AI GLM 4.5	$0.0026	20.2s	76%	97	86	86	86	85	82	82	81	81	81	85%
28	DeepSeek V3.1	$0.0014	35.2s	71%	96	96	92	89	88	88	81	73	69	69	84%
37	Claude 3.7 Sonnet	$0.023	10.0s	63%	100	100	92	92	89	78	77	73	70	62	83%
22	DeepSeek V3 (2024-12-26)	$0.0017	13.9s	71%	94	91	91	86	86	86	82	73	73	65	83%
40	Claude Haiku 4.5	$0.0070	5.0s	42%	97	97	97	94	94	89	86	86	85	3	83%
13	Claude 3.5 Haiku	$0.0047	6.1s	78%	85	85	85	85	85	85	85	82	75	75	83%
61	Qwen 3.5 397B A17B	$0.027	3.0m	43%	96	92	92	92	92	92	88	88	85	3	82%
30	GPT-4o, May 13th (temp=1)	$0.024	3.4s	73%	91	86	85	82	82	82	79	75	73	73	81%
20	GPT-4.1 Mini	$0.0013	5.3s	71%	89	86	86	86	85	82	82	69	69	69	80%
24	Mistral Small Creative	$0.0005	6.8s	71%	81	81	81	77	77	77	73	73	73	69	76%
33	DeepSeek V3 (2025-03-24)	$0.0013	23.4s	62%	86	86	82	82	77	77	73	73	73	51	76%
38	Hermes 3 405B	$0.0041	19.8s	57%	94	91	86	81	77	73	69	61	60	58	75%
53	GPT-4.1	$0.0076	8.2s	27%	96	94	94	94	94	94	89	85	3	3	75%
34	Llama 3.1 Nemotron 70B	$0.0051	16.2s	62%	82	82	82	79	73	69	69	69	69	65	74%
35	Llama 3.1 70B	$0.0018	15.9s	61%	82	82	77	77	76	73	73	73	61	51	73%
47	DeepSeek-V2 Chat	$0.0017	12.2s	41%	91	86	86	86	82	77	73	73	69	3	73%
44	Cohere Command R+ (Aug. 2024)	$0.013	11.8s	58%	88	82	77	77	73	70	66	65	60	55	71%
51	Gemini 2.5 Flash Lite	$0.0004	1.5s	24%	100	97	96	89	85	85	81	77	0	0	71%
39	WizardLM 2 8x22b	$0.0031	8.4s	57%	82	73	69	69	69	66	65	65	60	51	67%
45	GPT-4o Mini (temp=0)	$0.0005	7.0s	55%	71	66	60	59	59	59	59	59	59	59	61%
48	GPT-4o Mini (temp=1)	$0.0005	8.2s	48%	79	71	71	66	66	59	55	48	44	44	60%
52	Ministral 8B	$0.0004	4.0s	36%	78	78	74	71	66	66	62	56	33	0	58%
50	Claude 3 Haiku	$0.0016	4.3s	45%	75	65	60	59	55	55	51	51	48	40	56%
56	Hermes 3 70B	$0.0012	29.1s	36%	79	71	69	69	69	60	59	44	33	0	55%
49	Ministral 3B	$0.0002	2.5s	47%	65	62	59	58	58	53	53	50	47	39	54%
54	Mistral NeMO	$0.0006	4.2s	42%	65	61	61	53	50	48	47	44	44	42	52%
57	Llama 3.1 8B	$0.0001	8.1s	30%	71	69	61	55	43	42	39	36	33	27	48%
59	Qwen 2.5 72B	$0.0012	41.3s	29%	91	49	49	44	43	41	40	39	38	36	47%
58	Rocinante 12B	$0.0009	8.3s	34%	43	42	40	38	38	38	33	33	29	28	36%
60	GPT-4.1 Nano	$0.0004	4.1s	13%	33	33	24	17	17	17	3	3	0	0	15%
79.70%

40 codex entries

0-shot ToolingUtilityLogicRule following

Performance Score Distribution (Top 20)

Click a model name to view its detail page.