No hallucinated violations

Test: Codex Red Herring (False Positive Detection)

Avg. Score

51.9%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Grok 4.1 Fast	97.9%	$0.0019	12.5s	79%
2	ByteDance Seed 1.6 Flash	95.7%	$0.0008	9.1s	69%
3	o4 Mini	97.5%	$0.014	25.0s	78%
4	Gemini 2.5 Flash Lite (Reasoning)	94.4%	$0.0023	16.6s	68%
5	GPT-5.1	96.9%	$0.025	26.1s	76%
6	o4 Mini High	98.1%	$0.027	52.5s	81%
7	GPT-5 Mini	95.0%	$0.0059	37.8s	67%
8	GPT-5 Nano	95.0%	$0.0035	1.1m	70%
9	Gemini 2.5 Flash (Reasoning)	91.3%	$0.0085	14.2s	62%
10	ByteDance Seed 1.6	89.6%	$0.0043	32.7s	56%
11	GPT-5.2	89.3%	$0.013	14.5s	55%
12	Z.AI GLM 5	95.2%	$0.017	1.4m	69%
13	Aion 2.0	91.9%	$0.0096	1.3m	63%
14	Minimax M2.5	85.5%	$0.0032	25.9s	50%
15	GPT-4.1	85.9%	$0.0048	1.2s	43%
16	Grok 4 Fast	80.3%	$0.0019	19.2s	44%
17	Arcee AI: Trinity Mini	79.2%	$0.0009	26.3s	33%
18	Z.AI GLM 4.7 Flash	83.4%	$0.0039	2.5m	49%
19	GPT-5	83.5%	$0.048	1.5m	52%
20	Claude Opus 4.6 (Reasoning)	96.3%	$0.120	1.0m	74%
21	Ministral 3 8B	74.5%	$0.0013	17.9s	14%
22	Claude Opus 4.6	79.7%	$0.049	13.2s	34%
23	Z.AI GLM 4.6	74.6%	$0.015	59.0s	31%
24	Claude Haiku 4.5	51.9%	$0.0080	4.2s	28%
25	Z.AI GLM 4.7	75.9%	$0.023	2.2m	46%
26	Z.AI GLM 4.5	63.4%	$0.0038	25.0s	19%
27	MoonshotAI: Kimi K2.5	75.6%	$0.017	2.6m	48%
28	Claude Sonnet 4.6	66.2%	$0.033	17.4s	25%
29	GPT-4.1 Nano	58.4%	$0.0005	4.6s	8%
30	Ministral 8B	60.8%	$0.0007	4.4s	6%
31	Cohere Command R+ (Aug. 2024)	60.6%	$0.017	7.4s	15%
32	Gemini 3 Flash (Preview, Reasoning)	67.6%	$0.028	49.2s	20%
33	Gemini 2.5 Flash	31.4%	$0.0021	1.5s	23%
34	DeepSeek V3 (2024-12-26)	34.3%	$0.0023	6.6s	21%
35	GPT-4o, Aug. 6th (temp=1)	45.0%	$0.0094	2.4s	13%
36	Hermes 3 405B	47.5%	$0.0059	5.2s	9%
37	Claude Opus 4.5	52.4%	$0.036	4.4s	20%
38	Hermes 3 70B	43.4%	$0.0018	12.1s	7%
39	Gemini 3 Flash (Preview)	25.0%	$0.0034	3.2s	20%
40	Claude Sonnet 4	45.5%	$0.022	4.5s	13%
41	DeepSeek-V2 Chat	30.9%	$0.0023	8.3s	15%
42	Grok 4	75.5%	$0.074	1.6m	37%
43	DeepSeek V3 (2025-03-24)	31.5%	$0.0015	15.7s	15%
44	Mistral Medium 3.1	39.4%	$0.0032	4.7s	5%
45	GPT-4o, Aug. 6th (temp=0)	31.4%	$0.011	2.6s	14%
46	Qwen 2.5 72B	35.2%	$0.0009	11.3s	6%
47	Claude Sonnet 4.5	32.8%	$0.024	5.9s	19%
48	GPT-4.1 Mini	35.6%	$0.0018	6.9s	4%
49	Claude 3 Haiku	19.3%	$0.0020	3.5s	16%
50	Llama 3.1 70B	33.5%	$0.0031	16.4s	7%
51	Claude 3.7 Sonnet	35.5%	$0.022	4.3s	13%
52	Rocinante 12B	32.3%	$0.0015	13.3s	6%
53	GPT-4o Mini (temp=1)	28.2%	$0.0008	5.6s	6%
54	Gemini 2.5 Flash Lite	32.1%	$0.0010	4.0s	2%
55	Gemini 2.5 Pro	70.4%	$0.073	52.1s	21%
56	Mistral Small 3.2 24B	31.3%	$0.0010	11.9s	3%
57	Mistral Large 3	32.2%	$0.0040	11.7s	2%
58	Writer: Palmyra X5	24.0%	$0.0085	13.8s	12%
59	WizardLM 2 8x22b	32.6%	$0.0041	55.3s	12%
60	Gemini 3.1 Pro (Preview)	76.9%	$0.120	1.5m	45%
61	GPT-4o Mini (temp=0)	24.5%	$0.0009	14.5s	6%
62	Claude 3.5 Sonnet	39.1%	$0.045	6.9s	15%
63	DeepSeek V3.1	24.5%	$0.0019	36.6s	11%
64	Mistral Large 2	32.2%	$0.016	11.4s	2%
65	Claude Sonnet 4.6 (Reasoning)	87.5%	$0.145	2.2m	55%
66	DeepSeek V3.2	21.3%	$0.0018	42.8s	11%
67	Gemma 3 12B	13.7%	$0.0005	11.7s	10%
68	Ministral 3 14B	29.3%	$0.0019	29.2s	1%
69	Mistral Large	31.9%	$0.016	12.3s	2%
70	Gemma 3 27B	12.5%	$0.0007	12.6s	10%
71	GPT-4o, May 13th (temp=1)	23.5%	$0.033	3.0s	16%
72	Llama 3.1 Nemotron 70B	22.6%	$0.0077	21.4s	5%
73	Ministral 3B	9.9%	$0.0004	12.4s	6%
74	Mistral Small Creative	9.3%	$0.0011	8.9s	4%
75	Gemini 3 Pro (Preview)	75.6%	$0.144	1.7m	47%
76	Llama 3.1 8B	10.0%	$0.0003	33.8s	8%
77	Gemma 3 4B	5.6%	$0.0003	15.0s	5%
78	Ministral 3 3B	6.7%	$0.0013	31.1s	5%
79	GPT-4o, May 13th (temp=0)	16.7%	$0.043	6.4s	11%
80	Mistral NeMO	14.7%	$0.0018	1.6m	4%
81	Arcee AI: Trinity Large (Preview)	15.7%	$0.0000	1.7m	2%
82	Claude Opus 4	36.0%	$0.109	7.1s	22%
83	Qwen 3.5 Plus (2026-02-15)	22.0%	$0.021	2.0m	12%
84	Qwen 3.5 397B A17B	61.5%	$0.028	5.4m	23%
51.93%

Individual Scenarios

basic entries

▼

Long text (~1594 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	33	93.3%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	33	93.3%
GPT-5.2	100	100	100	100	100	100	100	100	100	33	93.3%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	20	92.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	2	90.2%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	1	90.1%
o4 Mini High	100	100	100	100	100	100	100	100	50	50	90.0%
o4 Mini	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	33	33	86.7%
Z.AI GLM 5	100	100	100	100	100	100	100	100	33	33	86.7%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	50	50	33	83.3%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	50	50	33	83.3%
Arcee AI: Trinity Mini	100	100	100	100	100	100	50	50	50	50	80.0%
Minimax M2.5	100	100	100	100	100	50	50	33	33	20	68.7%
Z.AI GLM 4.6	100	100	100	100	50	50	50	50	33	0	63.3%
ByteDance Seed 1.6	100	100	100	100	50	33	33	33	33	33	61.7%
GPT-5 Mini	100	100	100	50	50	50	50	33	33	33	60.0%
Claude Sonnet 4.6	100	100	100	100	50	33	33	33	20	7	57.7%
Gemini 3 Pro (Preview)	100	50	50	50	50	50	50	50	50	33	53.3%
Gemini 3.1 Pro (Preview)	100	100	50	50	50	50	33	33	33	25	52.5%
Z.AI GLM 4.7	100	50	50	50	50	33	33	33	33	33	46.7%
Rocinante 12B	100	100	100	50	25	20	20	20	10	6	45.1%
Gemini 2.5 Pro	50	50	50	50	50	50	50	33	33	33	45.0%
Grok 4	100	33	33	33	33	33	33	33	33	33	40.0%
GPT-4.1	100	100	50	33	33	20	17	17	17	10	39.7%
Hermes 3 405B	100	100	33	33	25	20	20	17	13	13	37.3%
Qwen 3.5 397B A17B	50	33	33	33	33	33	33	33	33	25	34.2%
Gemini 2.5 Flash Lite	100	100	100	8	8	4	4	3	1	1	32.9%
GPT-4.1 Mini	50	50	50	50	50	50	8	8	8	4	32.8%
Mistral NeMO	100	100	100	8	6	5	4	3	0	0	32.6%
Ministral 8B	100	100	100	6	3	3	2	2	2	2	32.0%
Claude Opus 4.5	50	33	33	33	33	33	33	25	25	17	31.7%
Grok 4 Fast	33	33	33	33	33	33	33	25	25	25	30.8%
Gemini 3 Flash (Preview, Reasoning)	50	33	33	33	25	25	25	25	25	20	29.5%
Z.AI GLM 4.5	100	50	33	33	25	20	11	8	5	2	28.8%
Claude Haiku 4.5	50	33	25	25	25	25	25	20	20	20	26.8%
Cohere Command R+ (Aug. 2024)	100	33	33	20	17	17	14	14	10	9	26.8%
Hermes 3 70B	100	50	25	14	13	11	10	9	8	8	24.8%
DeepSeek V3 (2024-12-26)	50	50	33	25	25	20	10	9	8	5	23.6%
Llama 3.1 70B	100	33	33	17	13	10	8	7	6	6	23.4%
Claude 3 Haiku	25	25	25	25	25	25	25	25	20	13	23.3%
Claude 3.5 Sonnet	33	33	25	20	20	20	20	20	17	14	22.3%
Gemini 2.5 Flash	33	33	25	25	25	20	17	14	10	10	21.3%
GPT-4o, Aug. 6th (temp=1)	50	50	20	17	17	14	14	11	10	7	21.0%
Claude Sonnet 4	33	25	20	20	20	20	17	17	17	17	20.5%
Claude Opus 4	25	25	25	20	20	17	17	17	11	9	18.5%
DeepSeek V3 (2025-03-24)	25	25	20	20	17	14	14	13	13	8	16.9%
DeepSeek-V2 Chat	50	33	17	13	13	11	11	9	5	4	16.6%
WizardLM 2 8x22b	50	50	25	20	7	4	3	3	1	1	16.4%
GPT-4o, May 13th (temp=1)	25	25	20	20	17	14	13	11	8	7	15.9%
Claude 3.7 Sonnet	20	17	17	17	17	17	14	14	13	9	15.3%
Claude Sonnet 4.5	20	17	17	17	14	14	14	14	14	11	15.3%
Gemini 3 Flash (Preview)	17	17	14	14	14	14	10	9	9	6	12.4%
GPT-4o Mini (temp=1)	25	13	11	11	10	10	8	8	8	7	11.1%
GPT-4o, Aug. 6th (temp=0)	14	14	13	13	11	11	9	8	8	7	10.9%
Qwen 3.5 Plus (2026-02-15)	17	14	13	13	11	9	8	8	7	2	10.1%
DeepSeek V3.1	17	17	13	13	9	9	7	7	1	1	9.3%
Mistral Large 2	17	11	11	11	10	8	7	6	6	5	9.2%
Llama 3.1 Nemotron 70B	13	11	11	10	10	9	9	7	7	6	9.2%
Gemma 3 27B	11	11	11	9	9	9	8	8	7	5	8.9%
Gemma 3 12B	25	9	8	8	8	7	6	6	5	4	8.6%
Qwen 2.5 72B	20	14	13	8	7	6	6	5	5	2	8.6%
Mistral Large 3	14	11	10	10	8	7	7	7	6	4	8.3%
Llama 3.1 8B	33	11	9	6	5	5	5	4	2	0	8.2%
Ministral 3B	25	14	9	7	6	5	5	3	3	3	8.0%
Writer: Palmyra X5	33	10	8	5	4	4	3	3	3	3	7.7%
Ministral 3 3B	33	8	6	4	4	3	3	2	1	0	6.4%
Mistral Large	14	7	7	6	6	6	5	5	5	4	6.4%
DeepSeek V3.2	13	10	7	7	7	5	4	3	2	2	5.9%
GPT-4o, May 13th (temp=0)	10	9	8	8	5	4	4	3	2	2	5.7%
Mistral Medium 3.1	7	7	6	6	6	6	6	5	5	3	5.6%
Mistral Small 3.2 24B	8	7	6	6	6	5	5	5	4	2	5.5%
GPT-4o Mini (temp=0)	7	7	6	6	6	6	6	5	3	1	5.2%
Mistral Small Creative	5	4	4	3	3	3	2	2	2	2	2.9%
Arcee AI: Trinity Large (Preview)	13	3	2	2	2	1	1	1	1	0	2.5%
Gemma 3 4B	3	3	3	2	2	2	2	2	2	2	2.3%
Ministral 3 14B	3	3	3	3	2	1	1	1	1	0	1.8%

▼

Long text (~1594 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	50	95.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	14	91.4%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	9	90.9%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	3	90.3%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	50	33	88.3%
Grok 4	100	100	100	100	100	100	100	100	50	25	87.5%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	33	33	13	77.9%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	13	11	11	73.5%
Z.AI GLM 4.7 Flash	100	100	100	100	100	50	50	50	50	33	73.3%
Gemini 3.1 Pro (Preview)	100	100	100	100	50	50	50	50	50	50	70.0%
Hermes 3 70B	100	100	100	100	100	100	33	17	14	13	67.7%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	33	25	25	63.3%
Z.AI GLM 4.6	100	100	100	100	50	50	50	50	14	8	62.2%
Z.AI GLM 4.5	100	100	100	100	100	50	14	10	8	5	58.8%
Arcee AI: Trinity Mini	100	100	100	100	50	33	33	33	25	11	58.6%
Ministral 8B	100	100	100	100	100	50	4	4	2	2	56.2%
Gemini 3 Flash (Preview, Reasoning)	100	50	50	50	50	50	50	50	50	33	53.3%
Gemini 3 Pro (Preview)	100	50	50	50	50	50	50	50	50	33	53.3%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	33	48.3%
GPT-4o, Aug. 6th (temp=1)	100	33	33	33	33	33	25	17	14	14	33.7%
Hermes 3 405B	100	50	33	25	25	25	20	20	20	17	33.5%
Claude 3.5 Sonnet	50	33	33	33	33	33	33	25	25	25	32.5%
Claude Sonnet 4	50	33	33	33	33	33	33	25	25	20	32.0%
Claude Opus 4	50	50	33	33	33	25	25	25	25	20	32.0%
WizardLM 2 8x22b	100	50	50	50	17	13	11	10	10	8	31.8%
DeepSeek-V2 Chat	50	50	33	33	25	25	20	20	20	11	28.8%
Qwen 2.5 72B	100	50	33	25	20	13	13	13	13	8	28.6%
DeepSeek V3.1	100	50	25	25	17	17	14	11	10	8	27.6%
Cohere Command R+ (Aug. 2024)	100	33	33	20	20	20	17	14	11	2	27.1%
DeepSeek V3 (2024-12-26)	50	33	25	25	25	25	25	25	17	11	26.1%
Mistral NeMO	100	100	11	8	8	7	7	7	7	2	25.6%
Claude Opus 4.5	33	33	33	25	25	25	20	20	20	20	25.5%
GPT-4o Mini (temp=1)	100	25	25	25	17	14	14	13	13	8	25.4%
Claude 3.7 Sonnet	33	33	25	25	25	25	25	20	20	20	25.2%
Rocinante 12B	100	33	20	17	17	14	13	11	8	6	23.9%
Llama 3.1 70B	50	33	33	20	20	20	20	13	9	6	22.4%
Llama 3.1 Nemotron 70B	100	25	20	14	13	13	10	10	7	6	21.7%
Gemini 3 Flash (Preview)	25	25	25	20	20	20	17	17	17	17	20.2%
DeepSeek V3 (2025-03-24)	33	20	20	20	20	20	20	17	17	14	20.1%
GPT-4o, May 13th (temp=1)	50	33	20	17	17	14	14	14	11	8	19.9%
Claude Opus 4.6	33	33	25	20	14	14	14	13	11	11	18.9%
Gemini 2.5 Flash	25	25	25	20	20	20	17	14	13	10	18.8%
Gemma 3 12B	33	17	17	17	17	17	14	14	14	14	17.4%
DeepSeek V3.2	33	33	33	25	9	9	9	8	8	5	17.4%
Claude Sonnet 4.5	20	20	20	17	17	17	17	17	14	14	17.2%
Qwen 3.5 Plus (2026-02-15)	50	20	20	17	13	13	13	13	10	4	17.1%
Gemma 3 27B	20	20	20	20	17	17	17	17	14	10	17.1%
GPT-4o, Aug. 6th (temp=0)	20	20	17	17	17	17	14	14	13	13	16.0%
GPT-4o Mini (temp=0)	17	17	17	17	17	17	17	17	13	13	15.8%
Claude 3 Haiku	20	20	17	17	17	14	14	14	13	11	15.6%
Ministral 3B	33	25	13	13	11	11	10	8	8	6	13.8%
Writer: Palmyra X5	33	20	14	14	13	10	9	9	8	4	13.5%
GPT-4.1 Mini	20	20	20	17	13	9	9	9	8	6	13.0%
Mistral Large 3	11	11	11	11	11	10	10	10	7	7	9.9%
Ministral 3 3B	25	14	10	9	8	8	8	7	4	1	9.5%
Llama 3.1 8B	14	11	11	10	10	8	8	6	6	5	9.1%
GPT-4o, May 13th (temp=0)	17	17	14	14	13	6	3	3	1	1	8.8%
Mistral Large	11	11	9	9	8	8	8	8	6	6	8.5%
Mistral Large 2	11	9	9	9	8	7	7	7	6	5	7.9%
Arcee AI: Trinity Large (Preview)	10	10	10	8	8	7	6	6	6	5	7.4%
Gemma 3 4B	7	6	6	6	6	5	5	5	4	4	5.4%
Mistral Small Creative	8	6	6	5	4	4	4	3	2	2	4.4%
Ministral 3 14B	6	6	6	5	5	4	3	3	3	3	4.3%

▼

Short text (~524 words), big codex (51 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	3	90.3%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	50	50	90.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	50	9	85.9%
Z.AI GLM 5	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-5 Nano	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-5.2	100	100	100	100	100	100	100	50	33	20	80.3%
Hermes 3 405B	100	100	100	100	100	100	100	50	33	20	80.3%
Minimax M2.5	100	100	100	100	100	100	50	50	50	50	80.0%
Claude Sonnet 4	100	100	100	100	100	100	50	33	33	33	75.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	50	25	20	10	70.5%
GPT-5	100	100	100	100	50	50	50	50	50	50	70.0%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	50	50	70.0%
MoonshotAI: Kimi K2.5	100	100	100	100	50	50	50	50	50	50	70.0%
Z.AI GLM 4.7	100	100	100	100	50	50	50	50	50	50	70.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	50	50	50	50	50	33	68.3%
Grok 4	100	100	100	100	100	50	33	33	33	33	68.3%
Claude 3.5 Sonnet	100	100	100	100	50	50	50	33	33	25	64.2%
Grok 4 Fast	100	100	100	50	50	50	50	50	50	33	63.3%
Ministral 8B	100	100	100	100	100	100	6	5	4	1	61.6%
Hermes 3 70B	100	100	100	100	100	33	25	20	17	14	60.9%
Z.AI GLM 4.6	100	100	100	100	100	50	25	8	5	0	58.9%
Claude Opus 4.5	100	50	50	50	50	50	50	50	50	50	55.0%
Claude Haiku 4.5	100	50	50	50	50	50	50	50	50	33	53.3%
GPT-4o, Aug. 6th (temp=1)	100	100	100	50	50	33	25	25	20	17	52.0%
WizardLM 2 8x22b	100	50	50	50	50	50	50	50	33	33	51.7%
Claude Opus 4	50	50	50	50	50	50	50	50	50	50	50.0%
Z.AI GLM 4.5	100	100	100	50	50	50	25	8	7	5	49.4%
Claude Sonnet 4.5	100	50	50	50	50	50	33	33	33	33	48.3%
Writer: Palmyra X5	100	100	50	50	50	33	33	33	25	7	48.2%
Claude 3.7 Sonnet	100	100	33	33	33	33	33	33	33	33	46.7%
GPT-4.1 Nano	100	100	100	100	50	3	2	2	2	2	46.0%
DeepSeek V3 (2024-12-26)	50	50	50	50	50	50	50	25	14	11	40.0%
GPT-4o, Aug. 6th (temp=0)	50	50	33	33	33	33	33	25	25	17	33.3%
Rocinante 12B	100	50	50	33	25	20	20	17	14	1	33.0%
Gemini 3 Flash (Preview)	50	33	33	33	33	33	33	25	25	25	32.5%
DeepSeek V3.2	50	50	50	33	33	33	17	17	17	9	30.9%
DeepSeek V3 (2025-03-24)	50	50	33	25	25	25	25	20	14	14	28.2%
DeepSeek V3.1	50	50	33	33	33	25	25	13	9	9	28.1%
Gemini 2.5 Flash	50	33	33	25	25	25	20	20	17	7	25.5%
Qwen 2.5 72B	100	33	25	25	20	11	11	10	8	8	25.2%
GPT-4o, May 13th (temp=1)	33	33	33	25	25	20	20	20	20	11	24.1%
DeepSeek-V2 Chat	50	33	25	25	20	20	17	13	13	10	22.5%
Llama 3.1 70B	100	20	17	14	13	13	11	11	10	7	21.5%
Claude Sonnet 4.6	33	25	25	25	20	20	17	17	17	14	21.3%
Mistral NeMO	100	33	13	10	10	10	7	6	4	3	19.5%
GPT-4o Mini (temp=1)	25	25	20	20	17	17	17	11	9	7	16.7%
Qwen 3.5 Plus (2026-02-15)	33	33	20	20	17	17	10	8	7	2	16.7%
Claude 3 Haiku	20	17	17	14	14	14	14	14	11	9	14.5%
Mistral Medium 3.1	25	17	17	11	11	10	10	9	9	9	12.8%
GPT-4.1 Mini	17	14	14	13	13	13	11	10	10	9	12.3%
Mistral Small 3.2 24B	14	14	14	13	13	13	11	11	10	8	12.1%
GPT-4o, May 13th (temp=0)	25	25	20	20	20	4	2	1	1	1	12.0%
GPT-4o Mini (temp=0)	14	14	14	14	14	13	13	11	10	2	11.9%
Mistral Large 3	14	14	14	14	10	10	10	9	8	8	11.2%
Mistral Large	14	14	14	14	11	11	8	8	8	8	11.1%
Gemini 2.5 Flash Lite	20	20	10	10	10	8	8	8	7	7	10.9%
Gemma 3 12B	14	14	13	11	11	11	10	10	8	4	10.6%
Gemma 3 27B	11	11	11	11	10	10	10	10	10	9	10.4%
Mistral Small Creative	33	9	8	8	7	7	7	7	7	6	10.0%
Llama 3.1 Nemotron 70B	17	11	11	10	10	9	9	8	7	7	9.9%
Mistral Large 2	17	17	11	9	8	8	8	7	7	7	9.8%
Ministral 3 3B	14	14	14	11	9	8	7	7	6	6	9.6%
Ministral 3B	14	11	10	10	9	8	7	7	7	5	8.8%
Llama 3.1 8B	17	14	9	8	7	6	5	4	3	2	7.4%
Gemma 3 4B	10	9	9	6	6	5	3	3	2	2	5.4%
Arcee AI: Trinity Large (Preview)	8	6	6	6	6	5	5	5	4	3	5.4%
Ministral 3 14B	6	6	6	5	5	5	4	3	3	1	4.3%

▼

Short text (~524 words), small codex (11 entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	50	95.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	50	95.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	5	90.5%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	50	50	90.0%
Aion 2.0	100	100	100	100	100	100	100	50	50	50	85.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
Grok 4 Fast	100	100	100	100	100	100	100	50	50	33	83.3%
GPT-5.2	100	100	100	100	100	100	100	50	50	25	82.5%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	11	0	81.1%
Z.AI GLM 4.5	100	100	100	100	100	100	100	50	50	7	80.7%
Gemini 2.5 Pro	100	100	100	100	100	100	50	50	50	50	80.0%
Qwen 3.5 397B A17B	100	100	100	100	100	50	50	50	50	33	73.3%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	50	50	50	50	33	73.3%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	50	20	17	17	70.3%
Claude Sonnet 4	100	100	100	100	50	50	50	50	50	33	68.3%
Grok 4	100	100	100	100	100	50	33	33	33	25	67.5%
Z.AI GLM 4.7	100	100	100	50	50	50	50	50	50	50	65.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	14	13	8	7	64.2%
GPT-5	100	100	50	50	50	50	50	50	50	50	60.0%
Claude Haiku 4.5	100	100	50	50	50	50	50	50	50	50	60.0%
DeepSeek-V2 Chat	100	100	50	50	50	50	50	50	50	33	58.3%
Llama 3.1 70B	100	100	100	100	100	33	20	9	9	8	57.9%
Hermes 3 405B	100	100	100	100	33	33	33	33	25	17	57.5%
Qwen 2.5 72B	100	100	100	100	50	33	33	17	14	8	55.6%
Claude Sonnet 4.5	100	100	50	50	50	50	50	33	33	33	55.0%
GPT-4.1 Nano	100	100	100	100	100	13	8	6	4	1	53.2%
Claude 3.5 Sonnet	100	100	100	50	33	33	25	25	25	20	51.2%
MoonshotAI: Kimi K2.5	50	50	50	50	50	50	50	50	50	50	50.0%
Claude Opus 4.5	50	50	50	50	50	50	50	50	50	50	50.0%
DeepSeek V3 (2024-12-26)	100	50	50	50	50	50	50	50	33	7	49.0%
GPT-4.1 Mini	100	100	100	100	20	17	14	11	11	9	48.2%
GPT-4.1	100	100	50	33	33	33	33	33	33	25	47.5%
Hermes 3 70B	100	100	100	50	33	25	25	17	14	9	47.3%
Gemini 2.5 Flash	50	50	50	50	50	50	50	33	33	33	45.0%
DeepSeek V3 (2025-03-24)	50	50	50	50	50	50	50	33	33	33	45.0%
GPT-4o, Aug. 6th (temp=0)	50	50	50	50	50	50	50	33	33	25	44.2%
Rocinante 12B	100	100	100	50	20	20	20	11	10	8	43.9%
Claude Opus 4	50	50	50	33	33	33	33	33	25	25	36.7%
Writer: Palmyra X5	50	50	50	50	50	33	33	25	8	6	35.5%
WizardLM 2 8x22b	50	50	50	50	50	33	25	17	17	11	35.3%
Gemini 3 Flash (Preview)	50	33	33	33	33	33	33	33	25	25	33.3%
GPT-4o, Aug. 6th (temp=1)	100	33	33	25	25	25	20	20	20	20	32.2%
Claude 3.7 Sonnet	50	33	33	33	33	25	25	25	25	25	30.8%
DeepSeek V3.1	50	50	50	50	33	20	17	14	11	7	30.3%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	20	20	20	17	14	9	30.0%
GPT-4o, May 13th (temp=0)	33	33	33	33	33	25	25	25	25	20	28.7%
GPT-4o, May 13th (temp=1)	33	33	33	33	33	25	25	20	17	11	26.4%
DeepSeek V3.2	50	50	50	33	33	14	8	8	7	6	26.0%
GPT-4o Mini (temp=1)	100	25	25	20	17	17	14	14	14	13	25.9%
Mistral Medium 3.1	50	33	33	33	33	20	14	13	10	9	24.9%
Claude Sonnet 4.6	33	25	25	25	25	20	20	20	20	17	23.0%
Gemini 2.5 Flash Lite	33	33	25	20	20	20	11	11	8	7	18.9%
Claude 3 Haiku	25	25	25	20	20	17	17	14	14	11	18.8%
GPT-4o Mini (temp=0)	20	20	20	20	20	20	17	17	17	13	18.3%
Arcee AI: Trinity Large (Preview)	100	20	11	8	8	8	7	6	6	0	17.4%
Gemma 3 27B	25	20	20	17	17	17	14	13	13	11	16.5%
Ministral 3 14B	50	50	10	9	9	8	8	8	7	6	16.5%
Mistral Large 2	20	20	20	17	17	14	13	13	13	9	15.4%
Mistral Large	20	20	17	17	14	14	13	11	11	9	14.6%
Mistral Large 3	14	14	14	13	13	13	13	13	13	9	12.7%
Ministral 3B	50	17	10	9	8	7	7	7	6	4	12.4%
Mistral Small 3.2 24B	20	20	14	11	11	11	10	8	8	8	12.1%
Gemma 3 4B	14	14	14	13	11	11	11	11	11	8	11.9%
Mistral NeMO	17	14	14	14	13	11	10	10	8	6	11.7%
Gemma 3 12B	14	11	11	10	10	10	10	10	8	7	10.2%
Llama 3.1 8B	14	11	10	10	10	9	8	8	7	5	9.2%
Mistral Small Creative	14	11	9	9	8	8	8	7	6	5	8.5%
Ministral 3 3B	13	11	8	8	8	8	8	6	5	4	7.9%
Ministral 8B	6	5	5	5	5	4	4	4	4	1	4.4%
Ministral 3 8B	6	6	5	4	4	4	3	1	0	0	3.3%

detailed entries

▼

Long text (~1594 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	33	93.3%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	20	92.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	5	90.5%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	50	33	88.3%
Minimax M2.5	100	100	100	100	100	100	100	100	50	4	85.4%
Grok 4	100	100	100	100	100	100	100	50	33	20	80.3%
Z.AI GLM 4.7	100	100	100	100	100	100	50	50	50	25	77.5%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	33	33	33	25	72.5%
Z.AI GLM 4.5	100	100	100	100	100	100	50	33	11	8	70.3%
Z.AI GLM 4.6	100	100	100	100	100	50	50	50	33	1	68.4%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	33	25	65.8%
Gemini 3 Pro (Preview)	100	100	100	100	50	50	50	33	25	25	63.3%
Qwen 2.5 72B	100	100	100	100	100	50	25	17	13	5	60.9%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	33	25	25	6	2	59.2%
Claude Sonnet 4.6	100	100	50	50	50	50	50	20	14	13	49.7%
Claude Haiku 4.5	50	50	50	50	50	50	50	50	50	33	48.3%
GPT-4.1 Nano	100	100	100	100	50	7	3	2	2	1	46.5%
Z.AI GLM 4.7 Flash	100	50	50	50	50	33	33	33	33	25	45.8%
GPT-4o, Aug. 6th (temp=1)	100	100	50	50	33	33	33	20	20	17	45.7%
Claude Sonnet 4.5	100	100	33	33	33	33	25	25	25	17	42.5%
Gemini 2.5 Flash	50	50	50	50	33	33	33	33	33	33	40.0%
Rocinante 12B	100	100	50	14	13	13	13	11	10	8	33.1%
Mistral Small 3.2 24B	100	100	100	4	4	4	3	3	2	0	31.9%
Gemini 2.5 Flash Lite	100	100	100	7	5	3	2	1	1	0	31.8%
WizardLM 2 8x22b	100	100	33	25	20	14	7	6	6	3	31.4%
Arcee AI: Trinity Large (Preview)	100	100	100	2	2	2	2	1	1	1	31.2%
Gemini 2.5 Pro	50	33	33	33	33	25	25	25	20	20	29.8%
Claude Opus 4.5	50	50	33	33	33	20	20	17	17	17	29.0%
GPT-4o, May 13th (temp=1)	50	50	33	33	25	20	17	17	14	13	27.2%
Claude 3.7 Sonnet	100	25	20	20	20	20	20	17	14	14	27.0%
Claude Opus 4	50	33	25	25	25	25	20	20	20	14	25.8%
Claude Sonnet 4	33	25	25	25	25	25	25	25	25	20	25.3%
Hermes 3 405B	33	33	33	25	25	25	20	20	14	13	24.2%
DeepSeek-V2 Chat	50	50	50	20	14	14	11	10	8	1	22.9%
Hermes 3 70B	100	20	17	17	14	11	10	10	9	8	21.6%
Gemini 3 Flash (Preview)	25	25	25	25	25	20	20	20	11	10	20.6%
Claude 3 Haiku	33	25	25	25	25	17	13	13	11	11	19.7%
GPT-4o, Aug. 6th (temp=0)	33	33	33	20	14	13	11	9	6	4	17.7%
DeepSeek V3 (2024-12-26)	50	33	20	14	13	13	10	10	8	7	17.7%
GPT-4o Mini (temp=1)	20	20	20	20	17	17	17	14	13	7	16.4%
Claude 3.5 Sonnet	20	17	17	17	17	17	14	14	14	14	16.0%
DeepSeek V3.2	50	50	20	14	6	3	3	2	1	1	15.0%
Mistral Medium 3.1	33	25	17	17	13	13	9	8	8	3	14.4%
Llama 3.1 70B	33	20	20	17	13	10	10	9	6	6	14.3%
Qwen 3.5 Plus (2026-02-15)	25	25	20	20	17	14	9	6	3	2	14.2%
Writer: Palmyra X5	33	33	25	11	10	7	5	4	3	2	13.4%
GPT-4o Mini (temp=0)	17	17	14	14	14	11	11	9	8	8	12.3%
Gemma 3 12B	17	17	17	13	11	10	8	8	8	3	11.0%
Llama 3.1 Nemotron 70B	17	13	13	11	11	11	11	8	6	6	10.6%
DeepSeek V3 (2025-03-24)	17	17	13	10	10	9	9	9	3	1	9.7%
GPT-4.1 Mini	50	9	6	6	6	5	5	3	3	3	9.6%
Llama 3.1 8B	17	17	17	13	13	7	5	4	4	1	9.6%
Mistral Large 3	17	11	11	10	8	7	6	6	4	3	8.3%
Mistral Large 2	17	11	10	8	8	7	7	6	5	3	8.2%
Mistral Large	11	10	9	9	8	7	5	4	4	3	7.0%
Gemma 3 27B	14	8	8	7	6	5	5	5	4	4	6.6%
Mistral NeMO	20	14	10	8	7	2	2	2	0	0	6.5%
GPT-4o, May 13th (temp=0)	9	9	7	7	7	6	6	5	5	4	6.5%
Mistral Small Creative	50	2	2	2	2	2	1	1	1	1	6.5%
DeepSeek V3.1	7	6	5	4	4	3	2	2	1	1	3.4%
Ministral 3B	5	5	4	3	2	2	1	1	1	0	2.3%
Gemma 3 4B	4	3	2	2	2	2	2	2	1	1	2.1%
Ministral 3 3B	3	3	2	1	1	1	0	0	0	0	1.1%

▼

Long text (~1594 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	50	95.0%
o4 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	50	50	90.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	50	17	86.7%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	50	50	50	85.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	50	50	6	80.6%
Minimax M2.5	100	100	100	100	100	100	50	50	50	50	80.0%
Z.AI GLM 4.7	100	100	100	100	100	50	50	50	50	50	75.0%
Grok 4 Fast	100	100	100	100	100	50	50	50	50	50	75.0%
Z.AI GLM 4.6	100	100	100	100	100	100	50	50	33	0	73.3%
GPT-5.2	100	100	100	100	100	50	50	50	50	33	73.3%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	33	25	25	20	70.3%
Aion 2.0	100	100	100	100	50	50	50	50	50	50	70.0%
Z.AI GLM 4.5	100	100	100	100	100	100	50	17	17	1	68.4%
ByteDance Seed 1.6	100	100	100	50	50	50	50	50	50	50	65.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	33	33	25	20	20	63.2%
Gemini 2.5 Pro	100	100	50	50	50	50	50	50	50	50	60.0%
Gemini 2.5 Flash (Reasoning)	100	100	50	50	50	50	50	50	50	50	60.0%
Grok 4	100	100	100	50	50	50	50	33	33	33	60.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	50	50	50	50	33	33	25	59.2%
MoonshotAI: Kimi K2.5	100	50	50	50	50	50	50	50	50	33	53.3%
Hermes 3 70B	100	100	100	50	50	50	25	17	17	14	52.3%
Mistral Medium 3.1	100	100	100	100	33	17	17	17	11	9	50.4%
Qwen 3.5 397B A17B	50	50	50	50	50	50	50	50	50	50	50.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	20	17	8	7	7	7	46.5%
Arcee AI: Trinity Mini	100	100	50	50	50	33	25	20	4	0	43.2%
Claude Haiku 4.5	50	50	50	50	50	50	50	33	25	20	42.8%
Hermes 3 405B	100	100	33	25	25	25	25	20	20	20	39.3%
Llama 3.1 70B	100	100	50	33	25	20	17	13	13	13	38.2%
Claude Opus 4.6	50	50	50	50	33	33	33	25	25	20	37.0%
Claude Sonnet 4	100	50	33	33	25	25	25	25	25	20	36.2%
Qwen 2.5 72B	100	100	25	20	20	20	17	14	13	11	34.0%
Llama 3.1 Nemotron 70B	100	100	33	20	20	14	13	11	11	5	32.7%
Ministral 8B	100	100	100	4	3	3	3	3	2	1	31.9%
Claude 3.7 Sonnet	50	33	33	33	33	25	25	25	25	25	30.8%
DeepSeek V3 (2024-12-26)	50	50	50	33	25	25	17	17	17	17	30.0%
Claude Opus 4	50	33	33	33	25	25	25	25	25	14	28.9%
Gemini 2.5 Flash Lite	100	100	17	14	14	10	9	8	7	2	28.1%
Claude Opus 4.5	33	33	33	33	25	25	25	25	25	20	27.8%
Claude 3.5 Sonnet	33	33	33	25	25	25	25	25	25	20	27.0%
DeepSeek V3 (2025-03-24)	50	33	25	25	25	25	25	20	17	17	26.2%
Qwen 3.5 Plus (2026-02-15)	50	50	50	17	14	11	11	11	9	7	23.0%
Ministral 3 8B	100	100	3	3	3	2	2	2	1	0	21.6%
GPT-4o, May 13th (temp=1)	50	25	25	17	17	17	17	17	17	14	21.4%
Gemini 3 Flash (Preview)	25	25	25	20	20	20	17	17	17	14	19.9%
DeepSeek V3.1	50	50	20	20	14	14	10	10	8	2	19.8%
GPT-4.1 Mini	50	50	17	17	17	14	11	8	7	3	19.4%
DeepSeek-V2 Chat	25	25	25	20	20	20	17	17	13	11	19.2%
Mistral Small 3.2 24B	100	14	13	11	11	11	9	8	8	5	19.1%
Claude 3 Haiku	25	25	25	25	20	17	17	17	11	10	19.1%
Gemma 3 12B	33	25	25	20	20	14	13	13	13	11	18.6%
Gemini 2.5 Flash	20	20	17	17	17	17	17	17	14	10	16.4%
Llama 3.1 8B	33	33	20	17	11	11	10	10	10	6	16.2%
GPT-4o, Aug. 6th (temp=0)	20	20	20	20	17	14	14	11	10	9	15.5%
Claude Sonnet 4.5	17	17	17	17	17	14	14	14	14	13	15.3%
DeepSeek V3.2	25	25	20	20	14	14	13	10	6	3	15.0%
Gemma 3 27B	20	17	17	17	14	14	14	14	13	9	14.9%
WizardLM 2 8x22b	50	25	13	10	8	8	7	7	7	6	14.1%
GPT-4o, May 13th (temp=0)	17	14	14	14	13	13	13	13	10	9	12.9%
Rocinante 12B	25	20	14	14	13	10	8	7	7	5	12.4%
Mistral NeMO	14	14	13	10	8	8	8	7	6	5	9.4%
Mistral Large	10	8	8	8	8	8	7	6	6	6	7.4%
Mistral Large 2	8	8	8	8	8	7	7	5	5	5	7.0%
Mistral Large 3	8	8	8	8	7	7	6	6	5	5	6.9%
Writer: Palmyra X5	10	9	9	8	7	6	5	5	4	3	6.7%
Gemma 3 4B	8	7	6	6	6	5	5	5	5	4	5.7%
Ministral 3B	14	11	8	6	4	3	3	3	2	2	5.6%
Ministral 3 14B	8	6	6	5	4	3	3	3	2	1	4.0%
Mistral Small Creative	5	5	5	4	4	4	3	3	3	3	3.9%
Ministral 3 3B	7	5	4	3	2	1	1	1	0	0	2.5%

▼

Short text (~524 words), big codex (51 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	50	95.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	50	95.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5.1	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	50	50	33	83.3%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	50	50	50	50	80.0%
GPT-5	100	100	100	100	100	100	50	50	50	50	80.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	50	50	50	50	80.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	33	33	1	76.8%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	50	33	68.3%
Gemini 3.1 Pro (Preview)	100	100	100	50	50	50	50	50	50	50	65.0%
Claude 3.7 Sonnet	100	100	100	100	50	50	50	33	33	33	65.0%
Claude Sonnet 4.6 (Reasoning)	100	100	50	50	50	50	50	50	50	33	58.3%
GPT-4.1 Mini	100	100	100	100	50	50	50	11	11	8	58.1%
Mistral Medium 3.1	100	100	100	100	50	33	33	9	8	7	54.1%
DeepSeek V3 (2025-03-24)	100	100	50	50	50	50	33	33	25	17	50.8%
Gemini 2.5 Flash	50	50	50	50	50	50	50	50	50	50	50.0%
WizardLM 2 8x22b	100	100	50	50	50	50	33	33	6	1	47.4%
Claude 3.5 Sonnet	100	100	50	50	33	33	33	25	25	20	47.0%
Claude Sonnet 4	100	100	100	33	25	25	20	20	20	20	46.3%
GPT-4o, Aug. 6th (temp=1)	50	50	50	50	50	50	50	50	33	25	45.8%
GPT-4o, Aug. 6th (temp=0)	50	50	50	50	50	50	50	33	33	33	45.0%
Claude Opus 4	100	50	50	50	33	33	33	33	33	25	44.2%
Claude Haiku 4.5	50	50	50	50	50	33	33	33	33	25	40.8%
Gemini 2.5 Flash Lite	100	100	100	25	25	25	7	5	5	4	39.5%
Qwen 2.5 72B	100	100	50	33	33	25	14	11	7	6	38.0%
DeepSeek V3 (2024-12-26)	50	50	50	50	50	33	33	25	20	17	37.8%
Qwen 3.5 Plus (2026-02-15)	50	50	50	50	50	33	33	17	14	13	36.0%
GPT-4o, May 13th (temp=0)	50	50	50	50	50	25	20	20	17	14	34.6%
Claude Sonnet 4.5	50	50	33	33	33	33	33	25	25	25	34.2%
DeepSeek V3.1	50	50	50	50	25	25	25	25	20	8	32.8%
DeepSeek-V2 Chat	50	50	50	33	33	33	25	25	17	9	32.6%
Writer: Palmyra X5	50	50	50	33	33	33	33	20	6	4	31.3%
Mistral Small 3.2 24B	100	100	25	25	11	11	10	10	10	10	31.2%
Hermes 3 405B	100	33	25	25	25	25	20	20	14	11	29.9%
DeepSeek V3.2	50	50	33	33	33	33	20	20	17	7	29.7%
Gemini 3 Flash (Preview)	50	33	33	33	25	25	25	25	20	20	29.0%
GPT-4o, May 13th (temp=1)	33	33	33	33	33	33	33	25	20	8	28.7%
Llama 3.1 70B	100	50	33	17	14	13	10	9	8	8	26.1%
GPT-4.1 Nano	100	100	33	5	5	4	2	2	2	1	25.4%
GPT-4o Mini (temp=0)	25	25	25	25	25	25	25	25	25	20	24.5%
Hermes 3 70B	50	25	25	25	20	17	17	17	14	14	22.4%
GPT-4o Mini (temp=1)	25	25	20	20	20	20	20	14	11	8	18.3%
Claude 3 Haiku	33	20	20	17	17	14	14	14	14	11	17.5%
Rocinante 12B	33	25	20	17	17	13	13	11	11	8	16.7%
Ministral 3 14B	100	7	7	6	5	3	2	1	1	0	13.1%
Ministral 3B	33	25	14	11	10	9	8	5	4	3	12.3%
Llama 3.1 Nemotron 70B	33	11	11	10	10	9	9	9	9	8	12.0%
Llama 3.1 8B	25	20	14	13	10	7	6	5	2	0	10.3%
Gemma 3 27B	11	11	10	10	10	10	10	10	9	8	10.0%
Gemma 3 12B	14	13	11	10	9	9	8	8	7	7	9.5%
Mistral Small Creative	11	11	8	6	6	5	5	5	5	3	6.5%
Ministral 3 3B	11	8	6	6	6	4	4	3	2	0	5.0%
Arcee AI: Trinity Large (Preview)	9	8	7	5	4	4	4	2	2	2	4.7%
Gemma 3 4B	5	5	3	3	2	2	2	2	2	2	2.8%
Mistral NeMO	7	6	4	3	3	2	1	0	0	0	2.5%

▼

Short text (~524 words), small codex (11 detailed entries)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	50	95.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	50	95.0%
o4 Mini	100	100	100	100	100	100	100	100	100	50	95.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	50	95.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	50	95.0%
Arcee AI: Trinity Mini	100	100	100	100	100	100	100	100	100	50	95.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	14	91.4%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	50	50	90.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	50	50	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	50	50	90.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	50	50	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	50	25	87.5%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	50	50	50	85.0%
Minimax M2.5	100	100	100	100	100	100	100	50	50	50	85.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	50	50	50	85.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	20	17	83.7%
Hermes 3 405B	100	100	100	100	100	100	100	33	25	25	78.3%
MoonshotAI: Kimi K2.5	100	100	100	100	100	50	50	50	50	50	75.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	50	50	50	50	50	75.0%
Gemini 2.5 Pro	100	100	100	100	100	50	50	50	50	50	75.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	50	50	50	50	50	50	70.0%
GPT-5	100	100	100	100	50	50	50	50	50	50	70.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	50	50	50	50	50	33	68.3%
Qwen 3.5 397B A17B	100	100	100	100	50	50	50	50	33	33	66.7%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	50	50	50	50	33	33	66.7%
Mistral Small 3.2 24B	100	100	100	100	100	100	17	13	11	10	65.0%
Llama 3.1 70B	100	100	100	100	50	50	50	33	33	25	64.2%
Gemini 3.1 Pro (Preview)	100	100	50	50	50	50	50	50	50	50	60.0%
Claude Sonnet 4	100	100	100	100	33	33	33	33	33	33	60.0%
Mistral Medium 3.1	100	100	100	50	50	50	50	33	33	14	58.1%
Z.AI GLM 4.5	100	100	100	100	100	25	8	8	7	6	55.4%
DeepSeek V3 (2025-03-24)	100	50	50	50	50	50	50	50	50	50	55.0%
Claude 3.5 Sonnet	100	100	100	50	33	33	33	33	25	20	52.8%
Claude Opus 4	100	100	50	50	50	33	33	33	33	33	51.7%
Rocinante 12B	100	100	100	100	50	20	13	8	8	6	50.4%
Hermes 3 70B	100	100	100	50	50	33	25	20	13	11	50.2%
DeepSeek V3 (2024-12-26)	50	50	50	50	50	50	50	50	50	50	50.0%
DeepSeek-V2 Chat	50	50	50	50	50	50	50	50	33	33	46.7%
DeepSeek V3.1	50	50	50	50	50	50	50	50	25	25	45.0%
Claude 3.7 Sonnet	100	100	50	33	25	25	25	25	25	20	42.8%
Writer: Palmyra X5	50	50	50	50	33	33	33	20	20	20	36.0%
Claude Sonnet 4.5	50	50	33	33	33	33	33	33	25	20	34.5%
Gemini 2.5 Flash	50	50	50	50	50	33	25	25	6	5	34.3%
GPT-4.1 Nano	100	100	100	9	7	7	6	5	4	4	34.1%
WizardLM 2 8x22b	50	50	50	50	33	25	25	20	17	8	32.8%
Gemini 3 Flash (Preview)	50	33	33	33	33	33	33	33	20	20	32.3%
Mistral Small Creative	100	100	33	20	17	14	8	8	8	7	31.5%
Qwen 2.5 72B	100	50	50	50	11	11	10	9	8	7	30.6%
DeepSeek V3.2	50	33	33	33	33	33	25	25	20	17	30.3%
Qwen 3.5 Plus (2026-02-15)	50	50	33	33	25	25	20	20	20	17	29.3%
Claude 3 Haiku	33	33	33	25	25	25	20	20	20	20	25.5%
GPT-4o, May 13th (temp=1)	50	33	33	33	25	20	20	17	8	7	24.7%
GPT-4o, May 13th (temp=0)	33	33	33	33	33	20	17	14	14	14	24.6%
Gemma 3 12B	33	33	25	25	25	25	25	17	14	11	23.4%
Llama 3.1 Nemotron 70B	33	25	25	25	20	20	20	17	11	8	20.4%
Gemini 2.5 Flash Lite	100	13	11	10	8	7	6	6	5	5	17.0%
Ministral 3B	50	33	17	14	10	8	8	7	7	7	16.2%
Gemma 3 27B	25	20	17	17	14	14	13	13	13	10	15.4%
GPT-4o Mini (temp=1)	20	17	14	13	11	10	10	8	8	8	11.8%
Ministral 3 3B	33	25	17	7	7	7	7	5	3	2	11.2%
Arcee AI: Trinity Large (Preview)	50	20	8	7	5	4	4	4	2	2	10.5%
Llama 3.1 8B	25	17	13	9	8	8	6	6	5	4	9.9%
Mistral NeMO	17	13	11	10	10	9	8	8	7	6	9.9%
Gemma 3 4B	11	11	10	10	9	9	9	8	7	7	9.2%
GPT-4o Mini (temp=0)	8	8	8	8	8	8	8	8	8	8	7.7%

No hallucinated violations

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

basic entries

Long text (~1594 words), big codex (51 entries)

Long text (~1594 words), small codex (11 entries)

Short text (~524 words), big codex (51 entries)

Short text (~524 words), small codex (11 entries)

detailed entries

Long text (~1594 words), big codex (51 detailed entries)

Long text (~1594 words), small codex (11 detailed entries)

Short text (~524 words), big codex (51 detailed entries)

Short text (~524 words), small codex (11 detailed entries)