Matches Regex

Test: Voice/dialogue sheets

Avg. Score

60.5%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Claude Sonnet 4.6	100.0%	$0.0033	1.9s	100%
2	Claude Sonnet 4	100.0%	$0.0033	2.8s	100%
3	Claude 3.7 Sonnet	100.0%	$0.0034	3.5s	100%
4	Claude Opus 4.6	100.0%	$0.0055	3.9s	100%
5	Claude Opus 4.6 (Reasoning)	100.0%	$0.0060	3.1s	100%
6	GPT-4.1	96.0%	$0.0017	2.6s	61%
7	GPT-4o, May 13th (temp=0)	96.0%	$0.0037	5.2s	61%
8	GPT-4o, Aug. 6th (temp=0)	92.0%	$0.0022	2.4s	46%
9	Gemini 2.5 Flash	88.0%	$0.0005	933ms	35%
10	ByteDance Seed 1.6	94.0%	$0.0013	14.2s	53%
11	Qwen 3.5 Plus (2026-02-15)	86.0%	$0.0005	6.6s	31%
12	Grok 4 Fast	84.0%	$0.0003	3.6s	27%
13	Grok 4.1 Fast	84.0%	$0.0004	4.4s	27%
14	GPT-4o, May 13th (temp=1)	90.0%	$0.0037	5.2s	40%
15	DeepSeek V3 (2025-03-24)	84.0%	$0.0003	6.1s	27%
16	Claude Haiku 4.5	82.0%	$0.0011	1.6s	23%
17	Gemini 3 Flash (Preview)	80.0%	$0.0006	1.8s	20%
18	Claude Sonnet 4.6 (Reasoning)	90.0%	$0.0053	3.3s	40%
19	Mistral Large 3	80.0%	$0.0004	4.1s	20%
20	GPT-4o, Aug. 6th (temp=1)	82.0%	$0.0022	2.3s	23%
21	DeepSeek-V2 Chat	80.0%	$0.0001	8.2s	20%
22	Claude Sonnet 4.5	84.0%	$0.0033	2.8s	27%
23	GPT-4.1 Mini	76.0%	$0.0003	2.7s	15%
24	Stealth: Healer Alpha	76.0%	$0.0000	5.0s	15%
25	Z.AI GLM 4.5	76.0%	$0.0004	5.3s	15%
26	Mistral Small 3.2 24B	72.0%	$0.0001	2.1s	10%
27	DeepSeek V3 (2024-12-26)	74.0%	$0.0003	4.6s	12%
28	Llama 3.1 70B	72.0%	$0.0005	1.5s	10%
29	Writer: Palmyra X5	78.0%	$0.0010	8.1s	17%
30	DeepSeek V3.1	76.0%	$0.0002	8.4s	15%
31	Gemini 3.1 Flash Lite (Preview)	70.0%	$0.0003	979ms	8%
32	Qwen3 235B A22B Instruct 2507	72.0%	$0.0001	4.9s	10%
33	Claude 3.5 Sonnet	80.0%	$0.0034	4.5s	20%
34	Gemini 2.5 Flash Lite (Reasoning)	70.0%	$0.0003	2.7s	8%
35	Gemini 2.5 Flash Lite	66.0%	$0.0001	610ms	5%
36	Z.AI GLM 5 Turbo	78.0%	$0.0030	7.7s	17%
37	Hermes 3 70B	68.0%	$0.0002	6.0s	7%
38	ByteDance Seed 1.6 Flash	66.0%	$0.0002	4.5s	5%
39	GPT-5.4 (Reasoning)	80.0%	$0.0049	5.4s	20%
40	Gemini 3 Flash (Preview, Reasoning)	72.0%	$0.0024	4.3s	10%
41	Mistral Large 2	70.0%	$0.0018	4.7s	8%
42	Cohere Command R+ (Aug. 2024)	70.0%	$0.0023	3.2s	8%
43	Mistral Small 4	60.0%	$0.0001	1.3s	2%
44	GPT-4o Mini (temp=0)	60.0%	$0.0001	3.7s	2%
45	Grok 4.20 (Beta)	60.0%	$0.0009	735ms	2%
46	Claude 3.5 Haiku	62.0%	$0.0009	4.6s	3%
47	Qwen 3.5 122B	88.0%	$0.0073	17.5s	35%
48	Llama 3.1 8B	54.0%	$0.0001	919ms	0%
49	GPT-5 Mini	72.0%	$0.0018	13.0s	10%
50	GPT-5.4 Mini (Reasoning)	64.0%	$0.0018	4.1s	4%
51	Stealth: Hunter Alpha	70.0%	$0.0000	18.7s	8%
52	ByteDance Seed 2.0 Lite	76.0%	$0.0018	20.6s	15%
53	Qwen 3.5 35B	82.0%	$0.0051	17.7s	23%
54	Gemma 3 12B	52.0%	$0.0000	4.1s	0%
55	MiniMax M2.7	56.0%	$0.0006	6.8s	1%
56	Z.AI GLM 4.7 Flash	70.0%	$0.0005	21.5s	8%
57	Arcee AI: Trinity Large (Preview)	48.0%	$0.0000	4.8s	0%
58	Claude Opus 4.5	70.0%	$0.0055	2.9s	8%
59	Hermes 3 405B	58.0%	$0.0000	13.3s	1%
60	Mistral Small 4 (Reasoning)	48.0%	$0.0004	4.2s	0%
61	DeepSeek V3.2	50.0%	$0.0002	6.7s	0%
62	GPT-4o Mini (temp=1)	60.0%	$0.0001	15.0s	2%
63	GPT-5.4 Mini (Reasoning, Low)	48.0%	$0.0010	2.2s	0%
64	Ministral 3 8B	40.0%	$0.0001	1.2s	0%
65	Qwen 3.5 Flash	70.0%	$0.0012	22.0s	8%
66	Rocinante 12B	52.0%	$0.0002	9.1s	0%
67	GPT-4.1 Nano	40.0%	$0.0001	2.0s	0%
68	GPT-5.4 Nano (Reasoning, Low)	40.0%	$0.0002	1.4s	0%
69	GPT-5.2	54.0%	$0.0025	2.1s	0%
70	Mistral Small Creative	38.0%	$0.0001	1.2s	0%
71	Grok 4.20 (Beta, Reasoning)	78.0%	$0.0075	9.9s	17%
72	GPT-5.4 (Reasoning, Low)	58.0%	$0.0034	3.2s	1%
73	GPT-5.1	64.0%	$0.0041	6.9s	4%
74	GPT-5.4 Mini	40.0%	$0.0009	1.1s	0%
75	MiniMax M2.5	48.0%	$0.0006	7.7s	0%
76	Claude 3 Haiku	40.0%	$0.0003	3.7s	0%
77	Inception Mercury	34.0%	$0.0001	961ms	0%
78	Aion 2.0	66.0%	$0.0015	20.9s	5%
79	WizardLM 2 8x22b	42.0%	$0.0004	7.6s	0%
80	o4 Mini	60.0%	$0.0036	8.6s	2%
81	Nemotron 3 Super	42.0%	$0.0000	9.8s	0%
82	Gemma 3 4B	30.0%	$0.0000	1.9s	0%
83	GPT-5.4 Nano	30.0%	$0.0002	1.4s	0%
84	Gemini 2.5 Flash (Reasoning)	38.0%	$0.0014	2.7s	0%
85	Inception Mercury 2	26.0%	$0.0004	823ms	0%
86	Gemma 3 27B	30.0%	$0.0001	5.0s	0%
87	GPT-5.4 Nano (Reasoning)	24.0%	$0.0003	1.5s	0%
88	Llama 3.1 Nemotron 70B	30.0%	$0.0002	6.9s	0%
89	Ministral 3 14B	20.0%	$0.0001	1.5s	0%
90	Mistral NeMO	22.0%	$0.0001	3.2s	0%
91	Mistral Medium 3.1	22.0%	$0.0004	3.2s	0%
92	Qwen 3.5 27B	66.0%	$0.0044	20.4s	5%
93	GPT-5.4	34.0%	$0.0030	2.0s	0%
94	Qwen 2.5 72B	20.0%	$0.0002	4.9s	0%
95	Qwen 3 32B	50.0%	$0.0006	24.6s	0%
96	Ministral 3B	10.0%	$0.0000	1.1s	0%
97	GPT-5 Nano	44.0%	$0.0007	21.1s	0%
98	ByteDance Seed 2.0 Mini	64.0%	$0.0007	37.7s	4%
99	Claude Opus 4	86.0%	$0.017	7.0s	31%
100	Ministral 8B	8.0%	$0.0001	1.5s	0%
101	Nemotron 3 Nano	22.0%	$0.0002	13.0s	0%
102	Ministral 3 3B	0.0%	$0.0001	835ms	0%
103	LFM2 24B	0.0%	$0.0000	2.9s	0%
104	Z.AI GLM 5	56.0%	$0.0035	26.7s	1%
105	Arcee AI: Trinity Mini	8.0%	$0.0001	8.1s	0%
106	Gemini 3.1 Pro (Preview)	74.0%	$0.013	12.9s	12%
107	Qwen 3.5 9B	76.0%	$0.0007	1.1m	15%
108	MoonshotAI: Kimi K2.5	50.0%	$0.0040	30.8s	0%
109	Z.AI GLM 4.6	54.0%	$0.0025	39.8s	0%
110	Grok 4	68.0%	$0.012	17.7s	7%
111	Stealth: Aurora Alpha	34.0%	—	1.7s	0%
112	Mistral Large	26.0%	$0.0070	6.2s	0%
113	GPT-5	58.0%	$0.011	19.2s	1%
114	Gemini 3 Pro (Preview)	68.0%	$0.016	10.7s	7%
115	Gemini 2.5 Pro	40.0%	$0.011	8.4s	0%
116	Qwen 3.5 397B A17B	82.0%	$0.0086	1.1m	23%
117	Z.AI GLM 4.7	60.0%	$0.0027	1.1m	2%
118	o4 Mini High	58.0%	$0.0052	58.3s	1%
60.47%

Individual Scenarios

▼

Multiple speakers

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3.2	100	100	100	100	100	100	100	100	0	0	80.0%
Aion 2.0	100	100	100	100	100	100	100	0	0	0	70.0%
DeepSeek V3.1	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	0	0	0	0	60.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 5	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4.1 Fast	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4	100	100	100	100	100	100	0	0	0	0	60.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4 Fast	100	100	100	100	100	100	0	0	0	0	60.0%
Stealth: Healer Alpha	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 27B	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Opus 4.5	100	100	100	100	100	0	0	0	0	0	50.0%
Z.AI GLM 4.6	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemma 3 12B	100	100	100	100	100	0	0	0	0	0	50.0%
Z.AI GLM 4.7	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 2.5 Pro	100	100	100	100	0	0	0	0	0	0	40.0%
Qwen 3.5 9B	100	100	100	100	0	0	0	0	0	0	40.0%
MiniMax M2.5	100	100	100	0	0	0	0	0	0	0	30.0%
Qwen 3.5 35B	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.7 Flash	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.1	100	100	0	0	0	0	0	0	0	0	20.0%
Grok 4.20 (Beta, Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-5.4 (Reasoning, Low)	100	100	0	0	0	0	0	0	0	0	20.0%
ByteDance Seed 2.0 Mini	100	100	0	0	0	0	0	0	0	0	20.0%
Qwen 3.5 Flash	100	100	0	0	0	0	0	0	0	0	20.0%
Qwen3 235B A22B Instruct 2507	100	100	0	0	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
ByteDance Seed 1.6 Flash	100	100	0	0	0	0	0	0	0	0	20.0%
Cohere Command R+ (Aug. 2024)	100	100	0	0	0	0	0	0	0	0	20.0%
MoonshotAI: Kimi K2.5	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Mini (Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.2	100	0	0	0	0	0	0	0	0	0	10.0%
MiniMax M2.7	100	0	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Stealth: Aurora Alpha	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash Lite	100	0	0	0	0	0	0	0	0	0	10.0%
Inception Mercury	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5	0	0	0	0	0	0	0	0	0	0	0.0%
o4 Mini High	0	0	0	0	0	0	0	0	0	0	0.0%
o4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini (Reasoning, Low)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
Nemotron 3 Super	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
Grok 4.20 (Beta)	0	0	0	0	0	0	0	0	0	0	0.0%
Inception Mercury 2	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 405B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 4 (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 3 32B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Writer: Palmyra X5	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano (Reasoning, Low)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 70B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Nemotron 3 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 4	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	0	90.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small 4	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	0	0	80.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Mini	100	100	100	100	100	100	100	0	0	0	70.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Large 2	100	100	100	100	100	100	100	0	0	0	70.0%
Llama 3.1 70B	100	100	100	100	100	100	100	0	0	0	70.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 122B	100	100	100	100	100	100	0	0	0	0	60.0%
Aion 2.0	100	100	100	100	100	100	0	0	0	0	60.0%
MiniMax M2.7	100	100	100	100	100	100	0	0	0	0	60.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 2.5 Flash	100	100	100	100	100	100	0	0	0	0	60.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	0	0	0	0	60.0%
Hermes 3 70B	100	100	100	100	100	100	0	0	0	0	60.0%
Rocinante 12B	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 397B A17B	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 27B	100	100	100	100	100	0	0	0	0	0	50.0%
MiniMax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
Stealth: Healer Alpha	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.2	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3 32B	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.2	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
Gemma 3 4B	100	100	100	100	0	0	0	0	0	0	40.0%
Llama 3.1 8B	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 5 Turbo	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.6	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Pro (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Flash (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Small 4 (Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 5	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-5.4 Mini (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
o4 Mini High	100	100	0	0	0	0	0	0	0	0	20.0%
Z.AI GLM 4.7	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Pro	100	100	0	0	0	0	0	0	0	0	20.0%
o4 Mini	100	100	0	0	0	0	0	0	0	0	20.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	0	0	0	0	0	0	0	0	20.0%
Gemma 3 27B	100	100	0	0	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Ministral 3B	100	100	0	0	0	0	0	0	0	0	20.0%
Grok 4	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash (Reasoning)	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Haiku 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Nemotron 3 Super	100	0	0	0	0	0	0	0	0	0	10.0%
Inception Mercury	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Nano	100	0	0	0	0	0	0	0	0	0	10.0%
WizardLM 2 8x22b	100	0	0	0	0	0	0	0	0	0	10.0%
Arcee AI: Trinity Mini	100	0	0	0	0	0	0	0	0	0	10.0%
Ministral 8B	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.1	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 (Reasoning, Low)	0	0	0	0	0	0	0	0	0	0	0.0%
MoonshotAI: Kimi K2.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Inception Mercury 2	0	0	0	0	0	0	0	0	0	0	0.0%
Stealth: Aurora Alpha	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano (Reasoning, Low)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Nemotron 3 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple (1-shot)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 12B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 4	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	0	90.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small Creative	100	100	100	100	100	100	100	100	100	0	90.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4	100	100	100	100	100	100	100	100	0	0	80.0%
Mistral Large 2	100	100	100	100	100	100	100	100	0	0	80.0%
Rocinante 12B	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3 32B	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	0	0	0	70.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	0	0	0	0	60.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	0	0	0	0	60.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 8B	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Aion 2.0	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
MiniMax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
Z.AI GLM 4.7	100	100	100	100	100	0	0	0	0	0	50.0%
Nemotron 3 Super	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.2	100	100	100	100	100	0	0	0	0	0	50.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 27B	100	100	100	100	0	0	0	0	0	0	40.0%
Stealth: Hunter Alpha	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-5 Nano	100	100	100	100	0	0	0	0	0	0	40.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral NeMO	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 5	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.6	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	0	0	0	0	0	0	0	30.0%
Stealth: Aurora Alpha	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Large	100	100	100	0	0	0	0	0	0	0	30.0%
Inception Mercury	100	100	100	0	0	0	0	0	0	0	30.0%
MiniMax M2.7	100	100	0	0	0	0	0	0	0	0	20.0%
Mistral Medium 3.1	100	100	0	0	0	0	0	0	0	0	20.0%
Ministral 3B	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Pro	100	0	0	0	0	0	0	0	0	0	10.0%
Inception Mercury 2	100	0	0	0	0	0	0	0	0	0	10.0%
Nemotron 3 Nano	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4 Nano	100	0	0	0	0	0	0	0	0	0	10.0%
Gemma 3 4B	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple (5-shot)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 12B	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 4	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small Creative	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100	100	100	100	100	100.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 4B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4	100	100	100	100	100	100	100	100	100	0	90.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.4	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	100	0	90.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	0	0	80.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3 32B	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5.4 Nano (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
Llama 3.1 8B	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 5	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
o4 Mini High	100	100	100	100	100	100	100	0	0	0	70.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	0	0	0	70.0%
DeepSeek V3.2	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Rocinante 12B	100	100	100	100	100	100	100	0	0	0	70.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	0	0	0	0	60.0%
Aion 2.0	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 4.6	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	0	0	0	0	60.0%
Mistral NeMO	100	100	100	100	100	100	0	0	0	0	60.0%
Nemotron 3 Super	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	0	0	0	0	0	0	0	30.0%
ByteDance Seed 2.0 Lite	100	100	100	0	0	0	0	0	0	0	30.0%
Stealth: Aurora Alpha	100	100	100	0	0	0	0	0	0	0	30.0%
Gemma 3 27B	100	100	100	0	0	0	0	0	0	0	30.0%
Ministral 8B	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 2.5 Flash (Reasoning)	100	100	0	0	0	0	0	0	0	0	20.0%
Inception Mercury 2	100	100	0	0	0	0	0	0	0	0	20.0%
Inception Mercury	100	100	0	0	0	0	0	0	0	0	20.0%
MiniMax M2.5	100	0	0	0	0	0	0	0	0	0	10.0%
Nemotron 3 Nano	100	0	0	0	0	0	0	0	0	0	10.0%
Ministral 3B	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Unattributed dialogue

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
MiniMax M2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 9B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury 2	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Inception Mercury	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 2.5 72B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100	100	0	90.0%
Aion 2.0	100	100	100	100	100	100	100	100	100	0	90.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100	100	0	90.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	0	90.0%
Nemotron 3 Nano	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 8B	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	0	0	80.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	0	0	80.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	0	0	0	0	60.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	0	0	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5.2	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.1	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3 32B	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	0	0	0	0	0	50.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Rocinante 12B	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Sonnet 4.5	100	100	100	100	0	0	0	0	0	0	40.0%
ByteDance Seed 2.0 Mini	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Opus 4	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.4 Nano	100	100	100	0	0	0	0	0	0	0	30.0%
Arcee AI: Trinity Mini	100	100	100	0	0	0	0	0	0	0	30.0%
Claude 3.5 Haiku	100	0	0	0	0	0	0	0	0	0	10.0%
Gemma 3 12B	100	0	0	0	0	0	0	0	0	0	10.0%
Mistral Small 4	100	0	0	0	0	0	0	0	0	0	10.0%
Mistral NeMO	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.4	0	0	0	0	0	0	0	0	0	0	0.0%
Grok 4.20 (Beta)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 405B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 2	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek V3.2	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5.4 Nano (Reasoning)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3B	0	0	0	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0	0	0	0.0%

Matches Regex

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

Multiple speakers

Simple

Simple (1-shot)

Simple (5-shot)

Unattributed dialogue