Matches Regex

Test: Voice/dialogue sheets

Avg. Score

61.5%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Claude Sonnet 4.6	100.0%	$0.0033	1.9s	100%
2	Claude Sonnet 4	100.0%	$0.0033	2.8s	100%
3	Claude 3.7 Sonnet	100.0%	$0.0034	3.5s	100%
4	Claude Opus 4.6	100.0%	$0.0055	3.9s	100%
5	GPT-4.1	96.0%	$0.0017	2.6s	61%
6	GPT-4o, May 13th (temp=0)	96.0%	$0.0037	5.2s	61%
7	GPT-4o, Aug. 6th (temp=0)	92.0%	$0.0022	2.4s	46%
8	Gemini 2.5 Flash	88.0%	$0.0005	933ms	35%
9	ByteDance Seed 1.6	94.0%	$0.0013	14.2s	53%
10	Qwen 3.5 Plus (2026-02-15)	86.0%	$0.0005	6.6s	31%
11	Grok 4 Fast	84.0%	$0.0003	3.6s	27%
12	Grok 4.1 Fast	84.0%	$0.0004	4.4s	27%
13	GPT-4o, May 13th (temp=1)	90.0%	$0.0037	5.2s	40%
14	DeepSeek V3 (2025-03-24)	84.0%	$0.0003	6.1s	27%
15	Claude Haiku 4.5	82.0%	$0.0011	1.6s	23%
16	Gemini 3 Flash (Preview)	80.0%	$0.0006	1.8s	20%
17	Mistral Large 3	80.0%	$0.0004	4.1s	20%
18	GPT-4o, Aug. 6th (temp=1)	82.0%	$0.0022	2.3s	23%
19	DeepSeek-V2 Chat	80.0%	$0.0001	8.2s	20%
20	Claude Sonnet 4.5	84.0%	$0.0033	2.8s	27%
21	GPT-4.1 Mini	76.0%	$0.0003	2.7s	15%
22	Z.AI GLM 4.5	76.0%	$0.0004	5.3s	15%
23	Mistral Small 3.2 24B	72.0%	$0.0001	2.1s	10%
24	DeepSeek V3 (2024-12-26)	74.0%	$0.0003	4.6s	12%
25	Llama 3.1 70B	72.0%	$0.0005	1.5s	10%
26	Writer: Palmyra X5	78.0%	$0.0010	8.1s	17%
27	DeepSeek V3.1	76.0%	$0.0002	8.4s	15%
28	Claude 3.5 Sonnet	80.0%	$0.0034	4.5s	20%
29	Gemini 2.5 Flash Lite	66.0%	$0.0001	610ms	5%
30	Hermes 3 70B	68.0%	$0.0002	6.0s	7%
31	ByteDance Seed 1.6 Flash	66.0%	$0.0002	4.5s	5%
32	Mistral Large 2	70.0%	$0.0018	4.7s	8%
33	Cohere Command R+ (Aug. 2024)	70.0%	$0.0023	3.2s	8%
34	GPT-4o Mini (temp=0)	60.0%	$0.0001	3.7s	2%
35	Claude 3.5 Haiku	62.0%	$0.0009	4.6s	3%
36	Llama 3.1 8B	54.0%	$0.0001	919ms	0%
37	GPT-5 Mini	72.0%	$0.0018	13.0s	10%
38	Gemma 3 12B	52.0%	$0.0000	4.1s	0%
39	Z.AI GLM 4.7 Flash	70.0%	$0.0005	21.5s	8%
40	Arcee AI: Trinity Large (Preview)	48.0%	$0.0000	4.8s	0%
41	Claude Opus 4.5	70.0%	$0.0055	2.9s	8%
42	Hermes 3 405B	58.0%	$0.0000	13.3s	1%
43	DeepSeek V3.2	50.0%	$0.0002	6.7s	0%
44	GPT-4o Mini (temp=1)	60.0%	$0.0001	15.0s	2%
45	Ministral 3 8B	40.0%	$0.0001	1.2s	0%
46	Rocinante 12B	52.0%	$0.0002	9.1s	0%
47	GPT-4.1 Nano	40.0%	$0.0001	2.0s	0%
48	GPT-5.2	54.0%	$0.0025	2.1s	0%
49	Mistral Small Creative	38.0%	$0.0001	1.2s	0%
50	GPT-5.1	64.0%	$0.0041	6.9s	4%
51	Minimax M2.5	48.0%	$0.0006	7.7s	0%
52	Claude 3 Haiku	40.0%	$0.0003	3.7s	0%
53	WizardLM 2 8x22b	42.0%	$0.0004	7.6s	0%
54	o4 Mini	60.0%	$0.0036	8.6s	2%
55	Gemma 3 4B	30.0%	$0.0000	1.9s	0%
56	Gemma 3 27B	30.0%	$0.0001	5.0s	0%
57	Llama 3.1 Nemotron 70B	30.0%	$0.0002	6.9s	0%
58	Ministral 3 14B	20.0%	$0.0001	1.5s	0%
59	Mistral NeMO	22.0%	$0.0001	3.2s	0%
60	Mistral Medium 3.1	22.0%	$0.0004	3.2s	0%
61	Qwen 2.5 72B	20.0%	$0.0002	4.9s	0%
62	Ministral 3B	10.0%	$0.0000	1.1s	0%
63	GPT-5 Nano	44.0%	$0.0007	21.1s	0%
64	Claude Opus 4	86.0%	$0.017	7.0s	31%
65	Ministral 8B	8.0%	$0.0001	1.5s	0%
66	Ministral 3 3B	0.0%	$0.0001	835ms	0%
67	Z.AI GLM 5	56.0%	$0.0035	26.7s	1%
68	Arcee AI: Trinity Mini	8.0%	$0.0001	8.1s	0%
69	Gemini 3.1 Pro (Preview)	74.0%	$0.013	12.9s	12%
70	MoonshotAI: Kimi K2.5	50.0%	$0.0040	30.8s	0%
71	Z.AI GLM 4.6	54.0%	$0.0025	39.8s	0%
72	Grok 4	68.0%	$0.012	17.7s	7%
73	Stealth: Aurora Alpha	34.0%	—	1.7s	0%
74	Mistral Large	26.0%	$0.0070	6.2s	0%
75	GPT-5	58.0%	$0.011	19.2s	1%
76	Gemini 3 Pro (Preview)	68.0%	$0.016	10.7s	7%
77	Gemini 2.5 Pro	40.0%	$0.011	8.4s	0%
78	Qwen 3.5 397B A17B	82.0%	$0.0086	1.1m	23%
79	Z.AI GLM 4.7	60.0%	$0.0027	1.1m	2%
80	o4 Mini High	58.0%	$0.0052	58.3s	1%
61.48%

Individual Scenarios

▼

Multiple speakers

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3.2	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3.1	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 5	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4.1 Fast	100	100	100	100	100	100	0	0	0	0	60.0%
Grok 4 Fast	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	0	0	0	0	60.0%
Claude Opus 4.5	100	100	100	100	100	0	0	0	0	0	50.0%
Z.AI GLM 4.6	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	0	0	0	0	0	50.0%
Gemma 3 12B	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 2.5 Pro	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 4.7	100	100	100	100	0	0	0	0	0	0	40.0%
Minimax M2.5	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.7 Flash	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	0	0	0	0	0	0	0	30.0%
GPT-5.1	100	100	0	0	0	0	0	0	0	0	20.0%
ByteDance Seed 1.6 Flash	100	100	0	0	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Cohere Command R+ (Aug. 2024)	100	100	0	0	0	0	0	0	0	0	20.0%
MoonshotAI: Kimi K2.5	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.2	100	0	0	0	0	0	0	0	0	0	10.0%
Stealth: Aurora Alpha	100	0	0	0	0	0	0	0	0	0	10.0%
Z.AI GLM 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Gemini 2.5 Flash Lite	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
o4 Mini High	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5	0	0	0	0	0	0	0	0	0	0	0.0%
o4 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Sonnet	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 3	0	0	0	0	0	0	0	0	0	0	0.0%
DeepSeek-V2 Chat	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3.5 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Writer: Palmyra X5	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 405B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Rocinante 12B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	100	100	0	90.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Mini	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Mistral Large 2	100	100	100	100	100	100	100	0	0	0	70.0%
Llama 3.1 70B	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash	100	100	100	100	100	100	0	0	0	0	60.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	0	0	0	0	60.0%
Hermes 3 70B	100	100	100	100	100	100	0	0	0	0	60.0%
Rocinante 12B	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Qwen 3.5 397B A17B	100	100	100	100	100	0	0	0	0	0	50.0%
Minimax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.2	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5.2	100	100	100	100	0	0	0	0	0	0	40.0%
Llama 3.1 8B	100	100	100	100	0	0	0	0	0	0	40.0%
Gemma 3 4B	100	100	100	100	0	0	0	0	0	0	40.0%
Gemini 3 Pro (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.6	100	100	100	0	0	0	0	0	0	0	30.0%
Gemini 3 Flash (Preview)	100	100	100	0	0	0	0	0	0	0	30.0%
o4 Mini High	100	100	0	0	0	0	0	0	0	0	20.0%
o4 Mini	100	100	0	0	0	0	0	0	0	0	20.0%
Z.AI GLM 5	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Pro	100	100	0	0	0	0	0	0	0	0	20.0%
Z.AI GLM 4.7	100	100	0	0	0	0	0	0	0	0	20.0%
Gemma 3 27B	100	100	0	0	0	0	0	0	0	0	20.0%
Arcee AI: Trinity Large (Preview)	100	100	0	0	0	0	0	0	0	0	20.0%
Ministral 3B	100	100	0	0	0	0	0	0	0	0	20.0%
Grok 4	100	0	0	0	0	0	0	0	0	0	10.0%
Claude Haiku 4.5	100	0	0	0	0	0	0	0	0	0	10.0%
Arcee AI: Trinity Mini	100	0	0	0	0	0	0	0	0	0	10.0%
Ministral 8B	100	0	0	0	0	0	0	0	0	0	10.0%
WizardLM 2 8x22b	100	0	0	0	0	0	0	0	0	0	10.0%
GPT-5.1	0	0	0	0	0	0	0	0	0	0	0.0%
MoonshotAI: Kimi K2.5	0	0	0	0	0	0	0	0	0	0	0.0%
Claude Opus 4.5	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5	0	0	0	0	0	0	0	0	0	0	0.0%
Stealth: Aurora Alpha	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-5 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple (1-shot)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 12B	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	100	0	90.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Small Creative	100	100	100	100	100	100	100	100	100	0	90.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4	100	100	100	100	100	100	100	100	0	0	80.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	0	0	80.0%
Mistral Large 2	100	100	100	100	100	100	100	100	0	0	80.0%
Rocinante 12B	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	0	0	0	70.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	0	0	0	0	60.0%
Llama 3.1 8B	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Z.AI GLM 4.7	100	100	100	100	100	0	0	0	0	0	50.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Minimax M2.5	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.2	100	100	100	100	100	0	0	0	0	0	50.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
GPT-5 Nano	100	100	100	100	0	0	0	0	0	0	40.0%
Mistral NeMO	100	100	100	100	0	0	0	0	0	0	40.0%
Z.AI GLM 5	100	100	100	0	0	0	0	0	0	0	30.0%
Z.AI GLM 4.6	100	100	100	0	0	0	0	0	0	0	30.0%
Stealth: Aurora Alpha	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Large	100	100	100	0	0	0	0	0	0	0	30.0%
Mistral Medium 3.1	100	100	0	0	0	0	0	0	0	0	20.0%
Ministral 3B	100	100	0	0	0	0	0	0	0	0	20.0%
Gemini 2.5 Pro	100	0	0	0	0	0	0	0	0	0	10.0%
Gemma 3 4B	100	0	0	0	0	0	0	0	0	0	10.0%
Gemma 3 27B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Simple (5-shot)

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 12B	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small Creative	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100	100	100	100.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 4B	100	100	100	100	100	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5	100	100	100	100	100	100	100	100	100	0	90.0%
GPT-5.2	100	100	100	100	100	100	100	100	100	0	90.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4	100	100	100	100	100	100	100	100	100	0	90.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100	100	0	90.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100	100	0	90.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	0	0	80.0%
o4 Mini	100	100	100	100	100	100	100	100	0	0	80.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	0	0	80.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	100	0	0	80.0%
DeepSeek V3.1	100	100	100	100	100	100	100	100	0	0	80.0%
Llama 3.1 8B	100	100	100	100	100	100	100	100	0	0	80.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	0	0	0	70.0%
o4 Mini High	100	100	100	100	100	100	100	0	0	0	70.0%
Z.AI GLM 5	100	100	100	100	100	100	100	0	0	0	70.0%
DeepSeek V3.2	100	100	100	100	100	100	100	0	0	0	70.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Rocinante 12B	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	0	0	0	0	60.0%
Z.AI GLM 4.6	100	100	100	100	100	100	0	0	0	0	60.0%
Mistral NeMO	100	100	100	100	100	100	0	0	0	0	60.0%
Gemini 2.5 Pro	100	100	100	0	0	0	0	0	0	0	30.0%
Stealth: Aurora Alpha	100	100	100	0	0	0	0	0	0	0	30.0%
Gemma 3 27B	100	100	100	0	0	0	0	0	0	0	30.0%
Ministral 8B	100	100	100	0	0	0	0	0	0	0	30.0%
Minimax M2.5	100	0	0	0	0	0	0	0	0	0	10.0%
Ministral 3B	100	0	0	0	0	0	0	0	0	0	10.0%
Qwen 2.5 72B	0	0	0	0	0	0	0	0	0	0	0.0%
Arcee AI: Trinity Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%

▼

Unattributed dialogue

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	# 8	# 9	# 10	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Minimax M2.5	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100	100	100	100.0%
Stealth: Aurora Alpha	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-5 Nano	100	100	100	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100	100	100	100.0%
Gemma 3 27B	100	100	100	100	100	100	100	100	100	100	100.0%
Qwen 2.5 72B	100	100	100	100	100	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100	100	0	90.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100	100	0	90.0%
Llama 3.1 8B	100	100	100	100	100	100	100	100	100	0	90.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100	0	0	80.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100	0	0	80.0%
Hermes 3 70B	100	100	100	100	100	100	100	100	0	0	80.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	0	0	0	70.0%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	100	0	0	0	70.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	0	0	0	70.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	0	0	0	0	60.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	0	0	0	0	60.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	0	0	0	0	60.0%
GPT-5.2	100	100	100	100	100	0	0	0	0	0	50.0%
DeepSeek V3.1	100	100	100	100	100	0	0	0	0	0	50.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	0	0	0	0	0	50.0%
Rocinante 12B	100	100	100	100	100	0	0	0	0	0	50.0%
Claude Sonnet 4.5	100	100	100	100	0	0	0	0	0	0	40.0%
Claude Opus 4	100	100	100	0	0	0	0	0	0	0	30.0%
Arcee AI: Trinity Mini	100	100	100	0	0	0	0	0	0	0	30.0%
Claude 3.5 Haiku	100	0	0	0	0	0	0	0	0	0	10.0%
Gemma 3 12B	100	0	0	0	0	0	0	0	0	0	10.0%
Mistral NeMO	100	0	0	0	0	0	0	0	0	0	10.0%
DeepSeek V3.2	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Mini	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large 2	0	0	0	0	0	0	0	0	0	0	0.0%
Hermes 3 405B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0	0	0	0.0%
Llama 3.1 Nemotron 70B	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Large	0	0	0	0	0	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0	0	0	0	0	0.0%
Claude 3 Haiku	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3 3B	0	0	0	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0	0	0	0.0%
Gemma 3 4B	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 8B	0	0	0	0	0	0	0	0	0	0	0.0%
WizardLM 2 8x22b	0	0	0	0	0	0	0	0	0	0	0.0%
Ministral 3B	0	0	0	0	0	0	0	0	0	0	0.0%

Matches Regex

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

Multiple speakers

Simple

Simple (1-shot)

Simple (5-shot)

Unattributed dialogue