Matches Regex

Test: Language Comprehension

Avg. Score

73.1%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Ministral 3 3B	100.0%	$0.0000	811ms	100%
2	Mistral Large 3	100.0%	$0.0001	4.2s	100%
3	DeepSeek V3 (2024-12-26)	100.0%	$0.0002	4.5s	100%
4	Qwen 3.5 Plus (2026-02-15)	100.0%	$0.0003	5.4s	100%
5	DeepSeek V3.1	100.0%	$0.0002	5.6s	100%
6	GPT-4o, May 13th (temp=0)	100.0%	$0.0022	2.9s	100%
7	Mistral Large 2	100.0%	$0.0012	4.9s	100%
8	DeepSeek-V2 Chat	100.0%	$0.0001	7.5s	100%
9	Claude Sonnet 4	100.0%	$0.0025	3.4s	100%
10	Claude Sonnet 4.6	100.0%	$0.0026	3.3s	100%
11	Claude Sonnet 4.5	100.0%	$0.0028	3.7s	100%
12	DeepSeek V3 (2025-03-24)	100.0%	$0.0001	9.8s	100%
13	Claude Opus 4.5	100.0%	$0.0043	3.7s	100%
14	Hermes 3 405B	100.0%	$0.0000	11.9s	100%
15	DeepSeek V3.2	100.0%	$0.0002	12.8s	100%
16	Claude Opus 4.6	100.0%	$0.0047	4.9s	100%
17	ByteDance Seed 1.6	100.0%	$0.0012	14.4s	100%
18	Gemini 2.5 Pro	100.0%	$0.0084	8.1s	100%
19	Z.AI GLM 4.7 Flash	100.0%	$0.0005	23.4s	100%
20	Claude Opus 4	100.0%	$0.0100	11.3s	100%
21	Gemini 3 Pro (Preview)	100.0%	$0.012	8.8s	100%
22	Gemini 3.1 Pro (Preview)	100.0%	$0.011	12.6s	100%
23	Z.AI GLM 4.6	100.0%	$0.0021	32.1s	100%
24	Qwen 3.5 397B A17B	100.0%	$0.0048	37.0s	100%
25	MoonshotAI: Kimi K2.5	100.0%	$0.0058	43.3s	100%
26	Z.AI GLM 4.7	100.0%	$0.0024	54.2s	100%
27	Z.AI GLM 5	100.0%	$0.0040	52.7s	100%
28	Mistral NeMO	90.0%	$0.0000	380ms	40%
29	Gemini 2.5 Flash Lite	90.0%	$0.0001	728ms	40%
30	Gemini 3 Flash (Preview)	90.0%	$0.0003	1.2s	40%
31	Z.AI GLM 4.5	90.0%	$0.0001	3.1s	40%
32	Claude Haiku 4.5	90.0%	$0.0009	2.1s	40%
33	GPT-4.1	90.0%	$0.0013	2.7s	40%
34	WizardLM 2 8x22b	90.0%	$0.0002	5.0s	40%
35	GPT-4o, May 13th (temp=1)	90.0%	$0.0020	2.8s	40%
36	GPT-5.2	90.0%	$0.0026	5.0s	40%
37	GPT-5 Mini	90.0%	$0.0015	12.5s	40%
38	GPT-5.1	90.0%	$0.0045	9.8s	40%
39	Minimax M2.5	90.0%	$0.0019	32.8s	40%
40	GPT-5	90.0%	$0.016	28.4s	40%
41	Gemini 2.5 Flash	80.0%	$0.0003	987ms	20%
42	Gemma 3 4B	80.0%	$0.0000	2.2s	20%
43	Claude 3.5 Haiku	80.0%	$0.0005	1.8s	20%
44	Mistral Large	80.0%	$0.0012	3.7s	20%
45	Claude 3.7 Sonnet	80.0%	$0.0024	3.4s	20%
46	Grok 4.1 Fast	80.0%	$0.0013	23.5s	20%
47	Claude 3 Haiku	70.0%	$0.0001	1.3s	8%
48	Grok 4 Fast	70.0%	$0.0004	5.7s	8%
49	Cohere Command R+ (Aug. 2024)	70.0%	$0.0028	5.5s	8%
50	o4 Mini	70.0%	$0.0026	7.4s	8%
51	o4 Mini High	70.0%	$0.0046	13.2s	8%
52	Arcee AI: Trinity Large (Preview)	60.0%	$0.0000	1.7s	2%
53	GPT-4.1 Mini	60.0%	$0.0004	2.8s	2%
54	Llama 3.1 70B	60.0%	$0.0002	3.4s	2%
55	Claude 3.5 Sonnet	60.0%	$0.0019	3.6s	2%
56	GPT-5 Nano	60.0%	$0.0006	17.5s	2%
57	Ministral 8B	50.0%	$0.0000	347ms	0%
58	GPT-4o Mini (temp=1)	50.0%	$0.0000	862ms	0%
59	GPT-4o Mini (temp=0)	50.0%	$0.0000	877ms	0%
60	Mistral Small 3.2 24B	50.0%	$0.0000	2.4s	0%
61	Hermes 3 70B	50.0%	$0.0001	3.2s	0%
62	Gemma 3 12B	50.0%	$0.0000	4.9s	0%
63	Rocinante 12B	50.0%	$0.0001	5.5s	0%
64	Writer: Palmyra X5	50.0%	$0.0017	7.8s	0%
65	Qwen 2.5 72B	40.0%	$0.0001	3.9s	0%
66	ByteDance Seed 1.6 Flash	40.0%	$0.0002	5.1s	0%
67	Stealth: Aurora Alpha	70.0%	—	1.4s	8%
68	Llama 3.1 Nemotron 70B	40.0%	$0.0002	13.5s	0%
69	Ministral 3B	30.0%	$0.0000	357ms	0%
70	Llama 3.1 8B	30.0%	$0.0000	1.4s	0%
71	GPT-4.1 Nano	30.0%	$0.0000	1.6s	0%
72	GPT-4o, Aug. 6th (temp=1)	30.0%	$0.0007	1.3s	0%
73	Gemma 3 27B	30.0%	$0.0001	8.2s	0%
74	Arcee AI: Trinity Mini	40.0%	$0.0001	32.5s	0%
75	Ministral 3 8B	0.0%	$0.0000	711ms	0%
76	Ministral 3 14B	0.0%	$0.0000	912ms	0%
77	Mistral Small Creative	0.0%	$0.0001	1.1s	0%
78	GPT-4o, Aug. 6th (temp=0)	0.0%	$0.0007	1.9s	0%
79	Mistral Medium 3.1	0.0%	$0.0004	3.6s	0%
80	Grok 4	70.0%	$0.110	3.2m	8%
73.13%

Individual Scenarios

▼

Asking for directions (Dutch)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100.0%
DeepSeek V3.2	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100.0%
DeepSeek V3.1	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100.0%
Ministral 3 3B	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	0	80.0%
o4 Mini High	100	100	100	100	0	80.0%
Minimax M2.5	100	100	100	100	0	80.0%
Gemini 3 Flash (Preview)	100	100	100	100	0	80.0%
Stealth: Aurora Alpha	100	100	100	100	0	80.0%
GPT-5 Nano	100	100	100	100	0	80.0%
Claude Haiku 4.5	100	100	100	100	0	80.0%
GPT-4.1 Mini	100	100	100	100	0	80.0%
Writer: Palmyra X5	100	100	100	100	0	80.0%
Gemini 2.5 Flash	100	100	100	100	0	80.0%
Gemini 2.5 Flash Lite	100	100	100	100	0	80.0%
Mistral NeMO	100	100	100	100	0	80.0%
WizardLM 2 8x22b	100	100	100	100	0	80.0%
Grok 4	100	100	100	0	0	60.0%
Claude 3.7 Sonnet	100	100	100	0	0	60.0%
Claude 3.5 Haiku	100	100	100	0	0	60.0%
Mistral Large	100	100	100	0	0	60.0%
Llama 3.1 8B	100	100	100	0	0	60.0%
Gemma 3 4B	100	100	100	0	0	60.0%
o4 Mini	100	100	0	0	0	40.0%
ByteDance Seed 1.6 Flash	100	100	0	0	0	40.0%
Arcee AI: Trinity Large (Preview)	100	100	0	0	0	40.0%
Claude 3 Haiku	100	100	0	0	0	40.0%
Arcee AI: Trinity Mini	100	100	0	0	0	40.0%
Cohere Command R+ (Aug. 2024)	100	100	0	0	0	40.0%
GPT-4.1 Nano	100	100	0	0	0	40.0%
Ministral 8B	100	100	0	0	0	40.0%
Ministral 3B	100	100	0	0	0	40.0%
Rocinante 12B	100	100	0	0	0	40.0%
Claude 3.5 Sonnet	100	0	0	0	0	20.0%
GPT-4o, Aug. 6th (temp=1)	100	0	0	0	0	20.0%
Llama 3.1 70B	100	0	0	0	0	20.0%
Llama 3.1 Nemotron 70B	100	0	0	0	0	20.0%
Hermes 3 70B	100	0	0	0	0	20.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0.0%
Gemma 3 27B	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0.0%
Qwen 2.5 72B	0	0	0	0	0	0.0%
Mistral Small 3.2 24B	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0.0%

▼

Asking for directions (German)

Model	# 1	# 2	# 3	# 4	# 5	Avg ▼
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100.0%
Minimax M2.5	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100.0%
Z.AI GLM 4.7 Flash	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100.0%
DeepSeek V3.2	100	100	100	100	100	100.0%
Claude 3.5 Haiku	100	100	100	100	100	100.0%
DeepSeek V3.1	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100.0%
GPT-4o Mini (temp=1)	100	100	100	100	100	100.0%
GPT-4o Mini (temp=0)	100	100	100	100	100	100.0%
Gemma 3 12B	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100.0%
Mistral Small 3.2 24B	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100.0%
Ministral 3 3B	100	100	100	100	100	100.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100.0%
Mistral NeMO	100	100	100	100	100	100.0%
Gemma 3 4B	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	0	80.0%
GPT-5	100	100	100	100	0	80.0%
GPT-5.2	100	100	100	100	0	80.0%
Grok 4	100	100	100	100	0	80.0%
GPT-4.1	100	100	100	100	0	80.0%
Z.AI GLM 4.5	100	100	100	100	0	80.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	0	80.0%
Gemini 2.5 Flash	100	100	100	100	0	80.0%
Qwen 2.5 72B	100	100	100	100	0	80.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	0	80.0%
Hermes 3 70B	100	100	100	100	0	80.0%
o4 Mini High	100	100	100	0	0	60.0%
Grok 4.1 Fast	100	100	100	0	0	60.0%
Stealth: Aurora Alpha	100	100	100	0	0	60.0%
Llama 3.1 Nemotron 70B	100	100	100	0	0	60.0%
Gemma 3 27B	100	100	100	0	0	60.0%
Ministral 8B	100	100	100	0	0	60.0%
Rocinante 12B	100	100	100	0	0	60.0%
GPT-5 Nano	100	100	0	0	0	40.0%
Grok 4 Fast	100	100	0	0	0	40.0%
GPT-4o, Aug. 6th (temp=1)	100	100	0	0	0	40.0%
GPT-4.1 Mini	100	100	0	0	0	40.0%
ByteDance Seed 1.6 Flash	100	100	0	0	0	40.0%
Arcee AI: Trinity Mini	100	100	0	0	0	40.0%
Writer: Palmyra X5	100	0	0	0	0	20.0%
GPT-4.1 Nano	100	0	0	0	0	20.0%
Ministral 3B	100	0	0	0	0	20.0%
GPT-4o, Aug. 6th (temp=0)	0	0	0	0	0	0.0%
Mistral Medium 3.1	0	0	0	0	0	0.0%
Mistral Small Creative	0	0	0	0	0	0.0%
Ministral 3 14B	0	0	0	0	0	0.0%
Ministral 3 8B	0	0	0	0	0	0.0%
Llama 3.1 8B	0	0	0	0	0	0.0%

Matches Regex

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

Asking for directions (Dutch)

Asking for directions (German)