Mara pronouns preserved (coreference test)

Test: Text Replacement

Avg. Score
89.5%
Scenarios
2

Overall Performance

Rank ▲ Model Score Avg. Cost Avg. Time Stability
1Gemini 3.1 Flash Lite (Reasoning)100.0%$0.00112.0s100%
2GPT-6 Luna100.0%$0.00044.8s100%
3Gemini 3.5 Flash Lite (Reasoning)100.0%$0.00171.6s100%
4Gemini 3.5 Flash Lite (Reasoning, Minimal)100.0%$0.00171.8s100%
5Gemini 3.1 Flash Lite100.0%$0.00113.9s100%
6Gemini 2.5 Flash100.0%$0.00172.4s100%
7GPT-6 Luna (Reasoning, Medium)100.0%$0.00046.7s100%
8GPT-6 Luna (Reasoning, High)100.0%$0.00057.6s100%
9Claude Haiku 5.5 (Adaptive)100.0%$0.00106.7s100%
10Z.AI GLM 5.3 Flash (Reasoning, Low)100.0%$0.00029.3s100%
11Gemini 3 Flash (Preview)100.0%$0.00213.6s100%
12Claude Haiku 5.5 (Reasoning, Medium)100.0%$0.00107.1s100%
13Claude Haiku 5.5 (Reasoning, Low)100.0%$0.00107.3s100%
14GPT-4.1 Mini100.0%$0.00127.1s100%
15Mistral Medium 3.1100.0%$0.00156.5s100%
16Mistral Large 4.0100.0%$0.00169.2s100%
17Gemma 4 26B100.0%$0.000313.5s100%
18GPT-5.6 Luna100.0%$0.00422.5s100%
19Gemini 3.1 Flash Lite (Preview)99.4%$0.00111.9s95%
20Claude Haiku 4.5100.0%$0.00423.3s100%
21Gemini 3.6 Flash (Reasoning, Minimal)100.0%$0.00552.6s100%
22Mistral Medium 3.5 (Reasoning)100.0%$0.00553.1s100%
23Mistral Medium 3.5100.0%$0.00553.3s100%
24GPT-5.6 Luna (Reasoning, Medium)100.0%$0.00534.0s100%
25DeepSeek V4.1 Flash (Reasoning, High)100.0%$0.002013.9s100%
26Xiaomi MIMO v2.6 Pro100.0%$0.001416.6s100%
27Gemini 3.5 Flash (Reasoning, Minimal)100.0%$0.00642.9s100%
28GPT-4.1100.0%$0.00614.7s100%
29Muse Glimmer 30B (Reasoning, Medium)100.0%$0.002817.2s100%
30GPT-4o, Aug. 6th (temp=0)100.0%$0.00762.8s100%
31GPT-4o, Aug. 6th (temp=1)100.0%$0.00763.2s100%
32Gemma 4 31B100.0%$0.000427.2s100%
33GPT-6 Sol100.0%$0.00726.8s100%
34Muse Spark 1.1 (Reasoning, Minimal)100.0%$0.00698.1s100%
35Hermes 3 405B100.0%$0.001425.1s100%
36DeepSeek V4 Flash (Reasoning)100.0%$0.000528.0s100%
37DeepSeek V4 Pro100.0%$0.001925.2s100%
38Gemini 3.7 Flash (Reasoning, Medium)100.0%$0.00877.0s100%
39Qwen 3.5 Plus (2026-02-15)98.7%$0.00177.7s91%
40GPT-5.6 Terra100.0%$0.0103.4s100%
41Qwen 3 32B100.0%$0.000633.7s100%
42Grok 4.20 (Reasoning)100.0%$0.006018.4s100%
43GPT-5.6 Terra (Reasoning, Medium)100.0%$0.0113.6s100%
44Gemini 3.8 Flash (Reasoning, Medium)100.0%$0.00997.5s100%
45GPT-5.4100.0%$0.0106.1s100%
46Gemini 3 Flash (Preview, Reasoning)100.0%$0.008313.3s100%
47Z.AI GLM 5.3 Flash (Reasoning, Max)100.0%$0.000638.4s100%
48GPT-5.4 (Reasoning, Low)100.0%$0.0116.7s100%
49GPT-6 Sol (Reasoning, Medium)100.0%$0.01010.2s100%
50Muse Spark 1.1 (Reasoning, Medium)100.0%$0.01011.4s100%
51Claude Sonnet 4.6100.0%$0.0135.1s100%
52Claude Sonnet 4.5100.0%$0.0135.2s100%
53Claude Sonnet 5.5 (Reasoning)100.0%$0.0125.9s100%
54Claude Sonnet 5100.0%$0.0119.5s100%
55Claude Sonnet 4100.0%$0.0136.5s100%
56Xiaomi MIMO v2.6 Flash100.0%$0.000543.0s100%
57Z.AI GLM 4.5100.0%$0.004132.4s100%
58Qwen 3.6 Flash100.0%$0.007522.4s100%
59Qwen 3.6 35B100.0%$0.005330.7s100%
60Muse Spark 1.2 (Reasoning, Medium)100.0%$0.01210.1s100%
61Gemini 2.5 Flash (Reasoning)98.7%$0.00598.8s91%
62ByteDance Seed 1.6100.0%$0.003537.7s100%
63Claude Sonnet 5 (Reasoning, Low)100.0%$0.01310.7s100%
64Claude Sonnet 5 (Reasoning)100.0%$0.01311.4s100%
65Grok 4.5 (Reasoning, Low)100.0%$0.01023.0s100%
66MiniMax M3100.0%$0.002846.7s100%
67Grok 4.3 (Reasoning)100.0%$0.007632.7s100%
68Aion 2.0100.0%$0.003847.6s100%
69Grok 4.5 (Reasoning, High)100.0%$0.01520.4s100%
70o4 Mini100.0%$0.01421.5s100%
71ByteDance Seed 2.0 Lite100.0%$0.004552.2s100%
72DeepSeek V4 Pro 0813 (Reasoning, High)100.0%$0.01131.5s100%
73Aion 3.0 Mini100.0%$0.003455.5s100%
74GPT-5.1100.0%$0.01716.5s100%
75Z.AI GLM 5.3 (Reasoning, Max)100.0%$0.01132.9s100%
76GPT-5.5100.0%$0.0215.3s100%
77Claude Opus 4.5100.0%$0.0215.8s100%
78Claude Opus 4.6100.0%$0.0216.5s100%
79Grok 4.7 (Reasoning, High)100.0%$0.01427.1s100%
80Muse Spark 1.3 (Reasoning, Medium)100.0%$0.01428.8s100%
81Gemma 4 31B (Reasoning)100.0%$0.00101.2m100%
82Grok 4.6 (Reasoning, High)100.0%$0.01530.3s100%
83Z.AI GLM 5100.0%$0.007454.4s100%
84Aion 3.5 Mini (Reasoning, High)100.0%$0.00461.1m100%
85Z.AI GLM 4.7100.0%$0.00591.0m100%
86GPT-5.6 Sol (Reasoning, Medium)100.0%$0.0248.2s100%
87GPT-5.5 (Reasoning, Low)100.0%$0.0256.6s100%
88Gemini 3.6 Flash (Reasoning)100.0%$0.02411.2s100%
89Mistral Large 4.0 (Reasoning)100.0%$0.009655.4s100%
90Qwen 3.5 27B100.0%$0.01249.5s100%
91Kimi K3 (Reasoning, Low)100.0%$0.01831.1s100%
92MiniMax M2.7100.0%$0.00651.1m100%
93Claude Opus 4.7100.0%$0.0285.1s100%
94Aion 3.5 (Reasoning, High)100.0%$0.01549.7s100%
95Claude Opus 5100.0%$0.0289.0s100%
96Claude Opus 4.8 (Reasoning, Low)100.0%$0.0299.1s100%
97Z.AI GLM 4.698.7%$0.00501.0m91%
98Gemini 3.5 Flash (Reasoning)100.0%$0.02912.0s100%
99Aion 3.0100.0%$0.01751.0s100%
100o4 Mini High100.0%$0.02233.0s100%
101Gemma 3 27B89.0%$0.000218.1s72%
102Inception Mercury 287.0%$0.00192.6s68%
103GPT-5.5 (Reasoning, Medium)100.0%$0.03110.1s100%
104Qwen 3.6 27B100.0%$0.0141.0m100%
105Gemini 2.5 Pro100.0%$0.02820.4s100%
106Qwen 3.5 122B100.0%$0.02046.9s100%
107Claude Opus 5.5 (Reasoning)100.0%$0.03212.5s100%
108Claude Opus 4.8 (Reasoning)100.0%$0.03310.4s100%
109Z.AI GLM 4.5 Air92.2%$0.001944.5s78%
110Z.AI GLM 5.1100.0%$0.0131.2m100%
111GPT-OSS 120B90.9%$0.001140.9s74%
112Ember-1 (Reasoning, Max)100.0%$0.02833.9s100%
113 Kimi K3 (Reasoning, High)100.0%$0.02545.1s100%
114GPT-5.6 Sol92.2%$0.0214.5s82%
115DeepSeek V4 Pro (Reasoning)100.0%$0.00801.7m100%
116Claude Opus 5 (Reasoning)100.0%$0.03812.9s100%
117Claude Opus 5 (Reasoning, Low)100.0%$0.03813.1s100%
118Laguna S 2.187.7%$0.000218.5s59%
119Muse Wonder (Reasoning, Medium)100.0%$0.0416.4s100%
120Kimi K2.6100.0%$0.0151.5m100%
121Claude Sonnet 4.6 (Reasoning)100.0%$0.03820.1s100%
122Kimi K2.5100.0%$0.00851.8m100%
123Thinking Machines Inkling (Reasoning)100.0%$0.0131.6m100%
124Claude Opus 4.7 (Reasoning)100.0%$0.0438.1s100%
125Gemma 4 26B (Reasoning)100.0%$0.00172.2m100%
126ByteDance Seed 1.6 Flash90.3%$0.000712.3s47%
127Claude Opus 4.6 (Reasoning)100.0%$0.04316.8s100%
128Xiaomi MIMO v2.592.9%$0.004317.1s48%
129Ministral 3 3B80.5%$0.00012.2s50%
130Thinking Machines Inkling92.9%$0.002925.2s48%
131Gemini 3.1 Pro (Preview)100.0%$0.03836.2s100%
132Qwen 3.7 Max100.0%$0.03254.8s100%
133GPT-5.4 Nano (Reasoning, Low)86.4%$0.00098.4s44%
134GPT-5.292.9%$0.0107.1s48%
135Grok 4.2088.3%$0.00224.8s41%
136Llama 3.1 70B92.9%$0.000536.2s48%
137GPT-5100.0%$0.03549.7s100%
138Qwen 3 235B A22B Instruct 250781.8%$0.000516.2s52%
139Z.AI GLM 5.2 (Reasoning, High)92.9%$0.007922.5s48%
140MiniMax M2.595.5%$0.00171.6m72%
141Z.AI GLM 5 Turbo92.9%$0.009519.7s48%
142Qwen 3.8 Flash (Reasoning)92.9%$0.002045.3s48%
143GPT-5 Mini92.2%$0.005732.9s49%
144GPT-5.4 Nano81.8%$0.00093.5s37%
145Arcee AI: Trinity Mini63.6%$0.00027.1s64%
146Ministral 3B77.3%$0.00012.3s41%
147Thinking Machines Inkling Small (Reasoning, High)92.9%$0.006241.6s48%
148Gemma 3 4B77.3%$0.00016.5s42%
149DeepSeek V4 Flash 0731 (Reasoning, High)99.4%$0.00112.8m95%
150Laguna XS 2.183.8%$0.000314.9s34%
151GPT-5.4 (Reasoning)92.9%$0.02016.0s48%
152Xiaomi MIMO v2.5 Pro85.7%$0.002913.0s30%
153Mistral Small 4 (Reasoning)84.4%$0.001916.2s30%
154Qwen 3.7 Flash (Reasoning)85.7%$0.000624.5s30%
155Claude Opus 4100.0%$0.0638.9s100%
156GPT-5.4 Mini (Reasoning)85.7%$0.00716.2s30%
157Qwen 3.6 Max Preview100.0%$0.0331.7m100%
158Writer: Palmyra X577.3%$0.004013.0s37%
159Qwen 3.8 27B (Reasoning, XHigh)92.9%$0.01157.6s48%
160Hy4 Preview (Reasoning, High)100.0%$0.0262.3m100%
161Cydonia 24B V4.173.4%$0.000513.6s27%
162Qwen 2.5 72B78.6%$0.000311.5s18%
163Qwen 3.5 Plus (2026-04-20)92.9%$0.0121.2m48%
164Nemotron 3 Super82.5%$0.000057.7s31%
165GPT-5.4 Mini (Reasoning, Low)72.1%$0.00362.9s19%
166GPT-5.4 Nano (Reasoning)68.8%$0.00103.7s20%
167Qwen 3.5 9B89.6%$0.00141.9m45%
168DeepSeek-V2 Chat71.4%$0.000917.7s10%
169Mistral Small 461.7%$0.00043.4s15%
170GPT-5.4 Mini55.8%$0.00312.3s27%
171Ministral 3 14B61.0%$0.00034.5s13%
172DeepSeek V3 (2025-03-24)68.8%$0.000726.6s11%
173GPT-5 Nano74.0%$0.00291.0m24%
174Qwen 3.5 35B80.5%$0.01652.7s30%
175DeepSeek V3 (2024-12-26)64.3%$0.000918.3s4%
176Mistral Small 3.2 24B56.5%$0.00025.5s6%
177Qwen 3.5 Flash71.4%$0.00331.0m17%
178Ministral 3 8B52.6%$0.00023.7s3%
179Qwen 3.5 397B A17B85.7%$0.00752.2m30%
180Ministral 8B50.6%$0.00013.9s1%
181Gemini 2.5 Flash Lite50.0%$0.00031.9s0%
182Mistral NeMO50.0%$0.00022.7s0%
183Z.AI GLM 4.7 Flash63.0%$0.00161.1m15%
184DeepSeek V4 Flash50.0%$0.00028.7s0%
185Mistral Large 350.0%$0.00128.4s0%
186GPT-6.1 Sol (Reasoning, Medium)57.1%$0.007810.6s1%
187Grok 4.350.0%$0.00245.9s0%
188WizardLM 2 8x22b57.1%$0.000840.0s1%
189Mistral Large 250.0%$0.00508.4s0%
190Gemini 4 Argon (Reasoning, Medium)50.0%$0.00754.8s0%
191GPT-4.1 Nano40.3%$0.00034.2s0%
192Gemini 2.5 Flash Lite (Reasoning)44.2%$0.002315.5s2%
193GPT-4o Mini (temp=0)31.8%$0.000510.8s12%
194GPT-4o Mini (temp=1)31.8%$0.000511.0s12%
195DeepSeek V3.250.0%$0.000638.9s0%
196Qwen 3.8 Max (Reasoning, XHigh)85.7%$0.0301.8m30%
197Gemma 3 12B29.2%$0.00019.6s8%
198DeepSeek V3.146.1%$0.000643.9s1%
199Cohere Command R+ (Aug. 2024)50.0%$0.007932.6s0%
200Hermes 3 70B35.7%$0.000421.2s0%
201ByteDance Seed 2.0 Mini50.0%$0.00202.0m0%
202Nemotron 3 Nano57.8%$0.00223.2m17%
89.47%

Individual Scenarios

Generic Prompt

Model # 1 # 2 # 3 # 4 # 5 # 6 # 7 Avg ▼
Claude Opus 5.5 (Reasoning)100100100100100100100100.0%
Gemini 3.8 Flash (Reasoning, Medium)100100100100100100100100.0%
Ember-1 (Reasoning, Max)100100100100100100100100.0%
Muse Wonder (Reasoning, Medium)100100100100100100100100.0%
Z.AI GLM 5.3 (Reasoning, Max)100100100100100100100100.0%
GPT-5.6 Sol (Reasoning, Medium)100100100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning)100100100100100100100100.0%
Gemini 3.7 Flash (Reasoning, Medium)100100100100100100100100.0%
Hy4 Preview (Reasoning, High)100100100100100100100100.0%
Muse Spark 1.2 (Reasoning, Medium)100100100100100100100100.0%
Qwen 3.7 Max100100100100100100100100.0%
DeepSeek V4.1 Flash (Reasoning, High)100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Max)100100100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100100100.0%
Muse Spark 1.3 (Reasoning, Medium)100100100100100100100100.0%
GPT-6 Luna (Reasoning, High)100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Medium)100100100100100100100100.0%
Z.AI GLM 5.1100100100100100100100100.0%
Qwen 3.6 Max Preview100100100100100100100100.0%
GPT-5.5 (Reasoning, Medium)100100100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100100100.0%
Kimi K3 (Reasoning, High)100100100100100100100100.0%
Claude Sonnet 5.5 (Reasoning)100100100100100100100100.0%
Grok 4.6 (Reasoning, High)100100100100100100100100.0%
GPT-6 Sol (Reasoning, Medium)100100100100100100100100.0%
Grok 4.7 (Reasoning, High)100100100100100100100100.0%
Kimi K2.6100100100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100100100.0%
GPT-5.6 Terra (Reasoning, Medium)100100100100100100100100.0%
GPT-6 Luna (Reasoning, Medium)100100100100100100100100.0%
Aion 3.5 Mini (Reasoning, High)100100100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100100100.0%
Claude Opus 4.6100100100100100100100100.0%
Mistral Large 4.0 (Reasoning)100100100100100100100100.0%
Kimi K3 (Reasoning, Low)100100100100100100100100.0%
Claude Opus 5 (Reasoning, Low)100100100100100100100100.0%
Aion 3.5 (Reasoning, High)100100100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100100100.0%
Xiaomi MIMO v2.6 Pro100100100100100100100100.0%
GPT-5100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Minimal)100100100100100100100100.0%
Claude Opus 5 (Reasoning)100100100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100100100.0%
Kimi K2.5100100100100100100100100.0%
Thinking Machines Inkling (Reasoning)100100100100100100100100.0%
GPT-5.6 Sol100100100100100100100100.0%
GPT-5.1100100100100100100100100.0%
Claude Sonnet 4.6100100100100100100100100.0%
GPT-5.6 Luna (Reasoning, Medium)100100100100100100100100.0%
MiniMax M3100100100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100100100.0%
Qwen 3.5 122B100100100100100100100100.0%
DeepSeek V4 Pro 0813 (Reasoning, High)100100100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100100100.0%
GPT-6 Sol100100100100100100100100.0%
Qwen 3.5 27B100100100100100100100100.0%
Claude Opus 5100100100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100100100.0%
Claude Opus 4.7100100100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100100100.0%
Claude Opus 4.5100100100100100100100100.0%
Z.AI GLM 5100100100100100100100100.0%
ByteDance Seed 1.6100100100100100100100100.0%
Claude Haiku 5.5 (Adaptive)100100100100100100100100.0%
GPT-5.5100100100100100100100100.0%
Qwen 3.6 Flash100100100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100100100.0%
Muse Glimmer 30B (Reasoning, Medium)100100100100100100100100.0%
Aion 3.0100100100100100100100100.0%
o4 Mini High100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Low)100100100100100100100100.0%
Gemini 2.5 Pro100100100100100100100100.0%
Qwen 3.6 27B100100100100100100100100.0%
GPT-5.6 Terra100100100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100100100.0%
Z.AI GLM 4.7100100100100100100100100.0%
Qwen 3.6 35B100100100100100100100100.0%
Z.AI GLM 4.6100100100100100100100100.0%
Claude Sonnet 4100100100100100100100100.0%
Claude Sonnet 4.5100100100100100100100100.0%
Claude Sonnet 5100100100100100100100100.0%
Xiaomi MIMO v2.6 Flash100100100100100100100100.0%
Claude Opus 4100100100100100100100100.0%
GPT-4.1100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning, Minimal)100100100100100100100100.0%
Aion 2.0100100100100100100100100.0%
o4 Mini100100100100100100100100.0%
MiniMax M2.7100100100100100100100100.0%
Qwen 3.5 Plus (2026-02-15)100100100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100100100.0%
Gemini 3 Flash (Preview)100100100100100100100100.0%
Gemini 3.1 Flash Lite (Preview)100100100100100100100100.0%
Mistral Medium 3.5 (Reasoning)100100100100100100100100.0%
Gemma 4 31B100100100100100100100100.0%
Gemini 3.1 Flash Lite100100100100100100100100.0%
GPT-5.6 Luna100100100100100100100100.0%
Z.AI GLM 4.5100100100100100100100100.0%
Gemma 4 26B100100100100100100100100.0%
GPT-OSS 120B100100100100100100100100.0%
Claude Haiku 5.5 (Reasoning, Low)100100100100100100100100.0%
Mistral Large 4.0100100100100100100100100.0%
Mistral Medium 3.5100100100100100100100100.0%
Gemini 3.5 Flash Lite (Reasoning, Minimal)100100100100100100100100.0%
Gemini 3.5 Flash Lite (Reasoning)100100100100100100100100.0%
GPT-5.4100100100100100100100100.0%
ByteDance Seed 2.0 Lite100100100100100100100100.0%
Claude Haiku 5.5 (Reasoning, Medium)100100100100100100100100.0%
GPT-6 Luna100100100100100100100100.0%
Aion 3.0 Mini100100100100100100100100.0%
Claude Haiku 4.5100100100100100100100100.0%
GPT-4o, Aug. 6th (temp=0)100100100100100100100100.0%
DeepSeek V4 Pro100100100100100100100100.0%
GPT-4.1 Mini100100100100100100100100.0%
GPT-4o, Aug. 6th (temp=1)100100100100100100100100.0%
Hermes 3 405B100100100100100100100100.0%
Gemini 2.5 Flash100100100100100100100100.0%
Qwen 3 32B100100100100100100100100.0%
Mistral Medium 3.1100100100100100100100100.0%
DeepSeek V4 Flash 0731 (Reasoning, High)1001001001001001009198.7%
Gemini 2.5 Flash (Reasoning)1001001001001001008297.4%
Z.AI GLM 4.5 Air1001001001001001008297.4%
Inception Mercury 210010010010082828292.2%
MiniMax M2.5100100100100100914590.9%
Qwen 3.8 Flash (Reasoning)100100100100100100085.7%
GPT-5.4 (Reasoning)100100100100100100085.7%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100085.7%
Z.AI GLM 5 Turbo100100100100100100085.7%
Qwen 3.8 27B (Reasoning, XHigh)100100100100100100085.7%
Qwen 3.5 Plus (2026-04-20)100100100100100100085.7%
GPT-5.2100100100100100100085.7%
Thinking Machines Inkling Small (Reasoning, High)100100100100100100085.7%
Qwen 3.5 9B100100100100100100085.7%
Xiaomi MIMO v2.5100100100100100100085.7%
Llama 3.1 70B100100100100100100085.7%
Thinking Machines Inkling100100100100100100085.7%
ByteDance Seed 1.6 Flash100100100100100100085.7%
GPT-5 Mini10010010010010091084.4%
GPT-5.4 Nano (Reasoning, Low)10010010010010045077.9%
Gemma 3 27B9191737373737377.9%
Grok 4.2010010010010010036076.6%
Laguna S 2.11001001007364454575.3%
Laguna XS 2.11001001001001009072.7%
Qwen 3.8 Max (Reasoning, XHigh)1001001001001000071.4%
Qwen 3.5 397B A17B1001001001001000071.4%
GPT-5.4 Mini (Reasoning)1001001001001000071.4%
Qwen 3.7 Flash (Reasoning)1001001001001000071.4%
Xiaomi MIMO v2.5 Pro1001001001001000071.4%
Mistral Small 4 (Reasoning)100100100100820068.8%
Nemotron 3 Super1001009182820064.9%
Qwen 3 235B A22B Instruct 25076464646464646463.6%
GPT-5.4 Nano100100100733627963.6%
Arcee AI: Trinity Mini6464646464646463.6%
Qwen 3.5 35B10010010064640061.0%
Ministral 3 3B6464646464555561.0%
GPT-5 Nano1001001009199058.4%
Ministral 3B6464645555555558.4%
GPT-5.4 Mini (Reasoning, Low)10010010010000057.1%
Qwen 2.5 72B10010010010000057.1%
Nemotron 3 Nano1001008255450054.5%
Writer: Palmyra X5646464646464054.5%
Gemma 3 4B5555555555555554.5%
GPT-5.4 Nano (Reasoning)100100735500046.8%
Cydonia 24B V4.11009164272718046.8%
Qwen 3.5 Flash100100643600042.9%
DeepSeek-V2 Chat100100100000042.9%
Z.AI GLM 4.7 Flash10010073000039.0%
DeepSeek V3 (2025-03-24)10010064000037.7%
GPT-5.4 Mini363636363627029.9%
DeepSeek V3 (2024-12-26)1001000000028.6%
Hermes 3 70B1001000000028.6%
Mistral Small 3.2 24B2727272727272727.3%
Mistral Small 4454518181818023.4%
Ministral 3 14B5555181890022.1%
GPT-6.1 Sol (Reasoning, Medium)10000000014.3%
Gemini 2.5 Flash Lite (Reasoning)10000000014.3%
WizardLM 2 8x22b10000000014.3%
Ministral 3 8B1818000005.2%
Ministral 8B90000001.3%
Gemini 4 Argon (Reasoning, Medium)00000000.0%
ByteDance Seed 2.0 Mini00000000.0%
Mistral Large 300000000.0%
DeepSeek V3.100000000.0%
DeepSeek V3.200000000.0%
DeepSeek V4 Flash00000000.0%
Mistral Large 200000000.0%
Gemini 2.5 Flash Lite00000000.0%
Grok 4.300000000.0%
GPT-4o Mini (temp=1)00000000.0%
GPT-4o Mini (temp=0)00000000.0%
Gemma 3 12B00000000.0%
GPT-4.1 Nano00000000.0%
Cohere Command R+ (Aug. 2024)00000000.0%
Mistral NeMO00000000.0%

Specific Prompt

Model # 1 # 2 # 3 # 4 # 5 # 6 # 7 Avg ▼
Claude Opus 5.5 (Reasoning)100100100100100100100100.0%
Qwen 3.8 Max (Reasoning, XHigh)100100100100100100100100.0%
GPT-6.1 Sol (Reasoning, Medium)100100100100100100100100.0%
Gemini 3.8 Flash (Reasoning, Medium)100100100100100100100100.0%
Ember-1 (Reasoning, Max)100100100100100100100100.0%
Muse Wonder (Reasoning, Medium)100100100100100100100100.0%
Z.AI GLM 5.3 (Reasoning, Max)100100100100100100100100.0%
GPT-5.6 Sol (Reasoning, Medium)100100100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning)100100100100100100100100.0%
Gemini 3.7 Flash (Reasoning, Medium)100100100100100100100100.0%
Hy4 Preview (Reasoning, High)100100100100100100100100.0%
Muse Spark 1.2 (Reasoning, Medium)100100100100100100100100.0%
Qwen 3.7 Max100100100100100100100100.0%
DeepSeek V4.1 Flash (Reasoning, High)100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Max)100100100100100100100100.0%
Qwen 3.8 Flash (Reasoning)100100100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100100100.0%
Muse Spark 1.3 (Reasoning, Medium)100100100100100100100100.0%
GPT-6 Luna (Reasoning, High)100100100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Medium)100100100100100100100100.0%
Gemini 4 Argon (Reasoning, Medium)100100100100100100100100.0%
Z.AI GLM 5.1100100100100100100100100.0%
Qwen 3.6 Max Preview100100100100100100100100.0%
GPT-5.5 (Reasoning, Medium)100100100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100100100.0%
Kimi K3 (Reasoning, High)100100100100100100100100.0%
Qwen 3.8 27B (Reasoning, XHigh)100100100100100100100100.0%
Claude Sonnet 5.5 (Reasoning)100100100100100100100100.0%
Grok 4.6 (Reasoning, High)100100100100100100100100.0%
GPT-6 Sol (Reasoning, Medium)100100100100100100100100.0%
Grok 4.7 (Reasoning, High)100100100100100100100100.0%
Kimi K2.6100100100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100100100.0%
GPT-5.6 Terra (Reasoning, Medium)100100100100100100100100.0%
GPT-6 Luna (Reasoning, Medium)100100100100100100100100.0%
Aion 3.5 Mini (Reasoning, High)100100100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100100100.0%
Claude Opus 4.6100100100100100100100100.0%
Mistral Large 4.0 (Reasoning)100100100100100100100100.0%
Kimi K3 (Reasoning, Low)100100100100100100100100.0%
Claude Opus 5 (Reasoning, Low)100100100100100100100100.0%
Aion 3.5 (Reasoning, High)100100100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100100100.0%
Xiaomi MIMO v2.6 Pro100100100100100100100100.0%
GPT-5100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Minimal)100100100100100100100100.0%
GPT-5 Mini100100100100100100100100.0%
Claude Opus 5 (Reasoning)100100100100100100100100.0%
Qwen 3.5 397B A17B100100100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100100100.0%
Kimi K2.5100100100100100100100100.0%
Thinking Machines Inkling (Reasoning)100100100100100100100100.0%
GPT-5.1100100100100100100100100.0%
Claude Sonnet 4.6100100100100100100100100.0%
GPT-5.6 Luna (Reasoning, Medium)100100100100100100100100.0%
MiniMax M3100100100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100100100.0%
Qwen 3.5 122B100100100100100100100100.0%
DeepSeek V4 Pro 0813 (Reasoning, High)100100100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100100100.0%
GPT-6 Sol100100100100100100100100.0%
Qwen 3.5 27B100100100100100100100100.0%
Claude Opus 5100100100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100100100.0%
Claude Opus 4.7100100100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100100100.0%
Claude Opus 4.5100100100100100100100100.0%
Z.AI GLM 5100100100100100100100100.0%
ByteDance Seed 1.6100100100100100100100100.0%
GPT-5.2100100100100100100100100.0%
Claude Haiku 5.5 (Adaptive)100100100100100100100100.0%
GPT-5.5100100100100100100100100.0%
Qwen 3.6 Flash100100100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100100100.0%
Muse Glimmer 30B (Reasoning, Medium)100100100100100100100100.0%
Aion 3.0100100100100100100100100.0%
o4 Mini High100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Low)100100100100100100100100.0%
Qwen 3.7 Flash (Reasoning)100100100100100100100100.0%
Gemini 2.5 Pro100100100100100100100100.0%
Qwen 3.6 27B100100100100100100100100.0%
GPT-5.6 Terra100100100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100100100.0%
DeepSeek V4 Flash 0731 (Reasoning, High)100100100100100100100100.0%
Z.AI GLM 4.7100100100100100100100100.0%
Qwen 3.6 35B100100100100100100100100.0%
Claude Sonnet 4100100100100100100100100.0%
Claude Sonnet 4.5100100100100100100100100.0%
Thinking Machines Inkling Small (Reasoning, High)100100100100100100100100.0%
Claude Sonnet 5100100100100100100100100.0%
Xiaomi MIMO v2.6 Flash100100100100100100100100.0%
Claude Opus 4100100100100100100100100.0%
Qwen 3.5 35B100100100100100100100100.0%
GPT-4.1100100100100100100100100.0%
MiniMax M2.5100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning, Minimal)100100100100100100100100.0%
Aion 2.0100100100100100100100100.0%
o4 Mini100100100100100100100100.0%
MiniMax M2.7100100100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100100100.0%
ByteDance Seed 2.0 Mini100100100100100100100100.0%
Qwen 3.5 Flash100100100100100100100100.0%
Gemini 3 Flash (Preview)100100100100100100100100.0%
Mistral Medium 3.5 (Reasoning)100100100100100100100100.0%
Gemma 4 31B100100100100100100100100.0%
Gemini 3.1 Flash Lite100100100100100100100100.0%
GPT-5.6 Luna100100100100100100100100.0%
Z.AI GLM 4.5100100100100100100100100.0%
Gemma 4 26B100100100100100100100100.0%
Claude Haiku 5.5 (Reasoning, Low)100100100100100100100100.0%
Mistral Large 4.0100100100100100100100100.0%
Mistral Medium 3.5100100100100100100100100.0%
Gemini 3.5 Flash Lite (Reasoning, Minimal)100100100100100100100100.0%
Gemini 3.5 Flash Lite (Reasoning)100100100100100100100100.0%
GPT-5.4100100100100100100100100.0%
Mistral Large 3100100100100100100100100.0%
ByteDance Seed 2.0 Lite100100100100100100100100.0%
Claude Haiku 5.5 (Reasoning, Medium)100100100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100100100.0%
DeepSeek-V2 Chat100100100100100100100100.0%
Xiaomi MIMO v2.5100100100100100100100100.0%
GPT-6 Luna100100100100100100100100.0%
Aion 3.0 Mini100100100100100100100100.0%
Claude Haiku 4.5100100100100100100100100.0%
DeepSeek V3 (2024-12-26)100100100100100100100100.0%
DeepSeek V3.2100100100100100100100100.0%
GPT-4o, Aug. 6th (temp=0)100100100100100100100100.0%
DeepSeek V4 Pro100100100100100100100100.0%
DeepSeek V4 Flash100100100100100100100100.0%
Nemotron 3 Super100100100100100100100100.0%
Mistral Large 2100100100100100100100100.0%
GPT-4.1 Mini100100100100100100100100.0%
GPT-4o, Aug. 6th (temp=1)100100100100100100100100.0%
Grok 4.20100100100100100100100100.0%
Hermes 3 405B100100100100100100100100.0%
Gemini 2.5 Flash100100100100100100100100.0%
Laguna S 2.1100100100100100100100100.0%
DeepSeek V3 (2025-03-24)100100100100100100100100.0%
Gemini 2.5 Flash Lite100100100100100100100100.0%
Mistral Small 4 (Reasoning)100100100100100100100100.0%
Qwen 3 32B100100100100100100100100.0%
Writer: Palmyra X5100100100100100100100100.0%
Qwen 3 235B A22B Instruct 2507100100100100100100100100.0%
Grok 4.3100100100100100100100100.0%
Llama 3.1 70B100100100100100100100100.0%
Mistral Medium 3.1100100100100100100100100.0%
Gemma 3 27B100100100100100100100100.0%
Mistral Small 4100100100100100100100100.0%
Thinking Machines Inkling100100100100100100100100.0%
Qwen 2.5 72B100100100100100100100100.0%
Cydonia 24B V4.1100100100100100100100100.0%
GPT-5.4 Nano100100100100100100100100.0%
WizardLM 2 8x22b100100100100100100100100.0%
Ministral 3 14B100100100100100100100100.0%
Ministral 3 8B100100100100100100100100.0%
Cohere Command R+ (Aug. 2024)100100100100100100100100.0%
Gemma 3 4B100100100100100100100100.0%
Ministral 3 3B100100100100100100100100.0%
Mistral NeMO100100100100100100100100.0%
Ministral 8B100100100100100100100100.0%
Gemini 3.1 Flash Lite (Preview)1001001001001001009198.7%
Z.AI GLM 4.61001001001001001008297.4%
Qwen 3.5 Plus (2026-02-15)1001001001001001008297.4%
Ministral 3B1001001001001001007396.1%
GPT-5.4 Nano (Reasoning, Low)100100100100100828294.8%
Laguna XS 2.1100100100100100828294.8%
ByteDance Seed 1.6 Flash1001001001001001006494.8%
Qwen 3.5 9B1001001001001001005593.5%
DeepSeek V3.1100100100100100915592.2%
GPT-5.4 Nano (Reasoning)1001001009182828290.9%
GPT-5 Nano1001001008282828289.6%
GPT-5.4 Mini (Reasoning, Low)100100828282828287.0%
Z.AI GLM 4.5 Air1001001008282737387.0%
Z.AI GLM 4.7 Flash10091918282828287.0%
Mistral Small 3.2 24B100100100100100100085.7%
GPT-5.6 Sol10082828282828284.4%
GPT-OSS 120B8282828282828281.8%
Inception Mercury 28282828282828281.8%
GPT-5.4 Mini8282828282828281.8%
GPT-4.1 Nano1001001001008282080.5%
Gemini 2.5 Flash Lite (Reasoning)1001001001007345074.0%
GPT-4o Mini (temp=1)6464646464646463.6%
GPT-4o Mini (temp=0)6464646464646463.6%
Arcee AI: Trinity Mini6464646464646463.6%
Nemotron 3 Nano1009182644545061.0%
Gemma 3 12B10055555555454558.4%
Hermes 3 70B100100100000042.9%