No hallucinated violations

Test: Codex Red Herring (False Positive Detection)

Avg. Score
68.2%
Scenarios
8

Overall Performance

Rank ▲ Model Score Avg. Cost Avg. Time Stability
1Z.AI GLM 5.3 Flash (Reasoning, Low)97.5%$0.00037.3s78%
2Inception Mercury 297.5%$0.00273.8s78%
3Nemotron 3 Super99.4%$0.00001.3m89%
4GPT-5.4 Mini (Reasoning, Low)96.9%$0.00344.0s76%
5Muse Spark 1.1 (Reasoning, Minimal)98.1%$0.01211.2s81%
6ByteDance Seed 1.6 Flash95.7%$0.00089.1s69%
7o4 Mini97.5%$0.01425.0s78%
8Z.AI GLM 5 Turbo96.9%$0.007116.0s73%
9GPT-5.4 Nano (Reasoning, Low)96.7%$0.00083.9s66%
10GPT-5.4 Nano (Reasoning)95.2%$0.001711.4s66%
11Gemini 2.5 Flash Lite (Reasoning)94.4%$0.002316.6s68%
12Laguna XS 2.194.1%$0.000516.4s66%
13Grok 4.5 (Reasoning, Low)95.6%$0.01217.0s69%
14GPT-5.196.9%$0.02526.1s76%
15Z.AI GLM 5.2 (Reasoning, High)95.4%$0.008125.8s68%
16GPT-5.6 Luna (Reasoning)93.8%$0.00484.7s62%
17o4 Mini High98.1%$0.02752.5s81%
18Muse Spark 1.1 (Reasoning, Medium)96.2%$0.01919.4s71%
19GPT-5 Mini95.0%$0.005937.8s67%
20MiniMax M2.793.1%$0.004734.6s66%
21GPT-5.4 Mini (Reasoning)92.3%$0.009010.8s63%
22DeepSeek V4 Flash (Reasoning)92.3%$0.000930.8s63%
23GPT-5 Nano95.0%$0.00351.1m70%
24Gemini 3.7 Flash (Reasoning, Medium)91.9%$0.0118.4s63%
25Gemini 2.5 Flash (Reasoning)91.3%$0.008514.2s62%
26Grok 4.3 (Reasoning)94.4%$0.01450.4s66%
27Xiaomi MIMO v2.590.4%$0.007530.3s60%
28Grok 4.5 (Reasoning, High)96.3%$0.0311.0m74%
29Muse Spark 1.2 (Reasoning, Medium)95.7%$0.02930.4s66%
30GPT-OSS 120B90.6%$0.001256.4s61%
31ByteDance Seed 1.689.6%$0.004332.7s56%
32Muse Spark 1.3 (Reasoning, Medium)94.2%$0.02452.0s67%
33Z.AI GLM 595.2%$0.0171.4m69%
34Mistral Small 4 (Reasoning)87.5%$0.002522.6s53%
35Laguna S 2.191.3%$0.000841.9s52%
36GPT-5.289.3%$0.01314.5s55%
37Aion 2.091.9%$0.00961.3m63%
38Kimi K3 (Reasoning, Low)90.6%$0.02028.5s59%
39Muse Glimmer 30B (Reasoning, Medium)88.5%$0.004437.4s54%
40Gemini 3.8 Flash (Reasoning, Medium)87.5%$0.02112.8s57%
41MiniMax M2.585.5%$0.003225.9s50%
42Qwen 3.5 Plus (2026-04-20)95.0%$0.0201.9m70%
43Qwen 3.7 Flash (Reasoning)84.4%$0.001245.9s54%
44GPT-4.185.9%$0.00481.2s43%
45Grok 4.20 (Reasoning)90.2%$0.01747.9s55%
46Xiaomi MIMO v2.5 Pro89.8%$0.0151.1m57%
47MiniMax M390.2%$0.00481.4m55%
48Aion 3.0 Mini93.9%$0.0112.0m63%
49GPT-5.6 Sol81.7%$0.00911.7s44%
50Gemma 4 26B (Reasoning)94.4%$0.00272.6m64%
51Qwen 3.6 Flash84.8%$0.01644.4s53%
52GPT-5.4 (Reasoning, Low)82.5%$0.0128.8s44%
53GPT-5.6 Terra (Reasoning)78.5%$0.00935.0s45%
54Z.AI GLM 5.194.8%$0.0242.1m66%
55Gemini 3.6 Flash (Reasoning, Minimal)82.7%$0.00781.6s39%
56Claude Opus 4.7 (Reasoning)94.4%$0.07513.1s66%
57Qwen 3 32B77.9%$0.001437.7s44%
58Gemma 4 31B77.3%$0.000911.7s39%
59Gemini 3.6 Flash (Reasoning)89.4%$0.05324.4s59%
60GPT-5.6 Luna77.2%$0.00201.2s36%
61Nemotron 3 Nano91.5%$0.00272.9m61%
62DeepSeek V4 Flash 0731 (Reasoning, High)86.7%$0.00192.0m52%
63Arcee AI: Trinity Mini79.2%$0.000926.3s33%
64Z.AI GLM 4.5 Air81.8%$0.002743.1s35%
65Qwen 3.6 35B80.8%$0.0171.2m50%
66DeepSeek V4.1 Flash (Reasoning, High)79.5%$0.00891.1m45%
67GPT-5.4 Nano69.5%$0.00051.5s28%
68 Kimi K3 (Reasoning, High)87.4%$0.0461.3m54%
69Qwen 3.6 27B90.7%$0.0332.3m57%
70GPT-5.4 (Reasoning)78.9%$0.03231.9s43%
71Grok 4.6 (Reasoning, High)93.1%$0.0572.2m66%
72Gemma 4 31B (Reasoning)87.7%$0.00253.1m52%
73Z.AI GLM 4.7 Flash83.4%$0.00392.5m49%
74Thinking Machines Inkling Small (Reasoning, High)73.4%$0.00941.0m36%
75Qwen 3.7 Max87.3%$0.0541.3m50%
76GPT-583.5%$0.0481.5m52%
77Claude Opus 4.6 (Reasoning)96.3%$0.1201.0m74%
78Gemma 4 26B68.8%$0.001239.3s27%
79Claude Sonnet 5 (Reasoning)86.3%$0.06957.0s52%
80GPT-5.4 Mini55.4%$0.00201.2s30%
81Ministral 3 8B74.5%$0.001317.9s14%
82Z.AI GLM 4.674.6%$0.01559.0s31%
83Claude Opus 4.679.7%$0.04913.2s34%
84Claude Sonnet 5 (Reasoning, Low)84.8%$0.06955.4s50%
85Gemini 3.5 Flash (Reasoning)81.0%$0.06729.5s46%
86Z.AI GLM 5.3 Flash (Reasoning, Max)79.0%$0.00243.0m45%
87GPT-5.6 Terra58.9%$0.00451.4s23%
88Z.AI GLM 4.775.9%$0.0232.2m46%
89Kimi K2.575.6%$0.0172.6m48%
90DeepSeek V4 Pro (Reasoning)82.5%$0.0173.4m52%
91Claude Opus 5 (Reasoning, Low)89.4%$0.11335.8s59%
92ByteDance Seed 2.0 Mini79.8%$0.00333.1m41%
93Claude Haiku 4.551.9%$0.00804.2s28%
94GPT-5.5 (Reasoning, Low)67.7%$0.02512.0s24%
95Z.AI GLM 4.563.4%$0.003825.0s19%
96Aion 3.087.9%$0.0582.4m53%
97GPT-5.6 Sol (Reasoning)65.6%$0.0269.1s24%
98GPT-5.440.2%$0.00483.1s34%
99Claude Opus 5 (Reasoning)87.3%$0.11336.5s56%
100GPT-5.555.9%$0.0132.9s21%
101Claude Sonnet 4.666.2%$0.03317.4s25%
102Gemini 3.5 Flash (Reasoning, Minimal)50.8%$0.0112.2s21%
103GPT-4.1 Nano58.4%$0.00054.6s8%
104Ministral 8B60.8%$0.00074.4s6%
105ByteDance Seed 2.0 Lite57.9%$0.00671.0m25%
106Cohere Command R+ (Aug. 2024)60.6%$0.0177.4s15%
107Gemini 3.5 Flash Lite (Reasoning)35.5%$0.00312.2s28%
108Gemini 3 Flash (Preview, Reasoning)67.6%$0.02849.2s20%
109Claude Opus 565.6%$0.0464.4s21%
110Qwen 3.5 9B82.8%$0.00384.7m44%
111Gemini 3.1 Flash Lite (Reasoning)40.3%$0.00151.9s19%
112Gemini 3.1 Flash Lite41.2%$0.00173.2s18%
113Gemini 3.5 Flash Lite (Reasoning, Minimal)34.6%$0.00312.1s23%
114Cydonia 24B V4.153.7%$0.00148.7s5%
115Qwen 3.5 Flash63.6%$0.00712.0m23%
116GPT-5.5 (Reasoning)66.7%$0.05124.1s23%
117Gemini 2.5 Flash31.4%$0.00211.5s23%
118DeepSeek V3 (2024-12-26)34.3%$0.00236.6s21%
119GPT-4o, Aug. 6th (temp=1)45.0%$0.00942.4s13%
120Hermes 3 405B47.5%$0.00595.2s9%
121Claude Opus 4.552.4%$0.0364.4s20%
122Gemini 3.1 Flash Lite (Preview)35.7%$0.00171.6s14%
123Claude Opus 4.8 (Reasoning, Low)89.2%$0.15052.5s58%
124Hermes 3 70B43.4%$0.001812.1s7%
125Claude Sonnet 445.5%$0.0224.5s13%
126Thinking Machines Inkling (Reasoning)84.8%$0.0325.2m52%
127DeepSeek-V2 Chat30.9%$0.00238.3s15%
128Gemini 3 Flash (Preview)25.0%$0.00343.2s20%
129Hy4 Preview (Reasoning, High)76.3%$0.0454.0m48%
130DeepSeek V3 (2025-03-24)31.5%$0.001515.7s15%
131Mistral Medium 3.139.4%$0.00324.7s5%
132Claude Opus 4.8 (Reasoning)89.0%$0.15856.0s55%
133DeepSeek V4 Pro36.1%$0.006942.0s17%
134Thinking Machines Inkling64.6%$0.0172.0m14%
135GPT-4o, Aug. 6th (temp=0)31.4%$0.0112.6s14%
136Qwen 2.5 72B35.2%$0.000911.3s6%
137Claude Opus 4.752.9%$0.0493.9s14%
138Claude Sonnet 4.532.8%$0.0245.9s19%
139Qwen 3.6 Max Preview82.8%$0.0723.8m50%
140GPT-4.1 Mini35.6%$0.00186.9s4%
141Grok 4.336.1%$0.00774.4s6%
142Claude Sonnet 537.6%$0.0206.0s11%
143Gemini 2.5 Pro70.4%$0.07352.1s21%
144Llama 3.1 70B33.5%$0.003116.4s7%
145Z.AI GLM 5.3 (Reasoning, Max)79.9%$0.0584.2m46%
146GPT-4o Mini (temp=1)28.2%$0.00085.6s6%
147Gemini 2.5 Flash Lite32.1%$0.00104.0s2%
148Gemini 3.1 Pro (Preview)76.9%$0.1201.5m45%
149Mistral Small 3.2 24B31.3%$0.001011.9s3%
150Mistral Large 332.2%$0.004011.7s2%
151WizardLM 2 8x22b32.6%$0.004155.3s12%
152Claude Sonnet 4.6 (Reasoning)87.5%$0.1452.2m55%
153DeepSeek V4 Pro 0813 (Reasoning, High)65.9%$0.0492.4m25%
154Qwen 3.5 35B64.5%$0.0442.4m23%
155Writer: Palmyra X524.0%$0.008513.8s12%
156Qwen 3 235B A22B Instruct 250726.3%$0.001132.0s10%
157GPT-4o Mini (temp=0)24.5%$0.000914.5s6%
158DeepSeek V3.124.5%$0.001936.6s11%
159Grok 4.2025.6%$0.008211.7s8%
160Kimi K2.676.9%$0.0475.2m47%
161Mistral Large 232.2%$0.01611.4s2%
162Ministral 3 14B29.3%$0.001929.2s1%
163DeepSeek V3.221.3%$0.001842.8s11%
164DeepSeek V4 Flash17.8%$0.000512.2s6%
165Gemma 3 12B13.7%$0.000511.7s10%
166Gemma 3 27B12.5%$0.000712.6s10%
167Qwen 3.8 27B (Reasoning, XHigh)65.6%$0.0353.6m23%
168Mistral Small 412.3%$0.00127.4s6%
169Ministral 3B9.9%$0.000412.4s6%
170Qwen 3.8 Flash (Reasoning)64.8%$0.00995.0m23%
171Gemma 3 4B5.6%$0.000315.0s5%
172Ministral 3 3B6.7%$0.001331.1s5%
173Mistral NeMO14.7%$0.00181.6m4%
174Qwen 3.5 Plus (2026-02-15)22.0%$0.0212.0m12%
175Claude Opus 436.0%$0.1097.1s22%
176Qwen 3.5 397B A17B61.5%$0.0285.4m23%
177Qwen 3.5 27B68.8%$0.0564.8m19%
178Qwen 3.8 Max (Reasoning, XHigh)70.6%$0.0955.7m24%
179Qwen 3.5 122B70.1%$0.1356.4m20%
68.22%

Individual Scenarios

basic entries

Model # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 9 # 10 Avg ▼
Gemini 3.8 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100100100100100100.0%
Gemini 3.7 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Muse Spark 1.2 (Reasoning, Medium)100100100100100100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100100100100100100.0%
Grok 4.6 (Reasoning, High)100100100100100100100100100100100.0%
Claude Opus 5 (Reasoning, Low)100100100100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Minimal)100100100100100100100100100100100.0%
GPT-5.1100100100100100100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100100100100100100.0%
Aion 2.0100100100100100100100100100100100.0%
Inception Mercury 2100100100100100100100100100100100.0%
GPT-5 Nano100100100100100100100100100100100.0%
GPT-5.4 Nano (Reasoning, Low)100100100100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning)1001001001001001001001001005095.0%
Z.AI GLM 5.3 Flash (Reasoning, Low)1001001001001001001001001005095.0%
Xiaomi MIMO v2.51001001001001001001001001005095.0%
Gemini 2.5 Flash Lite (Reasoning)1001001001001001001001001005095.0%
Nemotron 3 Super1001001001001001001001001005095.0%
Laguna XS 2.11001001001001001001001001005095.0%
Nemotron 3 Nano1001001001001001001001001005095.0%
ByteDance Seed 1.6 Flash1001001001001001001001001005095.0%
Muse Spark 1.3 (Reasoning, Medium)1001001001001001001001001003393.3%
GPT-51001001001001001001001001003393.3%
Mistral Small 4 (Reasoning)1001001001001001001001001003393.3%
Grok 4.3 (Reasoning)1001001001001001001001001003393.3%
GPT-5.21001001001001001001001001003393.3%
Claude Opus 4.61001001001001001001001001002092.0%
Ministral 3 8B100100100100100100100100100290.2%
GPT-4.1 Nano100100100100100100100100100190.1%
o4 Mini High100100100100100100100100505090.0%
o4 Mini100100100100100100100100505090.0%
GPT-OSS 120B100100100100100100100100505090.0%
Gemini 2.5 Flash (Reasoning)100100100100100100100100505090.0%
Aion 3.0 Mini100100100100100100100100505090.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100100100505090.0%
Claude Opus 5 (Reasoning)100100100100100100100100503388.3%
Muse Spark 1.1 (Reasoning, Medium)100100100100100100100100333386.7%
Claude Sonnet 4.6 (Reasoning)100100100100100100100100333386.7%
Z.AI GLM 5100100100100100100100100333386.7%
Laguna S 2.1100100100100100100100100333386.7%
Qwen 3.5 Plus (2026-04-20)10010010010010010010050505085.0%
DeepSeek V4 Flash (Reasoning)10010010010010010010050505085.0%
Kimi K2.510010010010010010010050503383.3%
Z.AI GLM 4.7 Flash10010010010010010010050503383.3%
GPT-5.6 Luna (Reasoning)10010010010010010010033333380.0%
Arcee AI: Trinity Mini1001001001001001005050505080.0%
Grok 4.5 (Reasoning, Low)10010010010010010010033333380.0%
Kimi K3 (Reasoning, High)10010010010010010010033332579.2%
Qwen 3.8 Max (Reasoning, XHigh)1001001001001001005050252575.0%
Qwen 3.7 Flash (Reasoning)100100100100100505050505075.0%
MiniMax M2.7100100100100100505050505075.0%
Z.AI GLM 5 Turbo1001001001001001005033333375.0%
Kimi K3 (Reasoning, Low)1001001001001001005033333375.0%
Kimi K2.61001001001001001003333333373.3%
Thinking Machines Inkling (Reasoning)1001001001001001003333333373.3%
Qwen 3 32B100100100100100505050502572.5%
MiniMax M2.5100100100100100505033332068.7%
Hy4 Preview (Reasoning, High)10010010010050505050503368.3%
Z.AI GLM 5.1100100100100100503333333368.3%
Qwen 3.6 27B100100100100100503333333368.3%
Xiaomi MIMO v2.5 Pro100100100100100503333333368.3%
Z.AI GLM 5.2 (Reasoning, High)100100100100100503333333368.3%
Cydonia 24B V4.1100100100100100100252525467.9%
GPT-5.4 (Reasoning)100100100100100333333333366.7%
Aion 3.0100100100100100333333333366.7%
GPT-5.4 Nano (Reasoning)100100100100100333333333366.7%
Z.AI GLM 4.5 Air10010010010010050503325766.5%
Z.AI GLM 4.61001001001005050505033063.3%
Qwen 3.6 35B1001001005050505050503363.3%
ByteDance Seed 1.610010010010050333333333361.7%
GPT-5 Mini1001001005050505033333360.0%
Claude Opus 4.7 (Reasoning)1001001005050505033333360.0%
GPT-5.5 (Reasoning, Low)10010010010033333333333360.0%
Grok 4.20 (Reasoning)10010010010033333333333360.0%
Qwen 3.6 Flash100100505050505050503358.3%
Claude Sonnet 4.61001001001005033333320757.7%
DeepSeek V4 Pro 0813 (Reasoning, High)1001001005050503333332557.5%
Claude Sonnet 5 (Reasoning)1001001005050333333333356.7%
Muse Glimmer 30B (Reasoning, Medium)1001001005050333333333356.7%
Qwen 3.8 Flash (Reasoning)1001001005033333333333355.0%
GPT-5.4 (Reasoning, Low)1001001005033333333333355.0%
Gemma 4 26B (Reasoning)1001001005033333333333355.0%
GPT-5.4 Nano100100100100503333205454.5%
GPT-5.6 Sol (Reasoning)1001001003333333333333353.3%
GPT-5.5 (Reasoning)1001001003333333333333353.3%
Gemini 3.1 Pro (Preview)100100505050503333332552.5%
GPT-5.4 Mini10050505050505050333351.7%
Qwen 3.6 Max Preview5050505050505050505050.0%
Claude Sonnet 5 (Reasoning, Low)100100503333333333333348.3%
GPT-5.4 Mini (Reasoning)5050505050505050503348.3%
Claude Opus 4.8 (Reasoning, Low)5050505050505050503348.3%
DeepSeek V4 Flash 0731 (Reasoning, High)100100503333333333333348.3%
Z.AI GLM 4.710050505050333333333346.7%
GPT-5.6 Terra (Reasoning)100100333333333333333346.7%
Gemma 4 31B (Reasoning)100100333333333333333346.7%
Gemini 2.5 Pro5050505050505033333345.0%
Claude Opus 4.8 (Reasoning)5050505050503333333343.3%
Qwen 3.5 35B5050505050503333332542.5%
Qwen 3.5 9B5050505050503333332542.5%
DeepSeek V4.1 Flash (Reasoning, High)10050333333333333332540.8%
Gemini 3.5 Flash (Reasoning)10033333333333333333340.0%
Qwen 3.5 Flash5050505033333333333340.0%
GPT-4.1100100503333201717171039.7%
Z.AI GLM 5.3 (Reasoning, Max)10033333333333333332539.2%
Hermes 3 405B100100333325202017131337.3%
MiniMax M35050333333333333333336.7%
Qwen 3.5 27B5050333333333333332535.8%
Qwen 3.7 Max5033333333333333333335.0%
GPT-5.6 Terra5033333333333333333335.0%
Qwen 3.5 397B A17B5033333333333333332534.2%
GPT-5.6 Sol5033333333333333332534.2%
Thinking Machines Inkling Small (Reasoning, High)3333333333333333333333.3%
ByteDance Seed 2.0 Lite5033333333333333252533.3%
Gemini 2.5 Flash Lite100100100884431132.9%
GPT-4.1 Mini505050505050888432.8%
GPT-5.6 Luna505050505020201414832.7%
Mistral NeMO100100100865430032.6%
Qwen 3.5 122B3333333333333333332532.5%
GPT-5.55050333333332525251732.5%
Ministral 8B100100100633222232.0%
ByteDance Seed 2.0 Mini3333333333333333332032.0%
Z.AI GLM 5.3 Flash (Reasoning, Max)3333333333333333252531.7%
Qwen 3.8 27B (Reasoning, XHigh)5033333333332525252531.7%
Claude Opus 4.55033333333333325251731.7%
Gemini 3 Flash (Preview, Reasoning)5033333325252525252029.5%
Z.AI GLM 4.510050333325201185228.8%
GPT-5.43333333325252525202027.3%
Claude Haiku 4.55033252525252520202026.8%
Cohere Command R+ (Aug. 2024)1003333201717141410926.8%
Claude Opus 4.73333332525252020202025.5%
Hermes 3 70B10050251413111098824.8%
Gemini 3.5 Flash (Reasoning, Minimal)2525252525252525201723.7%
DeepSeek V3 (2024-12-26)5050332525201098523.6%
Llama 3.1 70B1003333171310876623.4%
Gemini 3.6 Flash (Reasoning, Minimal)332525252525202020922.7%
DeepSeek V4 Pro505033332017854222.3%
Thinking Machines Inkling505025252525852221.7%
Gemini 2.5 Flash3333252525201714101021.3%
GPT-4o, Aug. 6th (temp=1)505020171714141110721.0%
Claude Sonnet 43325202020201717171720.5%
Grok 4.31001717141110996319.7%
Gemini 3.1 Flash Lite (Preview)2520202020202017171018.8%
Claude Opus 4252525202017171711918.5%
Gemma 4 31B2525202020171717131118.4%
Gemini 3.1 Flash Lite2020202020171717141417.9%
Claude Opus 52520202017171717141317.8%
Gemini 3.1 Flash Lite (Reasoning)2020202017171717131317.2%
Claude Sonnet 52020171717171717141416.9%
DeepSeek V3 (2025-03-24)252520201714141313816.9%
DeepSeek-V2 Chat5033171313111195416.6%
WizardLM 2 8x22b5050252074331116.4%
Gemma 4 26B3325171413131111111115.9%
Claude Sonnet 4.52017171714141414141115.3%
Gemini 3 Flash (Preview)1717141414141099612.4%
Gemini 3.5 Flash Lite (Reasoning)2525202077663212.0%
GPT-4o Mini (temp=1)251311111010888711.1%
GPT-4o, Aug. 6th (temp=0)141413131111988710.9%
Gemini 3.5 Flash Lite (Reasoning, Minimal)20171717136532210.1%
Qwen 3.5 Plus (2026-02-15)17141313119887210.1%
Mistral Small 4508876553339.8%
DeepSeek V3.1171713139977119.3%
Mistral Large 21711111110876659.2%
Gemma 3 27B11111199988758.9%
Gemma 3 12B259888766548.6%
Qwen 2.5 72B20141387665528.6%
Mistral Large 3141110108777648.3%
Ministral 3B2514976553338.0%
Writer: Palmyra X53310854433337.7%
Ministral 3 3B338644332106.4%
DeepSeek V3.21310777543225.9%
Mistral Medium 3.177666665535.6%
Mistral Small 3.2 24B87666555425.5%
Grok 4.20176555443335.4%
GPT-4o Mini (temp=0)77666665315.2%
Qwen 3 235B A22B Instruct 2507175554443325.1%
DeepSeek V4 Flash176543322114.4%
Gemma 3 4B33322222222.3%
Ministral 3 14B33332111101.8%
Model # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 9 # 10 Avg ▼
Qwen 3.8 Max (Reasoning, XHigh)100100100100100100100100100100100.0%
Gemini 3.8 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Z.AI GLM 5.3 (Reasoning, Max)100100100100100100100100100100100.0%
GPT-5.6 Sol (Reasoning)100100100100100100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100100100100100100.0%
Gemini 3.7 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Hy4 Preview (Reasoning, High)100100100100100100100100100100100.0%
Muse Spark 1.2 (Reasoning, Medium)100100100100100100100100100100100.0%
Qwen 3.7 Max100100100100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Max)100100100100100100100100100100100.0%
Qwen 3.8 Flash (Reasoning)100100100100100100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100100100100100100.0%
Muse Spark 1.3 (Reasoning, Medium)100100100100100100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Medium)100100100100100100100100100100100.0%
Z.AI GLM 5.1100100100100100100100100100100100.0%
Qwen 3.6 Max Preview100100100100100100100100100100100.0%
GPT-5.5 (Reasoning)100100100100100100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100100100100100100.0%
Kimi K3 (Reasoning, High)100100100100100100100100100100100.0%
Qwen 3.8 27B (Reasoning, XHigh)100100100100100100100100100100100.0%
Grok 4.6 (Reasoning, High)100100100100100100100100100100100.0%
Kimi K2.6100100100100100100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100100100100100100.0%
GPT-5.6 Terra (Reasoning)100100100100100100100100100100100.0%
Kimi K3 (Reasoning, Low)100100100100100100100100100100100.0%
GPT-5100100100100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Minimal)100100100100100100100100100100100.0%
GPT-5 Mini100100100100100100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100100100100100100.0%
Kimi K2.5100100100100100100100100100100100.0%
Thinking Machines Inkling (Reasoning)100100100100100100100100100100100.0%
GPT-5.6 Sol100100100100100100100100100100100.0%
GPT-5.1100100100100100100100100100100100.0%
GPT-5.6 Luna (Reasoning)100100100100100100100100100100100.0%
MiniMax M3100100100100100100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100100100100100100.0%
DeepSeek V4 Pro 0813 (Reasoning, High)100100100100100100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100100100100100100.0%
Z.AI GLM 5100100100100100100100100100100100.0%
ByteDance Seed 1.6100100100100100100100100100100100.0%
GPT-5.2100100100100100100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100100100100100100.0%
Muse Glimmer 30B (Reasoning, Medium)100100100100100100100100100100100.0%
o4 Mini High100100100100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Low)100100100100100100100100100100100.0%
Qwen 3.7 Flash (Reasoning)100100100100100100100100100100100.0%
Qwen 3.6 27B100100100100100100100100100100100.0%
GPT-5.6 Terra100100100100100100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100100100100100100.0%
DeepSeek V4 Flash 0731 (Reasoning, High)100100100100100100100100100100100.0%
Z.AI GLM 4.7100100100100100100100100100100100.0%
Thinking Machines Inkling Small (Reasoning, High)100100100100100100100100100100100.0%
GPT-4.1100100100100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning, Minimal)100100100100100100100100100100100.0%
o4 Mini100100100100100100100100100100100.0%
MiniMax M2.7100100100100100100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100100100100100100.0%
ByteDance Seed 2.0 Mini100100100100100100100100100100100.0%
Gemma 4 31B100100100100100100100100100100100.0%
GPT-5.6 Luna100100100100100100100100100100100.0%
GPT-OSS 120B100100100100100100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100100100100100100.0%
Xiaomi MIMO v2.5100100100100100100100100100100100.0%
Aion 3.0 Mini100100100100100100100100100100100.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100100100100100100.0%
Inception Mercury 2100100100100100100100100100100100.0%
Nemotron 3 Super100100100100100100100100100100100.0%
GPT-5 Nano100100100100100100100100100100100.0%
GPT-5.4 Nano (Reasoning)100100100100100100100100100100100.0%
GPT-5.4 Nano (Reasoning, Low)100100100100100100100100100100100.0%
Nemotron 3 Nano100100100100100100100100100100100.0%
ByteDance Seed 1.6 Flash100100100100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning)1001001001001001001001001005095.0%
Claude Opus 5 (Reasoning)1001001001001001001001001005095.0%
Qwen 3.6 Flash1001001001001001001001001005095.0%
DeepSeek V4 Pro (Reasoning)1001001001001001001001001005095.0%
MiniMax M2.51001001001001001001001001005095.0%
Aion 2.01001001001001001001001001005095.0%
Gemini 2.5 Flash Lite (Reasoning)1001001001001001001001001005095.0%
Z.AI GLM 4.5 Air1001001001001001001001001005095.0%
Laguna S 2.11001001001001001001001001005095.0%
Mistral Medium 3.11001001001001001001001001005095.0%
Claude Sonnet 4.61001001001001001001001001001491.4%
GPT-4.1 Nano100100100100100100100100100990.9%
Ministral 3 8B100100100100100100100100100390.3%
Claude Opus 5 (Reasoning, Low)100100100100100100100100505090.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100100100505090.0%
Aion 3.0100100100100100100100100505090.0%
Qwen 3.6 35B100100100100100100100100505090.0%
Gemini 3.1 Flash Lite100100100100100100100100505090.0%
Qwen 3.5 9B100100100100100100100100505090.0%
Laguna XS 2.1100100100100100100100100505090.0%
Gemini 2.5 Pro100100100100100100100100503388.3%
Gemini 3.5 Flash (Reasoning)10010010010010010010050505085.0%
Gemini 3.1 Flash Lite (Reasoning)10010010010010010010050503383.3%
Claude Opus 4.8 (Reasoning, Low)1001001001001001005050505080.0%
DeepSeek V4.1 Flash (Reasoning, High)10010010010010010010033333380.0%
Claude Opus 4.8 (Reasoning)1001001001001001005050503378.3%
Mistral Small 4 (Reasoning)1001001001001001005050503378.3%
Gemini 2.5 Flash Lite10010010010010010010033331377.9%
Gemini 3.1 Flash Lite (Preview)1001001001001001005050332575.8%
Qwen 3 32B1001001001001001005050332575.8%
Mistral Small 3.2 24B10010010010010010010013111173.5%
Z.AI GLM 4.7 Flash100100100100100505050503373.3%
Qwen 3.5 27B100100100100100505050332570.8%
Gemini 3.1 Pro (Preview)10010010010050505050505070.0%
Hermes 3 70B1001001001001001003317141367.7%
GPT-5.4 Nano1001001001001001003317141167.5%
Cydonia 24B V4.1100100100100100100171411764.9%
Qwen 3.5 397B A17B10010010010050505033252563.3%
Z.AI GLM 4.61001001001005050505014862.2%
Gemma 4 26B1001001005050505050333361.7%
Qwen 3.5 122B1001001005050505050333361.7%
Z.AI GLM 4.51001001001001005014108558.8%
Arcee AI: Trinity Mini10010010010050333333251158.6%
Ministral 8B10010010010010050442256.2%
Qwen 3.5 35B1001001005050333333252555.0%
Gemini 3 Flash (Preview, Reasoning)10050505050505050503353.3%
GPT-5.4 Mini10050505050505050332550.8%
Claude Haiku 4.55050505050505050503348.3%
ByteDance Seed 2.0 Lite5050505050505033333345.0%
Grok 4.201001001001001313844344.4%
Qwen 3.5 Flash10050505033333333252543.3%
Gemini 3.5 Flash (Reasoning, Minimal)10050505050333320202042.7%
GPT-5.45050505050505033251442.3%
Thinking Machines Inkling5050505050503333252541.7%
Gemini 3.5 Flash Lite (Reasoning)5050505050333325252038.7%
DeepSeek V4 Pro50505050503333337636.3%
Gemini 3.5 Flash Lite (Reasoning, Minimal)5050505050332525201036.3%
GPT-4o, Aug. 6th (temp=1)10033333333332517141433.7%
Hermes 3 405B10050332525252020201733.5%
Claude Sonnet 45033333333333325252032.0%
Claude Opus 45050333333252525252032.0%
GPT-5.55050503333252020201731.8%
WizardLM 2 8x22b1005050501713111010831.8%
DeepSeek-V2 Chat5050333325252020201128.8%
Qwen 2.5 72B1005033252013131313828.6%
DeepSeek V3.11005025251717141110827.6%
Cohere Command R+ (Aug. 2024)1003333202020171411227.1%
DeepSeek V3 (2024-12-26)5033252525252525171126.1%
Mistral NeMO10010011887777225.6%
Claude Opus 4.53333332525252020202025.5%
GPT-4o Mini (temp=1)1002525251714141313825.4%
Claude Opus 4.72525252525252525202024.0%
Grok 4.310033251414111188723.2%
Llama 3.1 70B50333320202020139622.4%
Claude Opus 52525252020202020202021.5%
DeepSeek V4 Flash50505011108777620.6%
Claude Sonnet 53325252020201717141420.5%
Gemini 3 Flash (Preview)2525252020201717171720.2%
DeepSeek V3 (2025-03-24)3320202020202017171420.1%
Claude Opus 4.63333252014141413111118.9%
Gemini 2.5 Flash2525252020201714131018.8%
Mistral Small 41001414131010986318.7%
Gemma 3 12B3317171717171414141417.4%
DeepSeek V3.23333332599988517.4%
Claude Sonnet 4.52020201717171717141417.2%
Qwen 3.5 Plus (2026-02-15)502020171313131310417.1%
Gemma 3 27B2020202017171717141017.1%
GPT-4o, Aug. 6th (temp=0)2020171717171414131316.0%
GPT-4o Mini (temp=0)1717171717171717131315.8%
Ministral 3B3325131311111088613.8%
Writer: Palmyra X5332014141310998413.5%
GPT-4.1 Mini20202017139998613.0%
Qwen 3 235B A22B Instruct 250733201411109887512.6%
Mistral Large 31111111111101010779.9%
Ministral 3 3B25141098887419.5%
Mistral Large 2119998777657.9%
Gemma 3 4B76666555445.4%
Ministral 3 14B66655433334.3%
Model # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 9 # 10 Avg ▼
Gemini 3.8 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Z.AI GLM 5.3 (Reasoning, Max)100100100100100100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning)100100100100100100100100100100100.0%
Gemini 3.7 Flash (Reasoning, Medium)100100100100100100100100100100100.0%
Qwen 3.7 Max100100100100100100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100100100100100100.0%
Muse Spark 1.1 (Reasoning, Medium)100100100100100100100100100100100.0%
Z.AI GLM 5.1100100100100100100100100100100100.0%
Qwen 3.6 Max Preview100100100100100100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100100100100100100.0%
Grok 4.6 (Reasoning, High)100100100100100100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100100100100100100.0%
Claude Opus 4.6100100100100100100100100100100100.0%
Claude Opus 5 (Reasoning, Low)100100100100100100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100100100100100100.0%
GPT-5 Mini100100100100100100100100100100100.0%
Claude Opus 5 (Reasoning)100100100100100100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100100100100100100.0%
GPT-5.6 Sol100100100100100100100100100100100.0%
MiniMax M3100100100100100100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100100100100100100.0%
Claude Opus 5100100100100100100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100100100100100100.0%
ByteDance Seed 1.6100100100100100100100100100100100.0%
GPT-5.5100100100100100100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100100100100100100.0%
o4 Mini High100100100100100100100100100100100.0%
Qwen 3.6 27B100100100100100100100100100100100.0%
DeepSeek V4 Flash 0731 (Reasoning, High)100100100100100100100100100100100.0%
GPT-4.1100100100100100100100100100100100.0%
Gemini 3.6 Flash (Reasoning, Minimal)100100100100100100100100100100100.0%
o4 Mini100100100100100100100100100100100.0%
Gemma 4 31B100100100100100100100100100100100.0%
GPT-5.6 Luna100100100100100100100100100100100.0%
Gemma 4 26B100100100100100100100100100100100.0%
Aion 3.0 Mini100100100100100100100100100100100.0%
Gemini 2.5 Flash Lite (Reasoning)100100100100100100100100100100100.0%
Inception Mercury 2100100100100100100100100100100100.0%
Nemotron 3 Super100100100100100100100100100100100.0%
GPT-5.4 Nano (Reasoning)100100100100100100100100100100100.0%
Laguna XS 2.1100100100100100100100100100100100.0%
ByteDance Seed 1.6 Flash100100100100100100100100100100100.0%
Z.AI GLM 5.3 Flash (Reasoning, Max)1001001001001001001001001005095.0%
Kimi K3 (Reasoning, Low)1001001001001001001001001005095.0%
GPT-5.11001001001001001001001001005095.0%
GPT-5.4 Mini (Reasoning)1001001001001001001001001005095.0%
Gemma 4 31B (Reasoning)1001001001001001001001001005095.0%
Qwen 3.6 Flash1001001001001001001001001005095.0%
Z.AI GLM 5.3 Flash (Reasoning, Low)1001001001001001001001001005095.0%
Aion 2.01001001001001001001001001005095.0%
MiniMax M2.71001001001001001001001001005095.0%
Gemini 2.5 Flash (Reasoning)1001001001001001001001001005095.0%
Qwen 3.5 9B1001001001001001001001001005095.0%
Mistral Small 4 (Reasoning)1001001001001001001001001005095.0%
Qwen 3 32B1001001001001001001001001005095.0%
Grok 4.3 (Reasoning)1001001001001001001001001003393.3%
Muse Glimmer 30B (Reasoning, Medium)1001001001001001001001001003393.3%
ByteDance Seed 2.0 Mini1001001001001001001001001003393.3%
GPT-5.4 Nano (Reasoning, Low)100100100100100100100100100890.8%
Ministral 3 8B100100100100100100100100100390.3%
Hy4 Preview (Reasoning, High)100100100100100100100100505090.0%
Muse Spark 1.2 (Reasoning, Medium)100100100100100100100100505090.0%
Muse Spark 1.1 (Reasoning, Minimal)100100100100100100100100505090.0%
Aion 3.0100100100100100100100100505090.0%
Gemini 2.5 Pro100100100100100100100100505090.0%
DeepSeek V4 Flash (Reasoning)100100100100100100100100505090.0%
Qwen 3.6 35B100100100100100100100100505090.0%
Xiaomi MIMO v2.5 Pro100100100100100100100100505090.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100100100505090.0%
Z.AI GLM 4.7 Flash100100100100100100100100505090.0%
Nemotron 3 Nano100100100100100100100100503388.3%
Arcee AI: Trinity Mini10010010010010010010010050985.9%
DeepSeek V4.1 Flash (Reasoning, High)10010010010010010010050505085.0%
Kimi K3 (Reasoning, High)10010010010010010010050505085.0%
Z.AI GLM 510010010010010010010050505085.0%
Qwen 3.7 Flash (Reasoning)10010010010010010010050505085.0%
Xiaomi MIMO v2.510010010010010010010050505085.0%
GPT-5 Nano10010010010010010010050505085.0%
Z.AI GLM 4.5 Air10010010010010010010010010981.9%
GPT-5.210010010010010010010050332080.3%
Hermes 3 405B10010010010010010010050332080.3%
Muse Spark 1.3 (Reasoning, Medium)1001001001001001005050505080.0%
MiniMax M2.51001001001001001005050505080.0%
GPT-OSS 120B1001001001001001005050505080.0%
GPT-5.6 Luna (Reasoning)10010010010010010010033333380.0%
Thinking Machines Inkling1001001001001001005050333376.7%
Thinking Machines Inkling (Reasoning)100100100100100505050505075.0%
Claude Opus 4.7100100100100100505050505075.0%
Claude Sonnet 41001001001001001005033333375.0%
GPT-5.4 Nano100100100100100100505011671.7%
Cohere Command R+ (Aug. 2024)1001001001001001005025201070.5%
Kimi K2.610010010010050505050505070.0%
GPT-510010010010050505050505070.0%
Qwen 3.5 397B A17B10010010010050505050505070.0%
Kimi K2.510010010010050505050505070.0%
DeepSeek V4 Pro (Reasoning)10010010010050505050505070.0%
Z.AI GLM 4.710010010010050505050505070.0%
Thinking Machines Inkling Small (Reasoning, High)10010010010050505050505070.0%
Gemini 3 Flash (Preview, Reasoning)10010010010050505050503368.3%
Laguna S 2.11001001001001001005077566.9%
GPT-5.5 (Reasoning)1001001005050505050505065.0%
Ministral 8B100100100100100100654161.6%
Hermes 3 70B100100100100100332520171460.9%
Qwen 3.5 122B1001001005050505033333360.0%
Qwen 3.5 27B1001001005050505033333360.0%
Z.AI GLM 4.6100100100100100502585058.9%
Qwen 3.5 35B100100505050505050503358.3%
Qwen 3.8 27B (Reasoning, XHigh)10050505050505050505055.0%
Claude Opus 4.510050505050505050505055.0%
Qwen 3.5 Flash10050505050505050505055.0%
ByteDance Seed 2.0 Lite10050505050505050505055.0%
GPT-5.4 Mini10050505050505050505055.0%
Claude Haiku 4.510050505050505050503353.3%
GPT-4o, Aug. 6th (temp=1)1001001005050332525201752.0%
GPT-5.6 Terra10050505050505050333351.7%
WizardLM 2 8x22b10050505050505050333351.7%
Qwen 3.8 Max (Reasoning, XHigh)5050505050505050505050.0%
Qwen 3.8 Flash (Reasoning)5050505050505050505050.0%
GPT-5.4 (Reasoning)10050505050505033333350.0%
GPT-5.5 (Reasoning, Low)5050505050505050505050.0%
DeepSeek V4 Pro 0813 (Reasoning, High)5050505050505050505050.0%
Claude Opus 45050505050505050505050.0%
Gemini 3.5 Flash (Reasoning, Minimal)5050505050505050505050.0%
Z.AI GLM 4.51001001005050502587549.4%
Claude Sonnet 4.510050505050503333333348.3%
GPT-5.6 Sol (Reasoning)5050505050505050503348.3%
GPT-5.6 Terra (Reasoning)5050505050505050503348.3%
Claude Sonnet 55050505050505050503348.3%
Writer: Palmyra X510010050505033333325748.2%
DeepSeek V4 Pro10010050505050332013847.4%
GPT-4.1 Nano100100100100503222246.0%
Cydonia 24B V4.11001001001001713986645.8%
GPT-5.4 (Reasoning, Low)10050505033333333333345.0%
Qwen 3 235B A22B Instruct 250710050505050333333252545.0%
DeepSeek V3 (2024-12-26)5050505050505025141140.0%
Grok 4.31001003333252520147636.4%
Gemini 3.5 Flash Lite (Reasoning)5050503333333333252036.2%
Gemini 3.5 Flash Lite (Reasoning, Minimal)5050503333252525252033.7%
GPT-4o, Aug. 6th (temp=0)5050333333333325251733.3%
Gemini 3 Flash (Preview)5033333333333325252532.5%
GPT-5.43333333333333333332532.5%
DeepSeek V3.2505050333333171717930.9%
Gemini 3.1 Flash Lite (Preview)3333333333333325252030.3%
Gemini 3.1 Flash Lite3333333333333325252030.3%
Gemini 3.1 Flash Lite (Reasoning)3333333333332525251428.9%
DeepSeek V3 (2025-03-24)5050332525252520141428.2%
DeepSeek V3.150503333332525139928.1%
Gemini 2.5 Flash503333252525202017725.5%
Grok 4.2050333333332520176425.5%
Qwen 2.5 72B100332525201111108825.2%
DeepSeek-V2 Chat5033252520201713131022.5%
Llama 3.1 70B1002017141313111110721.5%
Claude Sonnet 4.63325252520201717171421.3%
Mistral NeMO1003313101010764319.5%
GPT-4o Mini (temp=1)25252020171717119716.7%
Qwen 3.5 Plus (2026-02-15)3333202017171087216.7%
DeepSeek V4 Flash502017999766513.8%
Mistral Medium 3.12517171111101099912.8%
GPT-4.1 Mini171414131313111010912.3%
Mistral Small 3.2 24B141414131313111110812.1%
GPT-4o Mini (temp=0)141414141413131110211.9%
Mistral Large 31414141410101098811.2%
Gemini 2.5 Flash Lite20201010108887710.9%
Gemma 3 12B14141311111110108410.6%
Gemma 3 27B111111111010101010910.4%
Mistral Large 217171198887779.8%
Ministral 3 3B141414119877669.6%
Mistral Small 4131311109988779.4%
Ministral 3B141110109877758.8%
Gemma 3 4B109966533225.4%
Ministral 3 14B66655543314.3%