Structural validity

Test: Codex Extraction

Avg. Score
98.3%
Scenarios
4

Overall Performance

Rank ▲ Model Score Avg. Cost Avg. Time Stability
1Gemini 3.1 Flash Lite (Reasoning)100.0%$0.00182.0s100%
2Inception Mercury 2100.0%$0.00223.5s100%
3Gemini 3.1 Flash Lite (Preview)99.9%$0.00172.0s99%
4Gemini 2.5 Flash99.8%$0.00232.5s99%
5Ministral 3 8B99.6%$0.00063.3s98%
6Mistral Small 3.2 24B99.5%$0.00054.4s98%
7GPT-5.4 Mini (Reasoning, Low)100.0%$0.00423.7s100%
8Gemini 3.1 Flash Lite99.8%$0.00175.1s99%
9Ministral 3 3B99.1%$0.00041.8s97%
10GPT-5.4 Mini99.6%$0.00312.2s98%
11Grok 4.20100.0%$0.00484.9s100%
12GPT-4.1 Mini99.6%$0.00156.6s98%
13DeepSeek V3 (2024-12-26)100.0%$0.001713.5s100%
14Ministral 3B98.3%$0.00021.8s95%
15Claude Haiku 4.5100.0%$0.00734.5s100%
16DeepSeek-V2 Chat99.9%$0.001914.8s99%
17GPT-5.6 Luna99.4%$0.00433.6s97%
18Qwen 3.5 Plus (2026-02-15)99.8%$0.003010.6s99%
19Xiaomi MIMO v2.599.8%$0.003413.4s99%
20Gemma 3 27B99.4%$0.000514.5s97%
21Gemini 3.5 Flash (Reasoning, Minimal)100.0%$0.0103.1s100%
22GPT-4o, Aug. 6th (temp=1)99.8%$0.00923.8s99%
23DeepSeek V4 Pro99.7%$0.002116.0s98%
24Mistral Medium 3.198.6%$0.00265.8s95%
25GPT-5.6 Terra100.0%$0.0123.1s100%
26GPT-4o Mini (temp=1)98.0%$0.00068.0s95%
27GPT-4o, Aug. 6th (temp=0)99.7%$0.01004.0s99%
28GPT-5.6 Luna (Reasoning)99.8%$0.00787.2s98%
29DeepSeek V4 Flash99.1%$0.00037.7s93%
30Gemma 4 31B100.0%$0.000825.9s100%
31Gemini 2.5 Flash (Reasoning)100.0%$0.008211.8s100%
32Xiaomi MIMO v2.5 Pro100.0%$0.004818.7s100%
33MiniMax M2.799.9%$0.002221.7s99%
34GPT-4o Mini (temp=0)98.3%$0.00057.9s93%
35Gemini 3 Flash (Preview)98.2%$0.00273.9s93%
36Mistral Large 398.3%$0.00278.2s95%
37DeepSeek V3 (2025-03-24)100.0%$0.001427.0s100%
38Grok 4.5 (Reasoning, Low)100.0%$0.01013.1s100%
39Z.AI GLM 4.599.5%$0.002816.8s96%
40Mistral Small 4 (Reasoning)98.7%$0.001915.6s95%
41Z.AI GLM 5 Turbo99.8%$0.006816.0s98%
42GPT-5.4 Nano98.1%$0.00113.8s89%
43Z.AI GLM 5.2 (Reasoning, High)100.0%$0.007121.2s100%
44Hermes 3 405B98.9%$0.004017.8s97%
45MiniMax M2.599.7%$0.002330.1s99%
46DeepSeek V4 Flash (Reasoning)100.0%$0.000937.0s100%
47Qwen 2.5 72B97.0%$0.000811.0s92%
48WizardLM 2 8x22b99.9%$0.002631.3s99%
49DeepSeek V3.2100.0%$0.001136.7s100%
50Mistral Small 496.4%$0.00083.2s89%
51Mistral Large 298.9%$0.0118.3s96%
52Gemma 3 4B95.9%$0.00026.7s90%
53Gemini 3 Flash (Preview, Reasoning)100.0%$0.009622.2s100%
54Arcee AI: Trinity Mini95.8%$0.00035.8s90%
55GPT-5.6 Terra (Reasoning)99.8%$0.0167.3s99%
56Grok 4.398.7%$0.00564.3s89%
57Gemini 2.5 Flash Lite (Reasoning)97.7%$0.002115.6s92%
58GPT-5.498.2%$0.0116.4s95%
59Claude Sonnet 5100.0%$0.0198.6s100%
60GPT-5.4 (Reasoning, Low)99.8%$0.01710.4s98%
61o4 Mini100.0%$0.01421.5s100%
62Claude Sonnet 4.5100.0%$0.0226.6s100%
63Writer: Palmyra X596.7%$0.00527.5s90%
64Z.AI GLM 4.6100.0%$0.005738.8s100%
65MiniMax M399.8%$0.002340.5s98%
66Claude Sonnet 4100.0%$0.0227.9s100%
67GPT-5.6 Sol100.0%$0.0235.2s100%
68Claude Sonnet 4.6100.0%$0.0227.4s100%
69GPT-5.4 Nano (Reasoning, Low)96.6%$0.00113.7s83%
70Qwen 3 32B98.9%$0.001037.9s96%
71GPT-5.2100.0%$0.01816.8s100%
72Qwen 3.6 35B100.0%$0.007237.6s100%
73Qwen 3.6 Flash99.8%$0.009630.2s99%
74Ministral 8B96.0%$0.00043.8s83%
75Z.AI GLM 4.5 Air99.1%$0.002340.1s97%
76GPT-4.1 Nano94.1%$0.00032.8s85%
77Hermes 3 70B97.1%$0.001215.8s87%
78Qwen 3.5 Flash100.0%$0.003149.5s100%
79GPT-4.196.6%$0.00734.7s87%
80GPT-5.5100.0%$0.0276.0s100%
81GPT-5.4 Mini (Reasoning)100.0%$0.01725.6s100%
82Gemini 2.5 Flash Lite94.2%$0.00051.9s82%
83GPT-OSS 120B100.0%$0.001157.1s100%
84GPT-5 Mini100.0%$0.007045.5s100%
85Claude Sonnet 5 (Reasoning, Low)100.0%$0.02413.0s100%
86Grok 4.20 (Reasoning)100.0%$0.01237.2s100%
87Claude Sonnet 5 (Reasoning)100.0%$0.02513.6s100%
88Qwen3 235B A22B Instruct 250796.2%$0.000719.3s85%
89ByteDance Seed 1.6 Flash97.3%$0.001139.3s93%
90Cohere Command R+ (Aug. 2024)97.7%$0.01417.9s94%
91GPT-5.4 Nano (Reasoning)97.3%$0.002312.2s81%
92Nemotron 3 Super99.8%$0.00001.0m99%
93Z.AI GLM 5100.0%$0.009151.4s100%
94Qwen 3.5 35B99.9%$0.01547.3s99%
95Claude Opus 4.5100.0%$0.0377.7s100%
96Claude Opus 4.6100.0%$0.0378.1s100%
97GPT-5.5 (Reasoning, Low)100.0%$0.03813.3s100%
98o4 Mini High100.0%$0.02540.5s100%
99Gemini 2.5 Pro100.0%$0.03422.8s100%
100Aion 2.0100.0%$0.00801.2m100%
101Ministral 3 14B90.7%$0.00096.0s75%
102ByteDance Seed 1.6100.0%$0.00731.3m100%
103Grok 4.5 (Reasoning, High)100.0%$0.02445.2s100%
104Z.AI GLM 4.7 Flash98.7%$0.00191.2m94%
105Gemini 3.5 Flash (Reasoning)100.0%$0.04016.4s100%
106Z.AI GLM 5.1100.0%$0.0151.1m100%
107Claude Opus 4.7 (Reasoning)100.0%$0.0465.9s100%
108Claude Opus 4.7100.0%$0.0476.1s100%
109GPT-5 Nano99.5%$0.00431.3m96%
110Nemotron 3 Nano99.9%$0.00131.6m99%
111Claude Opus 4.8 (Reasoning, Low)100.0%$0.0478.3s100%
112Claude Opus 4.8 (Reasoning)100.0%$0.0478.3s100%
113ByteDance Seed 2.0 Lite99.6%$0.00781.4m98%
114GPT-5.6 Sol (Reasoning)100.0%$0.04517.0s100%
115Aion 3.0100.0%$0.02458.8s100%
116GPT-5.1100.0%$0.03543.6s100%
117Qwen 3.5 Plus (2026-04-20)100.0%$0.0151.4m100%
118Grok 4.3 (Reasoning)100.0%$0.0181.3m100%
119Z.AI GLM 4.7100.0%$0.00981.7m100%
120Cydonia 24B V4.192.3%$0.001012.2s60%
121DeepSeek V3.195.5%$0.001226.1s61%
122Claude Opus 4.6 (Reasoning)100.0%$0.05521.4s100%
123Gemma 4 31B (Reasoning)100.0%$0.00162.2m100%
124Gemma 3 12B89.8%$0.000314.1s59%
125Qwen 3.5 397B A17B100.0%$0.0121.8m100%
126GPT-5.5 (Reasoning)100.0%$0.05625.8s100%
127Llama 3.1 70B90.6%$0.001616.0s59%
128Claude Sonnet 4.6 (Reasoning)100.0%$0.05236.0s100%
129GPT-5.4 (Reasoning)100.0%$0.04453.6s100%
130Qwen3.7 Max100.0%$0.0411.1m100%
131Qwen 3.5 9B95.6%$0.00131.5m79%
132DeepSeek V4 Pro (Reasoning)100.0%$0.00932.3m100%
133Qwen 3.6 27B97.4%$0.0181.2m78%
134GPT-5100.0%$0.0491.3m100%
135MoonshotAI: Kimi K2.5100.0%$0.0132.4m100%
136MoonshotAI: Kimi K2.6100.0%$0.0262.4m100%
137Gemma 4 26B85.7%$0.000618.7s36%
138Gemini 3.1 Pro (Preview)99.5%$0.0651.0m96%
139Qwen 3.5 27B95.0%$0.0211.6m70%
140ByteDance Seed 2.0 Mini98.2%$0.00343.4m95%
141Claude Opus 4100.0%$0.11013.5s100%
142Qwen3.6 Max Preview100.0%$0.0452.4m100%
143Gemma 4 26B (Reasoning)95.5%$0.00232.6m61%
144Aion 3.0 Mini78.6%$0.00771.8m24%
145Qwen 3.5 122B100.0%$0.0793.6m100%
146Mistral NeMO30.0%$0.00061.4s0%
98.25%

Individual Scenarios

Model # 1 # 2 # 3 # 4 # 5 Avg ▼
GPT-5.6 Sol (Reasoning)100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100.0%
Qwen3.7 Max100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100.0%
Z.AI GLM 5.1100100100100100100.0%
Qwen3.6 Max Preview100100100100100100.0%
GPT-5.5 (Reasoning)100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100.0%
MoonshotAI: Kimi K2.6100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100.0%
GPT-5.6 Terra (Reasoning)100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100.0%
Claude Opus 4.6100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100.0%
GPT-5100100100100100100.0%
GPT-5 Mini100100100100100100.0%
Qwen 3.5 397B A17B100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100.0%
MoonshotAI: Kimi K2.5100100100100100100.0%
GPT-5.6 Sol100100100100100100.0%
GPT-5.1100100100100100100.0%
Claude Sonnet 4.6100100100100100100.0%
GPT-5.6 Luna (Reasoning)100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100.0%
Qwen 3.5 122B100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100.0%
Qwen 3.5 27B100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100.0%
Claude Opus 4.7100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100.0%
Claude Opus 4.5100100100100100100.0%
Z.AI GLM 5100100100100100100.0%
ByteDance Seed 1.6100100100100100100.0%
GPT-5.2100100100100100100.0%
GPT-5.5100100100100100100.0%
Qwen 3.6 Flash100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100.0%
Aion 3.0100100100100100100.0%
o4 Mini High100100100100100100.0%
Gemini 2.5 Pro100100100100100100.0%
GPT-5.6 Terra100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 4.7100100100100100100.0%
Qwen 3.6 35B100100100100100100.0%
Z.AI GLM 4.6100100100100100100.0%
Claude Sonnet 4100100100100100100.0%
Claude Sonnet 4.5100100100100100100.0%
Claude Sonnet 5100100100100100100.0%
Claude Opus 4100100100100100100.0%
Qwen 3.5 35B100100100100100100.0%
GPT-4.1100100100100100100.0%
Aion 2.0100100100100100100.0%
o4 Mini100100100100100100.0%
Qwen 3.5 Plus (2026-02-15)100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100.0%
Qwen 3.5 Flash100100100100100100.0%
Gemini 3 Flash (Preview)100100100100100100.0%
Gemini 3.1 Flash Lite (Preview)100100100100100100.0%
Gemma 4 31B100100100100100100.0%
Gemini 3.1 Flash Lite100100100100100100.0%
GPT-OSS 120B100100100100100100.0%
GPT-5.4100100100100100100.0%
ByteDance Seed 2.0 Lite100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100.0%
Claude Haiku 4.5100100100100100100.0%
DeepSeek V3 (2024-12-26)100100100100100100.0%
Inception Mercury 2100100100100100100.0%
Nemotron 3 Super100100100100100100.0%
GPT-4.1 Mini100100100100100100.0%
Grok 4.20100100100100100100.0%
GPT-5.4 Mini100100100100100100.0%
GPT-5 Nano100100100100100100.0%
GPT-5.4 Nano (Reasoning)100100100100100100.0%
Grok 4.3100100100100100100.0%
GPT-5.4 Nano (Reasoning, Low)100100100100100100.0%
Mistral Medium 3.1100100100100100100.0%
Nemotron 3 Nano100100100100100100.0%
GPT-5.4 Nano100100100100100100.0%
Z.AI GLM 4.51001001001009999.8%
Aion 3.0 Mini1001001001009999.8%
DeepSeek V3.21001001001009999.8%
DeepSeek V4 Pro1001001001009999.8%
DeepSeek V3 (2025-03-24)1001001001009999.8%
Qwen 3.6 27B1001001001009999.8%
Mistral Small 4 (Reasoning)1001001001009899.7%
Z.AI GLM 4.5 Air1001001001009899.6%
GPT-5.6 Luna1001001001009899.6%
Gemma 3 27B1001001001009899.6%
MiniMax M2.71001001001009899.6%
Gemini 2.5 Flash1001001001009899.6%
DeepSeek-V2 Chat100100100999999.6%
MiniMax M2.51001001001009899.6%
GPT-4o, Aug. 6th (temp=1)100100100999999.6%
WizardLM 2 8x22b1001001001009799.4%
Qwen 3 32B100100100999899.3%
Gemini 2.5 Flash Lite10010099999899.2%
Xiaomi MIMO v2.51009999999999.2%
Mistral Small 3.2 24B1009999999999.2%
MiniMax M31001001001009599.0%
Ministral 3 3B10010099989899.0%
GPT-4o, Aug. 6th (temp=0)999999999998.9%
Qwen 2.5 72B10010099999598.5%
Ministral 3 8B10010098989698.4%
Ministral 8B1009999989698.3%
GPT-4o Mini (temp=1)1009999979698.1%
Mistral Large 2999999999498.1%
Gemini 3.1 Pro (Preview)1001001001009098.0%
Mistral Small 410010098979497.9%
Z.AI GLM 4.7 Flash100100100989197.8%
Ministral 3B1009898979697.8%
Hermes 3 405B999999979597.7%
Llama 3.1 70B1009999969497.5%
Gemini 2.5 Flash Lite (Reasoning)100100100998897.5%
Gemma 3 12B10010099998997.5%
GPT-4o Mini (temp=0)10010096969497.4%
Writer: Palmyra X51009898969597.4%
DeepSeek V4 Flash1001001001008396.7%
ByteDance Seed 2.0 Mini999896959496.6%
ByteDance Seed 1.6 Flash999996949396.2%
Cohere Command R+ (Aug. 2024)989897969396.2%
Arcee AI: Trinity Mini999995959295.9%
Qwen3 235B A22B Instruct 2507969696969595.6%
Hermes 3 70B1009999988395.6%
Gemma 3 4B1009696959095.4%
Mistral Large 3989893939395.2%
Ministral 3 14B999695929294.9%
GPT-4.1 Nano1009898928093.6%
Cydonia 24B V4.1969695947891.7%
Qwen 3.5 9B100100100995390.4%
DeepSeek V3.11001001001001082.0%
Gemma 4 26B10010090101062.1%
Mistral NeMO000000.0%
Model # 1 # 2 # 3 # 4 # 5 Avg ▼
GPT-5.6 Sol (Reasoning)100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100.0%
Qwen3.7 Max100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100.0%
Z.AI GLM 5.1100100100100100100.0%
Qwen3.6 Max Preview100100100100100100.0%
GPT-5.5 (Reasoning)100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100.0%
MoonshotAI: Kimi K2.6100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100.0%
Claude Opus 4.6100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100.0%
GPT-5100100100100100100.0%
GPT-5 Mini100100100100100100.0%
Qwen 3.5 397B A17B100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100.0%
MoonshotAI: Kimi K2.5100100100100100100.0%
GPT-5.6 Sol100100100100100100.0%
GPT-5.1100100100100100100.0%
Claude Sonnet 4.6100100100100100100.0%
MiniMax M3100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100.0%
Qwen 3.5 122B100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100.0%
Qwen 3.5 27B100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100.0%
Claude Opus 4.7100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100.0%
Claude Opus 4.5100100100100100100.0%
Z.AI GLM 5100100100100100100.0%
ByteDance Seed 1.6100100100100100100.0%
GPT-5.2100100100100100100.0%
GPT-5.5100100100100100100.0%
Qwen 3.6 Flash100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100.0%
Aion 3.0100100100100100100.0%
o4 Mini High100100100100100100.0%
Gemini 2.5 Pro100100100100100100.0%
Qwen 3.6 27B100100100100100100.0%
GPT-5.6 Terra100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 4.7100100100100100100.0%
Qwen 3.6 35B100100100100100100.0%
Z.AI GLM 4.6100100100100100100.0%
Claude Sonnet 4100100100100100100.0%
Claude Sonnet 4.5100100100100100100.0%
Claude Sonnet 5100100100100100100.0%
Claude Opus 4100100100100100100.0%
Aion 2.0100100100100100100.0%
o4 Mini100100100100100100.0%
MiniMax M2.7100100100100100100.0%
Qwen 3.5 Plus (2026-02-15)100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100.0%
Qwen 3.5 Flash100100100100100100.0%
Gemini 3.1 Flash Lite (Preview)100100100100100100.0%
Gemma 4 31B100100100100100100.0%
Gemini 3.1 Flash Lite100100100100100100.0%
GPT-5.6 Luna100100100100100100.0%
Z.AI GLM 4.5100100100100100100.0%
GPT-OSS 120B100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100.0%
DeepSeek-V2 Chat100100100100100100.0%
Xiaomi MIMO v2.5100100100100100100.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100.0%
Claude Haiku 4.5100100100100100100.0%
DeepSeek V3 (2024-12-26)100100100100100100.0%
DeepSeek V3.1100100100100100100.0%
DeepSeek V3.2100100100100100100.0%
GPT-4o, Aug. 6th (temp=0)100100100100100100.0%
Inception Mercury 2100100100100100100.0%
Nemotron 3 Super100100100100100100.0%
Grok 4.20100100100100100100.0%
GPT-5.4 Mini100100100100100100.0%
GPT-5 Nano100100100100100100.0%
DeepSeek V3 (2025-03-24)100100100100100100.0%
Gemma 3 27B100100100100100100.0%
Nemotron 3 Nano100100100100100100.0%
WizardLM 2 8x22b100100100100100100.0%
Ministral 3 8B100100100100100100.0%
Ministral 8B100100100100100100.0%
ByteDance Seed 2.0 Lite1001001001009899.6%
DeepSeek V4 Flash1001001001009899.6%
GPT-4o, Aug. 6th (temp=1)1001001001009899.6%
Gemini 2.5 Flash1001001001009899.6%
Qwen 3.5 35B1001001001009899.6%
GPT-4.1 Mini1001001001009899.5%
Ministral 3 3B1001001001009899.5%
GPT-5.6 Terra (Reasoning)1001001001009799.3%
GPT-5.4 (Reasoning, Low)1001001001009699.3%
GPT-5.6 Luna (Reasoning)1001001001009699.3%
GPT-5.4 Nano (Reasoning)1001001001009699.3%
Mistral Large 2100100100989899.2%
MiniMax M2.5100100100989899.2%
Z.AI GLM 4.7 Flash1001001001009699.2%
Qwen 3 32B1001001001009699.2%
GPT-5.4 Nano1001001001009699.2%
Ministral 3B100100100989899.1%
Hermes 3 70B100100100989899.0%
DeepSeek V4 Pro100100100989698.9%
Hermes 3 405B100100100989698.7%
Mistral Small 3.2 24B10010098989898.7%
Gemma 4 26B100100100969698.6%
Cohere Command R+ (Aug. 2024)10010098989698.5%
Cydonia 24B V4.110010098989698.4%
GPT-4o Mini (temp=1)100100100979498.3%
Mistral Large 3989898989898.1%
GPT-4.1100100100969398.0%
Mistral Small 4 (Reasoning)100100100979397.9%
Gemini 2.5 Flash Lite (Reasoning)1009898969697.9%
Z.AI GLM 4.5 Air1009898969597.3%
GPT-5.410010096969397.1%
GPT-4o Mini (temp=0)1001001001008597.0%
Qwen 2.5 72B10010096959397.0%
ByteDance Seed 2.0 Mini1009896969497.0%
Writer: Palmyra X510010096959396.9%
Gemini 2.5 Flash Lite1009898969096.5%
Qwen 3.5 9B10010096968996.4%
ByteDance Seed 1.6 Flash1009896949296.1%
GPT-5.4 Nano (Reasoning, Low)10010097949096.1%
Grok 4.31001001001007595.0%
Mistral Medium 3.1969494949494.5%
Mistral Small 410010096918394.2%
Arcee AI: Trinity Mini989793929093.8%
GPT-4.1 Nano989493938993.4%
Qwen3 235B A22B Instruct 250710010096957593.3%
Gemini 3 Flash (Preview)1009393898992.9%
Gemma 3 4B969391919092.3%
Llama 3.1 70B989492908691.9%
Ministral 3 14B929189898890.0%
Gemma 3 12B908988881073.0%
Aion 3.0 Mini1001005025055.0%
Mistral NeMO000000.0%
Model # 1 # 2 # 3 # 4 # 5 Avg ▼
GPT-5.6 Sol (Reasoning)100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100.0%
Qwen3.7 Max100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100.0%
Z.AI GLM 5.1100100100100100100.0%
Qwen3.6 Max Preview100100100100100100.0%
GPT-5.5 (Reasoning)100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 5 Turbo100100100100100100.0%
MoonshotAI: Kimi K2.6100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100.0%
GPT-5.6 Terra (Reasoning)100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100.0%
Claude Opus 4.6100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100.0%
GPT-5100100100100100100.0%
GPT-5 Mini100100100100100100.0%
Qwen 3.5 397B A17B100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100.0%
MoonshotAI: Kimi K2.5100100100100100100.0%
GPT-5.6 Sol100100100100100100.0%
GPT-5.1100100100100100100.0%
Claude Sonnet 4.6100100100100100100.0%
GPT-5.6 Luna (Reasoning)100100100100100100.0%
MiniMax M3100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100.0%
Qwen 3.5 122B100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100.0%
Qwen 3.5 27B100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100.0%
Claude Opus 4.7100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100.0%
Claude Opus 4.5100100100100100100.0%
Z.AI GLM 5100100100100100100.0%
ByteDance Seed 1.6100100100100100100.0%
GPT-5.2100100100100100100.0%
GPT-5.5100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100.0%
Gemma 4 26B (Reasoning)100100100100100100.0%
Aion 3.0100100100100100100.0%
o4 Mini High100100100100100100.0%
Gemini 2.5 Pro100100100100100100.0%
Qwen 3.6 27B100100100100100100.0%
GPT-5.6 Terra100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 4.7100100100100100100.0%
Qwen 3.6 35B100100100100100100.0%
Z.AI GLM 4.6100100100100100100.0%
Claude Sonnet 4100100100100100100.0%
Claude Sonnet 4.5100100100100100100.0%
Claude Sonnet 5100100100100100100.0%
Claude Opus 4100100100100100100.0%
Qwen 3.5 35B100100100100100100.0%
MiniMax M2.5100100100100100100.0%
Aion 2.0100100100100100100.0%
o4 Mini100100100100100100.0%
MiniMax M2.7100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100.0%
ByteDance Seed 2.0 Mini100100100100100100.0%
Qwen 3.5 Flash100100100100100100.0%
Gemini 3 Flash (Preview)100100100100100100.0%
Gemini 3.1 Flash Lite (Preview)100100100100100100.0%
Gemma 4 31B100100100100100100.0%
GPT-OSS 120B100100100100100100.0%
GPT-5.4100100100100100100.0%
Mistral Large 3100100100100100100.0%
ByteDance Seed 2.0 Lite100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100.0%
DeepSeek-V2 Chat100100100100100100.0%
Xiaomi MIMO v2.5100100100100100100.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100.0%
Claude Haiku 4.5100100100100100100.0%
DeepSeek V3 (2024-12-26)100100100100100100.0%
DeepSeek V3.1100100100100100100.0%
DeepSeek V3.2100100100100100100.0%
GPT-4o, Aug. 6th (temp=0)100100100100100100.0%
DeepSeek V4 Pro100100100100100100.0%
DeepSeek V4 Flash100100100100100100.0%
Inception Mercury 2100100100100100100.0%
GPT-4o, Aug. 6th (temp=1)100100100100100100.0%
Grok 4.20100100100100100100.0%
Z.AI GLM 4.5 Air100100100100100100.0%
Gemini 2.5 Flash100100100100100100.0%
GPT-5 Nano100100100100100100.0%
DeepSeek V3 (2025-03-24)100100100100100100.0%
Mistral Small 3.2 24B100100100100100100.0%
GPT-4o Mini (temp=0)100100100100100100.0%
Mistral Medium 3.1100100100100100100.0%
WizardLM 2 8x22b100100100100100100.0%
Ministral 3 8B100100100100100100.0%
Ministral 8B100100100100100100.0%
Grok 4.31001001001009899.6%
Hermes 3 405B1001001001009899.6%
Nemotron 3 Nano1001001001009899.6%
Qwen 3.6 Flash1001001001009799.4%
GPT-5.4 Mini1001001001009799.4%
Qwen 3.5 Plus (2026-02-15)1001001001009799.3%
Gemini 3.1 Flash Lite1001001001009799.3%
Nemotron 3 Super1001001001009799.3%
GPT-5.6 Luna1001001001009599.0%
GPT-4o Mini (temp=1)1001001001009599.0%
Ministral 3 3B100100100989799.0%
GPT-4.1 Mini100100100989798.9%
Cydonia 24B V4.1100100100999598.7%
GPT-4.11001001001009398.7%
Arcee AI: Trinity Mini1001001001009298.3%
Mistral Large 2100100100979498.2%
Ministral 3B10010098979698.2%
Qwen 3 32B100100100989398.2%
ByteDance Seed 1.6 Flash100100100969598.2%
Z.AI GLM 4.51001001001009098.0%
Cohere Command R+ (Aug. 2024)10010098989598.0%
Gemma 3 27B10010098979597.9%
Z.AI GLM 4.7 Flash1001001001008997.9%
Mistral Small 4 (Reasoning)10010097949497.0%
Qwen 3.5 9B100100100949196.9%
Gemma 3 4B989796969596.6%
Mistral Small 4100100100988496.5%
Gemini 2.5 Flash Lite (Reasoning)100100100938695.8%
Qwen3 235B A22B Instruct 250710010096938995.7%
GPT-5.4 Nano1001001001007695.2%
Hermes 3 70B10010098987894.7%
Qwen 2.5 72B989693928893.5%
Gemma 3 12B979494919093.1%
GPT-5.4 Nano (Reasoning, Low)1001001001006492.8%
Writer: Palmyra X5969695908692.5%
Gemini 2.5 Flash Lite10010098937192.5%
GPT-4.1 Nano969391898891.3%
GPT-5.4 Nano (Reasoning)100100100945690.0%
Gemma 4 26B1001001001001082.0%
Ministral 3 14B888685706578.7%
Llama 3.1 70B959493911076.7%
Mistral NeMO1001001000060.0%
Aion 3.0 Mini100100980059.7%
Model # 1 # 2 # 3 # 4 # 5 Avg ▼
GPT-5.6 Sol (Reasoning)100100100100100100.0%
Claude Opus 4.6 (Reasoning)100100100100100100.0%
Qwen3.7 Max100100100100100100.0%
Grok 4.5 (Reasoning, High)100100100100100100.0%
Gemini 3.1 Pro (Preview)100100100100100100.0%
GPT-5.4 (Reasoning)100100100100100100.0%
Z.AI GLM 5.1100100100100100100.0%
Qwen3.6 Max Preview100100100100100100.0%
GPT-5.5 (Reasoning)100100100100100100.0%
Claude Sonnet 4.6 (Reasoning)100100100100100100.0%
Z.AI GLM 5.2 (Reasoning, High)100100100100100100.0%
Gemini 3.5 Flash (Reasoning)100100100100100100.0%
MoonshotAI: Kimi K2.6100100100100100100.0%
Claude Opus 4.7 (Reasoning)100100100100100100.0%
GPT-5.5 (Reasoning, Low)100100100100100100.0%
GPT-5.6 Terra (Reasoning)100100100100100100.0%
Claude Opus 4.8 (Reasoning)100100100100100100.0%
Claude Opus 4.6100100100100100100.0%
Claude Opus 4.8 (Reasoning, Low)100100100100100100.0%
GPT-5100100100100100100.0%
GPT-5 Mini100100100100100100.0%
Qwen 3.5 397B A17B100100100100100100.0%
Grok 4.3 (Reasoning)100100100100100100.0%
Grok 4.5 (Reasoning, Low)100100100100100100.0%
GPT-5.4 (Reasoning, Low)100100100100100100.0%
Grok 4.20 (Reasoning)100100100100100100.0%
MoonshotAI: Kimi K2.5100100100100100100.0%
GPT-5.6 Sol100100100100100100.0%
GPT-5.1100100100100100100.0%
Claude Sonnet 4.6100100100100100100.0%
GPT-5.6 Luna (Reasoning)100100100100100100.0%
MiniMax M3100100100100100100.0%
Claude Sonnet 5 (Reasoning)100100100100100100.0%
Qwen 3.5 122B100100100100100100.0%
Claude Sonnet 5 (Reasoning, Low)100100100100100100.0%
Gemini 3 Flash (Preview, Reasoning)100100100100100100.0%
Claude Opus 4.7100100100100100100.0%
GPT-5.4 Mini (Reasoning)100100100100100100.0%
Qwen 3.5 Plus (2026-04-20)100100100100100100.0%
Gemma 4 31B (Reasoning)100100100100100100.0%
Claude Opus 4.5100100100100100100.0%
Z.AI GLM 5100100100100100100.0%
ByteDance Seed 1.6100100100100100100.0%
GPT-5.2100100100100100100.0%
GPT-5.5100100100100100100.0%
Qwen 3.6 Flash100100100100100100.0%
DeepSeek V4 Pro (Reasoning)100100100100100100.0%
Aion 3.0100100100100100100.0%
o4 Mini High100100100100100100.0%
Gemini 2.5 Pro100100100100100100.0%
GPT-5.6 Terra100100100100100100.0%
DeepSeek V4 Flash (Reasoning)100100100100100100.0%
Z.AI GLM 4.7100100100100100100.0%
Qwen 3.6 35B100100100100100100.0%
Z.AI GLM 4.6100100100100100100.0%
Claude Sonnet 4100100100100100100.0%
Claude Sonnet 4.5100100100100100100.0%
Claude Sonnet 5100100100100100100.0%
Claude Opus 4100100100100100100.0%
Qwen 3.5 35B100100100100100100.0%
MiniMax M2.5100100100100100100.0%
Aion 2.0100100100100100100.0%
o4 Mini100100100100100100.0%
MiniMax M2.7100100100100100100.0%
Qwen 3.5 Plus (2026-02-15)100100100100100100.0%
Xiaomi MIMO v2.5 Pro100100100100100100.0%
Gemini 3.1 Flash Lite (Reasoning)100100100100100100.0%
Gemini 3.5 Flash (Reasoning, Minimal)100100100100100100.0%
Qwen 3.5 Flash100100100100100100.0%
Gemini 3 Flash (Preview)100100100100100100.0%
Gemma 4 31B100100100100100100.0%
Gemini 3.1 Flash Lite100100100100100100.0%
Z.AI GLM 4.5100100100100100100.0%
Gemma 4 26B100100100100100100.0%
GPT-OSS 120B100100100100100100.0%
Mistral Large 3100100100100100100.0%
Gemini 2.5 Flash (Reasoning)100100100100100100.0%
DeepSeek-V2 Chat100100100100100100.0%
Xiaomi MIMO v2.5100100100100100100.0%
Aion 3.0 Mini100100100100100100.0%
GPT-5.4 Mini (Reasoning, Low)100100100100100100.0%
Claude Haiku 4.5100100100100100100.0%
DeepSeek V3 (2024-12-26)100100100100100100.0%
DeepSeek V3.1100100100100100100.0%
DeepSeek V3.2100100100100100100.0%
Z.AI GLM 4.7 Flash100100100100100100.0%
GPT-4o, Aug. 6th (temp=0)100100100100100100.0%
DeepSeek V4 Pro100100100100100100.0%
DeepSeek V4 Flash100100100100100100.0%
Inception Mercury 2100100100100100100.0%
Nemotron 3 Super100100100100100100.0%
Mistral Large 2100100100100100100.0%
GPT-4.1 Mini100100100100100100.0%
GPT-4o, Aug. 6th (temp=1)100100100100100100.0%
Grok 4.20100100100100100100.0%
Gemini 2.5 Flash100100100100100100.0%
GPT-5.4 Nano (Reasoning)100100100100100100.0%
DeepSeek V3 (2025-03-24)100100100100100100.0%
Mistral Small 4 (Reasoning)100100100100100100.0%
Writer: Palmyra X5100100100100100100.0%
Qwen3 235B A22B Instruct 2507100100100100100100.0%
Grok 4.3100100100100100100.0%
Mistral Small 3.2 24B100100100100100100.0%
Mistral Medium 3.1100100100100100100.0%
Gemma 3 27B100100100100100100.0%
Nemotron 3 Nano100100100100100100.0%
WizardLM 2 8x22b100100100100100100.0%
Ministral 3 8B100100100100100100.0%
Gemini 2.5 Flash Lite (Reasoning)1001001001009899.6%
Gemini 3.1 Flash Lite (Preview)1001001001009899.5%
Hermes 3 405B1001001001009899.5%
Gemma 3 4B1001001001009899.5%
Z.AI GLM 4.5 Air1001001001009799.4%
ByteDance Seed 2.0 Mini1001001001009699.2%
GPT-5.4 Mini1001001001009699.2%
Ministral 3 14B100100100989899.2%
Z.AI GLM 5 Turbo1001001001009599.1%
GPT-5.6 Luna1001001001009599.1%
Qwen 2.5 72B1001001001009599.1%
Hermes 3 70B100100100989899.1%
Qwen 3 32B1001001001009599.0%
GPT-4o Mini (temp=0)1001001001009498.9%
Qwen 3.5 9B1001001001009498.8%
Ministral 3 3B1001001001009498.8%
ByteDance Seed 1.6 Flash10010098989898.7%
ByteDance Seed 2.0 Lite10010098989898.6%
Cohere Command R+ (Aug. 2024)100100100969598.3%
Ministral 3B100100100969598.3%
GPT-5 Nano1001001001009198.2%
GPT-4.1 Nano100100100969498.1%
GPT-5.4 Nano100100100979398.0%
GPT-5.4 Nano (Reasoning, Low)1001001001008897.7%
Mistral Small 41001001001008596.9%
GPT-4o Mini (temp=1)10010094949496.7%
Llama 3.1 70B1009897959296.4%
GPT-5.41009696969295.8%
Gemma 3 12B979796969295.4%
Arcee AI: Trinity Mini10010097908995.2%
Qwen 3.6 27B1001001001005090.0%
GPT-4.110010091837589.8%
Gemini 2.5 Flash Lite1009390857588.6%
Ministral 8B10010079777385.8%
Gemma 4 26B (Reasoning)1001001001001082.0%
Cydonia 24B V4.1100100100921080.3%
Qwen 3.5 27B100100100505080.0%
Mistral NeMO1001001000060.0%