Language Writing

Can the model generate text in different languages?

Model Total â–¼Character dialogue (Spanish) in a storyCharacter dialogue (French) in a storyCharacter dialogue (German) in a storyCharacter dialogue (Italian) in a storyCharacter dialogue (Hindi) in a story
Gemini 3 Flash (Preview)99%100%96%98%100%100%
Hermes 3 405B97%96%96%94%100%98%
Z.AI GLM 4.596%100%94%91%96%100%
o4 Mini95%95%94%93%91%100%
GPT-4o Mini (temp=1)95%96%90%92%95%100%
DeepSeek-V2 Chat93%93%96%84%94%100%
GPT-4.1 Mini93%96%86%88%95%100%
GPT-4o Mini (temp=0)92%95%87%90%90%100%
GPT-4o, Aug. 6th (temp=0)92%81%90%94%96%100%
Claude 2.091%93%94%100%97%72%
o4 Mini High91%85%95%89%89%98%
Gemini 2.5 Flash90%90%90%94%91%88%
GPT-4.189%90%87%92%89%89%
Llama 3.1 405B89%89%89%84%97%86%
MoonshotAI: Kimi K2.589%93%91%96%93%71%
1–15 of 93
Page 1 / 7

Character dialogue (Spanish) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Gemini 3 Flash (Preview)100%100%100%100%100%100%
Z.AI GLM 4.5100%100%100%100%100%100%
Z.AI GLM 4.7100%100%100%100%90%98%
GPT-4.1 Nano100%100%100%100%86%97%
Llama 3.2 11B (Vision)100%100%100%93%90%97%
GPT-4.1 Mini100%100%100%92%90%96%
GPT-4o Mini (temp=1)100%100%94%94%93%96%
Hermes 3 405B100%100%100%92%88%96%
GPT-4o Mini (temp=0)100%100%93%92%92%95%
o4 Mini100%100%93%92%90%95%
Mistral Large100%100%100%88%86%95%
GPT-4 Turbo100%100%93%92%85%94%
Claude 3 Haiku100%100%90%89%88%93%
MoonshotAI: Kimi K2.5100%94%93%91%88%93%
Claude 2.0100%100%100%86%80%93%
DeepSeek-V2 Chat100%100%100%83%80%93%
Qwen 2.5 72B100%94%92%89%87%92%
Qwen 2 72B100%100%100%88%71%92%
Llama 3.1 Euryale 70B v2.2100%100%100%100%58%92%
Claude Sonnet 4100%93%93%86%85%91%
Cohere Command R+ (Apr. 2024)100%100%100%86%70%91%
Claude 2.1100%100%100%86%67%90%
Llama 3.2 3B100%100%100%86%67%90%
Phi-3 Mini 128k100%100%100%100%50%90%
GPT-4.1100%100%100%93%57%90%
Claude Opus 4.593%93%88%87%87%90%
Gemini 2.5 Flash100%100%92%91%64%90%
lzlv 70B100%90%88%86%83%89%
Claude Opus 4.691%91%91%91%82%89%
Hermes 3 70B100%100%91%80%75%89%
GPT-4o, Aug. 6th (temp=1)100%90%89%86%80%89%
Llama 3.1 405B100%90%88%88%78%89%
Mistral Medium100%94%85%80%80%88%
Inflection 3 (Productivity)100%88%85%84%82%88%
Claude Opus 493%93%86%85%79%87%
Phi-3 Medium 128k100%100%100%67%67%87%
Llama 3 TenyxChat-DaybreakStorywriter 70B100%100%100%70%63%87%
Claude Sonnet 4.588%88%88%86%82%86%
Claude 3.5 Sonnet89%88%88%84%82%86%
Claude 3.7 Sonnet95%89%88%81%77%86%
Llama 3 70B100%87%82%80%80%86%
o4 Mini High100%91%82%77%76%85%
Inflection 3 (PI)91%87%87%85%78%85%
Claude 3.0 Sonnet92%91%89%83%71%85%
Llama 3.2 90B (Vision)100%90%86%83%67%85%
Gemini 3 Pro (Preview)94%93%89%78%69%85%
AI21 Jamba100%100%86%80%55%84%
Z.AI GLM 4.695%92%83%79%67%83%
Gemini 2.5 Pro100%86%83%72%71%82%
GPT-4o, May 13th (temp=1)100%93%80%69%65%81%
GPT-4o, Aug. 6th (temp=0)89%88%88%78%63%81%
Claude 3.5 Sonnet (new)90%83%83%80%67%81%
Magnum 72B100%77%75%73%67%78%
GPT-4o, May 13th (temp=0)86%83%82%80%58%78%
Toppy M 7B86%83%82%77%58%77%
Llama 3.2 1B100%100%92%56%38%77%
AI21 Jamba 1.5 Large95%93%86%60%50%77%
Cohere Command R+ (Aug. 2024)100%90%89%53%45%76%
Llama 3.1 70B100%75%75%71%55%75%
Magnum v2 72B100%100%91%44%38%75%
Liquid: LFM 40B MoE100%82%75%57%50%73%
Phi-3.5 Mini 128k100%100%60%50%50%72%
AI21 Jamba 1.5 Mini100%91%67%56%46%72%
Claude Haiku 4.589%82%71%63%53%71%
Hermes 2 Theta 8B89%80%78%56%50%70%
Llama 3.1 8B100%89%83%79%0%70%
Gemma 2 27B83%82%64%62%54%69%
Qwen 2 7B100%78%70%50%43%68%
Llama 3 Euryale 70B v2.1100%78%70%50%40%68%
Goliath 120B100%73%71%50%38%66%
Sao10K L3.1 70B Hanami x1100%71%60%50%50%66%
Z.AI GLM 4.7 Flash100%100%67%64%0%66%
MythoMax 13B100%67%60%50%50%65%
Mistral Large 2100%90%85%50%0%65%
Gemini Pro 1.588%73%67%67%29%64%
Mistral NeMO100%92%86%40%0%63%
WizardLM 2 8x22b92%80%79%67%0%63%
Claude 3.5 Haiku80%64%58%55%50%61%
Gemini 2.5 Flash Lite91%89%73%50%0%61%
Writer: Palmyra X5100%100%100%0%0%60%
Lumimaid v0.2 8B100%100%58%33%0%58%
Gemma 2 9B79%69%44%43%43%55%
Mistral Nemo 12B Celeste91%91%50%33%0%53%
Ministral 3B86%55%50%43%23%51%
MN GRAND Gutenberg Lyra4 12B Madness100%79%64%0%0%48%
Rocinante 12B64%50%45%38%33%46%
Fimbulvetr 11B v293%64%58%0%0%43%
Ministral 8B86%73%48%0%0%41%
MythoMist 7B76%60%50%20%0%41%
Gemini Flash 1.588%60%57%0%0%41%
EVA Qwen 2.5 14B100%50%0%0%0%30%
Mistral Small Creative50%50%0%0%0%20%
Llama 3.1 Nemotron 70B50%0%0%0%0%10%
76.99%

Character dialogue (French) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Hermes 3 70B100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%92%89%96%
DeepSeek-V2 Chat100%100%100%90%90%96%
Hermes 3 405B100%100%100%89%89%96%
o4 Mini High100%100%100%94%80%95%
Z.AI GLM 4.5100%100%91%91%90%94%
o4 Mini100%100%92%92%88%94%
Mistral Large100%100%100%100%71%94%
Claude 2.0100%100%100%90%80%94%
Z.AI GLM 4.7100%100%94%88%81%93%
Phi-3.5 Mini 128k100%100%100%100%63%93%
Claude 3.7 Sonnet100%94%94%89%83%92%
Cohere Command R+ (Apr. 2024)100%100%90%89%80%92%
Mistral Medium100%100%100%95%63%92%
MoonshotAI: Kimi K2.5100%100%87%86%81%91%
Claude 3.0 Sonnet100%100%89%86%78%90%
Llama 3.2 1B100%100%100%100%50%90%
GPT-4o, Aug. 6th (temp=0)100%100%90%82%78%90%
GPT-4o Mini (temp=1)93%92%92%87%86%90%
Gemini 2.5 Flash100%92%90%88%80%90%
Z.AI GLM 4.6100%95%94%87%74%90%
Phi-3 Medium 128k100%100%88%83%77%90%
Llama 3.1 405B100%100%88%80%75%89%
GPT-4.1 Nano100%100%90%88%64%88%
Inflection 3 (Productivity)94%93%90%83%80%88%
GPT-4.1100%91%85%82%79%87%
GPT-4 Turbo94%92%91%89%69%87%
GPT-4o Mini (temp=0)94%92%86%85%77%87%
MythoMist 7B100%100%83%80%67%86%
Claude 2.1100%100%83%80%67%86%
GPT-4.1 Mini91%89%89%88%73%86%
Claude Sonnet 4.596%89%84%82%77%86%
Llama 3.2 11B (Vision)92%90%83%80%80%85%
Claude Sonnet 493%92%82%80%77%85%
Gemini 3 Pro (Preview)94%93%83%82%71%85%
Hermes 2 Theta 8B94%92%85%82%70%84%
Z.AI GLM 4.7 Flash100%88%80%78%75%84%
Qwen 2 72B100%89%86%82%62%84%
GPT-4o, May 13th (temp=1)91%86%85%80%75%83%
AI21 Jamba 1.5 Large100%100%85%81%50%83%
Claude 3.5 Sonnet90%87%82%77%76%83%
Claude Haiku 4.5100%92%86%73%62%82%
Gemini 2.5 Flash Lite100%93%77%75%67%82%
Claude Opus 4.589%89%83%78%71%82%
Gemini 2.5 Pro86%86%81%79%77%82%
GPT-4o, May 13th (temp=0)91%90%82%77%69%82%
Llama 3.2 3B100%100%80%71%57%82%
GPT-4o, Aug. 6th (temp=1)100%82%79%73%73%81%
Claude Opus 4.688%81%79%78%78%81%
Gemma 2 27B86%86%78%78%77%81%
Claude 3.5 Sonnet (new)100%91%75%70%67%81%
Inflection 3 (PI)93%92%88%75%50%80%
Qwen 2 7B100%89%75%70%63%79%
Claude Opus 483%82%82%75%74%79%
Gemini Pro 1.5100%88%82%67%56%78%
Llama 3 70B89%88%80%70%64%78%
Llama 3.1 70B94%77%75%73%69%78%
Writer: Palmyra X5100%100%94%86%0%76%
Toppy M 7B100%90%73%67%50%76%
Fimbulvetr 11B v291%89%82%62%50%75%
Llama 3.2 90B (Vision)100%100%83%67%20%74%
Magnum v2 72B100%89%67%63%43%72%
Liquid: LFM 40B MoE100%100%100%57%0%71%
Llama 3.1 8B90%90%89%83%0%70%
lzlv 70B89%80%70%60%50%70%
Goliath 120B100%83%67%56%43%70%
Mistral Nemo 12B Celeste100%83%71%67%25%69%
Mistral Large 292%91%75%73%0%66%
AI21 Jamba85%83%62%50%50%66%
Phi-3 Mini 128k100%67%67%56%40%66%
Llama 3 TenyxChat-DaybreakStorywriter 70B75%70%67%60%50%64%
MythoMax 13B75%67%63%58%57%64%
Gemma 2 9B89%86%71%40%33%64%
Rocinante 12B100%67%63%50%38%63%
Ministral 3B100%94%71%50%0%63%
Llama 3.1 Euryale 70B v2.2100%85%67%64%0%63%
Ministral 8B100%75%67%64%0%61%
Lumimaid v0.2 8B100%53%50%50%40%59%
Magnum 72B100%83%57%50%0%58%
Cohere Command R+ (Aug. 2024)92%70%42%41%38%56%
Claude 3 Haiku83%50%50%50%43%55%
Sao10K L3.1 70B Hanami x1100%80%71%20%0%54%
Qwen 2.5 72B100%95%67%0%0%52%
Llama 3 Euryale 70B v2.1100%63%60%33%0%51%
Claude 3.5 Haiku64%60%45%45%36%50%
Mistral NeMO100%83%67%0%0%50%
Mistral Small Creative100%50%50%50%0%50%
AI21 Jamba 1.5 Mini100%55%50%43%0%49%
WizardLM 2 8x22b90%83%71%0%0%49%
MN GRAND Gutenberg Lyra4 12B Madness50%50%43%0%0%29%
Gemini Flash 1.589%0%0%0%0%18%
Llama 3.1 Nemotron 70B44%0%0%0%0%9%
EVA Qwen 2.5 14B33%0%0%0%0%7%
75.41%

Character dialogue (German) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Claude 2.0100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%92%98%
Z.AI GLM 4.7100%100%95%94%94%97%
MoonshotAI: Kimi K2.5100%100%100%94%87%96%
Claude 3.0 Sonnet100%100%100%89%88%95%
Hermes 3 405B100%100%100%89%82%94%
Gemini 2.5 Flash100%100%92%89%88%94%
GPT-4o, Aug. 6th (temp=0)100%100%90%89%89%94%
Claude Opus 4.6100%95%95%94%81%93%
GPT-4o, May 13th (temp=0)100%93%93%91%89%93%
o4 Mini100%100%93%90%82%93%
GPT-4.1100%100%92%85%83%92%
GPT-4o Mini (temp=1)100%94%92%92%80%92%
Z.AI GLM 4.594%92%92%90%88%91%
Inflection 3 (Productivity)100%92%91%90%82%91%
GPT-4 Turbo100%100%92%86%77%91%
Gemini 2.5 Flash Lite100%100%100%78%75%91%
GPT-4o Mini (temp=0)94%93%92%86%86%90%
Phi-3 Mini 128k100%100%100%80%69%90%
o4 Mini High100%90%89%86%81%89%
Claude Opus 4100%93%90%90%67%88%
Mistral Medium100%92%92%82%73%88%
GPT-4.1 Mini100%100%82%80%78%88%
Llama 3 70B100%90%88%82%80%88%
Claude 3.5 Sonnet94%90%88%87%80%88%
Claude 3.5 Sonnet (new)92%92%90%82%82%87%
AI21 Jamba 1.5 Large100%91%88%77%72%85%
Llama 3.2 11B (Vision)91%91%88%82%75%85%
Claude Opus 4.592%88%84%81%79%85%
DeepSeek-V2 Chat100%82%80%80%80%84%
GPT-4o, Aug. 6th (temp=1)90%88%83%80%80%84%
Mistral Large89%89%88%88%67%84%
Llama 3.1 405B100%86%85%75%73%84%
Claude 3.7 Sonnet93%87%84%79%75%84%
Inflection 3 (PI)100%91%91%90%40%82%
Claude Sonnet 4100%86%85%77%64%82%
Z.AI GLM 4.7 Flash100%100%75%75%62%82%
Goliath 120B100%88%86%82%55%82%
Gemma 2 9B85%82%81%80%80%82%
GPT-4.1 Nano100%100%91%67%50%82%
GPT-4o, May 13th (temp=1)100%86%82%73%64%81%
Claude Sonnet 4.591%85%82%75%68%80%
Claude 2.1100%100%100%100%0%80%
Llama 3.2 1B100%100%90%75%33%80%
Gemini 2.5 Pro86%80%80%77%75%80%
Qwen 2 72B100%90%90%83%33%79%
Llama 3.2 3B100%88%78%75%56%79%
MythoMist 7B100%100%100%57%38%79%
Phi-3.5 Mini 128k100%100%80%75%38%78%
Hermes 2 Theta 8B94%89%88%70%50%78%
Gemini 3 Pro (Preview)100%100%95%93%0%78%
Magnum 72B100%94%90%50%50%77%
lzlv 70B100%89%81%67%47%77%
Llama 3.2 90B (Vision)86%85%77%73%62%76%
Llama 3.1 70B100%80%71%67%56%75%
Claude Haiku 4.585%82%77%65%64%75%
Magnum v2 72B91%88%86%75%33%74%
Gemini Pro 1.5100%92%88%83%0%73%
MythoMax 13B90%90%60%56%50%69%
AI21 Jamba95%75%75%50%50%69%
Gemma 2 27B94%92%85%40%33%69%
Llama 3 TenyxChat-DaybreakStorywriter 70B89%82%63%57%54%69%
Hermes 3 70B100%100%91%50%0%68%
Z.AI GLM 4.6100%89%73%71%0%67%
Cohere Command R+ (Aug. 2024)86%75%62%60%50%66%
Llama 3.1 Euryale 70B v2.294%78%73%73%0%64%
Claude 3 Haiku100%73%50%50%43%63%
Claude 3.5 Haiku88%75%60%50%36%62%
Liquid: LFM 40B MoE81%80%80%67%0%62%
Fimbulvetr 11B v285%82%73%67%0%61%
Rocinante 12B100%100%70%33%0%61%
Llama 3.1 8B92%80%75%55%0%60%
Sao10K L3.1 70B Hanami x179%71%67%44%36%59%
Cohere Command R+ (Apr. 2024)71%57%57%57%55%59%
Qwen 2 7B85%85%64%55%0%58%
MN GRAND Gutenberg Lyra4 12B Madness90%70%67%50%0%55%
Phi-3 Medium 128k100%75%58%43%0%55%
Mistral Large 2100%94%77%0%0%54%
Lumimaid v0.2 8B83%75%70%40%0%54%
Ministral 8B100%92%62%0%0%51%
Llama 3.1 Nemotron 70B100%86%67%0%0%50%
Toppy M 7B91%55%55%46%0%49%
Llama 3 Euryale 70B v2.1100%80%56%0%0%47%
Ministral 3B69%59%56%42%0%45%
Mistral NeMO100%91%33%0%0%45%
AI21 Jamba 1.5 Mini82%60%50%29%0%44%
WizardLM 2 8x22b100%63%56%0%0%44%
Writer: Palmyra X5100%89%0%0%0%38%
Mistral Nemo 12B Celeste50%44%44%42%0%36%
Qwen 2.5 72B100%79%0%0%0%36%
Mistral Small Creative75%50%50%0%0%35%
EVA Qwen 2.5 14B75%50%0%0%0%25%
Gemini Flash 1.593%0%0%0%0%19%
73.38%

Character dialogue (Italian) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Hermes 3 405B100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%100%100%
Mistral Large 2100%100%100%100%91%98%
Claude 2.0100%100%100%100%86%97%
Magnum v2 72B100%100%100%94%91%97%
Llama 3.1 405B100%100%100%94%90%97%
GPT-4o, Aug. 6th (temp=0)100%100%100%92%90%96%
Z.AI GLM 4.6100%100%95%94%91%96%
Z.AI GLM 4.5100%100%94%92%92%96%
GPT-4o Mini (temp=1)100%100%100%93%85%95%
Z.AI GLM 4.7100%100%93%92%92%95%
GPT-4.1 Mini100%100%100%89%88%95%
DeepSeek-V2 Chat100%100%100%88%80%94%
Mistral Medium100%100%92%89%83%93%
MoonshotAI: Kimi K2.5100%100%92%86%86%93%
Qwen 2 7B100%100%94%88%82%93%
Claude Sonnet 4.5100%95%94%89%85%93%
Qwen 2 72B100%100%100%91%71%92%
Claude 3.0 Sonnet100%100%100%90%71%92%
Claude 2.1100%100%100%100%60%92%
GPT-4o, Aug. 6th (temp=1)100%92%91%90%87%92%
o4 Mini100%100%100%90%67%91%
Claude Opus 4.696%94%90%89%87%91%
Gemini 2.5 Pro100%93%93%85%85%91%
Gemini 2.5 Flash100%100%92%82%81%91%
Claude Sonnet 494%93%92%92%83%91%
Claude 3.7 Sonnet100%94%88%86%84%90%
Hermes 3 70B100%100%100%91%58%90%
Inflection 3 (PI)100%100%92%82%75%90%
GPT-4o Mini (temp=0)100%92%86%86%85%90%
Claude 3.5 Sonnet94%92%88%87%87%89%
Mistral Large100%100%100%100%46%89%
o4 Mini High100%100%93%78%75%89%
Gemini 2.5 Flash Lite100%93%90%83%79%89%
Claude Opus 4.5100%93%87%86%80%89%
Llama 3.2 90B (Vision)100%100%100%78%67%89%
Lumimaid v0.2 8B100%100%100%88%56%89%
GPT-4.1100%92%85%85%82%89%
Llama 3.2 3B100%100%90%78%71%88%
Gemini 3 Pro (Preview)95%92%90%82%79%88%
Liquid: LFM 40B MoE100%100%100%83%50%87%
lzlv 70B100%100%100%77%50%85%
GPT-4o, May 13th (temp=0)89%88%85%83%78%85%
Magnum 72B100%100%90%86%47%84%
Llama 3.2 11B (Vision)100%100%92%80%50%84%
Qwen 2.5 72B100%100%92%88%41%84%
Claude Haiku 4.594%94%93%73%67%84%
GPT-4 Turbo100%91%88%83%55%83%
Inflection 3 (Productivity)100%93%89%88%44%83%
Llama 3 TenyxChat-DaybreakStorywriter 70B93%89%80%75%75%82%
Gemini Pro 1.5100%90%88%78%56%82%
Llama 3 70B100%100%80%75%55%82%
Claude Opus 493%88%77%75%69%81%
Cohere Command R+ (Apr. 2024)100%100%88%57%54%80%
GPT-4.1 Nano100%89%86%71%50%79%
Claude 3.5 Sonnet (new)90%82%82%75%67%79%
Phi-3 Medium 128k100%100%88%67%40%79%
Claude 3 Haiku100%100%100%50%40%78%
Claude 3.5 Haiku100%86%80%64%60%78%
Gemma 2 27B100%90%79%70%50%78%
Phi-3 Mini 128k100%100%83%50%50%77%
AI21 Jamba100%100%77%58%44%76%
Llama 3.1 70B100%100%79%50%50%76%
Hermes 2 Theta 8B100%94%93%89%0%75%
Z.AI GLM 4.7 Flash100%100%81%50%43%75%
Gemma 2 9B86%75%73%69%68%74%
Mistral NeMO100%91%90%88%0%74%
Cohere Command R+ (Aug. 2024)100%86%78%50%50%73%
MythoMist 7B100%75%71%67%50%73%
MythoMax 13B100%86%67%56%50%72%
Goliath 120B100%70%67%60%57%71%
Mistral Small Creative100%92%90%69%0%70%
AI21 Jamba 1.5 Large100%100%100%50%0%70%
Phi-3.5 Mini 128k100%100%50%50%40%68%
Llama 3.1 Nemotron 70B100%100%86%50%0%67%
WizardLM 2 8x22b100%100%75%56%0%66%
Llama 3.2 1B89%71%60%56%50%65%
Toppy M 7B83%78%67%50%44%64%
AI21 Jamba 1.5 Mini100%63%62%55%43%64%
Llama 3.1 Euryale 70B v2.2100%90%80%50%0%64%
Sao10K L3.1 70B Hanami x1100%80%71%67%0%64%
Ministral 3B79%71%57%54%44%61%
GPT-4o, May 13th (temp=1)85%78%71%67%0%60%
Llama 3.1 8B83%83%75%50%0%58%
Gemini Flash 1.5100%81%60%50%0%58%
Writer: Palmyra X5100%100%89%0%0%58%
Ministral 8B100%73%64%46%0%57%
Rocinante 12B71%70%43%0%0%37%
Mistral Nemo 12B Celeste71%54%50%0%0%35%
EVA Qwen 2.5 14B67%58%33%0%0%32%
Fimbulvetr 11B v278%75%0%0%0%31%
MN GRAND Gutenberg Lyra4 12B Madness100%40%0%0%0%28%
Llama 3 Euryale 70B v2.175%0%0%0%0%15%
78.95%

Character dialogue (Hindi) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Claude 3 Haiku100%100%100%100%100%100%
GPT-4o, Aug. 6th (temp=0)100%100%100%100%100%100%
DeepSeek-V2 Chat100%100%100%100%100%100%
GPT-4o Mini (temp=0)100%100%100%100%100%100%
GPT-4o Mini (temp=1)100%100%100%100%100%100%
GPT-4.1 Mini100%100%100%100%100%100%
o4 Mini100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%100%100%
Z.AI GLM 4.5100%100%100%100%100%100%
GPT-4o, May 13th (temp=0)100%100%100%100%92%98%
Claude Opus 4.5100%100%100%96%95%98%
o4 Mini High100%100%100%100%91%98%
Hermes 3 405B100%100%100%100%88%98%
GPT-4o, Aug. 6th (temp=1)100%100%100%100%86%97%
Phi-3 Medium 128k100%100%100%100%80%96%
Z.AI GLM 4.6100%100%100%100%58%92%
Inflection 3 (Productivity)100%100%93%88%77%92%
Hermes 2 Theta 8B100%100%92%86%78%91%
GPT-4.1 Nano100%100%100%100%50%90%
GPT-4.1100%100%100%100%45%89%
Liquid: LFM 40B MoE100%100%100%100%44%89%
Claude 3.5 Sonnet100%100%100%94%50%89%
Gemini 2.5 Flash100%100%100%92%50%88%
Claude 3.7 Sonnet100%100%95%95%50%88%
Llama 3.1 405B100%93%93%91%55%86%
Claude 3.5 Haiku100%89%88%80%75%86%
Inflection 3 (PI)100%100%89%75%63%85%
Hermes 3 70B100%100%91%71%60%84%
GPT-4 Turbo100%100%94%70%45%82%
Gemini Pro 1.5100%100%83%64%60%81%
Phi-3.5 Mini 128k100%100%100%50%50%80%
Mistral NeMO100%100%100%100%0%80%
Claude Haiku 4.5100%100%93%53%53%80%
AI21 Jamba 1.5 Mini100%100%100%50%44%79%
Gemini 2.5 Pro100%100%100%94%0%79%
GPT-4o, May 13th (temp=1)100%100%100%88%0%78%
Llama 3.2 90B (Vision)100%90%78%60%57%77%
Claude Sonnet 493%92%92%53%50%76%
Goliath 120B100%100%67%63%50%76%
Mistral Small Creative92%90%86%50%50%74%
Claude 2.0100%100%62%57%43%72%
Z.AI GLM 4.7 Flash100%100%57%50%50%71%
MoonshotAI: Kimi K2.5100%100%54%54%47%71%
Qwen 2 7B100%100%57%50%47%71%
Claude 2.1100%100%50%50%50%70%
Cohere Command R+ (Apr. 2024)100%100%88%60%0%70%
Gemma 2 9B100%100%86%60%0%69%
Claude 3.0 Sonnet100%67%63%57%57%69%
Cohere Command R+ (Aug. 2024)100%100%88%55%0%68%
Llama 3.2 3B100%83%60%50%45%68%
Lumimaid v0.2 8B100%89%67%43%40%68%
Toppy M 7B100%78%64%46%46%67%
Gemini 2.5 Flash Lite93%64%60%60%56%67%
Gemma 2 27B100%67%63%60%43%66%
Llama 3.1 70B100%60%57%56%54%65%
Claude Opus 4.6100%59%55%55%54%64%
Llama 3.2 1B100%67%50%50%50%63%
Mistral Large 2100%100%62%50%0%62%
Phi-3 Mini 128k100%60%50%50%44%61%
Llama 3 Euryale 70B v2.1100%53%50%50%50%61%
AI21 Jamba100%64%53%44%40%60%
AI21 Jamba 1.5 Large100%100%50%50%0%60%
MythoMax 13B100%55%50%50%44%60%
lzlv 70B100%56%54%45%38%58%
Qwen 2.5 72B100%48%47%46%46%57%
Qwen 2 72B67%57%57%56%50%57%
Mistral Nemo 12B Celeste83%63%57%50%33%57%
Llama 3 70B60%58%57%56%55%57%
Claude Opus 483%50%50%47%47%55%
WizardLM 2 8x22b93%64%63%57%0%55%
Z.AI GLM 4.758%55%54%54%53%55%
Mistral Medium64%55%54%47%47%53%
MN GRAND Gutenberg Lyra4 12B Madness100%67%50%50%0%53%
Llama 3.1 8B64%60%50%46%44%53%
Llama 3.2 11B (Vision)100%57%57%50%0%53%
Gemini Flash 1.580%64%64%55%0%52%
Magnum v2 72B55%53%50%50%50%51%
Llama 3.1 Euryale 70B v2.2100%71%50%33%0%51%
Ministral 3B53%50%50%50%50%51%
Claude Sonnet 4.553%52%50%50%47%50%
Mistral Large50%50%50%50%50%50%
Claude 3.5 Sonnet (new)58%50%50%50%40%50%
Fimbulvetr 11B v2100%67%64%0%0%46%
Llama 3 TenyxChat-DaybreakStorywriter 70B78%57%50%43%0%46%
Magnum 72B60%50%50%40%25%45%
Gemini 3 Pro (Preview)56%54%54%53%0%43%
MythoMist 7B83%58%56%0%0%40%
EVA Qwen 2.5 14B67%63%56%0%0%37%
Ministral 8B64%56%50%0%0%34%
Sao10K L3.1 70B Hanami x193%29%29%0%0%30%
Rocinante 12B67%50%0%0%0%23%
Llama 3.1 Nemotron 70B60%50%0%0%0%22%
Writer: Palmyra X50%0%0%0%0%0%
69.75%