Language Writing

Can the model generate text in different languages?

Character dialogue (German) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Claude 2.0100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%92%98%
Z.AI GLM 4.7100%100%95%94%94%97%
MoonshotAI: Kimi K2.5100%100%100%94%87%96%
Claude 3.0 Sonnet100%100%100%89%88%95%
Hermes 3 405B100%100%100%89%82%94%
Gemini 2.5 Flash100%100%92%89%88%94%
GPT-4o, Aug. 6th (temp=0)100%100%90%89%89%94%
Claude Opus 4.6100%95%95%94%81%93%
GPT-4o, May 13th (temp=0)100%93%93%91%89%93%
o4 Mini100%100%93%90%82%93%
GPT-4.1100%100%92%85%83%92%
GPT-4o Mini (temp=1)100%94%92%92%80%92%
Z.AI GLM 4.594%92%92%90%88%91%
Inflection 3 (Productivity)100%92%91%90%82%91%
GPT-4 Turbo100%100%92%86%77%91%
Gemini 2.5 Flash Lite100%100%100%78%75%91%
GPT-4o Mini (temp=0)94%93%92%86%86%90%
Phi-3 Mini 128k100%100%100%80%69%90%
o4 Mini High100%90%89%86%81%89%
Claude Opus 4100%93%90%90%67%88%
Mistral Medium100%92%92%82%73%88%
GPT-4.1 Mini100%100%82%80%78%88%
Llama 3 70B100%90%88%82%80%88%
Claude 3.5 Sonnet94%90%88%87%80%88%
Claude 3.5 Sonnet (new)92%92%90%82%82%87%
AI21 Jamba 1.5 Large100%91%88%77%72%85%
Llama 3.2 11B (Vision)91%91%88%82%75%85%
Claude Opus 4.592%88%84%81%79%85%
DeepSeek-V2 Chat100%82%80%80%80%84%
GPT-4o, Aug. 6th (temp=1)90%88%83%80%80%84%
Mistral Large89%89%88%88%67%84%
Llama 3.1 405B100%86%85%75%73%84%
Claude 3.7 Sonnet93%87%84%79%75%84%
Inflection 3 (PI)100%91%91%90%40%82%
Claude Sonnet 4100%86%85%77%64%82%
Z.AI GLM 4.7 Flash100%100%75%75%62%82%
Goliath 120B100%88%86%82%55%82%
Gemma 2 9B85%82%81%80%80%82%
GPT-4.1 Nano100%100%91%67%50%82%
GPT-4o, May 13th (temp=1)100%86%82%73%64%81%
Claude Sonnet 4.591%85%82%75%68%80%
Claude 2.1100%100%100%100%0%80%
Llama 3.2 1B100%100%90%75%33%80%
Gemini 2.5 Pro86%80%80%77%75%80%
Qwen 2 72B100%90%90%83%33%79%
Llama 3.2 3B100%88%78%75%56%79%
MythoMist 7B100%100%100%57%38%79%
Phi-3.5 Mini 128k100%100%80%75%38%78%
Hermes 2 Theta 8B94%89%88%70%50%78%
Gemini 3 Pro (Preview)100%100%95%93%0%78%
Magnum 72B100%94%90%50%50%77%
lzlv 70B100%89%81%67%47%77%
Llama 3.2 90B (Vision)86%85%77%73%62%76%
Llama 3.1 70B100%80%71%67%56%75%
Claude Haiku 4.585%82%77%65%64%75%
Magnum v2 72B91%88%86%75%33%74%
Gemini Pro 1.5100%92%88%83%0%73%
MythoMax 13B90%90%60%56%50%69%
AI21 Jamba95%75%75%50%50%69%
Gemma 2 27B94%92%85%40%33%69%
Llama 3 TenyxChat-DaybreakStorywriter 70B89%82%63%57%54%69%
Hermes 3 70B100%100%91%50%0%68%
Z.AI GLM 4.6100%89%73%71%0%67%
Cohere Command R+ (Aug. 2024)86%75%62%60%50%66%
Llama 3.1 Euryale 70B v2.294%78%73%73%0%64%
Claude 3 Haiku100%73%50%50%43%63%
Claude 3.5 Haiku88%75%60%50%36%62%
Liquid: LFM 40B MoE81%80%80%67%0%62%
Fimbulvetr 11B v285%82%73%67%0%61%
Rocinante 12B100%100%70%33%0%61%
Llama 3.1 8B92%80%75%55%0%60%
Sao10K L3.1 70B Hanami x179%71%67%44%36%59%
Cohere Command R+ (Apr. 2024)71%57%57%57%55%59%
Qwen 2 7B85%85%64%55%0%58%
MN GRAND Gutenberg Lyra4 12B Madness90%70%67%50%0%55%
Phi-3 Medium 128k100%75%58%43%0%55%
Mistral Large 2100%94%77%0%0%54%
Lumimaid v0.2 8B83%75%70%40%0%54%
Ministral 8B100%92%62%0%0%51%
Llama 3.1 Nemotron 70B100%86%67%0%0%50%
Toppy M 7B91%55%55%46%0%49%
Llama 3 Euryale 70B v2.1100%80%56%0%0%47%
Ministral 3B69%59%56%42%0%45%
Mistral NeMO100%91%33%0%0%45%
AI21 Jamba 1.5 Mini82%60%50%29%0%44%
WizardLM 2 8x22b100%63%56%0%0%44%
Writer: Palmyra X5100%89%0%0%0%38%
Mistral Nemo 12B Celeste50%44%44%42%0%36%
Qwen 2.5 72B100%79%0%0%0%36%
Mistral Small Creative75%50%50%0%0%35%
EVA Qwen 2.5 14B75%50%0%0%0%25%
Gemini Flash 1.593%0%0%0%0%19%
73.38%