Language Writing

Can the model generate text in different languages?

Character dialogue (French) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Hermes 3 70B100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%92%89%96%
DeepSeek-V2 Chat100%100%100%90%90%96%
Hermes 3 405B100%100%100%89%89%96%
o4 Mini High100%100%100%94%80%95%
Z.AI GLM 4.5100%100%91%91%90%94%
o4 Mini100%100%92%92%88%94%
Mistral Large100%100%100%100%71%94%
Claude 2.0100%100%100%90%80%94%
Z.AI GLM 4.7100%100%94%88%81%93%
Phi-3.5 Mini 128k100%100%100%100%63%93%
Claude 3.7 Sonnet100%94%94%89%83%92%
Cohere Command R+ (Apr. 2024)100%100%90%89%80%92%
Mistral Medium100%100%100%95%63%92%
MoonshotAI: Kimi K2.5100%100%87%86%81%91%
Claude 3.0 Sonnet100%100%89%86%78%90%
Llama 3.2 1B100%100%100%100%50%90%
GPT-4o, Aug. 6th (temp=0)100%100%90%82%78%90%
GPT-4o Mini (temp=1)93%92%92%87%86%90%
Gemini 2.5 Flash100%92%90%88%80%90%
Z.AI GLM 4.6100%95%94%87%74%90%
Phi-3 Medium 128k100%100%88%83%77%90%
Llama 3.1 405B100%100%88%80%75%89%
GPT-4.1 Nano100%100%90%88%64%88%
Inflection 3 (Productivity)94%93%90%83%80%88%
GPT-4.1100%91%85%82%79%87%
GPT-4 Turbo94%92%91%89%69%87%
GPT-4o Mini (temp=0)94%92%86%85%77%87%
MythoMist 7B100%100%83%80%67%86%
Claude 2.1100%100%83%80%67%86%
GPT-4.1 Mini91%89%89%88%73%86%
Claude Sonnet 4.596%89%84%82%77%86%
Llama 3.2 11B (Vision)92%90%83%80%80%85%
Claude Sonnet 493%92%82%80%77%85%
Gemini 3 Pro (Preview)94%93%83%82%71%85%
Hermes 2 Theta 8B94%92%85%82%70%84%
Z.AI GLM 4.7 Flash100%88%80%78%75%84%
Qwen 2 72B100%89%86%82%62%84%
GPT-4o, May 13th (temp=1)91%86%85%80%75%83%
AI21 Jamba 1.5 Large100%100%85%81%50%83%
Claude 3.5 Sonnet90%87%82%77%76%83%
Claude Haiku 4.5100%92%86%73%62%82%
Gemini 2.5 Flash Lite100%93%77%75%67%82%
Claude Opus 4.589%89%83%78%71%82%
Gemini 2.5 Pro86%86%81%79%77%82%
GPT-4o, May 13th (temp=0)91%90%82%77%69%82%
Llama 3.2 3B100%100%80%71%57%82%
GPT-4o, Aug. 6th (temp=1)100%82%79%73%73%81%
Claude Opus 4.688%81%79%78%78%81%
Gemma 2 27B86%86%78%78%77%81%
Claude 3.5 Sonnet (new)100%91%75%70%67%81%
Inflection 3 (PI)93%92%88%75%50%80%
Qwen 2 7B100%89%75%70%63%79%
Claude Opus 483%82%82%75%74%79%
Gemini Pro 1.5100%88%82%67%56%78%
Llama 3 70B89%88%80%70%64%78%
Llama 3.1 70B94%77%75%73%69%78%
Writer: Palmyra X5100%100%94%86%0%76%
Toppy M 7B100%90%73%67%50%76%
Fimbulvetr 11B v291%89%82%62%50%75%
Llama 3.2 90B (Vision)100%100%83%67%20%74%
Magnum v2 72B100%89%67%63%43%72%
Liquid: LFM 40B MoE100%100%100%57%0%71%
Llama 3.1 8B90%90%89%83%0%70%
lzlv 70B89%80%70%60%50%70%
Goliath 120B100%83%67%56%43%70%
Mistral Nemo 12B Celeste100%83%71%67%25%69%
Mistral Large 292%91%75%73%0%66%
AI21 Jamba85%83%62%50%50%66%
Phi-3 Mini 128k100%67%67%56%40%66%
Llama 3 TenyxChat-DaybreakStorywriter 70B75%70%67%60%50%64%
MythoMax 13B75%67%63%58%57%64%
Gemma 2 9B89%86%71%40%33%64%
Rocinante 12B100%67%63%50%38%63%
Ministral 3B100%94%71%50%0%63%
Llama 3.1 Euryale 70B v2.2100%85%67%64%0%63%
Ministral 8B100%75%67%64%0%61%
Lumimaid v0.2 8B100%53%50%50%40%59%
Magnum 72B100%83%57%50%0%58%
Cohere Command R+ (Aug. 2024)92%70%42%41%38%56%
Claude 3 Haiku83%50%50%50%43%55%
Sao10K L3.1 70B Hanami x1100%80%71%20%0%54%
Qwen 2.5 72B100%95%67%0%0%52%
Llama 3 Euryale 70B v2.1100%63%60%33%0%51%
Claude 3.5 Haiku64%60%45%45%36%50%
Mistral NeMO100%83%67%0%0%50%
Mistral Small Creative100%50%50%50%0%50%
AI21 Jamba 1.5 Mini100%55%50%43%0%49%
WizardLM 2 8x22b90%83%71%0%0%49%
MN GRAND Gutenberg Lyra4 12B Madness50%50%43%0%0%29%
Gemini Flash 1.589%0%0%0%0%18%
Llama 3.1 Nemotron 70B44%0%0%0%0%9%
EVA Qwen 2.5 14B33%0%0%0%0%7%
75.41%