Language Writing

Can the model generate text in different languages?

Character dialogue (Hindi) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Claude 3 Haiku100%100%100%100%100%100%
GPT-4o, Aug. 6th (temp=0)100%100%100%100%100%100%
DeepSeek-V2 Chat100%100%100%100%100%100%
GPT-4o Mini (temp=0)100%100%100%100%100%100%
GPT-4o Mini (temp=1)100%100%100%100%100%100%
GPT-4.1 Mini100%100%100%100%100%100%
o4 Mini100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%100%100%
Z.AI GLM 4.5100%100%100%100%100%100%
GPT-4o, May 13th (temp=0)100%100%100%100%92%98%
Claude Opus 4.5100%100%100%96%95%98%
o4 Mini High100%100%100%100%91%98%
Hermes 3 405B100%100%100%100%88%98%
GPT-4o, Aug. 6th (temp=1)100%100%100%100%86%97%
Phi-3 Medium 128k100%100%100%100%80%96%
Z.AI GLM 4.6100%100%100%100%58%92%
Inflection 3 (Productivity)100%100%93%88%77%92%
Hermes 2 Theta 8B100%100%92%86%78%91%
GPT-4.1 Nano100%100%100%100%50%90%
GPT-4.1100%100%100%100%45%89%
Liquid: LFM 40B MoE100%100%100%100%44%89%
Claude 3.5 Sonnet100%100%100%94%50%89%
Gemini 2.5 Flash100%100%100%92%50%88%
Claude 3.7 Sonnet100%100%95%95%50%88%
Llama 3.1 405B100%93%93%91%55%86%
Claude 3.5 Haiku100%89%88%80%75%86%
Inflection 3 (PI)100%100%89%75%63%85%
Hermes 3 70B100%100%91%71%60%84%
GPT-4 Turbo100%100%94%70%45%82%
Gemini Pro 1.5100%100%83%64%60%81%
Phi-3.5 Mini 128k100%100%100%50%50%80%
Mistral NeMO100%100%100%100%0%80%
Claude Haiku 4.5100%100%93%53%53%80%
AI21 Jamba 1.5 Mini100%100%100%50%44%79%
Gemini 2.5 Pro100%100%100%94%0%79%
GPT-4o, May 13th (temp=1)100%100%100%88%0%78%
Llama 3.2 90B (Vision)100%90%78%60%57%77%
Claude Sonnet 493%92%92%53%50%76%
Goliath 120B100%100%67%63%50%76%
Mistral Small Creative92%90%86%50%50%74%
Claude 2.0100%100%62%57%43%72%
Z.AI GLM 4.7 Flash100%100%57%50%50%71%
MoonshotAI: Kimi K2.5100%100%54%54%47%71%
Qwen 2 7B100%100%57%50%47%71%
Claude 2.1100%100%50%50%50%70%
Cohere Command R+ (Apr. 2024)100%100%88%60%0%70%
Gemma 2 9B100%100%86%60%0%69%
Claude 3.0 Sonnet100%67%63%57%57%69%
Cohere Command R+ (Aug. 2024)100%100%88%55%0%68%
Llama 3.2 3B100%83%60%50%45%68%
Lumimaid v0.2 8B100%89%67%43%40%68%
Toppy M 7B100%78%64%46%46%67%
Gemini 2.5 Flash Lite93%64%60%60%56%67%
Gemma 2 27B100%67%63%60%43%66%
Llama 3.1 70B100%60%57%56%54%65%
Claude Opus 4.6100%59%55%55%54%64%
Llama 3.2 1B100%67%50%50%50%63%
Mistral Large 2100%100%62%50%0%62%
Phi-3 Mini 128k100%60%50%50%44%61%
Llama 3 Euryale 70B v2.1100%53%50%50%50%61%
AI21 Jamba100%64%53%44%40%60%
AI21 Jamba 1.5 Large100%100%50%50%0%60%
MythoMax 13B100%55%50%50%44%60%
lzlv 70B100%56%54%45%38%58%
Qwen 2.5 72B100%48%47%46%46%57%
Qwen 2 72B67%57%57%56%50%57%
Mistral Nemo 12B Celeste83%63%57%50%33%57%
Llama 3 70B60%58%57%56%55%57%
Claude Opus 483%50%50%47%47%55%
WizardLM 2 8x22b93%64%63%57%0%55%
Z.AI GLM 4.758%55%54%54%53%55%
Mistral Medium64%55%54%47%47%53%
MN GRAND Gutenberg Lyra4 12B Madness100%67%50%50%0%53%
Llama 3.1 8B64%60%50%46%44%53%
Llama 3.2 11B (Vision)100%57%57%50%0%53%
Gemini Flash 1.580%64%64%55%0%52%
Magnum v2 72B55%53%50%50%50%51%
Llama 3.1 Euryale 70B v2.2100%71%50%33%0%51%
Ministral 3B53%50%50%50%50%51%
Claude Sonnet 4.553%52%50%50%47%50%
Mistral Large50%50%50%50%50%50%
Claude 3.5 Sonnet (new)58%50%50%50%40%50%
Fimbulvetr 11B v2100%67%64%0%0%46%
Llama 3 TenyxChat-DaybreakStorywriter 70B78%57%50%43%0%46%
Magnum 72B60%50%50%40%25%45%
Gemini 3 Pro (Preview)56%54%54%53%0%43%
MythoMist 7B83%58%56%0%0%40%
EVA Qwen 2.5 14B67%63%56%0%0%37%
Ministral 8B64%56%50%0%0%34%
Sao10K L3.1 70B Hanami x193%29%29%0%0%30%
Rocinante 12B67%50%0%0%0%23%
Llama 3.1 Nemotron 70B60%50%0%0%0%22%
Writer: Palmyra X50%0%0%0%0%0%
69.75%