Language Writing

Can the model generate text in different languages?

Character dialogue (Italian) in a story

0-shot Language
Model Run 1 Run 2 Run 3 Run 4 Run 5 Total
Hermes 3 405B100%100%100%100%100%100%
Gemini 3 Flash (Preview)100%100%100%100%100%100%
Mistral Large 2100%100%100%100%91%98%
Claude 2.0100%100%100%100%86%97%
Magnum v2 72B100%100%100%94%91%97%
Llama 3.1 405B100%100%100%94%90%97%
GPT-4o, Aug. 6th (temp=0)100%100%100%92%90%96%
Z.AI GLM 4.6100%100%95%94%91%96%
Z.AI GLM 4.5100%100%94%92%92%96%
GPT-4o Mini (temp=1)100%100%100%93%85%95%
Z.AI GLM 4.7100%100%93%92%92%95%
GPT-4.1 Mini100%100%100%89%88%95%
DeepSeek-V2 Chat100%100%100%88%80%94%
Mistral Medium100%100%92%89%83%93%
MoonshotAI: Kimi K2.5100%100%92%86%86%93%
Qwen 2 7B100%100%94%88%82%93%
Claude Sonnet 4.5100%95%94%89%85%93%
Qwen 2 72B100%100%100%91%71%92%
Claude 3.0 Sonnet100%100%100%90%71%92%
Claude 2.1100%100%100%100%60%92%
GPT-4o, Aug. 6th (temp=1)100%92%91%90%87%92%
o4 Mini100%100%100%90%67%91%
Claude Opus 4.696%94%90%89%87%91%
Gemini 2.5 Pro100%93%93%85%85%91%
Gemini 2.5 Flash100%100%92%82%81%91%
Claude Sonnet 494%93%92%92%83%91%
Claude 3.7 Sonnet100%94%88%86%84%90%
Hermes 3 70B100%100%100%91%58%90%
Inflection 3 (PI)100%100%92%82%75%90%
GPT-4o Mini (temp=0)100%92%86%86%85%90%
Claude 3.5 Sonnet94%92%88%87%87%89%
Mistral Large100%100%100%100%46%89%
o4 Mini High100%100%93%78%75%89%
Gemini 2.5 Flash Lite100%93%90%83%79%89%
Claude Opus 4.5100%93%87%86%80%89%
Llama 3.2 90B (Vision)100%100%100%78%67%89%
Lumimaid v0.2 8B100%100%100%88%56%89%
GPT-4.1100%92%85%85%82%89%
Llama 3.2 3B100%100%90%78%71%88%
Gemini 3 Pro (Preview)95%92%90%82%79%88%
Liquid: LFM 40B MoE100%100%100%83%50%87%
lzlv 70B100%100%100%77%50%85%
GPT-4o, May 13th (temp=0)89%88%85%83%78%85%
Magnum 72B100%100%90%86%47%84%
Llama 3.2 11B (Vision)100%100%92%80%50%84%
Qwen 2.5 72B100%100%92%88%41%84%
Claude Haiku 4.594%94%93%73%67%84%
GPT-4 Turbo100%91%88%83%55%83%
Inflection 3 (Productivity)100%93%89%88%44%83%
Llama 3 TenyxChat-DaybreakStorywriter 70B93%89%80%75%75%82%
Gemini Pro 1.5100%90%88%78%56%82%
Llama 3 70B100%100%80%75%55%82%
Claude Opus 493%88%77%75%69%81%
Cohere Command R+ (Apr. 2024)100%100%88%57%54%80%
GPT-4.1 Nano100%89%86%71%50%79%
Claude 3.5 Sonnet (new)90%82%82%75%67%79%
Phi-3 Medium 128k100%100%88%67%40%79%
Claude 3 Haiku100%100%100%50%40%78%
Claude 3.5 Haiku100%86%80%64%60%78%
Gemma 2 27B100%90%79%70%50%78%
Phi-3 Mini 128k100%100%83%50%50%77%
AI21 Jamba100%100%77%58%44%76%
Llama 3.1 70B100%100%79%50%50%76%
Hermes 2 Theta 8B100%94%93%89%0%75%
Z.AI GLM 4.7 Flash100%100%81%50%43%75%
Gemma 2 9B86%75%73%69%68%74%
Mistral NeMO100%91%90%88%0%74%
Cohere Command R+ (Aug. 2024)100%86%78%50%50%73%
MythoMist 7B100%75%71%67%50%73%
MythoMax 13B100%86%67%56%50%72%
Goliath 120B100%70%67%60%57%71%
Mistral Small Creative100%92%90%69%0%70%
AI21 Jamba 1.5 Large100%100%100%50%0%70%
Phi-3.5 Mini 128k100%100%50%50%40%68%
Llama 3.1 Nemotron 70B100%100%86%50%0%67%
WizardLM 2 8x22b100%100%75%56%0%66%
Llama 3.2 1B89%71%60%56%50%65%
Toppy M 7B83%78%67%50%44%64%
AI21 Jamba 1.5 Mini100%63%62%55%43%64%
Llama 3.1 Euryale 70B v2.2100%90%80%50%0%64%
Sao10K L3.1 70B Hanami x1100%80%71%67%0%64%
Ministral 3B79%71%57%54%44%61%
GPT-4o, May 13th (temp=1)85%78%71%67%0%60%
Llama 3.1 8B83%83%75%50%0%58%
Gemini Flash 1.5100%81%60%50%0%58%
Writer: Palmyra X5100%100%89%0%0%58%
Ministral 8B100%73%64%46%0%57%
Rocinante 12B71%70%43%0%0%37%
Mistral Nemo 12B Celeste71%54%50%0%0%35%
EVA Qwen 2.5 14B67%58%33%0%0%32%
Fimbulvetr 11B v278%75%0%0%0%31%
MN GRAND Gutenberg Lyra4 12B Madness100%40%0%0%0%28%
Llama 3 Euryale 70B v2.175%0%0%0%0%15%
78.95%