Write N of X

Write exactly N words/sentences/paragraphs...

50 word summary

0-shot Rule following
Model Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 8 Run 9 Run 10 Total
Gemini 3 Pro (Preview)100%100%100%100%100%100%100%100%100%100%100%
o4 Mini High100%100%100%100%100%100%100%100%100%100%100%
Z.AI GLM 4.7100%100%100%100%100%100%100%100%100%100%100%
Z.AI GLM 4.7 Flash100%100%100%100%100%100%100%100%100%100%100%
GPT-4o Mini (temp=0)100%100%100%100%100%100%100%100%100%100%100%
MoonshotAI: Kimi K2.5100%100%100%100%100%100%100%100%100%98%100%
Gemini 2.5 Pro100%100%100%100%100%100%100%98%98%98%99%
Gemini 3 Flash (Preview)100%100%100%100%100%100%100%98%98%98%99%
GPT-4o, Aug. 6th (temp=0)100%100%100%100%100%100%100%98%98%98%99%
Llama 3.1 405B100%100%100%100%100%100%100%98%98%92%99%
GPT-4o, May 13th (temp=0)98%98%98%98%98%98%98%98%98%98%98%
Claude Opus 4.5100%100%100%100%100%100%100%98%92%92%98%
Claude Opus 4100%100%100%100%100%100%98%98%92%92%98%
GPT-4o Mini (temp=1)100%100%100%100%100%98%98%98%92%92%98%
GPT-4o, May 13th (temp=1)100%100%100%100%100%98%98%98%98%77%97%
o4 Mini100%100%100%100%100%100%100%98%92%77%97%
Claude Opus 4.6100%100%98%98%98%98%98%98%98%77%97%
Llama 3 70B100%100%100%100%100%98%98%92%92%77%96%
GPT-4.1 Mini100%100%100%100%100%98%98%92%92%77%96%
Z.AI GLM 4.6100%100%100%100%100%100%98%98%77%77%95%
Llama 3.1 Nemotron 70B100%100%100%100%100%98%98%98%98%54%95%
GPT-4o, Aug. 6th (temp=1)100%100%100%100%100%98%98%98%92%54%94%
Llama 3.1 70B100%100%100%100%100%92%77%77%77%54%88%
GPT-4.1 Nano100%100%100%98%98%98%98%92%54%27%87%
Gemini 2.5 Flash Lite100%100%100%98%98%92%92%92%77%2%85%
GPT-4.1100%100%100%98%98%92%92%77%54%27%84%
Claude Sonnet 4.5100%100%100%100%100%98%77%77%27%0%78%
Claude 3.7 Sonnet100%98%98%98%98%92%77%54%27%27%77%
AI21 Jamba 1.5 Large100%100%100%100%100%98%92%77%0%0%77%
Llama 3.2 3B100%100%100%100%98%92%92%54%27%0%76%
Sao10K L3.1 70B Hanami x1100%98%98%98%92%92%92%54%0%0%73%
Llama 3 TenyxChat-DaybreakStorywriter 70B100%98%98%92%92%77%77%54%9%2%70%
Claude 3.5 Sonnet100%100%100%98%77%54%54%54%54%9%70%
Llama 3.2 90B (Vision)100%100%100%100%98%98%54%27%9%0%69%
Cohere Command R+ (Apr. 2024)100%100%100%98%92%92%77%9%9%2%68%
Claude 3.5 Sonnet (new)92%92%77%77%77%77%54%54%54%9%66%
Llama 3.2 11B (Vision)100%100%98%98%98%77%54%0%0%0%63%
Hermes 3 405B100%100%100%98%92%92%27%9%2%0%62%
Claude Haiku 4.5100%100%100%92%77%54%54%27%9%0%61%
GPT-4 Turbo100%100%98%92%54%54%54%27%27%0%61%
Z.AI GLM 4.5100%100%98%92%92%27%27%27%9%2%58%
Llama 3.1 8B100%98%98%92%77%54%54%0%0%0%57%
Claude 2.1100%100%92%77%77%54%27%2%0%0%53%
Claude Sonnet 498%92%92%77%54%54%27%9%0%0%50%
Gemini Pro 1.5100%100%92%77%54%9%9%2%0%0%44%
Llama 3.2 1B100%100%100%77%27%27%9%0%0%0%44%
Qwen 2.5 72B100%98%77%77%54%27%2%0%0%0%44%
Mistral Large 298%92%77%54%54%27%0%0%0%0%40%
Phi-3.5 Mini 128k100%100%100%92%0%0%0%0%0%0%39%
Ministral 8B100%92%92%77%9%9%0%0%0%0%38%
Claude 3.0 Sonnet92%92%92%54%27%9%9%0%0%0%38%
Mistral NeMO98%92%92%54%27%2%0%0%0%0%37%
Inflection 3 (Productivity)100%98%98%27%27%9%2%0%0%0%36%
Inflection 3 (PI)100%77%54%54%54%2%0%0%0%0%34%
AI21 Jamba 1.5 Mini100%92%54%27%27%9%9%0%0%0%32%
Gemma 2 9B100%77%77%54%2%0%0%0%0%0%31%
Phi-3 Mini 128k100%98%98%9%0%0%0%0%0%0%31%
AI21 Jamba100%100%54%9%0%0%0%0%0%0%26%
lzlv 70B100%92%9%0%0%0%0%0%0%0%20%
Liquid: LFM 40B MoE100%54%0%0%0%0%0%0%0%0%15%
Gemma 2 27B98%54%0%0%0%0%0%0%0%0%15%
DeepSeek-V2 Chat54%27%27%9%9%9%0%0%0%0%14%
EVA Qwen 2.5 14B98%27%9%0%0%0%0%0%0%0%13%
Goliath 120B100%27%0%0%0%0%0%0%0%0%13%
Lumimaid v0.2 8B100%2%2%0%0%0%0%0%0%0%10%
Gemini Flash 1.5100%2%0%0%0%0%0%0%0%0%10%
Mistral Medium98%2%0%0%0%0%0%0%0%0%10%
Llama 3 Euryale 70B v2.198%0%0%0%0%0%0%0%0%0%10%
Phi-3 Medium 128k77%2%0%0%0%0%0%0%0%0%8%
Claude 2.054%0%0%0%0%0%0%0%0%0%5%
Llama 3.1 Euryale 70B v2.254%0%0%0%0%0%0%0%0%0%5%
MythoMist 7B54%0%0%0%0%0%0%0%0%0%5%
Ministral 3B27%2%2%2%0%0%0%0%0%0%3%
Claude 3.5 Haiku27%2%0%0%0%0%0%0%0%0%3%
Magnum v2 72B27%0%0%0%0%0%0%0%0%0%3%
Mistral Large9%0%0%0%0%0%0%0%0%0%1%
Writer: Palmyra X50%0%0%0%0%0%0%0%0%0%0%
Gemini 2.5 Flash0%0%0%0%0%0%0%0%0%0%0%
Qwen 2 7B0%0%0%0%0%0%0%0%0%0%0%
Rocinante 12B0%0%0%0%0%0%0%0%0%0%0%
Cohere Command R+ (Aug. 2024)0%0%0%0%0%0%0%0%0%0%0%
Hermes 3 70B0%0%0%0%0%0%0%0%0%0%0%
Claude 3 Haiku0%0%0%0%0%0%0%0%0%0%0%
Magnum 72B0%0%0%0%0%0%0%0%0%0%0%
Mistral Small Creative0%0%0%0%0%0%0%0%0%0%0%
MN GRAND Gutenberg Lyra4 12B Madness0%0%0%0%0%0%0%0%0%0%0%
Hermes 2 Theta 8B0%0%0%0%0%0%0%0%0%0%0%
MythoMax 13B0%0%0%0%0%0%0%0%0%0%0%
Fimbulvetr 11B v20%0%0%0%0%0%0%0%0%0%0%
WizardLM 2 8x22b0%0%0%0%0%0%0%0%0%0%0%
Qwen 2 72B0%0%0%0%0%0%0%0%0%0%0%
Toppy M 7B0%0%0%0%0%0%0%0%0%0%0%
Mistral Nemo 12B Celeste0%0%0%0%0%0%0%0%0%0%0%
46.94%