Rule Following
12 scenarios across 1 subcategory. 118 models scored.
Subcategories
| Subcategory | Avg Score | Best Model | Best Score |
|---|---|---|---|
| Constraint Adherence | 59.13% | Gemini 3.1 Pro (Preview) | 91.21% |
Model Leaderboard
All models ranked by their Rule Following category score.
| # | Model | Rule Following | Constraint Adherence | Overall |
|---|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 91.21% | 91.21% | 94.37% |
| 2 | Claude Opus 4.6 (Reasoning) | 89.78% | 89.78% | 95.02% |
| 3 | Z.AI GLM 5 Turbo | 86.78% | 86.78% | 94.27% |
| 4 | Claude Sonnet 4.6 (Reasoning) | 85.73% | 85.73% | 93.66% |
| 5 | Claude Opus 4.6 | 83.11% | 83.11% | 92.35% |
| 6 | Claude Sonnet 4.6 | 82.50% | 82.50% | 91.15% |
| 7 | Claude Sonnet 4 | 81.52% | 81.52% | 88.72% |
| 8 | Qwen 3.5 122B | 80.00% | 80.00% | 91.53% |
| 9 | Qwen 3.5 397B A17B | 79.39% | 79.39% | 91.73% |
| 10 | GPT-5.4 (Reasoning) | 79.29% | 79.29% | 93.24% |
| 11 | ByteDance Seed 1.6 | 77.71% | 77.71% | 90.70% |
| 12 | GPT-5 | 77.13% | 77.13% | 91.93% |
| 13 | Claude Sonnet 4.5 | 76.80% | 76.80% | 88.03% |
| 14 | GPT-5 Mini | 76.44% | 76.44% | 92.62% |
| 15 | Qwen 3.5 27B | 76.04% | 76.04% | 90.85% |
| 16 | Grok 4.20 (Beta, Reasoning) | 75.31% | 75.31% | 91.49% |
| 17 | Gemini 3 Flash (Preview, Reasoning) | 74.48% | 74.48% | 90.50% |
| 18 | GPT-4o, Aug. 6th (temp=0) | 74.19% | 74.19% | 82.45% |
| 19 | GPT-5.1 | 74.05% | 74.05% | 92.54% |
| 20 | Claude 3.7 Sonnet | 73.78% | 73.78% | 83.39% |
| 21 | GPT-4o, May 13th (temp=0) | 73.24% | 73.24% | 85.36% |
| 22 | o4 Mini High | 72.70% | 72.70% | 90.29% |
| 23 | Claude Opus 4.5 | 72.61% | 72.61% | 89.69% |
| 24 | MoonshotAI: Kimi K2.5 | 72.03% | 72.03% | 91.04% |
| 25 | Grok 4.1 Fast | 70.87% | 70.87% | 89.55% |
| 26 | Claude Opus 4 | 70.37% | 70.37% | 87.69% |
| 27 | Claude Haiku 4.5 | 70.35% | 70.35% | 85.14% |
| 28 | GPT-5.4 (Reasoning, Low) | 70.02% | 70.02% | 91.41% |
| 29 | GPT-4o, May 13th (temp=1) | 69.88% | 69.88% | 83.80% |
| 30 | Claude 3.5 Sonnet | 69.67% | 69.67% | 84.24% |
| 31 | Z.AI GLM 4.7 | 69.16% | 69.16% | 88.69% |
| 32 | MiniMax M2.7 | 68.90% | 68.90% | 89.10% |
| 33 | DeepSeek-V2 Chat | 68.78% | 68.78% | 84.83% |
| 34 | DeepSeek V3 (2025-03-24) | 67.94% | 67.94% | 81.99% |
| 35 | Grok 4 Fast | 67.91% | 67.91% | 86.15% |
| 36 | GPT-4o, Aug. 6th (temp=1) | 67.91% | 67.91% | 82.62% |
| 37 | Z.AI GLM 5 | 67.78% | 67.78% | 91.23% |
| 38 | Qwen 3.5 35B | 67.42% | 67.42% | 88.00% |
| 39 | Writer: Palmyra X5 | 67.19% | 67.19% | 79.57% |
| 40 | GPT-5.2 | 67.10% | 67.10% | 90.26% |
| 41 | Gemini 2.5 Flash Lite (Reasoning) | 66.81% | 66.81% | 85.75% |
| 42 | GPT-4.1 | 66.78% | 66.78% | 88.68% |
| 43 | DeepSeek V3 (2024-12-26) | 66.39% | 66.39% | 83.68% |
| 44 | DeepSeek V3.1 | 66.15% | 66.15% | 82.39% |
| 45 | Z.AI GLM 4.6 | 65.85% | 65.85% | 89.11% |
| 46 | Z.AI GLM 4.7 Flash | 65.63% | 65.63% | 84.82% |
| 47 | Qwen3 235B A22B Instruct 2507 | 65.42% | 65.42% | 80.10% |
| 48 | Gemini 3 Flash (Preview) | 65.14% | 65.14% | 85.35% |
| 49 | o4 Mini | 64.61% | 64.61% | 88.35% |
| 50 | Gemini 3 Pro (Preview) | 64.47% | 64.47% | 88.79% |
| 51 | Mistral Large 3 | 64.41% | 64.41% | 85.43% |
| 52 | Qwen 3.5 Plus (2026-02-15) | 64.21% | 64.21% | 85.96% |
| 53 | Claude 3.5 Haiku | 64.18% | 64.18% | 83.73% |
| 54 | Mistral Small 3.2 24B | 64.08% | 64.08% | 78.60% |
| 55 | Z.AI GLM 4.5 | 63.79% | 63.79% | 86.27% |
| 56 | Aion 2.0 | 63.77% | 63.77% | 89.21% |
| 57 | Stealth: Hunter Alpha | 63.63% | 63.63% | 87.34% |
| 58 | Llama 3.1 70B | 63.45% | 63.45% | 78.40% |
| 59 | Qwen 3.5 Flash | 63.19% | 63.19% | 86.38% |
| 60 | Grok 4 | 63.09% | 63.09% | 88.12% |
| 61 | Mistral Large 2 | 63.05% | 63.05% | 82.41% |
| 62 | MiniMax M2.5 | 62.69% | 62.69% | 88.71% |
| 63 | Mistral Small 4 | 62.17% | 62.17% | 76.46% |
| 64 | Gemma 3 12B | 61.05% | 61.05% | 78.41% |
| 65 | Qwen 3.5 9B | 60.98% | 60.98% | 86.05% |
| 66 | Gemini 2.5 Pro | 60.89% | 60.89% | 88.53% |
| 67 | Mistral Small 4 (Reasoning) | 60.28% | 60.28% | 82.39% |
| 68 | Gemini 2.5 Flash (Reasoning) | 59.97% | 59.97% | 86.51% |
| 69 | Gemini 2.5 Flash Lite | 59.96% | 59.96% | 81.08% |
| 70 | Hermes 3 405B | 59.17% | 59.17% | 82.86% |
| 71 | Gemini 3.1 Flash Lite (Preview) | 59.04% | 59.04% | 85.87% |
| 72 | GPT-4o Mini (temp=0) | 58.84% | 58.84% | 78.29% |
| 73 | ByteDance Seed 2.0 Mini | 58.77% | 58.77% | 86.91% |
| 74 | Cohere Command R+ (Aug. 2024) | 58.70% | 58.70% | 69.03% |
| 75 | GPT-4.1 Mini | 58.59% | 58.59% | 83.20% |
| 76 | GPT-5.4 | 58.11% | 58.11% | 84.32% |
| 77 | GPT-5 Nano | 57.57% | 57.57% | 82.60% |
| 78 | Gemini 2.5 Flash | 57.47% | 57.47% | 80.60% |
| 79 | Nemotron 3 Super | 57.43% | 57.43% | 84.56% |
| 80 | GPT-5.4 Mini (Reasoning) | 57.38% | 57.38% | 90.65% |
| 81 | GPT-4o Mini (temp=1) | 56.50% | 56.50% | 79.08% |
| 82 | Stealth: Healer Alpha | 56.03% | 56.03% | 85.93% |
| 83 | Inception Mercury 2 | 54.41% | 54.41% | 83.85% |
| 84 | Grok 4.20 (Beta) | 53.89% | 53.89% | 83.85% |
| 85 | DeepSeek V3.2 | 53.75% | 53.75% | 82.25% |
| 86 | Hermes 3 70B | 53.00% | 53.00% | 72.57% |
| 87 | Claude 3 Haiku | 51.15% | 51.15% | 71.19% |
| 88 | Ministral 3 14B | 50.83% | 50.83% | 72.54% |
| 89 | Llama 3.1 Nemotron 70B | 50.62% | 50.62% | 74.70% |
| 90 | Mistral Large | 49.87% | 49.87% | 80.15% |
| 91 | Mistral Medium 3.1 | 48.60% | 48.60% | 77.83% |
| 92 | Mistral Small Creative | 48.15% | 48.15% | 73.27% |
| 93 | Gemma 3 27B | 47.98% | 47.98% | 77.85% |
| 94 | ByteDance Seed 1.6 Flash | 47.15% | 47.15% | 73.27% |
| 95 | Qwen 3 32B | 46.83% | 46.83% | 82.21% |
| 96 | GPT-5.4 Mini | 46.32% | 46.32% | 82.43% |
| 97 | Stealth: Aurora Alpha | 44.19% | 44.19% | 83.79% |
| 98 | Nemotron 3 Nano | 43.47% | 43.47% | 77.73% |
| 99 | Rocinante 12B | 41.51% | 41.51% | 54.55% |
| 100 | GPT-4.1 Nano | 40.88% | 40.88% | 71.94% |
| 101 | Inception Mercury | 39.68% | 39.68% | 79.50% |
| 102 | Arcee AI: Trinity Large (Preview) | 38.52% | 38.52% | 73.33% |
| 103 | ByteDance Seed 2.0 Lite | 36.85% | 36.85% | 84.80% |
| 104 | Mistral NeMO | 34.11% | 34.11% | 65.04% |
| 105 | Llama 3.1 8B | 34.03% | 34.03% | 63.37% |
| 106 | GPT-5.4 Mini (Reasoning, Low) | 33.99% | 33.99% | 85.75% |
| 107 | GPT-5.4 Nano (Reasoning, Low) | 31.65% | 31.65% | 79.48% |
| 108 | Qwen 2.5 72B | 31.55% | 31.55% | 75.46% |
| 109 | Ministral 3 8B | 31.34% | 31.34% | 71.76% |
| 110 | WizardLM 2 8x22b | 28.27% | 28.27% | 71.07% |
| 111 | GPT-5.4 Nano (Reasoning) | 27.15% | 27.15% | 81.36% |
| 112 | Gemma 3 4B | 26.37% | 26.37% | 68.57% |
| 113 | Ministral 3B | 24.45% | 24.45% | 61.29% |
| 114 | LFM2 24B | 24.12% | 24.12% | 58.77% |
| 115 | Arcee AI: Trinity Mini | 23.57% | 23.57% | 70.90% |
| 116 | GPT-5.4 Nano | 20.94% | 20.94% | 74.40% |
| 117 | Ministral 3 3B | 15.87% | 15.87% | 67.22% |
| 118 | Ministral 8B | 15.27% | 15.27% | 64.87% |