Codex Violation Detection
Detects factual inconsistencies between a story bible and prose passages. The model must output structured XML identifying each violation with paragraph number and substring.
Performance Score Distribution (Top 20)
Click a model name to view its detail page.
Price-Performance Score Distribution (Top 20)
Click a model name to view its detail page.
| Score | Cost | Time | ||
|---|---|---|---|---|
| GPT-5.6 Terra | 98% | $0.011 | 3.8s | |
| GPT-5.4 | 97% | $0.013 | 8.8s | |
| Gemini 2.5 Flash (Reasoning) | 97% | $0.0079 | 12.5s | |
| Z.AI GLM 5 Turbo | 97% | $0.0072 | 17.1s | |
| Z.AI GLM 5.2 (Reasoning, High) | 97% | $0.0100 | 27.9s | |
| Gemini 3 Flash (Preview, Reasoning) | 97% | $0.011 | 18.0s | |
| Gemma 4 31B | 97% | $0.0009 | 37.9s | |
| Gemini 3.5 Flash (Reasoning, Minimal) | 95% | $0.011 | 3.6s | |
| Grok 4.5 (Reasoning, Low) | 99% | $0.013 | 29.6s | |
| GPT-5.6 Sol | 99% | $0.026 | 6.4s | |
| DeepSeek V4 Flash (Reasoning) | 97% | $0.0010 | 40.3s | |
| Xiaomi MIMO v2.5 | 95% | $0.0058 | 21.8s | |
| Qwen 3.6 Flash | 96% | $0.011 | 29.9s | |
| Gemini 2.5 Flash | 91% | $0.0025 | 2.8s | |
| GPT-5.4 (Reasoning, Low) | 96% | $0.020 | 13.5s | |
| Inception Mercury 2 | 92% | $0.0030 | 4.6s | |
| Gemini 3 Flash (Preview) | 94% | $0.0031 | 4.5s | |
| GPT-5.4 Mini (Reasoning, Low) | 93% | $0.0055 | 6.7s | |
| Xiaomi MIMO v2.5 Pro | 96% | $0.0091 | 37.0s | |
| Gemini 2.5 Flash Lite (Reasoning) | 93% | $0.0020 | 17.6s | |
Cost vs Performance
Compares total cost for this test against the test score. Quadrant lines are drawn at the median values. Only models with available cost data are shown.
14 low-scoring outliers hidden: Cydonia 24B V4.1 (64.3%), Mistral Small 4 (62.0%), Ministral 3 3B (61.5%), Cohere Command R+ (Aug. 2024) (61.4%), Hermes 3 70B (60.5%), Ministral 8B (59.6%), Ministral 3B (59.5%), Mistral NeMO (53.6%), Aion 3.0 Mini (47.3%), GPT-5.4 Nano (Reasoning, Low) (42.6%), Gemma 3 4B (39.2%), WizardLM 2 8x22b (36.9%), GPT-5.4 Nano (36.6%), GPT-4.1 Nano (21.9%).
Most Stable Models (Top 20)
Ranked by stability (median × consistency). Click a model name to view its detail page.
| Score | Consistency | Stability | ||
|---|---|---|---|---|
| Gemini 3.1 Pro (Preview) | 99% | 97% | 97% | |
| Claude Opus 4.5 | 99% | 97% | 97% | |
| Claude Opus 4.6 (Reasoning) | 99% | 96% | 96% | |
| Gemini 2.5 Pro | 99% | 95% | 95% | |
| Grok 4.5 (Reasoning, High) | 98% | 95% | 95% | |
| Grok 4.20 (Reasoning) | 98% | 95% | 95% | |
| GPT-5.6 Sol | 99% | 95% | 95% | |
| Qwen3.6 Max Preview | 98% | 95% | 95% | |
| Grok 4.5 (Reasoning, Low) | 99% | 95% | 95% | |
| GPT-5.6 Terra | 98% | 94% | 94% | |
| Gemini 3.5 Flash (Reasoning) | 98% | 94% | 94% | |
| Claude Opus 4.6 | 99% | 94% | 94% | |
| Qwen 3.5 27B | 98% | 94% | 94% | |
| Z.AI GLM 5.1 | 98% | 94% | 94% | |
| GPT-5.5 | 98% | 93% | 93% | |
| GPT-5.5 (Reasoning) | 97% | 93% | 93% | |
| Qwen 3.5 Plus (2026-04-20) | 97% | 93% | 93% | |
| GPT-5.5 (Reasoning, Low) | 97% | 93% | 93% | |
| Gemma 4 31B (Reasoning) | 97% | 93% | 93% | |
| Qwen3.7 Max | 98% | 93% | 93% | |
Top Overall Models (Top 20)
Ranked by composite score (performance, cost, speed & stability). Click a model name to view its detail page.
| Score | Cost | Speed | Stability | ||
|---|---|---|---|---|---|
| GPT-5.6 Terra | 98% | $0.011 | 3.8s | 94% | |
| Gemini 2.5 Flash (Reasoning) | 97% | $0.0079 | 12.5s | 92% | |
| Z.AI GLM 5 Turbo | 97% | $0.0072 | 17.1s | 92% | |
| GPT-5.4 | 97% | $0.013 | 8.8s | 91% | |
| Gemini 3 Flash (Preview, Reasoning) | 97% | $0.011 | 18.0s | 92% | |
| Gemini 3 Flash (Preview) | 94% | $0.0031 | 4.5s | 83% | |
| Gemma 4 31B | 97% | $0.0009 | 37.9s | 90% | |
| Grok 4.5 (Reasoning, Low) | 99% | $0.013 | 29.6s | 95% | |
| GPT-5.6 Sol | 99% | $0.026 | 6.4s | 95% | |
| Inception Mercury 2 | 92% | $0.0030 | 4.6s | 83% | |
| Gemini 3.5 Flash (Reasoning, Minimal) | 95% | $0.011 | 3.6s | 86% | |
| DeepSeek V4 Flash (Reasoning) | 97% | $0.0010 | 40.3s | 90% | |
| Gemini 2.5 Flash | 91% | $0.0025 | 2.8s | 82% | |
| Z.AI GLM 5.2 (Reasoning, High) | 97% | $0.0100 | 27.9s | 90% | |
| Gemini 2.5 Flash Lite (Reasoning) | 93% | $0.0020 | 17.6s | 84% | |
| GPT-5.6 Luna (Reasoning) | 95% | $0.010 | 9.3s | 85% | |
| GPT-5.4 Mini (Reasoning, Low) | 93% | $0.0055 | 6.7s | 82% | |
| GPT-5.5 | 98% | $0.030 | 7.7s | 93% | |
| Xiaomi MIMO v2.5 Pro | 96% | $0.0091 | 37.0s | 89% | |
| Grok 4.20 (Reasoning) | 98% | $0.015 | 45.4s | 95% | |
| matrix | tiers | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Total â–¼ | Small codex (7 entries), short passage (165 words) | Large codex (40 entries), short passage (165 words) | Small codex (7 entries), long passage (734 words) | Large codex (40 entries), long passage (1,019 words) | 5 codex entries | 10 codex entries | 20 codex entries | 40 codex entries |
| Claude Opus 4.5 | 99% | 100% | 100% | 100% | 96% | 100% | 100% | 100% | 98% |
| Gemini 3.1 Pro (Preview) | 99% | 100% | 99% | 99% | 96% | 100% | 100% | 99% | 100% |
| Claude Opus 4.6 (Reasoning) | 99% | 100% | 97% | 98% | 96% | 100% | 100% | 100% | 100% |
| GPT-5.6 Sol | 99% | 100% | 99% | 99% | 97% | 100% | 100% | 94% | 100% |
| Gemini 2.5 Pro | 99% | 99% | 100% | 99% | 96% | 100% | 99% | 97% | 99% |
| Claude Opus 4.6 | 99% | 100% | 98% | 99% | 91% | 100% | 100% | 100% | 100% |
| Grok 4.5 (Reasoning, Low) | 99% | 100% | 98% | 99% | 94% | 100% | 100% | 97% | 100% |
| Grok 4.5 (Reasoning, High) | 98% | 100% | 98% | 98% | 97% | 100% | 98% | 96% | 100% |
| Qwen3.6 Max Preview | 98% | 100% | 97% | 98% | 98% | 100% | 100% | 94% | 100% |
| Qwen3.7 Max | 98% | 100% | 98% | 95% | 98% | 100% | 100% | 95% | 100% |
| Grok 4.20 (Reasoning) | 98% | 100% | 97% | 97% | 98% | 100% | 97% | 96% | 100% |
| Gemini 3.5 Flash (Reasoning) | 98% | 100% | 98% | 98% | 95% | 100% | 99% | 95% | 100% |
| GPT-5.6 Terra | 98% | 100% | 96% | 97% | 94% | 100% | 100% | 96% | 99% |
| Z.AI GLM 5.1 | 98% | 100% | 98% | 96% | 96% | 100% | 99% | 93% | 99% |
| GPT-5.5 | 98% | 100% | 100% | 94% | 95% | 100% | 99% | 92% | 100% |
1–15 of 146
Page 1 / 10
matrix
Small codex (7 entries), short passage (165 words)
Performance Score Distribution (Top 20)
Click a model name to view its detail page.
Price-Performance Score Distribution (Top 20)
Click a model name to view its detail page.
| Score | Cost | Time | ||
|---|---|---|---|---|
| Gemini 3.1 Flash Lite | 100% | $0.0012 | 2.0s | |
| Gemini 3.1 Flash Lite (Preview) | 100% | $0.0012 | 1.9s | |
| Gemini 3.1 Flash Lite (Reasoning) | 90% | $0.0011 | 6.5s | |
| DeepSeek V3.2 | 98% | $0.0006 | 16.2s | |
| DeepSeek V3 (2024-12-26) | 100% | $0.0011 | 14.8s | |
| Mistral Medium 3.1 | 97% | $0.0018 | 5.6s | |
| DeepSeek-V2 Chat | 99% | $0.0011 | 12.7s | |
| Gemini 3 Flash (Preview) | 100% | $0.0025 | 3.6s | |
| DeepSeek V4 Flash | 96% | $0.0002 | 6.6s | |
| GPT-5.4 Nano (Reasoning) | 89% | $0.0022 | 9.4s | |
| GPT-5.6 Luna | 98% | $0.0036 | 3.6s | |
| Xiaomi MIMO v2.5 | 99% | $0.0031 | 11.9s | |
| Gemini 2.5 Flash Lite (Reasoning) | 95% | $0.0012 | 6.9s | |
| Llama 3.1 70B | 97% | $0.0009 | 14.9s | |
| Gemma 4 26B | 97% | $0.0003 | 24.6s | |
| DeepSeek V3 (2025-03-24) | 99% | $0.0007 | 36.3s | |
| Gemini 2.5 Flash | 97% | $0.0017 | 2.1s | |
| Inception Mercury 2 | 97% | $0.0018 | 2.6s | |
| Gemma 3 12B | 95% | $0.0001 | 14.1s | |
| Gemma 4 31B | 97% | $0.0004 | 23.8s | |
Most Stable Models (Top 20)
Ranked by stability (median × consistency). Click a model name to view its detail page.
| Score | Consistency | Stability | ||
|---|---|---|---|---|
| GPT-5.6 Sol (Reasoning) | 100% | 100% | 100% | |
| Qwen3.7 Max | 100% | 100% | 100% | |
| Grok 4.5 (Reasoning, High) | 100% | 100% | 100% | |
| Gemini 3.1 Pro (Preview) | 100% | 100% | 100% | |
| GPT-5.4 (Reasoning) | 100% | 100% | 100% | |
| Z.AI GLM 5.1 | 100% | 100% | 100% | |
| Qwen3.6 Max Preview | 100% | 100% | 100% | |
| GPT-5.5 (Reasoning) | 100% | 100% | 100% | |
| Claude Sonnet 4.6 (Reasoning) | 100% | 100% | 100% | |
| Gemini 3.5 Flash (Reasoning) | 100% | 100% | 100% | |
| Claude Opus 4.7 (Reasoning) | 100% | 100% | 100% | |
| GPT-5.6 Terra (Reasoning) | 100% | 100% | 100% | |
| Claude Opus 4.8 (Reasoning) | 100% | 100% | 100% | |
| Claude Opus 4.6 | 100% | 100% | 100% | |
| Claude Opus 4.8 (Reasoning, Low) | 100% | 100% | 100% | |
| GPT-5 | 100% | 100% | 100% | |
| GPT-5 Mini | 100% | 100% | 100% | |
| Grok 4.5 (Reasoning, Low) | 100% | 100% | 100% | |
| GPT-5.4 (Reasoning, Low) | 100% | 100% | 100% | |
| GPT-5.6 Sol | 100% | 100% | 100% | |
Top Overall Models (Top 20)
Ranked by composite score (performance, cost, speed & stability). Click a model name to view its detail page.
| Score | Cost | Speed | Stability | ||
|---|---|---|---|---|---|
| Gemini 3.1 Flash Lite (Preview) | 100% | $0.0012 | 1.9s | 100% | |
| Gemini 3 Flash (Preview) | 100% | $0.0025 | 3.6s | 100% | |
| Gemini 3.1 Flash Lite | 100% | $0.0012 | 2.0s | 97% | |
| DeepSeek V3 (2024-12-26) | 100% | $0.0011 | 14.8s | 98% | |
| Gemini 2.5 Flash (Reasoning) | 100% | $0.0052 | 7.8s | 100% | |
| Gemini 2.5 Flash | 97% | $0.0017 | 2.1s | 97% | |
| Inception Mercury 2 | 97% | $0.0018 | 2.6s | 97% | |
| GPT-5.4 Mini (Reasoning, Low) | 99% | $0.0041 | 4.4s | 97% | |
| Xiaomi MIMO v2.5 | 99% | $0.0031 | 11.9s | 97% | |
| Z.AI GLM 5 Turbo | 100% | $0.0047 | 10.5s | 98% | |
| DeepSeek-V2 Chat | 99% | $0.0011 | 12.7s | 95% | |
| GPT-5.6 Terra | 100% | $0.0086 | 3.3s | 100% | |
| DeepSeek V4 Flash | 96% | $0.0002 | 6.6s | 93% | |
| GPT-5.6 Luna | 98% | $0.0036 | 3.6s | 93% | |
| Xiaomi MIMO v2.5 Pro | 100% | $0.0051 | 20.3s | 100% | |
| DeepSeek V3.2 | 98% | $0.0006 | 16.2s | 94% | |
| Z.AI GLM 4.5 | 98% | $0.0019 | 15.2s | 95% | |
| Llama 3.1 70B | 97% | $0.0009 | 14.9s | 95% | |
| GPT-4.1 | 100% | $0.0065 | 7.7s | 97% | |
| GPT-4o, Aug. 6th (temp=0) | 99% | $0.0080 | 3.4s | 97% | |
| Median | Evaluator | Top 3 | Flop 3 |
|---|---|---|---|
| 96.5% | Accuracy (recall) | ||
| 100.0% | Precision | ||
| 100.0% | Structural validity |
Large codex (40 entries), short passage (165 words)
Performance Score Distribution (Top 20)
Click a model name to view its detail page.
Price-Performance Score Distribution (Top 20)
Click a model name to view its detail page.
| Score | Cost | Time | ||
|---|---|---|---|---|
| Z.AI GLM 5 Turbo | 98% | $0.0097 | 19.4s | |
| Grok 4.20 | 94% | $0.0081 | 8.6s | |
| Gemini 3 Flash (Preview) | 93% | $0.0039 | 6.2s | |
| DeepSeek V4 Flash | 92% | $0.0005 | 12.9s | |
| Inception Mercury 2 | 92% | $0.0040 | 5.5s | |
| Gemma 4 31B | 94% | $0.0013 | 1.2m | |
| DeepSeek V4 Flash (Reasoning) | 96% | $0.0015 | 58.3s | |
| Gemini 2.5 Flash | 90% | $0.0038 | 4.0s | |
| GPT-5.4 | 98% | $0.018 | 11.4s | |
| Xiaomi MIMO v2.5 | 94% | $0.0071 | 25.0s | |
| GPT-5.4 Mini (Reasoning, Low) | 92% | $0.0070 | 6.8s | |
| Xiaomi MIMO v2.5 Pro | 96% | $0.0093 | 35.3s | |
| Gemini 3.1 Flash Lite (Reasoning) | 90% | $0.0027 | 3.2s | |
| Qwen 3.6 Flash | 97% | $0.012 | 33.9s | |
| GPT-4.1 | 94% | $0.011 | 15.1s | |
| MiniMax M2.5 | 91% | $0.0023 | 30.1s | |
| Gemini 2.5 Flash (Reasoning) | 95% | $0.0098 | 14.4s | |
| Z.AI GLM 5.2 (Reasoning, High) | 97% | $0.012 | 35.5s | |
| Grok 4.5 (Reasoning, Low) | 98% | $0.018 | 20.6s | |
| Gemini 3.1 Flash Lite (Preview) | 90% | $0.0031 | 3.0s | |
Most Stable Models (Top 20)
Ranked by stability (median × consistency). Click a model name to view its detail page.
| Score | Consistency | Stability | ||
|---|---|---|---|---|
| Claude Opus 4.5 | 100% | 100% | 100% | |
| GPT-5.5 | 100% | 100% | 100% | |
| Claude Sonnet 4.5 | 100% | 100% | 100% | |
| GPT-5.6 Sol | 99% | 98% | 98% | |
| Gemini 3.1 Pro (Preview) | 99% | 98% | 98% | |
| Gemini 2.5 Pro | 100% | 97% | 97% | |
| GPT-5.5 (Reasoning, Low) | 99% | 97% | 96% | |
| Gemini 3.5 Flash (Reasoning) | 98% | 97% | 96% | |
| Z.AI GLM 5.1 | 98% | 97% | 96% | |
| GPT-5.5 (Reasoning) | 97% | 98% | 96% | |
| Grok 4.5 (Reasoning, Low) | 98% | 96% | 95% | |
| Qwen3.7 Max | 98% | 97% | 95% | |
| MiniMax M3 | 98% | 97% | 95% | |
| Grok 4.5 (Reasoning, High) | 98% | 98% | 95% | |
| Claude Sonnet 4.6 (Reasoning) | 97% | 97% | 95% | |
| Claude Opus 4.6 | 98% | 97% | 95% | |
| MoonshotAI: Kimi K2.6 | 98% | 97% | 95% | |
| Qwen3.6 Max Preview | 97% | 97% | 94% | |
| DeepSeek V4 Pro (Reasoning) | 97% | 96% | 94% | |
| Claude Sonnet 4.6 | 98% | 96% | 94% | |
Top Overall Models (Top 20)
Ranked by composite score (performance, cost, speed & stability). Click a model name to view its detail page.
| Score | Cost | Speed | Stability | ||
|---|---|---|---|---|---|
| Z.AI GLM 5 Turbo | 98% | $0.0097 | 19.4s | 94% | |
| GPT-5.4 | 98% | $0.018 | 11.4s | 93% | |
| Grok 4.5 (Reasoning, Low) | 98% | $0.018 | 20.6s | 95% | |
| Gemini 3 Flash (Preview) | 93% | $0.0039 | 6.2s | 88% | |
| GPT-5.6 Sol | 99% | $0.033 | 7.7s | 98% | |
| Claude Sonnet 4.5 | 100% | $0.036 | 11.3s | 100% | |
| Inception Mercury 2 | 92% | $0.0040 | 5.5s | 88% | |
| Gemini 3.5 Flash (Reasoning, Minimal) | 94% | $0.016 | 5.0s | 92% | |
| GPT-5.6 Terra | 96% | $0.018 | 5.2s | 91% | |
| Gemini 2.5 Flash (Reasoning) | 95% | $0.0098 | 14.4s | 90% | |
| Grok 4.20 | 94% | $0.0081 | 8.6s | 87% | |
| GPT-5.5 | 100% | $0.042 | 10.0s | 100% | |
| Qwen 3.6 Flash | 97% | $0.012 | 33.9s | 94% | |
| Gemini 3 Flash (Preview, Reasoning) | 96% | $0.012 | 19.4s | 91% | |
| GPT-5.4 Mini (Reasoning, Low) | 92% | $0.0070 | 6.8s | 86% | |
| GPT-5.6 Luna (Reasoning) | 95% | $0.016 | 13.3s | 90% | |
| Z.AI GLM 5.2 (Reasoning, High) | 97% | $0.012 | 35.5s | 92% | |
| DeepSeek V4 Flash | 92% | $0.0005 | 12.9s | 84% | |
| Xiaomi MIMO v2.5 | 94% | $0.0071 | 25.0s | 89% | |
| Gemini 3.1 Flash Lite (Preview) | 90% | $0.0031 | 3.0s | 84% | |
| Median | Evaluator | Top 3 | Flop 3 |
|---|---|---|---|
| 84.2% | Accuracy (recall) | ||
| 94.1% | Precision | ||
| 100.0% | Structural validity |
Small codex (7 entries), long passage (734 words)
Performance Score Distribution (Top 20)
Click a model name to view its detail page.
Price-Performance Score Distribution (Top 20)
Click a model name to view its detail page.
| Score | Cost | Time | ||
|---|---|---|---|---|
| Gemma 4 31B | 97% | $0.0005 | 44.4s | |
| GPT-5.6 Terra | 97% | $0.0097 | 3.6s | |
| Gemini 3.5 Flash (Reasoning, Minimal) | 96% | $0.0087 | 2.9s | |
| Claude Haiku 4.5 | 93% | $0.0051 | 4.6s | |
| GPT-4.1 Mini | 87% | $0.0011 | 4.4s | |
| Gemma 4 26B | 90% | $0.0005 | 15.6s | |
| DeepSeek V4 Flash | 89% | $0.0002 | 8.0s | |
| MiniMax M2.7 | 91% | $0.0021 | 28.0s | |
| Qwen 3.5 Plus (2026-02-15) | 95% | $0.0025 | 21.6s | |
| DeepSeek V4 Flash (Reasoning) | 94% | $0.0010 | 45.3s | |
| Z.AI GLM 4.5 Air | 92% | $0.0026 | 39.2s | |
| DeepSeek-V2 Chat | 85% | $0.0013 | 9.8s | |
| Z.AI GLM 5 Turbo | 94% | $0.0077 | 19.5s | |
| Gemini 3 Flash (Preview) | 88% | $0.0022 | 3.6s | |
| Gemini 2.5 Flash | 87% | $0.0020 | 2.4s | |
| Mistral Medium 3.1 | 79% | $0.0021 | 5.0s | |
| Xiaomi MIMO v2.5 | 93% | $0.0057 | 22.2s | |
| Gemini 2.5 Flash (Reasoning) | 95% | $0.0094 | 16.9s | |
| Inception Mercury 2 | 88% | $0.0032 | 4.9s | |
| Gemini 3 Flash (Preview, Reasoning) | 97% | $0.012 | 20.1s | |