open-mistral-nemo-2407
Mistral NeMO via mistral
Release Date
Jul 18th, 2024Parameters
–Context Size
128kCreative writing
27.72%Rule following
30.48%Utility
67.56%Mathematics
95.00%Tooling
68.21%Language
76.89%Logic
78.75%Data extraction
Extract key details from a given block of text.
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 0% | 0% | 80% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 50% | 95% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 50% | 50% | 50% | 50% | 50% | 50% | 50% | 50% | 50% | 55% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 85.83% | |||||||||||
Dialogue tags
Various tasks related to dialogue tags in text.
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 100% | 61% | 61% | 61% | 61% | 14% | 76% | |
| 69% | 50% | 50% | 34% | 0% | 0% | 0% | 0% | 0% | 0% | 20% | |
| 53% | 50% | 40% | 19% | 1% | 0% | 0% | 0% | 0% | 0% | 16% | |
| 50% | 50% | 50% | 50% | 50% | 50% | 50% | 49% | 47% | 46% | 49% | |
| 44% | 26% | 6% | 2% | 0% | 0% | 0% | 0% | 0% | 0% | 8% | |
| 32% | 31% | 22% | 20% | 2% | 1% | 0% | 0% | 0% | 0% | 11% | |
| 50% | 43% | 41% | 5% | 0% | 0% | 0% | 0% | 0% | 0% | 14% | |
| 27.72% | |||||||||||
Language Comprehension
Does the model understand more than just English?
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Total |
|---|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 0% | 80% | |
| 95.00% | ||||||
Language Writing
Can the model generate text in different languages?
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Total |
|---|---|---|---|---|---|---|
| 100% | 92% | 86% | 40% | 0% | 63% | |
| 100% | 83% | 67% | 0% | 0% | 50% | |
| 100% | 91% | 33% | 0% | 0% | 45% | |
| 100% | 91% | 90% | 88% | 0% | 74% | |
| 100% | 100% | 100% | 100% | 0% | 80% | |
| 62.40% | ||||||
Novel outline
Handle questions about the outline of a novel in various formats
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 0% | 90% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 0% | 0% | 0% | 0% | 0% | 0% | 40% | |
| 100% | 100% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 20% | |
| 100% | 100% | 100% | 100% | 0% | 0% | 0% | 0% | 0% | 0% | 40% | |
| 100% | 100% | 100% | 100% | 0% | 0% | 0% | 0% | 0% | 0% | 40% | |
| 100% | 100% | 50% | 50% | 0% | 0% | 0% | 0% | 0% | 0% | 30% | |
| 50% | 50% | 50% | 50% | 50% | 50% | 0% | 0% | 0% | 0% | 30% | |
| 65.83% | |||||||||||
Tool usage within Novelcrafter
Output messages that are related to tool usage within Novelcrafter
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 67% | 97% | |
| 96.67% | |||||||||||
N-Length Sentences
Write sentences with exactly N words
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 72% | 47% | 45% | 37% | 35% | 22% | 21% | 17% | 1% | 1% | 30% | |
| 57% | 48% | 45% | 24% | 24% | 15% | 14% | 11% | 10% | 9% | 26% | |
| 2% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 18.57% | |||||||||||
Voice/dialogue sheets
Extract dialogue from given text as voice sheets.
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 100% | 100% | 100% | 100% | 0% | 0% | 0% | 0% | 0% | 0% | 40% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 0% | 0% | 0% | 0% | 60% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 100% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 10% | |
| 22.00% | |||||||||||
Write N of X
Write exactly N words/sentences/paragraphs...
| Scenario | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 92% | 92% | 77% | 77% | 27% | 9% | 0% | 0% | 0% | 48% | |
| 92% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 9% | |
| 98% | 92% | 92% | 54% | 27% | 2% | 0% | 0% | 0% | 0% | 37% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 77% | 77% | 54% | 54% | 54% | 27% | 9% | 9% | 9% | 9% | 38% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | |
| 100% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 10% | |
| 100% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 10% | |
| 34.71% | |||||||||||