NC Bench
Overview
About
Tests
Models
Vendors
Compare
Run: 2026-02-21-01KJ0D2QQDS76DWKHPZVDGVMEJ
Run Details
Model:
Gemma 3 12B
Test:
Codex Extraction
Scenario:
Short: The Rusty Lantern (Explicit)
Duration:
11.9s
Cost:
$0.000190
Provider:
DeepInfra
Evaluation summary
Score
Evaluator
Details Link
70.79%
Accuracy
(details)
55.56%
Precision
(details)
54.11%
Recall
(details)
95.83%
Structural validity
(details)
69.0727%
Overall
Model input
System
You are a meticulous story bible editor for a fiction project. Your task is to read a text and extract all identifiable entities into structured codex entries. Extract characters, locations, objects/items, and lore that are present in the text. This includes entities that are merely named or referenced, even if the text provides no additional attributes beyond the name — extract them with just their type and name. Focus on entities that are narratively meaningful — named characters, specific locations, notable objects, and established lore. Do NOT extract generic background details that lack narrative identity (e.g. "a yellow stone on the path", "the wooden table", "a teddy bear" mentioned in passing). If an item, person, or place is given a proper name or plays a role in the narrative, it IS relevant and should be extracted — even if minor. Rules: 1. Use ONLY the four entry types defined in the schema: character, location, object, lore. 2. Use ONLY the fields defined for each entry type. Do not invent new fields. 3. For dropdown fields, use ONLY the listed values. If no listed value fits, omit the field. 4. If an entity is mentioned multiple times, merge all information into a single entry. 5. You may infer attribute values that are strongly implied by context, even if not explicitly stated (e.g. inferring a time period from surrounding dialogue). 6. Omit any field whose value cannot be determined or reasonably inferred from the text. 7. Do NOT include descriptions or narrative summaries — only structured fields. 8. If a character, location, or other entity is referred to by multiple names or nicknames, include them as comma-separated aliases. 9. Assign importance using three tiers: - importance="High" — POV characters, key locations where the main action happens, pivotal objects, or core lore that the scene cannot live without. - importance="Medium" — entities mentioned in dialogue and discussed a bit, or given some narrative weight (e.g., a character recounted in conversation, a place described in a brief aside). They matter to the story but are not central to the scene. - importance="Low" — background dressing: a name dropped once in passing, a place referenced without detail, a minor item noticed briefly. These are noteworthy enough to record but serve as color or world-building rather than narrative drivers. 10. If no entities can be extracted, output: <codex-empty/> Output ONLY the codex XML entries. Do not include any other text, preamble, or explanation outside the XML. { "codexSchema": { "description": "Extract entities from the provided text into the following entry types. Output as XML.", "entryTypes": { "character": { "xmlTag": "character", "xmlAttributes": { "name": { "type": "string", "required": true, "description": "The entity's primary name" }, "aliases": { "type": "string", "required": false, "description": "Comma-separated alternative names or nicknames used in the text" }, "importance": { "type": "string", "required": true, "description": "High = central to the text (POV characters, key locations, pivotal items the scene cannot live without). Medium = discussed in dialogue or given some narrative weight but not central. Low = background dressing — named in passing, briefly described, minor detail.", "enum": [ "High", "Medium", "Low" ] } }, "fields": { "species": { "type": "string", "enum": [ "Human", "Elf", "Dwarf", "Orc", "Halfling", "Gnome", "Dragon", "Demon", "Angel", "Undead", "Vampire", "Werewolf", "Fae", "Half-Elf", "Goblin", "Troll", "Giant", "Merfolk", "Centaur", "Tiefling" ] }, "gender": { "type": "string", "enum": [ "Male", "Female", "Non-binary" ] }, "age": { "type": "string" }, "eyeColor": { "type": "string", "enum": [ "Blue", "Green", "Brown", "Hazel", "Gray", "Amber", "Black", "Violet", "Red", "Golden", "Silver" ] }, "hairColor": { "type": "string", "enum": [ "Black", "Brown", "Blonde", "Red", "Auburn", "Gray", "White", "Silver", "Blue", "Green" ] } } }, "location": { "xmlTag": "location", "xmlAttributes": { "name": { "type": "string", "required": true, "description": "The entity's primary name" }, "aliases": { "type": "string", "required": false, "description": "Comma-separated alternative names or nicknames used in the text" }, "importance": { "type": "string", "required": true, "description": "High = central to the text (POV characters, key locations, pivotal items the scene cannot live without). Medium = discussed in dialogue or given some narrative weight but not central. Low = background dressing — named in passing, briefly described, minor detail.", "enum": [ "High", "Medium", "Low" ] } }, "fields": { "country": { "type": "string" }, "climate": { "type": "string", "enum": [ "Tropical", "Arid", "Temperate", "Continental", "Polar", "Humid", "Mediterranean", "Desert", "Oceanic", "Subarctic" ] }, "terrain": { "type": "string", "enum": [ "Forest", "Mountain", "Desert", "Plains", "Swamp", "Coastal", "Volcanic", "Tundra", "Underground", "Urban", "Island", "Jungle" ] } } }, "object": { "xmlTag": "object", "xmlAttributes": { "name": { "type": "string", "required": true, "description": "The entity's primary name" }, "aliases": { "type": "string", "required": false, "description": "Comma-separated alternative names or nicknames used in the text" }, "importance": { "type": "string", "required": true, "description": "High = central to the text (POV characters, key locations, pivotal items the scene cannot live without). Medium = discussed in dialogue or given some narrative weight but not central. Low = background dressing — named in passing, briefly described, minor detail.", "enum": [ "High", "Medium", "Low" ] } }, "fields": { "material": { "type": "string" }, "objectType": { "type": "string", "enum": [ "Weapon", "Armor", "Artifact", "Tool", "Jewelry", "Potion", "Book", "Key", "Instrument", "Clothing" ] }, "power": { "type": "string", "enum": [ "Fire", "Water", "Earth", "Air", "Lightning", "Ice", "Dark", "Light", "Arcane", "Nature", "Void", "Poison", "Healing", "Time", "Psychic" ] } } }, "lore": { "xmlTag": "lore", "xmlAttributes": { "name": { "type": "string", "required": true, "description": "The entity's primary name" }, "aliases": { "type": "string", "required": false, "description": "Comma-separated alternative names or nicknames used in the text" }, "importance": { "type": "string", "required": true, "description": "High = central to the text (POV characters, key locations, pivotal items the scene cannot live without). Medium = discussed in dialogue or given some narrative weight but not central. Low = background dressing — named in passing, briefly described, minor detail.", "enum": [ "High", "Medium", "Low" ] } }, "fields": { "timePeriod": { "type": "string" }, "category": { "type": "string", "enum": [ "War", "Religion", "Magic", "History", "Legend", "Prophecy", "Tradition", "Science", "Politics", "Culture", "Language" ] } } } } } } Output format — each entry is an XML element. name, aliases, and importance are XML attributes. All fields are child elements. Omit fields that cannot be determined from the text. Example: <character name="Adris" aliases="The Wanderer" importance="High"> <species>Half-Elf</species> <eyeColor>Green</eyeColor> </character> <location name="Thornwood" importance="Medium"> <terrain>Forest</terrain> </location> <lore name="The Grey Tide" importance="Low"> <category>History</category> </lore>
User/Human
<text> The Rusty Lantern sat crooked on its foundations, listing toward the harbor like a drunk leaning on a post. Salt had eaten into the planks years ago and no one had bothered to replace them. The ceiling beams were black with smoke. Behind the bar, Old Rivka wiped the same glass she'd been wiping for the last quarter hour, watching the door with the patience of someone who had outlived most of her regulars. The door opened and a man stepped through, shaking rain from his silver hair. Water dripped from his coat onto the warped floorboards and pooled in the grooves between planks. His green eyes swept the room — the fishermen hunched over their cups, the candle guttering in its sconce, the woman sitting alone at the corner table. He smelled of wet wool and the cold air off the harbor. Old Rivka raised her chin. "Stray," she called across the room. "Didn't think you'd show." He didn't answer. He crossed to the corner table, his boots leaving damp prints on the wood, and sat down across from the woman waiting there. Sable Dunmore was twenty-eight years old, human, and tired of waiting. Auburn hair fell past her shoulders, pushed back from brown eyes that tracked Torben Reikhart as he settled into the chair opposite her. On the table between them lay a sword — Kindling, she called it. A dark iron blade, the metal faintly warm even in the chill of the tavern. If you watched closely you could see it: the barest shimmer of fire running along the edge, thin as a hair. At her neck hung a silver locket. Petal's Locket, though she never called it that aloud. "It still has her hair inside," she said, touching it once. "That's all I have left of her." Torben waited. He'd learned when to be quiet. "My sister Petal," Sable said. "She was fourteen when The Felling reached Dunmore. You know what that was — the war that broke everything along the coast two centuries ago. Jurren — our father — he told me to take Petal and run. I did. She didn't make it." Torben didn't ask the obvious question. Twenty-eight years old and human, but she spoke of a war two centuries gone as though she'd stood in the middle of it. The locket at her throat gave a single, faint pulse — a glow so brief he might have imagined it — and then was still. Sable's hand moved to the locket again. "We couldn't give her a Kindling Rite. You know the tradition? You burn the dead with what they loved. Their tools. Their letters. You give it all to the fire so they carry it onward. But there was no body to burn. Nothing left of her but this locket and whatever I can remember." She looked at Torben. "Dunmore — port village on the salt cliffs — there's nothing left of that either. Just rock and old foundations." Torben said, "Then we'd better make it count." Sable picked up Kindling and sheathed it. They left the Rusty Lantern without another word. The rain had stopped. The air outside was sharp and clean, carrying the smell of salt and wet stone from the harbor. Somewhere down the waterfront a shutter banged in the wind. Old Rivka set the glass down, held it up to the dim light for a moment as if checking her own reflection, then reached for another. </text> Extract all codex entries from the text above.
Model response
Messages
Meta
AI
```xml <character name="Old Rivka" importance="Medium"> <age>Unknown</age> </character> <location name="Rusty Lantern" importance="High"> </location> <location name="harbor" importance="Medium"> </location> <object name="glass" importance="Low"> </object> <character name="Stray" importance="Medium"> <hairColor>Silver</hairColor> <eyeColor>Green</eyeColor> </character> <location name="corner table" importance="Medium"> </location> <character name="Sable Dunmore" importance="High"> <age>28</age> <gender>Female</gender> <hairColor>Auburn</hairColor> <eyeColor>Brown</eyeColor> <species>Human</species> </character> <object name="Kindling" importance="High"> <objectType>Weapon</objectType> <power>Fire</power> </object> <object name="silver locket" importance="High"> <aliases>Petal's Locket</aliases> </object> <lore name="The Felling" importance="Medium"> <category>War</category> <timePeriod>Two centuries ago</timePeriod> </lore> <character name="Jurren" importance="Medium"> </character> <character name="Petal" importance="Medium"> </character> <character name="Torben Reikhart" importance="High"> </character> <location name="Dunmore" importance="Medium"> <terrain>Port village</terrain> </location> <lore name="Kindling Rite" importance="Medium"> <category>Tradition</category> </lore> <location name="salt cliffs" importance="Low"> </location> <character name="Rivka" importance="Low"> </character> <location name="waterfront" importance="Low"> </location> ```
created_at
"2026-02-21T15:28:50.935Z"
2/21/2026, 3:28:50 PM
model
"google/gemma-3-12b-it"
app_id
182717
external_user
(null)
streamed
true
cancelled
false
latency
330
moderation_latency
(null)
generation_time
11843
tokens_prompt
3777
tokens_completion
415
native_tokens_prompt
3261
native_tokens_completion
459
native_tokens_completion_images
(null)
native_tokens_reasoning
0
native_tokens_cached
0
num_media_prompt
(null)
num_input_audio_prompt
(null)
num_media_completion
0
num_search_results
(null)
origin
"https://ncbench.com/"
is_byok
false
finish_reason
"stop"
native_finish_reason
"stop"
usage
0.00019011
router
(null)
provider_responses
0
endpoint_id
"eb06dc92-5a16-47ec-a776-6ef956457c47"
id
"chatcmpl-RlLl4excxHixcbtPThiIcQ2v"
is_byok
false
latency
45
model_permaslug
"google/gemma-3-12b-it"
provider_name
"DeepInfra"
status
200
api_type
"completions"
id
"gen-1771687730-zSxTT7Sc7x1WT3gfBpTZ"
upstream_id
"chatcmpl-RlLl4excxHixcbtPThiIcQ2v"
total_cost
0.00019011
cache_discount
(null)
upstream_inference_cost
0
provider_name
"DeepInfra"
Evaluation details
Result
Evaluator
Details
Meta Data
70.79%
Accuracy
Eligible: 10, Weighted avg: 0.71
eligibleCount
10
matchedCount
10
weightedAvg
0.7078947368421052
entryScores
0
name
"Torben Reikhart"
importance
"high"
score
0.5
details
type
expected
0
"character"
actual
"character"
correct
true
score
1
attributes
gender
expected
"Male"
actual
(null)
source
"explicit"
score
0
reason
"missing"
eyeColor
expected
"Green"
actual
"Green"
source
"explicit"
score
1
method
"dropdown"
hairColor
expected
"Silver"
actual
"Silver"
source
"explicit"
score
1
method
"dropdown"
aliases
score
0
matched
(empty)
missed
0
"stray"
extra
(empty)
importance
expected
"high"
alternatives
(empty)
actual
"Medium"
correct
false
score
0
rawScore
0.5
dimensions
6
entryScore
0.5
1
name
"Sable Dunmore"
importance
"high"
score
1
details
type
expected
0
"character"
actual
"character"
correct
true
score
1
attributes
species
expected
"Human"
actual
"Human"
source
"explicit"
score
1
method
"dropdown"
gender
expected
"Female"
actual
"Female"
source
"explicit"
score
1
method
"dropdown"
age
expected
"28"
actual
"28"
source
"explicit"
score
1
method
"text"
eyeColor
expected
"Brown"
actual
"Brown"
source
"explicit"
score
1
method
"dropdown"
hairColor
expected
"Auburn"
actual
"Auburn"
source
"explicit"
score
1
method
"dropdown"
importance
expected
"high"
alternatives
(empty)
actual
"High"
correct
true
score
1
rawScore
1
dimensions
7
entryScore
1
2
name
"Old Rivka"
importance
"low"
score
0.5
details
type
expected
0
"character"
actual
"character"
correct
true
score
1
attributes
gender
expected
"Female"
actual
(null)
source
"inferred"
score
0
reason
"missing"
species
expected
"Human"
actual
(null)
source
"inferred"
score
0
reason
"missing"
importance
expected
"low"
alternatives
0
"medium"
actual
"Medium"
correct
true
score
1
rawScore
0.5
dimensions
4
entryScore
0.5
3
name
"Petal Dunmore"
importance
"medium"
score
0.4
details
type
expected
0
"character"
actual
"character"
correct
true
score
1
attributes
age
expected
"14"
actual
(null)
source
"explicit"
score
0
reason
"missing"
gender
expected
"Female"
actual
(null)
source
"explicit"
score
0
reason
"missing"
species
expected
"Human"
actual
(null)
source
"inferred"
score
0
reason
"missing"
importance
expected
"medium"
alternatives
0
"high"
actual
"Medium"
correct
true
score
1
rawScore
0.4
dimensions
5
entryScore
0.4
4
name
"Jurren Dunmore"
importance
"low"
score
0.25
details
type
expected
0
"character"
actual
"character"
correct
true
score
1
attributes
gender
expected
"Male"
actual
(null)
source
"explicit"
score
0
reason
"missing"
species
expected
"Human"
actual
(null)
source
"inferred"
score
0
reason
"missing"
importance
expected
"low"
alternatives
(empty)
actual
"Medium"
correct
false
score
0
rawScore
0.25
dimensions
4
entryScore
0.25
5
name
"The Rusty Lantern"
importance
"low"
score
0.3333333333333333
details
type
expected
0
"location"
actual
"location"
correct
true
score
1
attributes
terrain
expected
"Urban"
actual
(null)
source
"inferred"
score
0
reason
"missing"
importance
expected
"low"
alternatives
0
"medium"
actual
"High"
correct
false
score
0
rawScore
0.3333333333333333
dimensions
3
entryScore
0.3333333333333333
6
name
"Dunmore"
importance
"low"
score
0.6666666666666666
details
type
expected
0
"location"
actual
"location"
correct
true
score
1
attributes
terrain
expected
"Coastal"
actual
"Port village"
source
"explicit"
score
0
method
"dropdown"
importance
expected
"low"
alternatives
0
"medium"
actual
"Medium"
correct
true
score
1
rawScore
0.6666666666666666
dimensions
3
entryScore
0.6666666666666666
7
name
"Kindling"
importance
"high"
score
0.8
details
type
expected
0
"object"
actual
"object"
correct
true
score
1
attributes
material
expected
"Dark iron"
actual
(null)
source
"explicit"
score
0
reason
"missing"
objectType
expected
"Weapon"
actual
"Weapon"
source
"explicit"
score
1
method
"dropdown"
power
expected
"Fire"
actual
"Fire"
source
"explicit"
score
1
method
"dropdown"
importance
expected
"high"
alternatives
0
"medium"
actual
"High"
correct
true
score
1
rawScore
0.8
dimensions
5
entryScore
0.8
8
name
"The Felling"
importance
"medium"
score
1
details
type
expected
0
"lore"
actual
"lore"
correct
true
score
1
attributes
timePeriod
expected
"Two centuries ago"
actual
"Two centuries ago"
source
"explicit"
score
1
method
"text"
category
expected
"War"
actual
"War"
source
"explicit"
score
1
method
"dropdown"
importance
expected
"medium"
alternatives
(empty)
actual
"Medium"
correct
true
score
1
rawScore
1
dimensions
4
entryScore
1
9
name
"The Kindling Rite"
importance
"medium"
score
1
details
type
expected
0
"lore"
actual
"lore"
correct
true
score
1
attributes
category
expected
"Tradition"
actual
"Tradition"
source
"explicit"
score
1
method
"dropdown"
importance
expected
"medium"
alternatives
0
"low"
actual
"Medium"
correct
true
score
1
rawScore
1
dimensions
3
entryScore
1
explicitAccuracy
correct
12
total
18
rate
0.6666666666666666
inferredAccuracy
correct
0
total
5
rate
0
55.56%
Precision
True positives: 10/18
totalReported
18
duplicateCount
0
duplicates
(empty)
truePositives
10
reportedDetails
0
parsed
type
"character"
name
"Old Rivka"
status
"true_positive"
matchedExpected
name
"Old Rivka"
types
0
"character"
1
parsed
type
"character"
name
"Stray"
status
"true_positive"
matchedExpected
name
"Torben Reikhart"
types
0
"character"
2
parsed
type
"character"
name
"Sable Dunmore"
status
"true_positive"
matchedExpected
name
"Sable Dunmore"
types
0
"character"
3
parsed
type
"character"
name
"Jurren"
status
"true_positive"
matchedExpected
name
"Jurren Dunmore"
types
0
"character"
4
parsed
type
"character"
name
"Petal"
status
"true_positive"
matchedExpected
name
"Petal Dunmore"
types
0
"character"
5
parsed
type
"character"
name
"Torben Reikhart"
status
"false_positive"
6
parsed
type
"character"
name
"Rivka"
status
"false_positive"
7
parsed
type
"location"
name
"Rusty Lantern"
status
"true_positive"
matchedExpected
name
"The Rusty Lantern"
types
0
"location"
8
parsed
type
"location"
name
"harbor"
status
"false_positive"
9
parsed
type
"location"
name
"corner table"
status
"false_positive"
10
parsed
type
"location"
name
"Dunmore"
status
"true_positive"
matchedExpected
name
"Dunmore"
types
0
"location"
11
parsed
type
"location"
name
"salt cliffs"
status
"false_positive"
12
parsed
type
"location"
name
"waterfront"
status
"false_positive"
13
parsed
type
"object"
name
"glass"
status
"false_positive"
14
parsed
type
"object"
name
"Kindling"
status
"true_positive"
matchedExpected
name
"Kindling"
types
0
"object"
15
parsed
type
"object"
name
"silver locket"
status
"false_positive"
16
parsed
type
"lore"
name
"The Felling"
status
"true_positive"
matchedExpected
name
"The Felling"
types
0
"lore"
17
parsed
type
"lore"
name
"Kindling Rite"
status
"true_positive"
matchedExpected
name
"The Kindling Rite"
types
0
"lore"
54.11%
Recall
Matched: 10/11, FP: 8
parsedCount
18
totalExpected
11
duplicates
(empty)
matched
10
falsePositives
8
weightedRecall
0.9047619047619048
fpPenalty
0.36363636363636365
expectedDetails
0
expected
name
"Torben Reikhart"
types
0
"character"
importance
"high"
status
"matched"
matchedBy
type
"character"
name
"Stray"
1
expected
name
"Sable Dunmore"
types
0
"character"
importance
"high"
status
"matched"
matchedBy
type
"character"
name
"Sable Dunmore"
2
expected
name
"Old Rivka"
types
0
"character"
importance
"low"
status
"matched"
matchedBy
type
"character"
name
"Old Rivka"
3
expected
name
"Petal Dunmore"
types
0
"character"
importance
"medium"
status
"matched"
matchedBy
type
"character"
name
"Petal"
4
expected
name
"Jurren Dunmore"
types
0
"character"
importance
"low"
status
"matched"
matchedBy
type
"character"
name
"Jurren"
5
expected
name
"The Rusty Lantern"
types
0
"location"
importance
"low"
status
"matched"
matchedBy
type
"location"
name
"Rusty Lantern"
6
expected
name
"Dunmore"
types
0
"location"
importance
"low"
status
"matched"
matchedBy
type
"location"
name
"Dunmore"
7
expected
name
"Kindling"
types
0
"object"
importance
"high"
status
"matched"
matchedBy
type
"object"
name
"Kindling"
8
expected
name
"Petal's Locket"
types
0
"object"
importance
"medium"
status
"missed"
9
expected
name
"The Felling"
types
0
"lore"
importance
"medium"
status
"matched"
matchedBy
type
"lore"
name
"The Felling"
10
expected
name
"The Kindling Rite"
types
0
"lore"
importance
"medium"
status
"matched"
matchedBy
type
"lore"
name
"Kindling Rite"
falsePositiveDetails
0
type
"character"
name
"Torben Reikhart"
1
type
"character"
name
"Rivka"
2
type
"location"
name
"harbor"
3
type
"location"
name
"corner table"
4
type
"location"
name
"salt cliffs"
5
type
"location"
name
"waterfront"
6
type
"object"
name
"glass"
7
type
"object"
name
"silver locket"
95.83%
Structural validity
Parsed: 18, Avg score: 0.96
parsedCount
18
avgScore
0.9583333333333334
entryDetails
0
name
"Old Rivka"
type
"character"
score
1
reason
"fully valid"
1
name
"Stray"
type
"character"
score
1
reason
"fully valid"
2
name
"Sable Dunmore"
type
"character"
score
1
reason
"fully valid"
3
name
"Jurren"
type
"character"
score
1
reason
"fully valid"
4
name
"Petal"
type
"character"
score
1
reason
"fully valid"
5
name
"Torben Reikhart"
type
"character"
score
1
reason
"fully valid"
6
name
"Rivka"
type
"character"
score
1
reason
"fully valid"
7
name
"Rusty Lantern"
type
"location"
score
1
reason
"fully valid"
8
name
"harbor"
type
"location"
score
1
reason
"fully valid"
9
name
"corner table"
type
"location"
score
1
reason
"fully valid"
10
name
"Dunmore"
type
"location"
score
0.75
reason
"invalid dropdown values"
dropdownIssues
0
"terrain: \"Port village\""
11
name
"salt cliffs"
type
"location"
score
1
reason
"fully valid"
12
name
"waterfront"
type
"location"
score
1
reason
"fully valid"
13
name
"glass"
type
"object"
score
1
reason
"fully valid"
14
name
"Kindling"
type
"object"
score
1
reason
"fully valid"
15
name
"silver locket"
type
"object"
score
0.5
reason
"invented fields"
inventedFields
0
"aliases"
16
name
"The Felling"
type
"lore"
score
1
reason
"fully valid"
17
name
"Kindling Rite"
type
"lore"
score
1
reason
"fully valid"
69.0727%