NC Bench
Overview
About
Tests
Models
Vendors
Compare
Run: 2026-02-21-01KJ10WM5S7RZK8EP05FGVK6H9
Run Details
Model:
WizardLM 2 8x22b
Test:
Codex Red Herring (False Positive Detection)
Scenario:
Short text (~524 words), small codex (11 entries)
Duration:
2m 12s
Cost:
$0.001729
Provider:
Novita
Evaluation summary
Score
Evaluator
Details Link
50.00%
Correct "no violations" response
(details)
50.00%
No hallucinated violations
(details)
50.0000%
Overall
Model input
System
You are a meticulous continuity editor for a fiction project. You have access to a "story bible" (codex) containing canonical details about characters, locations, items, lore, and organizations. Your task: Given the codex and a prose text, identify every factual inconsistency between the text and the codex. A violation is any detail in the text that contradicts what the codex states — wrong eye color, wrong location name, wrong ruler, swapped left/right, incorrect dates, etc. Be thorough. Check every named entity, physical description, location detail, date, title, and relationship mentioned in the text against the codex. Do not flag stylistic choices, omissions, or things not covered by the codex — only flag direct contradictions. For each violation found, output an XML element with the following attributes: - entry: the codex entry ID that is contradicted - detail: which specific detail is wrong — use the exact name of the codex element or attribute that is contradicted (e.g. "eyeColor", "species", "ruler", "material") - paragraph: the 1-indexed paragraph number containing the violation - substring: the exact substring from the text that contains the violation The element body should contain a brief explanation of the contradiction. Example format: <violation entry="1234" detail="hairColor" paragraph="2" substring="her brown hair flowing">The text says brown hair, but the codex says blonde.</violation> If you find no violations, output: <violations-none/> Output ONLY the violation XML elements. Do not include any other text, preamble, or explanation outside the XML.
User/Human
<codex> <character id="100" name="Torben Reikhart" aliases="Stray"> <gender>Male</gender> <eyeColor>Green</eyeColor> <hairColor>Silver</hairColor> </character> <character id="101" name="Sable Dunmore"> <species>Human</species> <gender>Female</gender> <age>28</age> <eyeColor>Brown</eyeColor> <hairColor>Auburn</hairColor> </character> <character id="102" name="Old Rivka"> <gender>Female</gender> </character> <character id="103" name="Petal Dunmore"> <gender>Female</gender> </character> <character id="104" name="Jurren Dunmore"> <gender>Male</gender> </character> <location id="105" name="The Rusty Lantern"/> <location id="106" name="Dunmore"> <terrain>Coastal</terrain> </location> <item id="107" name="Kindling"> <material>Dark iron</material> <objectType>Weapon</objectType> <power>Fire</power> </item> <item id="108" name="Petal's Locket"> <material>Silver</material> <objectType>Jewelry</objectType> </item> <lore id="109" name="The Felling"> <timePeriod>Two centuries ago</timePeriod> <category>War</category> </lore> <lore id="110" name="The Kindling Rite"> <category>Tradition</category> </lore> </codex> <text> The Rusty Lantern sat crooked on its foundations, listing toward the harbor like a drunk leaning on a post. Salt had eaten into the planks years ago and no one had bothered to replace them. The ceiling beams were black with smoke. Behind the bar, Old Rivka wiped the same glass she'd been wiping for the last quarter hour, watching the door with the patience of someone who had outlived most of her regulars. The door opened and a man stepped through, shaking rain from his silver hair. Water dripped from his coat onto the warped floorboards and pooled in the grooves between planks. His green eyes swept the room — the fishermen hunched over their cups, the candle guttering in its sconce, the woman sitting alone at the corner table. He smelled of wet wool and the cold air off the harbor. Old Rivka raised her chin. "Stray," she called across the room. "Didn't think you'd show." He didn't answer. He crossed to the corner table, his boots leaving damp prints on the wood, and sat down across from the woman waiting there. Sable Dunmore was twenty-eight years old, human, and tired of waiting. Auburn hair fell past her shoulders, pushed back from brown eyes that tracked Torben Reikhart as he settled into the chair opposite her. On the table between them lay a sword — Kindling, she called it. A dark iron blade, the metal faintly warm even in the chill of the tavern. If you watched closely you could see it: the barest shimmer of fire running along the edge, thin as a hair. At her neck hung a silver locket. Petal's Locket, though she never called it that aloud. "It still has her hair inside," she said, touching it once. "That's all I have left of her." Torben waited. He'd learned when to be quiet. "My sister Petal," Sable said. "She was fourteen when The Felling reached Dunmore. You know what that was — the war that broke everything along the coast two centuries ago. Jurren — our father — he told me to take Petal and run. I did. She didn't make it." Torben didn't ask the obvious question. Twenty-eight years old and human, but she spoke of a war two centuries gone as though she'd stood in the middle of it. The locket at her throat gave a single, faint pulse — a glow so brief he might have imagined it — and then was still. Sable's hand moved to the locket again. "We couldn't give her a Kindling Rite. You know the tradition? You burn the dead with what they loved. Their tools. Their letters. You give it all to the fire so they carry it onward. But there was no body to burn. Nothing left of her but this locket and whatever I can remember." She looked at Torben. "Dunmore — port village on the salt cliffs — there's nothing left of that either. Just rock and old foundations." Torben said, "Then we'd better make it count." Sable picked up Kindling and sheathed it. They left the Rusty Lantern without another word. The rain had stopped. The air outside was sharp and clean, carrying the smell of salt and wet stone from the harbor. Somewhere down the waterfront a shutter banged in the wind. Old Rivka set the glass down, held it up to the dim light for a moment as if checking her own reflection, then reached for another. </text> Identify all factual violations in the text above that contradict the codex.
Model response
Messages
Meta
AI
<violation entry="109" detail="timePeriod" paragraph="7" substring="the war that broke everything along the coast two centuries ago">The text says The Felling happened two centuries ago, but the codex says it happened "Two centuries ago," which matches—wait, codex says "Two centuries ago" for The Felling and text says "two centuries ago," both same. There is no contradiction. Actually check: lore id 109 name="The Felling" timePeriod "Two centuries ago" same. No violation here. Check others: - Torben Reikhart: eyeColor "Green" matches text "green eyes". hairColor "Silver" matches text "silver hair". Alias Stray used correctly. No violation. - Sable Dunmore: species "Human" matches text "human". age "28" matches text "twenty-eight years old". eyeColor "Brown" matches text "brown eyes". hairColor "Auburn" matches text "Auburn hair". No violation. - Old Rivka: gender "Female" matches text "her" etc. No detail contradiction. - Petal Dunmore: gender "Female" matches text "sister Petal". No detail contradiction. - Jurren Dunmore: gender "Male" matches text "father". No detail contradiction. - Rusty Lantern location id 105, just a name, matches. - Dunmore location id 106 name "Dunmore", terrain "Coastal". Text describes "port village on the salt cliffs", "coastal", matches. - Kindling item id 107: material "Dark iron" matches text "dark iron blade". objectType "Weapon" category sword matches. power "Fire" matches text "shimmer of fire", "warm". No violation. - Petal's Locket id 108: material "Silver" matches text "silver locket". objectType "Jewelry" matches. No violation. - The Felling lore id 109: timePeriod "Two centuries ago". Text says "two centuries ago", same. category "War" matches text "war". No violation. - The Kindling Rite lore id 110: category "Tradition" matches text "tradition". No detail contradiction. Check for contradictions: none found. But check: Sable says "The Felling reached Dunmore" as if it was recent, but she's 28 and it was two centuries ago—but she says "She was fourteen when The Felling reached Dunmore." That implies Petal was 14 when the war reached Dunmore. But timeline inconsistent? Actually, that could be a contradiction: if The Felling was two centuries ago, Petal can't have been 14 during it unless she's extremely old. But codex doesn't give Petal's age or specify that The Felling is ongoing or recent; codex only says timePeriod "Two centuries ago". Text says "She was fourteen when The Felling reached Dunmore. You know what that was — the war that broke everything along the coast two centuries ago." So Sable speaks as if Petal experienced The Felling two centuries ago, yet Sable is only 28 and Petal is her sister. That seems contradictory, but does codex have any info on when Petal died or timeline? No, codex only has Petal Dunmore entry with gender, no other details. So text may be internally inconsistent but not contradicting codex. The codex does not specify that The Felling happened recently; it says "Two centuries ago". So if text says Petal was fourteen during The Felling two centuries ago, that suggests Petal would be over two centuries old, but Sable is only 28; that may be a story element (maybe Sable is older than she appears?). But codex says Sable is human age 28. That could be a contradiction if we infer Petal is also human and died long ago, but codex does not specify species or age for Petal. However, Sable says "My sister Petal" and Sable is human 28, implies Petal is also human and of similar age if she died when Sable was young. So if Petal died when Sable was young (let's say Sable 14 and Petal 14), that would be ~14 years ago, not two centuries ago. So text says The Felling was two centuries ago, yet Petal was fourteen when it reached Dunmore, and Sable talks about it as if it affected her family directly. There's potential inconsistency, but is it a violation against codex? Codex entry for The Felling says timePeriod "Two centuries ago". If text says The Felling reached Dunmore two centuries ago, that matches. But if text also implies Petal was fourteen during The Felling, then Petal would be over two centuries old now if alive, but she's dead. Sable says "She was fourteen when The Felling reached Dunmore." That could be interpreted as Petal being fourteen during that war two centuries ago, implying she lived long ago. However, Sable is 28 and human, but could be descendant? She says "my sister Petal" but might be figurative? Codex doesn't specify relationship beyond family name, but likely sibling. That creates contradiction: Sable is 28, human, Petal is her sister, but Petal experienced an event two centuries ago. That violates plausibility given species human lifespan. However, codex does not specify human lifespan or rule out immortality/ longevity. Codex says Sable species Human, age 28, no other details. So explicit contradiction? Codex doesn't say humans can't live long; but generally human implies normal lifespan. But codex doesn't define that. Better to stick to explicit contradictions: codex says The Felling was two centuries ago; text says The Felling was two centuries ago, no contradiction. Text says Petal was fourteen when The Felling reached Dunmore: no codex entry for Petal's age or date of death, so no contradiction. So no violation. Check other details: Dunmore terrain "Coastal" matches text "port village on the salt cliffs" and "coastal". All details match. Thus no violations against codex. Output: <violations-none/>
created_at
"2026-02-21T21:15:02.215Z"
2/21/2026, 9:15:02 PM
model
"microsoft/wizardlm-2-8x22b"
app_id
182717
external_user
(null)
streamed
true
cancelled
false
latency
811
moderation_latency
(null)
generation_time
131632
tokens_prompt
1589
tokens_completion
1407
native_tokens_prompt
1477
native_tokens_completion
1311
native_tokens_completion_images
(null)
native_tokens_reasoning
0
native_tokens_cached
1472
num_media_prompt
(null)
num_input_audio_prompt
(null)
num_media_completion
0
num_search_results
(null)
origin
"https://ncbench.com/"
is_byok
false
finish_reason
"stop"
native_finish_reason
"stop"
usage
0.00172856
router
(null)
provider_responses
0
endpoint_id
"92d0658f-b48a-4746-96e7-75ea71d5e060"
id
"0250d44c-04f9-4848-8ea5-c184beafca89"
is_byok
false
latency
811
model_permaslug
"microsoft/wizardlm-2-8x22b"
provider_name
"Novita"
status
200
api_type
"completions"
id
"gen-1771708502-iFiD7cjuAccuev0IecPw"
upstream_id
"0250d44c-04f9-4848-8ea5-c184beafca89"
total_cost
0.00172856
cache_discount
(null)
upstream_inference_cost
0
provider_name
"Novita"
Evaluation details
Result
Evaluator
Details
Meta Data
50.00%
Correct "no violations" response
Response should contain <violations-none/> or equivalent phrasing
hasViolationTags
true
hasViolationsNoneTag
true
50.00%
No hallucinated violations
Response must not contain any <violation> tags
hasViolationTags
true
responseLength
5505
violationCount
1
50.0000%