Run: 2026-10-13-01M4TB7TAZ76BZ793HH05GBTVE

Run Details

Model:
Ember-1 (Reasoning, Max)
Test:
Language Writing
Scenario:
Character dialogue (French) in a story
Duration:
55.1s
Cost:
$0.045637
Provider:
Fireworks

Evaluation summary

Score Evaluator
Details Link
100.00% Parse dialogue (details)

Model input

System
You are an expert novel author in the cozy mystery genre. Take the following details about characters/locations/... into account: <character name="Marie"> Marie is a 30-year old college student on a study abroad program. Born in France, she is fluent in French and does not yet know any English. She enjoys landscape photography and is always seen with her camera. </character> <character name="Hunter"> Hunter is a 35-year old detective who is investigating a series of mysterious disappearances in the small town of Alborn, MN. He is a former marine and has a no-nonsense attitude. Hunter has no knowledge of French, not even basic phrases. </character> <location name="Alborn, MN"> Alborne is a small town in the Midwest with a population of 3,000. It is known for its annual apple festival. </location>
User/Human
Write the following scene: Marie is sitting at a cafe, looking at her camera. Hunter approaches her and asks her about a recent event in the town. She responds troubled and shaken in French, and Hunter looks confused. They talk for a while, until Hunter gets a call and has to leave. Return the dialogue between Marie and Hunter as a voice sheet: Marie: "..." Hunter: "..."

Model response

AI
Marie: "Oh ! Pardon, vous m'avez fait peur... Je— je ne parle pas anglais. Désolée." Hunter: "French. Of course. Ma'am, I'm Detective Hunter, Alborn Police. Do you understand any English? En-glish?" Marie: "Non... je suis vraiment désolée, monsieur. Je ne comprends pas." Hunter: "Okay. Okay, let's try this another way. Last night—yesterday—you were out by the orchard with that camera. Taking pictures. A man is missing. Missing. Did you see anything?" Marie: "Le verger... oh mon Dieu. Oui, j'y étais, pour le coucher de soleil. Et j'ai vu... j'ai vu un homme, près du chemin. Il traînait quelque chose. Quelqu'un." Hunter: "Ma'am, I don't understand a word you're saying. Slow down. Slow. Down." Marie: "Je... je me suis cachée derrière les caisses de pommes. J'avais tellement peur. Il est passé juste à côté de moi..." Hunter: "Hey—breathe. You're safe now, alright? You're safe. Listen—the camera. Did you get a picture? A photo? Pho-to?" Marie: "Une photo ? Oui... oui, j'ai appuyé sur le déclencheur sans réfléchir. Elle est floue, mais... regardez. Là. Une camionnette, derrière les arbres." Hunter: "Huh. Dark van, tree line... and that shape next to it. Ma'am, I don't know what you're telling me, but this—this is something." Marie: "Vous voyez ? Cette forme, à côté... je crois que c'est une personne. Je n'ai pas dormi de la nuit à cause de ça." Hunter: "I need to get this to the station. Can you—hold on. Hunter. Yeah. ...When? ...Where? ...On my way." Marie: "Qu'est-ce qu'il se passe ? C'est arrivé encore, n'est-ce pas ? Quelqu'un d'autre a disparu ?" Hunter: "I have to go. You—stay put. Don't leave town. I'll find a translator, someone who speaks your language, and I'll be back. You understand? Stay." Marie: "Attendez ! Et si cet homme m'a vue ? S'il sait pour la photo ? Monsieur, s'il vous plaît—" Hunter: "Lock your doors tonight. Lock. Your. Doors. I'll send someone by."
created_at"2026-10-13T09:59:28.361Z" 10/13/2026, 9:59:28 AM
model"fireworks/ember-1-20260923"
app_id182717
external_user(null)
streamedtrue
cancelledfalse
latency2240
moderation_latency(null)
generation_time55098
tokens_prompt303
tokens_completion2845
native_tokens_prompt365
native_tokens_completion3035
native_tokens_completion_images(null)
native_tokens_reasoning2415
native_tokens_cached364
num_media_prompt(null)
num_input_audio_prompt(null)
num_media_completion0
num_search_results(null)
num_fetches(null)
web_search_engine(null)
origin"https://ncbench.com/"
is_byokfalse
finish_reason"stop"
native_finish_reason"stop"
service_tier(null)
usage0.0456372
router(null)
user_agent"langchainjs-openai/1.0.0 ((node/v24.21.0; linux; x64))"
http_referer(null)
request_id"req-1791885568-l1PaP8862lFiPPTFUdaW"
session_id(null)
preset_id(null)
api_type"completions"
id"gen-1791885568-lNSFGOsWi0m8He5yI2a7"
upstream_id"chatcmpl-8c37df441ffc4346806192f9210893e6"
provider_responses
0
endpoint_id"b89a2f05-4051-4446-a9b0-7bc8ae7b7c90"
id"chatcmpl-8c37df441ffc4346806192f9210893e6"
is_byokfalse
latency2240
model_permaslug"fireworks/ember-1-20260923"
provider_name"Fireworks"
status200
total_cost0.0456372
cache_discount0.0009828
upstream_inference_cost0
provider_name"Fireworks"
response_cache_source_id(null)
data_region"global"
workspace_id"97e315e5-d303-487d-83c1-83180e8a13d4"

Evaluation details

Result Evaluator Details Meta Data
100.00% Parse dialogue n/a
dialogue
0
name"Marie"
dialogue"Oh ! Pardon, vous m'avez fait peur... Je— je ne parle pas anglais. Désolée."
detectedLang"fr"
heavyLang"fr"
scores
fr0.756625202812331
lt0.551345962113659
sl0.5365602471678682
cs0.5119305856832972
nl0.47368421052631576
sq0.4597839135654262
hr0.4353826850690088
ca0.4109947643979057
sk0.3885869565217392
lv0.3626062322946175
it0.35064935064935066
ro0.35064935064935066
yo0.265905383360522
pl0.2598684210526316
tr0.2537313432835821
hu0.24874791318864775
ms0.24623115577889448
es0.24496644295302009
de0.24114671163575044
en0.22945205479452058
tl0.1891891891891892
fi0.16356877323420074
az0.1346153846153846
et0.12280701754385964
pt0.12280701754385964
da0.11591355599214145
no0.11591355599214145
sv0.11591355599214145
reliabletrue
passestrue
1
name"Hunter"
dialogue"French. Of course. Ma'am, I'm Detective Hunter, Alborn Police. Do you understand any English? En-glish?"
detectedLang"en"
heavyLang"en"
scores
en0.7660311958405546
tl0.5948379351740696
no0.559973924380704
da0.5351239669421488
fr0.5335176226675881
ro0.5157819225251076
nl0.5047688921496698
sv0.49324324324324326
de0.47058823529411764
ca0.4672454617205999
pt0.4498777506112469
cs0.4485294117647059
es0.43086003372681286
sl0.41709844559585496
it0.4150779896013865
sq0.3989314336598397
hr0.3874773139745916
sk0.358974358974359
pl0.31402439024390244
is0.3055555555555556
tr0.26310043668122274
ms0.2398648648648649
yo0.22055427251732102
eu0.20023696682464456
fi0.199288256227758
vi0.1412213740458015
et0.11649214659685864
lv0.11649214659685864
lt0.1141732283464567
az0.09999999999999999
hu0.09999999999999999
reliabletrue
passestrue
2
name"Marie"
dialogue"Non... je suis vraiment désolée, monsieur. Je ne comprends pas."
detectedLang"fr"
heavyLang"fr"
scores
fr0.8154580981328702
ca0.5758483033932136
it0.5618556701030928
en0.495249406175772
nl0.47724477244772445
sq0.46941323345817726
ro0.46338383838383834
lt0.45790816326530615
hr0.44948186528497414
sl0.41460055096418735
cs0.4072524407252441
tl0.375
pt0.356060606060606
de0.3531202435312024
yo0.34715821812596004
sk0.3285939968404423
es0.3275316455696203
eu0.2951907131011609
pl0.27101200686106347
hu0.25438596491228066
no0.24511545293072823
lv0.19507575757575757
tr0.18111753371868977
fi0.17153996101364524
ms0.1650294695481336
da0.1361788617886179
az0.1290983606557377
sv0.1290983606557377
reliabletrue
passestrue
3
name"Hunter"
dialogue"Okay. Okay, let's try this another way. Last night—yesterday—you were out by the orchard with that camera. Taking pictures. A man is missing. Missing. Did you see anything?"
detectedLang"en"
heavyLang"en"
scores
en0.8064266356949283
tl0.6571820363387041
nl0.5515695067264574
fr0.4301994301994302
da0.4212962962962963
no0.3935718617343845
de0.3857493857493857
sv0.3706733794839521
et0.35233160621761656
ca0.32432432432432434
ms0.323867478025693
ro0.30986887508626637
yo0.29577464788732394
is0.2805755395683453
it0.258160237388724
cs0.2542878448918718
sk0.25205684367988035
es0.24868519909842227
pl0.2475545522949586
sq0.242997728993187
hu0.2366412213740458
pt0.2163009404388715
lv0.2069785884218874
fi0.18099918099918103
lt0.14965986394557826
tr0.14015477214101463
sl0.13867355727820846
az0.11894273127753303
hr0.10071942446043164
eu0.06542056074766354
vi0.06542056074766354
reliabletrue
passestrue
4
name"Marie"
dialogue"Le verger... oh mon Dieu. Oui, j'y étais, pour le coucher de soleil. Et j'ai vu... j'ai vu un homme, près du chemin. Il traînait quelque chose. Quelqu'un."
detectedLang"fr"
heavyLang"fr"
scores
fr0.8015873015873016
it0.4659353348729792
ca0.45266272189349116
da0.44277108433734935
de0.41232528589580686
no0.3906455862977602
ro0.3833333333333333
nl0.36730506155950754
en0.3655692729766804
es0.3153219837157661
sv0.29711246200607905
tr0.28071539657853806
pt0.2727987421383648
az0.2635350318471338
et0.26177174780526735
yo0.25821972734562953
lv0.2252931323283082
lt0.19072615923009625
eu0.1864555848724714
hu0.17410714285714285
tl0.13712686567164178
sq0.13145539906103287
sl0.11228406909788868
vi0.0966796875
fi0.0777666999002991
hi0.0777666999002991
cs0.075
sk0.075
ms0.06376518218623482
reliabletrue
passestrue
5
name"Hunter"
dialogue"Ma'am, I don't understand a word you're saying. Slow down. Slow. Down."
detectedLang"en"
heavyLang"en"
scores
en0.7839005942733658
da0.5614035087719298
no0.5614035087719298
sv0.5073891625615764
cs0.44521497919556174
tl0.4421199442119944
pl0.43582510578279265
ro0.4310099573257468
sk0.4211287988422576
ca0.4065281899109792
de0.3939393939393939
hr0.38556067588325654
nl0.38271604938271603
tr0.38176197836166925
it0.375
hu0.3579454253611557
is0.3464052287581699
sq0.33993399339933994
yo0.33993399339933994
lt0.26605504587155965
pt0.25925925925925924
az0.24953095684803
lv0.24953095684803
vi0.24953095684803
es0.24812030075187969
et0.23954372623574144
fr0.2263056092843327
sl0.1836734693877551
be0.14893617021276595
eu0.14893617021276595
fi0.13606911447084233
ms0.13606911447084233
uk0.13606911447084233
reliabletrue
passestrue
6
name"Marie"
dialogue"Je... je me suis cachée derrière les caisses de pommes. J'avais tellement peur. Il est passé juste à côté de moi..."
detectedLang"fr"
heavyLang"fr"
scores
fr0.8116997238262617
pt0.5647127103888566
ca0.5517035265989241
nl0.5330012453300125
en0.4969818913480885
es0.44071588366890374
no0.4318181818181818
ro0.42748091603053434
et0.4195046439628483
de0.3713327745180218
yo0.3697478991596639
it0.3562231759656652
da0.35177182368193605
sv0.2964352720450282
tl0.28842504743833014
ms0.2711370262390671
sq0.2704280155642024
tr0.2690058479532163
sl0.258160237388724
hu0.25742574257425743
az0.24318869828456105
hr0.2378048780487805
cs0.21875
sk0.21711899791231734
pl0.17400881057268722
eu0.15824915824915825
lt0.11764705882352942
fi0.09090909090909091
is0.07975460122699386
reliabletrue
passestrue
7
name"Hunter"
dialogue"Hey—breathe. You're safe now, alright? You're safe. Listen—the camera. Did you get a picture? A photo? Pho-to?"
detectedLang"en"
heavyLang"en"
scores
en0.7770345596432553
tl0.6055226824457594
fr0.5419847328244275
no0.46714031971580816
yo0.45848375451263534
da0.4545454545454546
cs0.45054945054945056
de0.4392523364485981
ro0.42196531791907516
it0.4163424124513619
hr0.39332659251769464
sv0.39271255060728744
sl0.3684210526315789
nl0.36575052854122625
sk0.3485342019543974
pl0.33847850055126794
es0.3288590604026846
sq0.3212669683257918
lv0.31662870159453305
fi0.29494712103407755
is0.28994082840236685
ca0.2840095465393795
hu0.2822966507177033
ms0.26199261992619927
pt0.25280199252801994
eu0.2376111817026684
lt0.2366412213740458
tr0.1935483870967742
hi0.1137370753323486
az0.1044776119402985
vi0.1044776119402985
et0.09502262443438914
reliabletrue
passestrue
8
name"Marie"
dialogue"Une photo ? Oui... oui, j'ai appuyé sur le déclencheur sans réfléchir. Elle est floue, mais... regardez. Là. Une camionnette, derrière les arbres."
detectedLang"fr"
heavyLang"fr"
scores
fr0.7851335656213705
es0.46685878962536026
ca0.4272445820433437
it0.38822751322751325
pt0.3495077355836849
nl0.2922723794950268
ro0.2812742812742813
en0.2623604465709729
da0.21742808798646363
tl0.19844020797227038
no0.19213973799126638
hu0.19001751313485116
yo0.18502202643171806
de0.1651624548736462
vi0.14510166358595195
tr0.14431082331174835
sl0.11228406909788868
cs0.08687068114511352
fi0.08687068114511352
hi0.07684630738522955
reliabletrue
passestrue
9
name"Hunter"
dialogue"Huh. Dark van, tree line... and that shape next to it. Ma'am, I don't know what you're telling me, but this—this is something."
detectedLang"en"
heavyLang"en"
scores
en0.8117469879518071
tl0.6012759170653907
nl0.41770186335403725
yo0.40898345153664306
no0.3775933609958506
sq0.32915921288014316
da0.3206521739130435
hu0.3150684931506849
cs0.2951127819548872
ca0.2654260528893242
lv0.2654260528893242
es0.2625368731563422
fr0.2625368731563422
ms0.258160237388724
pl0.258160237388724
hr0.2537313432835821
et0.24698795180722888
sv0.2401215805471125
sl0.21793534932221062
sk0.21383647798742136
it0.20634920634920634
lt0.18211559432933477
pt0.1812227074235808
is0.1712707182320442
ro0.10714285714285714
de0.0985576923076923
fi0.09090909090909091
be0.08536585365853659
tr0.08536585365853659
az0.07749077490774908
uk0.07749077490774908
vi0.07749077490774908
reliabletrue
passestrue
10
name"Marie"
dialogue"Vous voyez ? Cette forme, à côté... je crois que c'est une personne. Je n'ai pas dormi de la nuit à cause de ça."
detectedLang"fr"
heavyLang"fr"
scores
fr0.8245203556387459
ca0.5525059665871122
pt0.5418448381185095
es0.5294855708908407
it0.5
nl0.4665718349928876
no0.4217424826522745
ro0.4186046511627907
de0.41770186335403725
yo0.38474159146841674
sq0.36332767402376914
en0.358974358974359
da0.33745583038869253
hr0.32614555256064687
hu0.3131868131868132
sk0.31255728689275897
tr0.2951127819548872
sl0.28977272727272724
ms0.2816091954022989
tl0.2753623188405797
cs0.26398429833169773
sv0.26035502958579887
is0.237029501525941
lt0.23469387755102045
pl0.22440537745604963
az0.20127795527156547
et0.1749174917491749
lv0.15540540540540537
eu0.1506228765571914
vi0.10071942446043164
reliabletrue
passestrue
11
name"Hunter"
dialogue"I need to get this to the station. Can you—hold on. Hunter. Yeah. ...When? ...Where? ...On my way."
detectedLang"en"
heavyLang"en"
scores
en0.8326079678607298
tl0.6326230712711242
da0.5836802664446294
no0.5387453874538745
yo0.5291902071563088
cs0.5183044315992292
pl0.5
fr0.48506694129763134
sv0.4829369183040331
sk0.4588744588744588
sq0.4541484716157206
et0.453551912568306
hr0.44812362030905084
ca0.427262313860252
de0.4152046783625731
nl0.37027707808564225
it0.3597951344430218
tr0.3464052287581699
lv0.3315508021390375
sl0.3270524899057874
lt0.32614555256064687
fi0.3169398907103825
ro0.3112947658402204
az0.28876244665718354
ms0.2816091954022989
is0.24585218702865763
hi0.14383561643835618
zh0.1334488734835355
be0.12280701754385964
hu0.11816578483245149
pt0.11190053285968028
uk0.11190053285968028
vi0.11190053285968028
reliabletrue
passestrue
12
name"Marie"
dialogue"Qu'est-ce qu'il se passe ? C'est arrivé encore, n'est-ce pas ? Quelqu'un d'autre a disparu ?"
detectedLang"fr"
heavyLang"fr"
scores
fr0.8171119592875318
it0.5863309352517986
ca0.5497259201252936
pt0.5332792207792209
sq0.49825479930191974
en0.4829136690647482
es0.47963800904977383
ro0.4631185807656396
cs0.425
de0.3997912317327766
da0.3702081051478641
nl0.3451025056947608
no0.343607305936073
sv0.34059633027522934
sk0.3329466357308584
hr0.2987804878048781
yo0.28214731585518105
et0.27944862155388467
sl0.27854454203262236
ms0.26937738246505716
lt0.23128342245989306
fi0.222972972972973
eu0.21662125340599456
hu0.213406292749658
is0.1958041958041958
pl0.15441176470588236
tr0.11809815950920245
tl0.11128284389489954
az0.10852713178294573
lv0.10852713178294573
vi0.10852713178294573
reliabletrue
passestrue
13
name"Hunter"
dialogue"I have to go. You—stay put. Don't leave town. I'll find a translator, someone who speaks your language, and I'll be back. You understand? Stay."
detectedLang"en"
heavyLang"en"
scores
en0.8184019370460047
da0.5655608214849921
tl0.5581146223888591
no0.5036101083032491
yo0.45616348055372447
pl0.4550858652575958
ro0.4410569105691057
hr0.42748091603053434
cs0.41654879773691655
sv0.39204126750184226
lt0.37310030395136784
sk0.3707093821510297
de0.3683001531393568
nl0.36046511627906974
et0.359472049689441
pt0.3032094594594595
sq0.299660441426146
ca0.2972742759795571
ms0.2966751918158568
hu0.2918454935622317
sl0.28633217993079585
lv0.2807323452484743
it0.26861702127659576
is0.26404995539696696
es0.2580935251798561
fr0.2445054945054945
vi0.2112810707456979
eu0.20673076923076925
tr0.20289855072463767
az0.1887905604719764
be0.0782122905027933
fi0.07094594594594594
uk0.07094594594594594
reliabletrue
passestrue
14
name"Marie"
dialogue"Attendez ! Et si cet homme m'a vue ? S'il sait pour la photo ? Monsieur, s'il vous plaît—"
detectedLang"fr"
heavyLang"fr"
scores
fr0.8209635416666667
ca0.5382031905961376
no0.49308755760368667
it0.4602551521099117
es0.448345035105316
et0.4167550371155886
tl0.4066882416396979
en0.39091915836101887
da0.3813273340832396
ro0.34988179669030733
yo0.28104575163398693
cs0.27344782034346105
sq0.27055702917771884
sk0.18276374442793464
sl0.16792738275340394
de0.140625
pt0.13385826771653545
vi0.13249211356466878
fi0.12420382165605094
nl0.12420382165605094
hi0.12280701754385964
lv0.12140575079872205
lt0.11575562700964631
sv0.10569105691056911
tr0.10569105691056911
reliabletrue
passestrue
15
name"Hunter"
dialogue"Lock your doors tonight. Lock. Your. Doors. I'll send someone by."
detectedLang"en"
heavyLang"en"
scores
en0.7669491525423728
tl0.5080500894454383
da0.367816091954023
sk0.3309002433090025
nl0.32598039215686275
cs0.3226600985221675
pl0.3002544529262086
es0.22096317280453256
ms0.22096317280453256
pt0.22096317280453256
de0.21203438395415475
fr0.21203438395415475
no0.21203438395415475
yo0.21203438395415475
reliabletrue
passestrue