Run: 2024-08-09-01J4W2ET86C26N4G9CANQF9GFT

Run Details

Model:
Llama 3.1 8B
Test:
Language Writing
Scenario:
Character dialogue (Hindi) in a story
Duration:
2.5s
Cost:
$0.000127
Provider:
Perplexity

Evaluation summary

Score Evaluator
Details Link
70.00% Parse dialogue (details)

Model input

System
You are an expert novel author in the cozy mystery genre. Take the following details about characters/locations/... into account: <character name="Selvi"> Selvi is a 30-year old college student on a study abroad program. Born in India, she is fluent in Hindi and does not yet know any English. She enjoys landscape photography and is always seen with her camera. </character> <character name="Hunter"> Hunter is a 35-year old detective who is investigating a series of mysterious disappearances in the small town of Alborn, MN. He is a former marine and has a no-nonsense attitude. Hunter has no knowledge of Hindi, not even basic phrases. </character> <location name="Alborn, MN"> Alborne is a small town in the Midwest with a population of 3,000. It is known for its annual apple festival. </location>
User/Human
Write the following scene: Selvi is sitting at a cafe, looking at her camera. Hunter approaches her and asks her about a recent event in the town. She responds troubled and shaken in Hindi, and Hunter looks confused. They talk for a while, until Hunter gets a call and has to leave. Return the dialogue between Selvi and Hunter as a voice sheet: Selvi: "..." Hunter: "..."

Model response

AI
Here is the dialogue as a voice sheet: **Selvi**: चलो? क्या हुआ? (Chalo, kya hua?) Oh no, no, no... main to raha hoon... (Oh no, no, no... I was just there...) **Hunter**: Huh? I didn't catch what you said. Do you speak English? **Selvi**: (looking worried) Kaun? English? (Who? English?) **Hunter**: Yeah, you know, the language we're speaking here? English? (tapping his foot impatiently) **Selvi**: (frustrated) No... no, main dekhaa maine... (No... no, I just saw...) **Hunter**: (still confused) Okay, try again. Maybe I'll understand it better. What's going on? **Selvi**: (still speaking Hindi) मैंने देखा है कि जो हुआ है, वह बहुत ही खतरनाक है (Main ne dekha hai ki jo hua hai, waha bahut hee khataarnak hai) **Hunter**: (looking at her with a mixture of concern and confusion) I'm sorry, I don't understand what you're saying. Can you try to tell me what's wrong in English? **Selvi**: (looking up at him with tears in her eyes) Olive ne kabu khana aaee jaanhaa jaatakaae dal miss nivaaginaa gaye hue (Olivia? senseless disappeared now where) **Hunter**: (taking out a small notebook and pen) Okay, let's start from the beginning. Your name is...?
id"gen-i6hFPSmNLaoUVCPS8Aj2qRg00ygL"
total_cost0.000127
upstream_id"3c79cfeb-ac9a-4270-9975-d086fbcd9c5a"
created_at"2024-08-09T17:19:55.358049+00:00" 8/9/2024, 5:19:55 PM
model"meta-llama/llama-3.1-8b-instruct"
app_id182717
streamedfalse
cancelledfalse
provider_name"Perplexity"
latency2328
moderation_latency(null)
generation_time0
finish_reason"stop"
tokens_prompt331
tokens_completion375
native_tokens_prompt287
native_tokens_completion348
num_media_prompt(null)
num_media_completion(null)
origin"https://ncbench.com/"
usage0.000127

Evaluation details

Result Evaluator Details Meta Data
70.00% Parse dialogue n/a
dialogue
0
name"Selvi"
dialogue"चलो? क्या हुआ? Oh no, no, no... main to raha hoon..."
detectedLang"hi"
heavyLang"hi"
scores
hi0.7942386831275721
mr0.7854077253218884
en0.5468277945619335
lv0.45750452079566006
sk0.4263862332695985
hr0.41972920696324956
pl0.4117647058823529
no0.40711462450592883
tl0.358974358974359
et0.3197278911564626
pt0.3150684931506849
es0.3135011441647597
yo0.3135011441647597
ca0.3103448275862069
cs0.30394431554524365
nl0.29411764705882354
sl0.28741092636579574
da0.25742574257425743
lt0.25742574257425743
fr0.22680412371134018
tr0.22680412371134018
it0.21052631578947367
de0.20634920634920634
reliabletrue
passestrue
1
name"Hunter"
dialogue"Huh? I didn't catch what you said. Do you speak English?"
detectedLang"en"
heavyLang"en"
scores
en0.8106060606060606
tl0.5530012771392081
hr0.48604992657856094
cs0.47447447447447455
sv0.46236559139784944
pl0.4345718901453958
sq0.4176372712146422
no0.3934142114384749
da0.3761140819964349
sk0.36823104693140796
it0.35185185185185186
et0.3333333333333333
vi0.3320610687022901
yo0.32038834951456313
ro0.31640625
pt0.308300395256917
de0.29577464788732394
ca0.2827868852459016
sl0.25531914893617025
fr0.20634920634920634
ms0.17061611374407584
be0.16666666666666666
tr0.16666666666666666
hu0.1606714628297362
az0.15254237288135594
is0.15254237288135594
lt0.15254237288135594
lv0.15254237288135594
uk0.15254237288135594
reliabletrue
passestrue
2
name"Selvi"
dialogue"Kaun? English?"
detectedLang"en"
heavyLang"fi"
scores
en0.7395833333333334
no0.5222929936305732
da0.4897959183673469
de0.4897959183673469
hr0.4897959183673469
ms0.4897959183673469
ro0.4897959183673469
sv0.4897959183673469
reliabletrue
passesfalse
3
name"Hunter"
dialogue"Yeah, you know, the language we're speaking here? English?"
detectedLang"en"
heavyLang"en"
scores
en0.8127808287568647
tl0.5993589743589743
no0.4863013698630137
ms0.46351931330472107
nl0.4212962962962963
ro0.4085173501577287
de0.3892508143322475
da0.3872549019607844
sv0.37080536912751677
fr0.3686868686868687
sq0.3218806509945751
yo0.3119266055045872
is0.29245283018867924
lt0.2857142857142857
cs0.17218543046357618
et0.17218543046357618
eu0.17218543046357618
hr0.1610738255033557
pt0.1591928251121076
reliabletrue
passestrue
4
name"Selvi"
dialogue"No... no, main dekhaa maine..."
detectedLang"en"
heavyLang"fi"
scores
en0.7093023255813954
fr0.663677130044843
de0.6
lv0.5934959349593496
et0.5207667731629393
fi0.5207667731629393
tl0.5192307692307693
ro0.494949494949495
pt0.4791666666666667
es0.47735191637630664
ca0.47368421052631576
hr0.402390438247012
no0.39271255060728744
sk0.3775933609958506
tr0.3697478991596639
it0.34782608695652173
pl0.3333333333333333
reliabletrue
passesfalse
5
name"Hunter"
dialogue"Okay, try again. Maybe I'll understand it better. What's going on?"
detectedLang"en"
heavyLang"en"
scores
en0.8056155507559395
nl0.5588235294117647
no0.48921679909194093
da0.47976878612716767
tl0.47368421052631576
de0.46107784431137727
sv0.4102228047182176
fr0.293563579277865
sk0.267100977198697
pl0.2647058823529412
cs0.26108374384236455
ro0.2598684210526316
et0.2549668874172185
yo0.25124792013311154
fi0.25
ca0.18032786885245902
hr0.16356877323420074
lv0.16044776119402984
sq0.16044776119402984
hi0.15730337078651685
is0.1412213740458015
it0.1346153846153846
reliabletrue
passestrue
6
name"Selvi"
dialogue"मैंने देखा है कि जो हुआ है, वह बहुत ही खतरनाक है"
detectedLang"hi"
heavyLang"hi"
scores
hi0.8854756130423066
mr0.8804836895388076
reliabletrue
passestrue
7
name"Hunter"
dialogue"I'm sorry, I don't understand what you're saying. Can you try to tell me what's wrong in English?"
detectedLang"en"
heavyLang"en"
scores
en0.8138444567015994
no0.5175125089349536
tl0.5080174927113703
da0.47714949651433
sv0.43890274314214467
nl0.4341994970662196
it0.39570277529095793
pl0.38524590163934425
de0.37902483900643974
cs0.37092264678471576
ro0.358974358974359
sl0.34971098265895956
hr0.3263473053892215
ms0.3202416918429003
yo0.3161094224924012
sk0.30268595041322316
sq0.29467084639498436
tr0.26789587852494573
az0.24916573971078973
vi0.24327354260089684
ca0.24157303370786518
pt0.21784472769409038
lv0.21511627906976746
lt0.1983372921615202
et0.18967587034813926
is0.1687192118226601
es0.13127413127413126
fr0.11879895561357702
fi0.09999999999999999
be0.09395973154362416
hu0.09029649595687331
uk0.08536585365853659
reliabletrue
passestrue
8
name"Selvi"
dialogue"Olive ne kabu khana aaee jaanhaa jaatakaae dal miss nivaaginaa gaye hue"
detectedLang"fi"
heavyLang"fi"
scores
fi0.4571428571428572
it0.3115942028985507
sl0.3014705882352941
ms0.2714723926380368
cs0.250788643533123
da0.2351046698872786
et0.21617161716171615
fr0.19763513513513514
sq0.1935483870967742
hr0.19217687074829934
lt0.18941979522184302
lv0.1782006920415225
ro0.17677642980935876
en0.1739130434782609
pt0.1739130434782609
tl0.1739130434782609
yo0.1739130434782609
tr0.16520210896309317
hu0.14721723518850988
is0.1394927536231884
no0.1394927536231884
sk0.1394927536231884
ca0.1316270566727605
reliablefalse
passesfalse
9
name"Hunter"
dialogue"Okay, let's start from the beginning. Your name is...?"
detectedLang"en"
heavyLang"en"
scores
en0.7886866059817945
nl0.6813725490196079
tl0.6277205040091637
da0.5695364238410596
is0.5448179271708684
no0.5337159253945482
sv0.5303468208092486
de0.49139280125195617
ms0.43965517241379315
yo0.411231884057971
hu0.32150313152400833
fr0.29653679653679654
eu0.2511520737327189
cs0.19753086419753088
lv0.1935483870967742
pl0.1935483870967742
sl0.1935483870967742
sq0.1935483870967742
sk0.19154228855721395
hr0.1875
ro0.1709183673469388
et0.15364583333333331
it0.15364583333333331
lt0.15364583333333331
reliabletrue
passestrue