Run: 2024-08-26-01J68DCMWXJRAQF3M90NT3Y2C3

Run Details

Model:
Hermes 3 70B
Test:
Language Writing
Scenario:
Character dialogue (Hindi) in a story
Duration:
19.3s
Cost:
$0.000259
Provider:
Hyperbolic

Evaluation summary

Score Evaluator
Details Link
60.00% Parse dialogue (details)

Model input

System
You are an expert novel author in the cozy mystery genre. Take the following details about characters/locations/... into account: <character name="Selvi"> Selvi is a 30-year old college student on a study abroad program. Born in India, she is fluent in Hindi and does not yet know any English. She enjoys landscape photography and is always seen with her camera. </character> <character name="Hunter"> Hunter is a 35-year old detective who is investigating a series of mysterious disappearances in the small town of Alborn, MN. He is a former marine and has a no-nonsense attitude. Hunter has no knowledge of Hindi, not even basic phrases. </character> <location name="Alborn, MN"> Alborne is a small town in the Midwest with a population of 3,000. It is known for its annual apple festival. </location>
User/Human
Write the following scene: Selvi is sitting at a cafe, looking at her camera. Hunter approaches her and asks her about a recent event in the town. She responds troubled and shaken in Hindi, and Hunter looks confused. They talk for a while, until Hunter gets a call and has to leave. Return the dialogue between Selvi and Hunter as a voice sheet: Selvi: "..." Hunter: "..."

Model response

AI
Here is the scene with the dialogue between Selvi and Hunter: Selvi: (looking at her camera) So many strange things have been happening in this small town. I don't feel safe anymore. Hunter: (approaching her) Excuse me, miss. Did you happen to see anything unusual around the time of the recent disappearance? Selvi: (troubled and shaken, in Hindi) हाँ, मैंने कुछ अजीब देखा। एक आदमी एक लंबा कोट और चेहरे का निशान पहने हुए था। वह बहुत तेजी से चल रहा था। Hunter: (confused) I'm sorry, I don't understand Hindi. Could you please speak in English? Selvi: (apologetically) I...I don't speak English well. I saw...strange man. Long coat. Face mark. Fast walking. Hunter: (noting the information) I see. A man in a long coat with a facial mark, walking quickly. Did you see where he went? Selvi: (nervously) He went in...apple orchard. (points in the direction) Hunter: Thank you, that helps. If you think of anything else, here is my card. (hands her a business card) Please have someone who speaks English call me. Selvi: (takes the card) Okay. I hope you find him. Hunter: (gets a call) Excuse me, I have to take this. Stay safe. (answers phone) Hunter here. What's the report? (Hunter remains on the call as he walks away. Selvi looks at the card, then back at her camera, troubled by what she saw.) </SOLUTION>
id"gen-3IAPn5aC6ahTZ2eQTXRmT9BDBokZ"
total_cost0.0002592
upstream_id"chat-7b71d5e98fde4b55b504d66373abe5a9"
created_at"2024-08-26T22:37:50.400625+00:00" 8/26/2024, 10:37:50 PM
model"nousresearch/hermes-3-llama-3.1-70b"
app_id182717
streamedtrue
cancelledfalse
provider_name"Hyperbolic"
latency420
moderation_latency(null)
generation_time18729
finish_reason"stop"
tokens_prompt280
tokens_completion421
native_tokens_prompt284
native_tokens_completion364
num_media_prompt(null)
num_media_completion(null)
origin"https://ncbench.com/"
usage0.0002592

Evaluation details

Result Evaluator Details Meta Data
60.00% Parse dialogue n/a
dialogue
0
name"Selvi"
dialogue"So many strange things have been happening in this small town. I don't feel safe anymore."
detectedLang"en"
heavyLang"en"
scores
en0.8245881635143381
tl0.5740740740740741
no0.554953560371517
da0.5440126883425853
nl0.5168067226890757
de0.4824482448244825
sv0.4772727272727273
it0.4295634920634921
sl0.4059917355371901
sk0.3889479277364506
is0.36323366555924697
ro0.3480725623582766
yo0.343607305936073
sq0.33679354094579006
pl0.33371958285052145
hr0.28393524283935245
az0.26376440460947503
cs0.24639580602883354
vi0.21981004070556312
fr0.20689655172413793
ca0.1935483870967742
ms0.1935483870967742
lv0.18785310734463279
lt0.1843971631205674
tr0.18323863636363635
et0.17503586800573886
be0.10852713178294573
hu0.1043613707165109
uk0.0987460815047022
reliabletrue
passesfalse
1
name"Hunter"
dialogue"Excuse me, miss. Did you happen to see anything unusual around the time of the recent disappearance?"
detectedLang"en"
heavyLang"en"
scores
en0.7997535428219347
tl0.6147006520450504
nl0.5272727272727272
no0.48859166011014943
da0.4328097731239092
fr0.43082311733800355
cs0.3930905695611578
yo0.3902439024390244
ms0.38967136150234744
pt0.38271604938271603
sq0.37198067632850246
et0.36274509803921573
sv0.34804413239719156
sk0.3440968718466196
sl0.34010152284263956
hr0.33127572016460904
es0.3193717277486911
it0.3128964059196617
de0.2655367231638418
pl0.2388758782201405
lt0.20440636474908203
ca0.1761723700887199
ro0.1761723700887199
lv0.15803108808290156
hu0.12634408602150538
is0.12043301759133965
fi0.10344827586206896
reliabletrue
passestrue
2
name"Selvi"
dialogue"हाँ, मैंने कुछ अजीब देखा। एक आदमी एक लंबा कोट और चेहरे का निशान पहने हुए था। वह बहुत तेजी से चल रहा था।"
detectedLang"hi"
heavyLang"hi"
scores
hi0.8831355627472133
mr0.8756536156102538
reliabletrue
passestrue
3
name"Hunter"
dialogue"I'm sorry, I don't understand Hindi. Could you please speak in English?"
detectedLang"en"
heavyLang"en"
scores
en0.8004561003420753
tl0.6384297520661157
sv0.5014245014245015
no0.49952335557673977
da0.493731918997107
ro0.47657028913260224
it0.4691607684529828
de0.4397011739594451
nl0.38882421420256114
pl0.35424354243542433
cs0.3371212121212121
is0.30921052631578944
sq0.29624664879356566
hr0.2866847826086956
yo0.25
sk0.23469387755102045
et0.23020527859237538
sl0.22680412371134018
az0.21875
ms0.21875
ca0.2081447963800905
fr0.1758241758241758
pt0.16
hi0.12935323383084577
be0.11764705882352942
tr0.11764705882352942
hu0.11317567567567569
lt0.10714285714285714
lv0.10714285714285714
uk0.10714285714285714
vi0.10714285714285714
reliabletrue
passestrue
4
name"Selvi"
dialogue"I...I don't speak English well. I saw...strange man. Long coat. Face mark. Fast walking."
detectedLang"en"
heavyLang"en"
scores
en0.7940156114483955
tl0.5984784446322908
no0.5898100172711571
sv0.5770258236865539
da0.5670009115770283
de0.5278330019880716
nl0.4831338411316649
fr0.46145124716553293
ro0.4235436893203884
pl0.40550688360450565
hr0.3870967741935484
is0.3683510638297872
lv0.3365921787709497
ms0.33098591549295775
lt0.32045779685264664
it0.29420505200594355
sq0.28892215568862273
cs0.2835595776772248
vi0.2543171114599686
sk0.25196850393700787
pt0.2375601926163724
ca0.22512234910277326
tr0.21357615894039736
az0.20435510887772196
et0.20435510887772196
sl0.15630550621669628
yo0.15480427046263345
be0.12844036697247707
hu0.12361623616236161
uk0.1171003717472119
reliabletrue
passesfalse
5
name"Hunter"
dialogue"I see. A man in a long coat with a facial mark, walking quickly. Did you see where he went?"
detectedLang"en"
heavyLang"en"
scores
en0.8186528497409327
tl0.6206647398843931
sv0.5257452574525745
nl0.5209854014598541
de0.5138888888888888
ca0.5098039215686274
it0.5051837888784166
ro0.5028409090909091
pl0.453125
fr0.4519832985386221
da0.4366952789699571
yo0.4299674267100977
no0.42934782608695654
es0.42748091603053434
pt0.42434210526315785
sq0.4006849315068493
cs0.375
et0.36053593179049936
az0.35503685503685506
is0.35185185185185186
hr0.3234536082474227
sl0.31729518855656696
lv0.31462140992167104
lt0.31282722513089006
vi0.3064729194187582
hu0.2981283422459893
ms0.29149797570850206
sk0.2778541953232462
fi0.22680412371134018
tr0.21875
be0.11764705882352942
eu0.11764705882352942
uk0.10714285714285714
reliabletrue
passestrue
6
name"Selvi"
dialogue"He went in...apple orchard."
detectedLang"en"
heavyLang"en"
scores
en0.7697368421052632
de0.6812386156648452
nl0.6354166666666666
tl0.6004566210045662
fr0.580335731414868
es0.5742092457420924
it0.5679012345679012
yo0.5205479452054795
sv0.5084269662921348
sl0.46808510638297873
ro0.4479495268138801
ca0.34210526315789475
fi0.34210526315789475
az0.32432432432432434
pl0.32432432432432434
ms0.3
no0.3
reliabletrue
passesfalse
7
name"Hunter"
dialogue"Thank you, that helps. If you think of anything else, here is my card. Please have someone who speaks English call me."
detectedLang"en"
heavyLang"en"
scores
en0.831384892086331
tl0.5052770448548813
nl0.4261667941851569
sq0.3169398907103825
no0.2625368731563422
et0.2378048780487805
ms0.2378048780487805
yo0.237029501525941
da0.2260061919504644
is0.19441460794844254
es0.1935483870967742
fi0.1883116883116883
ro0.18655097613882862
sv0.18655097613882862
pt0.1812227074235808
cs0.17943107221006563
de0.1712707182320442
hu0.17035398230088497
vi0.16201117318435754
pl0.10926365795724466
fr0.10820451843043996
sl0.10501193317422433
sk0.10287081339712918
it0.09638554216867469
hr0.08759124087591241
reliabletrue
passestrue
8
name"Selvi"
dialogue"Okay. I hope you find him."
detectedLang"en"
heavyLang"en"
scores
en0.8
tl0.6323529411764706
da0.6143958868894601
cs0.5495495495495496
no0.4915254237288135
hr0.4809688581314879
ca0.4791666666666667
sv0.4791666666666667
sq0.471830985915493
pl0.45652173913043476
it0.43396226415094336
ro0.375
yo0.3670886075949367
sk0.32432432432432434
be0.3181818181818182
tr0.3181818181818182
hu0.3087557603686636
az0.29577464788732394
et0.29577464788732394
is0.29577464788732394
lt0.29577464788732394
lv0.29577464788732394
uk0.29577464788732394
vi0.29577464788732394
reliabletrue
passesfalse
9
name"Hunter"
dialogue"Excuse me, I have to take this. Stay safe. Hunter here. What's the report?"
detectedLang"en"
heavyLang"en"
scores
en0.8106060606060606
tl0.5982142857142857
da0.5766698024459078
no0.5635305528612997
sq0.5622568093385214
sv0.48571428571428565
sl0.47368421052631576
fr0.4693396226415094
cs0.4492044063647491
yo0.4238156209987196
it0.42159383033419023
pl0.42159383033419023
hr0.4109947643979057
nl0.4007989347536618
ro0.39271255060728744
ms0.3869209809264305
ca0.3767313019390581
sk0.36886395511921455
pt0.33135215453194655
et0.3253373313343328
lv0.2913385826771654
es0.2902208201892744
lt0.2706645056726094
de0.256198347107438
hu0.22945205479452058
tr0.2227979274611399
fi0.14285714285714285
be0.1346153846153846
az0.12280701754385964
is0.12280701754385964
uk0.12280701754385964
vi0.12280701754385964
reliabletrue
passestrue