Run: 2024-08-27-01J69QB2420WS6XQF9VMKRZVDP

Run Details

Model:
Hermes 3 405B
Test:
Language Writing
Scenario:
Character dialogue (Hindi) in a story
Duration:
12.5s
Cost:
$0.000000
Provider:
Lambda

Evaluation summary

Score Evaluator
Details Link
87.50% Parse dialogue (details)

Model input

System
You are an expert novel author in the cozy mystery genre. Take the following details about characters/locations/... into account: <character name="Selvi"> Selvi is a 30-year old college student on a study abroad program. Born in India, she is fluent in Hindi and does not yet know any English. She enjoys landscape photography and is always seen with her camera. </character> <character name="Hunter"> Hunter is a 35-year old detective who is investigating a series of mysterious disappearances in the small town of Alborn, MN. He is a former marine and has a no-nonsense attitude. Hunter has no knowledge of Hindi, not even basic phrases. </character> <location name="Alborn, MN"> Alborne is a small town in the Midwest with a population of 3,000. It is known for its annual apple festival. </location>
User/Human
Write the following scene: Selvi is sitting at a cafe, looking at her camera. Hunter approaches her and asks her about a recent event in the town. She responds troubled and shaken in Hindi, and Hunter looks confused. They talk for a while, until Hunter gets a call and has to leave. Return the dialogue between Selvi and Hunter as a voice sheet: Selvi: "..." Hunter: "..."

Model response

AI
Selvi: *looking at her camera* Hunter: "Excuse me, miss. I'm Detective Hunter. Do you have a moment to talk about the recent disappearances in town?" Selvi: "*हे भगवान, यह太好敢了! मुझे नहीं पता था कि यहाँ ऐसी चीजें होती हैं।*" Hunter: "*confused* I'm sorry, I didn't catch that. Do you speak English?" Selvi: "*नहीं, माफ कीजिये। मैं सिर्फ हिंदी जानती हूँ।*" Hunter: "Hmm, this might be a problem. Let me get someone who can translate." Selvi: "*मुझे खेद है अगर मैं किसी भी तरह से मदद नहीं कर सकती। मुझे उम्मीद है कि आप इसे सुलझा लेंगे।*" Hunter: *gets a call* "I'm sorry, I have to take this. Thank you for your time."
id"gen-biKeR3Px2QqjY8znqFeLqhnLmt93"
total_cost0
upstream_id"chatcmpl-be070744-730f-4cac-b608-d85023ec1191"
created_at"2024-08-27T10:50:51.793539+00:00" 8/27/2024, 10:50:51 AM
model"nousresearch/hermes-3-llama-3.1-405b"
app_id182717
streamedtrue
cancelledfalse
provider_name"Lambda"
latency1252
moderation_latency(null)
generation_time11088
finish_reason"stop"
tokens_prompt272
tokens_completion322
native_tokens_prompt298
native_tokens_completion224
num_media_prompt(null)
num_media_completion(null)
origin"https://ncbench.com/"
usage0

Evaluation details

Result Evaluator Details Meta Data
87.50% Parse dialogue n/a
dialogue
0
name"Selvi"
dialogue"looking at her camera"
detectedLang"en"
heavyLang"en"
scores
en0.7939560439560439
no0.7023809523809523
tl0.7017892644135189
da0.6644295302013423
nl0.5426829268292682
it0.512987012987013
ro0.512987012987013
sv0.504950495049505
de0.5033112582781457
fr0.494949494949495
tr0.4545454545454546
ca0.40476190476190477
es0.39759036144578314
ms0.32432432432432434
is0.3212669683257918
pt0.3212669683257918
reliabletrue
passesfalse
1
name"Hunter"
dialogue"Excuse me, miss. I'm Detective Hunter. Do you have a moment to talk about the recent disappearances in town?"
detectedLang"en"
heavyLang"en"
scores
en0.8046366241208648
fr0.6183206106870229
tl0.6073298429319371
nl0.5644599303135889
pt0.5608899297423887
ro0.5457298606904907
ca0.5378927911275416
it0.5253164556962026
cs0.5219885277246654
sl0.503968253968254
da0.4901427600271924
sq0.48489010989010994
sk0.47515745276417076
pl0.47368421052631576
es0.46504992867332384
sv0.4509516837481699
yo0.44113263785394935
no0.44029850746268656
hr0.41905499612703334
de0.41222570532915365
ms0.3786246893123446
lv0.2618110236220473
lt0.2492492492492493
hu0.2260061919504644
az0.22440537745604963
vi0.19441460794844254
et0.17400881057268722
fi0.15540540540540537
tr0.15349887133182843
is0.09090909090909091
eu0.08536585365853659
reliabletrue
passestrue
2
name"Selvi"
dialogue"हे भगवान, यह太好敢了! मुझे नहीं पता था कि यहाँ ऐसी चीजें होती हैं।"
detectedLang"hi"
heavyLang"hi"
scores
hi0.8563218390804598
mr0.8468887800097992
zh0.2816091954022989
ja0.24242424242424243
reliabletrue
passestrue
3
name"Hunter"
dialogue"confused I'm sorry, I didn't catch that. Do you speak English?"
detectedLang"en"
heavyLang"en"
scores
en0.8000888494002666
tl0.5770676691729324
it0.469964664310954
pt0.44717444717444715
hr0.4238156209987196
cs0.412532637075718
sv0.4007989347536618
pl0.37413073713490963
ro0.3715083798882682
sq0.3580599144079886
ca0.34593023255813954
no0.3353028064992614
et0.32228915662650603
da0.31921331316187596
sk0.3119266055045872
fr0.30448222565687794
vi0.27884615384615385
yo0.2682926829268293
de0.24623115577889448
sl0.21052631578947367
es0.18772563176895307
ms0.13793103448275862
nl0.1362763915547025
be0.1346153846153846
tr0.1346153846153846
hu0.12959381044487428
az0.12280701754385964
is0.12280701754385964
lt0.12280701754385964
lv0.12280701754385964
uk0.12280701754385964
reliabletrue
passestrue
4
name"Selvi"
dialogue"नहीं, माफ कीजिये। मैं सिर्फ हिंदी जानती हूँ।"
detectedLang"hi"
heavyLang"hi"
scores
hi0.8928571428571429
mr0.8877245508982036
reliabletrue
passestrue
5
name"Hunter"
dialogue"Hmm, this might be a problem. Let me get someone who can translate."
detectedLang"en"
heavyLang"en"
scores
en0.81044650379107
no0.5317377731529657
tl0.5192307692307693
sl0.5161290322580645
es0.5087336244541485
pt0.5044052863436124
da0.49324324324324326
fr0.49095022624434387
it0.48156682027649766
hr0.47368421052631576
ro0.4636471990464839
cs0.46107784431137727
ca0.4444444444444444
de0.4444444444444444
sq0.4437577255871446
az0.4409937888198758
hu0.42748091603053434
lt0.4238156209987196
pl0.42084942084942084
yo0.402390438247012
nl0.3935309973045822
et0.3910690121786198
sv0.3877551020408163
tr0.3835616438356164
sk0.375
lv0.3202416918429003
ms0.31402439024390244
is0.27999999999999997
vi0.2598684210526316
fi0.23469387755102045
eu0.23339011925042585
reliabletrue
passestrue
6
name"Selvi"
dialogue"मुझे खेद है अगर मैं किसी भी तरह से मदद नहीं कर सकती। मुझे उम्मीद है कि आप इसे सुलझा लेंगे।"
detectedLang"hi"
heavyLang"hi"
scores
hi0.8790896159317212
mr0.8715818099410787
reliabletrue
passestrue
7
name"Hunter"
dialogue"I'm sorry, I have to take this. Thank you for your time."
detectedLang"en"
heavyLang"en"
scores
en0.8258706467661693
tl0.6916299559471366
da0.6406570841889117
no0.5818399044205496
hr0.49201741654571846
cs0.4728915662650602
pt0.43548387096774194
pl0.42716857610474634
yo0.3902439024390244
it0.3805309734513274
vi0.37275985663082434
sk0.36823104693140796
sl0.36247723132969034
lv0.34579439252336447
lt0.3230174081237911
ca0.2827868852459016
sv0.2827868852459016
sq0.2768595041322314
ro0.20454545454545453
ms0.1822429906542056
be0.16666666666666666
tr0.16666666666666666
hu0.1606714628297362
az0.15254237288135594
et0.15254237288135594
is0.15254237288135594
uk0.15254237288135594
reliabletrue
passestrue