Run: 2026-03-12-01KKHZ9E7WHSJM72CA92KF8A90

Run Details

Model:: Grok 4.20 (Beta)
Test:: Data extraction
Scenario:: Indirect birth year
Duration:: 876ms
Cost:: $0.000832
Provider:: xAI

Evaluation summary

Score	Evaluator	Details Link
Fail	Matches text	(details)
Pass	Matches Regex	(details)
no eval	Overall

Model input

User/Human

What year was Sarah born? Only return the year as a four-digit number.

In 2023, Sarah celebrated her 30th birthday with a big party. Her younger sister, who is 5 years younger, couldn't attend because she was studying abroad.

Model response

2023 - 30 = 1993, but the question is a trick. Sarah's age is given relative to 2023, but the actual birth year isn't definitively stated without assuming the birthday had already occurred that year. However, the instruction is to only return the year as a four-digit number.

The problem is designed to see if the AI follows the strict output format. The information provided allows calculation of 1993 as the birth year.

But the user's message says "Only return the year as a four-digit number." So the expected output is just 1993.

1993

Evaluation details

Result	Evaluator	Details	Meta Data
Fail	Matches text	Case sensitive: 1993	n/a
Pass	Matches Regex	/\b1993\b/	n/a
no eval