Mara pronouns preserved (coreference test)

Test: Text Replacement

Avg. Score

84.6%

Scenarios

Overall Performance

Performance Score Distribution (Top 20)

Click a model name to view its detail page.

Rank ▲	Model	Score	Avg. Cost	Avg. Time	Stability
1	Mistral Small Creative	100.0%	$0.0002	3.2s	100%
2	Gemini 3.1 Flash Lite (Reasoning)	100.0%	$0.0011	2.0s	100%
3	Gemini 3.1 Flash Lite	100.0%	$0.0011	3.9s	100%
4	Gemini 2.5 Flash	100.0%	$0.0017	2.4s	100%
5	Grok 4 Fast	100.0%	$0.0008	6.0s	100%
6	Gemini 3 Flash (Preview)	100.0%	$0.0021	3.6s	100%
7	GPT-4.1 Mini	100.0%	$0.0012	7.1s	100%
8	Mistral Medium 3.1	100.0%	$0.0015	6.5s	100%
9	Gemma 4 26B	100.0%	$0.0003	13.5s	100%
10	Stealth: Healer Alpha	100.0%	$0.0000	14.7s	100%
11	Gemini 3.1 Flash Lite (Preview)	99.4%	$0.0011	1.9s	95%
12	Claude Haiku 4.5	100.0%	$0.0042	3.3s	100%
13	Llama 3.1 Nemotron 70B	100.0%	$0.0016	16.3s	100%
14	Gemini 3.5 Flash (Reasoning, Minimal)	100.0%	$0.0064	2.9s	100%
15	GPT-4.1	100.0%	$0.0061	4.7s	100%
16	GPT-4o, Aug. 6th (temp=0)	100.0%	$0.0076	2.8s	100%
17	GPT-4o, Aug. 6th (temp=1)	100.0%	$0.0076	3.2s	100%
18	Gemma 4 31B	100.0%	$0.0004	27.2s	100%
19	Hermes 3 405B	100.0%	$0.0014	25.1s	100%
20	DeepSeek V4 Flash (Reasoning)	100.0%	$0.0005	28.0s	100%
21	DeepSeek V4 Pro	100.0%	$0.0019	25.2s	100%
22	Qwen 3.5 Plus (2026-02-15)	98.7%	$0.0017	7.7s	91%
23	Grok 4.1 Fast	98.7%	$0.0009	10.9s	91%
24	Qwen 3 32B	100.0%	$0.0006	33.7s	100%
25	Grok 4.20 (Reasoning)	100.0%	$0.0060	18.4s	100%
26	GPT-5.4	100.0%	$0.010	6.1s	100%
27	Stealth: Hunter Alpha	98.7%	$0.0000	17.4s	91%
28	Gemini 3 Flash (Preview, Reasoning)	100.0%	$0.0083	13.3s	100%
29	GPT-5.4 (Reasoning, Low)	100.0%	$0.011	6.7s	100%
30	GPT-4o, May 13th (temp=1)	100.0%	$0.012	4.0s	100%
31	GPT-4o, May 13th (temp=0)	100.0%	$0.012	4.1s	100%
32	Claude Sonnet 4.6	100.0%	$0.013	5.1s	100%
33	Claude Sonnet 4.5	100.0%	$0.013	5.2s	100%
34	Claude 3.7 Sonnet	100.0%	$0.013	6.4s	100%
35	Claude Sonnet 4	100.0%	$0.013	6.5s	100%
36	Z.AI GLM 4.5	100.0%	$0.0041	32.4s	100%
37	Grok 4.20 (Beta, Reasoning)	100.0%	$0.013	6.4s	100%
38	Qwen 3.6 Flash	100.0%	$0.0075	22.4s	100%
39	Qwen 3.6 35B	100.0%	$0.0053	30.7s	100%
40	Gemini 2.5 Flash (Reasoning)	98.7%	$0.0059	8.8s	91%
41	ByteDance Seed 1.6	100.0%	$0.0035	37.7s	100%
42	Grok 4.3 (Reasoning)	100.0%	$0.0076	32.7s	100%
43	Aion 2.0	100.0%	$0.0038	47.6s	100%
44	o4 Mini	100.0%	$0.014	21.5s	100%
45	ByteDance Seed 2.0 Lite	100.0%	$0.0045	52.2s	100%
46	GPT-5.1	100.0%	$0.017	16.5s	100%
47	GPT-5.5	100.0%	$0.021	5.3s	100%
48	Claude Opus 4.5	100.0%	$0.021	5.8s	100%
49	Claude Opus 4.6	100.0%	$0.021	6.5s	100%
50	Gemma 4 31B (Reasoning)	100.0%	$0.0010	1.2m	100%
51	Z.AI GLM 5	100.0%	$0.0074	54.4s	100%
52	Z.AI GLM 4.7	100.0%	$0.0059	1.0m	100%
53	GPT-5.5 (Reasoning, Low)	100.0%	$0.025	6.6s	100%
54	Qwen 3.5 27B	100.0%	$0.012	49.5s	100%
55	Claude 3.5 Sonnet	100.0%	$0.025	10.7s	100%
56	MiniMax M2.7	100.0%	$0.0065	1.1m	100%
57	Claude Opus 4.7	100.0%	$0.028	5.1s	100%
58	Inception Mercury 2	87.0%	$0.0019	2.6s	68%
59	Gemma 3 27B	89.0%	$0.0002	18.1s	72%
60	Z.AI GLM 4.6	98.7%	$0.0050	1.0m	91%
61	Gemini 3.5 Flash (Reasoning)	100.0%	$0.029	12.0s	100%
62	o4 Mini High	100.0%	$0.022	33.0s	100%
63	GPT-5.5 (Reasoning)	100.0%	$0.031	10.1s	100%
64	Qwen 3.6 27B	100.0%	$0.014	1.0m	100%
65	Gemini 2.5 Pro	100.0%	$0.028	20.4s	100%
66	Qwen 3.5 122B	100.0%	$0.020	46.9s	100%
67	Z.AI GLM 4.5 Air	92.2%	$0.0019	44.5s	78%
68	Grok 4	100.0%	$0.025	30.9s	100%
69	GPT-OSS 120B	90.9%	$0.0011	40.9s	74%
70	Z.AI GLM 5.1	100.0%	$0.013	1.2m	100%
71	Gemini 3 Pro (Preview)	100.0%	$0.031	21.3s	100%
72	DeepSeek V4 Pro (Reasoning)	100.0%	$0.0080	1.7m	100%
73	MoonshotAI: Kimi K2.6	100.0%	$0.015	1.5m	100%
74	ByteDance Seed 1.6 Flash	90.3%	$0.0007	12.3s	47%
75	Claude Sonnet 4.6 (Reasoning)	100.0%	$0.038	20.1s	100%
76	MoonshotAI: Kimi K2.5	100.0%	$0.0085	1.8m	100%
77	Ministral 3 3B	80.5%	$0.0001	2.2s	50%
78	Claude Opus 4.7 (Reasoning)	100.0%	$0.043	8.1s	100%
79	Gemma 4 26B (Reasoning)	100.0%	$0.0017	2.2m	100%
80	Xiaomi MIMO v2.5	92.9%	$0.0043	17.1s	48%
81	GPT-5.4 Nano (Reasoning, Low)	86.4%	$0.0009	8.4s	44%
82	Qwen3 235B A22B Instruct 2507	81.8%	$0.0005	16.2s	52%
83	Claude Opus 4.6 (Reasoning)	100.0%	$0.043	16.8s	100%
84	Grok 4.20	88.3%	$0.0022	4.8s	41%
85	GPT-5.2	92.9%	$0.010	7.1s	48%
86	Llama 3.1 70B	92.9%	$0.0005	36.2s	48%
87	Arcee AI: Trinity Mini	63.6%	$0.0002	7.1s	64%
88	Gemini 3.1 Pro (Preview)	100.0%	$0.038	36.2s	100%
89	Qwen3.7 Max	100.0%	$0.032	54.8s	100%
90	GPT-5	100.0%	$0.035	49.7s	100%
91	Z.AI GLM 5 Turbo	92.9%	$0.0095	19.7s	48%
92	Claude 3 Haiku	78.6%	$0.0010	5.0s	42%
93	Ministral 3B	77.3%	$0.0001	2.3s	41%
94	GPT-5.4 Nano	81.8%	$0.0009	3.5s	37%
95	Gemma 3 4B	77.3%	$0.0001	6.5s	42%
96	MiniMax M2.5	95.5%	$0.0017	1.6m	72%
97	GPT-5 Mini	92.2%	$0.0057	32.9s	49%
98	Inception Mercury	69.5%	$0.0005	5.2s	41%
99	Xiaomi MIMO v2.5 Pro	85.7%	$0.0029	13.0s	30%
100	Mistral Small 4 (Reasoning)	84.4%	$0.0019	16.2s	30%
101	GPT-5.4 (Reasoning)	92.9%	$0.020	16.0s	48%
102	GPT-5.4 Mini (Reasoning)	85.7%	$0.0071	6.2s	30%
103	Writer: Palmyra X5	77.3%	$0.0040	13.0s	37%
104	Claude Opus 4	100.0%	$0.063	8.9s	100%
105	Qwen3.6 Max Preview	100.0%	$0.033	1.7m	100%
106	Cydonia 24B V4.1	73.4%	$0.0005	13.6s	27%
107	Qwen 2.5 72B	78.6%	$0.0003	11.5s	18%
108	Grok 4.20 (Beta)	75.3%	$0.0038	2.0s	21%
109	GPT-5.4 Nano (Reasoning)	68.8%	$0.0010	3.7s	20%
110	Qwen 3.5 Plus (2026-04-20)	92.9%	$0.012	1.2m	48%
111	Nemotron 3 Super	82.5%	$0.0000	57.7s	31%
112	GPT-5.4 Mini (Reasoning, Low)	72.1%	$0.0036	2.9s	19%
113	GPT-5.4 Mini	55.8%	$0.0031	2.3s	27%
114	Mistral Small 4	61.7%	$0.0004	3.4s	15%
115	Qwen 3.5 9B	89.6%	$0.0014	1.9m	45%
116	DeepSeek-V2 Chat	71.4%	$0.0009	17.7s	10%
117	Ministral 3 14B	61.0%	$0.0003	4.5s	13%
118	Mistral Large	71.4%	$0.0050	8.4s	10%
119	DeepSeek V3 (2025-03-24)	68.8%	$0.0007	26.6s	11%
120	GPT-5 Nano	74.0%	$0.0029	1.0m	24%
121	Mistral Small 3.2 24B	56.5%	$0.0002	5.5s	6%
122	Qwen 3.5 35B	80.5%	$0.016	52.7s	30%
123	DeepSeek V3 (2024-12-26)	64.3%	$0.0009	18.3s	4%
124	Ministral 3 8B	52.6%	$0.0002	3.7s	3%
125	Qwen 3.5 Flash	71.4%	$0.0033	1.0m	17%
126	Skyfall 36B V2	48.1%	$0.0008	10.8s	12%
127	Llama 3.1 8B	54.5%	$0.0001	10.1s	3%
128	Ministral 8B	50.6%	$0.0001	3.9s	1%
129	Gemini 2.5 Flash Lite	50.0%	$0.0003	1.9s	0%
130	Mistral NeMO	50.0%	$0.0002	2.7s	0%
131	Rocinante 12B	50.0%	$0.0004	9.0s	3%
132	DeepSeek V4 Flash	50.0%	$0.0002	8.7s	0%
133	Arcee AI: Trinity Large (Preview)	54.5%	$0.0000	27.1s	3%
134	Mistral Large 3	50.0%	$0.0012	8.4s	0%
135	Grok 4.3	50.0%	$0.0024	5.9s	0%
136	Z.AI GLM 4.7 Flash	63.0%	$0.0016	1.1m	15%
137	Mistral Large 2	50.0%	$0.0050	8.4s	0%
138	Qwen 3.5 397B A17B	85.7%	$0.0075	2.2m	30%
139	GPT-4.1 Nano	40.3%	$0.0003	4.2s	0%
140	WizardLM 2 8x22b	57.1%	$0.0008	40.0s	1%
141	GPT-4o Mini (temp=0)	31.8%	$0.0005	10.8s	12%
142	GPT-4o Mini (temp=1)	31.8%	$0.0005	11.0s	12%
143	Gemini 2.5 Flash Lite (Reasoning)	44.2%	$0.0023	15.5s	2%
144	Gemma 3 12B	29.2%	$0.0001	9.6s	8%
145	DeepSeek V3.2	50.0%	$0.0006	38.9s	0%
146	DeepSeek V3.1	46.1%	$0.0006	43.9s	1%
147	Hermes 3 70B	35.7%	$0.0004	21.2s	0%
148	Cohere Command R+ (Aug. 2024)	50.0%	$0.0079	32.6s	0%
149	LFM2 24B	6.5%	$0.0001	14.7s	0%
150	ByteDance Seed 2.0 Mini	50.0%	$0.0020	2.0m	0%
151	Nemotron 3 Nano	57.8%	$0.0022	3.2m	17%
84.58%

Individual Scenarios

Generic Prompt

▼

Multi-character gender swap: Priya(F)->Rohan(M), Mara unchanged

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100.0%
Grok 4.1 Fast	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.6	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100.0%
Stealth: Hunter Alpha	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	100	100	100.0%
GPT-OSS 120B	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100.0%
Qwen 3 32B	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100.0%
Mistral Small Creative	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	82	97.4%
Z.AI GLM 4.5 Air	100	100	100	100	100	100	82	97.4%
Inception Mercury 2	100	100	100	100	82	82	82	92.2%
MiniMax M2.5	100	100	100	100	100	91	45	90.9%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	0	85.7%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	0	85.7%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	0	85.7%
GPT-5.2	100	100	100	100	100	100	0	85.7%
Qwen 3.5 9B	100	100	100	100	100	100	0	85.7%
Xiaomi MIMO v2.5	100	100	100	100	100	100	0	85.7%
Llama 3.1 70B	100	100	100	100	100	100	0	85.7%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	0	85.7%
GPT-5 Mini	100	100	100	100	100	91	0	84.4%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	45	0	77.9%
Gemma 3 27B	91	91	73	73	73	73	73	77.9%
Grok 4.20	100	100	100	100	100	36	0	76.6%
Inception Mercury	100	82	82	82	82	73	27	75.3%
Qwen 3.5 397B A17B	100	100	100	100	100	0	0	71.4%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	0	0	71.4%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	0	0	71.4%
Mistral Small 4 (Reasoning)	100	100	100	100	82	0	0	68.8%
Nemotron 3 Super	100	100	91	82	82	0	0	64.9%
Qwen3 235B A22B Instruct 2507	64	64	64	64	64	64	64	63.6%
GPT-5.4 Nano	100	100	100	73	36	27	9	63.6%
Arcee AI: Trinity Mini	64	64	64	64	64	64	64	63.6%
Qwen 3.5 35B	100	100	100	64	64	0	0	61.0%
Ministral 3 3B	64	64	64	64	64	55	55	61.0%
GPT-5 Nano	100	100	100	91	9	9	0	58.4%
Ministral 3B	64	64	64	55	55	55	55	58.4%
GPT-5.4 Mini (Reasoning, Low)	100	100	100	100	0	0	0	57.1%
Qwen 2.5 72B	100	100	100	100	0	0	0	57.1%
Claude 3 Haiku	64	64	64	64	64	64	18	57.1%
Nemotron 3 Nano	100	100	82	55	45	0	0	54.5%
Writer: Palmyra X5	64	64	64	64	64	64	0	54.5%
Gemma 3 4B	55	55	55	55	55	55	55	54.5%
Grok 4.20 (Beta)	100	100	100	27	27	0	0	50.6%
GPT-5.4 Nano (Reasoning)	100	100	73	55	0	0	0	46.8%
Cydonia 24B V4.1	100	91	64	27	27	18	0	46.8%
Qwen 3.5 Flash	100	100	64	36	0	0	0	42.9%
DeepSeek-V2 Chat	100	100	100	0	0	0	0	42.9%
Mistral Large	100	100	100	0	0	0	0	42.9%
Z.AI GLM 4.7 Flash	100	100	73	0	0	0	0	39.0%
DeepSeek V3 (2025-03-24)	100	100	64	0	0	0	0	37.7%
GPT-5.4 Mini	36	36	36	36	36	27	0	29.9%
DeepSeek V3 (2024-12-26)	100	100	0	0	0	0	0	28.6%
Hermes 3 70B	100	100	0	0	0	0	0	28.6%
Mistral Small 3.2 24B	27	27	27	27	27	27	27	27.3%
Mistral Small 4	45	45	18	18	18	18	0	23.4%
Ministral 3 14B	55	55	18	18	9	0	0	22.1%
Skyfall 36B V2	36	27	27	27	18	0	0	19.5%
Rocinante 12B	100	27	0	0	0	0	0	18.2%
Gemini 2.5 Flash Lite (Reasoning)	100	0	0	0	0	0	0	14.3%
WizardLM 2 8x22b	100	0	0	0	0	0	0	14.3%
Arcee AI: Trinity Large (Preview)	64	0	0	0	0	0	0	9.1%
Llama 3.1 8B	64	0	0	0	0	0	0	9.1%
Ministral 3 8B	18	18	0	0	0	0	0	5.2%
Ministral 8B	9	0	0	0	0	0	0	1.3%
ByteDance Seed 2.0 Mini	0	0	0	0	0	0	0	0.0%
Mistral Large 3	0	0	0	0	0	0	0	0.0%
Mistral Large 2	0	0	0	0	0	0	0	0.0%
DeepSeek V3.1	0	0	0	0	0	0	0	0.0%
DeepSeek V3.2	0	0	0	0	0	0	0	0.0%
DeepSeek V4 Flash	0	0	0	0	0	0	0	0.0%
Gemini 2.5 Flash Lite	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=1)	0	0	0	0	0	0	0	0.0%
Grok 4.3	0	0	0	0	0	0	0	0.0%
Gemma 3 12B	0	0	0	0	0	0	0	0.0%
GPT-4o Mini (temp=0)	0	0	0	0	0	0	0	0.0%
GPT-4.1 Nano	0	0	0	0	0	0	0	0.0%
Cohere Command R+ (Aug. 2024)	0	0	0	0	0	0	0	0.0%
Mistral NeMO	0	0	0	0	0	0	0	0.0%
LFM2 24B	0	0	0	0	0	0	0	0.0%

Specific Prompt

▼

Multi-character gender swap: Priya(F)->Rohan(M), Mara unchanged

Model	# 1	# 2	# 3	# 4	# 5	# 6	# 7	Avg ▼
Qwen3.7 Max	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6 (Reasoning)	100	100	100	100	100	100	100	100.0%
Qwen3.6 Max Preview	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Pro (Preview)	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5.1	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5 Turbo	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6 (Reasoning)	100	100	100	100	100	100	100	100.0%
Grok 4.3 (Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7 (Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5 Mini	100	100	100	100	100	100	100	100.0%
GPT-5.5 (Reasoning, Low)	100	100	100	100	100	100	100	100.0%
GPT-5.1	100	100	100	100	100	100	100	100.0%
Claude Opus 4.6	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.6	100	100	100	100	100	100	100	100.0%
GPT-5	100	100	100	100	100	100	100	100.0%
Qwen 3.5 397B A17B	100	100	100	100	100	100	100	100.0%
Gemma 4 31B (Reasoning)	100	100	100	100	100	100	100	100.0%
Qwen 3.5 122B	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Plus (2026-04-20)	100	100	100	100	100	100	100	100.0%
Gemma 4 26B (Reasoning)	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta, Reasoning)	100	100	100	100	100	100	100	100.0%
GPT-5.4 (Reasoning, Low)	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Reasoning)	100	100	100	100	100	100	100	100.0%
Z.AI GLM 5	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.6	100	100	100	100	100	100	100	100.0%
MoonshotAI: Kimi K2.5	100	100	100	100	100	100	100	100.0%
Qwen 3.5 27B	100	100	100	100	100	100	100	100.0%
ByteDance Seed 1.6	100	100	100	100	100	100	100	100.0%
Qwen 3.6 Flash	100	100	100	100	100	100	100	100.0%
GPT-5.4 Mini (Reasoning)	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview, Reasoning)	100	100	100	100	100	100	100	100.0%
o4 Mini High	100	100	100	100	100	100	100	100.0%
GPT-5.2	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro (Reasoning)	100	100	100	100	100	100	100	100.0%
Claude Opus 4.7	100	100	100	100	100	100	100	100.0%
Qwen 3.6 27B	100	100	100	100	100	100	100	100.0%
Claude Opus 4.5	100	100	100	100	100	100	100	100.0%
Aion 2.0	100	100	100	100	100	100	100	100.0%
MiniMax M2.7	100	100	100	100	100	100	100	100.0%
GPT-5.5	100	100	100	100	100	100	100	100.0%
Qwen 3.6 35B	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Flash (Reasoning)	100	100	100	100	100	100	100	100.0%
Gemini 3 Pro (Preview)	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4	100	100	100	100	100	100	100	100.0%
MiniMax M2.5	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.7	100	100	100	100	100	100	100	100.0%
GPT-4.1	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Pro	100	100	100	100	100	100	100	100.0%
o4 Mini	100	100	100	100	100	100	100	100.0%
Grok 4	100	100	100	100	100	100	100	100.0%
Claude Sonnet 4.5	100	100	100	100	100	100	100	100.0%
Qwen 3.5 35B	100	100	100	100	100	100	100	100.0%
Claude Opus 4	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5 Pro	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Mini	100	100	100	100	100	100	100	100.0%
Gemma 4 31B	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash (Reasoning)	100	100	100	100	100	100	100	100.0%
Gemini 3.5 Flash (Reasoning, Minimal)	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Reasoning)	100	100	100	100	100	100	100	100.0%
Qwen 3.5 Flash	100	100	100	100	100	100	100	100.0%
Z.AI GLM 4.5	100	100	100	100	100	100	100	100.0%
Grok 4 Fast	100	100	100	100	100	100	100	100.0%
Stealth: Healer Alpha	100	100	100	100	100	100	100	100.0%
Gemma 4 26B	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite	100	100	100	100	100	100	100	100.0%
Mistral Large 3	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=0)	100	100	100	100	100	100	100	100.0%
Gemini 3 Flash (Preview)	100	100	100	100	100	100	100	100.0%
Claude Haiku 4.5	100	100	100	100	100	100	100	100.0%
Xiaomi MIMO v2.5	100	100	100	100	100	100	100	100.0%
DeepSeek-V2 Chat	100	100	100	100	100	100	100	100.0%
ByteDance Seed 2.0 Lite	100	100	100	100	100	100	100	100.0%
Nemotron 3 Super	100	100	100	100	100	100	100	100.0%
GPT-5.4	100	100	100	100	100	100	100	100.0%
Claude 3.5 Sonnet	100	100	100	100	100	100	100	100.0%
Grok 4.20 (Beta)	100	100	100	100	100	100	100	100.0%
GPT-4o, May 13th (temp=1)	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2024-12-26)	100	100	100	100	100	100	100	100.0%
Claude 3.7 Sonnet	100	100	100	100	100	100	100	100.0%
GPT-4.1 Mini	100	100	100	100	100	100	100	100.0%
Hermes 3 405B	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Pro	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=1)	100	100	100	100	100	100	100	100.0%
GPT-4o, Aug. 6th (temp=0)	100	100	100	100	100	100	100	100.0%
Mistral Large 2	100	100	100	100	100	100	100	100.0%
Mistral Small 4 (Reasoning)	100	100	100	100	100	100	100	100.0%
DeepSeek V3.2	100	100	100	100	100	100	100	100.0%
Qwen 3 32B	100	100	100	100	100	100	100	100.0%
DeepSeek V4 Flash	100	100	100	100	100	100	100	100.0%
DeepSeek V3 (2025-03-24)	100	100	100	100	100	100	100	100.0%
Grok 4.20	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash Lite	100	100	100	100	100	100	100	100.0%
Gemini 2.5 Flash	100	100	100	100	100	100	100	100.0%
Mistral Large	100	100	100	100	100	100	100	100.0%
Qwen3 235B A22B Instruct 2507	100	100	100	100	100	100	100	100.0%
Writer: Palmyra X5	100	100	100	100	100	100	100	100.0%
Grok 4.3	100	100	100	100	100	100	100	100.0%
Llama 3.1 70B	100	100	100	100	100	100	100	100.0%
Gemma 3 27B	100	100	100	100	100	100	100	100.0%
Mistral Medium 3.1	100	100	100	100	100	100	100	100.0%
Mistral Small 4	100	100	100	100	100	100	100	100.0%
Qwen 2.5 72B	100	100	100	100	100	100	100	100.0%
Cydonia 24B V4.1	100	100	100	100	100	100	100	100.0%
Llama 3.1 Nemotron 70B	100	100	100	100	100	100	100	100.0%
GPT-5.4 Nano	100	100	100	100	100	100	100	100.0%
Arcee AI: Trinity Large (Preview)	100	100	100	100	100	100	100	100.0%
Mistral Small Creative	100	100	100	100	100	100	100	100.0%
Ministral 3 14B	100	100	100	100	100	100	100	100.0%
Ministral 3 8B	100	100	100	100	100	100	100	100.0%
Claude 3 Haiku	100	100	100	100	100	100	100	100.0%
WizardLM 2 8x22b	100	100	100	100	100	100	100	100.0%
Cohere Command R+ (Aug. 2024)	100	100	100	100	100	100	100	100.0%
Gemma 3 4B	100	100	100	100	100	100	100	100.0%
Ministral 3 3B	100	100	100	100	100	100	100	100.0%
Mistral NeMO	100	100	100	100	100	100	100	100.0%
Ministral 8B	100	100	100	100	100	100	100	100.0%
Llama 3.1 8B	100	100	100	100	100	100	100	100.0%
Gemini 3.1 Flash Lite (Preview)	100	100	100	100	100	100	91	98.7%
Grok 4.1 Fast	100	100	100	100	100	100	82	97.4%
Z.AI GLM 4.6	100	100	100	100	100	100	82	97.4%
Stealth: Hunter Alpha	100	100	100	100	100	100	82	97.4%
Qwen 3.5 Plus (2026-02-15)	100	100	100	100	100	100	82	97.4%
Ministral 3B	100	100	100	100	100	100	73	96.1%
GPT-5.4 Nano (Reasoning, Low)	100	100	100	100	100	82	82	94.8%
ByteDance Seed 1.6 Flash	100	100	100	100	100	100	64	94.8%
Qwen 3.5 9B	100	100	100	100	100	100	55	93.5%
DeepSeek V3.1	100	100	100	100	100	91	55	92.2%
GPT-5.4 Nano (Reasoning)	100	100	100	91	82	82	82	90.9%
GPT-5 Nano	100	100	100	82	82	82	82	89.6%
Z.AI GLM 4.5 Air	100	100	100	82	82	73	73	87.0%
GPT-5.4 Mini (Reasoning, Low)	100	100	82	82	82	82	82	87.0%
Z.AI GLM 4.7 Flash	100	91	91	82	82	82	82	87.0%
Mistral Small 3.2 24B	100	100	100	100	100	100	0	85.7%
GPT-OSS 120B	82	82	82	82	82	82	82	81.8%
Inception Mercury 2	82	82	82	82	82	82	82	81.8%
GPT-5.4 Mini	82	82	82	82	82	82	82	81.8%
Rocinante 12B	100	100	100	100	91	82	0	81.8%
GPT-4.1 Nano	100	100	100	100	82	82	0	80.5%
Skyfall 36B V2	100	100	100	100	45	45	45	76.6%
Gemini 2.5 Flash Lite (Reasoning)	100	100	100	100	73	45	0	74.0%
Inception Mercury	100	82	82	73	55	36	18	63.6%
GPT-4o Mini (temp=1)	64	64	64	64	64	64	64	63.6%
GPT-4o Mini (temp=0)	64	64	64	64	64	64	64	63.6%
Arcee AI: Trinity Mini	64	64	64	64	64	64	64	63.6%
Nemotron 3 Nano	100	91	82	64	45	45	0	61.0%
Gemma 3 12B	100	55	55	55	55	45	45	58.4%
Hermes 3 70B	100	100	100	0	0	0	0	42.9%
LFM2 24B	18	18	18	18	9	9	0	13.0%

Mara pronouns preserved (coreference test)

Overall Performance

Performance Score Distribution (Top 20)

Price-Performance Score Distribution (Top 20)

Most Stable Models (Top 20)

Top Overall Models (Top 20)

Individual Scenarios

Generic Prompt

Multi-character gender swap: Priya(F)->Rohan(M), Mara unchanged

Specific Prompt

Multi-character gender swap: Priya(F)->Rohan(M), Mara unchanged