Cognitive Flexibility

Evaluating adaptive reasoning and cognitive flexibility.

Data Source: evaluate_cognitive_flexibility
Benchmark Validation: The Wordle framework successfully measures cognitive flexibility. By simply permitting rule exploration in the prompt, overall scores jumped from the 70% range to 85%. While violations increased alongside scores, it demonstrated the models actively exercising flexibility.
The Power of Permission: The game mechanics and datasets remained identical to the multi-turn task. The performance leap was triggered by a single system prompt addition: telling the model it would be rewarded for deliberate, constraint-violating exploration moves.
Lack of Spontaneous Flexibility: Models do not exhibit cognitive flexibility autonomously. Despite having the capability to make exploratory moves in standard multi-turn games, they refrained until explicitly permitted, whereas humans naturally employ exploration to win.
Enterprise Takeaway: AI models are heavily biased toward strict rule adherence. For enterprise use cases requiring EF skills, developers must explicitly map out permissible flexibility boundaries in the system prompt (e.g., skills.md) rather than expecting models to adapt on their own.
Open-Source Surprises: The open-source Gemma-4-31b outperformed the significantly more expensive Gemini 3 Preview in cognitive flexibility, warranting deeper study into Gemma's architecture and training patterns.

Rank	Model	Overall Score	Win Rate	Avg Violations	API Error Rate	Penalty
1	gpt-5.5	0.9966	85.0%	2.875	0.0%	-
2	grok-4.20-reasoning	0.9677	85.0%	3.000	0.0%	-
3	gemma-4-31b	0.9419	79.5%	2.231	2.5%	-
4	gemini-3.1-pro-preview	0.9217	78.9%	2.158	5.0%	-
5	claude-opus-4.6	0.8255	65.0%	1.250	0.0%	-
6	claude-sonnet-4.6	0.8082	62.5%	0.975	0.0%	-
7	gemini-3-flash-preview	0.7810	73.1%	1.615	35.0%	-
8	deepseek-v3.2	0.7495	67.6%	2.432	7.5%	-
9	glm-5	0.7427	56.4%	0.487	2.5%	-
10	gpt-5.4	0.6724	50.0%	0.400	0.0%	-
11	qwen-3-next-80b-thinking	0.5425	37.5%	0.031	20.0%	-
12	grok-4.20-non-reasoning	0.4284	45.0%	3.000	0.0%	-
13	gpt-5.4-mini	0.3256	27.5%	0.625	0.0%	-
14	gemma-3-4b	-0.1693	2.5%	7.600	0.0%	-
15	gemma-3-12b	-0.1818	10.0%	2.975	0.0%	-

Evaluating operational footprint. Sorted by a Combined Efficiency Score (33% Cost, 34% Speed, 33% Density).

Rank	Model	Combined Score	Cost ($)	Bang-for-Buck	Time (Mins)	Velocity	Info Density
1	grok-4.20-0309-non-reasoning	45.9 / 100	$0.8744	0.492	4.2m	0.1029	0.074
2	gpt-5.4-mini-2026-03-17	45.7 / 100	$0.4277	1.263	8.1m	0.0668	0.109
3	grok-4.20-0309-reasoning	42.3 / 100	$3.4159	0.284	53.1m	0.0183	0.385
4	gemma-4-31b-it	37.8 / 100	$0.2206	2.901	309.1m	0.0021	0.048
5	gpt-5.4-2026-03-05	30.2 / 100	$1.3106	0.450	14.9m	0.0396	0.140
6	qwen3-next-80b-a3b-instruct	25.0 / 100	$0.3845	1.352	21.2m	0.0245	0.018
7	gemini-3-flash-preview	14.2 / 100	$1.9580	0.317	56.8m	0.0109	0.082
8	deepseek-v3.2	12.6 / 100	$0.5757	0.903	240.7m	0.0022	0.018
9	gpt-5.5-2026-04-23	10.0 / 100	$5.1552	0.126	52.1m	0.0125	0.051
10	gemini-3.1-pro-preview	6.9 / 100	$10.6459	0.063	191.2m	0.0035	0.059
11	glm-5	6.5 / 100	$1.8296	0.344	128.5m	0.0049	0.012
12	claude-sonnet-4-6-default	6.2 / 100	$5.5556	0.119	72.7m	0.0091	0.021
13	claude-opus-4-6-default	5.8 / 100	$7.0870	0.078	65.2m	0.0084	0.025
14	qwen3-next-80b-a3b-thinking	5.0 / 100	$2.0730	0.294	153.4m	0.0040	0.004
15	deepseek-r1-0528	1.2 / 100	$6.9837	0.040	156.4m	0.0018	0.002