A benchmark suite for evaluating Executive Function capabilities of AI models using Wordle game and Information Gain optimization
Evaluating executive function from a single Wordle decision.
15 models evaluated
View Leaderboard →Evaluating executive function and information gain across multi-turn gameplay.
17 models evaluated
View Leaderboard →Evaluating adaptive reasoning and cognitive flexibility.
15 models evaluated
View Leaderboard →The mechanics of the Wordle game—requiring strict state-tracking, rule adherence, and dynamic adaptation—make it an exceptional sandbox for measuring Executive Function (EF). The benchmark results yield several fundamental insights into the current landscape of AI capabilities:
While traditional benchmarks measure static logic, EF requires dynamic cognitive control. Dedicated reasoning models that excel on standard logic tests (e.g., DeepSeek, Qwen) unexpectedly struggle here, failing to crack the Top 5. This divergence proves that EF—specifically maintaining context and suppressing errors under constraints—is a distinct, higher-tier capability than pure reasoning.
Top honors across the EF pillars are dominated by massive frontier models (e.g., GPT-5.5, Claude 4.6 Opus). Their superior performance likely stems from highly resilient working memory capacities, allowing them to track complex states and adapt seamlessly to mid-task rule changes. The exact architectural features enabling this remain a vital area for research.
Parameter count isn't the sole dictator of EF capability. The strong placements of mid-weight architectures like Gemma-4-31b demonstrate that highly efficient design and targeted alignment can yield strong executive control. Further evaluation of similar lightweight models is needed to definitively map EF scaling laws.
Our efficiency metrics expose a critical flaw in models designed to generate massive reasoning traces (e.g., DeepSeek-R1, Qwen-Thinking). While they output thousands of reasoning tokens to solve a turn, they frequently land in the "Over-thinkers" quadrant—burning massive compute without achieving proportional Information Gain. This proves that excessive verbosity and "thinking" tokens cannot compensate for a fundamental lack of Executive Function and rule retention.
The stark performance delta between Grok 4.20’s reasoning and non-reasoning variants provides a critical counterpoint to skeptics who argue that reasoning models aren't genuinely "thinking." While the non-reasoning Grok 4.20 languishes in the bottom 5, its reasoning counterpart secured a Top 5 finish across all the EF tasks. This massive gap demonstrates that at the frontier scale, the reasoning mechanism is not just superficial token generation. Instead, it actively amplifies cognitive control and strict state-tracking—proving that when paired with a sufficiently powerful base model, the "thinking" process directly and tangibly enhances Executive Function.
In our benchmark, models were equipped with a highly detailed system prompt (akin to a skills.md document) that explicitly mapped out Executive Function strategies—such as strict state-tracking, constraint checking, and adaptive planning. Both frontier and reasoning architectures successfully utilized these pathways. However, in a parallel experiment using a simplified prompt devoid of this cognitive scaffolding, overall rule adherence and performance collapsed, with smaller models suffering the most severe EF degradation. This reveals that robust instruction following acts as a powerful external working memory. By offloading cognitive control and strategic analysis directly into the prompt structure, smaller models can punch well above their weight class, effectively simulating the executive capabilities of massive frontier models.
While Wordle game performance itself serves as a perfect proxy for the EF skills of AI models, it can be helpful to identify the performance of each model under the individual core components of EF skills, namely: working memory, inhibitory control, and cognitive flexibility. We tried to isolate the performance of each model under each of these components, using the result outputs of individual Kaggle task runs. We then created a composite score-based ranking by giving equal weights to each component. This gave us a different ranking of the models for EF skills. The order of ranking is not very much different from the main benchmark, but the change in the ranking order provide some critical insights for analysis. This divergence likely stems from our equal-weighting methodology; in actual dynamic gameplay, cognitive demands constantly shift, rarely relying on a perfectly equal mathematical balance of all three EF components at any given moment.
Adjust the weights to see how different evaluation priorities affect the leaderboard.
Evaluating Multi-Turn cognitive performance against API cost. The ideal model sits in the top-left (Value Champions).
Measures how much a model's rule adherence degrades when transitioning from a single decision to multi-turn gameplay. A lower score indicates better context stability.
| Rank | Model | Single-Turn Violations | Multi-Turn Violations | Degradation Score |
|---|---|---|---|---|
| 1 | gpt-5.5 | 0.000 | 0.000 | 0.000 (Stable) |
| 2 | qwen-3-next-80b-thinking | 0.000 | 0.000 | 0.000 (Stable) |
| 3 | gemini-3.1-pro-preview | 0.000 | 0.041 | +0.041 (Slight Drift) |
| 4 | claude-opus-4.6 | 0.000 | 0.077 | +0.077 (Slight Drift) |
| 5 | grok-4.20-reasoning | 0.000 | 0.154 | +0.154 (Severe Drift) |
| 6 | claude-sonnet-4.6 | 0.000 | 0.173 | +0.173 (Severe Drift) |
| 7 | gemini-3-flash-preview | 0.000 | 0.258 | +0.258 (Severe Drift) |
| 8 | qwen-3-next-80b-instruct | 0.000 | 0.260 | +0.260 (Severe Drift) |
| 9 | gpt-5.4 | 0.000 | 0.365 | +0.365 (Severe Drift) |
| 10 | deepseek-r1 | 0.000 | 0.381 | +0.381 (Severe Drift) |
| 11 | glm-5 | 0.000 | 0.490 | +0.490 (Severe Drift) |
| 12 | deepseek-v3.2 | 0.000 | 0.872 | +0.872 (Severe Drift) |
| 13 | gpt-5.4-mini | 0.000 | 1.731 | +1.731 (Severe Drift) |
| 14 | grok-4.20-non-reasoning | 0.000 | 2.923 | +2.923 (Severe Drift) |
Mapping benchmark outcomes directly to cognitive science pillars.
Working Memory: MT Violations − ST Violations | Cognitive Flex: Task 3 Score | Inhibitory Control: Task 3 Violations − MT Violations
| Rank | Model | Working Memory (Drift) | Cognitive Flexibility | Inhibitory Control | Overall EF Score (0-100) |
|---|---|---|---|---|---|
| 1 | claude-opus-4.6 | 0.077 | 0.8255 | 1.173 | 88.3 |
| 2 | claude-sonnet-4.6 | 0.173 | 0.8082 | 0.802 | 88.2 |
| 3 | gemini-3.1-pro-preview | 0.041 | 0.9217 | 2.117 | 87.7 |
| 4 | gpt-5.5 | 0.000 | 0.9966 | 2.875 | 87.4 |
| 5 | gemma-4-31b | 0.149 | 0.9419 | 2.082 | 87.3 |
| 6 | glm-5 | 0.490 | 0.7427 | 0.000 | 85.9 |
| 7 | gpt-5.4 | 0.365 | 0.6724 | 0.035 | 84.8 |
| 8 | grok-4.20-reasoning | 0.154 | 0.9677 | 2.846 | 84.8 |
| 9 | qwen-3-next-80b-thinking | 0.000 | 0.5425 | 0.031 | 84.7 |
| 10 | gemini-3-flash-preview | 0.258 | 0.7810 | 1.357 | 83.9 |
| 11 | deepseek-v3.2 | 0.872 | 0.7495 | 1.560 | 74.9 |
| 12 | gemma-4 | 0.000 | 0.0000 | 0.000 | 66.7 |
| 13 | qwen-3-next-80b-instruct | 0.260 | 0.0000 | 0.000 | 63.7 |
| 14 | deepseek-r1 | 0.381 | 0.0000 | 0.000 | 62.3 |
| 15 | qwen-3-235b-A22b-instruct | 0.540 | 0.0000 | 0.000 | 60.5 |
| 16 | gpt-5.4-mini | 1.731 | 0.3256 | 0.000 | 57.8 |
| 17 | gpt-5.4-nano | 1.481 | 0.0000 | 0.000 | 49.8 |
| 18 | gemma-3-12b | 0.000 | -0.1818 | 2.975 | 47.5 |
| 19 | grok-4.20-non-reasoning | 2.923 | 0.4284 | 0.077 | 47.3 |
| 20 | gemma-3-4b | 0.000 | -0.1693 | 7.600 | 27.7 |
Scores normalized 0-100 (where 100 represents cohort state-of-the-art).
Compares a model's ability to win against its Information Gain strategy. Placed relative to the cohort median.
Ranks models based on format adherence and API stability. Models that trigger the >20% error penalty are placed in the Penalty category.