A comprehensive guide to the psychological concepts, dashboard visualizations, and exact mathematical formulas used to evaluate AI models.
Instead of binary win/loss outcomes, this benchmark uses Shannon Entropy and Information Theory. Every guess a model makes is evaluated mathematically on its Information Gain (IG)—how effectively it narrows down the remaining possible word space.
A guess that eliminates 90% of the remaining search space scores much higher than a redundant guess, providing a continuous, high-fidelity metric of logical reasoning. Conversely, Rule Violations (e.g., using a known gray letter) directly penalize the score, seamlessly merging constraint satisfaction with probabilistic deduction.
The benchmark replaces binary win/loss metrics with a nuanced, continuous scoring system rooted in Information Theory. The total score for any turn or game is a composite of three metrics:
Calculates the Shannon Entropy reduction of the remaining word pool. A perfectly optimizing guess that maximally splits the remaining candidates yields an IG of 1.0.
A binary bonus of +1.0 awarded exactly once when the model successfully identifies the final secret word.
If a model violates an established rule (e.g., guessing a word with a known 'gray' letter) and falsely claims it followed all constraints, a severe flat penalty of -0.3 is applied.
Because the formula is Information Gain + Success Bonus + Penalty, the absolute theoretical maximum score for any benchmark task is 2.0. However, scoring approaches vary by task:
,Y). This move protects the model from the -0.3 penalty but forfeits the Information Gain (0.0) for that turn. Consequently, scores on this task rarely approach 2.0, as models strategically sacrifice immediate points to secure the win.The dashboard utilizes AI performance data extracted from the rigorous Kaggle benchmarking environment. Frontier language models are evaluated against adversarial Wordle datasets and strictly scored based on information-theoretic metrics.
For detailed task construction, datasets, and competition rules, refer to the official Kaggle track:
Wordle InfoGain Benchmark on Kaggle
Executive Function (EF) is recognized as one of the critical AGI (Artificial General Intelligence) skills by institutions such as Google DeepMind. Derived directly from human cognitive psychology, Executive Function encompasses the higher-order processes necessary for the dynamic control of behavior.
Rather than relying purely on pre-trained factual knowledge, EF dictates a model's ability to plan, focus attention, adapt to novel situations, and regulate outputs under pressure. It constitutes three core components: Working Memory, Cognitive Flexibility, and Inhibitory Control.
A common question is: Why use a Wordle game instead of established benchmarks like BBH (BIG-bench Hard)?
While traditional benchmarks like BBH are excellent at measuring static reasoning abilities (e.g., solving a logic puzzle in a single prompt), research shows that raw reasoning is not the same as Executive Function. EF is highly dynamic. It requires state-tracking, impulse suppression, and real-time adaptation—necessitating a completely separate benchmark framework.
What it is: The cognitive system responsible for temporarily holding and manipulating information. In LLMs, this translates to context retention over extended interactions. It is important to note that the Wordle game in its entirety perfectly captures a model's working memory capability, as merely playing the game successfully requires maintaining a continuous, complex state.
How we measure it: While the overall game is already a holistic test of working memory, we try to quantify this specific component directly based on wordle game performance. We achieve this by defining an isolated proxy based on the game's outputs. By tracking multi-turn rule violations (e.g., forgetting a known "dead letter" on guess #4), we create a rigorous, mathematical proxy that explicitly measures context degradation and memory drift over time. The beauty of using the Wordle game as a benchmark is that it has tremendous potential for analyzing EF skills. By reconstructing datasets, changing scoring metrics, and evaluating different outputs, we can develop more complex and robust metrics to further isolate these critical AGI skills.
What it is: The mental ability to switch seamlessly between different concepts or adapt strategies when the environment suddenly changes.
How we measure it: Tested via the Cognitive Flexibility Task. Mid-evaluation, foundational rules (like dictionaries or logic constraints) are dynamically shifted. Flexible models adapt instantly to the new paradigm, while rigid models fail.
What it is: The ability to suppress impulsive, dominant responses in favor of deliberate, rule-aligned behavior, especially under stress.
How we measure it: Measured by observing the spike in rule violations when models transition from standard gameplay to the chaotic Cognitive Flexibility task. Emotionally/structurally stable models suppress the impulse to guess blindly and maintain composure.
A guide to the specific components displayed on the benchmark dashboard, in order of appearance:
An adjustable visualization that ranks models based on a weighted synthesis of the three cognitive pillars. Users can dynamically adjust the weights to see how different evaluation priorities reshape the leaderboard.
Tracks the rule violation rate between isolated single-turn states and continuous multi-turn states to highlight which models suffer from severe context drift over prolonged interactions. While some models achieve zero violations in simple states, future iterations of this benchmark will introduce higher-complexity adversarial constraints to establish an even more granular baseline for memory degradation.
A raw tabular breakdown of the three core psychological traits. It isolates the specific scores for Drift, Flexibility, and Inhibition used to calculate the overall EF index.
Maps a model's cognitive performance against its resource verbosity. Identifies models that are "strategically efficient" versus those that rely on brute-force token generation to achieve high scores.
Measures how strictly a model obeys the mathematical boundaries of the task. Tracks explicit logic violations, such as ignoring positional clues or reusing confirmed dead letters.
Evaluates the operational footprint and hardware cost-effectiveness of the models, tracking financial cost, velocity (speed), and output conciseness.
Drift = MT_Violations - ST_Violations
Note: Currently, Single-Turn (ST) violations are virtually zero across models due to strict instruction following in a constrained prompt. This effectively makes the MT violations themselves the baseline drift score. Future datasets and system prompts can be engineered to increase single-turn difficulty, providing a more dynamic baseline.
Inhibition Failure = MAX(0, CF_Violations - MT_Violations)
Score / Cost ($)
Score / Time (Minutes)
(Score / Output_Tokens) * 10,000