Single-turn Wordle

Task Insights

• Data Source: evaluate_wordle_single_turn_v2
• Metric Shift: Because single-turn win rates are inherently low, Information Gain proves to be a far more granular and robust metric for ranking models than standard win rates.
• Model Divergence: Claude Sonnet outperforms Claude Opus, with a noticeably wider performance gap here than in multi-turn scenarios.
• The Reasoning Imperative: A stark performance gap between Grok-reasoning and Grok-non-reasoning provides concrete proof that reasoning models are essential for Executive Function (EF) skills.
• Token Limits & Errors: DeepSeek and Qwen suffered lower rankings, likely because their highly verbose reasoning traces exceeded token limits, driving up error rates.
• Information Gain Validated: Qwen3-8b-instruct achieved the same win rate as the top-ranking Gemini 3.1 Pro but placed 12th overall due to a lower Information Gain score, validating the metric's precision.
• Trace Verbosity vs. Efficiency: Top rankers utilized extensive reasoning traces to score higher. Conversely, GPT models used fewer reasoning tokens, scoring lower on EF but dominating in speed, cost, and information density.
• Cost Realities: Factoring in cost shuffles the leaderboard significantly; Grok-reasoning drops to the bottom five, while Gemma-4 uniquely retains top-tier status by balancing high scores with the lowest cost.