Peter Gostev, head of the AI model evaluation platform Arena, released a post-mortem analysis of a chess experiment on October 16th (Beijing time), pointing out that current large models, relying solely on repeated practice, memory retention, and extended context, still struggle to achieve stable and continuous learning.
The experiment first had GPT-6 Astra and Claude Opus 5.5 each play 200 games against the Stockfish chess engine, autonomously choosing difficulty, recording experience, and adjusting strategies. Opus ultimately achieved an estimated Elo score of approximately 1760, higher than Astra's 1400 (Elo scoring is a rating system widely used in competitive chess and other sports to measure a player's relative skill level).
However, in the subsequent 30 direct matches, Astra decisively won 24.5 to 5.5. Another comparative test showed that GPT-6.1 Sol, using a context of approximately 800,000 tokens, actually performed worse than a version with approximately 300,000 tokens.
Gostev believes that while models may develop fixed strategies or repeatedly change tactics over long periods, they may not effectively accumulate experience. Experiments show that increasing context capacity and interaction frequency does not necessarily translate into continuous learning ability.
