Chess Arena: Watching LLMs Actually Play

Note: this is a personal evals/systems experiment, not a primary product. Public chess LLM benchmarks, including Kaggle's Game Arena, already exist; this project isn't positioned to compete with them.

Public chess benchmarks lead with win rate, which hides what actually matters: whether a model can reliably follow the rules it's given, whether its "efficiency" claims hold up once it's doing real work instead of answering a benchmark question, and, under tournament pressure across many runs, which model actually performs best rather than just who won the most individual games.

Chess Arena runs LLMs against each other in full games and knockout tournaments, enforces every rule itself so model output is never trusted blindly, and logs each move with legality, retries, tokens, and cost, producing data a results dashboard can render directly, with zero code changes required to add a new model.

Two independently deployed services share one database rather than one monolith. The Python runner plays LLM-vs-LLM matches and tournaments start-to-finish with no idle time, exactly what a Cloud Run Job is priced for. Human-vs-model play is the opposite: it waits on a person, so it runs synchronously inside a Next.js API route instead of ever being dispatched as a billable job.

Both services connect directly to the same DigitalOcean-managed database. The runner writes moves incrementally as a game is played, which is what makes live viewing possible: the frontend just polls for new rows on the current match.

The board is served to each model as a piece-location matrix, no PGN, no move history, plus turn, castling rights, en passant target, repetition count, and halfmove clock. That last pair matters: since the model gets no move history, it has no way to know it's stalling or repeating unless it's told the current-state facts directly.

Every proposed move is validated against python-chess's legal move list before it's applied, model output is always treated as untrusted input, never trusted on its own claim of legality. An illegal move gets re-prompted, up to a configurable retry cap; if it's still illegal after that, the engine auto-plays a random legal move and flags that ply as forced, so it never silently corrupts the results.

Games end only through real conditions: checkmate, stalemate, the 50-move rule, threefold repetition, or a hard move cap. No engine-based resignation in v1. The shared prompt template is identical across every model and explicitly instructs decisive play, pairing that incentive with the repetition/halfmove signals so a model can actually act on "don't stall."

V1 is fixed at a 4-model bracket, two semifinals into a final, with each matchup configurable as best-of-K games to dampen the real move-to-move variance a single LLM game has. Color assignment alternates by bracket position across repeated runs so no model is structurally favored.

Running the same bracket many times over answers a better question than a single knockout can: who actually performs best, not who got the friendlier draw. Each model's composite score leans heavily on average placement (about 90% of the score) with light efficiency penalties for retries and cost, enough to break ties between similarly-placed models without overriding placement outright.

Chess Arena turns "which model is best at chess" into a question with an actual evidentiary trail: every move, retry, token, and dollar logged, every rule enforced independently of the model, and every ranking backed by repeated runs instead of a single noisy bracket.

The engine is in active development against the architecture above: a Python runner for LLM-vs-LLM batches and tournaments, a Next.js dashboard for live viewing and human play, and a shared database that keeps both in sync without any real-time infrastructure.