Building a Chess Cheat Detector

Chess Cheat Detector is a free chess cheat checker. Paste a Chess.com or Lichess link, or a PGN, and it estimates how likely it is that each player had engine help.

The need seems to exist: Chess.com alone closed about 105,000 accounts a month for fair play violations in early 2025. And people are bad at finding cheaters by eye. When GM David Smerdon asked more than 4,000 players to judge seven games for cheating, with engine evaluations in hand, they were right about 54% of the time, only marginally better than a coin flip.

The first model on this website ran on hand-tuned rules. Matching the engine's top move was worth up to 45 points, a near-perfect move up to 25 more, and a match in a complex position another 20, all adjusted for rating and then blended into a single score for the game. Every constant was a guess, and I had no way to check any of them.

What I was missing was data. To build a cheat detector, or even to test one, you need games where you know who cheated, ideally down to the move. Platforms don't even have that, unless they pay people to cheat - an expensive and laborious process. A banned account on a chess platform is not conclusive proof that an account cheated.

So I generated my own games. Within a week of the first simulated game, a model trained on them was scoring every analysis on the site. This post covers the human-like engine that made it possible, the synthetic data, the evaluation (including where the model fails), why judging individual moves doesn't work, and what it took to run in production. There's no language model anywhere in the system: the detector is a classical machine learning model interpreting features for each player.

Maia: a chess engine that plays like a human

To simulate cheaters, you first need convincing honest players, and Stockfish can't provide them. It's one of the strongest chess programs ever written, and it plays the best move it can find. People don't. They play the natural-looking move, miss the tactic and, even at club level, they hang a piece now and then.

Maia, from the CSSLab at the University of Toronto, is built for the opposite job. It's a neural network trained on human games from Lichess to predict the move a person of a given rating would actually play. There's no search: each move is a single pass through the network.

Maia is on its third generation. The original (KDD 2020) trained a separate network for each rating band. Maia-2 (NeurIPS 2024) folded them into one model that takes skill as an input. Maia-3 (ICLR 2026) is built on a transformer architecture called Chessformer, and its top guess matches the move a human actually played 57.1% of the time, in positions that typically offer dozens of legal moves.

I chose Maia-3 as the human-simulation tool, for three reasons:

1. It's conditioned on both players' ratings, anywhere on Lichess's 600–2600 scale. A 1200 playing a 1250 is as easy to simulate as two 2500s.
2. It outputs a probability for every legal move, so you can sample rather than always taking its favorite. At temperature 1.0, a simulated 1200 plays with the variety of a real 1200, mistakes included. Always taking the most likely move would play like the consensus of many 1200s, which is steadier than any individual 1200.
3. It sees the last eight positions of the game, not just the current one, so its play remembers how the game got there. (Side note: My background is in textual AI - the ability to "remember" large context windows like this is a huge part of what has driven the emergence of LLMs.)

I ran the 23-million-parameter model on CPU as a UCI engine, driven from python-chess:

maia = chess.engine.SimpleEngine.popen_uci([
    sys.executable, "-m", "maia3.uci",
    "--model", "maia3-23m",
    "--use-uci-history",  # feed it the game's move history
    "--temperature", "1.0",
    "--seed", str(rng.getrandbits(32)),
    "--device", "cpu",
])
maia.configure({"SelfElo": rating, "OppoElo": opponent_rating})
move = maia.play(board, chess.engine.Limit(nodes=1)).move

What this means is that the "humans" in my dataset are Maia's imitation of Lichess players, not people. My chess cheating detector learns to tell Maia-plus-Stockfish from Maia, and how far that carries over to real players is a big open question. I come back to this at the end.

Generating training data: 6,613 games and counting

Each simulated game starts with three random draws:

• Who cheats. Nobody, White or Black, with equal odds. Color tells the model nothing, and a third of games have no cheater at all.
• The ratings. White's is drawn uniformly from 600 to 2600, and Black's within 100 points of it, an approximation of the way online matchmaking would pair them.
• How the cheater cheats. Each cheater gets their own settings.

An honest player is Maia at their rating. A cheater is the same Maia with a weighted coin: before every move, with probability p, they consult Stockfish. The coin flips double as labels, recording exactly which moves were assisted.

def get_move(self, board):
    # Determine with probability p whether to use Maia or Stockfish.
    use_stockfish = self._rng.random() < self._use_stockfish_probability

    # We save the per-move label as well as the game-level labels.
    self.engine_moves.append(use_stockfish)
    engine = self.stockfish_engine if use_stockfish else self.maia_engine
    return engine.get_move(board)

Honest players take the Maia branch on every move. A cheater takes the Stockfish branch with probability p.

The cheater's settings are where I've tried to create realism:

• How often. p is drawn log-uniformly between 5% and 100%, so light cheating is more common than heavy. The average (median) cheater used Stockfish on about 23% of their moves. This is to create hard cases, players who check the engine now and then, to dominate the data.
• How strong. Stockfish searches to a random depth between 12 and 20.
• How obvious. The cheater doesn't always play the engine's top choice. They pick from Stockfish's top N moves that are within L centipawns of the best, with L between 0 and 50, and N currently set to 5. (A centipawn is a hundredth of a pawn, the unit engines evaluate positions in.) Playing the second-best move after consulting an engine is still cheating.

Every game can be regenerated move for move from a single integer seed, including Maia's sampling. This makes the whole pipeline reproducible.

Games ran in parallel, distributed over CPUs.

Before trusting the data, I checked it:

• The results look like chess. White won 3,082 games, Black 2,829, and 702 were drawn. Maia never resigns, so 89% of games ended in checkmate.
• Maia isn't replaying one game. The 6,613 games began with 5,578 different eight-ply openings (four moves by each side). The most common, an Open Sicilian, appeared 55 times.
• Cheating works. In games with exactly one cheater, the cheater won 66.5% and lost 23.5%. Even light engine use is a big edge, and those games ended sooner: a median of 78 plies, against 86 with no cheater.
• Some labels were wrong. 111 "cheaters" by chance never once drew a Stockfish move, so they were actually honest and relabeled appropriately. Later I went further with this: a player now only counts as a cheater if Stockfish chose more than five of their moves. Below that there's almost nothing to detect, and calling it cheating mostly adds label noise. (Side note: I want to tune this figure with time.)

Then every position in every game was analyzed exactly the way production analyzes a submitted game: Stockfish 17.1 at depth 12, keeping its top five lines, on one thread, with the hash table cleared before each position. If the move actually played wasn't in the top five, a second search scores that move on its own. That came to 574,291 positions at about 0.18 seconds each on one core of an i7, or roughly 28 CPU-hours for every full pass over the dataset.

Finally, the split. Games, not players, are divided into training, validation and test sets by hashing each game's seed. This keeps the mapping between games and sets deterministic.

digest = hashlib.sha256(f"{SPLIT_SEED}:{game_id}".encode()).digest()
u = int.from_bytes(digest[:8], "big") / 2**64  # uniform in [0, 1)
split = "train" if u < 0.70 else "eval" if u < 0.85 else "test"

Both players in a game always land in the same split. They share a board, so putting one in training and the other in the test set could leak information across the split. And because the split depends only on the seed, it stays identical across every re-analysis at different depths and line counts, which is verified with a small script.

Evaluation: who it catches, and who it doesn't

The test set is 1,984 player-games from 992 games the model never saw: 441 cheaters, 1,337 honest players, and 206 players who used Stockfish on five moves or fewer and so count as honest.

The headline number is ROC AUC: 0.82 on the test set and 0.83 on validation. In plain terms, take one cheater and one honest player at random, and 82% of the time the model gives the cheater the higher score. Precision-recall AUC, which cares more about false alarms, is 0.62, against 0.22 for random guessing (the share of cheaters in the test set).

AUC is the number I tracked across experiments, but the site shows each player a percentage and a band. This is where the test set lands, counting as honest only the 1,337 players who never touched Stockfish:

Read it both ways. A "High" is rare for honest players, about 1 in 50, but only 28% of cheaters reach it, and 22% of cheaters look completely normal. A score of 50% or more catches 57% of cheaters and flags 11% of honest players.

The grey bar in each chart is the rate across the whole group, for comparison.

How much they cheated decides whether they're caught. Cheaters who let Stockfish choose more than three-quarters of their moves scored 50% or more 94% of the time. For those who used it on a quarter of their moves or fewer, it was 33%. Light cheating is harder to see.

Rating decides who gets wrongly flagged. As you might expect, the false-flag rate for honest players increases with strength: 1.2% of those rated 600–1000 scored 50% or more, against 21.4% of those rated 2200–2600. The model doesn't yet see a rating, so it can't know that a 2500 playing accurately is simply a 2500. This is probably its biggest weakness, and the first thing I want to improve next.

A stronger engine is harder to catch. One result I didn't expect: cheaters whose Stockfish searched to depth 18–20 reached 50% less often (47%) than those at depth 12–17 (61%). The detector analyzes at depth 12, and a deeper search doesn't always agree with a shallower one: 11% of engine moves in the dataset fell outside the depth-12 top five. With only about 130 players in the deep group, the data here is limited. I want to look into this more.

The base cheating rate changes what a prediction means. In this test set, 22% of players cheat. In the wild the share is far lower. If 1 player in 50 were cheating, a "High" would be right only about 23% of the time; at 1 in 10, about 61%. But people don't paste random games into a cheat checker. They paste the ones that already felt wrong, so the share among submitted games is probably higher than 1 in 50. Of course, I don't know what it is, so every prediction is purely statistical, and not proof. Human players do sometimes have extremely strong games.

All of this is measured on simulated games. It shows how well the model separates Maia-plus-Stockfish from Maia, which is only a proxy for separating cheaters from people.

Move level vs. game level: why one move doesn't prove much

Perhaps the obvious design for a model is a per-move detector: score every move, flag the suspicious ones, add them up. My initial rule-based version did work that way.

So I trained a classical machine learning model that works like this too. A model looks at features representing each move and predicts whether Stockfish had chosen the move.

Its move-level AUC was 0.74, which sounds respectable until you look closer:

• Engine moves are rare. Only 8.4% of the test set's moves came from Stockfish. At the threshold that maximized F1 on validation, just 17% of flagged moves were engine moves: about five incorrect predictions for every real one.
• Some of the signal was the scoreboard. Cheaters spend more of the game winning, so a good position predicts cheating without saying anything about the move.
• It couldn't pick out a cheater's engine moves. Within each cheater's game, it ranked their engine moves above their own with an AUC of 0.66. Sorting the same moves by centipawn loss alone also scored 0.66. Beyond "good moves look suspicious", it hadn't learned anything about individual moves.

Across every scored move in the dataset:

Honest players find the engine's top move a third of the time, and most of their moves are perfectly sound. A cheater's engine move, picked from Stockfish's top five, matches the top choice only half the time. Because honest moves vastly outnumber engine moves, any single excellent move is far more likely to be a human's good move than an engine's: 86% of the moves in the dataset that matched Stockfish's top choice were Maia's.

The better signal is in the aggregate. A player who lets Stockfish pick one move in five raises their top-move match rate by about five percentage points, while honest players range between about 20% and 48% from game to game (10th to 90th percentile). Only by combining many moves, and several kinds of evidence, does a shift that small become visible.

So I tested the aggregate directly. These experiments used an earlier version of the labels, where any engine use counted as cheating, so the absolute numbers are lower than the final model's.

• A game-level model, trained on 29 statistics summarizing each player's game, scored a per-player AUC of 0.763.
• Averaging the move-level model's log-odds across each player's moves scored 0.738. A paired bootstrap, resampling whole games, put the game-level model ahead by 0.011 to 0.039 (95% interval).
• Stacking, feeding the averaged move scores into the game-level model as extra features, changed nothing: −0.003 to +0.005.

So the game-level model won, and it's more simple. Per-move information didn't disappear from the site: the results page still shows every move's engine rank on a timeline, because that's how a person checks the model's work. But it's an explanation rather than evidence.

Training and deployment

The end-to-end production pipeline:

1. Analyze every position with Stockfish (depth 12, top five lines).
2. Compute the per-move features.
3. Drop each player's first four moves, which are often memorized opening theory, and any forced moves.
4. Summarize each player's remaining moves into game-level features.
5. A classical machine learning model turns the features into a probability.
6. Round it to a percentage, map it to a band, and write the explanation.

The game features come in four families:

• Match rates: how often a move was scored highly by the engine, in various different buckets.
• Centipawn loss: the share of moves losing different amounts of centipawns.
• Difficulty: the top-move rate in critical positions.
• Game state: the top-move rate and average loss when losing, winning, or even, and how many moves fell into each.

Finally, we also input the number of scored moves, so the model knows how much evidence it has.

Difficulty used to mean something more elaborate. The rule-based version estimated position complexity by watching how much Stockfish's move rankings shuffle as it searches deeper, following a method by Biswas and Regan. In my first benchmarks, at depth 19, that cost about 54 seconds per position. The gap between the top two moves comes free with the MultiPV search, and the model uses that instead.

Experiments that didn't make the cut:

• Deeper analysis. Depth 14 instead of 12 didn't improve the model.
• More engine lines. Analyzing the top ten moves instead of five scored a test AUC of 0.823 against 0.821. Production did briefly move to ten, then went back to five as this increased processing time by 1.6x.
• Streaks and stability. The longest run of top-engine moves, the best five-move window, and how often the engine changed its mind between depths. They traded recall for precision without clearly improving the model overall.

Two changes did help:

• Redefining a cheater as someone Stockfish chose more than five moves for lifted test AUC from 0.770 to 0.821. Part of that is a better-posed question rather than a better model. One or two engine moves are nearly invisible, and calling them cheating was mostly noise to the model.
• Keeping decided positions. At first I skipped moves played when already more than eight pawns up, since almost anything keeps the win. Putting them back added about 0.007 AUC under either definition.

Productionizing

The site runs as a handful of Docker containers on one server behind Cloudflare: nginx for the frontend, Django on gunicorn for the API, MySQL, RabbitMQ, and the analysis workers.

Engine analysis is the slowest part. At about 0.18 seconds per position, a typical 85-ply game needs around 15 seconds of Stockfish time, longer than a web request should wait. So submitting a game only saves it and publishes a message. A worker does the rest while the browser polls for progress.

A few details turned out to matter:

• Deterministic and parallel at once. Clearing Stockfish's hash before every position makes each position's analysis independent of the rest of the game. That makes results reproducible, and it also means a game's positions can be spread across a pool of Stockfish processes and still produce identical output.
• Honest progress. The engine stage owns the progress bar from 5% to 90% and advances one position at a time, so the processing screen moves steadily as work completes instead of stalling.
• Redelivery is normal. If a worker dies mid-game, from running out of memory or a redeploy, the queue redelivers the message. All of its writes are idempotent, so the worker simply starts again.
• Atomic results. Players' scores and the "complete" status are written in one transaction, so nobody ever sees a finished analysis without its scores.

No language model is used anywhere in the site. Each player gets a few templated sentences comparing their numbers with the honest players in the training data. The median simulated honest player matched the engine's top move on 33% of scored moves, found it in 56% of critical positions, lost 57 centipawns per move, and lost more than 100 centipawns on 18% of moves. Here's the output for one cheater in the test set, a 2335-rated player who let Stockfish choose 15 of their 43 moves and scored 81%:

The model found strong signs of engine-like play. 54% of their scored moves matched the engine's top choice, well above the typical 33% for human players. In the 15 positions where the next-best move was at least 50 centipawns worse, they found the engine's move 80% of the time, above the typical 56%. They lost 22 centipawns per move on average, well below the typical 57.

Next steps

1. Give the model the player's rating. Honest players rated 2200–2600 are wrongly flagged 21% of the time, against 1% of those rated 600–1000. Maia can help directly, because it estimates how likely a human of a given rating was to play each move. Moves that are consistently improbable for a player's rating are a good sign of engine help.
2. Use the clock. Chess.com and Lichess PGNs can include the clock after every move, and move times that don't fit the position, such as instant replies in sharp positions or long thinks over obvious recaptures, are a classic tell. Lichess's Irwin uses move times as a core input. My simulated games have no clocks, so this needs a model of human thinking time first.
3. Judge players, not just games. Going from moves to games was a big win, and going from games to players is the obvious next step. One "Moderate" is noise; ten in a row is a pattern. That's the idea behind the bulk profile review I'm building for premium.
4. Smarter simulated cheaters. Real cheaters can be strategic, switching the engine on when they're losing, at critical moments, or only to convert a won endgame. They might also use a different engine, while every simulated cheater currently uses the same Stockfish version the detector analyzes with. A detector trained only on coin-flippers may miss them, and now that the pipeline exists, new strategies are cheap to simulate.
5. Test on real games. Everything above is measured on simulated games. The next evaluation set should be real games from accounts closed for fair play violations. Those labels are noisy, since a closed account isn't ground truth and doesn't say which games were assisted, but they could be one of the best ways to measure the gap between Maia and real people.

If you've played a game that felt off, paste it into Chess Cheat Detector and see what the model makes of it. If you'd rather check by hand, I've written guides to spotting cheating on Chess.com and on Lichess.