Chess Cheat Detector has a new model. It catches more cheaters, and it wrongly flags about one honest player in 27, down from nearly one in five.
The numbers
v1 and v2 were scored on the same test set of 8,625 players, all from games that none of them saw during training:
| v0: hand-tuned rules | v1: first ML model | v2: today | |
|---|---|---|---|
| Picks the cheater over an honest player* | 71% | 80% | 93% |
| Cheaters flagged (score of 50% or more) | 0% | 63% | 69% |
| Honest players flagged | 0% | 19% | 4% |
| Honest players rated 2200+ flagged | 0% | 33% | 4.5% |
| Honest players scored "High" (70% or more) | 0% | 5.5% | 1.0% |
| Cheaters scored "High" | 0% | 32% | 52% |
*Take one cheater and one honest player at random. This is how often the cheater gets the higher score (ROC AUC).
v0, the hand-tuned rules the site launched with, needs a slow complexity search on every move, so it was scored on 7,874 players from a second set of similar games, which it also never saw. Its rules only gave real weight to engine moves in complex positions, to avoid accusing a strong player for finding obvious moves. That kept wrong flags at zero, and it still ranked cheaters above honest players 71% of the time. But its scores stayed below the 50% flag line, even for heavy engine users. Learning from data rather than hand-tuning is what let v1 and v2 get much more powerful.
This test set includes cheating styles v1 never saw, so v1's numbers here are lower than in the first write-up.
What changed: it knows who's playing
v1 judged everyone by the same yardstick. Accuracy that's normal for a 2400 is suspicious for a 1200, and v1 couldn't tell the two apart. This meant that it missed lower-rated cheaters when mixing human moves with Stockfish, and wrongly flagged a third of honest players rated 2200 and up.
v2 reads both players' ratings, and uses a neural network to ask one question about every move: how likely is a human at this rating to play it? A 1200 who keeps finding moves that 1200s almost never find stands out. A 2400 playing like a 2400 doesn't.
That one change does most of the work. Wrong flags are now flat across ratings, between 3% and 4.5% from 600 to 2600. For v1, wrong flags increased from 7% to 33% between 600 and 2600. These new features are now the model's most important inputs.
About 15% of games submitted to the site have no ratings. We still support this - v2 judges the moves alone and ignores ratings. It still ranks players better than v1 (83% against 80%) and flags far fewer honest players (11% against 19%). The trade-off is that it's more cautious, catching 53% of cheaters instead of 63%.
Five times the data, three ways to cheat
v1 learned from 6,613 simulated games, and every cheater in them worked in the same way: flip a weighted coin before each move. We know that real cheaters aren't that tidy.
The dataset is now over 33,000 games, with three different cheating styles:
- Coin flip. On any given move, there's a chance the cheater asks the engine.
- Streaks. Switch the engine on for a run of moves, then back off.
- Blunder checks. Play your own moves, but use an engine to prevent making a mistake. Most of these cheaters only use it a handful of times a game, which makes them the hardest to spot.
v1 flagged only 53% of blunder-checker cheaters, and v2 flags 65%. On ranking alone, v1 picked the blunder-checker over an honest player 75% of the time, and v2 does 94% of the time.
The fine print
- These are simulated games. The honest players are a simulated imitation of Lichess players, not people. In an early check on real games analyzed on this site, we found real players about as human-like as the simulated honest ones at almost every rating. It's encouraging, but not conclusive.
- Ratings from different sites aren't on the same scale. Chess.com ratings are typically below Lichess's, and new Lichess accounts start at 1500 before they've played. Correcting for both is next on the list.
- A score is just a probability. It's not a conclusion. Honest players do sometimes have extraordinary games.
Got a game that felt off? Paste it in and see what v2 makes of it.