Chess Cheat Detector has a new model. On real games between grandmasters, it wrongly rates about one honest player in 60 as Elevated or High, down from one in four.
The numbers
Both versions were scored on the same 14,229 players, from games that neither saw during training:
| v2 | v3: today | |
|---|---|---|
| Honest grandmasters scoring 70% or more | 25.9% | 1.6% |
| Honest players scoring 70% or more | 2.7% | 1.0% |
| Picks the cheater over an honest player* | 89.3% | 89.8% |
| Cheaters scoring 70% or more | 47.3% | 36.4% |
| Grandmaster-level cheaters scoring 70% or more | 53.3% | 25.5% |
*Take one cheater and one honest player at random. This is how often the cheater gets the higher score (ROC AUC).
A score of 70% or more is shown as Elevated or High. The honest grandmasters are 2,982 players from real games, all rated 2500 or above. Every other row comes from simulated games, including some that start like a grandmaster's.
v2's numbers here are lower than in its own write-up, which scored only games with ratings. These include a quarter of games without ratings, closer to what the site sees, as well as the harder grandmaster-style games.
What changed: it has seen a grandmaster
v2 learned from simulated players, and the strongest of them played at around 2600 on Lichess's scale. That's well below a real grandmaster. So v2 had never seen an honest player as accurate as one, and the only players in its training data who played that well were cheaters.
v3 also learned from two new sets of games, about 7,000 of each:
- Real grandmaster games. Games between players rated 2500+ FIDE, collected by PGN Mentor. Everyone in them is treated as honest.
- Games that start like a grandmaster's. Games that follow a real grandmaster game for the first 15 to 30 moves, then are played out by a simulated player, who sometimes cheats. Without these, a cheater who plays a strong opening would look like a grandmaster.
The trade-off
v3 is more cautious. It ranks players slightly better than v2, but it gives lower scores across the board. Fewer cheaters reach 70%, and the biggest drop is among those cheating at grandmaster level. Real grandmasters' play overlaps heavily with engine-assisted play, and v3 now gives strong players the benefit of the doubt.
We made that trade deliberately. Honest strong players vastly outnumber cheating ones, and wrongly accusing them does real damage. The real grandmaster games count for a quarter as much as the rest in training. At full weight, wrong flags fell even further, but v3 missed most grandmaster-level cheaters.
The fine print
- The same grandmasters appear in training and testing, in different games. v3 may do a little worse on a strong player it has never seen.
- Grandmaster-level cheaters are still simulated. We don't have real ones to test on.
- A score is just a probability. It's not a conclusion. Honest players do sometimes have extraordinary games.
Got a game that felt off? Paste it in and see what v3 makes of it.