Mismatch between www.maiachess.com/analysis model and python package model

[kawgit]

In the following position, the model at www.maiachess.com/analysis seems to correctly predict a reasonable distribution of responses from black.

See https://www.maiachess.com/analysis/7WCFYt0R/1

However, using the python package yields very different and disappointing results:

https://colab.research.google.com/drive/1E4mXo4UrUCOGx50a1ljo9mLCEEcE0Zbx#scrollTo=Ce2pPpE1kt3L

The blitz model predicts with >99% likelihood that black will choose to move it's knight back to where it started from (Ne8) which obviously makes no sense.

What's happening here?

[kawgit]

hmm, in the exact same position but with the opposite turn (white to move) the issue disappears and they end up having the exact same distributions. there may be an issue with how the python package parses fen

[lilv98]

Thanks for your interest in our research!

Maia2 did not include openings (first 5 plies) in training. So, opening behavior could be problematic. In maiachess.com, we sample from an opening book for openings, then use maia2 for other moves afterwards.

We did that with the goal of ensuring diversity in opening moves, otherwise maia2 will sample from the "weighted average" of human moves, which may be the most frequent/popular ones most of the time. We also removed moves under 30 seconds to avoid randomness in training signals.

Will this problem still exist in the middle games? It will be a separate problem/bug if so.

(We have the plan to update maia2 to include openings and endgames in training.)