The learning rate in the paper and the code are inconsistent

[kjzju]

In your paper, you use the learning rate above where β = 5000, which meaning the learning rate is less than 10−8. However, in your code, there is

Thus, the learning rate in your code is about 2e-5 with the parameter "scale_lr". Is there anything wrong?

[liming-ai]

There is a way to explain this. The value in the paper should be 2,048 (i.e., 2048) instead of 2.048, which can then correspond to the result in the code.