Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of Shampoo inconsistent with the paper #502

Open
siddharth9820 opened this issue May 29, 2023 · 0 comments
Open

Implementation of Shampoo inconsistent with the paper #502

siddharth9820 opened this issue May 29, 2023 · 0 comments

Comments

@siddharth9820
Copy link

pytorch-optimizer/torch_optimizer/shampoo.py

Line 130 in 910b414

inv_precond.copy_(_matrix_power(precond, -1 / order))

Shouldn't the second argument be -0.5/order? For example, with order 2, the authors raise the precondition matrices to the -1/4th power.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@siddharth9820