Propose fix perceptual loss sqrt nan #8414
Conversation
|
Thanks for the update, the changes looks fine to me. |
| def normalize_tensor(x: torch.Tensor, eps: float = 1e-8) -> torch.Tensor: | ||
| norm_factor = torch.sqrt(torch.sum(x**2, dim=1, keepdim=True) + eps) | ||
| return x / (norm_factor + eps) |
There was a problem hiding this comment.
| def normalize_tensor(x: torch.Tensor, eps: float = 1e-8) -> torch.Tensor: | |
| norm_factor = torch.sqrt(torch.sum(x**2, dim=1, keepdim=True) + eps) | |
| return x / (norm_factor + eps) | |
| def normalize_tensor(x: torch.Tensor, eps: float = 1e-8) -> torch.Tensor: | |
| norm_factor = torch.sqrt(torch.sum(x**2, dim=1, keepdim=True) + eps) | |
| return x / norm_factor |
Do we want to remove eps from the denominator? As proposed eps will contribute twice to the final result.
There was a problem hiding this comment.
This file should go into an appropriate subdirectory in the tests directory. We've changed the directory structure there recently so probably tests/losses.
| # Create tensor | ||
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) |
There was a problem hiding this comment.
I don't think the optimizer is needed for this test?
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) | ||
| x_scaled = x * scale |
There was a problem hiding this comment.
Since x is all 0, x_scaled is always going to be 0 unless you're expected float imprecision to create values here. If so, I would add a comment to mention this.
There was a problem hiding this comment.
I don't understand the point of this test with regards to the next one; instead of a zeros tensor, couldn't it be a random one which will be then multiplied by a really small number?
| x = torch.zeros(2, 3, 10, 10, requires_grad=True) | ||
|
|
||
| optimizer = optim.Adam([x], lr=0.01) | ||
| x_scaled = x * scale |
There was a problem hiding this comment.
I don't understand the point of this test with regards to the next one; instead of a zeros tensor, couldn't it be a random one which will be then multiplied by a really small number?
Fixes # 8412
Description
This PR fixes a numerical stability issue in the PerceptualLoss implementation where the
normalize_tensorfunction can produce NaN gradients when the input values are very small.Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.