Loss coming out to be "nan" on a pytorch lightning module

[asad-ak]

This is my notebook link https://colab.research.google.com/drive/1fDeXMOoek4E2i0C_CkAXNP8bozMbTknk?usp=sharing
Also I am providing the Model if there is a problem in that:

import torch.nn.functional as F

class VIPLitNet(pl.LightningModule):

  def __init__(self, num_cols, number_of_classes, learning_rate = 0.01):
  

    super().__init__()
        
    input_vector = len(num_cols)
    self.learning_rate = learning_rate
    self.num_cols = num_cols
    self.n_classes = number_of_classes
    self.l1 = torch.nn.Linear(input_vector, 128)
    self.l2 = torch.nn.Linear(128, 256)
    self.l3 = torch.nn.Linear(256, number_of_classes)


  def forward(self, x):
    x = self.l1(x)
    x = F.relu(x)
    x = self.l2(x)
    x = F.relu(x)
    x = self.l3(x)
    x = F.log_softmax(x)

    return x

  def training_step(self, batch, batch_idx):
    X, y = batch
    outputs = self(X)
    y = y.unsqueeze(1)
    y= y.float()
    loss = F.binary_cross_entropy_with_logits(outputs, y)
    return {"loss" : loss}

  def validation_step(self, batch, batch_idx):
    X, y = batch
    outputs = self(X)
    y = y.unsqueeze(1)
    y= y.float()
    loss = F.binary_cross_entropy_with_logits(outputs, y)
    return {"val_loss" : loss}

  def test_step(self, batch, batch_idx):
    X, y = batch
    outputs = self(X)
    y = y.unsqueeze(1)
    y= y.float()
    
    loss = F.binary_cross_entropy_with_logits(outputs, y)
    return {"test_loss" : loss}

  def configure_optimizers(self):
    return torch.optim.Adam(self.parameters(), lr=self.learning_rate)

  def validation_epoch_end(self,outputs):
    avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
    return {"val_loss" : avg_loss}

[rohitgr7]

try removing the F.log_softmax call since binary_cross_entropy_with_logits need logits.
ref: https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy_with_logits.html

[asad-ak]

Even after removing the log_softmax the loss is still coming out to be nan

[ananthsub]

Could you try running with Trainer(detect_anomaly=True) ? This should give an informative stacktrace of where the NaN might be coming from.

You can also check whether your data itself has bad inputs that are causing the loss to go NaN

[asad-ak]

Could you try running with Trainer(detect_anomaly=True) ? This should give an informative stacktrace of where the NaN might be coming from.

You can also check whether your data itself has bad inputs that are causing the loss to go NaN

Function 'BinaryCrossEntropyWithLogitsBackward0' returned nan values in its 0th output.

[asad-ak]

Could you try running with Trainer(detect_anomaly=True) ? This should give an informative stacktrace of where the NaN might be coming from.

You can also check whether your data itself has bad inputs that are causing the loss to go NaN

https://colab.research.google.com/drive/1fDeXMOoek4E2i0C_CkAXNP8bozMbTknk?usp=sharing

[rohitgr7]

the loss function might not be getting the correct logits since you are using softmax inside forward

  def forward(self, x):
    x = self.l1(x)
    x = F.relu(x)
    x = self.l2(x)
    x = F.relu(x)
    x = self.l3(x)
    x = F.softmax(x, dim=1)

    return x