Accuracy, Precision, Recall always the same value

[michaelgroeger]

Hello everyone,

Currently I am familiarising myself with pytorch-lightning. I train a simple CNN on Cifar-100 and log the metrics named in the title to be send to weights&biases. The whole process runs very smooth, however I get for each step & epoch the same values for my metrics. This happens on different datasets and runs always. That shouldn't be the case right? Any input/help on this is very much appreciated.

Here is my relevant code:

import torch
from torch import nn
import torch.nn.functional as F
import pytorch_lightning as pl
from torchmetrics import Accuracy, Precision, Recall

class Classifier(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.save_hyperparameters()
        self.train_accuracy_metric = Accuracy()
        self.train_precision_metric = Precision()
        self.train_recall_metric = Recall()
        self.val_accuracy_metric = Accuracy()
        self.val_precision_metric = Precision()
        self.val_recall_metric = Recall()
        self.test_accuracy_metric = Accuracy()
        self.test_precision_metric = Precision()
        self.test_recall_metric = Recall()
    
    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        # training_step defines the train loop
        x, y = batch
        y_hat = self.forward(x)
        loss = F.cross_entropy(y_hat, y)
        train_accuracy = self.train_accuracy_metric(y_hat, y)
        train_precision = self.train_precision_metric(y_hat, y)
        train_recall = self.train_recall_metric(y_hat, y)
        values = {"loss": loss, "train_accuracy": train_accuracy, "train_recall": train_recall, "train_precision": train_precision}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return loss
    
    def test_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        test_loss = F.cross_entropy(y_hat, y)
        test_accuracy = self.test_accuracy_metric(y_hat, y)
        test_precision = self.test_precision_metric(y_hat, y)
        test_recall = self.test_recall_metric(y_hat, y)
        values = {"test_loss": test_loss, "test_accuracy": test_accuracy, "test_recall": test_recall, "test_precision": test_precision}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)
    
    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        val_loss = F.cross_entropy(y_hat, y)
        val_accuracy = self.val_accuracy_metric(y_hat, y)
        val_precision = self.val_precision_metric(y_hat, y)
        val_recall = self.val_recall_metric(y_hat, y)
        values = {"val_loss": val_loss, "val_accuracy": val_accuracy, "val_recall": val_recall, "val_precision": val_precision}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)

class CNN_CIFAR_Classifier(Classifier):
    def __init__(self, num_outputs, learning_rate=0.001, *args, **kwarg):
        super().__init__(num_outputs)
        self.learning_rate = learning_rate
        self.net =  nn.Sequential(
            # input: 3 x 32 x 32
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            # output: 32 x 32 x 32
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            # output: 64 x 32 x 32
            nn.ReLU(),
            # output: 64 x 32 x 32
            nn.MaxPool2d(2, 2), # output: 64 x 16 x 16

            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 128 x 8 x 8

            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 256 x 4 x 4

            nn.Flatten(), 
            nn.Linear(256*4*4, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, num_outputs))

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        return optimizer

This is for instance the output of the trained network on the test set after 8 Epochs:

      Test metric             DataLoader 0
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   test_accuracy_epoch       0.382999986410141
     test_loss_epoch        2.4219934940338135
  test_precision_epoch       0.382999986410141
    test_recall_epoch        0.382999986410141

This the training output for Epoch 7:

Epoch 7: 100%|█| 197/197 [00:07<00:00, 26.02it/s, loss=1.73, v_num=jy1o, loss_step=1.630, train_accuracy_step=0.500, train_recall_step=0.500, train_precision_step=0.500, val_loss_step=2.240, val_accuraEpoch 7, global step 1256: 'val_loss' was not in top 1

[rohitgr7]

weird, this shouldn't happen. For started i'd start inspecting the step graphs, are they different or same? If they are different then the issue might be in the epoch accumulation. For that can you share a reproducible script?

[michaelgroeger]

Hello @rohitgr7,
Thank you so much that you took your time to respond.
Could you try it with this script? I assembled all the parts to be a working example from my modules. This should download CIFAR100 and train a CNN on it. It will log all three metrics:

from torchvision.datasets import CIFAR100
from torch import Generator
import pytorch_lightning as pl
import os

#####################################################################################
####################                DATA MODULE START               #################
#####################################################################################

from torch.utils.data import random_split, DataLoader
from torchvision import transforms

class GenericDataModule(pl.LightningDataModule):
    def __init__(self, pytorch_dataset, train_set_size, generator, num_workers, train_shuffle, data_dir: str = "./", batch_size: int=256):
        super().__init__()
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.pytorch_dataset = pytorch_dataset
        self.train_set_size = train_set_size
        self.generator = generator
        self.num_workers = num_workers
        self.train_shuffle = train_shuffle
        self.transform = transforms.Compose([transforms.ToTensor()])

    def prepare_data(self):
        # download
        self.pytorch_dataset(self.data_dir, train=True, download=True)
        self.pytorch_dataset(self.data_dir, train=False, download=True)

    def setup(self):
        self.full_dataset = self.pytorch_dataset(self.data_dir, train=True, transform=self.transform)
        train_set_split = int(len(self.full_dataset) * self.train_set_size)
        val_set_plit = len(self.full_dataset) - train_set_split
        self.dataset_train, self.dataset_val = random_split(self.full_dataset, [train_set_split, val_set_plit], generator=self.generator)
        self.dataset_test = self.pytorch_dataset(self.data_dir, train=False, transform=self.transform)

    def train_dataloader(self, collate_fn=None):
        return DataLoader(self.dataset_train, batch_size=self.batch_size, num_workers=self.num_workers, shuffle=self.train_shuffle, collate_fn=collate_fn)

    def val_dataloader(self, collate_fn=None):
        return DataLoader(self.dataset_val, batch_size=self.batch_size, num_workers=self.num_workers, shuffle=False, collate_fn=collate_fn)

    def test_dataloader(self, collate_fn=None):
        return DataLoader(self.dataset_test, batch_size=self.batch_size, num_workers=self.num_workers, shuffle=False, collate_fn=collate_fn)

    def prepare_for_training(self):
        self.prepare_data()
        self.setup()
        train_loader = self.train_dataloader()
        val_loader = self.val_dataloader()
        test_loader = self.test_dataloader()
        return train_loader, val_loader, test_loader

class CIFAR100DataModule(GenericDataModule):
    def __init__(self, pytorch_dataset, train_set_size, generator, num_workers, train_shuffle, data_dir: str = "./", batch_size: int = 256):
        super().__init__(pytorch_dataset, train_set_size, generator, num_workers, train_shuffle, data_dir, batch_size)
        self.transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])


#####################################################################################
####################                    MODEL START                 #################
#####################################################################################

import torch
from torch import nn
import torch.nn.functional as F
from torchmetrics import Accuracy, Precision, Recall


class Classifier(pl.LightningModule):
    def __init__(self, num_outputs, learning_rate=0.001):
        super().__init__()
        # self.save_hyperparameters()
        self.learning_rate = learning_rate
        self.accuracy_metric = Accuracy(num_classes=num_outputs)
        self.precision_metric = Precision(num_classes=num_outputs)
        self.recall_metric = Recall(num_classes=num_outputs)
    
    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        # training_step defines the train loop
        x, y = batch
        y_hat = self.forward(x)
        loss = F.cross_entropy(y_hat, y)
        train_accuracy = self.accuracy_metric(y_hat, y)
        train_precision = self.precision_metric(y_hat, y)
        train_recall = self.recall_metric(y_hat, y)
        values = {"loss": loss, "train_accuracy": train_accuracy, "train_precision": train_precision, "train_recall": train_recall}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return loss
    
    def test_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        test_loss = F.cross_entropy(y_hat, y)
        test_accuracy = self.accuracy_metric(y_hat, y)
        test_precision = self.precision_metric(y_hat, y)
        test_recall = self.recall_metric(y_hat, y)
        values = {"loss": test_loss, "test_accuracy": test_accuracy, "test_precision": test_precision, "test_recall": test_recall}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)
    
    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.forward(x)
        val_loss = F.cross_entropy(y_hat, y)
        val_accuracy = self.accuracy_metric(y_hat, y)
        val_precision = self.precision_metric(y_hat, y)
        val_recall = self.recall_metric(y_hat, y)
        values = {"loss": val_loss, "val_accuracy": val_accuracy, "val_precision": val_precision, "val_recall": val_recall}
        self.log_dict(values, on_step=True, on_epoch=True, prog_bar=True, logger=True)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        return optimizer

class CNN_CIFAR_Classifier(Classifier):
    def __init__(self, num_outputs, learning_rate, *args, **kwarg):
        super().__init__(num_outputs, learning_rate,)
        self.learning_rate = learning_rate
        self.net =  nn.Sequential(
            # input: 3 x 32 x 32
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            # output: 32 x 32 x 32
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            # output: 64 x 32 x 32
            nn.ReLU(),
            # output: 64 x 32 x 32
            nn.MaxPool2d(2, 2), # output: 64 x 16 x 16

            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 128 x 8 x 8

            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 256 x 4 x 4

            nn.Flatten(), 
            nn.Linear(256*4*4, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, num_outputs))


# Set Hyperparams
hparams = {
"MAX_EPOCHS" : 4,
"CLASSIFIER_NUM_OUTPUTS": 100, 
"ACCELERATOR" : "cpu",
"DEVICES" : 1,
"BATCH_SIZE" : 512,
"TRAIN_SHUFFLE": True,
"SEED": 42,
"LEARNING_RATE": 1e-2,
"OPTIMIZER" : "adam",
"NUM_WORKERS": 4,
"MODE" : "Local",
"DATASET": "CIFAR100",
"TRAINER_N_SANITY_CHECKS": 2,
"TRAIN_SET_SIZE": 0.8,
"TEST_WHICH_MODEL": "best",
}


def main(hparams=hparams):
    # Get and build paths
    os.getcwd()
    ROOT_DIR = os.getcwd().split('/')
    MODEL_DIR = ROOT_DIR.copy()
    LOGGER_DIR = ROOT_DIR.copy()
    DATA_DIR = ROOT_DIR.copy()
    MODEL_DIR.extend(['models', hparams["DATASET"]])
    LOGGER_DIR.extend(['logs', hparams["DATASET"]])
    DATA_DIR.extend('datasets')
    LOGGER_DIR = "/".join(LOGGER_DIR)
    MODEL_DIR = "/".join(MODEL_DIR)
    DATA_DIR = "/".join(DATA_DIR)
    if not os.path.exists(LOGGER_DIR):
        os.makedirs(LOGGER_DIR)
    if not os.path.exists(MODEL_DIR):
        os.makedirs(MODEL_DIR)
    ROOT_DIR = "/".join(ROOT_DIR)
    ## Data
    seed = Generator().manual_seed(hparams["SEED"])
    CIFAR100_DATASET = CIFAR100DataModule(CIFAR100, hparams["TRAIN_SET_SIZE"], generator=seed, num_workers=hparams["NUM_WORKERS"], data_dir=DATA_DIR, train_shuffle=hparams["TRAIN_SHUFFLE"], batch_size=hparams["BATCH_SIZE"])
    train_loader, val_loader, test_loader = CIFAR100_DATASET.prepare_for_training()
    ## model
    classifier = CNN_CIFAR_Classifier(num_outputs=hparams["CLASSIFIER_NUM_OUTPUTS"], learning_rate=hparams["LEARNING_RATE"])
    # train model
    trainer = pl.Trainer(max_epochs=hparams["MAX_EPOCHS"], accelerator=hparams["ACCELERATOR"], devices=hparams["DEVICES"], num_sanity_val_steps=hparams["TRAINER_N_SANITY_CHECKS"], default_root_dir=LOGGER_DIR, log_every_n_steps=5,  limit_train_batches=0.10, limit_val_batches=0.10, limit_test_batches=0.10)
    trainer.fit(classifier, train_loader, val_loader)
    trainer.test(ckpt_path=hparams["TEST_WHICH_MODEL"], dataloaders=test_loader)

if __name__ == "__main__":
    main(hparams=hparams)

I just ran it as a test and got those results with this script:

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       Test metric             DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       loss_epoch           4.6053361892700195
   test_accuracy_epoch         0.0087890625
  test_precision_epoch         0.0087890625
    test_recall_epoch          0.0087890625

[justusschock]

Hi,

this behavior is actually correct and results from the fact, that you didn't specify, how to average (which uses "micro" averaging by default in this case).

To account for that, this argument had been changed to "macro" as a default on master and you could specify this also on the current release explicitly.