[DEGRADATION] I see significant performance degradation with all TIMM models compare to TorchVision on iMet dataset

[Borda]

Describe the bug

Not sure if the problem is my (user) side but running the same pipeline with the same dataset I getting very different F1 performance measures, see the following notebook with the only switch between TV and TIMM model
https://github.com/Borda/kaggle_iMet-collection/blob/main/notebooks/iMet-with-Lightning.ipynb

To Reproduce
Steps to reproduce the behavior:

• used dataset - https://www.kaggle.com/c/imet-2021-fgvc8
• with timm.create_model('resnet50', pretrained=True, num_classes=dm.num_classes) I get f1 about 24%
• with net = LitResnet(arch='resnet50', num_classes=dm.num_classes) I get f1 about 42% ; the model is implemented as
  https://github.com/Borda/kaggle_iMet-collection/blob/d1757e5bcc50065b0f92bddb6e3b40924f2c8819/kaggle_imet/models.py#L11-L22

Expected behavior
there performances shall be almost the same, not difference as double

Desktop (please complete the following information):

• OS: google Colab
• This repository version: 0.4.9

Additional context
Add any other context about the problem here.

[rwightman]

@Borda moved to discussions as this sort of issue isn't likely a bug on the timm end of things, you can try tv_resnet50 as the model name in timm to get the torchvision weights with the same model code

[Borda]

I was running comparison with these three alternatives and got:

• TIMM resnet50: loss=0.689, v_num=0, valid_acc=0.877, valid_prec=0.112, valid_f1=0.0996, train_f1=0.118
• TIMM tv_resnet50: loss=0.68, v_num=1, valid_acc=0.904, valid_prec=0.791, valid_f1=0.420, train_f1=0.332
• TV original resnet50: loss=0.679, v_num=0, valid_acc=0.905, valid_prec=0.788, valid_f1=0.429, train_f1=0.456

[rwightman]

huh, if your training is that sensitive to the initial weights it could probably be improved

[Borda]

it did not change much after 10 epochs, and these numbers above are 10 epoch training...
I found that with simplifying augmentation it got better 🐰