Nest small and Nest base parameters for Cifar100 #1023
-
|
After following #751 (comment) to implement Nest tiny for Cifar100. I wanted to extend it to nest small but whatever changes I attempt to make, I end up with random guess accuracy. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
@abdohelmy maybe you are already on top of this, but according to the official implementation, the only params that change should be embedding size and number of heads, ie they double. If that is indeed what you are doing, then I'd be curious to know at what point it falls apart. Perhaps you could try changing only one of those two params? Or not doubling them, but only slightly increasing them? |
Beta Was this translation helpful? Give feedback.
-
|
I'm really really sorry for the late reply, i thought i replied all this time. From experimenting with the architecture,It is very sensitive to warm-up epochs. When i used the implementation with the warm up schedule of PyTorch-lightning it was diverging and end up in random guess accuracy. But when i followed their warm up implementation it worked fine. The same problem also happened with me once when i did not use any augmentation by mistake. |
Beta Was this translation helpful? Give feedback.
@abdohelmy maybe you are already on top of this, but according to the official implementation, the only params that change should be embedding size and number of heads, ie they double.
If that is indeed what you are doing, then I'd be curious to know at what point it falls apart. Perhaps you could try changing only one of those two params? Or not doubling them, but only slightly increasing them?