Adding FeatUp

[Jordan-Pierce]

@qubvel happen to see FeatUp? Might make for some interesting gains in model performance, where applicable.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[vedantdalimkar]

Hi @qubvel, I would like to work on this issue. Do you have any vision regarding it's implementation? If not, I will make a rough plan for the addition of this feature and will let you know.

[qubvel]

Hey @vedantdalimkar, thanks for the interest in contributing FeatUp, sounds super cool 🔥
I don't have any vision regarding implementation at the moment, feel free to suggest your way and ping me to discuss 👍

[vedantdalimkar]

Hi @qubvel, after reading the FeatUp paper I have some questions on how it can integrated in the library.
FeatUp extracts features from a frozen encoder of a trained model. It then trains a small MLP to learn a mapping from the extracted features to high resolution features. However, training the model on the high resolution feature maps output by FeatUp would require us to change the decoder input layers. So the steps would be

1. Training a segmentation model on a dataset
2. Freeze the encoder and train the FeatUp model
3. Change the decoder layer to accomodate the higher resolution of features
4. Finetune the decoder on the given dataset.

I am little confused on how the FeatUp module would be integrated inside a model given the above steps.
Also, how can we take care of the change in the decoder after FeatUp is trained?

[qubvel]

Hi @vedantdalimkar, yeah, it sounds a bit out of the scope for the library (at the moment), so we have to think how it would be better to integrate. As a first step it would be cool to have draft pipeline implemented in Jupiter notebook to understand all the steps, e.g. one of the example notebooks in examples/ folder can be used and extended. What do you think?

[vedantdalimkar]

Hi @vedantdalimkar, yeah, it sounds a bit out of the scope for the library (at the moment), so we have to think how it would be better to integrate. As a first step it would be cool to have draft pipeline implemented in Jupiter notebook to understand all the steps, e.g. one of the example notebooks in examples/ folder can be used and extended. What do you think?

Ok, I'll work on the example notebook once I have properly understood the FeatUp code.