Any recommendation for distillation PE？

[Jacobew]

Hi, thanks for your great work on PE!

I’ve tried it on my perception model and got good results.

Now I’m working on distilling a smaller ViT version and wonder:

1. Are there any recommended distillation strategies for small ViT models using only visual data? Would that be less effective than the CLIP-style distillation mentioned in Section C.3?

2. Is there any chance the distillation training code will be released?

Appreciate any guidance you can offer!

@shuminghu @dbolya @PeizeSun @mmaaz60 @Jiankai-Sun

[dbolya]

Hi!

1. This depends on how you want to use the model. If your goal is using the result as a CLIP model, then naturally you need to do the CLIP-style distillation. However, for more general distillation check out the losses used for spatial alignment (see appendix, though note that there's a missing minus sign in the current draft of the paper, see Question about the implementation of spatial alignment for perception encoder #43). Every distillation technique comes with benefits and drawbacks, so you will need to choose one tailored to your downstream use case.

2. There are no plans currently to release PE training code, but we may release some loss implementations if that is helpful.

[Jacobew]

Thanks for your suggestions! They're very helpful.

1. I'm currently working on a 3D perception task that utilizes a PE backbone with several dense decode heads. I'll definitely incorporate the distillation process from the spatial alignment section as you advised.

2. I agree that the loss implementation is crucial, and I'm very much looking forward to seeing it released soon. Your efforts are greatly appreciated!

I still have a few follow-up questions:

• Could you elaborate on the full distillation process for the newly released smaller models (e.g., pe-spatial-tiny/small/base)? Specifically:

  • Are these models distilled solely using the method described in the spatial alignment section?

  • Is the CLIP-style distillation process also involved in their training?

  • Is there any specific initialization strategy used when starting the training of these smaller models?

• Given the variety of distillation methods for Vision Transformers (ViT), I'm particularly interested in the highly promising approach used in ViT-22B. I'm curious if your team explored such well-established methods before developing the innovative alignment algorithm presented in your paper. If so, I'd be very grateful if you could share any insights or experiences gained from those trials.

@dbolya

[dbolya]

Sorry for the delay, the new smaller models have two losses:

• direct distillation (last layer -> linear layer) of the PE spatial G model
• pairwise cosine similarity distillation (last layer) of the PE spatial G model

So it's like you did the same losses for PE spatial G, but replaced both the semantic and locality teachers with PE spatial G. Likewise, they're finetuned directly on top of the PE core variants just with the position embedding interpolated up to 448 / 512.

I tried a bunch of things--e.g., additionally aligning to PE core, using the same technique from PE spatial G to produce PE spatial L from PE core L using core + SAM, etc., but all of those were lacking. Overall on all tasks we test (detection, depth, tracking, segmentation), the approach I described above worked the best. Any extra alignment to PE core (even DINO.txt-esque methods to enable zeroshot semantic segmentation) turned out to hurt performance of depth and semantic segmentation.

At some point, we'll update the paper with a more detailed description, but quite busy atm.

---

As for ViT 22B distillation, we did look into it before settling on our CLIP distillation method since it looked promising. However, what the ViT 22B distillation really solves is distilling for ImageNet, not distilling in general. The innovation comes from 500x pre-computed augmentations of ImageNet 🫠. We needed something more general in the end.

[Jacobew]

Thank you for your detailed and thoughtful reply—it has truly clarified my questions.

Just to confirm my understanding:

• Is the "direct distillation loss" the cosine similarity loss computed on the spatial features of size H × W?
• And is the "pairwise cosine similarity distillation loss" the MSE loss applied to the flattened HW × HW similarity matrix?

Additionally, have you explored generating the PE spatial series from the PE lang series? In my experiments, the PE lang series outperforms PE core on perception tasks, which is quite interesting.

I would greatly appreciate it if more details on these design choices, along with the exact loss implementations, could be included in the paper or released in the codebase when available.

Looking forward to your insights!

@dbolya