Positional embedding for time in temporal attention layers in SVD

[jeanne-wang]

It seems that SVD are not actually using positional embedding for time in the temporal attention layers in the released model. Are there any specific reason not doing this?

[patrickvonplaten]

Hey @jeanne-wang,

I'm not 100% sure what you mean by "positional embedding for time", but the difference between SVD and say SDXL is that SVD conditions the unet not on discrete timesteps (e.g. 1, 2, ..., 999), but on continous "noise" values such as (3.2341).
This is explained in the paper in section 4.1:

As a first step, we finetune the
fixed discrete noise schedule from our image model towards
continuous noise [87] using the network preconditioning
proposed in Karras et al. [51] for images of size 256 × 384.

You should also probably take a look at https://arxiv.org/abs/2206.00364

[lim142857]

Hey @patrickvonplaten Is it possible to perform DDIM inversion on real video using SVD? I am concerned that DDIM is generally applied with discrete-time schedulers.