Pixel Space to SD-XL's Latent Space?

[tolgacangoz]

Shouldn't each value in the SD-XL's latent space represent 8x8x3÷4 = 48 pixels in the pixel space?

[asomoza]

why are you mixing the channels with the pixels?

That's the code I demoed with the image preview, if you just save the image it literally saves a 128x128 pixel image, the channels are for the information stored in each pixel. They don't affect the number of pixels in the latent space or the pixel space.

[tolgacangoz]

Thanks for your answer; but I couldn't understand, could you elaborate?
VAE takes 3x1024×1024 and compresses it to 4x128x128. Then UNet works on a 4x128x128 tensor. Doesn't this mean 48 times compression? The blog post only evaluates the spatial compression part. Where is wrong in this reasoning? Could you tell me what the compression ratio is here?

[asomoza]

Maybe I'm the one not understanding what you're trying to say, but I don't think you can't mix the compression with the channels as you're doing it.

This text:

For a 1024×1024px image generated by SDXL, the latents tensor is 128×128px, where every pixel in the latent space represents 64 (8×8) pixels in the pixel space.

Just means that a 1024x1024 image gets converted to a 128x128 tensor in the latent space, that means that 1 pixel in the latent space is equivalent to 64 pixels (8x8) not taking into account what information each pixel contains or the number of channels, it could be 9 channels and it will still be 1 pixel = 64 pixels. The author never mentions that each value represents a pixel, just the equivalent pixel per pixel in each space.

Each pixel in the latent space has 4 channels and each channel means something completely different than the 3 channels in the RGB space, so they're not comparable in terms of information or compression, without the VAE the latents are pretty much useless except for previews.

For example, in the pixel space, if you convert a RGB to a RGBA, it just means that each pixel also has information about the transparency, the image will still have the same number of pixels.

Finally I not sure if we can call a 48 times compression, if you use full precision to encode the latents of an image and save them, you'll see that the compression is not even close to it, for example a 1.5MB png image gets saved as a 513kb file and at float16 it's gets saved as a 257kb file, so at most is a little over a 6x compression, this is obvious since even if we're making the tensor smaller we're adding a whole new dimension to it with a higher range of values in some channels.

Edit: Updated size values since I was saving the latents wrong