Art Style Caused by VAE Latent Representation?

[berpat01]

I have adapted a script (https://github.com/zyinghua/uncond-image-generation-ldm) to train an unconditional Latent Diffusion Model with a pretrained VAE from Hugging Face (https://huggingface.co/stabilityai/sd-vae-ft-mse). I'm using the CelebA-HQ dataset (256x256). The VAE was finetuned to improve the reconstruction of faces and it performs very well in that regard — reconstructions look great. However, the samples generated by the LDM tend to resemble paintings rather than realistic faces:

When I train my own VAE on CelebA-HQ and use it to train the LDM, the generated samples look realistic, as expected. Could this "Art-Style" of the samples be caused by the latent representations of the VAE, even though its reconstructions are high quality?

I haven’t found any information on this effect — any guidance would be greatly received!