Waifu2x CUnet vs. RealCUGAN

[ehobson]

Can you explain on a technical level the differences between the nunif cunet2x model and the cugan2x model? I see a lot of similarities between them but I've noticed some minor differences as well. Overall, nunif cunet2x is incredibly close to the lower denoising-level cugan variants (level 1, level 0, conservative). But it performs slightly differently and I've also noticed that some GUIs cannot handle it, while they have no problem with CUGAN. So, I'm curious to know more about the model.

[ehobson]

@nagadomi, This would be a question for you

[nagadomi]

I am not the author of CUGAN so I don't know much about it.

From https://github.com/bilibili/ailab/blob/main/Real-CUGAN/README_EN.md ,
CUGAN is a model trained waifu2x's CUNet architecture with Real-ESRGAN training code.
It is a GAN-based model, also trained with various types of noise and blur.

GAN-based models generate fake textures, lines etc., without much respect for the input image/original image, but it is supposed to have high perceptual quality for humans.

waifu2x's cunet is more conservative than that because it is optimized for PSNR (reconstruction loss).
waifu2x's cunet is aimed to denoise JPEG compression artifact only.

In nunif/waifu2x, swin_unet/art_scan is a GAN-based model similar to CUGAN and supports various types of denoising.

[ehobson]

RealCUGAN (conservative model) preserves the original colors slightly better than nunif cunet and is also moderately sharper. But it has a strange issue with generating artifacts around outlines - especially during scenes with a lot of motion - that I can't seem to fix. Nunif cunet also removes compression artifacts more efficiently than Cugan (conservative model).

The outline issue is the reason I switched from Cugan to Nunif CUnet. Actually, I came across multiple models with this issue, but Nunif Cunet was unique in that it did not produce those artifacts. That's why I was curious to know what it is doing differently.

But, aside from those minor differences, the outputs look remarkably similar.

[nagadomi]

waifu2x cunet intends to repair the color degradation caused by 4:2:0 chroma subsampling (which is used in JPEG and YUV420 video).
So, at high denoising levels, the colors may change from the source image, but it is intended to restore the “original” color (Not sure if that is accurate).

[ehobson]

Ah, that explains why the colors look a bit "brighter"/more vibrant than the original source file.

[nagadomi]

Also, if you are referring to converting videos, it is possible that the software you are using is incorrectly handling color_range or colorspace.
That is a very confusing part of video processing.
In nunif/waifu2x-gui, this should be handled properly with Colorspace=auto option.