about Inpainting models

[KolaKater]

How do you think about lama inpaint and ZITS inpaint?
It seems that they are better than forward_inpaint and mlbw_l2_inpaint

[nagadomi]

Regarding ZITS, I am not familiar with it, but Lama has been a strong reference for me.
For image models, a discriminator uses FFC. However, it does not seem that its effect is being utilized effectively, so there may be something wrong.
Video models are currently trained only with L1+LPIPS loss and do not use GANs.

Regarding the generator, FFT is a resolution-dependent module and If a model trained at 256×256 is run at HD, quality degrades significantly, and the inpainting boundaries become visible.
For this reason, it was not adopted. Practically, it is desirable for a model to work similarly at both 360p and 4K.
For now, instead, a Shifted Window mechanism — often used in tasks with arbitrary input resolution, such as super-resolution — is used.
However, it is possible to reduce FFT’s resolution dependence with techniques like tiling or shifted window.
At present, there is little perceived benefit in pursuing FFT to that extent.
As mentioned above, the effect feels less significant than expected, so it might be worth fully reproducing Lama to confirm whether my understanding and implementation of FFC are correct.

There are issues with directly using Lama’s architecture and pretrained models in iw3:

• Slow and high VRAM usage (my requirements prioritize speed and VRAM efficiency over accuracy).
• Not a video model, so flickering occurs.
• Inpainting for warp holes is different in domain from general inpainting tasks, so results are not optimal.
  (Warp holes follow foreground edges, with the right view having background on the right side, meaning domain knowledge allows for easier filling. Retraining would likely give better results.)

These are based on my intuition, not actual tests.
If this intuition cannot be trusted, it is recommended to test it yourself.

Regarding speed, some may think I care too much about performance, but video conversion itself is a very time-consuming process.
Most people already consider two hours to be a long time.
If the process becomes 30× or 5000× slower, it would be unacceptable.
Without actually testing, it is difficult to perceive the scale of this processing time issue.

[KolaKater]

If we keep the left-side image from the original and only generate the right-side view, we can avoid having two potentially incorrect images. This approach is faster and makes it easier for inpainting models to repair holes and artifacts. It would also simplify the training process for the inpainting models.
Of course, the perceived depth might be stronger when both the left and right images are generated from the 2D source. However, this approach doubles the processing time and also doubles the potential errors introduced during conversion and inpainting.

[nagadomi]

Yes, as I mentioned before, that option already exists.

There are advantages and disadvantages to both methods: one that synthesizes only the right view from the left source image, and another that synthesizes both views.

To achieve the same parallax, if only one eye is synthesized, the shift amount must be doubled. This means the hole size will also double, making warping and inpainting more difficult.
When both eyes are synthesized, the shift can be distributed between them, effectively halving it. This reduces both the difficulty and the distortion caused by shifting.

Additionally, people have a dominant eye, which can be either the left or the right depending on the individual. So all three modes—both eyes, right eye only, and left eye only—are supported.

As for training the inpainting model, it can be trained using only one view. To adapt it to the opposite view, the image can simply be flipped horizontally before inference and flipped back afterward. Therefore, it is unnecessary to train separate models for both views.

[nagadomi]

It was an overstatement to say that training separate models for both views is unnecessary.
In cases where the model also performs warping and handles left–right consistency, separate training may indeed be required.
However, the current implementation of iw3 does not do that.
When the task is only hole filling, each view’s missing regions are visible only from one eye, so they are treated independently.

[kaelsonofkrypto]

Hey Nagadomi, thanks for mlbw 12 inpainting, i'm curious, will you be releasing an inpainting version of mlbw 14 in the future?

[nagadomi]

I can support it in the future, but changing anything would require retraining everything, so until the experiments related to the inpaint are finished, I will use only l2.