I made a patch to reduce artefatcs

[math-artist]

The title says it all. I wanted to increase the forward fill function with a 3D strength up to 10, because when placing the screen closer to our face, the perceived depth decrease. So, this is an attempt to get good depth at close range.

To do that, I patched the results of ordered_index_copy. The problem is that when the divergence is high, pixels can move and expand so much that they can skip a pixel indice. If that pixel is painted by an object under, it's not managed by the inpainting function.

The patch finds those isolated pixels with an arbitrary value:
gap = int(order_max * 0.1) and fill them with the neighbours of higher order.
I haven't really tested the performance impact but it works great on my side.

Also, the problem with the algo is that each source pixel is associated to only one destination pixel. If there was a way to conceptualize a different approach where one pixel could have more than one destination, we would not have those artefacts.

Because the patch only looks for isolated pixels, this prevent artefatcs for about 2X the divergence, but values higher than 10 will generate artefatcs.

def ordered_index_copy(c, src_index, dest_index, index_order, undefined_value=-1):
    B, _, H, W = c.shape
    c = c.permute(0, 2, 3, 1).reshape(-1, c.shape[1])
    if torch.is_tensor(undefined_value):
        out = undefined_value.view(1, -1).repeat(c.shape[0], 1)
    else:
        out = torch.empty_like(c).fill_(undefined_value)

    # Create order map in output space
    order_map = torch.empty((B, H, W), dtype=torch.long, device=c.device).fill_(-1)
    order_map.view(-1)[dest_index[index_order]] = torch.arange(len(index_order), device=c.device)

    deterministic = torch.are_deterministic_algorithms_enabled()
    torch.use_deterministic_algorithms(True)
    try:
        out.index_copy_(0, dest_index[index_order], c[src_index[index_order]])
    finally:
        torch.use_deterministic_algorithms(deterministic)

    out = out.view(B, H, W, -1).permute(0, 3, 1, 2)

    # Create masks for both conditions
    # First mask: undefined pixels surrounded by valid pixels
    undefined = out[:, 1, :, 1:-1] == -1
    has_valid_left = out[:, 1, :, :-2] != -1
    has_valid_right = out[:, 1, :, 2:] != -1
    mask1 = undefined & has_valid_left & has_valid_right

    # Second mask: pixels with lower order surrounded by higher order pixels
    order_max = order_map.max()
    gap = int(order_max * 0.1)
    order_center = order_map[:, :, 1:-1]
    order_left = order_map[:, :, :-2]
    order_right = order_map[:, :, 2:]
    has_lower_order = (order_center != -1) & (order_left != -1) & (order_right != -1) & \
                      (order_center < order_left - gap) & (order_center < order_right - gap)
    mask2 = has_lower_order

    # Combine masks
    mask = F.pad(mask1 | mask2, (1, 1), mode='constant', value=False)

    kernel = torch.tensor([[0.5, 0, 0.5]], dtype=out.dtype, device=out.device)
    kernel = kernel.view(1, 1, 1, 3).expand(4, 1, 1, 3)

    out_conv = F.conv2d(out, kernel, padding=(0, 1), groups=4)
    out = torch.where(mask.unsqueeze(1), out_conv, out)

    return out

[francdn]

This is for "forward fill" only, right?

[math-artist]

Yes, but row flow is supposed to be a model based on forward fill data, right? So the results are similar. I assume that this could be used to improve the training data to make row flow usable with higher values in the future.

[francdn]

You have to replace the function "ordered_index_copy" on the file "iw3/forward_warp.py". It does not give me any error and performance seems to be similar as before. I have not checked the results yet though.

[andy500]

I have replace the function "ordered_index_copy" but still taking over 2 hours to convert a 1 minute clip in 4k

[math-artist]

I saw no speed change on my end, but I was already using forward fill, which is slower than row flow. You may have a vram issue? Is it faster if you check the low VRAM checkbox?

[andy500]

I have replace the function "ordered_index_copy" but l am have a error name warp is not defined

[andy500]

it is working now VRAM was the issue with speed thank you

[hanweinjq]

I tested it with an intensity of 10, and the stereoscopic effect was greatly enhanced—it’s truly amazing!

[nagadomi]

Thank you. I assume that this patch fixes for this comment.

nunif/iw3/forward_warp.py

Line 19 in 3397eab

# TODO: If holes exist between different depth layers, they are not masked and may cause artifact

This type of artifact is a rare case when tested on images, but can occur frequently on video because of the large number of frames in the video.

Also, the problem with the algo is that each source pixel is associated to only one destination pixel. If there was a way to conceptualize a different approach where one pixel could have more than one destination, we would not have those artefacts.

I think backward warp(--method backward) applies to this.
However, PyTorch's backward warp (F.grid_sample) produces different type of artifacts due to the undefined behavior of the overlapped area.
In row_flow_v3, this type of artifact is reduced by eliminating complex overlaps before executing backward warp.

Here is illustration of iw3's image warp methods in my understanding

backward(grid_sample)

row_flow_v3

forward/forward_fill (depth ordered copy)

If 3D strength=10 is really needed, I will train a row_flow model for 3D strength=5-10 next time and switch the model to be used according to the setting.

[andy500]

I don't know if l have do the patch right can you show me what the forward_warp.py would look like with your patch thank you

[math-artist]

You just copy the code above and replace the ordered_index_copy function with it. If you do it in an IDE and there is no error then it's probably fine. This is essentially the original ordered_index_copy function, but with 2 masks and a kernel filter at the end of it. If you look at both on a split screen you will see the difference between both functions.

Maybe I should have done a PR instead, but since this patch alters the image slightly (it will round concave corners by 1 pixel) I tought it would be better to discuss it first.

[math-artist]

@hanweinjq You know,this does not add intensity, it was always possible. It just removes the vertical stripe artefacts that appear when the 3D strength is above 5.

[nagadomi]

@math-artist
For me, forward_fill is only used to create synthetic data for row_flow training. So I am not too concerned about compatibility and little processing speed change.
And, I think this change will make some parts better, but not worse, so if you send a PR I will merge it.
However, I might make a few changes on our end later. If anyone is using forward-fill for video conversion, I am planning to make (existing) inpaint a little better.

[andy500]

is the the ordered_index_copy function till l get to def warp(batch, width, height, c, x_index, index_shift, src_index, index_order):
floor_index, ceil_index, floor_weight, ceil_weight = make_bilinear_data(batch, width, height, x_index, index_shift)

[hanweinjq]

train a row_flow model for 3D strength=5-10 next time ,please!

[nagadomi]

@math-artist
I fixed forward_fill artifacts caused by layered holes in dev branch.
a1776a3
This should fix all layred holes.
The basic idea is that the source index of width(x axis) must be monotonically increasing, even after warping. Pixels that do not fit that condition are holes. This is pointed out by your patch.

You can try using the dev branch or overwriting the following file to forward_warp.py.
https://raw.githubusercontent.com/nagadomi/nunif/7dab3eb27f844cb8e8811f3a9456357ae018ebff/iw3/forward_warp.py

[hanweinjq]

train a row_flow model for 3D strength=5-10 next time ,please!

[beosliege]

@math-artist I fixed forward_fill artifacts caused by layered holes in dev branch. a1776a3 This should fix all layred holes. The basic idea is that the source index of width(x axis) must be monotonically increasing, even after warping. Pixels that do not fit that condition are holes. This is pointed out by your patch.

You can try using the dev branch or overwriting the following file to forward_warp.py. https://raw.githubusercontent.com/nagadomi/nunif/7dab3eb27f844cb8e8811f3a9456357ae018ebff/iw3/forward_warp.py

Hi @nagadomi ,
a1776a3, will it / should it move to master? I've been using dev branch assuming this is better, but correct me if I should not be.

Seems to be in dev:

user@NAME:/mnt/d/PythonProjects/nunif$ git checkout dev
Switched to branch 'dev'
Your branch is up to date with 'origin/dev'.

user@NAME:/mnt/d/PythonProjects/nunif$ git log --oneline | grep a1776a3
a1776a3 iw3: Fix layered hole artifact in forward_fill #274

user@NAME:/mnt/d/PythonProjects/nunif$ git branch --contains a1776a3

• dev

Not in master:

user@NAME:/mnt/d/PythonProjects/nunif$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

user@NAME:/mnt/d/PythonProjects/nunif$ git log --oneline | grep a1776a3

Thanks

[nagadomi]

@beosliege
dev branch will be merged into master at some point.
However, since there is unfinished work and I will be away for a few weeks starting the day after tomorrow, I am stopping the merge to avoid any problems in windows package.
For now, I've pushed the dev branch, which includes all changes in the master branch.

[andy500]

you will all was get artifacts in the background of your 3D The high you set your 3D strength because a depth map is one image movie studios create two images a foreground plate and a background plate which the 3d depth models can not do so with two plates you don't have a issue artifacts in the background you have separate the foreground and background so for me l set my 3d strength at 1.5 and l get a good depth 3d full tb and l input at 1080p and output at 4k l don't use forward_fill it is not as good as row_flow_v3_sym which l find better and to the person who what's you to include 3d 6-10 strength must understand the your program as limitation to it

[diverswan9]

I'm trying to understand this stuff. So what's the difference between @nagadomi's code and @math-artist's code? These are related to using...

Method? (which one?)
3D Strength? (what depths?)
Forward Fill? (a certain direction or none?)
Depth Model? (all or a certain one?)

These sound like they are for helping reduce artifacts but for which conditions?

[loawizard]

Would like to know this too

[nagadomi]

This change fixes a specific problem in forward_fill. It is a bug fix rather than an improvement. It does not fix inpainting quality or stretching.

input

with option --depth-model ZoeD_Any_N --divergence 10 --method forward_fill --edge-dilation 1

without patch

with (my) patch

filled this holes

Also, I think something is still wrong with my patch. (Is it misaligned?)

[gituser123456789000]

Thank you for providing results, with patch looks like a clear improvement

[loawizard]

should I be using the last dev version or should i use the code that @math-artist put up to get best effect at this moment?
should i use forword fill or can i also use row flow v3?
what 3depth should i use

[KolaKater]

now with the impainting models, it looks 99% perfect: