Implement Flip transforms with CVCUDA backend

[zy1git]

Summary:
Implemented _horizontal_flip_image_cvcuda and _vertical_flip_image_cvcuda kernels using cvcuda.flip operator. The kernels are automatically registered when CVCUDA is available and route cvcuda.Tensor inputs appropriately.

Test Plan:

• Added test_functional_cvcuda and test_image_correctness_cvcuda tests
• Verified parity between PyTorch and CVCUDA implementations
• All tests pass with CVCUDA backend

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9277

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3eb5cb6 with merge base dccf466 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[justincdavis]

@zy1git What is the strategy for creating the tests for the transforms with CV-CUDA backends? Do we want to have all the tests live entirely inside the existing classes or make a new class?

The PRs for gaussian_blur, normalize, and to_dtype I made all use new classes, but I can switch it be more centralized.

[AntoineSimoulin]

Thanks a lot for submitting this PR! This is looking good. I added some comments to make sure we have an extensive test coverage:)

[NicolasHug]

@justincdavis replying to your question in #9277 (comment): we prefer centralizing the tests in the existing test class. The idea is that, as much as possible, we'd just add CV-CUDA as a parametrization entry with pytest.mark.parametrize to the existing tests. Antoine and I left a few comments related to that above. Does that make sense?

[justincdavis]

@NicolasHug Makes sense! I will follow the comments you and Antoine left on this PR

[AntoineSimoulin]

Thanks a lot for addressing the initial comments! I left some final adjustments to make. Let's also make sure linting and tests are passing!

[justincdavis]

@AntoineSimoulin @NicolasHug @zy1git

I would propose that we take an approach like this to simplify the comparison of CV-CUDA tensors with PIL references.

We would add the following function to common_utils.py

def cvcuda_to_pil_compatible_tensor(tensor):
    tensor = cvcuda_to_tensor(tensor)
    if tensor.ndim != 4:
        raise ValueError(f"CV-CUDA Tensor should be 4 dimensional. Got {tensor.ndim} dimensions.")
    if tensor.shape[0] != 1:
        raise ValueError(f"CV-CUDA Tensor should have batch dimension 1 for comparison with PIL.Image.Image. Got {tensor.shape[0]}.")
    return tensor.squeeze(0).cpu()

Then we would modify ImagePair as follows

class ImagePair(TensorLikePair):
    def __init__(
        self,
        actual,
        expected,
        *,
        mae=False,
        **other_parameters,
    ):
        if all(isinstance(input, PIL.Image.Image) for input in [actual, expected]):
            actual, expected = (to_image(input) for input in [actual, expected])
        elif CVCUDA_AVAILABLE and all(isinstance(input, _import_cvcuda().Tensor) for input in [actual, expected]):
            actual, expected = (cvcuda_to_tensor(input) for input in [actual, expected])
        elif CVCUDA_AVAILABLE and isinstance(actual, _import_cvcuda().Tensor) and isinstance(expected, PIL.Image.Image):
            actual = cvcuda_to_pil_compatible_tensor(actual)
            expected = to_image(expected)
        elif CVCUDA_AVAILABLE and isinstance(actual, _import_cvcuda().Tensor):
            actual = cvcuda_to_pil_compatible_tensor(actual)

        super().__init__(actual, expected, **other_parameters)
        self.mae = mae

Then when we compare the actual tensor (cvcuda.Tensor) to the expected (PIL.Image.Image), the conversion will be handled automatically for us.

Additionally, with the helper function cvcuda_to_pil_compatible_tensor, we can simplify the logic for handling CV-CUDA specific comparisions. For example, in TestRgbToGrayscale::test_image_correctness, the logic for handling CV-CUDA goes from:

if make_input is make_image_cvcuda:
        actual = F.cvcuda_to_tensor(actual).to(device="cpu")
        actual = actual.squeeze(0)
        # drop the batch dimension
        image = F.cvcuda_to_tensor(image).to(device="cpu")
        image = image.squeeze(0)

to

# here the conversion of actual is handled in either assert_close or assert_equal itself
if make_input is make_image_cvcuda:
        image = cvcuda_to_pil_compatible_tensor(image)

These changes are integrated in this PR

[NicolasHug]

Thanks for the great work @zy1git ! I made another pass.

[NicolasHug]

This seems unnecessary, we should be able to compare tensors where the batch dim is 1. Try to remove it, if it doesn't work for any reason let me know.

[NicolasHug]

EDIT: ah, OK, it's for when we compare a 3D PIL image to a 4D cvcuda tensor. That's... fine. Let's explain why then (addition in bold):

Remove batch dimension if it's 1 for easier comparison against 3D PIL images

[justincdavis]

Are we able to split the logic to drop batch and move to cpu into a helper function? We do the same thing in the test itself, and I think it would improve clarity to have an explicit helper.

Line in reference

[NicolasHug]

It's a 2-line helper, I'd say let's leave it out for now. We might need it later, but it'll be more obvious to me when I get to see more code and more usage. There might be some refactoring opportunities that you're seeing @justincdavis and that I'm not yet seeing - sorry for that, I'm sure things will be easier to decide for me once I've reviewed more of the coming PRs.

[abhi-glitchhg]

is it feature still in prototype state? can we also add other transforms like resizing others?
i couldnt find any issue about the cv-cuda backend features so maybe im missing something or if its internal
thank you.

[NicolasHug]

@abhi-glitchhg This will likely be released as Beta. Either in the next version in late Jan, or the following one. We do plan to add more transforms support, we just need to make sure that the output results are the same with our current tensor GPU backend.

[justincdavis]

Maybe a bit of a nitpick, but could we rename the function to _horizontal_flip_cvcuda, CV-CUDA only operates on one datatype so the extra "image" in the funcname does not add value IMO. Removing it also mirrors the cvcuda_to_tensor and tensor_to_cvcuda functions

[NicolasHug]

the cvcuda_to_tensor and tensor_to_cvcuda functions are a bit of outliers in that sense, but most other kernels specify the nature of the input they work on. We have e.g.

• horizontal_flip_image for tensors and tv_tensor.Image
• _horizontal_flip_image_pil
• horizontal_flip_mask
• horizontal_flip_bounding_boxes
• etc.

The CVCUDA backend is basically of the same nature as the PIL backend. So It makes sense to keep it named _horizontal_flip_cvcuda (EDIT: meant _horizontal_flip_image_cvcuda!!) IMO., like we have _horizontal_flip_image_pil.

[AntoineSimoulin]

@NicolasHug just to be sure, you are saying it makes sense to keep it named _horizontal_flip_cvcuda, I guess you mean it makes sense to keep it named _horizontal_flip_image_cvcuda?

[NicolasHug]

Yes, thanks for catching! I'll edit above to avoid further confusion

[abhi-glitchhg]

can i help implementing some transforms? cc @justincdavis as you are handling most of the prs regarding cv-cuda.

[NicolasHug]

Thank you so much for offering to help @abhi-glitchhg ! We would have loved to make this a community-driven project, like the various ones you participated in in the past, but for this one we're directly partnering with the CV-CUDA folks from NVidia. @justincdavis has already implemented almost all transforms (see current PRs + many in the backlog), and what's left is for us to review the PRs.

I'll definitely let you know in the future if there are any interesting projects that could be in scope for the community. Thanks again!

[NicolasHug]

Oh, oops, that's a hard dependency and it crashes on the jobs where CVCUDA isn't available. Let's hack something for now, we'll think of a better way to handle that later:

Suggested change

	cvcuda.Tensor,
	None,

Then in the code below:

def test_functional_signature(self, kernel, input_type):
    if kernel is F._geometry._horizontal_flip_image_cvcuda:
        input_type = _import_cvcuda().Tensor

[justincdavis]

I have been using the string "cvcuda.Tensor" and then checking input_type == "cvcuda.Tensor". If we want some kind of better engineered solution as opposed to None or strings, we could make a cvcuda_tensor type which is None if CV-CUDA isn't installed.

Maybe something like:

cvcuda_tensor = None
if CVCUDA_AVAILABLE:
    cvcuda_tensor = _import_cvcuda().Tensor

This at least keeps the naming consistent and drops the if statements in the signature tests.

[AntoineSimoulin]

Hey @zy1git, this is looking good! Just leaving a small comments here, let me know what you think:)

[AntoineSimoulin]

Is this import needed? I feel we are not using cvcuda module directly anywhere in this file?

[AntoineSimoulin]

This is looking good to me. Just left a last comment regarding imports in torchvision/transforms/v2/_geometry.py‎.

[NicolasHug]

Thank you for the great effort @zy1git !

@justincdavis @AntoineSimoulin thank you so much for your help with the review and your insightful comments! Let's merge this, we have a quick test follow-up to do but this can be done in a separate PR (I'll describe that below)

[github-actions]

Hey @NicolasHug!

You merged this PR, but no labels were added.
The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

[NicolasHug]

So looking at the logs of the cvcuda job (e.g. from the hud) you'll see that some of the new cvcuda tests are being skipped.

The reason is: they are missing the needs_cuda pytest mark.

We have a mechanism in place (link) which:

• skips the needs_cuda tests on CPU CI instances
• skips the cpu tests on GPU CI instances.

The new cvcuda CI job is a GPU CI instance. It skips the CPU tests. But since some of the CV-CUDA tests aren't marked with needs_cuda they are considered to be CPU tests and they are then skipped.

Some of the newly added tests here implicitly have the needs_cuda mark because they rely on the

 @pytest.mark.parametrize("device", cpu_and_cuda())

parametrization, which will be adding the needs_cuda mark when needed. But for the tests that do not use that, they're missing the mark and we need to add it.

I think what we can do is, for the tests that are currently using

marks=pytest.mark.skipif(not CVCUDA_AVAILABLE, reason="CVCUDA is not available"),

we should do something like:

CVCUDA_MARK = (pytest.mark.skipif(not CVCUDA_AVAILABLE, reason="CVCUDA is not available"), needs_cuda)
marks=CVCUDA_MARK

(not sure it'll work as-is, but you get the idea.

I hope that makes sense, let's chat offline if needed.