Problem updating neighborhood search on GPU: unsupported call to modify!()

[marklau34]

I was trying to play around with PointNeighbors on the GPU and was running into the following problem. Totally understand that GPU compatibility is still work in progress, so please let me know if this is expected, being worked on, and/or a duplicate of something referenced in another issue e.g. #26. If so, let me know if I should close this issue.

When attempting to run update!() on the GPU, I get an error:

ERROR: InvalidIRError: compiling MethodInstance for PointNeighbors.gpu_generic_kernel(::KernelAbstractions.CompilerMetadata{…}, ::PointNeighbors.var"#11#12"{…}) resulted in invalid LLVM IR
Reason: unsupported dynamic function invocation (call to modify!)

Here's a working example:

using PointNeighbors
using CUDA

# Generate some points
pts = rand(2, 100)
search_radius = 0.25
min_corner = (minimum(pts[1,:]), minimum(pts[2,:]))
maxcorner = (maximum(pts[1,:]), maximum(pts[2,:]))

# Create neighborhood search
cell_list = FullGridCellList(min_corner=min_corner, max_corner=maxcorner, search_radius=search_radius, 
                                backend=PointNeighbors.DynamicVectorOfVectors{Int32}, max_points_per_cell = 100)
nhs = GridNeighborhoodSearch{2}(search_radius=search_radius, n_points=size(pts,2), cell_list=cell_list, update_strategy=nothing)

# Initialize neighborhood search
PointNeighbors.initialize!(nhs, pts, pts)

# Copy neighborhood search onto GPU
nhs_gpu = PointNeighbors.Adapt.adapt(CuArray, nhs)

# Attemp to update points
pts_gpu = cu(pts) # copy points onto GPU
PointNeighbors.update!(nhs_gpu, pts_gpu, pts_gpu);

Please let me know if I'm doing something wrong here. I assumed the initialization had to happen on the CPU first before adapting to the GPU because I would get the common scalar indexing error, but maybe I am wrong.

Thanks in advance for any help!

[marklau34]

Perhaps defining this temporary workaround would work for now just so the code runs?

@inline function PointNeighbors.pushat_atomic!(vov::PointNeighbors.DynamicVectorOfVectors{<:Any, <:CUDA.CuDeviceArray}, i, value)
    (; backend, lengths) = vov
    @boundscheck checkbounds(vov, i)

    # Increment the column length with an atomic add to avoid race conditions.
    # Store the new value since it might be changed immediately afterwards by another
    # thread.
    new_length = CUDA.@atomic lengths[i] += 1

    # We can write here without race conditions, since the atomic add guarantees
    # that `new_length` is different for each thread.
    backend[new_length, i] = value

    return vov
end

[efaulhaber]

1. CUDA.@atomic doesn't work the same way. It returns the old value of lengths[i], so you need to use backend[new_length + 1, i] = value in the next line.
2. I could reproduce this error and investigated a bit. I tried updating and downgrading packages, but nothing worked. When I tried with Julia 1.10, I had to delete the Manifest.toml, and then things worked. When trying with 1.11 again, things still work, so I think it was indeed a package issue. Could you please try deleting your Manifest.toml and updating all packages?

[marklau34]

Wonderful! Both 1) and 2) solved it. After deleting Manifest.toml and updating all packages the redefinition of pushat_atomic!() wasn't needed anymore. Closing the issue because we figured out what was going on. Thanks for the help!