GPU utilization at large atomic scales

[Seunghyo-Noh]

Hi,

I have a question regarding GPU utilization for large-scale atomic systems when running grace-fs on a GPU.

When the number of atoms is small, the GPU seems to be utilized efficiently. However, as the system size increases, the GPU utilization drops noticeably.

Since the rattle() step is relatively fast, it appears that the main bottleneck in the total runtime comes from the grace-fs energy evaluation. In particular, as shown in the attached figure, the GPU does not seem to be continuously busy during the energy calculation. Instead, it appears to be used only intermittently, with repeated idle gaps between bursts of activity, which seems to reduce the overall efficiency.

I am wondering what causes this behavior.

For example:

• Could this be related to neighbor list construction for large systems?
• Is there any known bottleneck in the energy calculation path for large atom counts?
• My main question is how GPU utilization can be improved in this case.

I would appreciate any advice.

Best regards

Test code

from tensorpotential.calculator import TPCalculator, grace_fm
from ase.build import molecule
import numpy as np
import time

calc = grace_fm("GRACE-FS-OAM")

benzene = molecule("C6H6")
benzene.set_cell([[6, 0, 0], [0, 6, 0], [0, 0, 2]])
benzene.center()
benzene.calc = calc

system_size = 16
benzenes = benzene * [system_size, system_size, system_size]
benzenes.calc = calc

print("number_of_atoms:", len(benzenes))

for i in range(1000):
    t0 = time.perf_counter()
    benzenes.rattle()
    t1 = time.perf_counter()

    e = benzenes.get_potential_energy()
    t2 = time.perf_counter()

    print(
        i,
        e,
        "rattle_time=", t1 - t0,
        "energy_time=", t2 - t1
    )

Calculation results

number_of_atoms: 49152
0 -290843.59752837033 rattle_time= 0.00409810000564903 energy_time= 17.13231819999055
1 -290838.4689753966 rattle_time= 0.007699099980527535 energy_time= 6.416746700007934
2 -290829.99310491676 rattle_time= 0.007239300000946969 energy_time= 6.539110300014727
3 -290818.16925198934 rattle_time= 0.013635700015584007 energy_time= 6.361958299996331
4 -290802.99637909676 rattle_time= 0.013899599987780675 energy_time= 6.596613500005333
5 -290784.47307868826 rattle_time= 0.012828900013118982 energy_time= 6.228629199991701
6 -290762.5975761879 rattle_time= 0.003804200008744374 energy_time= 6.552064099989366
7 -290737.3677334663 rattle_time= 0.013860999984899536 energy_time= 6.354261500004213
8 -290708.78105277533 rattle_time= 0.01365879998775199 energy_time= 6.443483000010019
9 -290676.8346811471 rattle_time= 0.013648399995872751 energy_time= 6.243714500014903
10 -290641.52541525767 rattle_time= 0.004216400004224852 energy_time= 6.281998299993575
11 -290602.8497067569 rattle_time= 0.004405600018799305 energy_time= 6.423533099994529
...

[AntBoc]

Hi,

Since your energy timings are seem to be constant, the likely explanation is that the python overheads for moving data for 50k atoms is significantly larger than the compute time itself.
You can add compute_time = benzene.calc.eval_time to you list and will likely see that the number is significantly lower than the energy_time. Advice is to use LAMMPS for such large scale calculations.