feat: AdePT physics list plugin for e+/e-/gamma offload to GPU [WIP] by wdconinc · Pull Request #1606 · AIDASoft/DD4hep

wdconinc · 2026-04-11T17:14:36Z

BEGINRELEASENOTES

feat: AdePT physics list plugin for e+/e-/gamma offload to GPU

ENDRELEASENOTES

This PR adds an AdePT physics list plugin for DD4hep (similar to the celeritas physics list plugin in https://github.com/celeritas-project/celeritas/).

Notes on AdePT integration approach:

AdePT is added as a Geant4AdePTPhysics action, and must be added with a helper function to setupUserPhysics in DDsim. This is added through the steering file.
The use of callUserTrackingAction=true is required, since we need a Geant4AdePTUserParticleHandler to 'repair' the track/particle after it comes back from the GPU to the CPU. This is also added by the steering file.
The rest of the example steering file only serves to make it self-consistent and runnable, but does not have AdePT functionality.

Notes on DD4hep core changes:

SiD.xml gets an EcalRegion for testing.
In Geant4ParticleHandler, the loop over user handlers is moved earlier to allow them to 'repair' m_currTrack:
https://github.com/wdconinc/DD4hep/blob/a70d61afc812dc027990f29f8a351274fae11d59/DDG4/src/Geant4ParticleHandler.cpp#L349-L350
To accept the new AdePT particle handler, we allow in DDSim/Helper/ParticleHandler.py‎ the passthrough of a generic particle handler.

MarkusFrankATcernch

We do not want to make plugin headers public.
If necessary, we can put the header in a public directory and define the factory instance elsewhere....
There is a good reason to separate plugins from code.

github-actions · 2026-04-11T18:11:04Z

Test Results

18 files 18 suites 6h 4m 48s ⏱️
357 tests 354 ✅ 0 💤 3 ❌
3 143 runs 3 116 ✅ 0 💤 27 ❌

For more details on these failures, see this check.

Results for commit b2ddfbd.

♻️ This comment has been updated with latest results.

wdconinc · 2026-04-11T18:24:29Z

We do not want to make plugin headers public.
If necessary, we can put the header in a public directory and define the factory instance elsewhere....
There is a good reason to separate plugins from code.

Yes, I noticed when filing the PR that the unrelated DDCore change snuck in. I'll remove it when I am back at a computer.

andresailer · 2026-04-13T08:47:13Z

cc @SeverinDiederichs (FYI)

wdconinc · 2026-04-13T20:59:49Z

Together with apt-sim/AdePT#546 this now offloads tracks to my (very crappy) GPU.

ddsim --steeringFile DDG4/examples/AdePTSteeringFile.py --compactFile /opt/local/DDDetectors/compact/SiD.xml

When AdePT's callUserTrackingAction=false (the default for performance), GPU-produced hits and hadronic secondaries carry trackID/parentID=0 from the dummy HostTrackData. This caused two classes of errors: 1. 'No Equivalent particle for track:0' (from Geant4ParticleMap::particleID) GPU hits have trackID=0, and when Geant4Output2ROOT tries to remap them to final particle IDs, it calls particleID(0) which fails. Fix: in Geant4ParticleHandler::endEvent(), after rebaseSimulatedTracks, add m_equivalentTracks[0] pointing to the primary particle (g4id=1) so that all GPU hits with dummy trackID=0 are correctly attributed. 2. Hadronic secondaries returned from GPU with parentID=0 breaking the MC truth parent chain. Fix: Geant4AdePTUserParticleHandler::begin() remaps particle.g4Parent from 0 to the entering primary's G4 track ID. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ePTPhysics LastNParticlesOnCPU: when the in-flight count drops below this threshold the remaining particles are leaked back to Geant4/HepEm on CPU, terminating the GPU transport loop early. Setting this to a small value (e.g. 10-100) avoids launching many near-empty kernels during the long shower tail. Default 0 preserves the previous behaviour (always finish on GPU). SpeedOfLight: debug/benchmark mode that kills all e-/e+/gamma immediately without tracking them (equivalent to setting their mean free path to zero). Useful for measuring geometry or non-EM overhead in isolation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document and expose the property in the example steering file with a comment explaining its effect on GPU kernel launch efficiency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

In ddsim-mt (PR#1240), Geant4ParticleHandler is created inside __setupGeneratorActions, which runs as a UserInitialization callback during G4RunManager::Initialize() -- after setupPhysics has returned. The previous approach of looking up the handler via kernel.generatorAction().get('ParticleHandler') inside setup_physics no longer works: the object either doesn't exist yet or its adopt() method is not available on the returned wrapper. The FIXME in ParticleHandler.py noted that setupUserParticleHandler was not extensible: it hardcoded only Geant4TCUserParticleHandler and Geant4TVUserParticleHandler and called exit(1) for anything else. Add an 'else' branch that supports arbitrary DDG4 action plugin class names: create the action and call part.adopt(user) without any special tracker-region configuration. This allows plugins such as Geant4AdePTUserParticleHandler to be registered simply via: runner.part.userParticleHandler = "Geant4AdePTUserParticleHandler" Update AdePTSteeringFile.py to use this clean mechanism in place of the monkey-patch workaround introduced in the previous commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

As a defense-in-depth measure alongside the primary fix in AdePT's HostTrackDataMapper (cpuAncestorG4id propagation), force any track whose G4 track ID is in the GPU-assigned range (>= INT_MAX/2, counting down from INT_MAX) to have G4PARTICLE_ABOVE_ENERGY_THRESHOLD set in particle.reason. This ensures such tracks enter the m_particleMap if-branch in Geant4ParticleHandler::end() rather than the else-branch that walks the parent chain and emits 'FATAL: No real particle parent present' when the chain is broken by an unregistered GPU-assigned parent ID. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When AdePT returns GPU-processed tracks to CPU, the returned tracks carry GPU-assigned track IDs (counting down from INT_MAX). When G4HepEm then handles these tracks and produces hadronic secondaries (e.g. photo-nuclear products W182/W183/W184/W186, neutrons, protons) inline -- before the GPU track PostUserTrackingAction fires -- those secondaries have a GPU-range parentID that is not yet registered in m_particleMap/m_equivalentTracks, causing "FATAL: No real particle parent present" errors. Fix in Geant4AdePTUserParticleHandler: - In begin(): if track->GetParentID() is GPU-range, resolve g4Parent by looking up the parent in m_trackCache (populated when the GPU parent itself began). This replaces the GPU parent ID with the CPU ancestor ID. - In end() fallback: if track->GetParentID() is GPU-range, apply the same cache-based resolution rather than blindly using GetParentID(), which would undo the begin()-time fix for hadronic secondaries not in m_trackCache. - In end(): GPU-assigned track IDs are forced into m_particleMap (if-branch) via G4PARTICLE_ABOVE_ENERGY_THRESHOLD so they are always registered. - Update cache on end() instead of erasing, so a second end() call (for tracks that re-enter the GPU region) can still restore correct state. Fix in Geant4ParticleHandler (core, minimal): - Add cycle detection (std::set<int> visited) in the m_equivalentTracks walk in end() and rebaseSimulatedTracks() to prevent infinite loops if a self-referential entry is created by any remaining edge case. Also update AdePTSteeringFile.py to use adequate slot sizes (10M) for realistic simulation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

wdconinc · 2026-04-28T21:29:21Z

(Copilot got ahead of itself, merged ddsim-mt into this, and pushed it up. I'll revert this.)

Fixed up so these commits are all 'unique', but the AdePT functionality will require merging into ddsim-mt first since running with AdePT on a single thread is not performant.

wdconinc mentioned this pull request Apr 11, 2026

g4adept: new package spack/spack-packages#4150

Merged

2 tasks

MarkusFrankATcernch reviewed Apr 11, 2026

View reviewed changes

wdconinc force-pushed the adept branch 2 times, most recently from 62336b6 to 06dc416 Compare April 15, 2026 17:01

This comment was marked as resolved.

Sign in to view

wdconinc and others added 16 commits April 28, 2026 16:02

feat: AdePT physics list plugin for e+/e-/gamma

55402a9

fix: lower slots; nthread fixes; correct AdePT::G4Integration linkage

032182e

fix(cmake): revert the core plugin header installation

d13c9b3

feat: add EcalRegion to SiD (EcalBarrel, EcalEndcap)

c44c6d6

feat: add complete set of properties to example

c6f2818

DDG4: add LastNParticlesOnCPU to AdePTSteeringFile.py

68f5a0f

Document and expose the property in the example steering file with a comment explaining its effect on GPU kernel launch efficiency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: always use CallUserTrackingAction=true

80444c7

fix: use both Geant4AdePTUserParticleHandler::begin/end

e99dee1

fix: mv user end calls before reason use in Geant4ParticleHandler

21e3bbf

fix: rm unreachable code now that callUserTrackingAction=true

726e548

fix: mv PropertyMask ref handler too

6db8911

wdconinc force-pushed the adept branch from 5bae5b7 to 626f8b4 Compare April 28, 2026 21:28

fix: ensure AdePT is correctly available to downstream DD4hep project

b2ddfbd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: AdePT physics list plugin for e+/e-/gamma offload to GPU [WIP]#1606

feat: AdePT physics list plugin for e+/e-/gamma offload to GPU [WIP]#1606
wdconinc wants to merge 17 commits into
AIDASoft:masterfrom
wdconinc:adept

wdconinc commented Apr 11, 2026 •

edited

Loading

Uh oh!

MarkusFrankATcernch left a comment •

edited

Loading

Uh oh!

github-actions Bot commented Apr 11, 2026 •

edited

Loading

Uh oh!

wdconinc commented Apr 11, 2026

Uh oh!

andresailer commented Apr 13, 2026

Uh oh!

wdconinc commented Apr 13, 2026

Uh oh!

This comment was marked as resolved.

wdconinc commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wdconinc commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarkusFrankATcernch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

wdconinc commented Apr 11, 2026

Uh oh!

andresailer commented Apr 13, 2026

Uh oh!

wdconinc commented Apr 13, 2026

Uh oh!

This comment was marked as resolved.

wdconinc commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wdconinc commented Apr 11, 2026 •

edited

Loading

MarkusFrankATcernch left a comment •

edited

Loading

github-actions Bot commented Apr 11, 2026 •

edited

Loading