test: coarsen ASan sampling rate/workload to stop nightly heap OOMs#653
Merged
Conversation
Fixed-interval CPU/wall sampling (100us/1ms) over an iteration-bounded workload scales unboundedly under ASan (signal handling itself is instrumented), and each test materializes a String per sample, OOMing the ASan test-runner heap. Sample coarser and shrink the workload under isAsan(), matching the pattern already used in BoundMethodHandleProfilerTest.
![dd-octo-sts[bot]](https://avatars.githubusercontent.com/in/1157446?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 869ba8d074
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
CI Test ResultsRun: #29244926453 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-13 11:56:47 UTC |
Unlike the other tests here, this one was never observed failing in CI, already samples at a coarser cpu=1ms, and drives pure-Java virtual dispatch that ASan doesn't slow down — stacking a 10x coarser interval with a 10x smaller workload risked starving the narrow vtable-receiver stack-pattern search instead of fixing a real problem.
Contributor
Benchmark Results (commit 869ba8d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124278010 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10386 ms (21 iters) | ✅ 10314 ms (21 iters) | ≈ -0.7% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8896 ms (24 iters) | ✅ 8760 ms (24 iters) | ≈ -1.5% (±9.9%) | — / — |
| finagle-chirper | 21 | ✅ 5960 ms (33 iters) | ✅ 6003 ms (33 iters) | ≈ +0.7% (±25.4%) | |
| finagle-chirper | 25 | ✅ 5473 ms (36 iters) | ✅ 5491 ms (36 iters) | ≈ +0.3% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2694 ms (70 iters) | ✅ 2766 ms (67 iters) | ≈ +2.7% (±2.8%) | — / — |
| fj-kmeans | 25 | ✅ 2822 ms (66 iters) | ✅ 2828 ms (66 iters) | ≈ +0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2153 ms (87 iters) | ✅ 2047 ms (90 iters) | 🟢 -4.9% | — / — |
| future-genetic | 25 | ✅ 2021 ms (92 iters) | ✅ 2048 ms (90 iters) | ≈ +1.3% (±2.8%) | — / — |
| naive-bayes | 21 | ✅ 1326 ms (129 iters) | ✅ 1266 ms (135 iters) | ≈ -4.5% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1003 ms (170 iters) | ✅ 1010 ms (169 iters) | ≈ +0.7% (±31.7%) | — / — |
| reactors | 21 | ✅ 16276 ms (15 iters) | ✅ 16443 ms (15 iters) | ≈ +1% (±8.6%) | — / — |
| reactors | 25 | ✅ 18707 ms (15 iters) | ✅ 18656 ms (15 iters) | ≈ -0.3% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1952 / 2049 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2264 / 2187 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8544 / 8822 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8090 / 8333 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1279 / 1263 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1291 / 1283 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2901 / 2958 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2873 / 2840 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 3543 / 3578 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 5 | 3470 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1592 / 1528 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1897 / 1899 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 1f3b41e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124281533 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Coarsens the CPU/wall sampling interval and shrinks the driven workload, under ASan only, in four tests that were producing an unbounded number of profiler samples under ASan and then materializing a
Stringstack trace for every single one of them:MegamorphicCallTest(wall=100us→wall=1ms; 1M → 100k iterations)MetadataNormalisationTest(cpu=100us→cpu=1ms; 100k → 10k iterations)CTimerSamplerTest(cpu=100us→cpu=1ms)LightweightContextCpuTest(cpu=100us→cpu=1ms)Non-ASan behavior/rates are unchanged.
VtableReceiverFrameTestwas initially included in this set but the change was reverted after review (discussion): it was never observed failing in CI, already sampled at a coarsercpu=1ms, and drives pure-Java virtual dispatch that ASan doesn't meaningfully slow down — stacking a 10x coarser interval with a 10x smaller workload risked starving its narrow.vtable stub()/vtable_receiverpattern search instead of fixing a real problem.Motivation:
Nightly
test-linux-glibc-aarch64 (*, asan)runs have been failing deterministically withJava heap spacein the Gradle test executor for at least 4 consecutive nights (e.g. run 29223441349), always breaking atMegamorphicCallTest.testITableStubs().Root cause: these tests use a very tight, fixed signal-based sampling interval (e.g.
wall=100us) over a workload bounded by a fixed iteration count rather than a fixed wall-clock duration. Under ASan, signal handling itself is instrumented and much slower, so the same fixed-rate sampler collects a disproportionately larger number of samples (242,177 in the confirmed failing run) — and each test then builds a JavaStringstack trace for every sample to search for a substring, which exhausts the ASan test JVM's-Xmx1024mheap.Two prior fixes (#632 forkEvery, #640 raising
-Xmx512m→1024m) only widened the margin without bounding the sample count, so the flake persisted. This applies the pattern already established forBoundMethodHandleProfilerTest(#630/#533): sample coarser and shrink the workload specifically underisAsan().I audited every other test using a similarly tight sampling interval; the four listed above are the ones confirmed or strongly suspected to combine a tight interval, an iteration-bounded (not time-bounded) workload, and full per-sample materialization, without a countervailing reason (like
VtableReceiverFrameTest's) to leave them alone.Additional Notes:
N/A
How to test the change?:
Ran
:ddprof-test:compileTestJavato confirm the changes compile; the actual regression only reproduces under ASan on aarch64 in CI, so the real validation is the next few nightlyNightly Sanitized Runexecutions no longer failing withJava heap space.For Datadog employees:
Unsure? Have a question? Request a review!