feat(crashtracker): add Go crashtracker API#5026
Draft
kakkoyun wants to merge 16 commits into
Draft
Conversation
Add the compilable skeleton for the new crashtracker package: - Public API (Start/Stop/Option) stubs - Report, StackTrace, Frame, Thread, OSInfo, SigInfo structs (errorsintake model) - parse, upload, monitor stubs for parallel workstreams - Register DD_CRASHTRACKING_ENABLED and DD_CRASHTRACKING_IS_MONITOR_PROCESS via configinverter
Add the crashtracker package, which captures Go application crashes via runtime/debug.SetCrashOutput (Go 1.23+) and uploads structured reports to Datadog Error Tracking. Architecture: - Start() re-execs the current binary as a monitor child process. In the app process, SetCrashOutput pipes the runtime's crash dump to the child's stdin. In the monitor process, Start() hijacks control: it reads the pipe, parses the dump, uploads the report, and calls os.Exit — never returning. - GOMEMLIMIT and GOGC are filtered from the monitor child env to avoid inheriting the application's runtime tuning (see golang/go#73490). - Crash output reads are bounded at 32 MiB to handle large goroutine dumps. - Start() is idempotent via sync.Once; Stop() calls SetCrashOutput(nil). Report format: - Parses Go crash text into the errorsintake data model: error.type, error.message, structured error.stack (StackTrace with per-frame function/file/line), error.threads (flat []Thread, one per goroutine), os_info (architecture, os_type), and sig_info for signal crashes. - ddtags carries service/env/version/language:go/go.version/library_version and git metadata from ReadBuildInfo. Configuration: - DD_CRASHTRACKING_ENABLED: gate, default true (opt-out). - DD_CRASHTRACKING_IS_MONITOR_PROCESS: internal re-exec marker. - Standard DD_SITE / DD_API_KEY / DD_TRACE_AGENT_URL for routing. Upload: - Agent path: EVP proxy /evp_proxy/v2/api/v2/errors (TODO: confirm exact subdomain with Error Tracking team). - Agentless: site-templated https://errorsintake.agent.<site>/api/v2/errors.
- Add package init() to intercept monitor role before user init()s run, preventing app-level side-effects (DB, gRPC, signals) in the monitor - Add cmd.Wait() goroutine to reap the monitor child and avoid fd/zombie leak - Log upload failures to stderr so operators know when crash reporting fails
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: b6efd81 | Docs | Datadog PR Page | Give us feedback! |
Contributor
Config Audit |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…igurations DD_CRASHTRACKING_IS_MONITOR_PROCESS is an internal re-exec marker read directly via os.Getenv, not via env.Get. It is not a user-facing config and does not belong in supported_configurations.json. DD_CRASHTRACKING_ENABLED remains and requires registration in the internal configuration registry by a Datadog maintainer.
Member
Author
|
@codex review |
… 32-bit Passing bitSize 64 to strconv.ParseInt and then casting to int is unsafe on 32-bit platforms where int is 32 bits. bitSize 0 bounds the result to the native int width, making the int() conversion always safe.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6987c29b3e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Set DDTags on every crash report (was always empty) - Add UDS-aware client for unix:// agent URLs - Fix goroutineHeaderRe to parse GOTRACEBACK=system headers - Fix funcName to handle pointer-receiver methods (main.(*T).M) - Drain crash pipe after 32 MiB cap so crashing process is not blocked - Preserve first Start error; return it on subsequent idempotent calls - Add SIGTRAP (5) to signal number table - Document: monitor options are env-var only, GOTRACEBACK=all for full goroutine list, init order nuance
BenchmarksBenchmark execution time: 2026-07-15 13:25:03 Comparing candidate commit b6efd81 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 326 metrics, 0 unstable metrics, 1 flaky benchmarks without significant changes.
|
… 'A' .NET already registered this key at version A in the Feature Parity Dashboard. Using implementation C caused a mismatch; changing to A makes validate_supported_configurations_v2_local_file pass.
…ing them As an observability product, we should surface failures in Stop() rather than silently dropping them. SetCrashOutput and pipe close errors are now logged at WARN level via internal/log.
Baseline: ~7µs/parse, ~50 allocs/op on M4 Max across the three dump types (panic, concurrent-map-write, SIGSEGV). SetBytes enables MB/s throughput tracking for future comparisons.
- parse.go: check sc.Err() after splitDump scan loop; scanner truncation now returns an Incomplete report instead of silently partial data - parse.go: preallocate lines slice with len(dump)/80 capacity hint - monitor.go: log cmd.Wait() errors; monitor panics are now visible - monitor.go: extract buildChildEnv() helper; spawnMonitor drops from 63 to 45 lines, under the 60-line TigerStyle limit - parse_bench_test.go: add tb.Helper() to loadFixture
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…eguard The codebase uses err.Error() not bare err with %v in log calls. Matches the pattern in profiler/ and ddtrace/tracer/.
…eline TestE2ECrashReport_Panic: spawns a crash-victim subprocess that panics with crashtracker.Start() active; waits for the monitor grandchild to POST an errorsintake report to a mock server; asserts ddsource, error.type, error.is_crash, error.stack.frames, error.threads, one crashed goroutine, os_info.architecture, and ddtags language:go. TestE2ECrashReport_CleanExit: verifies clean exit produces no report. Uses the TestMain subprocess-role pattern: the test binary re-execs itself with _CRASHTRACKER_E2E set; crashtracker.init() intercepts the monitor grandchild role before TestMain runs.
hannahkm
reviewed
Jul 14, 2026
hannahkm
left a comment
hannahkm
left a comment
Contributor
There was a problem hiding this comment.
I did a really quick skim-through of the code! Thanks for making the first implementation. Two small nits, and also we should update the READMEs and AGENT files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
crashtrackerpackage that captures Go application crashes and uploads structured reports to Datadog Error Tracking. The feature covers unhandled panics, runtime fatals (concurrent map writes, stack exhaustion), and signal-triggered crashes (SIGSEGV, SIGABRT).How it works:
The package uses
runtime/debug.SetCrashOutput(Go 1.23+) and the monitor-process pattern:Start()re-execs the current binary as a lightweight monitor child (identified byDD_CRASHTRACKING_IS_MONITOR_PROCESS=1).init(), before any userinit()functions run, so app-level side-effects (DB connections, gRPC dials) don't fire in the monitor.SetCrashOutput(pipeWriteEnd), wiring the runtime's crash dump to the monitor's stdin.Configuration:
DD_CRASHTRACKING_ENABLED: enables/disables crash reporting (default:true).DD_SITE,DD_API_KEY,DD_TRACE_AGENT_URLfor routing.Report format:
Structured errorsintake payload:
error.type,error.message,error.stack(per-frame function/file/line),error.threads(flat goroutine list),os_info,sig_infofor signal crashes, andddtagswith service/env/version/language:go.Upload goes via the Agent EVP proxy (
/evp_proxy/v2/api/v2/errors) or directly to the site-templated intake whenDD_API_KEYis set. Two TODOs remain for the Error Tracking team to confirm the exact intake path.Usage:
Test plan
buildDDTagsoutput shape, idempotentStart(), upload againsthttptest.Server(agent path, agentless path, 5xx error)-race -count=1golangci-lint run ./crashtracker/...— 0 issuesgo run ./scripts/configinverter/main.go check— in sync