-
Notifications
You must be signed in to change notification settings - Fork 536
feat(crashtracker): orchestrion auto-injection aspect #5027
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
kakkoyun
wants to merge
6
commits into
feat/crashtracker-api
Choose a base branch
from
feat/crashtracker-orchestrion
base: feat/crashtracker-api
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
7655e43
feat(crashtracker): add orchestrion auto-injection aspect
kakkoyun e4ba9a4
test(crashtracker): add orchestrion e2e test for crash injection aspect
kakkoyun 2b2f86f
fix(crashtracker): orchestrion e2e explicitly calls Start() in subpro…
kakkoyun fe5ca42
docs(crashtracker): document ordering relative to tracer/profiler in …
kakkoyun 360b3e1
Merge branch 'feat/crashtracker-api' into feat/crashtracker-orchestrion
kakkoyun f083758
fix(crashtracker): harden orchestrion startup
kakkoyun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Unless explicitly stated otherwise all files in this repository are licensed | ||
| # under the Apache License Version 2.0. | ||
| # This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| # Copyright 2016-present Datadog, Inc. | ||
| --- | ||
| # yaml-language-server: $schema=https://datadoghq.dev/orchestrion/schema.json | ||
| meta: | ||
| name: github.com/DataDog/dd-trace-go/v2/crashtracker | ||
| description: |- | ||
| Starts the Datadog crashtracker as the first statement of main(), so the | ||
| monitor process is established before any goroutine can crash. The injected | ||
| start uses DD_* environment configuration; a later programmatic Start call | ||
| in main is a no-op because crashtracker keeps the first successful start. | ||
|
|
||
| aspects: | ||
| - id: func main() | ||
| join-point: | ||
| all-of: | ||
| - package-name: main | ||
| - test-main: false | ||
| - function-body: | ||
| function: | ||
| - name: main | ||
| - signature: {} | ||
| advice: | ||
| - prepend-statements: | ||
| imports: | ||
| crashtracker: github.com/DataDog/dd-trace-go/v2/crashtracker | ||
| log: log | ||
| template: |- | ||
| if err := crashtracker.Start(); err != nil { | ||
| log.Printf("failed to start crashtracker: %s", err.Error()) | ||
| } | ||
148 changes: 148 additions & 0 deletions
148
internal/orchestrion/_integration/crashtracker/crashtracker.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| // Unless explicitly stated otherwise all files in this repository are licensed | ||
| // under the Apache License Version 2.0. | ||
| // This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| // Copyright 2016-present Datadog, Inc. | ||
|
|
||
| package crashtracker | ||
|
|
||
| import ( | ||
| "context" | ||
| "encoding/json" | ||
| "io" | ||
| "net/http" | ||
| "net/http/httptest" | ||
| "os" | ||
| "os/exec" | ||
| "strings" | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/DataDog/orchestrion/runtime/built" | ||
|
|
||
| "github.com/DataDog/dd-trace-go/v2/internal/orchestrion/_integration/internal/trace" | ||
| ) | ||
|
|
||
| const ( | ||
| // e2eRoleEnv drives subprocess behaviour in TestMain. | ||
| e2eRoleEnv = "_CRASHTRACKER_E2E_ORCH" | ||
|
|
||
| // crashRoleOrch is the panic-victim role. | ||
| crashRoleOrch = "panic" | ||
|
|
||
| // orchCrashMsg is the panic string asserted in the received crash report. | ||
| orchCrashMsg = "orchestrion e2e crash" | ||
| ) | ||
|
|
||
| // TestCase is the orchestrion integration test for the crashtracker aspect. | ||
| // | ||
| // It verifies the full crash-reporting chain under an orchestrion-built test | ||
| // binary: a subprocess starts crashtracker explicitly, panics, and the monitor | ||
| // child uploads a structured report to the mock intake. The aspect deliberately | ||
| // excludes test binaries' main functions with test-main:false, so injection into | ||
| // real non-test main functions is validated by TestCrashtrackerMainInjection. | ||
| type TestCase struct { | ||
| mockSrv *httptest.Server | ||
| received chan []byte | ||
| } | ||
|
|
||
| func (tc *TestCase) Setup(_ context.Context, t *testing.T) { | ||
| tc.received = make(chan []byte, 1) | ||
| tc.mockSrv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| if !isCrashtrackerRequest(r) { | ||
| w.WriteHeader(http.StatusAccepted) | ||
| return | ||
| } | ||
| body, _ := io.ReadAll(r.Body) | ||
| select { | ||
| case tc.received <- body: | ||
| default: | ||
| } | ||
| w.WriteHeader(http.StatusAccepted) | ||
| })) | ||
| t.Cleanup(tc.mockSrv.Close) | ||
| } | ||
|
|
||
| func (tc *TestCase) Run(_ context.Context, t *testing.T) { | ||
| cmd := spawnSubprocess(t, crashRoleOrch, tc.mockSrv.URL) | ||
| _ = cmd.Wait() // non-zero exit expected (panic) | ||
|
|
||
| select { | ||
| case body := <-tc.received: | ||
| assertOrchCrashReport(t, body) | ||
| case <-time.After(15 * time.Second): | ||
| t.Fatal("timed out waiting for crash report from explicit crashtracker.Start() subprocess") | ||
| } | ||
| } | ||
|
|
||
| func (*TestCase) ExpectedTraces() trace.Traces { | ||
| return trace.Traces{} | ||
| } | ||
|
|
||
| // spawnSubprocess re-execs this binary as a crash-victim subprocess. It is | ||
| // called from TestCase.Run. Only runs when the binary was built with orchestrion. | ||
| func spawnSubprocess(t *testing.T, role, agentURL string) *exec.Cmd { | ||
| t.Helper() | ||
| if !built.WithOrchestrion { | ||
| t.Skip("subprocess e2e requires orchestrion-built binary; run via orchestrion go test") | ||
| } | ||
|
|
||
| cmd := exec.Command(os.Args[0], "-test.run=^$", "-test.v=false") //nolint:gosec | ||
| cmd.Env = append(filterOrchEnv(os.Environ()), | ||
| e2eRoleEnv+"="+role, | ||
| "DD_TRACE_AGENT_URL="+agentURL, | ||
| "DD_CRASHTRACKING_ENABLED=true", | ||
| ) | ||
| cmd.Stdout = io.Discard | ||
| cmd.Stderr = io.Discard | ||
| if err := cmd.Start(); err != nil { | ||
| t.Fatalf("spawn subprocess: %v", err) | ||
| } | ||
| return cmd | ||
| } | ||
|
|
||
| // filterOrchEnv strips variables that must not pollute the subprocess environment. | ||
| func filterOrchEnv(env []string) []string { | ||
| filtered := make([]string, 0, len(env)) | ||
| for _, kv := range env { | ||
| if strings.HasPrefix(kv, e2eRoleEnv+"=") || | ||
| strings.HasPrefix(kv, "DD_TRACE_AGENT_URL=") || | ||
| strings.HasPrefix(kv, "DD_CRASHTRACKING_ENABLED=") { | ||
| continue | ||
| } | ||
| filtered = append(filtered, kv) | ||
| } | ||
| return filtered | ||
| } | ||
|
|
||
| func isCrashtrackerRequest(r *http.Request) bool { | ||
| return r.URL.Path == "/evp_proxy/v4/api/v2/errorsintake" && | ||
| r.Header.Get("X-Datadog-EVP-Subdomain") == "error-tracking-intake" | ||
| } | ||
|
|
||
| // assertOrchCrashReport validates the key fields of the received crash report. | ||
| func assertOrchCrashReport(t *testing.T, body []byte) { | ||
| t.Helper() | ||
| if len(body) == 0 { | ||
| t.Fatal("received empty crash report body") | ||
| } | ||
| var report map[string]any | ||
| if err := json.Unmarshal(body, &report); err != nil { | ||
| t.Fatalf("unmarshal crash report: %v\nbody: %s", err, body) | ||
| } | ||
| if report["ddsource"] != "crashtracker" { | ||
| t.Errorf("ddsource = %q, want \"crashtracker\"", report["ddsource"]) | ||
| } | ||
| errObj, ok := report["error"].(map[string]any) | ||
| if !ok { | ||
| t.Fatalf("error field missing or not an object") | ||
| } | ||
| if errObj["is_crash"] != true { | ||
| t.Errorf("error.is_crash = %v, want true", errObj["is_crash"]) | ||
| } | ||
| if got, _ := errObj["type"].(string); got != "panic" { | ||
| t.Errorf("error.type = %q, want \"panic\"", got) | ||
| } | ||
| if msg, _ := errObj["message"].(string); !strings.Contains(msg, orchCrashMsg) { | ||
| t.Errorf("error.message = %q, want it to contain %q", msg, orchCrashMsg) | ||
| } | ||
| } |
35 changes: 35 additions & 0 deletions
35
internal/orchestrion/_integration/crashtracker/e2e_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| // Unless explicitly stated otherwise all files in this repository are licensed | ||
| // under the Apache License Version 2.0. | ||
| // This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| // Copyright 2016-present Datadog, Inc. | ||
|
|
||
| package crashtracker | ||
|
|
||
| import ( | ||
| "os" | ||
| "testing" | ||
|
|
||
| ct "github.com/DataDog/dd-trace-go/v2/crashtracker" | ||
| ) | ||
|
|
||
| // TestMain intercepts re-executions of this binary that serve as crash-victim | ||
| // subprocesses (see TestCase.Run in crashtracker.go). | ||
| // | ||
| // The orchestrion.yml join-point uses test-main:false, which deliberately | ||
| // excludes test binaries' main() functions from injection. This subprocess role | ||
| // calls crashtracker.Start() explicitly to validate the crash pipeline | ||
| // (spawn → SetCrashOutput → panic → monitor → upload) under an orchestrion-built | ||
| // test binary. TestCrashtrackerMainInjection validates injection into a real | ||
| // non-test main function. | ||
| func TestMain(m *testing.M) { | ||
| switch os.Getenv(e2eRoleEnv) { | ||
| case crashRoleOrch: | ||
| // Explicit Start() — orchestrion does not inject into test binaries. | ||
| if err := ct.Start(); err != nil { | ||
| os.Stderr.WriteString("crashtracker.Start: " + err.Error() + "\n") | ||
| os.Exit(1) | ||
| } | ||
| panic(orchCrashMsg) | ||
| } | ||
| os.Exit(m.Run()) | ||
| } |
18 changes: 18 additions & 0 deletions
18
internal/orchestrion/_integration/crashtracker/generated_test.go
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With
test-main: false, this aspect never touches the syntheticgo testmain. The new e2e fixture re-execs the test binary and itsTestMainin packagecrashtrackerpanics without callingStart, but that function also does not satisfypackage-name: main/function main; the only callable main in that subprocess is therefore excluded here. As a resultTestCase.Runwaits 15s for a report that can never be produced underorchestrion go test, so CI will fail rather than validating the aspect.Useful? React with 👍 / 👎.