Skip to content

Troubleshooting.md

Latest commit

 

History

History
1060 lines (874 loc) · 44 KB

Troubleshooting.md

File metadata and controls

1060 lines (874 loc) · 44 KB
Raw

Troubleshooting

Examples below assume the generated repository is named test_optimization_data. If you used a different name, replace labels and repository names accordingly. For config-gated Go and Python repositories, include --config=test-optimization in sync, test, doctor, and uploader commands. Other companions retain their current activation contract.

If Bazel reports that sync requires WORKSPACE support, add --enable_workspace to sync commands in this document.

Test Optimization config and disabled mode

For config-gated Go and Python onboarding, --config=test-optimization is the single user-facing enablement switch. Both languages set the metadata repository environment:

common:test-optimization --repo_env=DD_TEST_OPTIMIZATION_ENABLED=1

Go additionally sets the existing rules_go Orchestrion flag:

build:test-optimization --@rules_go//go/private/orchestrion:enabled=true

When the config is omitted, the public Go extension's config-gated default and repositories explicitly configured with enabled_by_env = True generate the documented no-fetch stubs. Go aliases select local empty targets; Python keeps the normal consumer runner without Test Optimization metadata or payload wiring. Python-only consumers omit the Go line. For WORKSPACE Go, replace @rules_go with the apparent repository name used by that workspace. Java, NodeJS, .NET, and Ruby retain their existing enablement contract in this release.

Quick triage map

Symptom First checks Likely section
No files fetched or stale data DD_API_KEY forwarded with --repo_env; use FETCH_SALT only for an explicit force-refresh sync Repository rule not fetching data
Uploader says no payload files tracer file-mode contract + payload files under bazel-testlogs/*/test.outputs/ Uploader not finding payloads
Doctor reports msgpack payloads tracer is not in Bazel JSON file mode Doctor failures
Doctor reports missing Git or Bazel metadata sync metadata context or sidecar metadata is absent Doctor failures
Uploaded tests miss Git or Bazel tags run uploader dry-run enrichment validation Uploader enrichment dry-run
Upload network errors credential mode (agentless vs EVP), intake reachability Tests not uploading (network errors)
CI failure requires log archaeology archive the support bundle from the failing run Collect diagnostic reports
Module selection misses bazel query for module_* targets and importpath/module label expectations Per-module files not found
Go build fails with a tracer version mismatch dd_trace_go_pin_files, explicit version escape hatches, local go.mod/go.sum Go tracer version drift
Bazel resolves an older tracer or Orchestrion module in WORKSPACE mode checked-in go_repository(...) pins WORKSPACE go_repository drift
WORKSPACE archive pins fail after a PR was squash-merged generated pins commit reachability and archive SHA Published Go pins
Private/internal WORKSPACE fetch returns 404 SSH git or authenticated archive access Private repository fetch
Bazel downloads unrelated toolchains or analyzes unrelated packages cold monorepo state or root-package doctor/uploader placement Monorepo analysis cost
Windows path/policy failures PowerShell policy + path separators Windows-specific issues

Collect diagnostic reports

When a failure is hard to classify from logs alone, archive a support bundle from the same run. For the simplest customer ask after tests have already run, use the doctor directly:

bazel run --config=test-optimization //:dd_test_optimization_doctor -- \
  --support-bundle .topt/reports/dd-test-optimization-support.zip

For BEP/BwoB CI investigations, include the same BEP and artifact flags used by the failing job:

bazel run --config=test-optimization //:dd_test_optimization_doctor -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --artifact-staging-dir="$artifact_staging_dir" \
  --support-bundle=.topt/reports/dd-test-optimization-support.zip

The doctor bundle is doctor-only: it captures expected targets, BEP freshness, artifact staging, discovered payload directories, payload counts, status, exit code, effective doctor flags, runtime metadata, and selected BEP summaries. It does not prove uploader dry-run, enrichment, API-key, or upload behavior.

Use this escalation ladder:

Situation Ask for Why
First customer response after tests already ran Doctor --support-bundle=<path> Smallest command; built into the doctor target; no helper scripts required
CI failure where upload, enrichment, or dry-run behavior matters Wrapper --report-dir=<path> --support-bundle=<path> Includes doctor, uploader dry-run, optional upload report, BEP summaries, and effective wrapper flags
Repository cannot run the wrapper or doctor bundle Raw --report-json files plus manual create_support_bundle.py output Fallback only; raw reports may include internal paths until the redacted zip is created

Ask the customer to attach the zip, not screenshots of terminal output. If the bundle is not available, ask for the individual JSON reports and the exact bazel test, doctor, and uploader commands.

Use the wrapper flow when the CI job should also validate uploader dry-run or perform the real upload:

# Vendor the full tools/test_optimization/ helper directory, or set
# DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE_COLLECTOR to create_support_bundle.py.
tools/test_optimization/run_test_optimization_ci.sh \
  --report-dir .topt/reports \
  --support-bundle .topt/reports/dd-test-optimization-support.zip \
  //...
# Vendor the full tools/test_optimization/ helper directory, or set
# DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE_COLLECTOR to create_support_bundle.py.
.\tools\test_optimization\run_test_optimization_ci.ps1 `
  -ReportDir .topt\reports `
  -SupportBundle .topt\reports\dd-test-optimization-support.zip `
  //...

The wrapper writes doctor-report.json and uploader-dry-run-report.json under the report directory. If the same wrapper run includes the real upload, it also writes uploader-upload-report.json, preserving the dry-run report. When --support-bundle or -SupportBundle is configured, it also writes dd-test-optimization-support.zip. Each report has a result.reason_code, human-readable result.reason, and result.next_steps so support can distinguish cached tests, remote-only BEP artifacts, missing payloads, enrichment failures, dry-run no-upload, and real upload failures without reading the full log.

When opening a support ticket, attach .topt/reports/dd-test-optimization-support.zip. The bundle is redacted by default and does not contain raw payload files, raw CI logs, raw environment variables, or the raw BEP. Depending on whether the doctor or wrapper created it, it contains bounded doctor-only or doctor/uploader reports, selected BEP summaries, command/flags.json, environment/runtime.json, summary.md, and diagnostics.json.

Triage the bundle in this order:

  1. Open summary.md for the human-readable failure classification.
  2. Check diagnostics.json for summary.status, summary.primary_reason_code, payload counts, and included report paths.
  3. Check reports/doctor-report.json for expected targets, seen targets, fresh/cached/remote-only BEP outputs, artifact staging, and doctor errors.
  4. If the wrapper created the bundle, check reports/uploader-dry-run-report.json for payload discovery, enrichment validation, skipped payloads, and no-upload reasons.
  5. If upload was enabled, check reports/uploader-upload-report.json for the real upload attempt count, HTTP status, and terminal upload failure.
  6. Check command/flags.json to confirm the test run used a unique BEP file and that doctor/uploader ran with the matching --bep-json, --freshness-source=bep, --freshness-mode=required, --artifact-source=bep, and artifact staging flags.

Manual doctor/uploader flow:

bazel run --config=test-optimization //:dd_test_optimization_doctor -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --artifact-staging-dir="$artifact_staging_dir" \
  --report-json=.topt/doctor-report.json

bazel run --config=test-optimization //:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --artifact-staging-dir="$artifact_staging_dir" \
  --dry-run \
  --validate-enrichment \
  --report-json=.topt/uploader-report.json

Archive the JSON files as CI artifacts when a repository cannot use the wrapper support bundle. If the repository has the helper script available, create the same redacted zip from existing reports:

python3 tools/test_optimization/create_support_bundle.py \
  --report-dir .topt/reports \
  --report-json .topt/reports/doctor-report.json \
  --report-json .topt/reports/uploader-dry-run-report.json \
  --output .topt/reports/dd-test-optimization-support.zip \
  --workspace-root "$PWD" \
  --output-base "$(bazel info output_base)"

Use repeatable --bep-json=<path> flags with that command when the reports came from BEP/BwoB runs. If only a Markdown summary is possible, render a short fallback summary:

python3 tools/test_optimization/render_report_summary.py \
  .topt/reports/doctor-report.json \
  .topt/reports/uploader-dry-run-report.json \
  --output .topt/reports/upload-diagnostics.md

Before sharing raw reports outside the trusted project boundary, review them for repository paths, target names, and service metadata. They must not contain API keys, but they can contain internal labels and filesystem paths. Prefer a support bundle for external escalation because it redacts and bounds those values automatically.

Repository rule not fetching data

Symptom: Build succeeds but test optimization files are empty or stale.

Solutions:

  1. Verify DD_API_KEY is set:

    env | grep '^DD_API_KEY='
grep -n 'common --repo_env=DD_API_KEY' .bazelrc
    $env:DD_API_KEY
Select-String -Path .bazelrc -Pattern "common --repo_env=DD_API_KEY"

    If missing, set DD_API_KEY in your shell/CI secret store and add to .bazelrc:

    common --repo_env=DD_API_KEY

  2. Force refetch only when intentional with a cache-busting salt:

    bazel sync --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT="$(date +%s)"
    bazel sync --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT="$(Get-Date -UFormat %s)"

    Do not put FETCH_SALT in .bazelrc, bazel test, doctor, or uploader commands. It deliberately breaks the repository-rule cache key and should be used only when you want fresh backend metadata.

  3. Check repository cache to see if the rule ran:

    # Find the external repository directory
bazel info output_base
# Repository contents at: $(bazel info output_base)/external/<repo_name>
ls -la $(bazel info output_base)/external/<repo_name>/.testoptimization/
    # Find the external repository directory
$outputBase = bazel info output_base
# Repository contents at: "$outputBase/external/<repo_name>"
Get-ChildItem -Force "$outputBase/external/<repo_name>/.testoptimization/"

  4. Enable debug logging in your MODULE.bazel:

    test_optimization_sync.test_optimization_sync(
    name = "test_optimization_data",
    debug = True,  # Verbose logging
)

    Preserve the repository's existing enabled_by_env value; enabling debug logging must not change its activation contract.

Manifest-driven managed runs

These checks apply only to the automatic Go/Python path. Static single-service and static multi-service consumers do not need an invocation manifest.

Enabled run reports a missing manifest

The aggregate repository intentionally fails before HTTP when enabled without the command-owned manifest. Run the consumer repository's managed Test Optimization command. Do not add the internal manifest handoff to .bazelrc or ordinary CI jobs.

If a direct bazel test --config=test-optimization bypasses the managed command, the failure is expected: target discovery must run first so the repository rule receives exact targets and runtime contexts.

Doctor or uploader triggers another metadata fetch

One managed command invocation must pass the same temporary manifest path to test, doctor, uploader dry-run, and optional upload. If the request log shows another fetch during a post-test phase, check that the command did not create a new temporary directory or change DD_TEST_OPTIMIZATION_SERVICES_MANIFEST between child Bazel processes.

A later managed command intentionally creates a new manifest path and fetches current backend state once. If its selected settings/module files are unchanged, tests should remain Bazel cache hits even though telemetry_facts.json contains new request timings.

Manifest is malformed or unsupported

The error names the schema field rejected before metadata requests. Verify that the consumer resolver emitted:

  • schema version 1;
  • canonical local labels such as //pkg:test, not patterns or external labels;
  • unique, used contexts;
  • only Go/Python runtimes;
  • deterministic context keys matching service and runtime;
  • application or domain_fallback for every target.

Fix the resolver or its target input. Do not hand-edit the temporary manifest.

Unsupported target is absent from the aggregate mapping

Only targets recognized by the consumer's central Go/Python macro policy are eligible. A target absent from topt_data_by_target correctly keeps its raw behavior. Inspect the managed command summary for unsupported labels and confirm that the target is a runtime test, not a build-only companion.

Java, NodeJS, .NET, and Ruby are intentionally outside automatic manifest onboarding in this release.

Service or context collision

The resolver and repository reject two logical services that sanitize to the same context key, conflicting module paths or runtime versions for one service/runtime, duplicate labels, and one label assigned to multiple contexts. Change the consumer's deterministic naming grammar or split the conflicting runtime contexts; do not append positional suffixes whose result depends on input order.

Doctor expected-target mismatch

The aggregate :expected_targets file is exact. A mismatch means the execution set, BEP inputs, or doctor wiring does not match the manifest. The managed command must test the fully expanded labels from the same discovery phase and pass that invocation's BEP file to doctor.

Cached tests do not emit fresh payloads. With configured expected targets, strict BEP freshness accepts those cache hits as covered, validates no stale local outputs for them, and lets uploader treat an all-cached invocation as a no-op. If a cached target is reported as missing instead, verify that doctor received the BEP from the exact matching bazel test invocation. Use separate cache-isolation tests when validating action-key behavior.

Disabled mode or enabled bootstrap invokes host Go

Disabled manifest sync must not read the manifest, contact Datadog, or resolve the real Orchestrion tool repository. Enabled Go bootstrap uses the Bazel-managed Go SDK and must not depend on a host go binary. If a host-Go sentinel fires, verify the patched rules_go profile, generated SDK wiring, and central wrapper before installing Go on the host.

Published Go pins

Symptom: A consumer cannot fetch the published Go/Orchestrion archive, or a pin that worked during review stops working after a squash merge.

Cause: The consumer was given a feature-branch commit, a stale archive hash, or a tuple that was not generated from the real GitHub codeload archive.

Solutions:

  1. Regenerate pins from origin/main in this repository:

    ./bazelw run //tools/dev:print_go_onboarding_pins -- \
  --commit "$(git rev-parse origin/main)" \
  --rules-go-upstream v0_60_0 \
  --variant base \
  --verify-main-reachable

  2. If you are in a consumer repo, use the bootstrap helper with the squash-merged commit:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --print-published-pins \
  --rto-commit <published-origin-main-sha> \
  --rules-go-upstream v0_60_0 \
  --rules-go-variant base

  3. Keep the generated tuple together. If RTO_COMMIT, RTO_ARCHIVE_URL, RTO_ARCHIVE_SHA256, and RTO_ARCHIVE_PREFIX come from different commits, archive mode will fail or fetch the wrong source. The current published tuple is tracked in docs/Installation_Reference.md.

  4. Authenticate private archive downloads. The helper uses GITHUB_TOKEN, GH_TOKEN, or gh auth token when available. If codeload returns 404 for a commit you can see in GitHub, refresh local GitHub CLI authentication or set one of those token variables before regenerating pins.

  5. Use the generated summary for handoff when a monorepo needs a permanent local guide:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --write-onboarding-summary=TEST_OPTIMIZATION_GUIDE.md \
  --rto-commit <published-origin-main-sha> \
  --rules-go-upstream v0_60_0 \
  --rules-go-variant base

Private repository fetch

Symptom: A WORKSPACE consumer gets 404 when fetching the rules archive or companion archive from a private/internal repository.

Cause: Anonymous codeload archive URLs can return 404 for private repositories even when the commit exists and the user can see it in GitHub.

Solutions:

  1. Prefer SSH git fetch for internal Datadog consumers:

    git_repository(
    name = "datadog-rules-test-optimization",
    commit = "<commit-sha>",
    remote = "ssh://git@github.com/DataDog/rules_test_optimization.git",
)

  2. Use archive fetch only when Bazel has compatible authentication, for example .netrc or another repository-approved token mechanism.

  3. Do not commit local archive paths or local checkouts as a workaround for CI. Use local paths only for temporary local validation.

Monorepo analysis cost

Symptom: Tests already produced payload files, but running doctor or uploader appears slow because Bazel downloads unrelated toolchains or analyzes unrelated packages.

Cause: Some cost is normal cold-cache monorepo behavior. Another common cause is placing doctor/uploader in the workspace root package, which can load global root BUILD.bazel wiring unrelated to Test Optimization.

Solutions:

  1. Move the logical doctor/uploader pair to a lightweight package:

    # tools/test_optimization/BUILD.bazel
load("@datadog-rules-test-optimization//tools/core:test_optimization_targets.bzl", "dd_test_optimization_targets")

dd_test_optimization_targets(
    name = "test_optimization",
    sync_repo_name = "test_optimization_data",
    expected_targets = ["//app:service_py_test"],
)

  2. Run package-local labels:

    bep_json="${bep_json:-/path/from/matching/bazel-test.bep.json}"
bazel run --config=test-optimization //tools/test_optimization:dd_test_optimization_doctor -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required
bazel run --config=test-optimization //tools/test_optimization:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --dry-run \
  --validate-enrichment

  3. If Bazel repeatedly refetches Test Optimization metadata, check whether a script or .bazelrc is setting FETCH_SALT by default.

Uploader not finding payloads

Symptom: Uploader runs but says "no payload files found".

Solutions:

  1. Check if tests wrote payloads:

    find bazel-testlogs -name "test.outputs" -type d
ls bazel-testlogs/*/test.outputs/payloads/tests/
    Get-ChildItem -Recurse -Directory -Path bazel-testlogs -Filter "test.outputs"
Get-ChildItem -Recurse -Path bazel-testlogs -File -Filter "*.json" | Where-Object { $_.FullName -match "test\.outputs[\\/]+payloads[\\/]+tests" }

  2. Verify tracer support: Ensure your tracer/runtime supports file mode via DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES and manifest discovery via DD_TEST_OPTIMIZATION_MANIFEST_FILE.

  3. Check DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES: The macro should set this to "true". Verify your test environment:

    bazel test --config=test-optimization //your:test --test_output=all 2>&1 | grep DD_TEST_OPTIMIZATION
    bazel test --config=test-optimization //your:test --test_output=all *>&1 | Select-String "DD_TEST_OPTIMIZATION"

    PowerShell uses *>&1 (not Bash 2>&1) to merge stderr/stdout.

  4. For RBE/remote-cache users: Put --remote_download_minimal, --remote_download_regex=.*test[.]outputs.*, and --zip_undeclared_test_outputs in the active test .bazelrc config. Run tests with a fresh --build_event_json_file=..., then run doctor/uploader with repeatable --bep-json=<path>, --freshness-source=bep, --freshness-mode=required, --artifact-source=bep, and --artifact-staging-dir=<temp-dir> so cached outputs are skipped and local outputs.zip carriers are extracted through BEP staging.

Doctor failures

Symptom: bazel run --config=test-optimization //:dd_test_optimization_doctor fails before upload.

Solutions:

  1. Msgpack payloads: Go/Orchestrion Bazel mode must write JSON payloads to TEST_UNDECLARED_OUTPUTS_DIR. If the doctor finds .msgpack or .msgpack.gz, verify that the target uses the Go test optimization macro or wrapper and that the configured dd-trace-go version includes Bazel JSON file-mode support.

  2. No JSON payloads: Confirm the test target actually ran with Test Optimization enabled and inspect:

    find bazel-testlogs -path '*/test.outputs/payloads/*/*.json' -type f

    For Python WORKSPACE consumers, confirm the target loads dd_topt_py_test from @datadog-rules-test-optimization-python//:topt_py_test.bzl and that the companion repository was declared with datadog_python_test_optimization_workspace_repositories(...). For Python runner_mode = "consumer_runner", also verify that the consumer-owned wrapper really executes pytest, propagates the env passed by dd_topt_py_test, and keeps PYTEST_ADDOPTS=--ddtrace unless it intentionally sets --no-ddtrace. If env is a select(...), the macro cannot add PYTEST_ADDOPTS inside each branch; include --ddtrace in every relevant selected environment. The target must also depend on ddtrace and pytest.

  3. Missing Git metadata: The sync metadata fetch must see repository URL, commit SHA, and branch or tag. Put DD_GIT_* values in .bazelrc as common:test-optimization --repo_env=DD_GIT_<NAME>, not --test_env. The doctor scans versioned .bazelrc files for --test_env=DD_GIT_*, but it cannot detect a bad --test_env=DD_GIT_* flag typed directly on the CLI.

  4. Missing Bazel metadata: The target should emit bazel_target_metadata.json next to payload files. Use the companion macro or generated wrapper instead of invoking the raw language test rule directly.

    For Python consumer-owned wrappers, the wrapper must expose an executable target with RunEnvironmentInfo preserved from the raw py_test; otherwise the public metadata wrapper can run but the actual pytest process may miss the required DD_TEST_OPTIMIZATION_* environment.

    In WORKSPACE mode, also verify that the Python helper maps the companion's internal @rules_python dependency to the consumer repository name:

    datadog_python_test_optimization_workspace_repositories(
    rto_commit = "<commit-sha>",
    rules_python_repo_name = "rules_python",  # or the consumer's custom name
)

  5. full_bundle_no_match: The Go macro could not map the test target to a per-module bundle. Prefer embed = [":lib"] so the macro can read the same importpath that rules_go uses, or set module_label_override only when the module label is intentionally known. module and module_override are valid successful selections; full_bundle_disabled is valid when backend module data is disabled. Generic inferred/derived selection may intentionally use the canonical full bundle and report full_bundle_no_match; the doctor rejects that state by default, so only an intentionally generic consumer should set forbid_full_bundle_no_match = False.

  6. Expected target output missing: Run the exact target listed in expected_targets before the doctor. With remote execution or remote cache, use the recommended test .bazelrc config: --remote_download_minimal --remote_download_regex=.*test[.]outputs.* --zip_undeclared_test_outputs. Export DD_TEST_OPTIMIZATION_ARTIFACT_SOURCE=bep for doctor/uploader; if BEP references remote-only artifacts, enable BEP artifact resolution with --artifact-source=bep --remote-artifacts=download.

BEP artifact resolution failures

Symptom: The doctor or uploader is configured with BEP freshness, but loose test.outputs/payloads/... files are not present under bazel-testlogs/ because the Bazel invocation used remote execution/cache, zipped undeclared outputs, or left the selected BEP artifacts remote-only.

Solutions:

  1. Prefer selective local materialization plus zipped undeclared outputs when Bazel can download the needed outputs:

    test:test-optimization --remote_download_minimal
test:test-optimization --remote_download_regex=.*test[.]outputs.*
test:test-optimization --zip_undeclared_test_outputs

  2. Pass the same BEP file to doctor/uploader and enable local BEP artifact staging. The checked-in CI wrapper does this with repeatable CLI flags:

    tools/test_optimization/run_test_optimization_ci.sh //...

    Manual invocations should use the same temporary BEP and staging paths:

    bep_json="${bep_json:-/path/from/matching/bazel-test.bep.json}"
artifact_staging_dir="${artifact_staging_dir:-$(mktemp -d "${TMPDIR:-/tmp}/dd-topt-artifacts.XXXXXX")}"
bazel run --config=test-optimization //:dd_test_optimization_doctor -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --artifact-staging-dir="$artifact_staging_dir"

bazel run --config=test-optimization //:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --artifact-staging-dir="$artifact_staging_dir" \
  --dry-run \
  --validate-enrichment

    --remote-artifacts is not required for local outputs.zip files.

  3. If CI cannot download test.outputs or outputs.zip locally, pass the same BEP file to doctor and uploader and enable remote staging. HTTP/HTTPS outputs.zip carriers can be staged without a downloader:

    artifact_staging_dir="${artifact_staging_dir:-$(mktemp -d "${TMPDIR:-/tmp}/dd-topt-artifacts.XXXXXX")}"
bazel run --config=test-optimization //:dd_test_optimization_doctor -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --remote-artifacts=download \
  --artifact-staging-dir="$artifact_staging_dir"

bazel run --config=test-optimization //:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --artifact-source=bep \
  --remote-artifacts=download \
  --artifact-staging-dir="$artifact_staging_dir" \
  --dry-run \
  --validate-enrichment

  4. For bytestream/CAS/custom provider BEP artifact references, or HTTP/HTTPS endpoints that require custom auth, configure one downloader executable:

    export DD_TEST_OPTIMIZATION_BEP_ARTIFACT_DOWNLOADER=/path/to/download-outputs-zip

    The executable receives --uri, --name, and --output arguments and must write an outputs.zip archive to --output before --bep-artifact-downloader-timeout-sec expires. If your provider needs an interpreter, fixed arguments, custom auth headers, mTLS, signed header refresh, or a CAS client, wrap it in a script and configure the wrapper path. The public rule does not ship credentials or a Datadog-internal CAS client.

Native HTTP/HTTPS staging is unauthenticated. It retries transient network failures, truncated responses, HTTP 408, HTTP 429, and HTTP 5xx responses three times with exponential backoff. It does not add custom headers, read credentials, use cookies, or consult netrc. Query strings, fragments, usernames, and passwords are redacted from doctor/uploader warnings and JSON reports.

Common messages:

  • --artifact-source=bep requires --bep-json or DD_TEST_OPTIMIZATION_BEP_JSON: pass the BEP JSON produced by the matching bazel test invocation.
  • BEP artifact is remote-only and no downloader is configured: either use --remote_download_minimal --remote_download_regex=.*test[.]outputs.* --zip_undeclared_test_outputs with --artifact-source=bep, enable --remote-artifacts=download for HTTP/HTTPS outputs.zip carriers, configure DD_TEST_OPTIMIZATION_BEP_ARTIFACT_DOWNLOADER for bytestream/CAS/custom-auth providers, or use --remote-artifacts=disabled outside required freshness mode.
  • BEP HTTP artifact download failed ... HTTP Error 401 or 403: the endpoint likely requires auth. Use a downloader wrapper that injects the required credentials or headers.
  • BEP HTTP artifact download failed ... HTTP Error 404: confirm the BEP URI still exists for the current job and has not expired.
  • BEP HTTP artifact download failed ... timed out: native HTTP retries transient failures three times; raise --bep-artifact-downloader-timeout-sec only when each request attempt needs more time.
  • invalid BEP outputs.zip: confirm the URL serves Bazel's outputs.zip, not HTML, JSON, or an error page.
  • BEP HTTP artifact is too large: inspect undeclared test outputs for unexpected files; the compressed download and extracted tree both have size guards.
  • BEP artifact is not available locally: the BEP points at a local path or file:// URI that does not exist in this workspace. Check that the BEP file matches the current test invocation and machine.
  • remote-artifacts=required: strict artifact mode is all-or-nothing. Missing, ambiguous, unsafe, or undownloadable selected BEP artifacts fail the command instead of falling back to local payloads.
  • unsafe path in BEP outputs.zip: the archive contains an absolute path or parent traversal entry. Regenerate the archive or fix the downloader.
  • BEP outputs.zip has too many entries: the archive exceeds the staging entry limit and is rejected before extraction.
  • BEP outputs.zip is too large after decompression: the archive exceeds the staging decompressed-size limit.
  • BEP artifact staging requires PYTHON, python3, or python: the uploader needs Python only when artifact staging is enabled. Set DD_TEST_OPTIMIZATION_PYTHON or PYTHON when Python is not discoverable as python3 or python on PATH.

Uploader enrichment dry-run

Symptom: Payload files exist, but Datadog UI or JSON inspection suggests the uploaded test is missing Git or Bazel tags.

Explanation: Raw test payloads on disk are not the final upload body. The uploader adds context.json and bazel_target_metadata.json tags immediately before upload.

Solution: Run the uploader dry-run after tests and doctor:

bep_json="${bep_json:-/path/from/matching/bazel-test.bep.json}"
bazel run --config=test-optimization //:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required \
  --dry-run \
  --validate-enrichment
$env:BEP_JSON = "C:\path\from\matching\bazel-test.bep.json"
bazel run --config=test-optimization //:dd_upload_payloads -- `
  --bep-json="$env:BEP_JSON" `
  --freshness-source=bep `
  --freshness-mode=required `
  --dry-run `
  --validate-enrichment

Dry-run mode does not upload, does not delete payload files, and does not need DD_API_KEY for agentless mode. By default it requires git.repository_url, git.commit.sha, bazel.target, and bazel.package in the enriched test payload. Add --expected-enriched-tag=bazel.go.payload_selection when a Go rollout must prove per-module selection. If this fails:

  1. Ensure the uploader target has the right data = ["@...//:test_optimization_context"].
  2. Ensure bazel_target_metadata.json exists beside the payloads.
  3. Ensure jq is available on Linux/macOS when using --validate-enrichment.
  4. Use --expected-enriched-tag=<tag> for repository-specific required tags.

Non-standard bazel-testlogs location

Symptom: Uploader cannot find the bazel-testlogs directory.

Solution: Set TESTLOGS_DIR explicitly using the same Bazel flags:

# Bash - use array for multiple flags
BAZEL_FLAGS=("--output_base=/custom/base")
bep_json="${bep_json:-/path/from/matching/bazel-test.bep.json}"
TESTLOGS_DIR=$(bazel "${BAZEL_FLAGS[@]}" info bazel-testlogs) bazel "${BAZEL_FLAGS[@]}" run //:dd_upload_payloads -- \
  --bep-json="$bep_json" \
  --freshness-source=bep \
  --freshness-mode=required
# PowerShell
$BazelFlags = @("--output_base=/custom/base")
$env:BEP_JSON = "C:\path\from\matching\bazel-test.bep.json"
$env:TESTLOGS_DIR = (bazel @BazelFlags info bazel-testlogs)
bazel @BazelFlags run //:dd_upload_payloads -- `
  --bep-json="$env:BEP_JSON" `
  --freshness-source=bep `
  --freshness-mode=required

Tests not uploading (network errors)

Symptom: Uploader fails with network errors.

Solutions:

  1. Verify credentials:

    • Agentless mode requires: DD_API_KEY, DD_SITE
    • EVP proxy mode requires: DD_TEST_OPTIMIZATION_AGENT_URL

  2. Check firewall/proxy allows HTTPS to:

    • https://citestcycle-intake.datadoghq.com
    • https://citestcov-intake.datadoghq.com
    • (or equivalent for your DD_SITE)

  3. Enable debug logging:

    dd_payload_uploader(
    name = "dd_upload_payloads",
    debug = True,
    ...
)

Per-module files not found

Symptom: dd_topt_go_test fails with "module_X not found" or falls back to full bundle.

Solutions:

  1. List available modules:

    bazel query 'kind(".*", @<repo_name>//...)' | grep module_
    bazel query 'kind(".*", @<repo_name>//...)' | Select-String "module_"

  2. Override module label explicitly (workaround):

    dd_topt_go_test(
    name = "my_test",
    module_label_override = "my_expected_module",  # Matches :module_my_expected_module
    orchestrion_mode = "test_optimization",
    ...
)

Go tracer version drift

Symptom: Go bootstrap or Bazel build fails with a message showing a configured tracer version map and a different resolved local dd-trace-go module version.

Solutions:

  1. If you use guided bootstrap, rerun it with the query you want:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --dd-trace-go-version <tag-or-sha>

    Bootstrap accepts tags, pseudo-versions, branches, and commit SHAs. It rewrites the workspace to the exact resolved versions that Go actually uses. By default it uses targeted module sync, not go mod tidy.

  2. If targeted sync reports readonly module errors, rerun bootstrap with explicit targeted sync and the same Go SDK your repository expects:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --go-mod-sync=targeted \
  --go-binary /path/to/go

    If the workspace uses checked-in go_repository declarations, refresh those declarations after go.mod or go.sum changes so Bazel and the Go module graph agree.

  3. If you wire Orchestrion manually, use the checked-in module as the normal source of truth:

    orchestrion.from_source(
    dd_trace_go_pin_files = ["@//:go.mod", "@//:go.sum"],
    go_sdk_root = "@test_optimization_go_sdk//:ROOT",
    go_sdk_version = "<go-version>",
    version = "<orchestrion-version>",
)

    Export both root files from their BUILD package. Bazel resolves all supported direct and transitive tracer modules with hermetic Go and -mod=readonly; it does not update go.sum.

  4. If pin-file resolution reports an absent module, unsupported replacement, or read-only failure, fix the module with its normal dependency workflow. If that graph intentionally cannot use pin-file mode, configure the exact dd_trace_go_versions map as an escape hatch. Do not combine pin files with either explicit version field.

  5. If you omitted every selection mode, remember the legacy default is v2.9.0.

The build fails on purpose here. It is preventing Bazel from injecting one set of tracer versions while the local Go module still resolves another.

orchestrion.tool.go still matters, but as required import/config wiring for Orchestrion, not as the source of truth for tracer versions.

Enabled Orchestrion cannot find Go

Symptom:

Could not find 'go' binary. Please ensure Go is installed.

Cause: the workspace uses Orchestrion wiring generated before the Bazel-managed SDK contract was added, or a manual block omitted the SDK root and version. Disabled mode still avoids the real repository, but enabled mode must compile the patched Orchestrion binary on a cold cache miss.

Solution:

  1. For Bzlmod, rerun guided bootstrap with the workspace's real Go toolchain version: 
    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --guided \
  --service <datadog-service> \
  --runtime-version <go-version> \
  --write-bazelrc
  2. For WORKSPACE, regenerate the reviewed snippet with --workspace-mode --print-workspace-snippet --runtime-version <go-version>.
  3. Verify the generated SDK version equals the version passed to the repository's central go_register_toolchains(...) call and the Test Optimization sync repository.

Do not install Go on the analysis host as the fix. The supported wiring uses the Bazel-managed SDK on a cache miss and can restore a compatible warm Orchestrion bootstrap before materializing that SDK.

WORKSPACE go_repository drift

Symptom: go.mod and go.sum look correct, but Bazel still resolves an older github.com/DataDog/orchestrion or dd-trace-go module from a checked-in repositories.bzl file.

Cause: WORKSPACE repositories often keep generated go_repository(...) declarations separate from go.mod. Updating the Go module graph is not enough if the generated repository file still pins old versions.

Solutions:

  1. Run bootstrap diagnostics after targeted sync:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --workspace-mode \
  --write-orchestrion-files \
  --go-mod-sync=targeted \
  --check-go-repositories \
  --go-repositories-file repositories.bzl

    The checker compares only the modules bootstrap owns: github.com/DataDog/orchestrion, github.com/DataDog/dd-trace-go/v2, github.com/DataDog/dd-trace-go/contrib/net/http/v2, and github.com/DataDog/dd-trace-go/contrib/log/slog/v2.

  2. Let your repo-owned refresh command repair the file:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --workspace-mode \
  --write-orchestrion-files \
  --go-mod-sync=targeted \
  --check-go-repositories \
  --go-repositories-file repositories.bzl \
  --go-repositories-refresh-command './tools/update-go-repositories.sh'

    Bootstrap runs the refresh command only after targeted sync succeeds, then validates repositories.bzl again. It never edits repositories.bzl directly.

  3. Print expected versions for manual updates:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --workspace-mode \
  --go-mod-sync=off \
  --check-go-repositories \
  --print-go-repository-updates

    This is useful when the repository has a custom Gazelle/update-repos flow that must be run manually or in a separate review step.

Large monorepo validation is easy to run incorrectly

Symptom: onboarding works in small fixtures but large WORKSPACE repos need a long command sequence, local controls, disk checks, and an explicit upload step.

Solutions:

  1. Generate a local validation script instead of hand-copying commands:

    bazel run @datadog-rules-test-optimization-go//:dd_topt_go_bootstrap -- \
  --workspace-mode \
  --write-validation-script \
  --bazel-command bzl \
  --bazel-config test-optimization \
  --sync-repo-name test_optimization_data \
  --control-target //pkg/plain:go_default_test \
  --expected-target //pkg:go_default_test \
  --large-monorepo \
  --shutdown-bazel-on-exit

  2. Run without upload first:

    ./tools/test_optimization/validate_go_pilot.sh --no-upload

  3. Upload only after tests and doctor pass:

    DD_API_KEY="$DD_API_KEY" DD_SITE="$DD_SITE" \
  ./tools/test_optimization/validate_go_pilot.sh --upload

The generated script never deletes caches. In --large-monorepo mode it warns when free disk drops below --min-free-disk-gb, runs phases serially, and can shut down Bazel on exit. It still depends on the normal Bazel config for --remote_download_minimal --remote_download_regex=.*test[.]outputs.* and --zip_undeclared_test_outputs, plus BEP runtime configuration through DD_TEST_OPTIMIZATION_* or equivalent CLI flags; no rule can force that client-side Bazel behavior.

Windows-specific issues

Symptom: PowerShell errors or path issues.

Solutions:

  1. Verify PowerShell execution policy:

    Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

  2. Check paths use forward slashes in Starlark/Bazel contexts (backslashes are auto-converted).

  3. Use native PowerShell harness on Windows:

    • .\tools\tests\integration\run_mock_server_tests.ps1
    • Git Bash is not required for Windows integration runs.

Safe command patterns

  • Prefer explicit env assignment over shell interpolation in troubleshooting commands:
    • Unix: DD_API_KEY="$DD_API_KEY" DD_SITE="$DD_SITE" bazel run --config=test-optimization //:dd_upload_payloads
    • PowerShell: set $env:DD_API_KEY and $env:DD_SITE first, then run bazel run --config=test-optimization //:dd_upload_payloads
  • Quote paths containing spaces and avoid eval-style wrappers.
  • For refetch debugging, use:
    • bazel sync --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT=<timestamp>
    • if required by workspace mode: bazel sync --enable_workspace --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT=<timestamp>

Getting help

If issues persist:

  1. Enable debug mode and capture full output:

    bazel sync --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT=<timestamp> 2>&1 | tee debug.log
    bazel sync --config=test-optimization --only=<repo_name> --repo_env=FETCH_SALT=<timestamp> *>&1 | Tee-Object -FilePath debug.log

  2. Collect diagnostic info:

    • Bazel version: bazel version
    • OS: uname -a (Linux/macOS) or systeminfo (Windows)
    • Repository rule outputs (as shown above)
    • Doctor and uploader JSON reports, if the failing run produced them
    • Sanitized logs (remove API keys before sharing)

  3. File an issue:

    • open an issue in the repository issue tracker with sanitized logs