Add Azure VM and AKS integration tests for the default OTel config#730
Open
movence wants to merge 21 commits into
Open
Add Azure VM and AKS integration tests for the default OTel config#730movence wants to merge 21 commits into
movence wants to merge 21 commits into
Conversation
Adds test/azurevm (TestAzureVM, //go:build integration, TestMain/pre-provisioned pattern) validating the agent on a real Azure VM: Azure IMDS detection, OTLP metrics/logs/traces reaching CloudWatch, and the Azure web-identity credential chain (oidctoken -> AssumeRoleWithWebIdentity -> sigv4auth). Includes the terraform/azurevm harness (references the AWS OIDC provider by ARN; no role creation; uploads the runner-built .deb to the VM over SSH), AZUREVM compute type, and a one-time Azure-side setup.sh.
- Default VM size Standard_D2s_v5 is capacity-restricted in eastus for some subscriptions (SkuNotAvailable 409); switch to Standard_D2s_v7. - Add a network security group with an inbound SSH rule scoped to runner_ip and associate it to the NIC; Azure's implicit default denies all inbound, so the file/remote-exec provisioners could not reach port 22. null_resource now depends on the association so SSH is reachable before provisioning.
Align setup.sh defaults with the shared integ-test resource group and vnet naming so it reuses existing shared infrastructure instead of creating a differently-named set.
Azure VMs have no AWS instance metadata, so the agent (with the new env fallback) resolves region from AWS_REGION at translation time - pass it on the sudo'd fetch-config invocation. Replace the 'tee -a etc/env-config' lines: the agent reads etc/env-config.json (strict JSON) at service startup, so plain KEY=VALUE lines in etc/env-config were never loaded and the systemd-spawned agent would miss CWAGENT_ROLE_ARN needed to expand the role_arn placeholder in the translated YAML. Use the supported '-setenv ... -envconfig' merge instead. USE_DEFAULT_CONFIG is translation-time only and needs no persistence.
Namespaced as test/azure/vm and terraform/azure/vm so future Azure compute targets (e.g. AKS) can live alongside under the same prefix. Assertion design: - Use lowercase "azure" marker (OTel structured logs are lowercase) - No negative log-marker checks in the CredentialChain subtest: AssumeRoleWithWebIdentity is the successful API call name, and transient retries leave permanent markers even on recovery. The Metrics/Logs/Traces subtests already prove delivery end-to-end.
… retry window X-Ray's OTLP endpoint silently drops traces whose first 4 bytes are not a valid Unix epoch timestamp (seconds). Our previous format used nanoseconds which overflowed the 4-byte field. Generate compliant IDs (first 4 bytes = unix seconds, remaining 12 bytes = sequence padding). Also revert the t.Skip() workaround — Traces now hard-fails — and extend the retry window to 5 attempts × 90s to accommodate X-Ray OTLP indexing latency (typically 3-5 minutes).
GetTraceSummaries depends on X-Ray's indexing pipeline which can lag
5-10+ minutes for OTLP-ingested traces, causing persistent test failures.
Switch to BatchGetTraces with the exact trace IDs we generated — this
queries the raw trace store directly and bypasses indexing entirely.
Record generated OTLP trace IDs during the load window, convert to
X-Ray format (1-{8hex}-{24hex}), and fetch them directly.
The X-Ray OTLP endpoint (xray.{region}.amazonaws.com/v1/traces) requires
xray:PutSpans — a separate IAM action from the legacy PutTraceSegments.
Without it, the agent's trace exports are silently rejected with 403,
which is why traces never appeared in X-Ray despite the endpoint
accepting connections.
Also add xray:BatchGetTraces which the test binary needs for the
direct trace-ID-based validation (bypasses GetTraceSummaries indexing).
Dump agent log lines containing error/warn/xray/traces keywords after the load window to diagnose why the otlphttp/traces exporter silently fails to deliver spans to X-Ray. The IAM policy is now correct (PutSpans granted) but traces still don't arrive — need visibility into agent-side export errors.
The X-Ray OTLP endpoint requires Transaction Search (trace segment destination = CloudWatchLogs). Spans ingested this way are stored in the aws/spans log group in full; GetTraceSummaries and BatchGetTraces only see the indexed subset (1% by default), so they return empty for low-volume test traffic. Query aws/spans for the exact generated trace IDs instead and require all of them to be present.
Provisions an AKS cluster with OIDC issuer, creates an AWS IAM role trusting the cluster's service account token, deploys the CWA DaemonSet from the pre-built ECR image with projected STS token auth, runs a hostNetwork load-generator Job pushing OTLP for 3 minutes, then validates all three signals (metrics/logs/traces) reach CloudWatch. Traces validated via aws/spans (Transaction Search) same as the VM test.
The agent entrypoint checks IsRunningInContainer() to decide whether to use the container config path (--input-dir). Without this env var set, translateConfig falls back to host paths and fails inside the container.
…sform
The agent's transform_identity_k8s fills service.namespace from
k8s.namespace.name, so the routed stream is
{k8s.namespace.name}/{service.namespace}/{service.name} — verified live
in CloudWatch for the previous run's cluster.
- Protect generatedTraceIDs/traceSeq with sync.Mutex; snapshot under lock after sendTelemetry stops so the read has a happens-before. - Add AssertLogsNotEmpty to validateLogs (matches the AKS path). - Close resp.Body in post() to avoid leaking connections across the ~54 POST calls per test run.
movence
force-pushed
the
hsookim/azurevm-integ-test
branch
from
July 24, 2026 15:37
ba81e53 to
03d7a2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
The agent supports running on non-AWS hosts (Azure VM and AKS) with the default OTel config, authenticating to AWS via web-identity federation — but there was no integration test coverage proving the credential chain and end-to-end telemetry delivery (metrics, logs, traces) to CloudWatch from either environment.
Description of changes
Adds two new test suites, each provisioned by terraform and dispatched from the companion workflow PR (aws/amazon-cloudwatch-agent#2216):
Azure VM (
terraform/azure/vm,test/azure/vm).deb, and starts the agent with the default OTel config.AssumeRoleWithWebIdentity).aws/spansvia Logs Insights — the OTLP endpoint requiresxray:PutSpansand X-Ray trace IDs with Unix seconds in the first 4 bytes).AKS (
terraform/azure/aks,test/azure/aks)sub/audconditions), and deploys the agent as a DaemonSet from the integration-test ECR image using a projected service-account token (audiencests.amazonaws.com).AZUREVMandAKScompute types inenvironment/computetype.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
03d7a2a2). Prior runs: 30048181167, 30029802668, 30018222183. AKS:TestAKS/Metrics,TestAKS/Logs,TestAKS/Tracesall PASS; Azure VM passing across 11 consecutive runs.AssertLogsNotEmptyto prevent vacuous passes.go vet/gofmtclean on new packages;terraform validateclean on both stacks.Worth noting
sts:AssumeRoleon the same role that the projected token federates into, which since the 2022 IAM change requires the role to explicitly trust itself (account-root principal +aws:PrincipalArncondition).host.idwith the node VMSS UUID, so metrics are scoped by thek8s.cluster.nameresource attribute instead.{k8s.namespace.name}/{service.namespace}/{service.name}(the identity transform fillsservice.namespacefromk8s.namespace.name); the test asserts the verified destination.aws_ecr_authorization_token) so no JSON passes through shell quoting.