Skip to content

[python_lambda] Add scenario for testing the python tracer for AWS Lambda #4891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
florentinl merged 14 commits into from
Aug 20, 2025
Merged

[python_lambda] Add scenario for testing the python tracer for AWS Lambda #4891

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b3a7c3f
Add helper container to translate http requests to API Gateway Events
florentinl Jun 30, 2025
1cc7b63
[python] Add aws apigw-rest weblog container
florentinl Jun 30, 2025
d794866
Add scenario for running lambdas inside Docker
florentinl Jun 30, 2025
f319696
Add lambda-proxy to build image job
florentinl Jul 1, 2025
e455383
Move lambda weblog to a different library
florentinl Jul 4, 2025
e0b9a07
Add manifest for python_lambda
florentinl Jul 4, 2025
9ee01f0
Fix build lambda layer issue
florentinl Jul 4, 2025
00ccdf6
Easy fixes after review
florentinl Jul 8, 2025
c87c323
Add aws_lambda documentation
florentinl Jul 8, 2025
f9ce6fe
[python_lambda] fetch artifact from github action workflow
florentinl Jul 11, 2025
f20a5de
[python_lambda] simplify installation of datadog_lambda
florentinl Jul 11, 2025
b51825f
[python_lambda] update manifest
florentinl Jul 23, 2025
9c2310d
[python_lambda] fix issues preventing some tests
florentinl Jul 24, 2025
219f30f
fix minor tweaks
florentinl Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/compute-impacted-libraries.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@ jobs:
# temporary print to see what's hapenning on differents events
print(json.dumps(github_context, indent=2))

libraries = "cpp|cpp_httpd|cpp_nginx|dotnet|golang|java|nodejs|php|python|ruby|java_otel|python_otel|nodejs_otel"
libraries = "cpp|cpp_httpd|cpp_nginx|dotnet|golang|java|nodejs|php|python|ruby|java_otel|python_otel|nodejs_otel|python_lambda"
result = set()

# do not include otel in system-tests CI by default, as the staging backend is not stable enough
# all_libraries = {"cpp", "dotnet", "golang", "java", "nodejs", "php", "python", "ruby", "java_otel", "python_otel", "nodejs_otel"}
all_libraries = {"cpp", "cpp_httpd", "cpp_nginx", "dotnet", "golang", "java", "nodejs", "php", "python", "ruby"}
all_libraries = {"cpp", "cpp_httpd", "cpp_nginx", "dotnet", "golang", "java", "nodejs", "php", "python", "ruby", "python_lambda"}

if github_context["ref"] == "refs/heads/main":
print("Merge commit to main => run all libraries")
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/compute-workflow-parameters.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ jobs:
with:
name: binaries_dev_${{ inputs.library }}
path: binaries/
include-hidden-files: ${{ inputs.library == 'python' }}
include-hidden-files: ${{ inputs.library == 'python' || inputs.library == 'python_lambda' }}
- name: Set unique ID
id: unique_id
run: echo "value=$(openssl rand -hex 8)" >> $GITHUB_OUTPUT
6 changes: 6 additions & 0 deletions .github/workflows/run-end-to-end.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,9 @@ jobs:
- name: Build weblog
id: build
run: SYSTEM_TEST_BUILD_ATTEMPTS=3 ./build.sh ${{ inputs.library }} -i weblog -w ${{ inputs.weblog }}
- name: Build Lambda Proxy
if: ${{ endsWith(inputs.library, '_lambda') }}
run: ./build.sh python_lambda -i lambda-proxy

- name: Run APPSEC_STANDALONE scenario
if: always() && steps.build.outcome == 'success' && contains(inputs.scenarios, '"APPSEC_STANDALONE"')
Expand Down Expand Up @@ -420,6 +423,9 @@ jobs:
DD_APP_KEY_2: ${{ secrets.DD_APP_KEY_2 }}
DD_API_KEY_3: ${{ secrets.DD_API_KEY_3 }}
DD_APP_KEY_3: ${{ secrets.DD_APP_KEY_3 }}
- name: Run APPSEC_LAMBDA_DEFAULT scenario
if: always() && steps.build.outcome == 'success' && contains(inputs.scenarios, '"APPSEC_LAMBDA_DEFAULT"')
run: ./run.sh APPSEC_LAMBDA_DEFAULT
- name: Run all scenarios in replay mode
if: success() && steps.build.outcome == 'success' && inputs.enable_replay_scenarios
run: utils/scripts/replay_scenarios.sh
Expand Down
63 changes: 63 additions & 0 deletions docs/scenarios/aws_lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Lambda Testing scenario

The Lambda scenario is a variation on the [classical architecture](../architecture/overview.md#what-are-the-components-of-a-running-test) of the system-tests tailored to evaluate the `AWS Lambda` variants of the tracers when used to serve HTTP requests.

To achieve this we simulate the following AWS deployment architecture inside the system-tests using AWS provided tools :

```mermaid
graph LR
A[Incoming HTTP Request] -->|HTTP| B[AWS Managed Load Balancer]
B -->|event: request as JSON| C[AWS Lambda]
```

The AWS Managed Load Balancer could be any of the following ones:
- API Gateway
- Application Load Balancer
- Lambda function url service

To do this, we rely on two tools from AWS to emulate Lambda and Load Balancers:
- [AWS Lambda Runtime Interface Emulator](https://github.com/aws/aws-lambda-runtime-interface-emulator)
- [AWS SAM cli](https://github.com/aws/aws-sam-cli)

>Note: for now only the python variant ([`datadog_lambda`](https://github.com/DataDog/datadog-lambda-python)) is being tested simulating an `API Gateway`

## Key differences with end to end scenarios

To replace the **AWS Managed Load Balancer**, we run a dedicated container in front of the weblog named **Lambda Proxy**. It is responsible for converting the incoming request to a *lambda event* representation, invoking the lambda function running inside the weblog and converting back the return value of function to an http response.

The **Lambda Function** runs inside the **Weblog Container** thanks to the *AWS Lambda Runtime Interface Emumlator*.


There is no **Agent Container**, the **Datadog Extension** (equivalent to the **Datadog Agent** in the context of lambda) needs to run inside the **Weblog Container**, the [**Application Proxy Container**](../architecture/overview.md#application-proxy-container) therefore needs to send traces back to the **Weblog Container**.


```mermaid
flowchart TD
TESTS[Tests Container] -->|Send Requests| LambdaProxy
LambdaProxy[Lambda Proxy] -->|Send Lambda Event| Application
subgraph APP[Application Container]
socat[socat *:8127] --> Extension
Extension[Extension localhost:8126]
Application[Application *:8080]
end
Application --> | Send Traces | APPPROXY
APPPROXY[Application Proxy] --> | Send back traces | socat
APPPROXY -->|mitmdump| TESTS
Extension --> AGENTPROXY
AGENTPROXY[Agent Proxy] -->|remote request| BACKEND
AGENTPROXY -->|mitmdump| TESTS
BACKEND[Datadog] -->|trace API| TESTS
```

## Specific considerations for the weblogs

On top of responding to the regular [`/healthcheck`](../weblog/README.md#get-healthcheck) endpoint.

Lambda Weblogs should respond the same JSON dict response to the non HTTP event:
```json
{
"healthcheck": true
}
```

This is because the healthcheck is sent by the Lambda Weblog container itself which has no knowledge of how to serialize it as the event type expected by the weblog.
21 changes: 21 additions & 0 deletions manifests/python_lambda.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
tests/:
appsec/:
test_alpha.py:
Test_Basic: 7.112.0
test_only_python.py:
Test_ImportError: 7.112.0
test_reports.py:
Test_ExtraTagsFromRule: 7.112.0
Test_Info: 7.112.0
Test_RequestHeaders: 7.112.0
Test_StatusCode: 7.112.0
test_traces.py:
Test_AppSecEventSpanTags: 7.112.0
Test_AppSecObfuscator: 7.112.0
Test_CollectDefaultRequestHeader: 7.112.0
Test_CollectRespondHeaders: 7.112.0
Test_ExternalWafRequestsIdentification: 7.112.0
Test_RetainTraces: 7.112.0
test_versions.py:
Test_Events: 7.112.0
10 changes: 10 additions & 0 deletions tests/appsec/api_security/test_schemas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ def equal_value(t1, t2):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_Headers:
"""Test API Security - Request Headers Schema"""
Expand All @@ -63,6 +64,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_Cookies:
"""Test API Security - Request Cookies Schema"""
Expand All @@ -87,6 +89,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_Query_Parameters:
"""Test API Security - Request Query Parameters Schema"""
Expand All @@ -107,6 +110,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_Path_Parameters:
"""Test API Security - Request Path Parameters Schema"""
Expand All @@ -128,6 +132,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_Json_Body:
"""Test API Security - Request Body and list length"""
Expand All @@ -148,6 +153,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Request_FormUrlEncoded_Body:
"""Test API Security - Request Body and list length"""
Expand Down Expand Up @@ -188,6 +194,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Response_Headers:
"""Test API Security - Response Header Schema"""
Expand All @@ -207,6 +214,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Response_Body:
"""Test API Security - Response Body Schema with urlencoded body"""
Expand All @@ -233,6 +241,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Schema_Response_on_Block:
"""Test API Security - Response Schemas with urlencoded body
Expand Down Expand Up @@ -293,6 +302,7 @@ def test_request_method(self):

@rfc("https://docs.google.com/document/d/1OCHPBCAErOL2FhLl64YAHB8woDyq66y5t-JGolxdf1Q/edit#heading=h.bth088vsbjrz")
@scenarios.appsec_api_security
@scenarios.appsec_lambda_api_security
@features.api_security_schemas
class Test_Scanners:
"""Test API Security - Scanners"""
Expand Down
1 change: 1 addition & 0 deletions tests/appsec/test_alpha.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_Basic:
"""Detect attacks on raw URI and headers with default rules"""

Expand Down
5 changes: 3 additions & 2 deletions tests/appsec/test_only_python.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@
@scenarios.appsec_runtime_activation
@scenarios.appsec_standalone
@scenarios.default
@scenarios.appsec_lambda_default
@features.language_specifics
@irrelevant(context.library != "python", reason="specific tests for python tracer")
@irrelevant(context.library not in ("python", "python_lambda"), reason="specific tests for python tracer")
class Test_ImportError:
"""Tests to verify that we don't have import errors due to tracer instrumentation."""

@flaky(context.library == "python@3.2.1" and "flask" in context.weblog_variant, reason="APMRP-360")
def test_circular_import(self):
"""Test to verify that we don't have a circular import in the weblog."""
assert context.library == "python"
assert context.library in ("python", "python_lambda")
interfaces.library_stdout.assert_absence("most likely due to a circular import")
4 changes: 4 additions & 0 deletions tests/appsec/test_reports.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ def _check_service(span, appsec_data): # noqa: ARG001
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_RequestHeaders:
"""Request Headers for IP resolution"""

Expand Down Expand Up @@ -107,6 +108,7 @@ def test_http_request_headers(self):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_TagsFromRule:
"""Tags tags from the rule"""

Expand Down Expand Up @@ -135,6 +137,7 @@ def test_category(self):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_ExtraTagsFromRule:
"""Extra tags may be added to the rule match since libddwaf 1.10.0"""

Expand Down Expand Up @@ -164,6 +167,7 @@ def _get_appsec_triggers(request):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_AttackTimestamp:
"""Attack timestamp"""

Expand Down
22 changes: 18 additions & 4 deletions tests/appsec/test_traces.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_RetainTraces:
"""Retain trace (manual keep & appsec.event = true)"""

Expand Down Expand Up @@ -59,6 +60,7 @@ def validate_appsec_event_span_tags(span):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_AppSecEventSpanTags:
"""AppSec correctly fill span tags."""

Expand All @@ -71,9 +73,15 @@ def test_custom_span_tags(self):

spans = [span for _, span in interfaces.library.get_root_spans()]
assert spans, "No root spans to validate"
spans = [s for s in spans if s.get("type") == "web"]
assert spans, "No spans of type web to validate"
spans = [s for s in spans if s.get("type") in ("web", "serverless")]
assert spans, "No spans of type web or serverless to validate"
for span in spans:
if span.get("type") == "serverless" and "_dd.appsec.unsupported_event_type" in span["metrics"]:
# For serverless, the `healthcheck` event is not supported
assert (
span["metrics"]["_dd.appsec.unsupported_event_type"] == 1
), "_dd.appsec.unsupported_event_type should be 1 or 1.0"
continue
assert "_dd.appsec.enabled" in span["metrics"], "Cannot find _dd.appsec.enabled in span metrics"
assert span["metrics"]["_dd.appsec.enabled"] == 1, "_dd.appsec.enabled should be 1 or 1.0"
assert "_dd.runtime_family" in span["meta"], "Cannot find _dd.runtime_family in span meta"
Expand All @@ -84,14 +92,15 @@ def test_custom_span_tags(self):
def setup_header_collection(self):
self.r = weblog.get("/headers", headers={"User-Agent": "Arachni/v1", "Content-Type": "text/plain"})

@bug(library="python_lambda", reason="APPSEC-58202")
@bug(context.library < f"python@{PYTHON_RELEASE_GA_1_1}", reason="APMRP-360")
@bug(context.library < "java@1.2.0", weblog_variant="spring-boot-openliberty", reason="APPSEC-6734")
@bug(
context.library < "nodejs@5.57.0",
weblog_variant="fastify",
reason="APPSEC-57432", # Response headers collection not supported yet
)
@irrelevant(context.library not in ["golang", "nodejs", "java", "dotnet"], reason="test")
@irrelevant(context.library not in ["golang", "nodejs", "java", "dotnet", "python_lambda"], reason="test")
@irrelevant(context.scenario is scenarios.external_processing, reason="Irrelevant tag set for golang")
def test_header_collection(self):
"""AppSec should collect some headers for http.request and http.response and store them in span tags.
Expand All @@ -113,7 +122,7 @@ def test_header_collection(self):
@bug(context.library < "java@0.93.0", reason="APMRP-360")
def test_root_span_coherence(self):
"""Appsec tags are not on span where type is not web, http or rpc"""
valid_appsec_span_types = ["web", "http", "rpc"]
valid_appsec_span_types = ["web", "http", "rpc", "serverless"]
spans = [span for _, _, span in interfaces.library.get_spans()]
assert spans, "No spans to validate"
assert any("_dd.appsec.enabled" in s.get("metrics", {}) for s in spans), "No appsec-enabled spans found"
Expand All @@ -134,6 +143,7 @@ def test_root_span_coherence(self):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_AppSecObfuscator:
"""AppSec obfuscates sensitive data."""

Expand Down Expand Up @@ -285,6 +295,7 @@ def validate_appsec_span_tags(span, appsec_data): # noqa: ARG001
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_CollectRespondHeaders:
"""AppSec should collect some headers for http.response and store them in span tags."""

Expand All @@ -295,6 +306,7 @@ def setup_header_collection(self):
context.scenario is scenarios.external_processing,
reason="The endpoint /headers is not implemented in the weblog",
)
@bug(library="python_lambda", reason="APPSEC-58202")
def test_header_collection(self):
def assert_header_in_span_meta(span, header):
if header not in span["meta"]:
Expand All @@ -313,6 +325,7 @@ def validate_response_headers(span):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_CollectDefaultRequestHeader:
HEADERS = {
"User-Agent": "MyBrowser",
Expand Down Expand Up @@ -346,6 +359,7 @@ def test_collect_default_request_headers(self):
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_ExternalWafRequestsIdentification:
def setup_external_wafs_header_collection(self):
self.r = weblog.get(
Expand Down
1 change: 1 addition & 0 deletions tests/appsec/test_versions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
@features.envoy_external_processing
@scenarios.external_processing
@scenarios.default
@scenarios.appsec_lambda_default
class Test_Events:
"""AppSec events uses events in span"""

Expand Down
Loading