[aws-sdk]Added Initial auto-instrumentation library for aws-sdk #361
Conversation
|
Thanks for opening your first pull request! If you haven't yet signed our Contributor License Agreement (CLA), then please do so that we can accept your contribution. A link should appear shortly in this PR if you have not already signed one. |
| $cmd = $params[0]; | ||
| $builder = $inst->tracer() | ||
| ->spanBuilder("{$c->getApi()->getServiceName()}.{$cmd->getName()}") | ||
| ->setSpanKind(SpanKind::KIND_CLIENT) |
There was a problem hiding this comment.
If ext-curl and the instrumentation were used, we'd end up with 2 client spans in a row, which messes up the hierarchy
There was a problem hiding this comment.
So that is one of the issues i mentioned above. Same goes with Guzzle instrumentation. For OTEL in other languages (like dotnet), there is the option to suppress downstream http instrumentations so the SDK Span is the final Client span in this case. In the case of AWS Instrumentation, we do want the aws span to be the client span. Any ideas what can be done here or any tips?
There was a problem hiding this comment.
In an instrumentation extension i worked before, it was on the most specific instrumentation to check the hierarchy of the currently open spans and modify them as needed.
Can you link to an example for what you mentioned?
There was a problem hiding this comment.
Here's an example. Some context, this is from a slim app where an API call is made to /aws-sdk-call. This function then makes a ListBuckets call to S3 using the AWS SDK. There are 4 spans, the main server span with name = GET /aws-sdk-call -> Internal span with name closure -> Client Span with name s3.ListBuckets -> Client Span with name GET (from guzzle instrumentation).
my_php_app | [Wed Apr 23 05:30:57 2025] 127.0.0.1:55342 [200]: GET /aws-sdk-call
my_php_app | [
my_php_app | {
my_php_app | "name": "GET",
my_php_app | "context": {
my_php_app | "trace_id": "381643ab3693053e6ba9f3ee19d3b2b7",
my_php_app | "span_id": "5da488bc98610623",
my_php_app | "trace_state": "",
my_php_app | "trace_flags": 1
my_php_app | },
my_php_app | "resource": {
my_php_app | "host.name": "160a5c84428f",
my_php_app | "host.arch": "aarch64",
my_php_app | "os.type": "linux",
my_php_app | "os.description": "6.10.14-linuxkit",
my_php_app | "os.name": "Linux",
my_php_app | "os.version": "#1 SMP Thu Mar 20 16:32:56 UTC 2025",
my_php_app | "process.pid": 7,
my_php_app | "process.executable.path": "\/usr\/local\/bin\/php",
my_php_app | "process.owner": "root",
my_php_app | "process.runtime.name": "cli-server",
my_php_app | "process.runtime.version": "8.2.28",
my_php_app | "telemetry.sdk.name": "opentelemetry",
my_php_app | "telemetry.sdk.language": "php",
my_php_app | "telemetry.sdk.version": "dev-main",
my_php_app | "telemetry.distro.name": "opentelemetry-php-instrumentation",
my_php_app | "telemetry.distro.version": "1.1.2",
my_php_app | "service.name": "php-sample-ec2",
my_php_app | "service.version": "dev-main"
my_php_app | },
my_php_app | "parent_span_id": "a8806c84389623ff",
my_php_app | "kind": "KIND_CLIENT",
my_php_app | "start": 1745386257530134205,
my_php_app | "end": 1745386257842614538,
my_php_app | "attributes": {
my_php_app | "url.full": "https:\/\/s3.amazonaws.com\/",
my_php_app | "http.request.method": "GET",
my_php_app | "network.protocol.version": "1.1",
my_php_app | "user_agent.original": "aws-sdk-php\/3.342.32 ua\/2.1 OS\/Linux#6.10.14-linuxkit lang\/php#8.2.28 m\/P,Z,g,b GuzzleHttp\/7",
my_php_app | "http.request.body.size": "",
my_php_app | "server.address": "s3.amazonaws.com",
my_php_app | "url.path": "\/",
my_php_app | "code.function.name": "transfer",
my_php_app | "code.namespace": "GuzzleHttp\\Client",
my_php_app | "code.filepath": "\/app\/vendor\/guzzlehttp\/guzzle\/src\/Client.php",
my_php_app | "code.line.number": 326,
my_php_app | "http.response.status_code": 200,
my_php_app | "http.response.body.size": ""
my_php_app | },
my_php_app | "status": {
my_php_app | "code": "Unset",
my_php_app | "description": ""
my_php_app | },
my_php_app | "events": [],
my_php_app | "links": [],
my_php_app | "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.30.0"
my_php_app | }
my_php_app | ]
my_php_app | [
my_php_app | {
my_php_app | "name": "s3.ListBuckets",
my_php_app | "context": {
my_php_app | "trace_id": "381643ab3693053e6ba9f3ee19d3b2b7",
my_php_app | "span_id": "a8806c84389623ff",
my_php_app | "trace_state": "",
my_php_app | "trace_flags": 1
my_php_app | },
my_php_app | "resource": {
my_php_app | "host.name": "160a5c84428f",
my_php_app | "host.arch": "aarch64",
my_php_app | "os.type": "linux",
my_php_app | "os.description": "6.10.14-linuxkit",
my_php_app | "os.name": "Linux",
my_php_app | "os.version": "#1 SMP Thu Mar 20 16:32:56 UTC 2025",
my_php_app | "process.pid": 7,
my_php_app | "process.executable.path": "\/usr\/local\/bin\/php",
my_php_app | "process.owner": "root",
my_php_app | "process.runtime.name": "cli-server",
my_php_app | "process.runtime.version": "8.2.28",
my_php_app | "telemetry.sdk.name": "opentelemetry",
my_php_app | "telemetry.sdk.language": "php",
my_php_app | "telemetry.sdk.version": "dev-main",
my_php_app | "telemetry.distro.name": "opentelemetry-php-instrumentation",
my_php_app | "telemetry.distro.version": "1.1.2",
my_php_app | "service.name": "php-sample-ec2",
my_php_app | "service.version": "dev-main"
my_php_app | },
my_php_app | "parent_span_id": "8ec2305971576a88",
my_php_app | "kind": "KIND_CLIENT",
my_php_app | "start": 1745386257527632330,
my_php_app | "end": 1745386257846341288,
my_php_app | "attributes": {
my_php_app | "rpc.system": "aws-api",
my_php_app | "rpc.method": "ListBuckets",
my_php_app | "rpc.service": "s3",
my_php_app | "aws.region": "us-east-1",
my_php_app | "code.function": "execute",
my_php_app | "code.namespace": "Aws\\AwsClient",
my_php_app | "code.filepath": "\/app\/vendor\/aws\/aws-sdk-php\/src\/AwsClientTrait.php",
my_php_app | "code.line_number": 56,
my_php_app | "http.status_code": 200,
my_php_app | "aws.requestId": "60PQSDPPNH5CSPMD"
my_php_app | },
my_php_app | "status": {
my_php_app | "code": "Unset",
my_php_app | "description": ""
my_php_app | },
my_php_app | "events": [],
my_php_app | "links": [],
my_php_app | "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.30.0"
my_php_app | }
my_php_app | ]
my_php_app | [
my_php_app | {
my_php_app | "name": "{closure}",
my_php_app | "context": {
my_php_app | "trace_id": "381643ab3693053e6ba9f3ee19d3b2b7",
my_php_app | "span_id": "8ec2305971576a88",
my_php_app | "trace_state": "",
my_php_app | "trace_flags": 1
my_php_app | },
my_php_app | "resource": {
my_php_app | "host.name": "160a5c84428f",
my_php_app | "host.arch": "aarch64",
my_php_app | "os.type": "linux",
my_php_app | "os.description": "6.10.14-linuxkit",
my_php_app | "os.name": "Linux",
my_php_app | "os.version": "#1 SMP Thu Mar 20 16:32:56 UTC 2025",
my_php_app | "process.pid": 7,
my_php_app | "process.executable.path": "\/usr\/local\/bin\/php",
my_php_app | "process.owner": "root",
my_php_app | "process.runtime.name": "cli-server",
my_php_app | "process.runtime.version": "8.2.28",
my_php_app | "telemetry.sdk.name": "opentelemetry",
my_php_app | "telemetry.sdk.language": "php",
my_php_app | "telemetry.sdk.version": "dev-main",
my_php_app | "telemetry.distro.name": "opentelemetry-php-instrumentation",
my_php_app | "telemetry.distro.version": "1.1.2",
my_php_app | "service.name": "php-sample-ec2",
my_php_app | "service.version": "dev-main"
my_php_app | },
my_php_app | "parent_span_id": "03dc30cfb39542ec",
my_php_app | "kind": "KIND_INTERNAL",
my_php_app | "start": 1745386257464926371,
my_php_app | "end": 1745386257846492121,
my_php_app | "attributes": {
my_php_app | "code.function.name": "__invoke",
my_php_app | "code.namespace": "Slim\\Handlers\\Strategies\\RequestResponse",
my_php_app | "code.filepath": "\/app\/vendor\/slim\/slim\/Slim\/Handlers\/Strategies\/RequestResponse.php",
my_php_app | "code.line.number": 28
my_php_app | },
my_php_app | "status": {
my_php_app | "code": "Unset",
my_php_app | "description": ""
my_php_app | },
my_php_app | "events": [],
my_php_app | "links": [],
my_php_app | "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.30.0"
my_php_app | }
my_php_app | ]
my_php_app | [
my_php_app | {
my_php_app | "name": "GET \/aws-sdk-call",
my_php_app | "context": {
my_php_app | "trace_id": "381643ab3693053e6ba9f3ee19d3b2b7",
my_php_app | "span_id": "03dc30cfb39542ec",
my_php_app | "trace_state": "",
my_php_app | "trace_flags": 1
my_php_app | },
my_php_app | "resource": {
my_php_app | "host.name": "160a5c84428f",
my_php_app | "host.arch": "aarch64",
my_php_app | "os.type": "linux",
my_php_app | "os.description": "6.10.14-linuxkit",
my_php_app | "os.name": "Linux",
my_php_app | "os.version": "#1 SMP Thu Mar 20 16:32:56 UTC 2025",
my_php_app | "process.pid": 7,
my_php_app | "process.executable.path": "\/usr\/local\/bin\/php",
my_php_app | "process.owner": "root",
my_php_app | "process.runtime.name": "cli-server",
my_php_app | "process.runtime.version": "8.2.28",
my_php_app | "telemetry.sdk.name": "opentelemetry",
my_php_app | "telemetry.sdk.language": "php",
my_php_app | "telemetry.sdk.version": "dev-main",
my_php_app | "telemetry.distro.name": "opentelemetry-php-instrumentation",
my_php_app | "telemetry.distro.version": "1.1.2",
my_php_app | "service.name": "php-sample-ec2",
my_php_app | "service.version": "dev-main"
my_php_app | },
my_php_app | "parent_span_id": "",
my_php_app | "kind": "KIND_SERVER",
my_php_app | "start": 1745386257463754788,
my_php_app | "end": 1745386257846579830,
my_php_app | "attributes": {
my_php_app | "code.function.name": "handle",
my_php_app | "code.namespace": "Slim\\App",
my_php_app | "code.filepath": "\/app\/vendor\/slim\/slim\/Slim\/App.php",
my_php_app | "code.line.number": 207,
my_php_app | "url.full": "http:\/\/localhost:8080\/aws-sdk-call",
my_php_app | "http.request.method": "GET",
my_php_app | "http.request.body.size": "",
my_php_app | "user_agent.original": "curl\/7.88.1",
my_php_app | "server.address": "localhost",
my_php_app | "server.port": 8080,
my_php_app | "url.scheme": "http",
my_php_app | "url.path": "\/aws-sdk-call",
my_php_app | "http.route": "\/aws-sdk-call",
my_php_app | "http.response.status_code": 200,
my_php_app | "network.protocol.version": "1.1",
my_php_app | "http.response.body.size": ""
my_php_app | },
my_php_app | "status": {
my_php_app | "code": "Unset",
my_php_app | "description": ""
my_php_app | },
my_php_app | "events": [],
my_php_app | "links": [],
my_php_app | "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.30.0"
my_php_app | }
my_php_app | ]
There was a problem hiding this comment.
It makes sense for an instrumentation like Guzzle to create an outgoing http call as a client span because it assumes that span will be the last outgoing span. That's why in dotnet and ruby for example, there is a mechanism to suppress downstream instrumentation calls (in this case, suppress the downstream outgoing call).
There was a problem hiding this comment.
In terms of direction, it's actually both
Neither of those are the responsibility of this package, and should already be handled by other packages provided that the appropriate context propagators are configured (ie, x-ray)
So the problem to be solved seems to be: how can we signal to a guzzle, curl, etc (or more generally, CLIENT-span-producing instrumentation libraries) that they shouldn't record? Or, can they be updated to figure this out for themselves? Can agree that having >1 nested CLIENT spans is bad (does spec say this, @cedricziel ?)
This is possibly something that could be done with a Sampler (something like AvoidNestedClientSpansSampler), but I'm not sure that we have access to the parent's span kind. A Sampler is part of the SDK, so we've got some scope to get creative in how we access it.
There was a problem hiding this comment.
I played around with this idea, and I have a working sampler (actually, an addition to our existing RulesBased sampler) which allows skipping sampling of repeated CLIENT spans.
Context propagation should still work, however I'm pretty sure that visually the trace will be broken because the propagated span context (ie, from the last-executed CLIENT span which sets the outgoing headers, most likely curl), will be from a non-recording span.
Is there a way to suppress the http instrumentation for this specific scenario? I know this is doable in other languages such as Java/DotNet.
@AsakerMohd how do Java/dotnet do this?
There was a problem hiding this comment.
Can agree that having >1 nested CLIENT spans is bad
I agree that having SERVER -> CLIENT -> CLIENT is bad. From the spec "According to the OpenTelemetry specification, the parent of a server span is often a remote client span, and the child of a client span is usually a server span".
About the sampler, how would that work with existing sampler configs tho? For example would this override the existing sampler or would it work in tandem?
In DotNet, seems like each trace has it's own scope and you can suppress downstream instrumentations there: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/SuppressInstrumentationScope.cs. TBH, I haven't dug deep on how exactly this works so I might need to spend some time on it.
There was a problem hiding this comment.
About the sampler, how would that work with existing sampler configs tho?
It would require users to know about and configure the sampler to work with the default/existing sampler.
I think what we should do is treat this as a general problem, and break it out into its own issue. That un-blocks this PR, and we can get an initial version stamped.
edit: created open-telemetry/opentelemetry-php#1579 so I think we can mark this as resolved?
There was a problem hiding this comment.
From my side, this should be ready to merge. The other thing I will probably work on is adding the propagators to the correct places. I guess my question for that: do I need to follow the same pattern as something like this:
?
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #361 +/- ##
=========================================
Coverage 82.13% 82.13%
Complexity 1861 1861
=========================================
Files 137 137
Lines 7804 7804
=========================================
Hits 6410 6410
Misses 1394 1394 Flags with carried forward coverage won't be shown. Click here to find out more. Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Description:
There currently only exists a manual instrumentation library for the AWS SDK. It would be very helpful to also have an auto instrumentation library where customers only need to add one line in composer.json like
"open-telemetry/opentelemetry-auto-aws-sdk": "*",to instrument their AWS SDK calls.This is an initial version of the auto instrumentation agent where it sets attributes such as
rpc.system, rpc.method, rpc.service, aws.region and aws.requestId.. In the future, it should also support aws specific attributes for specific services such as bedrock, sns, sqs, etc. that have attributes such asgen_ai.agent.id, gen_ai.system, etcAnother thing that needs to be supported is propagation which I'm having difficulty figuring out where to add. I see it's being added to libraries such as Slim here:
opentelemetry-php-contrib/src/Instrumentation/Slim/src/SlimInstrumentation.php
Line 108 in 3835977
Enabling http instrumentation like slim produces duplicate spans. Lets say we make a call to S3. In this case, we have an aws-sdk span with
rpc.service = S3and an outgoing http span with"url.full": "https://s3.amazonaws.com/". Is there a way to suppress the http instrumentation for this specific scenario? I know this is doable in other languages such as Java/DotNet.For testing:
I have this sample app: https://github.com/AsakerMohd/phpSample
I added the instrumentation path to that repo in the
composer.jsonfile:Then running the app through docker and making an API call, this is an example trace where only the aws-sdk instrumentation is enabled (notice the parent_id is empty. This is because this is the only instrumentation enabled. When something like slim is also enabled, the id is populated correctly):
For the integration test: Running
vendor/bin/phpunit tests/Integration/AwsSdkInstrumentationTest.phpruns the test there that asserts for the attributes and span and it passes as expected.