Django Middleware not working on AWS Lambda

[levesquejf]

I tried to use the Django middleware with an app deployed on Lambda (using Zappa with ELB, not API Gateway) but it’s not working. If the middleware is enabled, the subsegments are sent to the Lambda X-Ray Daemon but they are discarded because the parent_id is not good (according to AWS Support & AWS X-Ray teams).

Without the middleware, it is possible to use patch_db() to get the SQL queries inside X-Ray.

SDK Version: 2.4.2
Django Version: 2.2.5

[chanchiem]

Hey,

Thank you very much for reporting the issue. Can you provide any logs from Lambda that are related to the errors that are happening?

Thanks,
Chan Chiem Jeffery Saeteurn

[levesquejf]

I don't have much logs from lambda except that the subsegments are sent to lambda. It's @haotianw465 that confirmed the issue with the parent_id. You might find some useful information in AWS Support case.

[chanchiem]

Thanks for giving me the support case number. I've filed a request to access the support case and I will get back to you as soon as I get more information about it.

[levesquejf]

Any updates on this @chanchiem ?

[awssandra]

Hi levesquejf,

This developer is unfortunately no longer on this project - This seems to have been missed in the shuffle - apologies! We'll get this prioritized to take a look.

[willarmiros]

Hi @levesquejf,
The issue is that in lambda environments, a facade segment is generated for use in application code because the true segment that represents the request to Lambda is generated by Lambda's backend and unavailable to application code. However, the Django middleware appears to incorrectly be attempting to overwrite the facade segment with a new segment, which is causing this error. The middleware should behave consistently with the other SDKs and perform a no-op instead of generating a new segment when in Lambda environments.

The longer term strategy is to include two segments when serving requests via a web framework in Lambda, one representing the Lambda function (like we have today) and one generated for the middleware like we do in other environments. Related is this issue for node.js.

[levesquejf]

@willarmiros Do you have any quick short-term workaround for this issue? What is the ETA for a fix in the SDK?

[willarmiros]

Calling our internal _refresh_context method defined in the Lambda Context should work. You'd have to do this at the beginning of each flask route you define:

xray_recorder.context._refresh_context()

This creates a new facade segment with the appropriate trace ID and segment ID, which will be able to properly parent subsequent subsegments. Can't provide an official timeline on a fix but we will prioritize it as soon as we can.

[rajbeard]

We are having this same issue. I noticed the fix for Flask, is there a workaround for Django?

[bhautikpip]

Hi @rajbeard ,

calling _refresh_context method basically resolves the inconsistent trace_id issue by generating facade segment again with appropriate trace ID and segment ID. You can try above solution to see if this is leaking issue but I would like to get more information of error you are receiving to make sure you're experiencing the same issue or not. Would you mind providing log information and repro steps (if possible) ?

[rajbeard]

Hello,

As it turns out, we weren't having this issue. What was actually happening was our XRAY_RECORDER config was overwriting the streaming threshold inside of xray_recorder (which is set to 0). Thus, it was giving the appearance that our subsegments were being discarded.