Improve debuggability of SQS, Lambda, EC2, and RDS hooks#64661
Open
shivaam wants to merge 1 commit intoapache:mainfrom
Open
Improve debuggability of SQS, Lambda, EC2, and RDS hooks#64661shivaam wants to merge 1 commit intoapache:mainfrom
shivaam wants to merge 1 commit intoapache:mainfrom
Conversation
- Add debug logging before/after API calls in all four hooks, following the pattern from the Airbyte provider (apache#51503) and DynamoDB hook (apache#64629) - Fix `raise e` → bare `raise` in RDS hook (get_export_task_state, get_event_subscription_state) to avoid misleading extra traceback frames
boring-cyborg
bot
added
area:providers
provider:amazon
eladkal
reviewed
Apr 3, 2026
|
|
||
| return self.conn.stop_instances(InstanceIds=instance_ids) | ||
| result = self.conn.stop_instances(InstanceIds=instance_ids) | ||
| self.log.debug("stop_instances response: %s", result.get("StoppingInstances")) |
Contributor
There was a problem hiding this comment.
To be honest I think adding log.debug everywhere has very little value if any.
To leverage this users must change the logging configuration in the Airflow cluster. I don't know if anyone is actually doing it in production. It cary risks and interruption for other jobs, also logs becomes so spammy that it's very hard to figure out what you actually need.
I think that the improve debuggability track should be about finding a way to change a specific DAG logging level without affecting the whole cluster - then, PRs like this one has real value.
Contributor
There was a problem hiding this comment.
But regardless of my above comment happy to accept it.
eladkal
approved these changes
Apr 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #64629 (DynamoDB hook), applying the same debug logging
and exception handling pattern to four more AWS provider hooks.
Debug logging — adds
self.log.debug()before/after API calls,following the Airbyte provider pattern (#51503):
create_queue,send_message,asend_messageinvoke_lambda,create_lambdaget_instance,stop/start/terminate_instances,get_instance_state_async,wait_for_stateget_*_statemethodsAll logs are at
DEBUGlevel — no-ops unless explicitly enabled.Exception handling (RDS only):
from eexception chaining to all 6raise AirflowNotFoundException(e)to preserve
__cause__in tracebacksraise ewith bareraiseinget_export_task_stateandget_event_subscription_stateto avoid adding a misleading extraframe to the traceback
Testing: All 53 existing tests across the four hook test files
pass unchanged.
Part of the Debugging Improvements — Airflow 3 initiative.
related: #64629, #51503
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Claude Opus 4.6) following the guidelines