Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Elasticsearch reindexing step of the image data refresh triggers DB alerts #5379

Open
krysal opened this issue Feb 18, 2025 · 0 comments
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs

Comments

@krysal
Copy link
Member

krysal commented Feb 18, 2025

Problem

We keep getting this unactionable alarm during the DR process, and to avoid creeping the alerts channel, it'd be better to silence it while the process is complete, as we do with ES alarms.

Image


Image

Description

Include the name of the alarm, API Production DB CPU above 50% utilization, in the list of alarms to silence.

cw_wrapper = CloudWatchWrapper(cloudwatch.get_conn())
for percentage in [50, 90]:
cw_wrapper.enable_alarm_actions(
f"ES Production CPU utilization above {percentage}%", enable
)

Or change the mechanism to get the list from an Airflow variable; this way, future changes won't require code modifications, as it can be tweaked through the Airflow UI.

@krysal krysal added ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Feb 18, 2025
@openverse-bot openverse-bot moved this to 📋 Backlog in Openverse Backlog Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant