fix(common-services): add connector.log retention to curator - #64
fix(common-services): add connector.log retention to curator#64gnanirahulnutakki wants to merge 1 commit into
Conversation
connector.log is shipped to Elasticsearch by the fid-exporter sidecar (fid chart source 'connector' -> index connector.log) but has no delete_indices action in curator.logs, so its daily indices are never reclaimed while every other shipped log is pruned at 7 days. Observed on the BSWH clusters before cleanup: bswh-use1 90 indices 55.8 GB oldest 2026-05-13 (78% of index data) bswh-use2 348 indices 167.6 GB oldest 2025-08-28 (85% of index data) Adds connector.log to curator.logs so it inherits the same 7-day delete_indices action as the other 24 prefixes.
|
Superseded by #65, which folds this change into the in-flight common-services 2.0.3 release (#63) instead of landing it on Closing here rather than merging, because this PR cannot go green on its own: its only CI failure is the — and the whole point of leaving #65 carries the commit from this branch cherry-picked unmodified, plus a The measurements and rationale in this PR remain the reference for the change; #65 links back here rather than restating them. |
Problem
connector.logis shipped to Elasticsearch by the fid-exporter sidecar — the fid chart defines the source incharts/fid/values.yaml:…but
curator.logsin this chart has no entry for it. Every other prefix the exporter produces is pruned at 7 days;connector.logis the one exception, so its daily indices accumulate indefinitely.It is also the highest-volume index in the stack — one document per connector event, across every sync agent — so it becomes the dominant consumer of Elasticsearch disk within a few months.
Evidence
Measured on the BSWH clusters (before a manual cleanup):
connector.logindicesbswh-use12026-05-13bswh-use22025-08-28On
bswh-use2that is ~12 months of unpruned data. Curator was running nightly and healthy the whole time — it simply had no action matching this prefix, so every run loggedSkipping action "delete_indices" due to empty listfor the prefixes that had already aged out and never touchedconnector.log.Change
One entry added to
curator.logs. It inherits the chart's existing defaults (action: delete_indices,unit: days,unit_count: 7,direction: older), so it behaves exactly like the other 24 prefixes.Verification
Rendered
templates/elasticcurator/configmap.yamlbefore and after and diffed the result:connector.log, removed noneunit_count: 7action_file.ymlparses as valid YAML in both rendersThe only other diff is the positional renumbering of actions after index 11, which is inherent to
range $indexin the template — curator treats these as opaque action IDs executed in order, so the renumbering is behaviourally inert.Rollout note
This is a values-only change with no chart version bump, to avoid colliding with the in-flight
common-services/2.0.3release (#63). It needs to be picked up by the next published chart version.Consuming tenants pin the dependency explicitly, so they will each need a bump before this reaches a cluster — e.g.
radiantlogic-saas/bswh-use1is oncommon-services 2.0.1andradiantlogic-saas/bswh-use2on2.0.0, both inapp/charts/common-services/Chart.yaml.Curator runs once per cluster in the
-svcnamespace, so this single change covers every-svcnamespace that consumes the chart. The fid chart ships its own curator, but it defaults toenabled: falseand is not deployed on these clusters, so no corresponding change is needed there.