Fix monthly quota baseline recovery after aggregation gaps

[crliao]

Summary

Fixes AMBRY-14502.

During the Aug 2026 prod-lva1 incident, account 1085's monthly STORAGE_IN_GB baseline was about 18 TB below peer fabrics. Once aggregation caught up, the apparent monthly delta jumped from about 13 TB to 31.5 TB and uploads began receiving 429s.

The actual task order is current aggregation write first, monthly rollover second. Because AggregatedAccountReports.updatedAt is refreshed by changed-row writes and host report timestamps do not prove report completeness, neither is a safe completed-run signal. This change adds a small durable aggregation state instead:

• persist the last successful aggregation completion time
• when a gap of at least 16 hours crosses a month boundary, continue writing the fresh aggregate but defer the monthly snapshot for exactly one successful cycle
• replace the current month's baseline from the next successful aggregate, then clear recovery state atomically
• ignore long same-month gaps so legitimate monthly usage is not erased
• treat initial null state as schema-rollout bootstrap, not an outage
• use optimistic locked state transitions so retries, restarts, and overlapping task executions cannot repeatedly reset the baseline
• replace monthly rows and advance snapshot state/version in one transaction
• make quota readers reload a corrected same-month baseline when its month or snapshot version changes, and atomically replace/retry local backup persistence

This can slightly under-report usage for the remainder of a recovered month because the corrected baseline starts near zero. That is intentional: avoiding customer-facing 429s from a stale baseline is the higher-priority failure mode.

Deployment and rollback

The DDL adds nullable/defaulted columns and is backward compatible. Required rollout order is:

1. apply DDL
2. deploy all aggregation-task hosts and allow old tasks to drain
3. deploy quota readers

That order matters because an old aggregator can rewrite a same-month monthly table without incrementing the new snapshot version. Fully detecting that case would require a broader trigger/schema protocol or expensive snapshot fingerprinting.

For rollback, roll back quota readers first and aggregators second; retain the additive DDL. Existing binaries ignore the added columns, and retaining them preserves recovery state for a subsequent redeploy.

Durability risk

This does not change blob write/delete paths or client success callbacks. It changes quota metadata only. Monthly snapshot replacement and its state/version transition are transactional; SQL failures roll back rather than exposing an empty or partially replaced monthly table. Current aggregate writes are never suppressed, including when the recovery-state read fails.

Testing Done

• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-clustermap:test --tests 'com.github.ambry.clustermap.MySqlReportAggregatorTaskTest' :ambry-mysql:test --tests 'com.github.ambry.accountstats.AggregatedAccountReportsDaoTest' :ambry-quota:test --tests 'com.github.ambry.quota.storage.MySqlStorageUsageRefresherUnitTest'
• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-api:test :ambry-mysql:test :ambry-clustermap:test :ambry-quota:test
• JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :ambry-mysql:compileIntegrationTestJava :ambry-quota:compileIntegrationTestJava
• MySQL-backed integration execution was not run locally because no ambry_container_storage_stats MySQL service is available; the integration test sources compile.
• ./gradlew build -x test reaches an existing unrelated repository enforcement failure at HostThrottleConfig.java:120 for direct ObjectMapper construction.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 45.84615% with 176 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.65%. Comparing base (52ba813) to head (977ad56).
⚠️ Report is 420 commits behind head on master.

Files with missing lines	Patch %	Lines
...mbry/accountstats/AggregatedAccountReportsDao.java	53.61%	71 Missing and 6 partials ⚠️
...ub/ambry/clustermap/MySqlReportAggregatorTask.java	0.00%	60 Missing ⚠️
...ry/accountstats/AggregatedAccountReportsState.java	42.10%	11 Missing ⚠️
.../ambry/accountstats/InmemoryAccountStatsStore.java	0.00%	11 Missing ⚠️
...mbry/quota/storage/MySqlStorageUsageRefresher.java	78.43%	7 Missing and 4 partials ⚠️
...hub/ambry/accountstats/AccountStatsMySqlStore.java	62.50%	6 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3298       +/-   ##
=============================================
- Coverage     64.24%   50.65%   -13.59%     
+ Complexity    10398     8722     -1676     
=============================================
  Files           840      940      +100     
  Lines         71755    81058     +9303     
  Branches       8611     9786     +1175     
=============================================
- Hits          46099    41061     -5038     
- Misses        23004    36585    +13581     
- Partials       2652     3412      +760     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.