Per-route HTTP rate limiting as a backstop against client overload

[nicolastakashi]

Context

prom-analytics-proxy currently has no per-route or global rate limiting at the HTTP layer. A misbehaving or runaway client (typically an operator with too many concurrent reconcile workers, or a client without exponential backoff after errors) can drive request rate well past what the underlying postgres can sustain.

When that happens the proxy doesn't reject; it simply queues — postgres saturates, queries serialize through a small connection pool, individual request latency climbs past client-side deadlines, and the resulting client cancellations create the pq: canceling statement storm pattern (see related issue on statement_timeout).

The system has no proactive shedding behaviour. The only feedback to the client is "your request was too slow and you cancelled it" — which the client may interpret as transient and retry.

Proposal

Add a per-route, per-client-IP token-bucket rate limit, off by default but easy to enable. Conservative defaults so it doesn't surprise existing deployments.

Options:

1. A simple in-process limiter (e.g. golang.org/x/time/rate.Limiter keyed on remote IP) per route or per route-group. Stateless, no external dependency, sheds load with 429 responses. Good enough as a backstop.
2. For multi-replica deployments, a distributed limiter (Redis-backed) — but only worth doing once option 1 is proven insufficient. Same Redis the catalog seenCache already uses could be reused.

Sensible defaults to start:

• seriesMetadata read endpoints: 50 req/s per source IP.
• Ingester POST /api/v1/metrics: 1000 req/s per source IP (writes are batched).
• Configurable per-route via flag or config block; 0 = unlimited.

On exceeding the limit, return 429 Too Many Requests with Retry-After set to the bucket-refill window. Emit a metric (http_requests_rate_limited_total{route,reason}) for visibility.

Impact

This is defensive, not corrective — it doesn't make slow queries faster. But it converts "saturated, no signal" into "saturated, client gets a clear 429" which:

• Lets well-behaved clients back off rather than continuing to push.
• Gives operators a metric signal (429 rate) that lights up before postgres saturation does.
• Bounds the worst case from a single client.

This complements rather than replaces the structural improvements (statement timeout, response cache, partial indexes).

Acceptance criteria

• Configurable limiter, off by default.
• 429 responses with Retry-After.
• http_requests_rate_limited_total{route,client} metric emitted.
• Documented in README with the recommended default per route.

[miguelbernadi]

I think it's worth implementing some rate-limiting. The values proposed here seem too high for what we've seen in practice, and extremely overengineered:

• We do not have multi-deployments right now to require a Redis
• We do not have enough clients for per-IP rules to be relevant
• We do not know if the per-endpoint access pattern is worth discriminating, nor how

I think a global rate limit to protect the number of pool clients we have is enough. It could default to match the DB pool size, or a factor over the pool size (e.g. 1.5x) and it's likely to suffice for the situations we have seen so far.

I'd only invest in a more complex solution if we saw practical issues in the wild with numbers to analyze. Most of the clients are under our control, and fixing the client would be a more effective endeavour in most cases where issues appear.