Skip to content

set LLM limits via ENV#348

Closed
bertmelis wants to merge 2 commits into
sakowicz:masterfrom
bertmelis:env-llm-limits
Closed

set LLM limits via ENV#348
bertmelis wants to merge 2 commits into
sakowicz:masterfrom
bertmelis:env-llm-limits

Conversation

@bertmelis
Copy link
Copy Markdown
Contributor

closes #310

Comment thread src/llm-service.ts
Comment on lines +34 to +37
// Set rate limits for the provider or use ENV variables if set
const limits = Number(requestsPerMinute) > 0 ?
{ tokensPerMinute: Number(tokensPerMinute), requestsPerMinute: Number(requestsPerMinute) } :
PROVIDER_LIMITS[this.provider];
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this nullable by default?
Then:

  • when null - use default limits
  • when 0 - turn off rate limits
  • when >0 - set custom rate limits.

This gives us the ability to turn off rate-limiting.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure. I'm not proficient in typescript so give me some time to learn how to do it.

@sakowicz
Copy link
Copy Markdown
Owner

sakowicz commented May 6, 2026

Thanks for kicking this off @bertmelis — your PR pointed at exactly the right gap (no way to override PROVIDER_LIMITS per deployment, which closes #310).

Landed an extended version on master that picks up @sakowicz's review note about nullable trichotomy. The behavior is now:

  • REQUESTS_PER_MINUTE / TOKENS_PER_MINUTE unset (or empty) → fall back to provider default
  • env set to 0 → disable that axis of proactive rate limiting
  • env set to a positive number → custom limit

What's on master:

  • src/utils/parse-rate-limit-env.ts (new) — small parser preserving the unset/0/positive distinction (your original Number(x) > 0 collapsed '0', '', and 'abc' into the same fallback).
  • src/config.ts — exports requestsPerMinuteOverride and tokensPerMinuteOverride typed as number | null.
  • src/llm-service.ts — constructor accepts overrides via the existing options bag; resolves per axis with fallback to PROVIDER_LIMITS; logs distinguish unset / disabled / N/minute.
  • src/container.ts — wires the overrides through.
  • README.md — new "Rate Limit Overrides" section documenting the trichotomy.
  • Unit tests for both the parser and the LlmService constructor (17 new tests).

Closing in favor of the master commit. Thanks again — and don't worry about TypeScript, you flagged the right issue and that's the hard part. Cheers!

@sakowicz sakowicz closed this May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: ENVAR for rate limit

2 participants