[Feat] Dynamic Rate Limiter - Allow defining a rate limit policy by model + error #16513
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| f"[Dynamic Rate Limit] Tracked failure for deployment {deployment_id}, " | ||
| f"provider {custom_llm_provider}, error type {error_type}" |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 1 day ago
To fix this problem, avoid logging sensitive identifiers such as API keys, their hashes, deployment IDs (if derived from credentials), or any values that originate from authorization secrets. In this context, the deployment_id logged on line 1553 potentially exposes sensitive data, so we should redact it or replace it with a non-sensitive stand-in (e.g., "[REDACTED]"). Changing only the log output while keeping the rest of the logic intact preserves functionality.
How to fix:
- On line 1553, when logging, mask or redact the deployment_id to prevent leakage of sensitive information.
- You may use a general placeholder like "[REDACTED]" or something more descriptive, depending on real use-case knowledge.
- No new imports are needed, and the edit only affects the single logging line.
Files/regions to change:
- Edit the logging statement in
litellm/proxy/hooks/parallel_request_limiter_v3.py, specifically lines 1553-1555.
| @@ -1550,7 +1550,7 @@ | ||
| error_type=error_type, | ||
| ) | ||
| verbose_proxy_logger.debug( | ||
| f"[Dynamic Rate Limit] Tracked failure for deployment {deployment_id}, " | ||
| f"[Dynamic Rate Limit] Tracked failure for deployment [REDACTED], " | ||
| f"provider {custom_llm_provider}, error type {error_type}" | ||
| ) | ||
|
|
| rpm_limit_type = metadata.get("rpm_limit_type") | ||
| tpm_limit_type = metadata.get("tpm_limit_type") | ||
|
|
||
| # Get dynamic rate limit policy from general_settings |
There was a problem hiding this comment.
Is this only going to work with rpm_limit_type/tpm_limit_type at key level. We also have that at team level.
[Feat] Dynamic Rate Limiter - Allow defining a rate limit policy by model + error
Fixes LIT-1389
This PR implements a dynamic rate limit policy that allows fine-grained control over when rate limits are enforced based on provider-specific error thresholds. When using dynamic rate limiting (
rpm_limit_type: "dynamic"ortpm_limit_type: "dynamic"), rate limits will only be enforced when specific error types exceed configured thresholds per provider.Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🆕 New Feature
✅ Test
Changes