feat: add token budget validation to simpleReadFileTool #9906
Conversation
- Import validateFileTokenBudget and truncateFileContent helpers - Import getModelMaxOutputTokens and ANTHROPIC_DEFAULT_MAX_TOKENS - Add token budget validation in normal file read path - Truncate file content if it exceeds available context budget - Display appropriate notice when truncation occurs - Mirrors the behavior of ReadFileTool for consistency Fixes context overflow issues in Grok and other models using simpleReadFileTool
Test coverage added successfully. All 15 tests pass, covering token budget validation for files exceeding budget, within budget, preview mode, and model-specific context calculations.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| // Validate if file content fits within token budget | ||
| const budgetResult = await validateFileTokenBudget( | ||
| fullPath, | ||
| contextWindow - maxOutputTokens, | ||
| contextTokens || 0, | ||
| ) |
There was a problem hiding this comment.
The new token budget validation logic lacks test coverage. While the helper functions in fileTokenBudget.ts are tested, there are no tests verifying that simpleReadFileTool correctly integrates this functionality. Tests should cover: (1) files exceeding the token budget that require truncation, (2) files within budget limits, (3) large files triggering preview mode, and (4) the token budget calculation using model-specific context windows. This is particularly important given the PR addresses context overflow errors in production (Sentry issue 7079272598).
Fix it with Roo Code or mention @roomote and request a fix.
There was a problem hiding this comment.
Fixaroo
See task on Roo Cloud
Added test coverage for simpleReadFileTool token budget validation. All local checks passed.
hannesrudolph
added
the
Issue/PR - Triage
- Add tests for files within budget limits (no truncation) - Add tests for files exceeding token budget (require truncation) - Add tests for large files triggering preview mode - Add tests for model-specific context window calculations - Add tests for line count tracking with truncation - Add tests for integration with existing maxReadFileLine behavior
Adds token budget validation to simpleReadFileTool() to prevent context window overflow errors, mirroring the behavior already implemented in ReadFileTool.
Changes
validateFileTokenBudgetandtruncateFileContenthelpers fromfileTokenBudget.tsgetModelMaxOutputTokensandANTHROPIC_DEFAULT_MAX_TOKENSfor context calculationImpact
This fix addresses context overflow issues in models that use simpleReadFileTool (particularly Grok), preventing "context overflow" errors like those reported in the Sentry issues.
Related Issues
View task on Roo Code Cloud