fix: prevent context window overflow for directory mentions with MiniMax

[roomote]

This PR attempts to address Issue #9131. Feedback and guidance are welcome.

Problem

When using @ to reference directories in a new task with the MiniMax model, users were encountering a "context window exceeded" error (2013). The MiniMax model has a 192K token context window, and when referencing directories with many files, all file contents were being included without any size limits, causing the overflow.

Solution

Added size limits for directory content when processing @ mentions to prevent context window overflow:

• 50KB total size limit for directory content (approximately 12-15K tokens)
• Maximum 20 files included per directory mention
• 10KB per-file size limit when reading directory contents
• Files are truncated with warning messages if they exceed limits
• Informative notes are added when content is limited or skipped

Changes

• Modified src/core/mentions/index.ts to add configurable size limits in the getFileOrFolderContent function
• Added file counting to limit the number of files processed per directory
• Implemented truncation for large files within directories
• Added clear warning messages when content is limited

Testing

• All existing tests pass successfully
• Tested with npx vitest run core/mentions/__tests__/index.spec.ts
• Tested with npx vitest run core/mentions/__tests__/processUserContentMentions.spec.ts

Impact

• Prevents MiniMax context window overflow when referencing large directories
• Maintains backward compatibility for single file mentions
• Provides clear feedback to users when content is limited

Fixes #9131

---

Important

Adds size limits to directory mentions in index.ts to prevent MiniMax context window overflow, with truncation and warnings for large files.

• Behavior:
  • Adds size limits in getFileOrFolderContent in index.ts to prevent context window overflow for directory mentions.
  • Limits: 50KB total directory content, 20 files per directory, 10KB per file.
  • Truncates files exceeding limits with warning messages.
  • Adds notes when content is limited or skipped.
• Testing:
  • All existing tests pass.
  • Tested with npx vitest run core/mentions/__tests__/index.spec.ts and processUserContentMentions.spec.ts.
• Impact:
  • Prevents MiniMax context window overflow for large directories.
  • Maintains backward compatibility for single file mentions.
  • Provides user feedback when content is limited.

^{This description was created by for e69fd19. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

Rooviewer   See task on Roo Cloud

Review completed. Found 3 issues that need attention:

• Race condition in concurrent file size tracking (lines 297-300)
• Inconsistent size limit enforcement logic (line 323)
• Inefficient file size check before extraction (lines 339-348)

_{Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.}

[ellipsis-dev]

Beware: the shared counters (filesIncluded, totalContentSize) are updated inside asynchronous functions, but the loop’s synchronous check (if (filesIncluded >= MAX_FILES_TO_INCLUDE || totalContentSize >= MAX_DIR_CONTENT_SIZE)) uses values that haven’t yet been updated. This may lead to a race condition where limits are not correctly enforced. Consider processing entries sequentially or otherwise synchronizing updates.

[roomote]

Race condition: totalContentSize, filesIncluded, filesTruncated, and filesSkipped are updated inside async promises that execute concurrently via Promise.all(). Multiple promises can read and update these shared counters simultaneously, leading to inaccurate counts. For example, two promises could both read totalContentSize as 40KB, each add 15KB of content, and both increment the counter, resulting in a final value of 55KB instead of the correct 70KB. This undermines the size limiting logic and could allow the context window to be exceeded.

_{Fix it with Roo Code or mention @roomote and request a fix.}

[roomote]

Inconsistent size limit enforcement: Line 323 checks totalContentSize >= MAX_DIR_CONTENT_SIZE before processing files, but line 351 checks totalContentSize + content.length > MAX_DIR_CONTENT_SIZE after reading. This means when totalContentSize equals or exceeds the limit, the code still attempts to read files (wasting I/O), only to skip them afterward. The pre-check should use the same logic as the post-check to avoid unnecessary file reads.

_{Fix it with Roo Code or mention @roomote and request a fix.}

[roomote]

Inefficient file size check: The code checks fileStats.size > MAX_FILE_SIZE at line 340, then calls extractTextFromFile(absoluteFilePath, 50) at line 342 to read 50 lines before truncating. For large files, extractTextFromFile performs expensive I/O operations (file stats, line counting, file reading) before the content is truncated. The file size check should happen before any extractTextFromFile call to avoid unnecessary disk reads on files that will be truncated anyway.

_{Fix it with Roo Code or mention @roomote and request a fix.}