Improves tokenization and editor performance

[malore350]

Pull Request

Description

This commit introduces several improvements to tokenization and editor performance, especially for large files:

• Optimizes tokenization by handling ASCII content more efficiently and using character offsets instead of byte offsets in the editor.
• Adds logic to skip git gutter updates for large files, which significantly improves the editor's responsiveness.
• Introduces a new utility for detecting large files based on character and line count thresholds, which helps prevent performance bottlenecks.
• Implements a content hash to improve the git gutter change detection.

These changes reduce memory usage and improve the overall user experience, particularly when working with large files.

Related Issues

Fixes #345

Tests

Tested with files of 5,000, 10,000, 20,000, and 50,000 lines of code to verify performance improvements and large file handling.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}