fix(core): StreamString performance improvements with large buffers (positional index instead of memory move) #11232
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👋 Hello mathieucarbou, we appreciate your contribution to this project! 📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more. 🖊️ Please also make sure you have read and signed the Contributor License Agreement for this project. Click to see more instructions ...
Review and merge process you can expect ...
|
mathieucarbou
changed the title
mathieucarbou
changed the title
Test Results 76 files 76 suites 12m 48s ⏱️ Results for commit b1b6615. |
Memory usage test (comparing PR against master branch)The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.
Click to expand the detailed deltas report [usage change in BYTES]
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Question about this PR: if there are 2 tasks (producer/consumer), one writes data to the StreamString and the other reads data from the same StreamString, wouldn't its String Buffer grow for ever? |
|
For the issue with buffers, I'd suggest using some other structure, such as this one: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Using
StreamStringon large buffers causes large movements of memory data when reading which can cause the TWDT to trigger, heap fragmentation or crash due to allocation failures.StreamStringis definitely slower than cbuf.h for large buffers.Explanation of the issue with a small MRE: ESP32Async/ESPAsyncWebServer#148 (comment)
This issue was discovered in the OpenDTU project : tbnobody/OpenDTU#2535
Note: the code is not tested yet: this is a proposal meant to open a discussion on a solution.
Sadly it does not solve the issue fully but aims at being backward compatible. It avoids at least to do a memory move for each byte read.
This implementation version (unlike #11229) aims at avoiding any memory move when reading, so that the reading can be really fast. Heap consumption stays the same since the String is not manipulated. As such, the backing String, if accessed through the API, will not change, which is a difference in behaviour, so this implementation is not really backward compatible.