Swap meta allocators with much more reserve. #202
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We increase the reserve from 2x to 3x the minimum number of blocks needed for our reserve, and change the algorithm that determines when to swap them. The old algorithm swaps them if _avail is just larger than _freed. While the simplest algorithm, it suffers from the problem that in practice, when we hit ENOSPC conditions, it will almost always swap on every iteration when we hit our low water mark. In our testing, we regularly see failures because what is effectively happening is that we starve both allocators by slowly draining them block by block, trying to do work. The work of course requires us to drain more blocks to commit changes. This cycle doesn't end until both allocators are almost completely drained and not enough blocks remain to do any real work anymore. The new algorithm will not swap allocators unless _freed is 50% larger than _avail. The outcome is that, during meta space pressure, we're allowing _avail to slowly drain down to the same levels as before, effectively. However, we then swap to _freed which is now 1.5x larger. This results in us being able to do a whole chunk of work without needing to swap. While draining _avail for longer, we allow work to commit and recycle blocks back into _freed muich more effectively. Signed-off-by: Auke Kok <[email protected]>
|
retest |
1 similar comment
|
retest |
8 tasks
|
retest |
2 similar comments
|
retest |
|
retest |
|
Ironically, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the core of the meta allocator swapping change: swap a little bit earlier, but always swap only when _freed is much larger than _avail.
From my observations I can see that without this change, both allocators bleed slowly to empty without either of them regaining enough over time to continue work. In effect, we never recover from ENOSPC. Everything grinds to a halt, the allocators just swap endlessly and transactions take forever to complete.
With this change we avoid all that - we purposely let our _avail allocator go much lower than _freed before we swap. This assures that whenever we swap we let enough blocks recycle post transactions into _freed so that it gets us out of ENOSPC again, and into a healthy state again.
I think.