[WIP] KAFKA-17928: Make remote log manager thread-pool configs dynamic #17793
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add isReady() method to RemoteLogMetadataManager interface to check if a partition is ready to serve remote storage operations. Implement the method across concrete classes with appropriate checks: - Add default implementation in RemoteLogMetadataManager interface - Add implementation in NoOpRemoteLogMetadataManager - Add implementation in ClassLoaderAwareRemoteLogMetadataManager - Add implementation in TopicBasedRem
…heck Add ability to dynamically update remote storage thread pool sizes and implement partition readiness check: - Add methods to RemoteLogManager to update thread pool sizes: - updateCopyThreadPoolSize - updateExpirationThreadPoolSize - updateReaderThreadPoolSize - Add thread pool configuration properties to DynamicRemoteLogConfig - Add RLMM readiness check before executing remote storage tasks - Update validation and reconfiguration logic in DynamicRemoteLogConfig This change allows operators to tune thread pool sizes at runtime and prevents operations on partitions that aren't ready for remote storage operations.
github-actions
bot
added
core
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add ability to dynamically update remote storage thread pool sizes and implement
partition readiness check:
This change allows operators to tune thread pool sizes at runtime and prevents
operations on partitions that aren't ready for remote storage operations.
Related Link
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)