Re-ingest COMMITTING LLC segments when pauseless is disabled #16811
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #16811 +/- ##
============================================
- Coverage 63.47% 63.47% -0.01%
Complexity 1424 1424
============================================
Files 3086 3086
Lines 182340 182355 +15
Branches 27979 27981 +2
============================================
- Hits 115746 115743 -3
- Misses 57662 57687 +25
+ Partials 8932 8925 -7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
2 times, most recently
from
f9ddae8 to
50cd487
Compare
noob-se7en
approved these changes
Sep 15, 2025
...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
Outdated
Show resolved
Hide resolved
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
from
50cd487 to
241ead5
Compare
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
from
241ead5 to
d3f1043
Compare
noob-se7en
reviewed
Sep 17, 2025
...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/apache/pinot/controller/validation/RealtimeSegmentValidationManager.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Pull Request Overview
This PR extends segment re-ingestion capabilities to work for all realtime tables, not just those with pauseless consumption enabled. The change allows the controller to re-ingest COMMITTING LLC segments when pauseless consumption is disabled, while maintaining safety for partial upsert and deduplication scenarios.
Key Changes
- Modified RealtimeSegmentValidationManager to always call repair method regardless of pauseless status
- Updated logging and error handling in PinotLLCRealtimeSegmentManager for better clarity
- Added comprehensive unit tests to verify the behavior for both pauseless enabled and disabled scenarios
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| RealtimeSegmentValidationManager.java | Removes pauseless consumption check and always calls repair method |
| PinotLLCRealtimeSegmentManager.java | Improves logging messages and error handling flow |
| RealtimeSegmentValidationManagerTest.java | Adds unit tests for both pauseless enabled/disabled scenarios and reformats existing test data |
...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
Outdated
Show resolved
Hide resolved
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
2 times, most recently
from
9479c6c to
4421d16
Compare
somandal
approved these changes
Sep 23, 2025
| if (aliveServer == null) { | ||
| LOGGER.warn("No alive server found to re-ingest segment: {} in table: {}, skipping re-ingestion", segmentName, | ||
| realtimeTableName); | ||
| LOGGER.warn("No alive server found to re-ingest segment: {} in table: {}", segmentName, realtimeTableName); |
There was a problem hiding this comment.
nit: was there some reason to remove ", skipping re-ingestion"? seems to be easier to understand the log if this is included, right?
|
I have fixed some bugs in method: Will request to wait before merging this PR. |
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
2 times, most recently
from
f255a74 to
2f22da2
Compare
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
2 times, most recently
from
d8afa42 to
2a1987b
Compare
…ant reingest method - Reuse repairSegmentsInErrorStateForPauselessConsumption() to repair errored segments for both pauseless and non-pauseless tables - Remove reingestCommittingSegmentsForPauselessDisabled() and route call sites to the shared repair flow - Keep a single reingestion entrypoint: reingestSegment(table, segment, instances) with simple predicates (hasOnlineInstance, maybeResetIfNotInProgress) - Update RealtimeSegmentValidationManager to always invoke the shared repair flow (and preserve optional auto-reset behavior) - Adjust RealtimeSegmentValidationManagerTest expectations accordingly Behavioral notes: - Only re-ingests COMMITTING LLC segments with start/end offsets and no download URL when all replicas are in ERROR and the segment is ONLINE in IdealState - Otherwise resets segments not in IN_PROGRESS to fetch from deep store/peer - Applies only to tables without dedup or partial upsert unless explicitly allowed via allowRepairOfErrorSegments()
xiangfu0
force-pushed
the
re-ingest-when-segments-in-commintting-mode
branch
from
2a1987b to
cec9993
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Controller: re-ingest COMMITTING LLC segments when pauseless is disabled.
Add PinotLLCRealtimeSegmentManager#reingestCommittingSegmentsForPauselessDisabled() and wire from RealtimeSegmentValidationManager when pauseless=false
Target only COMMITTING LLC segments with start/end offsets and no download URL that are ONLINE in IS
Respect partial upsert/dedup safety via allowRepairOfErrorSegments
Add unit tests in RealtimeSegmentValidationManagerTest
Build: checkstyle clean, targeted unit test passing.