Add index operation listener to update translog source #2629
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmazanec15
force-pushed
the
derived-translog-fix
branch
2 times, most recently
from
a7e5c58 to
45f159c
Compare
jmazanec15
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
naveentatikonda,
junqiu-lei,
martin-gaievski,
ryanbogan,
luyuncheng,
shatejas,
0ctopus13prime and
Vikasht34
as code owners
jmazanec15
force-pushed
the
derived-translog-fix
branch
from
45f159c to
1cac747
Compare
jmazanec15
added
Bug Fixes
jmazanec15
force-pushed
the
derived-translog-fix
branch
from
1cac747 to
50e57c4
Compare
...va/org/opensearch/knn/index/codec/KNN10010Codec/KNN10010DerivedSourceStoredFieldsWriter.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
jmazanec15
force-pushed
the
derived-translog-fix
branch
2 times, most recently
from
6915d4c to
da2376d
Compare
|
Also wondering, if the performance is concerning, do we need to quickly run a small benchmark to have tentative numbers on the impact? |
jmazanec15
force-pushed
the
derived-translog-fix
branch
2 times, most recently
from
f3a0284 to
0401483
Compare
jmazanec15
commented
Mar 26, 2025
| @@ -48,7 +48,7 @@ public KNN10010DerivedSourceStoredFieldsWriter(StoredFieldsWriter delegate, List | |||
| if (vectorFieldTypes.isEmpty() == false) { | |||
| this.vectorMask = XContentMapValues.transform( | |||
| vectorFieldTypes.stream().collect(Collectors.toMap(k -> k, k -> (Object o) -> o == null ? o : MASK)), | |||
| false | |||
| true | |||
There was a problem hiding this comment.
Performing in place map modification - we create and own the lifecycle of the map so we can do this.
...va/org/opensearch/knn/index/codec/KNN10010Codec/KNN10010DerivedSourceStoredFieldsWriter.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
...n/java/org/opensearch/knn/index/codec/derivedsource/DerivedSourceIndexOperationListener.java
Show resolved
Hide resolved
jmazanec15
force-pushed
the
derived-translog-fix
branch
2 times, most recently
from
93b4066 to
f7d0d4d
Compare
Yes, Im going to run some benchmarks after I get it working functionally. |
jmazanec15
force-pushed
the
derived-translog-fix
branch
4 times, most recently
from
23ec8f4 to
4e85167
Compare
Adds an index operation listener to update the source to match the source getting stored to the stored fields. This ensures that on certain recovery events, we can be sure that duplicate operations do not cause conflicts. Along with this, updated integration tests to be more robust. Signed-off-by: John Mazanec <[email protected]>
jmazanec15
force-pushed
the
derived-translog-fix
branch
from
4e85167 to
1027836
Compare
0ctopus13prime
approved these changes
Mar 27, 2025
shatejas
approved these changes
Mar 27, 2025
Vikasht34
approved these changes
Mar 27, 2025
navneet1v
approved these changes
Mar 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds an index operation listener to update the source to match the source getting stored to the stored fields. This ensures that on certain recovery events, we can be sure that duplicate operations do not cause conflicts.
Along with this, updated integration tests to be more robust. See DerivedSourceTestCase for cases covered so far. Logic is shared with the bwc test.
Also, as part of #2626, I made source changes in place.
In future PR, I will add more optimizations and also add logic for blocking in certain cases.
Related Issues
Resolves #2625
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.