Skip to content

Issue 5873 - Duplicate data files in very large compaction system test #5874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Oct 24, 2025
Merged

Issue 5873 - Duplicate data files in very large compaction system test #5874

merged 18 commits into from
Oct 24, 2025

Conversation

@patchwork01
Copy link
Collaborator

@patchwork01 patchwork01 commented Oct 20, 2025 •
edited
Loading

Make sure you have checked all steps below.

Issue

  • My PR fully resolves the following issues. I've referenced an issue in the PR title, for example "Issue 1234 - My
    Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
    for your progress.

Tests

  • My PR adds the following tests based on our test strategy OR does not need testing for this extremely good reason:
    • DataFileDuplicationTest
    • DataFileDuplicationIT
    • SystemTestIngestToStateStoreTest
    • Updated CompactionVeryLargeST
    • Ran CompactionVeryLargeST in AWS

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
    separate issue for that below.
  • If I have added new Java code, I have added Javadoc that explains it following our conventions and style.
  • If I have added or removed any dependencies from the project, I have updated the NOTICES file.

@patchwork01 patchwork01 added the pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking label Oct 21, 2025
Base automatically changed from 5871-configure-rust-logging to develop October 21, 2025 07:29
@patchwork01 patchwork01 added pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base) and removed pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking labels Oct 21, 2025
@patchwork01 patchwork01 marked this pull request as ready for review October 21, 2025 14:18
@patchwork01 patchwork01 added the needs-reviewer Pull requests that need a reviewer to be assigned label Oct 21, 2025
@patchwork01 patchwork01 marked this pull request as draft October 21, 2025 15:19
@patchwork01 patchwork01 removed the needs-reviewer Pull requests that need a reviewer to be assigned label Oct 21, 2025
@patchwork01
5873 In memory implementation of data file duplication
578b379
@patchwork01
5873 Duplicate files in AWS
21a673b
@patchwork01
5873 Move DataFileDuplication to util package
1750033
@patchwork01
5873 Refactor DataFileDuplicationTest
fbd9281
@patchwork01
5873 Add DataFileSetDuplication
578355b
@patchwork01
5873 Add DataFileDuplications
744a4c5
@patchwork01
5873 Refactoring
a1f17ae
@patchwork01
5873 Fix duplicating file multiple times
ff1aa29
@patchwork01
5873 Set up test to compact from duplicates
ef7c27b
@patchwork01
5873 Add CompactionDriver.createJobBatches
4aec348
@patchwork01
5873 Create compactions from duplicates
8f0eac2
@patchwork01
5873 Create separate compactions for original and duplicates
231aeb8
@patchwork01
5873 Gather CompletableFutures to list before joining
3be100a
@patchwork01
5873 Use CompletableFuture.allOf
f4b0696
@patchwork01
5873 Remove SystemTestClients.skipAssumeRole
9934f8d
@patchwork01
5873 Merge DataFileDuplicationIT into AwsDataFilesDriverIT
d5cf2a6
@patchwork01
5873 Simplify AwsDataFilesDriverIT
e3c973b
@patchwork01
5873 Fix AwsDataFilesDriverIT
504893c
@patchwork01 patchwork01 force-pushed the 5873-duplicate-data-files branch from 6376473 to 504893c Compare October 22, 2025 10:20
@patchwork01 patchwork01 changed the base branch from develop to 5883-refactor-create-compactions October 22, 2025 10:21
@patchwork01 patchwork01 added pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking and removed pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base) labels Oct 22, 2025
@patchwork01 patchwork01 marked this pull request as ready for review October 22, 2025 10:35
@patchwork01 patchwork01 added the needs-reviewer Pull requests that need a reviewer to be assigned label Oct 22, 2025
Base automatically changed from 5883-refactor-create-compactions to develop October 23, 2025 07:36
@patchwork01 patchwork01 added pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base) and removed pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking labels Oct 23, 2025
@rtjd6554 rtjd6554 self-assigned this Oct 23, 2025
@rtjd6554 rtjd6554 removed the needs-reviewer Pull requests that need a reviewer to be assigned label Oct 23, 2025
@rtjd6554 rtjd6554 assigned patchwork01 and unassigned rtjd6554 Oct 24, 2025
@patchwork01 patchwork01 merged commit 6fbbef2 into develop Oct 24, 2025
8 checks passed
@patchwork01 patchwork01 deleted the 5873-duplicate-data-files branch October 24, 2025 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rtjd6554 rtjd6554 rtjd6554 approved these changes

Assignees

@patchwork01 patchwork01

Labels

pr-base-for-stacking Base for stacked pull requests (a dependency for others, where this PR's branch will be the base)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Duplicate data files in system tests

2 participants

@patchwork01 @rtjd6554