Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test shared mod file #14048

Draft
wants to merge 77 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
7dc39f1
temp save
jt2594838 Oct 12, 2024
469f3b6
temp save
jt2594838 Oct 17, 2024
f115dc6
Merge branch 'master' into table_data_deletion
jt2594838 Oct 17, 2024
480b4b1
Change modification format
jt2594838 Oct 23, 2024
6104265
Merge branch 'master' into table_data_deletion
jt2594838 Oct 23, 2024
45e33a8
update tsfile version
jt2594838 Oct 23, 2024
87baed0
fix deviceId match
jt2594838 Oct 23, 2024
2ec1aa4
Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…
jt2594838 Oct 23, 2024
9bb80e3
fix identitySinOperatorTest
jt2594838 Oct 23, 2024
1dacc0b
refactor interface hierachy
jt2594838 Oct 23, 2024
acb1a47
refactor package structure
jt2594838 Oct 23, 2024
088009b
spotless
jt2594838 Oct 23, 2024
05ba9c8
support table deletion
jt2594838 Oct 25, 2024
ba59a3f
Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…
jt2594838 Oct 25, 2024
2ffd6b4
Merge branch 'master' into table_data_deletion
jt2594838 Oct 25, 2024
cb83826
fix test
jt2594838 Oct 25, 2024
2dc417f
remove v1 mod file
jt2594838 Oct 25, 2024
1a234ba
fix read empty mod
jt2594838 Oct 25, 2024
ffc3eba
add table deletion IT
jt2594838 Oct 30, 2024
655c84f
Merge branch 'master' into table_data_deletion
jt2594838 Oct 30, 2024
ba01a6f
Fix nullability check in buildTsBlock().
jt2594838 Oct 31, 2024
bec444a
fix partialPath type in TreeDeletionEntry
jt2594838 Nov 1, 2024
d64b0a0
add predicate ut
jt2594838 Nov 4, 2024
0dd0c71
allow multiple mods in a plan node
jt2594838 Nov 4, 2024
2798a66
Merge branch 'master' into table_data_deletion
jt2594838 Nov 4, 2024
bcf8ccb
implment deleteDataForDropTable & fix IT
jt2594838 Nov 5, 2024
120acb3
fix log level
jt2594838 Nov 5, 2024
d0bfa97
Merge branch 'master' into table_data_deletion
jt2594838 Nov 5, 2024
7d5c3a6
ignore one test
jt2594838 Nov 5, 2024
9b511dc
Added table IT (#13978)
Caideyipi Nov 5, 2024
4319dd7
fix RelationalDeleteDataPlan serialization
jt2594838 Nov 5, 2024
e42f159
Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…
jt2594838 Nov 5, 2024
80197cd
Merge branch 'master' into table_data_deletion
jt2594838 Nov 5, 2024
f8b1bd7
spotless
jt2594838 Nov 5, 2024
5c2230c
add license
jt2594838 Nov 5, 2024
2e1f6b1
Support more compicated deletion predicate
jt2594838 Nov 5, 2024
7d660b2
fix tests
jt2594838 Nov 5, 2024
64d6d39
update tsfile version & update tests
jt2594838 Nov 6, 2024
3f4e130
fix modEntry merge
jt2594838 Nov 6, 2024
592e595
parallel file deletion & fix ut
jt2594838 Nov 6, 2024
c511b33
add performance test
jt2594838 Nov 7, 2024
2a86245
all shared mod file framework
jt2594838 Nov 7, 2024
67359fa
fix TsFileResourceSerialization
jt2594838 Nov 7, 2024
d97bc8b
add mod file manager
jt2594838 Nov 7, 2024
f592321
fix TsFileResource deserialization
jt2594838 Nov 8, 2024
6ae9faf
fix exception handle
jt2594838 Nov 8, 2024
946b536
spotless
jt2594838 Nov 8, 2024
9ebc4b7
fulfill deletion framework
jt2594838 Nov 8, 2024
884ec96
ignore perf test
jt2594838 Nov 8, 2024
00a44b5
use buffered stream to read mods
jt2594838 Nov 8, 2024
09ba21b
fix ut
jt2594838 Nov 8, 2024
002d385
spotless
jt2594838 Nov 8, 2024
5faf169
Merge branch 'master' into table_data_deletion
jt2594838 Nov 11, 2024
76a5b3f
fix metric ut
jt2594838 Nov 11, 2024
55dd6c3
Merge branch 'table_data_deletion' into share_mod_file
jt2594838 Nov 11, 2024
182e069
spotless
jt2594838 Nov 11, 2024
aaeb322
add mod file exists marker and log condition
jt2594838 Nov 11, 2024
4645ddc
fix tests
jt2594838 Nov 12, 2024
687c606
fix tests
jt2594838 Nov 12, 2024
40b63a5
Merge branch 'table_data_deletion' into share_mod_file
jt2594838 Nov 12, 2024
6f185de
fix test
jt2594838 Nov 12, 2024
541fd73
fix recover deadlock
jt2594838 Nov 12, 2024
cc4df81
Merge branch 'table_data_deletion' into share_mod_file
jt2594838 Nov 12, 2024
41e821c
Drop column adaptation (#14073)
Caideyipi Nov 13, 2024
a87f0e7
adaptation (#14077)
Caideyipi Nov 13, 2024
17f0a7f
fix comments
jt2594838 Nov 13, 2024
d0e6e34
Merge branch 'table_data_deletion' of github.com:apache/iotdb into ta…
jt2594838 Nov 13, 2024
1fe88ff
Merge branch 'master' into table_data_deletion
jt2594838 Nov 13, 2024
67c986b
Fixed the adaptation of delete device (#14081)
Caideyipi Nov 14, 2024
bf2587f
multiple fixes
jt2594838 Nov 14, 2024
a31ff27
Merge branch 'master' into table_data_deletion
jt2594838 Nov 14, 2024
c35051d
Merge branch 'table_data_deletion' into share_mod_file
jt2594838 Nov 14, 2024
ede6028
Merge branch 'master' into share_mod_file
jt2594838 Nov 14, 2024
fcf3f8b
update maxTime in TVList after deletion
jt2594838 Nov 14, 2024
6d01f4b
Merge branch 'table_data_deletion' into share_mod_file
jt2594838 Nov 14, 2024
741d889
fix tests
jt2594838 Nov 14, 2024
ba7ab7c
spotless
jt2594838 Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@

import org.apache.iotdb.commons.exception.IllegalPathException;
import org.apache.iotdb.commons.path.MeasurementPath;
import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
import org.apache.iotdb.db.storageengine.dataregion.modification.ModEntry;
import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
import org.apache.iotdb.db.storageengine.dataregion.modification.TreeDeletionEntry;

import org.apache.tsfile.common.conf.TSFileConfig;
import org.apache.tsfile.exception.write.WriteProcessException;
Expand Down Expand Up @@ -250,23 +251,21 @@ private void generateTEXT(final Object obj, final int row) {
public void generateDeletion(final String device, final int number)
throws IOException, IllegalPathException {
try (final ModificationFile modificationFile =
new ModificationFile(tsFile.getAbsolutePath() + ModificationFile.FILE_SUFFIX)) {
new ModificationFile(ModificationFile.getExclusiveMods(tsFile))) {
writer.flushAllChunkGroups();
final TreeSet<Long> timeSet = device2TimeSet.get(device);
if (timeSet.isEmpty()) {
return;
}

final long fileOffset = tsFile.length();
final long maxTime = timeSet.last() - 1;
for (int i = 0; i < number; i++) {
final int endTime = random.nextInt((int) (maxTime)) + 1;
final int startTime = random.nextInt(endTime);
for (final IMeasurementSchema measurementSchema : device2MeasurementSchema.get(device)) {
final Deletion deletion =
new Deletion(
final ModEntry deletion =
new TreeDeletionEntry(
new MeasurementPath(device, measurementSchema.getMeasurementId()),
fileOffset,
startTime,
endTime);
modificationFile.write(deletion);
Expand Down
Loading
Loading