Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] extensive enhance for fullsync #43576

Open
2 of 3 tasks
dataroaring opened this issue Nov 11, 2024 · 1 comment
Open
2 of 3 tasks

[Enhancement] extensive enhance for fullsync #43576

dataroaring opened this issue Nov 11, 2024 · 1 comment
Labels

Comments

@dataroaring
Copy link
Contributor

dataroaring commented Nov 11, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

Description

Fullsync means sync all data via backup and restore again. It happens when ccr can not handle binlog incrementally, then it starts from beginning again. Internally, fullsync using bakcup with alias and atomic restore to reuse data as mush as possible , thus avoid sync all data again.

So we should let fullsync happen frequently to test it thoughroughly.

Solution

  1. add cases for some operations listed by experience as below, test_ds_tbl_truncate is an example.
Resource Operation Db sync case Table sync case Table sync with alias case Comment
Database          
Table Create + Drop   - -  
Create + Alter   - -    
Drop + Create   - - Schemas are same  
Drop + Create   - - Schemas are different  
Drop   - -    
Truncate test_ds_tbl_truncate        
Rename   - -    
Replace          
Alter properties          
Alter comment          
Partition Add        
  Drop        
  Replace        
  Modify        
  Rename        
  Add + Drop        
  Drop + Add        
  Add + rename        
Column Add key        
Drop key          
Add value          
Drop value          
Order by          
Rename          
Modify comment          
Add key + drop          
Drop key + add          
Add value + drop          
Drop value + add          
Add column value + modify          
Add key column + modify          
Rollup Add        
Drop          
Rename   - -    
Add column          
Drop column          
Modify column          
Rename column          
View Create        
Alter          
Drop          
Drop + Create          
Create + Drop          
Index Add inverted index        
Drop inverted index          
Add bloom filter          
Drop bloom filter          
Add NG bloom filter          
Drop NG bloom filter          
MV Create        
Drop          
DML INSERT INTO OVERWRITE        
explicit transaction          
  1. writing a test tool which generate random operation and ccr syncer take full sync in action randomly.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@dataroaring
Copy link
Contributor Author

This is a huge issue, please create a new issue for specific task you pick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant