Skip to content

db: incorporate blob files into EstimateDiskUsage #4621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jbowens opened this issue Apr 24, 2025 · 0 comments · Fixed by #4703
Closed

db: incorporate blob files into EstimateDiskUsage #4621

jbowens opened this issue Apr 24, 2025 · 0 comments · Fixed by #4703
Assignees
@annrpom
Labels
A-storage T-storage

Comments

@jbowens
Copy link
Collaborator

jbowens commented Apr 24, 2025 •
edited by exalate-issue-sync bot
Loading

When estimating the disk usage between two bounds, we should incorporate contribution from blob files referenced from overlapping sstables. When there's partial overlap with a sstable, we don't have a means of accurately determining how much of a blob reference comes from the span passed into EstimateDiskUsage specifically. I suppose we could crudely estimate it using linear interpolation.

We use (*sstable.Reader).EstimateDiskUsage(start, end) to get an accurate estimate of the disk usage within the sstable itself. We divide that by the TableMetadata.Size to get an estimate of the fraction of the table's blob references that fall within the range. Then for each blob reference, we scale the uncompressd ValueLen by that fraction to get estimates of the uncompressed value quantities. Then we scale each of those by the compression ratio of the blob file and sum them all together.

Jira issue: PEBBLE-424

Epic CRDB-20379
@annrpom annrpom self-assigned this May 5, 2025
annrpom added a commit to annrpom/pebble that referenced this issue May 7, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
0dce6de 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
annrpom added a commit to annrpom/pebble that referenced this issue May 8, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
604f874 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
@annrpom annrpom mentioned this issue May 8, 2025
Merged
annrpom added a commit to annrpom/pebble that referenced this issue May 8, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
2927d79 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
annrpom added a commit to annrpom/pebble that referenced this issue May 8, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
7263776 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
annrpom added a commit to annrpom/pebble that referenced this issue May 9, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
81bd754 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
annrpom added a commit to annrpom/pebble that referenced this issue May 13, 2025
@annrpom
db: incorporate blob files into EstimateDiskUsage
35e91df 
This patch ensures that blob files are also included
in disk usage estimation.

Fixes: cockroachdb#4621
@jbowens jbowens mentioned this issue May 14, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@annrpom annrpom

Labels
A-storage T-storage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

db: incorporate blob files into EstimateDiskUsage annrpom/pebble
2 participants
@jbowens @annrpom