Skip to content

feat(rust/sedona): Auto-configure spilled batch in-memory size threshold based on global memory limit#680

Draft
Kontinuation wants to merge 1 commit intoapache:mainfrom
Kontinuation:auto-configure-spill-batch-size-threshold
Draft

feat(rust/sedona): Auto-configure spilled batch in-memory size threshold based on global memory limit#680
Kontinuation wants to merge 1 commit intoapache:mainfrom
Kontinuation:auto-configure-spill-batch-size-threshold

Conversation

@Kontinuation
Copy link
Member

Summary

  • Automatically configure the spilled_batch_in_memory_size_threshold based on the global memory limit, setting it to 5% of the per-partition memory limit (with a minimum of 10MB).
  • This is a by-product of writing documentation for configuring memory management and spilling (docs: add memory management and spill configuration guide #679). We decided to configure this option automatically instead of requiring users to set it up manually based on the global memory limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant