docs: add memory management and spill configuration guide#679
docs: add memory management and spill configuration guide#679Kontinuation wants to merge 5 commits intoapache:mainfrom
Conversation
Add documentation covering memory limits, memory pool types (greedy vs fair), temporary spill directory configuration, operators that support spilling, advanced DataFusion spill settings, and system-level file descriptor tuning.
Clarify memory pool behavior without a limit, add guidance on fair pool usage, use SQL-centric operator descriptions for spill-capable plans, and document DataFusion spill settings and file descriptor requirements more precisely.
There was a problem hiding this comment.
Pull request overview
Adds end-user documentation for configuring SedonaDB/DataFusion memory limits and spill behavior, and exposes it in the MkDocs navigation so users can discover it alongside existing guides.
Changes:
- Registers a new “Memory Management” page in the MkDocs
nav. - Adds a new documentation page describing memory limits, pool types, spill directory configuration, DataFusion spill settings, and OS file-descriptor tuning.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| mkdocs.yml | Adds the new “Memory Management” page to the SedonaDB docs navigation. |
| docs/memory-management.md | New guide covering memory limits, pool behavior, spill/temp-dir configuration, and operational tuning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
paleolimbot
left a comment
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you!
Can you make this a .ipynb file where the Python code is runnable chunks? This ensures that we can check that the examples actually run in the future. The docs build script will render the .ipynb file automatically although I have been rendering them anyway because the .md files are easier to review.
Summary