Skip to content

Stream temp files when saving workbook#2343

Open
1436761858 wants to merge 1 commit into
qax-os:masterfrom
1436761858:saveas-stream-temp-files
Open

Stream temp files when saving workbook#2343
1436761858 wants to merge 1 commit into
qax-os:masterfrom
1436761858:saveas-stream-temp-files

Conversation

@1436761858

Copy link
Copy Markdown

Related Issue

#2342

Description

This change updates the save path for temp-file-backed workbook parts.

Currently, when writeToZip writes entries from f.tempFiles, it calls f.readBytes(path). If the entry is backed by a temp file, readBytes opens the temp file, reads the entire content into memory with io.ReadAll, stores it back into f.Pkg, and then writes the resulting byte slice to the ZIP writer.

This change avoids materializing the whole temp file in heap. Instead, writeToZip opens the temp file with f.readTemp(path) and streams it directly to the ZIP writer with io.Copy.

The saved workbook content is intended to be functionally equivalent:

Before:
temp file -> []byte in memory -> ZIP writer

After:
temp file -> ZIP writer

The same temp-file-backed entry is written to the same ZIP path with the same bytes. The change only affects how the bytes are transported during save, not the generated workbook content or workbook structure.
ZIP64 handling is preserved by using the byte count returned by io.Copy, which is equivalent to the previous Write byte count for this purpose.

A new test, TestWriteToZipStreamsTempFiles, was added to cover the temp-file-backed save path.
The test verifies that:
A temp-file-backed entry is still written into the generated ZIP output.
The written ZIP entry content matches the original temp file content.
The temp-file-backed content is not loaded back into f.Pkg during save.
The third assertion is important because readBytes(path) stores temp file content into f.Pkg after reading it. Verifying that f.Pkg does not contain the entry after save helps ensure this path remains streaming and does not regress back to full in-memory loading.

@xuri xuri added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 24, 2026
@xuri

xuri commented Jun 30, 2026

Copy link
Copy Markdown
Member

Thanks for your PR. Any benchmark data on the performance impact of the changes? Specifically, how much memory is saved, and any speed impact? I suggest test with excelize-benchmark for performance related changes.

@xuri

xuri commented Jun 30, 2026

Copy link
Copy Markdown
Member

The following benchmark report compares memory usage before and after the changes in this PR. The benchmarks were conducted by generating a workbook generated by xuri/excelize-benchmark containing 512K cells.

Environment:
Windows 11 Enterprise, 13th Gen Intel® Core™ i9-13900 (2.00 GHz), 64GB DDR5 RAM, 500GB SSD, Go version go1.25.5 windows/amd64

Workbook generated using the following command:

./v2.11.0-master -func=StreamWriter -rows=102400 -cols=50 -n=6

Base version 09f0656

Func: StreamWriter_r102400xc50.xlsx  RSS = 182 MB  Alloc = 134 MB  TotalAlloc = 1470 MB  Sys = 189 MB  NumGC = 71  Cost = 3.1179301s

This PR: #2343

Func: StreamWriter_r102400xc50.xlsx  RSS = 180 MB  Alloc = 134 MB  TotalAlloc = 1470 MB  Sys = 181 MB  NumGC = 71  Cost = 3.0502737s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants