diff: align the chunk lists to count added/removed bytes

[ThomasWaldmann]

What

borg diff derived the added/removed byte counts of a modified file from the difference of the two chunk id sets. A set knows nothing about order or multiplicity, so it misses real content changes:

change	before	after
chunks reordered	0 B / 0 B	the moved bytes
a chunk duplicated ([A] -> [A,A,A])	0 B added	2 chunks added
a chunk de-duplicated ([A,A,A] -> [A])	0 B removed	2 chunks removed
a block moved within a file	0 B / 0 B	removed + added once

This PR aligns the two chunk lists as sequences instead, the way a text diff aligns lines (difflib.SequenceMatcher). The chunks that are part of the alignment are the unchanged content; everything else is counted, the chunks of ARCHIVE1 as removed and the ones of ARCHIVE2 as added. Insertions, removals, moves and duplicates are all accounted for now.

Real run, a 9 MB file whose two 3 MB blocks were swapped (default chunker):

before:  modified:      0 B      0 B in/doc.bin
after:   modified:  +5.1 MB  -5.1 MB in/doc.bin

Runtime

SequenceMatcher is quadratic in the worst case, so the new chunks_diff_size() has two guards:

1. The common prefix and suffix are stripped first. This makes the usual cases nearly free (a file that was appended to, a file that was edited in one place) and it also keeps the matcher away from the long runs of identical chunks it is slow on. A 200k-chunk sparse file with one changed chunk in the middle: 9 ms.
2. Above MAX_ALIGN_CHUNKS (65536) or MAX_ALIGN_WORK (2^20) the alignment is skipped and the chunk ids are only counted per id, i.e. approximately what borg did before (but multiplicity-aware). MAX_ALIGN_WORK is the estimated matcher work, sum over list1 of the chunk's number of occurrences in list2; it is what catches the pathological case, chunk lists that repeat one id very often (sparse files, VM images with big all-zero ranges), where 8k chunks already took 5.6 s.

Measured on an M3 Pro, worst cases per file after both guards:

case	time
500k chunks, one appended (prefix strip)	15 ms
200k identical chunks, 1 changed (prefix/suffix strip)	10 ms
200k unique chunks, 1% edits (length cap -> counted)	67 ms
20k chunks, 50% duplicates, shuffled (work cap -> counted)	6 ms
65536 unique chunks fully shuffled (worst aligned case)	150 ms

Without the guards the last two rows would be several seconds each.

Notes

• The change is behind the existing "chunk ids are comparable" branch, so archives created with different --chunker-params are unaffected (they are still compared by content and still report no byte counts).
• test_reordered_chunks (from diff: report a file whose chunks were only reordered or duplicated as modified #10350) is updated: the swap of two 1 KB chunks now reports +1.0 kB / -1.0 kB rather than 0 B / 0 B, which is the point of this PR. New tests: test_duplicated_chunks, test_inserted_chunk, and unit tests for chunks_diff_size() including both fallback paths.
• Item.get_size(consider_ids=...) has no caller any more, it existed only for the old set-based counting. Left in place to keep the diff small - say the word and I will drop it.
• Docs updated: the borg diff epilog and docs/internals/frontends.rst.
• A possible follow-up if the caps ever become a problem: patience diff (align only on chunk ids that are unique in both lists, then LIS) is O(n log n) with no pathological case, at the price of a hand-written matcher.

Full archiver test suite passes locally (1177 passed, 623 skipped).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.68%. Comparing base (05debca) to head (da0c417).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10358      +/-   ##
==========================================
- Coverage   88.02%   85.68%   -2.35%     
==========================================
  Files         103      103              
  Lines       18913    18913              
  Branches     2919     2919              
==========================================
- Hits        16649    16205     -444     
- Misses       1572     2026     +454     
+ Partials      692      682      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.