Compute linear block-combo search chunk size from block file sizes.

[bbuschkaemper]

After the fixes in #968 #965 we restarted a 48B sequence clustering run. Even though the blocks are now correctly capped at 2^32-1 sequences, we encountered another problem downstream: For the linear block-combo search, the algorithm assumes a maximum chunk size of 1024GB, which causes problems downstream.

Run logfile:
20260626_153616.log

Summary

Fixes a crash in the multinode linear search path by preventing Search::run from re-splitting length-sorted input blocks.

Problem

The multinode workflow first creates length-sorted block files in len_sort() and then runs linear block-combo searches over those exact VolumedFile entries.

However, run_block_combo() previously forced:

config.chunk_size = 1024;

This means that a length-sorted block larger than 1024GB on disk could be split again inside the generic Search::run path:

• query blocks are loaded via load_seqs(config.block_size(), ...)
• reference blocks are loaded via load_seqs(config.block_size(), ...)
• config.block_size() is config.chunk_size * 1e9

For our dataset, one generated input0.faa block was ~1.11 TB on disk. Although length sorting had already produced a representation-safe block, the fixed 1024GB search chunk size caused Search::run to create additional internal query/reference blocks. That broke the multinode block-combo assumption that each (r, i) search operates on the already generated length-sort blocks.

Fix

Instead of hardcoding 1024, run_block_combo() now derives config.chunk_size from the actual block files being compared:

config.chunk_size = block_combo_chunk_size(volumes, r, i);

The helper takes the larger of the database/query block file sizes and rounds it up to whole decimal GB. This keeps the search chunk size large enough that Search::run does not split the length-sorted block again. This change makes the later generic search stage respect those precomputed block boundaries.

Validation

We will rerun the 48B clustering and report back if this fixes it. However, we suspect more possible problems arising downstream, e.g. greedy_vertex_cover() uses an unordered_map<string, OId> without respecting memory limits. This, as well as other iterations/maps/buffers that go over the whole input db, can cause OOM crashes on very-large dbs where whole-db sample mappings are non-trivial.

[bbuschkaemper]

@bbuchfink Will leave this as draft PR until the clustering rerun will hit that point again (approx. in 4 days). If (as I suspect) there will be more downstream scaling issues with very-large dbs I think bundling them as one PR might make sense..

[bbuchfink]

Very well, makes sense. Thanks for staying tuned.

[bbuchfink]

On a side note, your command will not only run linclust but all-vs-all alignment, that should be super expensive on a dataset this size and with a 90% id cutoff, meaning the first round will not reduce by that much. To save time I'd recommend to use e.g. --cluster-steps faster_lin fast_lin instead