41: TN93 preliminary distance heuristic

[glstott]

This is an implementation of GH issue #41 . This solution provides faster initial load times at the cost of longer load times for exact calculations overall. To reduce the cost of the feature we only have it used when there are sufficient data for performance benefits.

Summary

• Adds progressive TN93 distance computation to reduce time-to-first-network for large sequence datasets.
• Uses consensus-based radial distances to identify likely link candidates and renders those first.
• Computes deferred sequence pairs in the background, then updates the network with the exact final matrix.
• Activates progressively only for workloads with at least 5,000 pairs and no more than 80% candidate coverage; otherwise uses the existing exhaustive path.
• Keeps the initial network clearly labeled as provisional because the heuristic may temporarily omit qualifying links.
• Leaves SNP, imported distance data, edge lists, and Newick workflows unchanged.

Worker and lifecycle improvements

• Adds a dedicated TN93 worker with:

  • Packed transferable sequence buffers.
  • Compact computed-pair tracking.
  • Sparse Float32 distance batches.
  • Backpressure acknowledgements.
  • Chunked execution for responsive cancellation and threshold changes.

• Rejects stale results using run IDs, load generations, and immutable input signatures.

• Cancels active work when datasets, sessions, metrics, sequences, or ambiguity settings change.

• Falls back to exhaustive computation if the worker fails before the first network is rendered.

• Retains provisional state and supports retry if background completion fails.

• Prioritizes newly eligible pairs when the link threshold increases and reuses existing results when it decreases.

User experience

• Adds a persistent progress banner showing TN93 computation status and completed-pair progress.
• Warns users that links, clusters, and statistics may change while the network is provisional.
• Defers exact-matrix features—including Heatmap, inferred phylogeny, MST/nearest-neighbor processing, link exports, and session saves—until computation completes.
• Adds revision-based notifications so open views refresh after foreground, threshold-promotion, and final-completion milestones.
• Coalesces Heatmap and phylogeny refreshes to avoid duplicate or stale rendering work.

Correctness and compatibility

• Preserves TN93 ambiguity strategies, including RESOLVE and HIVTRACE-G behavior.
• Excludes synthetic all-gap placeholder sequences from consensus construction.
• Handles ties, raw threshold equality, non-finite radial distances, variable sequence lengths, IUPAC symbols, and gaps.
• Preserves final Float32 link-distance behavior and synchronizes the link cache with the complete distance matrix.
• Recomputes derived clusters, statistics, visibility, MST/nearest-neighbor state, and inferred trees after exact completion.

Test and benchmark coverage

• Adds unit coverage for:

  • Consensus and radial-distance planning.
  • Candidate/complement partitioning and triangular pair indexing.
  • TN93 ambiguity strategies and RESOLVE parity.
  • Packed worker payloads.
  • Batch acknowledgements and background-worker coordination.
  • Threshold promotion, cancellation, stale-result isolation, and failure fallback.

• Adds Cypress coverage for:

  • Provisional UI and Heatmap gating.
  • Deferred qualifying-edge correction at exact completion.
  • Higher- and lower-threshold behavior.
  • Metric-change cancellation and stale-run isolation.

• Adds deterministic performance fixtures covering:

  • Progressive and adaptive-fallback workloads.
  • Dataset scaling from 100 to 1,000 sequences.
  • Candidate density, sequence length, topology, ambiguity, and non-finite distances.
  • A known TN93 consensus-window counterexample.

• Adds five-run p50 performance comparisons, exhaustive-result parity checks, fallback-regression limits, scaling benchmarks, and a sharded worker benchmark command.

No tests were run during this summary-only pass.

[glstott]

Progressive TN93 vs dev: apples-to-apples performance report

Generated: 2026-08-05

Executive summary

• The intended speedup is real at meaningful scale. Compared with dev, the feature reached the first 2D network 5.6% faster at 500 sequences and 31.1% faster at 1,000 sequences.
• The TN93 link-computation stage itself improved more substantially: 60.0% faster at 500 sequences and 71.7% faster at 1,000 sequences.
• The benefit is effectively neutral at the smallest activating fixture. At 180 sequences, the first-view difference was 0.1%, because the legacy TN93 calculation takes only about 83 ms and overall UI work dominates the load.
• Adaptive fallback matched dev: 0.4% faster to the first view and 0.4% faster to exact completion.
• Exact completion is the primary cost. It was 24.8% slower at 500 sequences and 63.6% slower at 1,000 sequences, even though users saw the provisional network sooner.
• Every measured run produced the same final pair count, visible-link count, finite-distance count, and Float32 distance hash as the deterministic fixture oracle.

Test design

• Feature commit: c6bfb561ae131e42778dbcc7b787f07104f614b7
• Baseline: dev at 6a15280bf5f06a884768c7c8ec4f5416763938fe
• Browser: Edge 150, headless, 1280 x 720
• Runtime: Node.js 22.16.0 and Cypress 15.16.0
• Server: isolated Angular development servers, same configuration and port
• Sampling: one excluded warm-up plus five measured runs for each fixture on each branch
• First-view timing: file-upload start through ready 2D network
• Exact-completion timing: the same start point through full pair materialization and, where available, TN93 complete status; readiness is captured before parity-hash validation
• Correctness gates: pair count, visible-link count, finite-distance count, and deterministic Float32 distance hash

Results

Negative change means the feature branch was faster than dev; positive change means it was slower.

Fixture	Candidate pairs	Feature first view	dev first view	First-view change	Feature exact	dev exact	Exact change
Diverse 180	2,978 / 16,110 (18.5%)	7,220 ms	7,225 ms	0.1% faster	7,489 ms	7,227 ms	3.6% slower
Adaptive fallback 180	16,110 / 16,110 (100%)	7,100 ms	7,130 ms	0.4% faster	7,103 ms	7,132 ms	0.4% faster
Scaling 500	23,688 / 124,750 (19.0%)	8,189 ms	8,673 ms	5.6% faster	10,833 ms	8,677 ms	24.8% slower
Scaling 1,000	95,337 / 499,500 (19.1%)	9,464 ms	13,745 ms	31.1% faster	22,505 ms	13,756 ms	63.6% slower

TN93 computation and responsiveness

Fixture	Feature computeLinks p50	dev p50	Change	Feature first-view long-task time	dev long-task time	Change
Diverse 180	50 ms	83 ms	39.8% faster	335 ms	358 ms	6.4% lower
Adaptive fallback 180	96 ms	96 ms	no change	469 ms	550 ms	14.7% lower
Scaling 500	174 ms	435 ms	60.0% faster	2,424 ms	2,551 ms	5.0% lower
Scaling 1,000	565 ms	1,998 ms	71.7% faster	3,831 ms	6,893 ms	44.4% lower

At 500 and 1,000 sequences, the feature both shortened the first-view wall-clock time and reduced total main-thread blocking before that view. At 180 sequences, UI and orchestration work dominate the load, so reducing TN93 pair computation does not translate into a faster visible result.

Costs and tradeoffs

• Longer path to exactness: at 1,000 sequences, the provisional view arrived 4.28 seconds sooner, but the exact matrix completed 8.75 seconds later than dev.
• More result-transfer overhead: the progressive worker streams source index, target index, and distance for each pair. The completed 1,000-sequence run transferred 5.99 MB across 289 batches. The legacy dense result represents the same 499,500 distances as one Float32 value per pair, approximately 2.00 MB, although worker-input transfer differs between implementations.
• More merge stages: foreground, background, threshold promotion, and final refresh improve responsiveness and cancellation but add acknowledgements, cache invalidations, merges, and final view-refresh work.
• Provisional correctness window: the first view can temporarily omit a qualifying edge. Final parity passed, and the dedicated counterexample coverage verifies that deferred qualifying links are restored at exact completion.
• Memory evidence is inconclusive: browser heap deltas varied widely and included negative values after garbage collection. The current five-run data is not reliable enough to claim either a memory improvement or regression.
• Asymptotic cost is unchanged: every pair is still computed and stored, so total CPU and memory remain O(n^2).

Correctness validation

• All four fixtures passed on both branches.
• All runs ended with the expected complete pair counts:
  • 16,110 pairs at 180 sequences
  • 124,750 pairs at 500 sequences
  • 499,500 pairs at 1,000 sequences
• Final visible-link counts matched the deterministic fixture manifests.
• Final finite-distance counts and Float32 distance hashes matched exactly on every run.
• Progressive runs rendered only the planned candidate set initially and reached the full matrix before final validation.