fix(offline-diarizer): pyannote-parity clustering — threshold semantics, constraint count, constrained assignment#802
Conversation
…cs, constraint count, constrained assignment (#801) Three porting bugs against pyannote community-1, all surfaced by #801 where auto clustering and numSpeakers=2 returned the same speaker count (2) but materially different partitions (69% vs 92% correctly-attributed speech): 1. AHC threshold: the config value was documented as a Euclidean cut distance (pyannote applies it directly via fcluster(..., "distance")), but the implementation reinterpreted it as a cosine similarity and cut at sqrt(2 - 2t). This inverted the knob's direction (raising the value split more instead of merging more) and shifted the default 0.6 to an effective cut of 0.894. The value is now applied directly, clamped to [0, 2]; config validation and CLI help updated to match. Old configs map to new ones via sqrt(2 - 2*old): old 0.6 ≈ new 0.894, old 0.7 ≈ new 0.775. 2. Speaker-count constraints compared against the AHC warm-start cluster count instead of the number of clusters VBx actually kept (pi > 1e-7, pyannote's auto_num_clusters). Since the user-visible detected count is the post-collapse survivor count, numSpeakers=N triggered K-Means re-clustering even when VBx had already collapsed to exactly N, producing a different partition at the same count. The check now uses VBxOutput.activeClusterCount. 3. Embedding-to-centroid assignment was a plain per-embedding argmax, which lets two local speakers sharing a segmentation chunk snap to the same centroid — silently absorbing one speaker's turns into another's (the mechanism behind the bad 82s-boundary partition in #801). Ported pyannote's constrained_argmax: per chunk, local speakers are matched to distinct clusters via Hungarian assignment maximizing total similarity. Controlled by Clustering.constrainedAssignment (default true) and automatically disabled when the count is forced via K-Means, matching pyannote. The Hungarian solver moved from DiarizationDER.swift into an internal HungarianAssignment enum with a rectangular max-score wrapper. Docs: Benchmarks.md threshold guidance rewritten for the new semantics (the old --threshold 0.7 advice corresponds to 0.775 today); min/max/ numSpeakers doc comments now state that the floor alone cannot correct an under-partition and that numSpeakers is a target, not a guarantee.
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m11s • 07/16/2026, 05:22 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
PocketTTS Smoke Test ✅
Runtime: 0m25s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 7m8s • 07/16/2026, 05:27 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 159.1s processing • Test runtime: 2m 42s • 07/16/2026, 05:28 PM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 38.9s diarization time • Test runtime: 2m 42s • 07/16/2026, 05:29 PM EST |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 3m 14s • 2026-07-16T21:31:53.534Z |
Supertonic3 Smoke Test ✅
Runtime: 0m11s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
1 participant
Why is this change needed?
Fixes #801:
OfflineDiarizerManagerwith.defaultconfig and withclustering.numSpeakers = 2returned the same speaker count (2) but materially different partitions (69% vs 92% correctly-attributed speech on the reporter's 2-speaker clip). Every observation in the reporter's 20-config sweep decodes exactly to three porting bugs against pyannote community-1'sVBxClustering:1. AHC threshold was inverted (issue Q2 — confirmed). The config value is documented as a "Euclidean distance threshold for unit-normalized embeddings", and pyannote applies it directly as the dendrogram cut:
fcluster(dendrogram, threshold, criterion="distance"). Our port reinterpreted it as a cosine similarity and cut atsqrt(2 − 2t), which inverts the knob (raising it split more instead of merging more) and shifted the default 0.6 to an effective cut of 0.894. The reporter's sweep maps perfectly through this formula: 0.3/0.4/0.45 → cuts 1.18/1.10/1.05 (merged to 1 speaker), while 0.6–1.1 → cuts 0.894–0.0 (byte-identical near-singleton AHC output). The value is now applied directly, clamped/validated to (0, 2].2. Speaker-count constraints compared against the wrong count (issue Q1 — it's a bug). pyannote decides whether to fall back to K-Means by comparing the requested count against
auto_num_clusters— the clusters VBx actually kept (pi > 1e-7). We compared against the AHC warm-start count, which on this audio was far above 2 (AHC produced near-singletons, VBx collapsed them to 2). SonumSpeakers=2always triggered K-Means re-clustering — a completely different algorithm — even though auto had already detected 2. Hence same N, different partition. The check now uses the newVBxOutput.activeClusterCount. This also explains the reporter's side observations:minSpeakers=2never bound (compared against the inflated count), andnumSpeakers=2 + threshold=0.5"cancelling the win" (cut 1.0 made AHC itself return 2, so the constraint stopped binding).3. Missing constrained assignment (the actual quality gap). Fixing (2) alone would regress the reporter's clip to the bad 69% partition — the K-Means path was accidentally winning. pyannote assigns embeddings with
constrained_argmax: local speakers sharing a segmentation chunk must map to distinct clusters (Hungarian matching per chunk, maximizing total similarity). Our plain per-embedding argmax let two co-chunk speakers snap to the same centroid, silently absorbing one speaker's turns into another's — the mechanism behind the bad 82s-boundary partition. Ported it behindClustering.constrainedAssignment(default true), auto-disabled when the count is forced via K-Means, matching pyannote. The Hungarian solver moved fromDiarizationDER.swiftinto an internalHungarianAssignmentenum with a rectangular max-score wrapper.Behavior change / migration
The
thresholdknob keeps its numeric default (0.6, the community-1 value) but its meaning changes to pyannote's: larger = more merging = fewer speakers. Old values map to new ones viasqrt(2 − 2·old)— old 0.6 behaved like 0.894, and the old AMI advice of--threshold 0.7corresponds to0.775today.Benchmarks.mdand CLI help are updated; the AMI-SDM offline table should be re-benchmarked on this branch since it was measured under the old semantics.Validation
activeClusterCount, the Hungarian solver, and the constrained assigner (including the exact Offline diarizer: auto clustering and numSpeakers=2 return the same speaker count but different partitions (69% vs 92% correctly-attributed speech) #801 failure mode: two co-chunk embeddings both nearest to the same centroid).swift build+ swift-format clean; XCTest runs in CI (unavailable locally)..defaultconfig is the ideal check; the prediction is the auto path now finds the 62s boundary on its own.config.yamlis gated on HF, so the shippedahc_threshold(assumed 0.6-as-distance, within the code'sUniform(0.5, 0.8)prior) is inferred from source semantics, not read from the config.