ENH: deterministic ordering for better Parquet compression

[fedorov]

Summary

• Add ORDER BY to all ARRAY_AGG/STRING_AGG calls that lacked deterministic ordering (21 aggregations across 7 files), ensuring identical logical arrays are always encoded the same way across query runs
• Add compression-friendly final ORDER BY to all queries (9 files), clustering rows by semantically meaningful columns (anatomy, staining, collection, algorithm type) rather than arbitrary UIDs
• Replace STRING_AGG(DISTINCT collection_id) with ANY_VALUE(collection_id) in sm_index.sql since a series always belongs to one collection
• Document that independently-aggregated DISTINCT columns in seg_index.sql do not preserve positional correspondence

These changes improve Parquet dictionary encoding and run-length encoding without removing any content from query results.

Files changed

File	Changes
scripts/sql/idc_index.sql	Add ORDER BY collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID
scripts/sql/prior_versions_index.sql	Add ORDER BY version to STRING_AGG + final ORDER BY collection_id, min_idc_version, Modality
scripts/sql/collections_index.sql	Add ORDER BY collection_id
scripts/sql/analysis_results_index.sql	Add ORDER BY analysis_result_id
assets/sm_index.sql	Fix 7 aggregations + ORDER BY primaryAnatomicStructure, staining, collection_id
assets/sm_instance_index.sql	Fix 3 aggregations + reorder to staining, TransferSyntaxUID, SeriesInstanceUID
assets/seg_index.sql	Fix 5 aggregations + expand to ORDER BY SegmentationType, AlgorithmType, AlgorithmName
assets/contrast_index.sql	Fix 3 aggregations + ORDER BY ContrastBolusAgent, Ingredient, Route
assets/ann_index.sql	Add ORDER BY AnnotationCoordinateType, referenced_SeriesInstanceUID
assets/ann_group_index.sql	Add ORDER BY category, type, algorithm
scripts/gdc/idc_gdc_selection.sql	Fix 1 ARRAY_AGG

Test plan

• Verify queries execute without errors against BigQuery
• Compare row counts before/after to confirm no data loss
• Compare Parquet file sizes before/after to measure compression improvement

🤖 Generated with Claude Code

[fedorov]

original	updated