tests on range partitioned data

[jayshrivastava]

See

datafusion-distributed/tests/join.rs

Line 120 in a6cff42

│ t1: DataSourceExec: file_groups={4 groups: [[/testdata/join/parquet/dim/d_dkey=B/data0.parquet], [/testdata/join/parquet/dim/d_dkey=D/data0.parquet], [], []]}, projection=[env, service, host, d_dkey], output_partitioning=Hash([d_dkey@3], 4), file_type=parquet, predicate=service@1 = log, pruning_predicate=service_null_count@2 != row_count@3 AND service_min@0 <= log AND log <= service_max@1, required_guarantees=[service in (log)]

It's not valid in datafusion to claim your data source is distributed by hash(expr), so this test isn't useful. We should migrate it to a distribution which maps to range partitioning.

There's several open questions that can only really be solved with tests

• file_scan_config_scale_up_leaf_node in src/events/defaults/file_scan_config.rs if the file scan is claiming range partitioning and we rearrange files, that can cause correctness issues right? Should we call FileScanConfig::repartitioned
• Say target_partitions is 10 and our data source has 100 range partitions. Does this distribute correctly to 10 leaf tasks? Does datafusion introduce range partitions?
• Are all instances of scale_partitioning correct when it comes to range partitioned data?

  datafusion-distributed/src/execution_plans/common.rs

  Line 17 in a6cff42

  pub(super) fn scale_partitioning(

[shinzoxD]

Taking this. I'll migrate the join test off the invalid hash(expr) data-source distribution and add coverage for range-partitioned sources (file-scan scale-up, target_partitions vs range partition count, and scale_partitioning).

[jayshrivastava]

@shinzoxD thanks! You may want to base on top of #540. I'll upgrade and merge that once the release is finalized: apache/datafusion#22393