Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[comet-parquet-exec] update some stability plans #1083

Merged
merged 1 commit into from
Nov 14, 2024
+420 −420
Merged

[comet-parquet-exec] update some stability plans #1083

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/explain.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,30 +18,30 @@
: : : : : +- CometExchange (4)
: : : : : +- CometProject (3)
: : : : : +- CometFilter (2)
: : : : : +- CometScan parquet spark_catalog.default.catalog_sales (1)
: : : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales (1)
: : : : +- CometSort (9)
: : : : +- CometExchange (8)
: : : : +- CometProject (7)
: : : : +- CometScan parquet spark_catalog.default.catalog_sales (6)
: : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales (6)
: : : +- CometSort (15)
: : : +- CometExchange (14)
: : : +- CometProject (13)
: : : +- CometScan parquet spark_catalog.default.catalog_returns (12)
: : : +- CometNativeScan parquet spark_catalog.default.catalog_returns (12)
: : +- CometBroadcastExchange (20)
: : +- CometProject (19)
: : +- CometFilter (18)
: : +- CometScan parquet spark_catalog.default.date_dim (17)
: : +- CometNativeScan parquet spark_catalog.default.date_dim (17)
: +- CometBroadcastExchange (26)
: +- CometProject (25)
: +- CometFilter (24)
: +- CometScan parquet spark_catalog.default.customer_address (23)
: +- CometNativeScan parquet spark_catalog.default.customer_address (23)
+- CometBroadcastExchange (32)
+- CometProject (31)
+- CometFilter (30)
+- CometScan parquet spark_catalog.default.call_center (29)
+- CometNativeScan parquet spark_catalog.default.call_center (29)


(1) CometScan parquet spark_catalog.default.catalog_sales
(1) CometNativeScan parquet spark_catalog.default.catalog_sales
Output [8]: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_warehouse_sk#4, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7, cs_sold_date_sk#8]
Batched: true
Location [not included in comparison]/{warehouse_dir}/catalog_sales]
Expand All @@ -64,7 +64,7 @@ Arguments: hashpartitioning(cs_order_number#5, 5), ENSURE_REQUIREMENTS, CometNat
Input [7]: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_warehouse_sk#4, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7]
Arguments: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_warehouse_sk#4, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7], [cs_order_number#5 ASC NULLS FIRST]

(6) CometScan parquet spark_catalog.default.catalog_sales
(6) CometNativeScan parquet spark_catalog.default.catalog_sales
Output [3]: [cs_warehouse_sk#9, cs_order_number#10, cs_sold_date_sk#11]
Batched: true
Location [not included in comparison]/{warehouse_dir}/catalog_sales]
Expand All @@ -91,7 +91,7 @@ Arguments: [cs_order_number#5], [cs_order_number#10], LeftSemi, NOT (cs_warehous
Input [7]: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_warehouse_sk#4, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7]
Arguments: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7], [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7]

(12) CometScan parquet spark_catalog.default.catalog_returns
(12) CometNativeScan parquet spark_catalog.default.catalog_returns
Output [2]: [cr_order_number#12, cr_returned_date_sk#13]
Batched: true
Location [not included in comparison]/{warehouse_dir}/catalog_returns]
Expand All @@ -114,7 +114,7 @@ Left output [6]: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_
Right output [1]: [cr_order_number#12]
Arguments: [cs_order_number#5], [cr_order_number#12], LeftAnti

(17) CometScan parquet spark_catalog.default.date_dim
(17) CometNativeScan parquet spark_catalog.default.date_dim
Output [2]: [d_date_sk#14, d_date#15]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
Expand Down Expand Up @@ -142,7 +142,7 @@ Arguments: [cs_ship_date_sk#1], [d_date_sk#14], Inner, BuildRight
Input [7]: [cs_ship_date_sk#1, cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7, d_date_sk#14]
Arguments: [cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7], [cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7]

(23) CometScan parquet spark_catalog.default.customer_address
(23) CometNativeScan parquet spark_catalog.default.customer_address
Output [2]: [ca_address_sk#16, ca_state#17]
Batched: true
Location [not included in comparison]/{warehouse_dir}/customer_address]
Expand Down Expand Up @@ -170,7 +170,7 @@ Arguments: [cs_ship_addr_sk#2], [ca_address_sk#16], Inner, BuildRight
Input [6]: [cs_ship_addr_sk#2, cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7, ca_address_sk#16]
Arguments: [cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7], [cs_call_center_sk#3, cs_order_number#5, cs_ext_ship_cost#6, cs_net_profit#7]

(29) CometScan parquet spark_catalog.default.call_center
(29) CometNativeScan parquet spark_catalog.default.call_center
Output [2]: [cc_call_center_sk#18, cc_county#19]
Batched: true
Location [not included in comparison]/{warehouse_dir}/call_center]
Expand Down
12 changes: 6 additions & 6 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q16/simplified.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,24 +21,24 @@ WholeStageCodegen (2)
CometExchange [cs_order_number] #2
CometProject [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit]
CometFilter [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
CometScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
CometNativeScan parquet spark_catalog.default.catalog_sales [cs_ship_date_sk,cs_ship_addr_sk,cs_call_center_sk,cs_warehouse_sk,cs_order_number,cs_ext_ship_cost,cs_net_profit,cs_sold_date_sk]
CometSort [cs_warehouse_sk,cs_order_number]
CometExchange [cs_order_number] #3
CometProject [cs_warehouse_sk,cs_order_number]
CometScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
CometNativeScan parquet spark_catalog.default.catalog_sales [cs_warehouse_sk,cs_order_number,cs_sold_date_sk]
CometSort [cr_order_number]
CometExchange [cr_order_number] #4
CometProject [cr_order_number]
CometScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
CometNativeScan parquet spark_catalog.default.catalog_returns [cr_order_number,cr_returned_date_sk]
CometBroadcastExchange [d_date_sk] #5
CometProject [d_date_sk]
CometFilter [d_date_sk,d_date]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
CometNativeScan parquet spark_catalog.default.date_dim [d_date_sk,d_date]
CometBroadcastExchange [ca_address_sk] #6
CometProject [ca_address_sk]
CometFilter [ca_address_sk,ca_state]
CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
CometNativeScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_state]
CometBroadcastExchange [cc_call_center_sk] #7
CometProject [cc_call_center_sk]
CometFilter [cc_call_center_sk,cc_county]
CometScan parquet spark_catalog.default.call_center [cc_call_center_sk,cc_county]
CometNativeScan parquet spark_catalog.default.call_center [cc_call_center_sk,cc_county]
24 changes: 12 additions & 12 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q19/explain.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,26 +16,26 @@
: : : : +- CometBroadcastHashJoin (7)
: : : : :- CometProject (3)
: : : : : +- CometFilter (2)
: : : : : +- CometScan parquet spark_catalog.default.date_dim (1)
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim (1)
: : : : +- CometBroadcastExchange (6)
: : : : +- CometFilter (5)
: : : : +- CometScan parquet spark_catalog.default.store_sales (4)
: : : : +- CometNativeScan parquet spark_catalog.default.store_sales (4)
: : : +- CometBroadcastExchange (12)
: : : +- CometProject (11)
: : : +- CometFilter (10)
: : : +- CometScan parquet spark_catalog.default.item (9)
: : : +- CometNativeScan parquet spark_catalog.default.item (9)
: : +- CometBroadcastExchange (17)
: : +- CometFilter (16)
: : +- CometScan parquet spark_catalog.default.customer (15)
: : +- CometNativeScan parquet spark_catalog.default.customer (15)
: +- CometBroadcastExchange (22)
: +- CometFilter (21)
: +- CometScan parquet spark_catalog.default.customer_address (20)
: +- CometNativeScan parquet spark_catalog.default.customer_address (20)
+- CometBroadcastExchange (27)
+- CometFilter (26)
+- CometScan parquet spark_catalog.default.store (25)
+- CometNativeScan parquet spark_catalog.default.store (25)


(1) CometScan parquet spark_catalog.default.date_dim
(1) CometNativeScan parquet spark_catalog.default.date_dim
Output [3]: [d_date_sk#1, d_year#2, d_moy#3]
Batched: true
Location [not included in comparison]/{warehouse_dir}/date_dim]
Expand All @@ -50,7 +50,7 @@ Condition : ((((isnotnull(d_moy#3) AND isnotnull(d_year#2)) AND (d_moy#3 = 11))
Input [3]: [d_date_sk#1, d_year#2, d_moy#3]
Arguments: [d_date_sk#1], [d_date_sk#1]

(4) CometScan parquet spark_catalog.default.store_sales
(4) CometNativeScan parquet spark_catalog.default.store_sales
Output [5]: [ss_item_sk#4, ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, ss_sold_date_sk#8]
Batched: true
Location: InMemoryFileIndex []
Expand All @@ -75,7 +75,7 @@ Arguments: [d_date_sk#1], [ss_sold_date_sk#8], Inner, BuildRight
Input [6]: [d_date_sk#1, ss_item_sk#4, ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, ss_sold_date_sk#8]
Arguments: [ss_item_sk#4, ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7], [ss_item_sk#4, ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7]

(9) CometScan parquet spark_catalog.default.item
(9) CometNativeScan parquet spark_catalog.default.item
Output [6]: [i_item_sk#9, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, i_manager_id#14]
Batched: true
Location [not included in comparison]/{warehouse_dir}/item]
Expand Down Expand Up @@ -103,7 +103,7 @@ Arguments: [ss_item_sk#4], [i_item_sk#9], Inner, BuildRight
Input [9]: [ss_item_sk#4, ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, i_item_sk#9, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13]
Arguments: [ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13], [ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13]

(15) CometScan parquet spark_catalog.default.customer
(15) CometNativeScan parquet spark_catalog.default.customer
Output [2]: [c_customer_sk#15, c_current_addr_sk#16]
Batched: true
Location [not included in comparison]/{warehouse_dir}/customer]
Expand All @@ -127,7 +127,7 @@ Arguments: [ss_customer_sk#5], [c_customer_sk#15], Inner, BuildRight
Input [9]: [ss_customer_sk#5, ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, c_customer_sk#15, c_current_addr_sk#16]
Arguments: [ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, c_current_addr_sk#16], [ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, c_current_addr_sk#16]

(20) CometScan parquet spark_catalog.default.customer_address
(20) CometNativeScan parquet spark_catalog.default.customer_address
Output [2]: [ca_address_sk#17, ca_zip#18]
Batched: true
Location [not included in comparison]/{warehouse_dir}/customer_address]
Expand All @@ -151,7 +151,7 @@ Arguments: [c_current_addr_sk#16], [ca_address_sk#17], Inner, BuildRight
Input [9]: [ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, c_current_addr_sk#16, ca_address_sk#17, ca_zip#18]
Arguments: [ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, ca_zip#18], [ss_store_sk#6, ss_ext_sales_price#7, i_brand_id#10, i_brand#11, i_manufact_id#12, i_manufact#13, ca_zip#18]

(25) CometScan parquet spark_catalog.default.store
(25) CometNativeScan parquet spark_catalog.default.store
Output [2]: [s_store_sk#19, s_zip#20]
Batched: true
Location [not included in comparison]/{warehouse_dir}/store]
Expand Down
12 changes: 6 additions & 6 deletions spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q19/simplified.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,20 @@ WholeStageCodegen (1)
CometBroadcastHashJoin [d_date_sk,ss_item_sk,ss_customer_sk,ss_store_sk,ss_ext_sales_price,ss_sold_date_sk]
CometProject [d_date_sk]
CometFilter [d_date_sk,d_year,d_moy]
CometScan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy]
CometNativeScan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy]
CometBroadcastExchange [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ext_sales_price,ss_sold_date_sk] #2
CometFilter [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ext_sales_price,ss_sold_date_sk]
CometScan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ext_sales_price,ss_sold_date_sk]
CometNativeScan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ext_sales_price,ss_sold_date_sk]
CometBroadcastExchange [i_item_sk,i_brand_id,i_brand,i_manufact_id,i_manufact] #3
CometProject [i_item_sk,i_brand_id,i_brand,i_manufact_id,i_manufact]
CometFilter [i_item_sk,i_brand_id,i_brand,i_manufact_id,i_manufact,i_manager_id]
CometScan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_brand,i_manufact_id,i_manufact,i_manager_id]
CometNativeScan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_brand,i_manufact_id,i_manufact,i_manager_id]
CometBroadcastExchange [c_customer_sk,c_current_addr_sk] #4
CometFilter [c_customer_sk,c_current_addr_sk]
CometScan parquet spark_catalog.default.customer [c_customer_sk,c_current_addr_sk]
CometNativeScan parquet spark_catalog.default.customer [c_customer_sk,c_current_addr_sk]
CometBroadcastExchange [ca_address_sk,ca_zip] #5
CometFilter [ca_address_sk,ca_zip]
CometScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_zip]
CometNativeScan parquet spark_catalog.default.customer_address [ca_address_sk,ca_zip]
CometBroadcastExchange [s_store_sk,s_zip] #6
CometFilter [s_store_sk,s_zip]
CometScan parquet spark_catalog.default.store [s_store_sk,s_zip]
CometNativeScan parquet spark_catalog.default.store [s_store_sk,s_zip]
Loading
Loading