Feat: support array_except function #1343
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1343 +/- ##
============================================
+ Coverage 56.12% 57.49% +1.36%
- Complexity 976 978 +2
============================================
Files 119 121 +2
Lines 11743 11944 +201
Branches 2251 2254 +3
============================================
+ Hits 6591 6867 +276
+ Misses 4012 3919 -93
- Partials 1140 1158 +18 ☔ View full report in Codecov by Sentry. |
# Conflicts: # native/core/src/execution/planner.rs # native/proto/src/proto/expr.proto # spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala # spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
# Conflicts: # spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala # spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
| sql("SELECT array_except(array(_2, _3, _4), array(_3, _4)) from t1")) | ||
| checkSparkAnswerAndOperator(sql("SELECT array_except(array(_18), array(_19)) from t1")) | ||
| checkSparkAnswerAndOperator(spark.sql( | ||
| "SELECT array_except((CASE WHEN _2 = _3 THEN array(_2, _3, _4) END), array(_4)) FROM t1")) |
There was a problem hiding this comment.
_2 = _3 would be always true?
There was a problem hiding this comment.
You are right, it is necessary to add the condition is not null, thank you.
There was a problem hiding this comment.
I saw you added WHERE _2 IS NOT NULL but not sure how it helps... _2 = _3 will be always true is it intentional?
There was a problem hiding this comment.
This is a test case for deduplication checking.
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Outdated
Show resolved
Hide resolved
|
Looks there are some format issues need to be fixed @kazantsev-maksim otherwise I think it is good to go |
|
Thanks @kazuyukitanimura. I've fixed the formatting in the |
|
@kazantsev-maksim it seems there are test failures |
Which issue does this PR close?
Related to Epic: #1042
array_except: SELECT array_except(array(1, 2, 3), array(1, 3, 5)) => array(2)
DataFusion' s array_except has same behavior with Spark 's array_except function
Spark: https://docs.databricks.com/en/sql/language-manual/functions/array_except.html
DataFusion: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#array-except
Rationale for this change
Defined under Epic: #1042
What changes are included in this PR?
planner.rs: Maps Spark 's array_except function to DataFusion array_except_udf physical expression from Spark physical expression
expr.proto: array_except array_except has been added,
QueryPlanSerde.scala: arrays_except pattern matching case has been added,
CometExpressionSuite.scala: A new UT has been added for array_except function.
How are these changes tested?
A new UT has been added.