feat: Define function signatures in CometFuzz #2614

andygrove · 2025-10-20T20:22:20Z

Which issue does this PR close?

Part of #2611

Rationale for this change

The fuzz tester currently passes random inputs to functions without checking if they are the correct type. For example, it could try and pass a string to a numeric function. Although this can be a valid test (because Spark will add a cast to coerce the input type) it also means that many generated queries are not valid, so is not very efficient.

What changes are included in this PR?

Define signatures for functions
Add all functions that Comet currently supports
Update query generator to only generate queries using valid input columns for the functions
Improve error handling

How are these changes tested?

Manually

codecov-commenter · 2025-10-20T21:16:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.15%. Comparing base (f09f8af) to head (6058427).
⚠️ Report is 632 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2614      +/-   ##
============================================
+ Coverage     56.12%   59.15%   +3.02%     
- Complexity      976     1444     +468     
============================================
  Files           119      147      +28     
  Lines         11743    13735    +1992     
  Branches       2251     2356     +105     
============================================
+ Hits           6591     8125    +1534     
- Misses         4012     4386     +374     
- Partials       1140     1224      +84

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mbutrovich · 2025-10-21T17:31:10Z

How does this change with different Spark versions, or does it?

andygrove · 2025-10-21T17:44:19Z

How does this change with different Spark versions, or does it?

It really doesn't. As an example, if we add a new function that only exists in Spark 4.0 then run the fuzz test against an older version, the query will fail both with Spark and Comet, so that is a pass.

mbutrovich · 2025-10-21T17:50:54Z

It really doesn't. As an example, if we add a new function that only exists in Spark 4.0 then run the fuzz test against an older version, the query will fail both with Spark and Comet, so that is a pass.

What if the signature changes in a new Spark release? Then it would start failing for Spark and Comet (and thus pass)?

I'm just trying to understand what the maintenance process is for future releases, and how to potentially document that.

comphead · 2025-10-21T18:05:21Z

fuzz-testing/src/main/scala/org/apache/comet/fuzz/Meta.scala

-
-  val dateScalarFunc: Seq[Function] =
-    Seq(Function("year", 1), Function("hour", 1), Function("minute", 1), Function("second", 1))
+  private def createFunctionWithInputs(name: String, inputs: Seq[SparkType]): Function = {


Suggested change

private def createFunctionWithInputs(name: String, inputs: Seq[SparkType]): Function = {

private def createFunctionWithInputParams(name: String, inputs: Seq[SparkType]): Function = {

inputs might be confused with input data

I renamed to createFunctionWithInputTypes

andygrove · 2025-10-21T18:09:03Z

It really doesn't. As an example, if we add a new function that only exists in Spark 4.0 then run the fuzz test against an older version, the query will fail both with Spark and Comet, so that is a pass.

What if the signature changes in a new Spark release? Then it would start failing for Spark and Comet (and thus pass)?

I'm just trying to understand what the maintenance process is for future releases, and how to potentially document that.

That is true. This is all very manual at the moment.

I briefly looked into using Spark APIs to get the signature, but there are some challenges. We can look at classes to see if they extend UnaryExpression or BinaryExpression, but to determine the valid input data types we would need to create an instance of the class, which seems challenging.

comphead · 2025-10-21T18:09:32Z

fuzz-testing/src/main/scala/org/apache/comet/fuzz/Meta.scala

+  }
+
+  // Math expressions (corresponds to mathExpressions in QueryPlanSerde)
  val mathScalarFunc: Seq[Function] = Seq(


perhaps we can validate that expressions in QueryPlanSerde not covered by fuzzer? it would help us having a consistent functions set we fuzz testing especially if a person added a new function

That would be nice. The main challenge is that in the expr map in QueryPlanSerde we just have class names and no mapping to SQL function name.

My current approach is to use AI to detect any expressions in QueryPlanSerde that are not covered in the fuzz test.

I filed #2627 for finding a way to automate this

comphead · 2025-10-21T18:11:39Z

fuzz-testing/src/main/scala/org/apache/comet/fuzz/Meta.scala

+    createUnaryStringFunction("ascii"),
+    createUnaryStringFunction("bit_length"),
+    createUnaryStringFunction("chr"),
+    createFunctionWithInputs("concat_ws", Seq(SparkStringType, SparkStringType)),


we should be supporting concat with strings input in 50.3.0 #2604 so need to add it there.

Btw @andygrove concat supports string or arrays as input, looks like this design supports it?

Yes, the framework supports it. In this case we could add two signatures to the function. One that takes two strings and one that takes two arrays. I did not implement support for variadic functions yet. I will file an issue for that.

I added support for concat with two arguments. According to the docs:

The function works with strings, numeric, binary and compatible array columns.

comphead · 2025-10-21T18:14:25Z

fuzz-testing/src/main/scala/org/apache/comet/fuzz/QueryRunner.scala

        a.length == b.length && a.zip(b).forall(x => same(x._1, x._2))
+      case (a: Row, b: Row) =>
+        // struct support
+        format(a) == format(b)


I'm not sure what is compared here? is it text representation of structs?

Yes. This could probably be made more efficient.

I updated this

comphead · 2025-10-21T18:16:33Z

fuzz-testing/src/main/scala/org/apache/comet/fuzz/QueryRunner.scala

+      return l == null && r == null
+    }
    (l, r) match {
      case (a: Float, b: Float) if a.isInfinity => b.isInfinity


should we also check negInfinity and posInfinity

andygrove · 2025-10-21T18:22:59Z

It really doesn't. As an example, if we add a new function that only exists in Spark 4.0 then run the fuzz test against an older version, the query will fail both with Spark and Comet, so that is a pass.

What if the signature changes in a new Spark release? Then it would start failing for Spark and Comet (and thus pass)?
I'm just trying to understand what the maintenance process is for future releases, and how to potentially document that.

That is true. This is all very manual at the moment.

I briefly looked into using Spark APIs to get the signature, but there are some challenges. We can look at classes to see if they extend UnaryExpression or BinaryExpression, but to determine the valid input data types we would need to create an instance of the class, which seems challenging.

@mbutrovich One option I would like to explore is to produce a summary report after running the queries, which would show how many successful queries ran for each expression and show error messages for any expressions that always failed

edit: I filed an isssue for this: #2618

comphead · 2025-10-21T18:48:52Z

For comparison, should we delegate this to Spark itself for simplest cases? 🤔

   sparkDf.count = = cometDf.count // to see duplicates or missing rows
   sparkDf.except(comeDf).union(cometDF.except(sparkDf)) // column-level checks

andygrove · 2025-10-21T19:17:56Z

For comparison, should we delegate this to Spark itself for simplest cases? 🤔
   sparkDf.count = = cometDf.count // to see duplicates or missing rows
   sparkDf.except(comeDf).union(cometDF.except(sparkDf)) // column-level checks

Will this involve re-executing the queries?

comphead · 2025-10-21T19:29:20Z

For comparison, should we delegate this to Spark itself for simplest cases? 🤔
   sparkDf.count = = cometDf.count // to see duplicates or missing rows
   sparkDf.except(comeDf).union(cometDF.except(sparkDf)) // column-level checks
Will this involve re-executing the queries?

Yes, both rows would be actions and trigger the dataframe evaluation. to avoid this we can cache/checkpoint dataframes, so it would be evaluated only once. Another option is to save both result df to disk and then read them for correctness checks, this option may also help in the future when Comet support the parquet writer.

wForget · 2025-10-22T10:00:04Z

I briefly looked into using Spark APIs to get the signature, but there are some challenges.

We may be able to obtain function signatures in these ways:

Get input types from ExpectsInputTypes trait
Parse function signatures from example sql, refer: ExpressionsSchemaSuite.scala
Parse argument number from function desc, like _FUNC_(str, charset) in org.apache.spark.sql.catalyst.expressions.Encode desc

andygrove · 2025-10-22T13:59:13Z

Get input types from ExpectsInputTypes trait

Parse function signatures from example sql, refer: ExpressionsSchemaSuite.scala

Parse argument number from function desc, like _FUNC_(str, charset) in org.apache.spark.sql.catalyst.expressions.Encode desc

Thanks. I filed an issue for this #2627

andygrove · 2025-10-22T14:06:19Z

For comparison, should we delegate this to Spark itself for simplest cases? 🤔
   sparkDf.count = = cometDf.count // to see duplicates or missing rows
   sparkDf.except(comeDf).union(cometDF.except(sparkDf)) // column-level checks

I'm not sure how we can run cometDF.except(sparkDf) with cometDF running with Comet enabled and sparkDf running with Comet disabled

andygrove · 2025-10-22T14:11:42Z

@mbutrovich @comphead Thanks for the reviews so far. CometFuzz is still quite experimental/hacky, but this PR expands coverage of tested functions and reduces the number of invalid queries generated now that we have signatures, so it seems worth merging in my opinion. It would be better to automate the discovery of function signatures rather than hand-code them. I filed #2627 to explore this.

Here are stats from a recent run of this version:

Total queries: 853; Invalid queries: 317; Comet failed: 3; Comet succeeded: 533

So far, this version of CometFuzz has found three bugs:

andygrove · 2025-10-22T17:19:39Z

Moved to draft until #2629 is merged

andygrove added 6 commits October 20, 2025 14:19

start to define function signatures

6042822

update more signatures

e41325b

update more signatures

166f139

update more signatures

0db4999

update more signatures

984ceef

test

b087839

andygrove added 9 commits October 20, 2025 17:41

save progress

4bf6939

save [skip ci]

5e20dd8

save [skip ci]

73c9185

update more signatures

af4ccf8

convert remaining signatures

64d40ab

add more functions

3c1feaf

refactor [skip ci]

9e185eb

query gen updates

d13301d

error handling

4d4362e

andygrove marked this pull request as ready for review October 21, 2025 15:50

andygrove changed the title ~~feat: Define function signatures in CometFuzz [WIP]~~ feat: Define function signatures in CometFuzz Oct 21, 2025

andygrove added 2 commits October 21, 2025 10:22

update agg query gen

a90e64f

fix copy paste

ad71962

fix

9841b5c

comphead reviewed Oct 21, 2025

View reviewed changes

andygrove added 2 commits October 21, 2025 12:50

partially address feedback

9509bfa

concat

903482c

andygrove added 6 commits October 21, 2025 16:09

reduce number of invalid queries

aa979e6

save

1147374

Merge remote-tracking branch 'apache/main' into fuzz-sig

638db06

save

f0b89a2

skip first/last

bb0dfa1

offheap

28b0509

andygrove mentioned this pull request Oct 22, 2025

[CometFuzz] Automate keeping function signatures up-to-date with Spark #2627

Open

andygrove marked this pull request as draft October 22, 2025 15:10

andygrove added 3 commits October 22, 2025 15:14

upmerge

6fc6607

address feedback

7c38dfa

improve reporting

871b571

andygrove marked this pull request as ready for review October 22, 2025 21:21

format

6058427

comphead mentioned this pull request Oct 22, 2025

chore: add TPC queries to be run by fuzzer correctness checker #2632

Open

	private def createFunctionWithInputs(name: String, inputs: Seq[SparkType]): Function = {
	private def createFunctionWithInputParams(name: String, inputs: Seq[SparkType]): Function = {

feat: Define function signatures in CometFuzz #2614

Are you sure you want to change the base?

feat: Define function signatures in CometFuzz #2614

Conversation

andygrove commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

codecov-commenter commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mbutrovich commented Oct 21, 2025

Uh oh!

andygrove commented Oct 21, 2025

Uh oh!

mbutrovich commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Oct 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Oct 21, 2025

Uh oh!

andygrove commented Oct 21, 2025

Uh oh!

comphead commented Oct 21, 2025

Uh oh!

wForget commented Oct 22, 2025

Uh oh!

andygrove commented Oct 22, 2025

Uh oh!

andygrove commented Oct 22, 2025

Uh oh!

andygrove commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andygrove commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

andygrove commented Oct 20, 2025 •

edited

Loading

codecov-commenter commented Oct 20, 2025 •

edited

Loading

mbutrovich commented Oct 21, 2025 •

edited

Loading

andygrove commented Oct 21, 2025 •

edited

Loading

andygrove commented Oct 22, 2025 •

edited

Loading