Vortex has some specified behavior for aggregation functions, but it's not formalized. Duckdb and Datafusion as our main query engines have other specified behaviour. Our vortex-duckdb and vortex-datafusion integrations have a third set of behavior. Goal of this epic is to bring our integrations to the main query engine processing.
NaN means both + and - nan since both Duckdb and Datafusion treat both the same.
Integer (BIGINT / i64)
| Query |
ANSI |
DuckDB |
DataFusion |
Vortex |
Vortex + Duckdb |
sum([1,2,NULL]) |
3 |
3 |
3 |
3 |
3 (pushed) |
min / max([1,2,NULL]) |
1 / 2 |
1 / 2 |
1 / 2 |
1 / 2 |
1 / 2 (pushed) |
avg([1,2,NULL]) |
1.5 |
1.5 |
1.5 |
1.5 |
1.5 (pushed) |
count([1,2,NULL]) |
2 |
2 |
2 |
2 |
2 (pushed) |
sum(all-null) |
NULL |
NULL |
NULL |
0 |
MISMATCH 0 (#9084) |
min / max(all-null) |
NULL |
NULL |
NULL |
NULL |
NULL |
avg(all-null) |
NULL |
NULL |
NULL |
NaN |
MISMATCH NaN (#9083) |
count(all-null) |
0 |
0 |
0 |
0 |
0 |
Floating point
Floating point pushdown is turned off since #9069 but we can revert that for all functions but max.
Therefore last column is the situation where we push to floats
| Query |
ANSI |
DuckDB |
DataFusion |
Vortex (skip_nans) |
Vortex + Duckdb |
count([NaN,1,2,NULL]) |
impl-def |
3 |
3 |
2 |
3 |
sum([NaN,1,2]) |
impl-def |
NaN |
NaN |
3 |
NaN |
min([NaN,1,2]) |
impl-def |
1 |
1 |
1 |
1 (not pushed) |
max([NaN,1,2]) |
impl-def |
NaN |
NaN |
2 |
NaN |
avg([NaN,1,2]) |
impl-def |
NaN |
NaN |
1.5 |
NaN |
count([NaN,NaN]) |
impl-def |
2 |
2 |
0 |
2 |
sum([NaN,NaN]) |
impl-def |
NaN |
NaN |
0 |
NaN |
min([NaN,NaN]) |
impl-def |
NaN |
NaN |
NULL |
NaN (not pushed) |
max([NaN,NaN]) |
impl-def |
NaN |
NaN |
NULL |
NaN |
avg([NaN,NaN]) |
impl-def |
NaN |
NaN |
NaN |
NaN |
sum([inf,-inf,0]) |
impl-def |
NaN |
NaN |
NaN |
NaN |
min / max([inf,-inf,0]) |
impl-def |
-inf / inf |
-inf / inf |
-inf / inf |
-inf (not pushed) / inf |
sum([inf,1,2]) |
impl-def |
inf |
inf |
inf |
inf |
sum(all-null) |
NULL |
NULL |
NULL |
0 |
MISMATCH 0 |
min / max(all-null) |
NULL |
NULL |
NULL |
NULL |
NULL |
avg(all-null) |
NULL |
NULL |
NULL |
NaN |
MISMATCH NaN |
count(all-null) |
0 |
0 |
0 |
0 |
0 |
String (VARCHAR / Utf8)
| Query |
ANSI |
DuckDB |
DataFusion |
Vortex |
Vortex + DuckDB |
min / max(['banana','apple',NULL]) |
apple / banana |
apple / banana |
apple / banana |
apple / banana |
apple / banana (pushed) |
count(['banana','apple',NULL]) |
2 |
2 |
2 |
2 |
2 (pushed) |
min / max(all-null) |
NULL |
NULL |
NULL |
NULL |
NULL |
count(all-null) |
0 |
0 |
0 |
0 |
0 |
sum(x) / avg(x) |
type error |
error |
error |
unsupported |
not pushed |
Vortex has some specified behavior for aggregation functions, but it's not formalized. Duckdb and Datafusion as our main query engines have other specified behaviour. Our vortex-duckdb and vortex-datafusion integrations have a third set of behavior. Goal of this epic is to bring our integrations to the main query engine processing.
NaN means both + and - nan since both Duckdb and Datafusion treat both the same.
Integer (
BIGINT/i64)sum([1,2,NULL])33333(pushed)min/max([1,2,NULL])1/21/21/21/21/2(pushed)avg([1,2,NULL])1.51.51.51.51.5(pushed)count([1,2,NULL])22222(pushed)sum(all-null)NULLNULLNULL00(#9084)min/max(all-null)NULLNULLNULLNULLNULLavg(all-null)NULLNULLNULLNaNNaN(#9083)count(all-null)00000Floating point
Floating point pushdown is turned off since #9069 but we can revert that for all functions but max.
Therefore last column is the situation where we push to floats
count([NaN,1,2,NULL])3323sum([NaN,1,2])NaNNaN3NaNmin([NaN,1,2])1111(not pushed)max([NaN,1,2])NaNNaN2NaNavg([NaN,1,2])NaNNaN1.5NaNcount([NaN,NaN])2202sum([NaN,NaN])NaNNaN0NaNmin([NaN,NaN])NaNNaNNULLNaN(not pushed)max([NaN,NaN])NaNNaNNULLNaNavg([NaN,NaN])NaNNaNNaNNaNsum([inf,-inf,0])NaNNaNNaNNaNmin/max([inf,-inf,0])-inf/inf-inf/inf-inf/inf-inf(not pushed) /infsum([inf,1,2])infinfinfinfsum(all-null)NULLNULLNULL00min/max(all-null)NULLNULLNULLNULLNULLavg(all-null)NULLNULLNULLNaNNaNcount(all-null)00000String (
VARCHAR/Utf8)min/max(['banana','apple',NULL])apple/bananaapple/bananaapple/bananaapple/bananaapple/banana(pushed)count(['banana','apple',NULL])22222(pushed)min/max(all-null)NULLNULLNULLNULLNULLcount(all-null)00000sum(x)/avg(x)