Support group by span over time based column with Span UDF #3421

songkant-aws · 2025-03-13T11:25:22Z

Description

Support group by span over time based column with Span UDF

Related Issues

Resolves #3354

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Songkan Tang <[email protected]>

…ngine Signed-off-by: Songkan Tang <[email protected]>

qianheng-aws · 2025-03-14T02:43:19Z

core/src/main/java/org/opensearch/sql/calcite/udf/SpanFunction.java

+                date.atStartOfDay().atZone(ZoneOffset.UTC).toInstant().toEpochMilli(), interval);
+        return SqlFunctions.timestampToDate(dateEpochValue);
+      case SqlTypeName.TIME:
+        if (dateTimeUnit.getId() > 4) {


Could you explain why need this limitation?

Added comment inline. The TIME type usually means date format without year, month, etc like the field '17:59:59.99'

qianheng-aws · 2025-03-14T03:42:55Z

core/src/main/java/org/opensearch/sql/calcite/udf/SpanFunction.java

+ *       day, 1 month, 1 hour
+ * </ol>
+ */
+public class SpanFunction implements UserDefinedFunction {


Maybe leave a TODO here to do refactoring in future. Your previous idea of implementing a self-defined implementor seems to be a better approach.

Looks like it could be implemented by replacing ScalarFunction(we currently use) with ImplementableFunction, and move logic for handling different filed type to the implementor. In that way, this span function should be simpler with only handling timestamp type.

Then, we may have different functions like dateToTs, tsToDate, span(ForTs) and provide us more flexibility to reuse them or combine them.

…ngine

Signed-off-by: Songkan Tang <[email protected]>

LantaoJin · 2025-03-17T04:15:47Z

core/src/main/java/org/opensearch/sql/calcite/utils/BuiltinFunctionUtils.java

@@ -100,6 +101,9 @@ static SqlOperator translate(String op) {
        return SqlLibraryOperators.DATE_ADD_SPARK;
      case "DATE_ADD":
        return SqlLibraryOperators.DATEADD;
+        // UDF Functions
+      case "SPAN":
+        return TransferUserDefinedFunction(SpanFunction.class, "SPAN", ReturnTypes.ARG0);


ReturnTypes.ARG0 or ReturnTypes.ARG0_NULLABLE? What if the timestamp is null in span(timestamp, ...)?

I changed it to ReturnTypes.ARG0_NULLABLE but it will transform it to default long or int value like 0L or 0 in linq4j generated code. Still looking

Resolved the groupby nulls issue due to sql type creation conflicts in RexNodeBuilder.

LantaoJin · 2025-03-17T04:18:00Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLAggregationIT.java

  public void testAvgByTimeSpanAndFields() {
    JSONObject actual =
        executeQuery(
            String.format(
-                "source=%s | stats avg(balance) by span(birthdate, 1 day) as age_balance",
+                "source=%s | stats avg(balance) by span(birthdate, 1 month) as age_balance",


we need some ITs for such as 15 minutes

Added 15 minutes IT.

Signed-off-by: Songkan Tang <[email protected]>

…ngine

Signed-off-by: Songkan Tang <[email protected]>

LantaoJin · 2025-03-18T10:57:32Z

PPLQueryDataAnonymizer and PPLQueryDataAnonymizerTest are required to update for span expression. @songkant-aws

LantaoJin · 2025-03-18T11:02:43Z

Flaky test

CalcitePPLStringBuiltinFunctionIT > testPosition FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([EFDC39F9CD048074:F34E991E6AAE7EDB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.sql.util.MatcherUtils.verify(MatcherUtils.java:192)
        at org.opensearch.sql.util.MatcherUtils.verifyDataRows(MatcherUtils.java:160)
        at org.opensearch.sql.calcite.standalone.CalcitePPLStringBuiltinFunctionIT.testPosition(CalcitePPLStringBuiltinFunctionIT.java:161)

Let me trigger a re-run attempt.

cc @yuancu

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws · 2025-03-19T06:22:01Z

PPLQueryDataAnonymizer and PPLQueryDataAnonymizerTest are required to update for span expression. @songkant-aws

Added corresponding change and more IT to cover span over different kinds of date formats.

songkant-aws · 2025-03-19T07:15:33Z

Looks like we have some flaky tests, this time there is one in build-windows-macos, which is irrelevant to this PR's change.

org.opensearch.sql.calcite.standalone.CalcitePPLStringBuiltinFunctionIT.testSubstring(CalcitePPLStringBuiltinFunctionIT.java:148)

LantaoJin · 2025-03-19T09:03:23Z

Failure due to flaky test, merged.

Support group by span over time based column with Span UDF

5cd75c5

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin and noCharger as code owners March 13, 2025 11:25

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

f94632d

…ngine Signed-off-by: Songkan Tang <[email protected]>

qianheng-aws reviewed Mar 14, 2025

View reviewed changes

songkant-aws added 2 commits March 17, 2025 10:19

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

a4137a2

…ngine

Minor fix UT and IT after merge

e2ea5ea

Signed-off-by: Songkan Tang <[email protected]>

LantaoJin reviewed Mar 17, 2025

View reviewed changes

songkant-aws added 3 commits March 17, 2025 17:27

Add custom minute span IT

3a39f38

Signed-off-by: Songkan Tang <[email protected]>

Merge branch 'feature/calcite-engine' into songkant/feature/calcite-e…

8fe3727

…ngine

Remove print

7ae4dc8

Signed-off-by: Songkan Tang <[email protected]>

songkant-aws added 4 commits March 17, 2025 18:14

Fix spotless style

e81326b

Signed-off-by: Songkan Tang <[email protected]>

Fix groupby nulls issue

cac6023

Signed-off-by: Songkan Tang <[email protected]>

Remove unnecessary sql type creation in RexNodeBuilder

70afc6c

Signed-off-by: Songkan Tang <[email protected]>

Fix rounding for date and time type

92d3857

Signed-off-by: Songkan Tang <[email protected]>

LantaoJin added the calcite calcite migration releated label Mar 18, 2025

songkant-aws added 2 commits March 19, 2025 14:12

Add more tests

ee7d03f

Signed-off-by: Songkan Tang <[email protected]>

Revert unrelated change in build.gradle

f515620

Signed-off-by: Songkan Tang <[email protected]>

LantaoJin approved these changes Mar 19, 2025

View reviewed changes

LantaoJin merged commit b99130a into opensearch-project:feature/calcite-engine Mar 19, 2025
18 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support group by span over time based column with Span UDF #3421

Support group by span over time based column with Span UDF #3421

songkant-aws commented Mar 13, 2025 •

edited by LantaoJin

Loading

qianheng-aws Mar 14, 2025

songkant-aws Mar 17, 2025 •

edited

Loading

qianheng-aws Mar 14, 2025 •

edited

Loading

songkant-aws Mar 17, 2025

LantaoJin Mar 17, 2025

songkant-aws Mar 17, 2025 •

edited

Loading

songkant-aws Mar 18, 2025

LantaoJin Mar 17, 2025

songkant-aws Mar 17, 2025

LantaoJin commented Mar 18, 2025 •

edited

Loading

LantaoJin commented Mar 18, 2025

songkant-aws commented Mar 19, 2025

songkant-aws commented Mar 19, 2025

LantaoJin commented Mar 19, 2025

Support group by span over time based column with Span UDF #3421

Support group by span over time based column with Span UDF #3421

Conversation

songkant-aws commented Mar 13, 2025 • edited by LantaoJin Loading

Description

Related Issues

Check List

qianheng-aws Mar 14, 2025

Choose a reason for hiding this comment

songkant-aws Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

qianheng-aws Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

songkant-aws Mar 17, 2025

Choose a reason for hiding this comment

LantaoJin Mar 17, 2025

Choose a reason for hiding this comment

songkant-aws Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

songkant-aws Mar 18, 2025

Choose a reason for hiding this comment

LantaoJin Mar 17, 2025

Choose a reason for hiding this comment

songkant-aws Mar 17, 2025

Choose a reason for hiding this comment

LantaoJin commented Mar 18, 2025 • edited Loading

LantaoJin commented Mar 18, 2025

songkant-aws commented Mar 19, 2025

songkant-aws commented Mar 19, 2025

LantaoJin commented Mar 19, 2025

songkant-aws commented Mar 13, 2025 •

edited by LantaoJin

Loading

songkant-aws Mar 17, 2025 •

edited

Loading

qianheng-aws Mar 14, 2025 •

edited

Loading

songkant-aws Mar 17, 2025 •

edited

Loading

LantaoJin commented Mar 18, 2025 •

edited

Loading