Skip to content

Add custom DDL parser - #174

Merged
ryannedolan merged 6 commits into
mainfrom
custom-ddl-parser
Nov 20, 2025
Merged

ryannedolan merged 6 commits into
mainfrom
custom-ddl-parser

Conversation

@ryannedolan

@ryannedolan ryannedolan commented Nov 8, 2025

Copy link
Copy Markdown
Collaborator

Summary

  • Add custom DDL parser impl.
  • Replace Calcite's server DDL parser with new custom one.
  • Bump Calcite version.

Details

The Calcite project has a bunch of machinery to generate parsers. It's not practical to replicate that machinery in this project. Instead, I've added a custom parser definition to Calcite, had it generate a parser impl, and then copy-pasted the resulting files here.

The new parser supports the following custom statements:

create materialized view foo
  refreshed 'cron-schedule'
  with (k v, ...)
  as select ...;
drop materialized view foo;

create trigger foo on bar as 'jobTemplate' in 'namespace'
  scheduled 'cron-schedule'
  with (k v, ...) ;
drop trigger foo;

create function foo as 'jobTemplate' in 'namespace'
  with (k v, ...);
drop function foo;

The new refreshed <cron>, scheduled <cron>, with (options), and in <namespace> clauses are optional. The with clause is intended for passing in options that pop out when rendering templates. For example with (foo 'bar') should result in {{foo}} expanding as bar. This could be used to dynamically enable/disable conditional templates, override defaults, etc.

The refreshed clause is intended for creating cron jobs that fire any triggers created as part of the resulting pipeline. Without specifying refreshed, any TableTriggers in the pipeline would only get fired by external event sources. With refreshed, we can drive those TableTriggers even without an external event source.

The scheduled clause likewise creates a cron job that drives user-defined triggers.

Testing Done

create trigger DDL is tested in #175:

> create or replace trigger bar on foo as 'my-app' in 'my-mp' scheduled '@daily' with (key 'value');

@ryannedolan
ryannedolan force-pushed the custom-ddl-parser branch 3 times, most recently from 7000a72 to e9adfc3 Compare November 8, 2025 18:45
// In lieu CREATE_TRIGGER, re-use similar CREATE_FUNCTION as operator kind.
private static final SqlOperator OPERATOR =
new SqlSpecialOperator("CREATE TRIGGER",
SqlKind.CREATE_FUNCTION);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intentional, or do we need to introduce a dedicated SqlKind for CREATE_TRIGGER?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an enum, so we can't introduce a new type without contributing it upstream to Calcite. Per their docs, we can use an UNKNOWN kind in the meantime, but triggers and functions look very similar so I figure we can use the same kind for both. Looking ahead, I'm planning to have triggers be zero-parameter functions under the hood, s.t. we can manually fire a trigger the same way you'd invoke a function. At that point, triggers would just be syntactic sugar for functions anyway.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, makes sense.

// In lieu DROP_TRIGGER, re-use similar DROP_FUNCTION as operator kind.
private static final SqlOperator OPERATOR =
new SqlSpecialOperator("DROP TRIGGER",
SqlKind.DROP_FUNCTION);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mismatched SqlKind here as well? Also would we need a DROP_FUNCTION DDL as well ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we'll need to implement DROP TRIGGER at some point.

@srnand srnand left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! (just need to include the template files)

@jogrogan jogrogan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on including template files and maybe outlining exactly how you got these files to generate in the README.

We should also figure out how to add testing for this kind of stuff to prevent regressions in the future.

@ryannedolan
ryannedolan enabled auto-merge (squash) November 20, 2025 18:24
@ryannedolan
ryannedolan merged commit 81de128 into main Nov 20, 2025
1 check passed
@ryannedolan
ryannedolan deleted the custom-ddl-parser branch November 20, 2025 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants