feat(snowflake): support pushdown allow + using LIKE rather than exact match for query log optimization #13649
Conversation
github-actions
bot
added
the
ingestion
Codecov ReportAttention: Patch coverage is ✅ All tests successful. No failed tests found.
❌ Your patch check has failed because the patch coverage (13.33%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. 📢 Thoughts on this report? Let us know! |
datahub-cyborg
bot
added
the
needs-review
| deny_conditions = [] | ||
| for pattern in deny_usernames: | ||
| # Ensure pattern is uppercase for case-insensitive LIKE | ||
| deny_conditions.append(f"UPPER(user_name) NOT LIKE '{pattern.upper()}'") |
| default=[], | ||
| description="List of snowflake usernames (SQL LIKE patterns, e.g., 'ANALYST_%', '%_USER', 'MAIN_ACCOUNT') which WILL be considered for lineage/usage/queries extraction. " | ||
| "This is primarily useful for improving performance by filtering in only specific users. " | ||
| "If not specified, all users not in deny list are included.", |
There was a problem hiding this comment.
are the allow and deny config flags mutually exclusive? it's not super clear if you can use both together, or if that'd even make sense to do
There was a problem hiding this comment.
also should call out that these are case insensitive
There was a problem hiding this comment.
thanks- ill clarify
| if allow_conditions: | ||
| user_filters.append(f"({' OR '.join(allow_conditions)})") | ||
|
|
||
| users_filter_clause = " AND ".join(user_filters) if user_filters else "TRUE" |
There was a problem hiding this comment.
might make sense to have a _build_user_filter(field_name) -> str helper method, ideally with some more focused unit tests
datahub-cyborg
bot
added
pending-submitter-merge
and removed
needs-review
The
pushdown_deny_usernames, param gave us an effective strategy for reducing noise in audit log. However, it didn't have the full flexibility to completely optimize our query. This approach adds two new improvements - support for LIKE and the ability to write an allowlist rather than deny list.