Add a framework to validate each of the Ingestion Transformation functions #17039
+86
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a validation framework for Pinot transform functions used in ingestion configs. It provides validation hooks for datatype checks in the TransformFunction interface that individual functions can implement to validate their configurations during table creation.
We can include validationMode in the transform function specification:
Purpose: Backward compatibility - allows all existing type conversions
Behavior: No validation, accepts everything (current Pinot behavior)
Use Case: Existing tables that shouldn't break
Purpose: Safe type conversions allowed
Behavior: Allows safe conversions like INT→LONG, FLOAT→DOUBLE, but blocks unsafe ones like STRING→INT
Use Case: New tables that want some safety but flexibility
Purpose: Maximum type safety
Behavior: No automatic type conversions, exact type matching required
Use Case: Critical tables where type safety is paramount