-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Description
Search before asking
- I had searched in the feature and found no similar feature requirement.
Background
Following the merge of #10586, we propose to upgrade the current DataSource SPI to a more comprehensive Metadata SPI. This change will also integrate the improvements from #10402 to provide better metadata management capabilities.
Motivation
- Improve abstraction clarity between connector and metadata provider
- Align naming with actual functionality
- Enable better extensibility for external metadata services
- Consolidate metadata-related improvements from [Feature][seatunnel-api] Integrate Gravitino as metadata service for non-relational connectors #10402
- Provide a consistent configuration model
Proposal
1. Interface Upgrade
Current: DataSource interface
Proposed: MetaDataProvider interface
public interface MetaDataProvider extends AutoCloseable {
/**
* Returns the metadata provider type
* Example: "gravitino", "hive", "iceberg"
*/
String kind();
/**
* Initialize provider with configuration
*/
void init(Config config);
/**
* Retrieve table metadata by ID
*/
Optional<CatalogTable> getTable(CatalogTableId tableId);
/**
* Returns datasource mapper configurations
*/
Map<String, Object> dataSourceMappers();
/**
* Release resources
*/
void close();
}2. YAML Configuration Update
metadata:
enabled: true
kind: gravitino
gravitino:
uri: http://127.0.0.1:8090
metalake: test_metalake3. Field Name Optimizations
| Old Field Name | New Field Name | Rationale |
|---|---|---|
datasource_id |
metadata_datasource_id |
Clearly indicates that the identifier belongs to the metadata context rather than a connector datasource |
schema_url |
metadata_table_id |
Aligns with catalog-based metadata terminology |
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels