Skip to content

[Improve][Api] Upgrade DataSource SPI to Metadata SPI #10640

@chl-wxp

Description

@chl-wxp

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Background

Following the merge of #10586, we propose to upgrade the current DataSource SPI to a more comprehensive Metadata SPI. This change will also integrate the improvements from #10402 to provide better metadata management capabilities.

Motivation

Proposal

1. Interface Upgrade

Current: DataSource interface
Proposed: MetaDataProvider interface

public interface MetaDataProvider extends AutoCloseable {

    /**
     * Returns the metadata provider type
     * Example: "gravitino", "hive", "iceberg"
     */
    String kind();

    /**
     * Initialize provider with configuration
     */
    void init(Config config);

    /**
     * Retrieve table metadata by ID
     */
    Optional<CatalogTable> getTable(CatalogTableId tableId);

    /**
     * Returns datasource mapper configurations
     */
    Map<String, Object> dataSourceMappers();

    /**
     * Release resources
     */
    void close();
}

2. YAML Configuration Update

metadata:
  enabled: true
  kind: gravitino
  gravitino:
    uri: http://127.0.0.1:8090
    metalake: test_metalake

3. Field Name Optimizations

Old Field Name New Field Name Rationale
datasource_id metadata_datasource_id Clearly indicates that the identifier belongs to the metadata context rather than a connector datasource
schema_url metadata_table_id Aligns with catalog-based metadata terminology

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions