[DISCUSS] Long-term architecture for Lance support in Gravitino

[FANNG1]

Background

Gravitino has started to support Lance through the Lance REST server and generic lakehouse Lance tables. Recent discussions around authorization, credential vending, and Lance path namespace compatibility show that we need to align on the long-term architecture before adding more features.

This discussion proposes a long-term direction for Lance support in Gravitino.

Overall Picture

In the long term, Gravitino should expose two major entry points:

Users / governance systems
  -> Gravitino REST / client
  -> Lance catalog

Compute engines
  -> Gravitino Lance REST server
  -> Lance REST Namespace API

The goals are:

• Expose a first-class Lance catalog to users.
• Expose a Lance REST server to engines such as Spark, Ray, and Daft.
• Reuse Gravitino authorization, credential vending, and catalog/table lifecycle capabilities.
• Move the current generic catalog Lance table capabilities into shared internal services.

Core Decision

The largest architectural decision is whether Gravitino should support non-Gravitino namespace backends for Lance.

This proposal assumes that we should support them. In that model, Gravitino Lance REST is not only a protocol adapter for Gravitino metadata. It is also an engine-facing Lance namespace gateway.

Supported metadata authorities:

path backend      -> Lance path namespace
rest backend      -> remote Lance REST namespace
gravitino backend -> Gravitino metadata

If we only support the Gravitino backend, the architecture is simpler:

Lance REST
  -> Gravitino internal dispatcher
  -> Gravitino Lance catalog metadata

But that would give up support for existing Lance path namespaces, remote Lance REST namespaces, and lightweight standalone Lance REST deployments.

Proposed Long-Term Model

Introduce a first-class Lance catalog provider:

provider = lance
namespace-backend = path | rest | gravitino

Ignoring nested namespaces for now, the object model can be viewed as:

metalake
  catalog
    namespace
      table

Each Lance catalog instance should choose exactly one namespace backend.

Backend Semantics

Path backend

• Metadata comes from Lance dataset metadata.
• Schema is read from the Lance dataset.
• Location is resolved from namespace root and table identifier.
• Lance REST can generate temporary credentials based on table location.

REST backend

• Metadata comes from a remote Lance REST Namespace service.
• Gravitino Lance REST acts as a gateway.
• Credential vending is delegated to the remote Lance REST service by default.

Gravitino backend

• Metadata is stored in Gravitino.
• Lance REST accesses metadata through Gravitino internal dispatchers.
• Authorization and credential configuration come from Gravitino.
• Supported only in auxiliary mode, not standalone mode.

Deployment Modes

auxiliary mode:
  supports path / rest / gravitino backends

standalone mode:
  supports path / rest backends
  does not support gravitino backend

Standalone Lance REST should not directly read the Gravitino entity store or act as a generic HTTP proxy for Gravitino metadata.

Authorization and Credential Vending

Metadata authorization should be table-oriented:

metalake / lance_catalog / namespace / table

When an engine accesses a Gravitino Lance catalog through Lance REST, authorization should follow a pattern similar to Iceberg REST:

Lance REST request
  -> resolve catalog / namespace / table
  -> build Gravitino request context and principal
  -> map Lance REST operation to Gravitino metadata privilege
  -> authorize the table under the Gravitino Lance catalog
  -> execute the metadata operation

Credential vending should be executed through the Lance REST pipeline:

engine
  -> Lance REST describeTable
  -> resolve table metadata and location
  -> authorize table/data access
  -> generate or fetch temporary credential
  -> return schema, location, storage_options

For the Gravitino backend, metadata authority and policy authority still come from Gravitino.

Role of Generic Catalog

The following should remain as a compatibility path, but not the long-term user-facing model:

provider = lakehouse-generic
table_format = lance

The main external path should be:

provider = lance

The existing Lance capabilities in the generic catalog should be moved into shared internal services and reused by both paths.

[yuqi1129]

@FANNG1

Quick observation + question before we go deeper into the three-backend split:

For all three backends (path / rest / gravitino), the ultimately reliable metadata is whatever you read from the dataset's location. Lance is path-self-describing — unlike Iceberg, it does not need an external metadata pointer to locate the schema; opening the path gives you schema, versions, fragments. And since applications can (and routinely do) reach the storage directly — Python notebooks doing lance.dataset("s3://..."), ETL jobs writing fragments, training pipelines opening shards — they will bypass Lance REST entirely. Any catalog-held metadata is at best a cache or projection of what the path already says. The location-resolved view is the universal fallback / reconciliation source in every backend.

The Gravitino Lance REST as it stands today already runs into this in several concrete ways:

1. No way to manage existing Lance tables. For a dataset already living at some path, today we can only register the path; there is no story for adopting the dataset as a first-class catalog table with its versions, schema, history surfaced through the catalog.
2. Multi-writer / out-of-band write drift. If a user writes directly to the dataset path (which Lance encourages), the Lance REST service's metadata view is not refreshed in time, so reads going through REST diverge from what the path actually holds.
3. Surrounding gaps. Authentication / authorization, governance (lineage, tags, owners), engine integration (Spark / Ray / Daft), and general usability all still need to be defined per backend.

Given the above, could you spell out the concrete primary use case for each backend — specifically what each one gives users that the other two do not, and how each handles the "an app wrote directly to the path and bypassed me" case? Would be good to have one paragraph per backend in the proposal that names the target user, the differentiator, and the bypass story.

[FANNG1]

Thanks @yuqi1129, this is a very good point.

I agree that whether Gravitino should support non-Gravitino namespace backends is the key decision here. I do not have a strong preference yet, and would like to hear more feedback from the community.

From my perspective, the main benefit of supporting path and rest backends is compatibility with existing Lance datasets and deployments.

If we support path, users can expose existing Lance datasets through Lance REST more easily. If we support rest, Gravitino Lance REST can integrate with an existing Lance REST namespace deployment.

However, from a platform/governance perspective, if a platform chooses to use a managed REST namespace, we probably should not allow or encourage users to access the same datasets through the path namespace. Path namespace requires users or engines to manage storage options, AK/SK, and storage-level configuration directly, which is hard to govern consistently.

So I think this is exactly the major tradeoff we need to align on first: whether Lance REST in Gravitino should be only a Gravitino-managed catalog endpoint, or also a gateway for existing path/rest Lance namespaces.

[FANNG1]

If we do not support non-Gravitino namespace backends for now, the architecture can be simplified around the existing Gravitino namespace backend.

1. Make the Gravitino backend auxiliary-only

   The Gravitino backend should only be supported when Lance REST runs as a Gravitino auxiliary service.

   A standalone Lance REST server talking to a remote Gravitino server over HTTP does not seem very useful for this mode, and it increases the system complexity. It introduces extra questions around authentication forwarding, double authorization, credential vending over HTTP, cache consistency, and failure boundaries.

   So for the Gravitino backend, the intended path should be:

   Engine
     -> Lance REST
     -> Gravitino internal dispatchers
     -> Gravitino Lance metadata
   

   We should not support:

   standalone Lance REST
     -> Gravitino HTTP client
     -> remote Gravitino server
   

2. Centralize metadata authorization in the Lance REST service

   Similar to the Gravitino Iceberg REST server, metadata authorization should be handled in the Gravitino Lance REST server.

   Lance REST resolves the Lance catalog / namespace / table from the request, builds the Gravitino request context and principal, maps the Lance REST operation to the corresponding Gravitino metadata privilege, and authorizes the table under the Gravitino Lance catalog.

   Since the Gravitino backend is auxiliary-only, there is no remote Gravitino server call and no double authorization problem.

   The metadata access path becomes:

   Lance REST request
     -> resolve catalog / namespace / table
     -> authorize against Gravitino Lance catalog objects
     -> call internal metadata dispatchers
   

3. Use internal credential vending instead of remote HTTP calls

   Credential vending for the Gravitino backend should use Gravitino internal credential managers / dispatchers.

   Lance REST is still the engine-facing execution point: it decides whether the Lance operation needs read or write credentials, asks the internal credential manager for credentials according to Gravitino policy, and translates the result into Lance storage_options.

   Lance REST
     -> internal credential manager
     -> temporary credentials
     -> Lance storage_options
   

4. Expose engine access through the Lance REST protocol

   External engines should access Lance tables through the Lance REST protocol. This is similar to the direction in [MINOR] docs: Add Spark REST catalog registration design #11280, where Spark can derive Lance REST catalog configuration from Gravitino catalog metadata.

   The user-facing model becomes:

   Users manage catalogs/tables in Gravitino
   Engines access them through Lance REST
   

5. Handle the empty schema problem in the Gravitino namespace

   Today, the Gravitino namespace path may not have a complete schema story for Lance REST.

   Short term, Lance REST can return only the table location if that is all the engine currently needs. The engine can open the Lance dataset from the location. For Gravitino REST APIs, Gravitino can probe the Lance dataset by version and return the schema.

   Long term, Gravitino should support the Lance REST commit protocol so that Gravitino can maintain the latest schema/version information as part of the Lance table metadata flow.

   So the short-term and long-term split is:

   short term:
     Lance REST returns location
     engine opens Lance dataset directly by location
     Gravitino REST probes schema by version
   
   long term:
     Lance REST commit protocol updates Gravitino-visible schema/version
   

[MehulBatra]

Feature request:
Add gravitino.lance-rest.gravitino-auth-type (simple/oauth2) to the Lance REST service, for parity with gravitino.iceberg-rest.gravitino-auth-type Without it, a standalone Lance REST server cannot authenticate to a secured Gravitino server which blocks multi-tenant deployments that run per-tenant standalone REST servers (separate namespaces for isolation) against a central secured hub. Iceberg REST already supports this; Lance REST should match.

Support for AuthN/Z for Lance Rest Catalog backed by Gravitino RBAC running lance as a auxillary.