Proposal: Add storage_options to BasePath

[cmccabe]

Lance supports multi-base tables, in which objects are sharded across multiple base paths. Currently, base paths are defined as follows:

message BasePath {
  uint32 id = 1;
  optional string name = 2;
  bool is_dataset_root = 3;
  string path = 4;
}

Unfortunately, this is not enough information to uniquely identify base paths in several scenarios. For example, in Azure, a user may have several containers with the same name, that are in different storage accounts. In essence, the storage account is meaningfully a component of the location.

Another example is that a user may have some data in Cloudflare R2, and some data in Amazon S3. Both of them use the s3 storage provider, but with different "s3 endpoint URLs."

In order to address these scenarios, I propose to add a storage_options map to the protocol.

  map<string, string> storage_options = 5;

We could then add designated storage options to each base path for disambiguation purposes.

Note that this mechanism is not intended to store secrets, or options such as the number of retries. It is strictly intended for storage options which affect the identity of the path itself. In order to enforce this we could set up a list of allowed storage keys (the number is not that large).

[westonpace]

I wonder if we can maybe just call it properties or metadata so it is clear it isn't mean to be a 1:1 mapping with storage_options? Also we need to make sure to update https://lance.org/format/table/layout/?h=base#basepath-message. Anyways, I'm ok with +1 regardless of the above nit.

[wjones127]

question: is your goal just to have a place to put these kind of options or is your goal to have a standard place that all Lance implementations understand? For example, do you expect LanceDB, Spark, and Trino to all parse these options consistently?

If what you want is the former, then I think you could do this without a format change. We already have a place for table-level config here:

lance/protos/table.proto

Lines 172 to 177 in 7f6fd0e

	// Table config.
	//
	// Keys with the prefix "lance." are reserved for the Lance library. Other
	// libraries may wish to similarly prefix their configuration keys
	// appropriately.
	map<string, string> config = 16;

You could put some JSON in here under the key lancedb.storage_options, for example.

If you are aiming for consistency across engines, then I think we need to define a detailed spec for what options there are and how engines should interpret them. This is a lot more work, but I think it's necessary if you want to bake this into the table format spec. If we go this route, it's worth researching what different engines expect when it comes to configuring and mounting different storage systems.

[cmccabe]

You could put some JSON in here under the key lancedb.storage_options, for example.

If we did put the metadata into the existing map inside `Manifest, we would have to prefix it with the storage directory index so that we could understand which base it applied to. So, for example, we could have lance.storage_options.base0.azure_storage_account, lance.storage_options.base1.azure_storage_account, etc.

Another option that we previously discussed was using azure paths of type az://container@account/path to disambiguate azure paths using different storage accounts. This also would not require a format change (actually, it might not even require a code change). But it doesn't solve all the cases described above, just the Azure one.

If you are aiming for consistency across engines, then I think we need to define a detailed spec for what options there are and how engines should interpret them. This is a lot more work, but I think it's necessary if you want to bake this into the table format spec. If we go this route, it's worth researching what different engines expect when it comes to configuring and mounting different storage systems.

It seems to me like the spec here is actually controlled mostly by the storage providers (Microsoft, Amazon, etc. etc.) But I suppose we would have to write about how specific keys in the format mapped to specific object storage parameters, which would involve some work.

[wjones127]

It seems to me like the spec here is actually controlled mostly by the storage providers (Microsoft, Amazon, etc. etc.)

On some level, yes. For example, Azure has the storage account name. They define that property of their service. But in our spec we need to define exactly how that's stored. For example, is the key azure_storage_account or azureStorageAccount? Little details like that.

Another option that we previously discussed was using azure paths of type az://container@account/path to disambiguate azure paths using different storage accounts. This also would not require a format change (actually, it might not even require a code change). But it doesn't solve all the cases described above, just the Azure one.

I think this is seriously worth looking into. We should investigate how other table formats have solved this, as I imagine we aren't the first. For example, it looks like there's one system that uses ¹:

abfs[s]://{container-name}@{storage-account-name}.dfs.core.windows.net/{path}

Footnotes

1. https://gravitino.apache.org/docs/1.2.0/lakehouse-iceberg-catalog/ ↩

[jackye1995]

the @ notation is a pretty common one, for using Azure/OCI when accessing them from their S3 compatible endpoint

[westonpace]

On some level, yes. For example, Azure has the storage account name. They define that property of their service. But in our spec we need to define exactly how that's stored. For example, is the key azure_storage_account or azureStorageAccount? Little details like that.

I'm not sure I entirely agree.

• Azure can come up with whatever mechanisms they want for how Azure is accessed
• The Lance table format is responsible for defining how to describe a base
• The "Azure Plugin for Lance" is responsible for mapping between the two systems

So the decision to use azure_storage_account vs. azureStorageAccount seems like it belongs to the "Azure Plugin for Lance" and not part of the format itself. Though, I'll admit that the "Azure Plugin for Lance" (e.g. the implementation of the storage provider trait) is something we own, I do not think it's decisions affect the Lance table spec (from governance terms). I certainly don't think we need to put any Azure-specific information in the spec itself.

Another option that we previously discussed was using azure paths of type az://container@account/path to disambiguate azure paths using different storage accounts. This also would not require a format change (actually, it might not even require a code change). But it doesn't solve all the cases described above, just the Azure one.

I appreciate the abuse of authority 😜

On a more serious note, avoiding a spec change might be practical. URIs are quite flexible. If your goal was to get a string/string key=value mapping then you could also use query parameters which, combined with percent-encoding, should give you all the flexibility you need. So I think it is possible to solve all the cases the storage_options introduces without adding the concept of storage_options.

[jackye1995]

So, looks like at least we are good on Azure? Since Arrow object store already supports

az://container@account.blob.core.windows.net/path 

The different S3 endpoint story is interesting, but is rather hypothetical, given S3 users already typically have the habit of making the name globally unique anyway. But we should at least support per-base storage options to be supplied at runtime, that seems to be always beneficial.