feat: add ANN proto codecs and extract table_identifier module by LuQQiu · Pull Request #6503 · lance-format/lance

LuQQiu · 2026-04-13T23:49:59Z

Add protobuf encode/decode for ANNIvfSubIndexExec

codecov · 2026-04-14T00:22:09Z

Codecov Report

❌ Patch coverage is 89.97555% with 41 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/ann_proto.rs	90.57%	10 Missing and 16 partials ⚠️
rust/lance/src/io/exec/table_identifier.rs	90.83%	5 Missing and 6 partials ⚠️
rust/lance/src/io/exec/knn.rs	75.00%	3 Missing ⚠️
rust/lance/src/io/exec/filtered_read_proto.rs	0.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

BubbleCal · 2026-04-14T08:51:30Z

protos/ann.proto

+message ANNIvfSubIndexExecProto {
+  VectorQueryProto query = 1;
+  TableIdentifier table = 2;
+  repeated lance.table.IndexMetadata indices = 3;


do we need to serialize prefilter_source for this?

prefilter source is

Why this works: PreFilterSource is one of three variants: - FilteredRowIds(Arc<dyn ExecutionPlan>) — an execution plan child - ScalarIndexQuery(Arc<dyn ExecutionPlan>) — an execution plan child - None — no prefilter

When ANNIvfSubIndexExec has a prefilter, it reports the prefilter plan as a child in children() .
DataFusion's proto serialization framework automatically serializes/deserializes all children. So the codec receives the decoded children in the inputs slice and reconstructs the PreFilterSource from them. It doesn' need to be in the proto. @westonpace FYI

westonpace

A few nits but approve pending @BubbleCal 's observation re: prefilter source

Is the eventual goal to use PhysicalExtensionCodec?

westonpace · 2026-04-14T12:56:26Z

protos/ann.proto

+  float dist_q_c = 12;
+}
+
+message ANNIvfSubIndexExecProto {


Can we just add a brief comment that the purpose of this message is to serialize the ANNIvfSubIndexExec node? (Though that might be obvious from the name)

No problem, added

westonpace · 2026-04-14T13:02:01Z

protos/ann.proto

+// Serialized vector query parameters.
+message VectorQueryProto {
+  // Query vector as Arrow IPC bytes (supports Float16, Float32, Float64, UInt8, etc.)
+  bytes key_arrow_ipc = 1;


Can we do query_arrow_ipc or query_vector_arrow_ipc? We don't use the term key anywhere else that I know of.

changed to query_vector_arrow_ipc

westonpace · 2026-04-14T13:03:35Z

protos/ann.proto

+  optional uint32 refine_factor = 9;
+  // Distance metric type as string ("l2", "cosine", "dot", "hamming").
+  // Absent means None (use the index's default metric).
+  optional string metric_type = 10;


Can we use

lance/protos/index.proto

Line 152 in d37348d

enum VectorMetricType {

Change to use that
but because lance-index depends on lance-datafusion
protos/ann.proto cannot belongs to lance-datafusion
i have to move that to lance crate

westonpace · 2026-04-14T13:08:47Z

protos/ann.proto

+  // Absent means None (use the index's default metric).
+  optional string metric_type = 10;
+  bool use_index = 11;
+  float dist_q_c = 12;


Should this be optional?

change to optional

Double checked float dist_q_c is not optional in Query struct, callers send 0.0 explicitly

protos/ann.proto

westonpace · 2026-04-14T13:10:03Z

The Cargo.lock for python needs updated.

Add serialization support for ANNIvfPartitionExec and ANNIvfSubIndexExec execution plan nodes, enabling distributed ANN query execution. - Add ann_ivf.proto with VectorQueryProto, ANNIvfPartitionExecProto, ANNIvfSubIndexExecProto - Add knn_proto.rs with encode/decode for ANN IVF exec nodes - Extract table_identifier.rs from filtered_read_proto.rs for reuse - Add query/dataset/indices accessors on ANNIvfSubIndexExec Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Remove ANNIvfPartitionExecProto (will be handled separately) - Use lance.table.IndexMetadata directly via extern_path instead of raw bytes encoding - Move table_identifier tests from filtered_read_proto to table_identifier module Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Rename test_array_ipc_roundtrip_* to test_query_key_ipc_roundtrip_* - Move resolve_dataset to table_identifier.rs, use from both ann_proto.rs and filtered_read_proto.rs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…Type enum Address PR review feedback: - Move ann.proto from lance-datafusion to lance crate (package lance.pb) so it can reference types from lance-index without circular deps - Use lance.index.pb.VectorMetricType enum instead of string for metric_type - Rename key_arrow_ipc to query_vector_arrow_ipc - Make minimum_nprobes and dist_q_c optional - Add comment about prefilter_source handled by DataFusion child serialization - Add comment explaining ANNIvfSubIndexExecProto purpose Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ery_key → query_vector - Remove extern_path and lance-table dep from lance-datafusion (only needed when ann.proto was compiled there) - Rename query_key_to/from_ipc_bytes → query_vector_to/from_ipc_bytes to match the proto field name query_vector_arrow_ipc Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…t to 1 - use_index and dist_q_c are required values (no centralized default), keep them as bare proto fields - minimum_nprobes: optional with unwrap_or(1) matching documented default Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Needed by distributed execution codecs to inspect the prefilter source type when rebuilding plans for remote PEs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

LuQQiu · 2026-04-14T21:11:33Z

@westonpace @BubbleCal resolve the comments, please take a look

github-actions bot added the enhancement New feature or request label Apr 13, 2026

LuQQiu requested review from BubbleCal and westonpace April 13, 2026 23:50

BubbleCal reviewed Apr 14, 2026

View reviewed changes

westonpace approved these changes Apr 14, 2026

View reviewed changes

github-actions bot added the python label Apr 14, 2026

LuQQiu and others added 8 commits April 14, 2026 10:19

Rename ann_ivf.proto to ann.proto and knn_proto.rs to ann_proto.rs

77c9aa1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rename array IPC helpers to query_key_to/from_ipc_bytes

93c8a42

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: rustfmt formatting — alphabetize modules, condense chain expression

2f834b7

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

LuQQiu force-pushed the lu/ann branch from 475a6c2 to eb00681 Compare April 14, 2026 17:27

LuQQiu and others added 5 commits April 14, 2026 11:13

fix: update python Cargo.lock for lance-datafusion dep change

3e1d3b5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: restore main's Cargo.lock to avoid yanked core2 resolution failure

d32039b

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: make dist_q_c optional with default 0.0, keep use_index required

267f06c

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat: add prefilter_source() accessor to ANNIvfSubIndexExec

ec6dc00

Needed by distributed execution codecs to inspect the prefilter source type when rebuilding plans for remote PEs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Conversation

LuQQiu commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

westonpace commented Apr 14, 2026

Uh oh!

LuQQiu commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LuQQiu commented Apr 13, 2026 •

edited

Loading

codecov bot commented Apr 14, 2026 •

edited

Loading