RFC: Lance C/C++ Library (liblance)

[jja725]

RFC: Lance C++ Library (liblance)

Summary

This RFC proposes creating a first-class C/C++ library for Lance (liblance) that exposes the full Lance feature set through a stable C-ABI with idiomatic C++ wrapper headers. This enables native integration with C++ query engines (Velox, DuckDB), ML frameworks, and any language with C FFI capabilities.

Motivation

Lance currently provides bindings for Python (via PyO3) and Java (via JNI). Both are mature and production-grade, but the lack of a C/C++ interface creates several gaps:

1. Query engine integration: Velox, DuckDB, DataFusion (C++), and other C++ query engines cannot natively read Lance datasets. The lance-duckdb extension already implements a project-specific C-ABI to bridge this gap, but it would be better to have a unified interface for all applications.

2. Ecosystem reach: A C-ABI library is the universal FFI target. Languages like Go, Ruby, Swift, Zig, and others can bind to it without project-specific bridges.

3. ML infrastructure: Many ML serving systems (TensorRT, ONNX Runtime, TFServing) and feature stores are C++-native. A Lance C++ library enables direct integration without Python overhead.

4. Velox integration: Velox is Meta's open-source C++ vectorized execution engine. A Lance C++ library would enable building a Velox connector for Lance, unlocking Lance as a data source for Velox-powered query workloads — including vector search capabilities that Parquet/ORC cannot provide.

Design Principles

1. C-ABI first, C++ wrappers second. The core interface is extern "C" functions callable from any language. Idiomatic C++ wrappers (RAII, smart pointers, iterators) are a thin layer on top.

2. Arrow C Data Interface for data exchange. All data crosses the FFI boundary as ArrowArray/ArrowSchema/ArrowArrayStream structs — the industry-standard zero-copy interchange format. No custom serialization.

3. Opaque handle pattern. All Rust objects are exposed as void* opaque pointers with explicit lifecycle functions (lance_*_open / lance_*_close). No struct layout leaks across the boundary.

4. Feature parity with Python/Java. The C API should expose the same capabilities available in the Python and Java bindings: dataset CRUD, scanning with pushdowns, vector/scalar indexing, versioning, and merge-insert.

5. Panic safety. The Rust crate compiles with panic = "abort" to prevent undefined behavior from unwinding across FFI boundaries.

6. Dual threading model. Expose both synchronous (block_on) and non-blocking (poll + waker) APIs. Simple integrators and scripts use the blocking API; high-performance query engines (Velox, Presto) use the poll-based API to avoid thread starvation.

Architecture

┌─────────────────────────────────────────────────────────┐
│  C++ Applications / Query Engines                       │
│  (Velox, DuckDB, custom apps)                           │
├─────────────────────────────────────────────────────────┤
│  C++ Header-Only Wrappers (lance.hpp)        [optional] │
│  RAII handles, iterators, builder pattern                │
├─────────────────────────────────────────────────────────┤
│  C Header (lance.h)                          [stable]   │
│  extern "C" functions, opaque handles, Arrow C Data     │
├─────────────────────────────────────────────────────────┤
│  lance-c crate (Rust)                        [new]      │
│  Tokio runtime, sync + async FFI exports                 │
├─────────────────────────────────────────────────────────┤
│  Existing Lance Rust crates                             │
│  (lance, lance-core, lance-index, lance-io, etc.)       │
└─────────────────────────────────────────────────────────┘

Crate Structure

A new rust/lance-c/ crate is added to the workspace:

rust/lance-c/
├── Cargo.toml          # crate-type = ["cdylib", "staticlib"]
├── cbindgen.toml       # Auto-generate lance.h from Rust
├── build.rs            # cbindgen header generation, cc crate for C++ tests
├── include/
│   ├── lance.h         # Generated C header (stable ABI)
│   └── lance.hpp       # C++ RAII wrappers (header-only)
├── src/
│   ├── lib.rs          # Module root, Tokio runtime init
│   ├── error.rs        # Thread-local error handling
│   ├── dataset.rs      # Dataset open/write/close
│   ├── scanner.rs      # Scan creation and iteration
│   ├── index.rs        # Index create/drop/list
│   ├── fragment.rs     # Fragment-level operations
│   ├── schema.rs       # Schema inspection
│   ├── writer.rs       # Write path (append/overwrite)
│   ├── merge_insert.rs # Merge-insert / upsert
│   ├── version.rs      # Version management
│   └── arrow_bridge.rs # Arrow C Data Interface export
└── tests/
    └── c_api_test.cpp  # C/C++ integration tests (compiled via cc crate)

Build Outputs

Output	Description
liblance.so / liblance.dylib / lance.dll	Shared library (for dynamic linking)
liblance.a / lance.lib	Static library (for embedding, e.g., DuckDB extension)
lance.h	C header (auto-generated via cbindgen)
lance.hpp	C++ wrapper header (hand-written, header-only)
lance.pc	pkg-config file for downstream C/C++ consumers

Proposed API

Error Handling

Following the pattern proven in lance-duckdb, errors use thread-local storage:

// Error codes
typedef enum {
    LANCE_OK = 0,
    LANCE_ERR_INVALID_ARGUMENT = 1,
    LANCE_ERR_IO = 2,
    LANCE_ERR_NOT_FOUND = 3,
    LANCE_ERR_DATASET_ALREADY_EXISTS = 4,
    LANCE_ERR_INDEX = 5,
    LANCE_ERR_INTERNAL = 6,
    // ...
} LanceErrorCode;

// Check last error after any function returns non-zero
LanceErrorCode lance_last_error_code(void);
const char    *lance_last_error_message(void);  // caller must free
void           lance_free_string(const char *s);

Dataset Operations

// Opaque handles
typedef struct LanceDataset LanceDataset;
typedef struct LanceScanner  LanceScanner;
typedef struct LanceWriter   LanceWriter;
typedef struct LanceBatch    LanceBatch;
typedef struct LanceSchema   LanceSchema;
typedef struct LanceIndex    LanceIndex;
typedef struct LanceVersion  LanceVersion;

// --- Dataset lifecycle ---

// Open a dataset at the given URI (file://, s3://, az://, gs://)
// storage_options: NULL-terminated key-value pairs ["key1", "val1", "key2", "val2", NULL]
// Returns NULL on error (check lance_last_error_*).
LanceDataset *lance_dataset_open(
    const char *uri,
    const char *const *storage_options,  // nullable
    uint64_t version                     // 0 = latest
);

void lance_dataset_close(LanceDataset *dataset);

// Metadata
uint64_t lance_dataset_count_rows(LanceDataset *dataset);
uint64_t lance_dataset_version(LanceDataset *dataset);
uint64_t lance_dataset_latest_version(LanceDataset *dataset);

// Schema (returned as Arrow C Data Interface)
int32_t lance_dataset_schema(LanceDataset *dataset, struct ArrowSchema *out);

Scanning / Reading

// --- Scanner builder ---

// Create a scanner with optional column projection and filter.
// columns: NULL-terminated array of column names (NULL = all columns)
// filter:  SQL filter expression string (NULL = no filter)
LanceScanner *lance_scanner_new(
    LanceDataset *dataset,
    const char *const *columns,   // nullable
    const char *filter            // nullable, SQL expression
);

// Scanner configuration (builder pattern — call before iterating)
int32_t lance_scanner_set_limit(LanceScanner *scanner, int64_t limit);
int32_t lance_scanner_set_offset(LanceScanner *scanner, int64_t offset);
int32_t lance_scanner_set_batch_size(LanceScanner *scanner, int64_t batch_size);
int32_t lance_scanner_set_fragment_ids(LanceScanner *scanner, const uint64_t *ids, size_t len);
int32_t lance_scanner_with_row_id(LanceScanner *scanner, bool enable);

// Nearest-neighbor vector search
int32_t lance_scanner_nearest(
    LanceScanner *scanner,
    const char *column,
    const float *query_vector,
    size_t dimension,
    uint32_t k,
    const char *metric,           // "L2", "cosine", "dot"
    uint32_t nprobes              // 0 = default
);

// Full-text search
int32_t lance_scanner_full_text_search(
    LanceScanner *scanner,
    const char *column,
    const char *query
);

// --- Iteration ---

// Open the scan as an Arrow C Data stream.
// This is the preferred API for simple integrations — blocks the calling thread.
int32_t lance_scanner_to_arrow_stream(
    LanceScanner *scanner,
    struct ArrowArrayStream *out
);

// Synchronous (blocking) batch iteration.
// Blocks the calling thread until data is available.
// Returns: 0 = batch available, 1 = end of stream, -1 = error
int32_t lance_scanner_next(LanceScanner *scanner, LanceBatch **out);

// --- Non-blocking (poll + waker) iteration ---
// Preferred for high-performance query engines (Velox, Presto) that
// must not block worker threads on I/O.

typedef enum {
    LANCE_POLL_READY    =  0,  // Batch available in *out
    LANCE_POLL_PENDING  =  1,  // I/O in progress; waker will fire when ready
    LANCE_POLL_FINISHED =  2,  // End of stream
    LANCE_POLL_ERROR    = -1,  // Error (check lance_last_error_*)
} LancePollStatus;

// Waker callback: called when data becomes available (may be called from any thread).
// Contract:
//   - Must be thread-safe (may fire on a Tokio worker thread or inline during poll_next)
//   - Single-use: fires exactly once per poll_next call that returns PENDING
//   - Must NOT call back into any lance_scanner_* function (will deadlock)
//   - Must tolerate being called after the scanner is closed (stale waker)
//   - Caller must tolerate spurious wakes (re-poll may return PENDING again)
// Typical use: signal a folly::Promise, post to an event loop, or set an atomic flag.
typedef void (*LanceWaker)(void *ctx);

// Poll for the next batch without blocking.
// - If data is already buffered, returns LANCE_POLL_READY immediately (zero overhead).
// - If I/O is needed, returns LANCE_POLL_PENDING and schedules the waker callback
//   on the Tokio runtime. The caller should yield the thread and re-poll after
//   the waker fires.
// - The waker is single-use: it fires at most once per poll call that returns PENDING.
// - A fresh waker must be provided on each poll_next call.
LancePollStatus lance_scanner_poll_next(
    LanceScanner *scanner,
    LanceWaker waker,        // called when data becomes available
    void *waker_ctx,         // opaque context passed back to waker
    LanceBatch **out         // populated when READY is returned
);

// Common helpers for both paths
int32_t lance_batch_to_arrow(LanceBatch *batch, struct ArrowArray *out_array, struct ArrowSchema *out_schema);
void    lance_batch_free(LanceBatch *batch);

void lance_scanner_close(LanceScanner *scanner);

// Random access
int32_t lance_dataset_take(
    LanceDataset *dataset,
    const uint64_t *indices,
    size_t num_indices,
    const char *const *columns,   // nullable
    struct ArrowArrayStream *out
);

Writing

typedef enum {
    LANCE_WRITE_CREATE    = 0,
    LANCE_WRITE_APPEND    = 1,
    LANCE_WRITE_OVERWRITE = 2,
} LanceWriteMode;

// Write from an Arrow stream
int32_t lance_dataset_write(
    const char *uri,
    struct ArrowArrayStream *data,
    LanceWriteMode mode,
    const char *const *storage_options,  // nullable
    LanceDataset **out_dataset           // nullable, returns opened dataset
);

// Incremental writer (for batch-at-a-time writing)
LanceWriter *lance_writer_open(
    const char *uri,
    struct ArrowSchema *schema,
    LanceWriteMode mode,
    const char *const *storage_options   // nullable
);
int32_t lance_writer_write_batch(LanceWriter *writer, struct ArrowArray *batch, struct ArrowSchema *schema);
int32_t lance_writer_finish(LanceWriter *writer, LanceDataset **out_dataset);
void    lance_writer_abort(LanceWriter *writer);

Data Modification

// Delete rows matching a SQL predicate
int32_t lance_dataset_delete(LanceDataset *dataset, const char *predicate);

// Update rows
int32_t lance_dataset_update(
    LanceDataset *dataset,
    const char *const *set_expressions,  // ["col1 = val1", "col2 = val2", NULL]
    const char *where_clause             // nullable
);

// Merge-insert (upsert)
typedef struct LanceMergeInsertBuilder LanceMergeInsertBuilder;

LanceMergeInsertBuilder *lance_merge_insert_new(
    LanceDataset *dataset,
    const char *on_column
);
int32_t lance_merge_insert_when_matched_update_all(LanceMergeInsertBuilder *builder);
int32_t lance_merge_insert_when_not_matched_insert_all(LanceMergeInsertBuilder *builder);
int32_t lance_merge_insert_when_not_matched_by_source_delete(LanceMergeInsertBuilder *builder);
int32_t lance_merge_insert_execute(
    LanceMergeInsertBuilder *builder,
    struct ArrowArrayStream *new_data,
    LanceDataset **out_dataset
);
void lance_merge_insert_free(LanceMergeInsertBuilder *builder);

Indexing

// Vector index
int32_t lance_dataset_create_vector_index(
    LanceDataset *dataset,
    const char *column,
    const char *index_type,    // "IVF_PQ", "IVF_FLAT", "IVF_SQ", "HNSW_PQ", "HNSW_SQ"
    const char *metric,        // "L2", "cosine", "dot"
    uint32_t num_partitions,   // 0 = auto
    uint32_t num_sub_vectors,  // 0 = auto
    bool replace
);

// Scalar index
int32_t lance_dataset_create_scalar_index(
    LanceDataset *dataset,
    const char *column,
    const char *index_type     // "BTREE", "BITMAP", "INVERTED", "LABEL_LIST"
);

// List indices (returns JSON for simplicity; or a dedicated struct array)
int32_t lance_dataset_list_indices(
    LanceDataset *dataset,
    char **out_json            // caller must free with lance_free_string
);

int32_t lance_dataset_drop_index(LanceDataset *dataset, const char *index_name);

Versioning

// Version management
int32_t lance_dataset_checkout(LanceDataset *dataset, uint64_t version);
int32_t lance_dataset_restore(LanceDataset *dataset, uint64_t version);

// List versions (returns JSON array)
int32_t lance_dataset_list_versions(
    LanceDataset *dataset,
    char **out_json
);

Schema Evolution

// Add columns via SQL expressions
int32_t lance_dataset_add_columns(
    LanceDataset *dataset,
    const char *const *expressions,  // ["new_col = old_col * 2", NULL]
    struct ArrowArrayStream *data    // alternative: provide data directly (nullable)
);

int32_t lance_dataset_drop_columns(
    LanceDataset *dataset,
    const char *const *columns       // NULL-terminated
);

int32_t lance_dataset_alter_columns(
    LanceDataset *dataset,
    const char *alterations_json     // JSON array of {path, name?, nullable?, data_type?}
);

Optimization

// Compact fragments
int32_t lance_dataset_compact(LanceDataset *dataset);

// Clean up old versions
int32_t lance_dataset_cleanup_old_versions(
    LanceDataset *dataset,
    uint64_t older_than_days
);

C++ Wrapper Design (lance.hpp)

The C++ wrapper is a header-only library providing RAII semantics, iterators, and builder patterns:

#include "lance.h"
#include <memory>
#include <string>
#include <vector>
#include <optional>
#include <stdexcept>
#include <arrow/c/bridge.h>  // Arrow C++ <-> C Data Interface

namespace lance {

class LanceError : public std::runtime_error {
public:
    LanceErrorCode code;
    LanceError(LanceErrorCode code, const char* msg)
        : std::runtime_error(msg), code(code) {}
};

// Check last error and throw if non-OK
inline void check_error() {
    auto code = lance_last_error_code();
    if (code != LANCE_OK) {
        auto msg = lance_last_error_message();
        std::string message(msg ? msg : "Unknown error");
        if (msg) lance_free_string(msg);
        throw LanceError(code, message.c_str());
    }
}

// RAII handle wrapper
template<typename T, void(*Deleter)(T*)>
class Handle {
    T* ptr_;
public:
    explicit Handle(T* ptr) : ptr_(ptr) {}
    ~Handle() { if (ptr_) Deleter(ptr_); }
    Handle(Handle&& o) noexcept : ptr_(o.ptr_) { o.ptr_ = nullptr; }
    Handle& operator=(Handle&& o) noexcept {
        if (ptr_) Deleter(ptr_);
        ptr_ = o.ptr_; o.ptr_ = nullptr;
        return *this;
    }
    Handle(const Handle&) = delete;
    Handle& operator=(const Handle&) = delete;
    T* get() const { return ptr_; }
    T* release() { auto p = ptr_; ptr_ = nullptr; return p; }
    explicit operator bool() const { return ptr_ != nullptr; }
};

using DatasetHandle = Handle<LanceDataset, lance_dataset_close>;
using ScannerHandle = Handle<LanceScanner, lance_scanner_close>;

class Scanner;

class Dataset {
    DatasetHandle handle_;
public:
    // Open a dataset
    static Dataset open(
        const std::string& uri,
        const std::vector<std::pair<std::string, std::string>>& storage_options = {},
        uint64_t version = 0
    ) {
        // Convert storage_options to C format ...
        auto* ds = lance_dataset_open(uri.c_str(), /* ... */ nullptr, version);
        if (!ds) { check_error(); }
        return Dataset(DatasetHandle(ds));
    }

    uint64_t count_rows() const { return lance_dataset_count_rows(handle_.get()); }
    uint64_t version() const { return lance_dataset_version(handle_.get()); }

    // Create a scanner with builder pattern
    Scanner scan() const;

    // Write data
    static Dataset write(
        const std::string& uri,
        ArrowArrayStream* data,
        LanceWriteMode mode = LANCE_WRITE_CREATE
    );

    // Delete rows
    void delete_rows(const std::string& predicate) {
        if (lance_dataset_delete(handle_.get(), predicate.c_str()) != 0)
            check_error();
    }

    // Index creation
    void create_vector_index(
        const std::string& column,
        const std::string& index_type = "IVF_PQ",
        const std::string& metric = "L2",
        uint32_t num_partitions = 0,
        uint32_t num_sub_vectors = 0
    );

private:
    explicit Dataset(DatasetHandle h) : handle_(std::move(h)) {}
};

class Scanner {
    ScannerHandle handle_;
public:
    explicit Scanner(LanceScanner* s) : handle_(ScannerHandle(s)) {}

    Scanner& columns(const std::vector<std::string>& cols) { /* ... */ return *this; }
    Scanner& filter(const std::string& expr) { /* ... */ return *this; }
    Scanner& limit(int64_t n) { lance_scanner_set_limit(handle_.get(), n); return *this; }
    Scanner& offset(int64_t n) { lance_scanner_set_offset(handle_.get(), n); return *this; }

    Scanner& nearest(
        const std::string& column,
        const std::vector<float>& query,
        uint32_t k,
        const std::string& metric = "L2"
    ) {
        lance_scanner_nearest(
            handle_.get(), column.c_str(),
            query.data(), query.size(), k,
            metric.c_str(), 0
        );
        return *this;
    }

    // Get results as ArrowArrayStream (zero-copy)
    void to_arrow_stream(ArrowArrayStream* out) {
        if (lance_scanner_to_arrow_stream(handle_.get(), out) != 0)
            check_error();
    }
};

} // namespace lance

Threading Model

The Problem with block_on()

Both the Java JNI bindings and the lance-duckdb extension use a simple block_on() approach: a global Tokio runtime is initialized once, and every async Lance operation is called via runtime.block_on(future), which blocks the calling thread until the I/O completes.

This works acceptably for DuckDB because it achieves parallelism at the fragment level — each DuckDB worker thread gets its own stream over a different fragment, so multiple block_on() calls run concurrently on different threads. However, within a single stream, the calling thread sits idle waiting for I/O.

For high-performance engines like Velox or Presto, this creates thread starvation: engine worker threads block on Lance I/O instead of yielding to process other tasks. This wastes thread pool capacity and can cascade into pipeline stalls.

Dual API: Sync + Poll

The C library exposes both APIs. The right choice depends on the integrator:

API	When to use	Example consumer
lance_scanner_next() (blocking)	Simple tools, scripts, single-threaded apps	CLI tools, DuckDB, batch jobs
lance_scanner_poll_next() (non-blocking)	Async engines with their own task scheduler	Velox (folly), Presto, custom async runtimes
lance_scanner_to_arrow_stream() (blocking)	Standard Arrow consumers	Any ArrowArrayStream-compatible code

The blocking API is a thin wrapper over the poll API internally — both share the same Tokio-backed stream. This avoids maintaining two separate code paths.

How Poll + Waker Works

The poll-based API mirrors Rust's Future::poll semantics across the FFI boundary:

C++ engine thread                     Rust / Tokio runtime
─────────────────                     ────────────────────

1. poll_next(waker, ctx, &batch)
   ──────────────────────────────►
                                      Check internal buffer:

   Case A: Data buffered              ◄── batch already in memory
   ◄── returns LANCE_POLL_READY ──
   Process batch, poll again.

   Case B: I/O needed                 ◄── no data ready
   ◄── returns LANCE_POLL_PENDING ─
   Yield thread to other work.
                                      Tokio spawns async I/O...
                                      ... I/O completes ...
                                      waker(ctx)  ──────────►  Engine reschedules task

2. poll_next(waker, ctx, &batch)      ◄── data now buffered
   ◄── returns LANCE_POLL_READY ──
   Process batch.

   ...

N. poll_next(waker, ctx, &batch)
   ◄── returns LANCE_POLL_FINISHED ─
   Stream exhausted.

Internal Design

On the Rust side, lance_scanner_poll_next wraps a Tokio RecordBatchStream behind a two-state machine:

• Idle: The stream is ready to be polled. The call spawns a Tokio task to fetch the next batch and transitions to Polling.
• Polling: A Tokio task is in flight. If the task has completed, the result is returned immediately (READY or FINISHED). If not, the waker is registered and PENDING is returned.

The waker notification uses a shared signaling mechanism (e.g., Arc<Notify> or oneshot channel) between the I/O task and a lightweight waker task. When the I/O task completes, it signals the waker task, which calls the C function pointer. This avoids the mistake of trying to share a JoinHandle across two tasks.

Waker Contract

The waker callback has strict thread-safety requirements that must be documented in lance.h:

1. Thread-safety: The waker may be called from any thread (a Tokio worker thread, or potentially inline during poll_next if data is immediately available). The callback implementation must be thread-safe.

2. Single-use: Each poll_next call that returns LANCE_POLL_PENDING will fire the waker exactly once. The caller must provide a fresh waker on each poll_next call.

3. No reentrancy: The waker callback must NOT call back into any lance_scanner_* function. Doing so will deadlock. Typical implementations should signal a promise, post to an event loop, or set an atomic flag.

4. Stale waker safety: If the scanner is closed while a waker is pending, the waker may still fire. The callback must tolerate being called after the scanner has been destroyed (e.g., by checking a validity flag in the context).

5. Spurious wake tolerance: The caller must tolerate re-polling after a waker fires and receiving LANCE_POLL_PENDING again. This can happen if the Tokio task was cancelled or if the I/O result was consumed by another path. The correct response is simply to re-register a new waker.

Implementation Plan

Phase 1: Core Read Path + C++ Wrappers (MVP)

Goal: A usable C++ library for reading Lance datasets, sufficient to build a Velox connector or integrate with any C++ application.

Component	Description
Infrastructure
lance-c crate scaffold	Cargo.toml, cbindgen, Tokio runtime
Error handling	Thread-local error codes/messages
C header generation	cbindgen-based lance.h
C API — Read Path
Dataset open/close	URI + storage options + version
Schema export	Arrow C Data Interface
Scanner builder	Column projection, SQL filters, limit/offset, batch size
ArrowArrayStream export	Blocking stream for simple consumers
Blocking batch iteration	lance_scanner_next() for simple integrations
Poll + waker iteration	lance_scanner_poll_next() for async engines (Velox, Presto)
Random access (take)	Index-based row retrieval
C++ Wrappers
lance.hpp header-only lib	RAII handles (Dataset, Scanner), exception-based error handling
Builder pattern for Scanner	Fluent .columns().filter().limit().nearest() API
Arrow C++ bridge helpers	Convenience functions for arrow::RecordBatch ↔ C Data Interface

Deliverable: liblance.so + lance.h + lance.hpp + lance.pc. A C++ developer can open a dataset, scan with filters/projections, and consume Arrow batches. Everything builds with cargo build.

Phase 2: Vector Search & Indexing

Component	Description
Vector search via scanner	nearest() with metric, k, nprobes
Full-text search	FTS query through scanner
Vector index creation	IVF_PQ, IVF_FLAT, IVF_SQ, HNSW variants
Scalar index creation	BTree, Bitmap, Inverted
Index listing/dropping	Management operations
C++ wrappers for indexing	Dataset::create_vector_index(), Dataset::create_scalar_index()

Deliverable: Vector search and index management from C++. Enables Velox IndexSource for ANN lookups.

Phase 3: Write Path & Mutations

Component	Description
Dataset write	Create / append / overwrite from ArrowArrayStream
Incremental writer	Batch-at-a-time writing with C++ Writer RAII class
Delete	Predicate-based deletion
Update	Expression-based updates
Merge-insert	Upsert operations with C++ builder
Schema evolution	Add/drop/alter columns
Version management	Checkout, restore, list versions

Deliverable: Full read-write C++ library. Enables Velox DataSink.

Phase 4: Advanced Features

Component	Description
Fragment-level access	Split-level parallelism for Velox
Compaction	Fragment consolidation
Statistics export	Row counts, column stats for query planning
Cloud storage (S3/GCS/Azure)	Full cloud backend support
Package distribution	vcpkg / Conan recipes

Deliverable: Production-ready library with optimization and distribution support.

Comparison with Existing Bindings

Capability	Python (PyO3)	Java (JNI)	C/C++ (proposed)
Dataset open/close	Yes	Yes	Phase 1
Scan with filters	Yes	Yes	Phase 1
C++ RAII wrappers	N/A	N/A	Phase 1
Vector search	Yes	Yes	Phase 2
Full-text search	Yes	Yes	Phase 2
Vector index creation	Yes	Yes	Phase 2
Scalar index creation	Yes	Yes	Phase 2
Write (create/append/overwrite)	Yes	Yes	Phase 3
Delete / Update	Yes	Yes	Phase 3
Merge-insert	Yes	Yes	Phase 3
Schema evolution	Yes	Yes	Phase 3
Version management	Yes	Yes	Phase 3
Fragment-level access	Yes	Yes	Future
Compaction	Yes	Yes	Future
Cloud storage (S3/GCS/Azure)	Yes	Yes	Future
Namespace / catalog	Yes	Yes	Future

Lessons from lance-duckdb

The lance-duckdb extension validates the core architectural decisions:

1. Static library linking works well. DuckDB bundles liblance_duckdb_ffi.a directly. We should support both static and shared library builds.

2. Arrow C Data Interface is the right data exchange format. Zero-copy, no custom serialization, works with both DuckDB and Velox.

3. Thread-local error handling is simple and effective. Avoids complex return types while remaining thread-safe.

4. Custom binary IR for filter pushdown is powerful but complex. lance-duckdb serializes DuckDB expressions into a LFT1 binary format and deserializes them into DataFusion expressions on the Rust side. For the general-purpose C library, we should start with SQL string filters (simpler, already supported by Lance) and add binary IR as an optional optimization later.

5. block_on() works but has limitations. Both lance-duckdb and the Java bindings use a global Tokio runtime with OnceLock and block_on(). This is simple and correct, and DuckDB mitigates the blocking cost through fragment-level parallelism (each worker thread gets its own stream). However, for cooperative async engines like Velox, blocking is a thread starvation risk. The C library should support both block_on() (for simplicity) and a poll-based API (for performance).

6. panic = "abort" is mandatory. Prevents undefined behavior from Rust panics unwinding into C/C++ frames.

Lessons from Python and Java Bindings

The existing bindings inform several design decisions:

1. Builder pattern for scanners. Both Python (ScannerBuilder) and Java (ScanOptions) use builders. The C API follows suit with lance_scanner_new() + lance_scanner_set_*() functions.

2. block_on() is a starting point, not a long-term design. Java's BlockingDataset and BlockingScanner were implemented as expedient shortcuts. For a general-purpose C library targeting high-performance engines, we go further with a poll + waker API that avoids blocking engine threads on I/O.

3. Arrow for data exchange. Python uses PyArrow arrays; Java uses Arrow C Data Interface structs. The C library should exclusively use Arrow C Data Interface.

4. Comprehensive feature surface. Both bindings expose dataset CRUD, scanning, indexing, versioning, merge-insert, schema evolution, and compaction. The C library should aim for parity.

5. Storage options as key-value pairs. Both bindings accept HashMap<String, String> for cloud storage configuration. The C API uses NULL-terminated const char** arrays.

Alternatives Considered

1. Wrap lance-duckdb's FFI

lance-duckdb already has a Rust FFI layer. We could extract and generalize it.

Rejected because: lance-duckdb's FFI is tightly coupled to DuckDB's execution model (custom IR formats, DuckDB-specific scan patterns). A general-purpose library needs a cleaner, more universal API.

2. Use Substrait for filter pushdown

Substrait is the emerging standard for cross-engine expression exchange.

Deferred: Substrait support can be added later as an alternative filter format alongside SQL strings. SQL strings are simpler for initial adoption and already supported by Lance's scanner.

Open Questions

1. Header generation tooling: cbindgen vs hand-written headers? cbindgen is automated but sometimes produces suboptimal output. lance-duckdb uses hand-written headers.

2. Thread safety guarantees: Should a single LanceDataset* be safe to use from multiple threads? The Rust Dataset is Send + Sync, so this is feasible but needs documentation.

3. Waker thread safety contract: Resolved — see "Waker Contract" in the Threading Model section. The waker is documented as single-use, callable from any thread, no-reentrancy, and must tolerate stale/spurious invocations.

References

• lance-duckdb extension — Existing Rust-to-C FFI for DuckDB
• Velox Connector API — Velox's data source abstraction
• WIP Velox Lance Connector — Active PR adding Lance support to Velox
• Arrow C Data Interface — Zero-copy cross-language data exchange
• Lance Python bindings — PyO3-based Python API
• Lance Java bindings — JNI-based Java API

[jja725]

@beinan @Xuanwo @jackye1995 would appreciate some early feedback while polishing the API.

[beinan]

Great proposal! I'm really excited to see a stable C-ABI for Lance.

I do want to push back gently on using the JNI bindings as a reference for the threading model. When I implemented block_on there, it was a shortcut for a quick implementation, not a long-term design choice.

For high-performance engines like Velox or Presto, calling a function that blocks the thread on Rust's Tokio runtime is dangerous. It leads to thread starvation where the engine's worker threads sit idle waiting for I/O, rather than yielding to other tasks.

Suggestion: A "Poll + Waker" Pattern

Instead of a blocking next() or a complex callback-hell approach, could we expose a state-machine API similar to Rust’s Future::poll? This allows C++ engines (which often use folly::Future or similar async runtimes) to integrate cooperatively.

The C signature could look something like this:

typedef void (*LanceWaker)(void* ctx);

// Returns:
// 0 = LANCE_POLL_READY (Batch is available in *out)
// 1 = LANCE_POLL_PENDING (IO needed; 'waker' will be called when ready)
// 2 = LANCE_POLL_FINISHED (End of stream)
// <0 = Error
int lance_scanner_poll_next(
    LanceScanner* scanner,
    LanceWaker waker,      // Callback to wake up the C++ task
    void* waker_ctx,       // Context passed back to the waker
    LanceBatch** out
);

How this helps Velox:

1. Non-Blocking: If data isn't ready, Lance returns PENDING immediately. Velox can suspend the task and use the thread for other work.
2. Efficient Wake-up: When Rust finishes the I/O in the background, it calls the waker (e.g., folly::Promise::setValue), prompting Velox to schedule the task again.
3. Zero Overhead: If data is already buffered/cached, it returns READY instantly, just like a synchronous call.

This approach bridges the gap between Rust's async model and C++'s async executors without forcing a blocking impedance mismatch.

[beinan]

A PR for non-blocking JNI interface #6102 cc: @jackye1995

[wangmj17]

Many OLAP engines and data lakes use deletion vector to mark deleted row ids in a data file. Is it possible to add a method to pass excluded row ids to lance scanner?

[jja725]

Is there a need for external row id? I feel this can be scanner internal functionality to read the delete vectors in lance.

[wangmj17]

What do you mean by external row id?

[vicaya]

A common use case of the c binding is for existing c/c++ systems (esp. robotics) to ingest data into lance tables efficiently. Exposing a minimal local fragment creation api with explicit schema would suffice, as a sidecar process in either rust or python can upload/finalize the ingestion async. cf. lance-format/lance-c#4

[jja725]

Hi @vicaya , love to hear such use case in robotics. And given recent evolution in robotics that it need massive amount of multimodal data, I believe lance can play a critical role there as well. Let's having fun together in lance-c!