Writing to a vortex stream

[ak-gupta]

Hi!

I was wondering if there's a way to write to a stream using vortex in Python. Right now, I'm using Polars with PyArrow IPC streams:

import polars as pl
import pyarrow as pa

def serialize(df: pl.LazyFrame) -> pa.Buffer:
    sink = pa.BufferOutputStream()
    obj.collect().write_ipc_stream(sink, ...)

    return sink.getvalue()

def deserialize(buf: pa.Buffer) -> pl.LazyFrame:
    return pl.read_ipc_stream(obj, use_pyarrow=True).lazy()

I would love to be able to write to the stream using vortex. Something along the lines of

def deserialize(buf: pa.Buffer) -> pl.LazyFrame:
    ds = vx.open(buf).to_dataset()

    return pl.scan_pyarrow_dataset(ds)

Thanks!

[gatesn]

I believe @onursatici was planning to take a look at this in the next couple of weeks. Right now the writer is pull-based so it's a little hard to invert the control flow in this manner

[ak-gupta]

Sounds good, thanks for the reply!

[ak-gupta]

Hey @gatesn @onursatici just checking in on this!

[gatesn]

Do you think this RFC would help address the issue? vortex-data/rfcs#54

[ak-gupta]

Yes!