Direct ingest from Arrow tables

[mauropagano]

I was wondering how you guys feel about exposing a convenient way to load an Arrow table straight into a db table.
Right now it requires going Arrow -> Python -> Oracle types, would be nice to skip python in between.
Doing it in the library also means executemany() could be aligned to a RecordBatch, making streaming easier, etc etc

Is this a big no because it would add a potentially unnecessary dependency?
Is this more of a ODPI-level question?

[anthony-tuininga]

As an optional feature it could be considered. Doing so in ODPI-C, however, would limit it to thick mode. Can you provide additional information on what you would like to see? Are you ingesting serialized Arrow tables? What would the input format look like?

[mauropagano]

Personally having to go thick mode route wouldn't be an issue on my end but I can see how others might not like that.

Ideally, I'd like to be able to pass the whole table as bind (granted the SQL would match the columns in the arrow table, etc etc), something like cursor.executemany(sql, arrow_table).

Are you ingesting serialized Arrow tables

Not exactly, more producing Arrow data / working with it and just want to move that to Oracle (Pandas, Polars, DuckDB, etc very often offer zero-copy via Arrow so lots of ways to end up with data in Arrow format)

What would the input format look like

I would imagine a "flat" Arrow table, it gets messy with nested types but maybe they can just not be supported (like collections from python already aren't)

To be clear, I'm not asking for it as a convenience function, more like you can unlock performance in the library that users can't without dropping into ODPI-C

[anthony-tuininga]

I was referring to this: https://github.com/oracle/python-oracledb/issues, but if you create an SR with Oracle Support that might increase visibility.

[mauropagano]

Got it, I might be able to do both, let me start with a GH issue first as I can turn that around quickly

[mauropagano]

One clarification, so that I can help tailor the ER, you said

generating arrow format in python-oracledb

Is the focus on emitting arrow from Oracle (i.e. cursor.execute().fetcharrow() or something like that) and NOT ingesting arrow (i.e. cursor.executemany("insert ....", <<arrow object>>)) ?

I can see value in having both, and happy to suggest we do both, but just to be clear, my initial suggestion was on the latter (so that I wasn't going to trip on the "no quick ways to get data out of Oracle"

[anthony-tuininga]

Most would be interested in getting data out of the database, so that is the current focus, but I don't see any reason why we couldn't do both, though! You are more interested in getting data into the database?

[mauropagano]

I'm interested in both, and I'm assuming the gain as well as use cases solved will be the same (skip python representation altogether).
I was only asking about ingest arrow into database as historically there wasn't a lot of appetite to expose convenient ways to get data out of Oracle.
VERY EXCITED if both in/out can be implemented!

[cjbj]

This is being tracked in #375