Apache ORC Support in TensorFlow IO

[oliverhu]

(Creating this issue for visibility so people interested can join the discussion... )

Overview

Load Apache ORC formatted data natively into TensorFlow from file system supported by TensorFlow, e.g. HDFS, local disk, etc.

Motivation

We traditionally use Avro to store our dataset but it is becoming inefficient to use row based format for big data analytics processing. Historically we selected ORC as our columnar storage format. (not planning to argue Parquet vs ORC here ;))

Design Discussions

• Apache ORC would be brought in via https://github.com/bazelbuild/rules_foreign_cc
• Feature wise, I expect the APIs to be similar to Parquet or Arrow reader.

Milestones

• Add Apache ORC build dependency.
• Implement a simple ORC dataset that maps records in ORC files into Tensors.
• add a tutorial for ORC reader.
• feature schemas support: support sparseTensor and VarLenFeature.
• feature schemas support: support denseTensor FixedLenFeature only. (follow parse_example_v2.)
• usability improvements
• performance tuning
• feature schemas support: support raggedTensor

[kvignesh1420]

@oliverhu any update on this?

[oliverhu]

no update recently @kvignesh1420

[kvignesh1420]

@oliverhu can we document the current feature in the form of a tutorial?

[oliverhu]

sure, will add that !

[kvignesh1420]

Reference FYKI: https://github.com/tensorflow/io/tree/master/docs/tutorials

[372046933]

Is HDFS supported now? Loading from HDFS path results in coredump

dataset = tfio.IODataset.from_orc("hdfs://xxx/yy/iris.orc", capacity=15).batch(1)

[372046933]

Is HDFS supported now? Loading from HDFS path results in coredump

dataset = tfio.IODataset.from_orc("hdfs://xxx/yy/iris.orc", capacity=15).batch(1)

HDFS supported (with kerberos) by #1674