Add support for Apache ORC format

[yimin-yang]

Hi Velox community,

We are from Meituan's big data team and are currently working on adding support for the Apache ORC format in Velox.

1. We have enhanced the ORC Reader/Writer based on the current implementation of dwio/dwrf. This primarily involves adding RLEv2 Encoder/Decoder, as well as column readers for date/string/timestamp/decimal types.

2. The current approach of incorporating Apache ORC into Dwrf seems to have some issues, especially since the encapsulation of Dwrf and ORC is only halfway complete. We have some ideas to optimize this part and would like to discuss them with the Velox community.

3. We hope to continue contributing with the Velox community in supporting the ORC format by enhancing, refactoring, and optimizing it.

[zuochunwei]

I will explain why the encapsulation of Dwrf and ORC is halfway complete.

the module velox/dwio uses ProtoWrapperBase to encapsulate Dwrf & ORC.

the classes, for example, PostScript、StripeInformationWrapper、TypeWrapper、UserMetadataItemWrapper、FooterWrapper defined in the file of FileMetadata.h have two constructors for each class. One takes the parameter of proto::Footer for Dwrf, and the other takes the parameter of proto::orc::Footer for ORC. they work well.

But for some other classes, they are not well designed.

For example, StripeReaderBase has a member of proto::StripeFooter* footer_ but does not also has a member of proto::orc::StripeFooter* footer_. it handles both formats Dwrf and ORC, it may work, but it's dangerous, and we should avoid coding like this.

let's see another example, DwrfStreamIdentifier has a constructor with a parameter of proto::Stream_Kind, but doesn't has a constructor with a parameter of proto::orc::Stream_Kind, we use static_cast but it's not a good way.

We should complete this work in the same way as FileMetadata.h.

[Yuhta]

These components (StripeReaderBase and DwrfStreamIdentifier) are already used in production systems outside Velox (actually it was moved from those systems into Velox), so it would be risky to change the interface (all constructor and public/protected methods) at this point. If you can fix it by adding extra constructors it would be best, but the current interface need to be preserved and we need to be careful there is no performance regression (mainly dynamic dispatch cost in this case).

[zuochunwei]

I want to add extra constructors for the class DwrfStreamIdentifier, and add extra members for StripeReaderBase.

In my mind, I think StripeReaderBase should have two getStripeFooter interfaces, one for Dwrf, and another for Orc.

In this way, StripeStreamsImpl::loadStreams can work in the safe branch code.

Now, the function StripeStreamsImpl::loadStreams() contains a statement auto& footer = reader_.getStripeFooter();, the type of footer is proto::StripeFooter, We cannot use this to process Orc directly, it's dangerous and not reasonable.

[zuochunwei]

I have proposed a draft PR for oap-project/velox, and this work is still in progress, but the refactor thinking is present.
Could you please have a look for me? thank you all.
@Yuhta @yimin-yang

[Yuhta]

The PR looks good so far. As long as we do not change the existing interface and performance, we should be ok. Ideally we want to reuse as much as possible code though (e.g. in StripeStreamsImpl::loadStreamsOrc).

[zuochunwei]

The PR looks good so far. As long as we do not change the existing interface and performance, we should be ok. Ideally we want to reuse as much as possible code though (e.g. in StripeStreamsImpl::loadStreamsOrc).

OK, thanks for your reply, we will finish it.

[Yuhta]

We can start from 1 by adding support for decoders and types. 2 is a little risky but we can give it a try.

[zuochunwei]

I have completed almost this discussion, please review this PR for me PR