Optimize parquet GZIP compress/decompress with ISA-L

[zhouyuan]

Description

Hi Velox community,

Parquet w/ GZIP compression is still widely used in some production envs as it can provide high compression ratio. However gzip based I/O throughput will slower comparing with new codec like lz4, zstd.
ISA-L provides optimized low-level functions(CRC, Compression, Erasure coding) which are commonly used in storage applications. Its igzip provides much better performance on compression/decompression, and is also compatible with zlib based algorithms. With this new API, gzip performance is almost able to catch up with zstd[1].

I have made a draft patch(#4533) on parquet gzip decompression code path and its performance did improve a lot.

[1] https://s3.us-east-2.amazonaws.com/intel-builders/day_2_intelligent_storage_acceleration_library.pdf

CC: @Yuhta @mbasmanova

Thanks, -yuan

[yingsu00]

@zhouyuan Currently Parquet decompression is separate from DWRF and ORC. I think it would be better to provide the same decompressors for all file formats, not just Parquet. And it's better to have a central location where the decompressed input streams are created. I left a comment in your draft PR about this. What do you think? @Yuhta

[yingsu00]

This is related to #4411 . Besides these two efforts, Im also working on a customized decompressor. I think it's time to think of a unified solution for decompressor managements. @Yuhta @zhouyuan Shall we discuss this in the Parquet reader meeting this Friday? Velox native parquet reader project meeting
Friday, April 14 · 2:00 – 2:30pm
Google Meet joining info
Video call link: https://meet.google.com/cbg-ugjf-mok
Or dial: ‪(US) +1 484-364-5970‬ PIN: ‪936 707 945‬#
More phone numbers: https://tel.meet/cbg-ugjf-mok?pin=8803191019981

[mbasmanova]

CC: @oerling

[zhouyuan]

@yingsu00 @mbasmanova
Thanks for looking, +1 on consolidating the code of decompression for both parquet and DWRF. Will check further on this.
note ISAL is not bounded with any Intel hardware, it can run on common x86_64 and aarch64 - if this can address some of your concerns.

thanks, -yuan

[Yuhta]

I think it's good to consolidate the decompressors.

Only one thing to keep in mind is that ReaderBase::createDecompressedStream is used outside Velox and we must preserve this interface and its performance.

We could merge QPL support first though, since the integration point is not large, only a few lines in PageHeader, shall not be too hard to move around later