25% faster decompression with cloudflare's zlib fork

[aminnj]

I came across an AWS recommendation to use a fork of zlib that cloudfare maintains because it's faster.

Basically,

git clone https://github.com/cloudflare/zlib.git
cd zlib
./configure --prefix=`pwd`
make
make install

and then to hackily swap between the two

# copy original zlib back
cp ~/julia/julia-1.7.0-beta4/lib/julia/bck_libz.so.1.2.11 ~/julia/julia-1.7.0-beta4/lib/julia/libz.so.1.2.11

# copy cloudflare over original
cp ~/julia/cloudflare/zlib/libz.so.1.2.8 ~/julia/julia-1.7.0-beta4/lib/julia/libz.so.1.2.11

Trying out @btime transcode(ZlibDecompressor, a) between the two on raw compressed basket bytes, I saw between a 25 and 40% speedup. We can still see a similar speedup with

julia> using UnROOT; const t = LazyTree(ROOTFile("Run2012BC_DoubleMuParked_Muons.root"), "Events");

# regular zlib
julia> @btime sum(t.nMuon) # 770.774 ms (11366 allocations: 544.33 MiB)
julia> @btime sum(length,t.Muon_pt) # 5.322 s (18391 allocations: 2.93 GiB)

# cloudflare fork
julia> @btime sum(t.nMuon) # 542.433 ms (11366 allocations: 544.33 MiB)
julia> @btime sum(length,t.Muon_pt) # 3.975 s (18391 allocations: 2.93 GiB)

This was on a x86_64 AMD EPYC 7702P, but hopefully one can reproduce it on other systems. From another thread/PR, I think we wrote down that zlib-ng was giving a 10-15% improvement over the usual zlib library, but this cloudflare fork is even faster!

I bet since it's a fork, not some next-gen rewrite, that it'll be easier to make into binaries (@Moelf ? ;) ). And I think this could benefit many workflows due to ROOT using zlib by default.

[aminnj]

Based on http://www.htslib.org/benchmarks/zlib.html, I tried the Intel fork but got the same speed as the default zlib. I guess it makes sense that cloudflare would want a super optimized zlib...

My findings are consistent with https://www.snellman.net/blog/archive/2014-08-04-comparison-of-intel-and-cloudflare-zlib-patches.html (for decompression, Intel is the same as the usual zlib, but cloudflare is clearly faster)

Google/Chromium has a fork too (https://serverfault.com/questions/160969/faster-zlib-alternatives) but I don't want to figure out their build system, and apparently cloudflare is faster.

[aminnj]

As a further note, it could be worth it to look into
https://github.com/jakobnissen/LibDeflate.jl

Technically we allow a streaming-API for decompressing each basket, but from what I saw, we only enter the while decompressedbytes < totalexpectedbytes loop once.

[Moelf]

we only enter the while decompressedbytes < totalexpectedbytes loop once.

at some point I hit this bug so I'm pretty sure it "happens"

[aminnj]

we only enter the while decompressedbytes < totalexpectedbytes loop once.

at some point I hit this bug so I'm pretty sure it "happens"

True. If LibDeflate.jl does bring a drastic improvement, then we could just check if uncompbytes, tkey.fObjlen match when we first enter the loop. If so, we know it's a single decompression and we break out after using LibDeflate.

[tamasgal]

Oh that's interesting, thanks @aminnj for the information. To me it sounds like it would make sense to provide a jll for the cloudfare zlib, but this is of course much more work and I am sure there are more people (outside of the ROOT world) who could potentially help out. Currently I even struggle to get the hdf5lib compiled on an M1, let alone to automate it: JuliaIO/HDF5.jl#870

LibDeflate.jll is also quite interesting and I think it totally makes sense since as you said, we are not streaming.

[Moelf]

jakobnissen/LibDeflate.jl#1 (comment)

sorry to break the news for you but LibDeflate is not our out ;)

[Moelf]

JuliaPackaging/Yggdrasil#3946

I can't get the same branch to build cross-platform, unfortunately.

[aminnj]

Looks like LibDeflate.jl is giving the same performance gain as cloudflare. Just need to drop the first two header bytes which are associated with zlib (rest is DEFLATE data):

nbytes = decompress!(Decompressor(), uncomp_data, rawbytes[3:end])

PR incoming