Is it a must to let Aggregation's Accumulator's memory allocation handled by HashStringAllocator?

[xumingming]

We are trying to implement a Aggregation function which need to use CRoaring(See [1]), from the existing aggregation function implementation, seems memory allocation of Accumulator should be handled by HashStringAllocator, e.g.

velox/velox/functions/prestosql/aggregates/HistogramAggregate.cpp

Line 36 in 063f2b6

AlignedStlAllocator<std::pair<const T, int64_t>, 16>>;

But since we are using an external library, it seems hard to make CRoaring accept HashStringAllocator as memory allocator, is it a must?

[1]. https://github.com/RoaringBitmap/CRoaring

[lingbin]

I noticed there is a chapter in CRoaring README.md that says about this, https://github.com/RoaringBitmap/CRoaring/tree/master?tab=readme-ov-file#custom-memory-allocators, does it help?

[xumingming]

@lingbin CRoaring's custom memory allocators is a global hook, but in Velox every query has a separate memory pool(which is NOT global), so that does not seems to work.

[lingbin]

It seems that a little hacking is needed. After all, the CRoaring library does not expose more interfaces.

The requirement here can be simplified to: when the hook it provides (such as the my_malloc callback function) is called, the corresponding MemoryPool(may be an operator-pool) should be found and updated, right?

I'm not an expert in CRoaring, but I can provide an implementation idea:
we can set the MemoryPool*(may be an operator-pool) object in thread-local before entering the CRoaring library, and then obtain the MemoryPool object of the current thread in the callback, and then call MemoryPool::allocateContinuous() interface.
Is this method feasible?

[xumingming]

@lingbin Thanks for the response, Hmm... I didn't expect to write so hacky code to use an external library.

[Yuhta]

The idea is not that hacky I think, it's quite common workaround when the library does not allow us to pass the needed information (MemoryPool*) through stack.