Is there a document underlying benefits/working on inmemory async data cache

[jaystarshot]

Background is that we are seeing a lot of server ooms due to inmemory async data cache. We are investigating using the linux memory manager for pushback which does reduce ooms.
The cache is greedy and takes all the available memory (without pushback). There is no config limiting its max bounds so wanted to understand how performant this setup would be vs not using the inmemory cache/using ssd cache more.

We are seeing around 5%-10% hit rate (cache hit bytes / large bytes)

[jaystarshot]

cc: @aditi-pandit @majetideepak cc: @xiaoxmeng

[aditi-pandit]

@jaystarshot : Interesting that you only see 5% hit rate. That is very low. But ofcourse, this depends a lot on your workload. Are you'll not repeating the same input tables in queries ?

@majetideepak has more detailed numbers. We mostly used TPC-DS queries and those show more locality in table accesses so we do see better cache hit rate.

[jaystarshot]

This is our production workload with diverse tables

[jaystarshot]

E20241106 15:39:30.451709  429 PrestoMain.cpp:41]   @   0x7fbd023ed8f9 (unknown)
E20241106 15:39:30.456590  429 PrestoMain.cpp:41]   @   0x7fbd023f8f6a (unknown)
E20241106 15:39:30.461366  429 PrestoMain.cpp:41]   @   0x7fbd023f8fd5 std::terminate()
E20241106 15:39:30.465986  429 PrestoMain.cpp:41]   @   0x7fbd023f9cd3 __cxa_pure_virtual
E20241106 15:39:30.476809  429 PrestoMain.cpp:41]   @     0x47a6ddc facebook::velox::memory::Allocation::~Allocation()
E20241106 15:39:30.485215  429 PrestoMain.cpp:41]   @     0x1469f27 facebook::velox::dwio::common::DirectCoalescedLoad::~DirectCoalescedLoad()
E20241106 15:39:30.485824  429 PrestoMain.cpp:41]   @     0x1466121 _ZN5folly6detail8function13DispatchSmall4execIZN8facebook5velox4dwio6common19DirectBufferedInput9makeLoadsESt6vectorIPNS7_11LoadRequestESaISB_EEbEUlvE6_EEmNS1_2OpEPNS1_4DataESH_
E20241106 15:39:30.497891  429 PrestoMain.cpp:41]   @     0x488c305 folly::ThreadPoolExecutor::runTask()
E20241106 15:39:30.501258  429 PrestoMain.cpp:41]   @     0x4883c69 _ZN5folly6detail8function5call_IZNS_20IOThreadPoolExecutor3addENS_8FunctionIFvvEEENSt6chrono8durationIlSt5ratioILl1ELl1000EEEES6_EUlvE_Lb0ELb0EvJEEET2_DpT3_RNS1_4DataE
E20241106 15:39:30.507431  429 PrestoMain.cpp:41]   @     0x4913230 folly::AtomicNotificationQueue<>::drive<>()
E20241106 15:39:30.517274  429 PrestoMain.cpp:41]   @     0x4913f39 folly::EventBaseAtomicNotificationQueue<>::handlerReady()
E20241106 15:39:30.528376  429 PrestoMain.cpp:41]   @     0x49194f7 folly::EventHandler::libeventCallback()
E20241106 15:39:30.538313  429 PrestoMain.cpp:41]   @   0x7fbd027e3b4f (unknown)
E20241106 15:39:30.543954  429 PrestoMain.cpp:41]   @   0x7fbd027e428f event_base_loop
E20241106 15:39:30.555012  429 PrestoMain.cpp:41]   @     0x490d9ae folly::EventBase::loopMain()
E20241106 15:39:30.566761  429 PrestoMain.cpp:41]   @     0x490e00e folly::EventBase::loopBody()
E20241106 15:39:30.572649  429 PrestoMain.cpp:41]   @     0x490e096 folly::EventBase::loop()
E20241106 15:39:30.580574  429 PrestoMain.cpp:41]   @     0x490ff87 folly::EventBase::loopForever()
E20241106 15:39:30.587162  429 PrestoMain.cpp:41]   @     0x4885281 folly::IOThreadPoolExecutor::threadRun()
E20241106 15:39:30.594511  429 PrestoMain.cpp:41]   @     0x4891c18 folly::detail::function::call_<>()
E20241106 15:39:30.599946  429 PrestoMain.cpp:41]   @   0x7fbd02424903 (unknown)
E20241106 15:39:30.605088  429 PrestoMain.cpp:41]   @   0x7fbd02950ea7 start_thread
E20241106 15:39:30.609648  429 PrestoMain.cpp:41]   @   0x7fbd0210facf clone

Also we are seeing crashes like this once we turn off the async data cache with no abnormality in metrics

[jaystarshot]

Actually it seems like after a while like 1-2 days with pushback enabled the cache size decreases significantly