EPIC: Storage node memory footprint optimisation

[the-ancient-1]

Translation of Task Description from the Ticket
DoD: VDisk index consumes 20+% less memory

• Buffer in interconnect is 256 kilobytes - TOutgoingStream - can reduce buffer size to 32 kilobytes, for example (possibly dynamically adjust buffer size depending on channel load - if a lot of data is being transmitted, use larger buffers)
• Learn to reproduce high memory consumption by storage nodes on a test slice
• Understand who and why keeps 13 gigabytes in interconnect session tcp on storage node -- threshold for copying from buffer - some percentage of buffer size
• Take a snapshot of the vdisk index from production and try different algorithms - see how much space the index will occupy
• Write a document describing memory consumption by storage node
• Remove size from the index
• Evaluate the benefit of index compression by converting to a prefix tree
• Consider converting to B-tree
• tablet id, channel - can be moved to part of the tree
• Use custom allocator for fixed-size elements with short addresses-indices in the pool

2 parts:

Loading any index and converting it to a compact form in memory
Writing a compact index and loading it quickly
Memory consumption by storage node nbs

cloud prod vla
34 gigabytes total

interconnect session tcp - 13 gigabytes
vdisk compaction - 8 gigabytes
bs level segment loader - 7 gigabytes
vdisk skeleton - 2.5 gigabytes
bs async level index - 2.4 gigabytes
pdisk actor 1 gigabyte
interconnect proxy tcp 0.5
interconnect proxy wrapper 0.5

23 gigabytes total (node 400)

interconnect session tcp - 13 gigabytes --- 52 vdisks, 128*52= 6.656
vdisk compaction - 3.5 gigabytes
vdisk skeleton - 2.5 gigabytes
bs async level index - 2.3 gigabytes
bs level segment loader - 0.7 gigabytes
pdisk actor 1 gigabyte
interconnect proxy tcp 0.5
interconnect proxy wrapper 0.5

according to graphs - indices 3.8 gigabytes
fresh - 2.8 gigabytes
(total by vdisk graph - 7.6)