Implementation of KV Compression in Llama.cpp for Single-User Long-Context Scenarios?

[i-LOVE-cplusplus]

The attention mechanism, which has a time complexity of $O(l^2)$, can lead to significant performance degradation when dealing with extremely long sequences, such as $l = 10,000$. By incorporating KV compression techniques, we may potentially enhance the inference efficiency for long-context scenarios. (https://arxiv.org/pdf/2406.11430 and https://arxiv.or/pdf/2504.09936)

Compared to the current context shifting approach that discards most information prior to the ctx-size, KV compression appears to offer a more nuanced solution by preserving key details from distant parts of the sequence while selectively discarding less relevant content.

I would greatly appreciate any insights or suggestions you might have on this matter. Would it be feasible for llama.cpp to introduce a specialized mode that supports KV compression, tailored specifically for single-user conversational tasks, while also allowing prompts with lengths exceeding cparams.n_ctx?