Native "SmartCache" System with Configurable RAM/Disk Persistence & Hybrid/RNN Model Optimization

[CrazyAce25]

1. The Critical Problem: Wasted Time, Energy & Hardware Wear

Currently, Jan discards the Key-Value (KV) cache and recurrent state whenever a user switches chats or minimizes a window. When the user returns, the backend (llama.cpp) must re-process the entire conversation history from scratch.

• Energy Waste: Re-processing a 200k context window on a modern GPU (e.g., RTX 4090, 7900XTX) consumes 300–400W of steady power for 1–2 minutes per switch. This is a massive, unnecessary electricity cost for local AI users.
• Hardware Degradation: Continuous, redundant high-load compute cycles generate excessive heat, accelerating thermal wear on VRAM and GPU cores.
• The Hybrid/RNN Bottleneck: This is catastrophic for Qwen3.6 and other hybrid models (Attention + RNN/DeltaNet). Unlike standard Transformers, these models maintain a recurrent hidden state that accumulates context. Discarding this state forces the GPU to re-compute complex, non-parallelizable recurrence loops for every historical token. Without caching, the primary efficiency benefit of hybrid architectures is completely lost.

2. Native Capabilities Already Available in llama.cpp

The underlying engine llama.cpp (which powers Jan) already includes mature, production-ready flags to solve this. Jan simply needs to expose them via the GUI.

• --cache-ram <MB>: Allocates a specific pool of System RAM to store pre-computed prompt prefixes and KV states. Restoring from this pool is instantaneous (milliseconds) and bypasses GPU compute entirely.
  • Default: 8192 MB (often too low for heavy users).
  • Capability: Accepts any value (e.g., -1 for unlimited) up to total system RAM.
• --slot-save-path <directory>: Enables the REST API endpoints (/slots/<id>/save and /slots/<id>/restore) to persist KV cache snapshots to Disk (SSD/NVMe).
  • Benefit: Ensures cache survival across app restarts and system reboots.
  • Performance: On NVMe drives, restoring a 200MB cache takes ~40ms.
  • API Detail: The endpoints accept a filename parameter in the JSON body, allowing Jan to namespace files (e.g., qwen3_chat_01.bin) to prevent conflicts in Router Mode.
• --swa-full: Critical for Qwen3.6. Forces full-size Sliding Window Attention cache. Without this, hybrid models will silently truncate context or fail restoration.
• --ctx-checkpoints 512: Increases checkpoint density for faster recurrent state restoration (default 32 is too low for long contexts).
• --kv-unified: Critical for Apple Silicon and integrated GPUs to ensure the host RAM cache and GPU memory allocator do not conflict.

3. Precedents: How Others Have Solved This

• KoboldCPP (SmartCache): Automatically saves KV snapshots to RAM when a context is switched. It intelligently manages these slots (default 6) and is mandatory for RNN/Hybrid models to prevent re-processing. It requires zero user configuration.
• Community Scripts: Advanced users currently run standalone llama-server instances with custom Bash/Python wrappers to manually call /slots/save and /slots/restore. This proves the workflow is viable but highlights the need for a native GUI solution.

4. Proposed Implementation for Jan

We request a "Context Persistence" panel in Jan’s Model Settings with the following specific features:

A. User Interface Controls

1. Toggle: Enable Smart Context Caching (Default: Off).
2. RAM Allocation Slider: System RAM Cache Limit.
   • Range: 0 GB to 80% of Total System RAM (Dynamic; no arbitrary cap).
   • Logic: Passes the selected value (in MB) to --cache-ram. A "Max" option should pass -1.
3. Disk Persistence Toggle: Save Contexts to Disk (Persistent Across Restarts).
   • Logic: When enabled, launches server with --slot-save-path ./jan_cache_data.
4. Cache Rotation Manager:
   • Input: Max Cached Conversations (e.g., 50).
   • Visualizer: A list view showing cached chats, their size, and last accessed time, with "Clear All" and "Delete Selected" buttons.
   • Logic: Jan monitors the cache directory. When the limit is reached, it automatically deletes the Least Recently Used (LRU) .bin file before saving a new one to prevent unbounded disk growth.
5. Advanced: Checkpoint Granularity:
   • Input: Context Checkpoints (Default: 512).
   • Logic: Passes --ctx-checkpoints. Critical for reducing restore time in hybrid models.

B. Backend Logic (The "Smart" Layer)

Jan must implement a frontend listener to orchestrate the native API, specifically addressing Router Mode and Hybrid Model requirements:

1. Required Backend Flags:
   When enabled, Jan must inject these flags into the global llama-server router command:

   • --cache-ram <value>
   • --slot-save-path <jan_data_dir>/cache_slots
   • --swa-full (Mandatory for Qwen3.6 to prevent context truncation).
   • --ctx-checkpoints 512 (Optimizes recurrent state restoration).
   • --kv-unified (Ensures Apple Silicon compatibility).

2. Per-Model Slot Isolation (Router Mode):
   Since Jan uses a unified router, cache files must be namespaced to prevent conflicts between models.

   • Implementation: When calling the API POST /slots/{id}?action=save, Jan must pass a unique filename parameter in the JSON body: <model_id>_<chat_id>.bin.
   • Rotation Logic: The frontend must monitor the cache_slots directory. If file count > Max Cached Conversations, delete the file with the oldest "Last Modified" timestamp before saving the new one.

3. Error Handling (Silent Failure Prevention):
   Hybrid models often fail restoration silently if token positions mismatch.

   • Requirement: On restore, Jan must verify the n_restored token count (returned in API response) matches the chat history length.
   • UI Action: If mismatch occurs, display a toast notification: "Cache restore failed (position mismatch). Re-processing context..." and trigger a full re-prefill. Do not show an empty chat.

4. Automation Triggers:

   • On Chat Switch/Minimize: Automatically trigger POST /slots/<id>?action=save with the unique filename.
   • On Chat Restore: Automatically trigger POST /slots/<id>?action=restore before rendering the chat.
   • Fallback: If a restore fails (e.g., cache evicted), gracefully fall back to standard re-processing.

5. Expected Benefits

• Instant Switching: Returning to a 200k context chat becomes instantaneous (<100ms), regardless of model size.
• Massive Energy Savings: Eliminates the 300W+ power spikes associated with re-processing, reducing electricity costs and heat output by >90% during multitasking.
• Hybrid Model Optimization: Unlocks the full potential of Qwen3.6, Mamba, and RWKV by preserving their recurrent states, which are otherwise discarded.
• Hardware Longevity: Significantly reduces thermal cycling and GPU load, extending the lifespan of consumer graphics cards.

6. References & Precedents

• KoboldCPP SmartCache: Automatically manages RAM snapshots for RNN/Hybrid models. Release Notes
• llama.cpp Flags:
  • --swa-full: Documentation
  • --ctx-checkpoints: Discussion on Hybrid Models
  • --slot-save-path: API Tutorial