Hey guys @zhongkaifu and team, thanks you for the amazing work on TensorSharp :3.
I am currently blocked by two missing features that are critical for users with limited VRAM (e.g., 12 GB). I would like to ask whether these are on the roadmap, or if there are any workarounds I might have missed.
My Setup
- GPU: NVIDIA 12 GB VRAM
- Model:
Qwen-35B-A3B-UD-Q4_K_M.gguf (MoE)
- Context: 256K tokens
- Current engine:
llama.cpp with the following flags:
-ngl 99 -c 256000 -ctv turbo2 -ctk turbo2 --n-cpu-moe 32
Request 1: MoE Expert Offload to CPU (--n-cpu-moe equivalent)
In llama.cpp, --n-cpu-moe N allows offloading a subset of MoE experts to system RAM, keeping only the active ones on the GPU. Without this, the entire MoE router + all expert weights must reside in VRAM, which makes a 35B-A3B model in Q4_K_M impossible to fit on a 12 GB card while still leaving room for the KV cache.
Question: Is there any plan to support partial or full MoE expert offload to CPU? Even a simple "run all MoE FFN ops on CPU" fallback would be a huge enabler for low-VRAM users.
Request 2: 2-bit KV Cache Quantization (turbo2 equivalent)
For 256K context, the KV cache dominates memory usage. llama.cpp supports -ctv turbo2 and -ctk turbo2, which quantize the KV cache down to ~2 bits, drastically reducing memory pressure. TensorSharp currently supports f32, f16, and q8_0 for the KV cache.
Question: Are there plans to support more aggressive KV cache quantization (e.g., Q4, Q2, or a custom TurboQuant 2-bit codec)? I noticed TurboQuantKvCodec already exists for Q4/Q8 paged blocks—would extending it to 2 bits be feasible?
Why
With these two features, TensorSharp could become the best inference engine for low-VRAM, long-context MoE workloads. Right now, users like me are forced to stay on llama.cpp despite TensorSharp's superior prefill performance.
I am happy to help test any experimental branch or provide memory profiling data if useful.
Thanks for considering this!
Hey guys @zhongkaifu and team, thanks you for the amazing work on TensorSharp :3.
I am currently blocked by two missing features that are critical for users with limited VRAM (e.g., 12 GB). I would like to ask whether these are on the roadmap, or if there are any workarounds I might have missed.
My Setup
Qwen-35B-A3B-UD-Q4_K_M.gguf(MoE)llama.cppwith the following flags:Request 1: MoE Expert Offload to CPU (
--n-cpu-moeequivalent)In
llama.cpp,--n-cpu-moe Nallows offloading a subset of MoE experts to system RAM, keeping only the active ones on the GPU. Without this, the entire MoE router + all expert weights must reside in VRAM, which makes a 35B-A3B model in Q4_K_M impossible to fit on a 12 GB card while still leaving room for the KV cache.Question: Is there any plan to support partial or full MoE expert offload to CPU? Even a simple "run all MoE FFN ops on CPU" fallback would be a huge enabler for low-VRAM users.
Request 2: 2-bit KV Cache Quantization (
turbo2equivalent)For 256K context, the KV cache dominates memory usage.
llama.cppsupports-ctv turbo2and-ctk turbo2, which quantize the KV cache down to ~2 bits, drastically reducing memory pressure. TensorSharp currently supportsf32,f16, andq8_0for the KV cache.Question: Are there plans to support more aggressive KV cache quantization (e.g., Q4, Q2, or a custom
TurboQuant2-bit codec)? I noticedTurboQuantKvCodecalready exists for Q4/Q8 paged blocks—would extending it to 2 bits be feasible?Why
With these two features, TensorSharp could become the best inference engine for low-VRAM, long-context MoE workloads. Right now, users like me are forced to stay on
llama.cppdespite TensorSharp's superior prefill performance.I am happy to help test any experimental branch or provide memory profiling data if useful.
Thanks for considering this!