support MoE offload to cpu - #128
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 448f04fe72
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| gateBias: null, upBias: null, downBias: null, | ||
| activation: GgmlBasicOps.MoEActivation.SwiGLUSplit); | ||
| activation: GgmlBasicOps.MoEActivation.SwiGLUSplit, | ||
| runOnCpu: MoeCpuOffloadConfig.IsLayerOnCpu(layer)); |
There was a problem hiding this comment.
When Qwen 3.5/3.6 runs with --tp > 1, the default TpFusedModelDecodeAvailable() path remains enabled because it has no MoeCpuOffloadConfig.IsEnabled guard. TryBuildTpFdLayerDescs() also leaves CpuMoe at zero, so decode bypasses this newly offloaded per-op call and continues executing all routed experts on the GPUs. This makes --cpu-moe ineffective for the default TP decode path and can retain enough VRAM to OOM; make the fused TP path stand down while CPU offload is enabled, as the usage documentation promises.
Useful? React with 👍 / 👎.
| gateBias: null, upBias: null, downBias: null, | ||
| activation: GgmlBasicOps.MoEActivation.GEGLUSplit); | ||
| activation: GgmlBasicOps.MoEActivation.GEGLUSplit, | ||
| runOnCpu: MoeCpuOffloadConfig.IsLayerOnCpu(layer)); |
There was a problem hiding this comment.
With tensor parallelism enabled, this correctly sends the selected layer's matmuls to the host, but PrepareGgmlQuantizedWeightsForInferenceTP() still uploads every _tpQuantWeights shard, and the Qwen/Gemma auxiliary preload hooks likewise upload every stacked expert shard without checking IsLayerOnCpu. Consequently the routed experts remain fully device-resident even though computation runs on the CPU, defeating the flag's primary VRAM-saving purpose and potentially causing the same load-time OOM it is intended to avoid. The TP generic and stacked preload loops need to skip expert shards belonging to offloaded layers.
Useful? React with 👍 / 👎.
| protected virtual bool ShouldPreloadCudaQuantWeightToDevice(string weightName) | ||
| => !MoeCpuOffloadConfig.IsOffloadedExpertWeightName(weightName); |
There was a problem hiding this comment.
When users combine these public flags with --backend cuda, the new predicate is never consulted: PrepareDirectCudaQuantizedWeightsForInference() still uploads every routed expert and releases its host data, while the new host MoE dispatches are only reached through GGML model paths. The CLI nevertheless logs CPU offload as active, so users expecting the advertised VRAM reduction instead get a silent no-op and may OOM. Either apply the residency and host-dispatch policy to direct CUDA or fail fast for that backend.
Useful? React with 👍 / 👎.
Engine comparison — TensorSharp vs llama.cpp (PR smoke)No report artifact was produced — the benchmark failed before generating results (see the workflow logs). |
1 participant
No description provided.