Feature/support muse glimmer model inference - #137
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e1321a823
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| _preparedVisionEmbeddings.Add(new PreparedEmbeddingSpan( | ||
| cached, | ||
| tokenPosition + 1, | ||
| tokenPosition, | ||
| tokenPosition + cached.TokenCount + 2)); |
There was a problem hiding this comment.
For Muse-Glimmer image requests using the shared multimodal path (including BatchExecutor and InteractiveSession), this records a prepared span, but both QueuePreparedVisionEmbeddings switches later in this class omit MuseGlimmerModel. Consequently SetVisionEmbeddings is never called, HasPendingVisionEmbeddings remains false, and inference uses the repeated <|patch|> token embeddings instead of the encoded image. Add Muse-Glimmer cases to both whole-prompt and sliced queue paths.
Useful? React with 👍 / 👎.
| protected override void ResetKVCacheCore() | ||
| { | ||
| _cacheSeqLen = 0; | ||
| _cachedSWAMaskStartPos = -1; | ||
| ResetFusedDecodeCache(); |
There was a problem hiding this comment.
When fused DFlash is enabled and a completed sequence exceeds _dflashRingRows, resetting the target for another request leaves _dflashRingFilled at the previous sequence length because ResetKVCacheCore never calls ResetFusedDFlashCaches. BuildRingSlotPositions then labels slots freshly written at positions starting from zero as positions from the old wrapped range, so they are masked out for the new short prompt and drafting loses its context until the new sequence catches up to the old length. Reset the DFlash ring bookkeeping whenever the target KV cache is reset.
Useful? React with 👍 / 👎.
Engine comparison — TensorSharp vs llama.cpp (PR smoke)No report artifact was produced — the benchmark failed before generating results (see the workflow logs). |
1 participant
No description provided.