Skip to content

fix: handle llama.cpp context overflow in model adapter - #3371

Open
wefstratis-nv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
wefstratis-nv:fix/llamacpp-context-overflow
Open

wefstratis-nv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
wefstratis-nv:fix/llamacpp-context-overflow

Conversation

@wefstratis-nv

Copy link
Copy Markdown

What does this PR do?

Handle llama.cpp context-overflow errors in the OpenAI-compatible model adapter.

Previously, llama.cpp HTTP 400 errors such as exceed_context_size_error were not recognized and caused evaluation failures. These errors now follow the existing context-length handling: return an empty completion with finish_reason="length".

The change covers both chat-completions and completions paths. Unrelated errors continue to propagate.

Validation:

  • All 177 adapter tests pass, including 26 new regression cases.
  • Pre-commit checks pass for the two changed files. The full --all-files command has not been run.
  • Adapter-only coverage increased from 84.74% to 87.75%, but remains below the configured 95% threshold.
  • No live benchmark validation has been performed.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format/copyright pass).
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Signed-off-by: wefstratis <wefstratis@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 14, 2026

@ananthsub ananthsub left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @wefstratis-nv for the contribution! could you address the merge conflicts with the latest main so we can merge?

@yaoyu-33 yaoyu-33 added area:model Model servers, inference providers, and model adapters bug Something isn't working waiting-on-customer Waiting on the original author to respond and removed community-request Issue reported or requested by someone from the community labels Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model servers, inference providers, and model adapters bug Something isn't working waiting-on-customer Waiting on the original author to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants