[None][docs] fix incorrect auto sampler behavior description for beam search#14487
Open
fuergaosi233 wants to merge 1 commit into
Open
[None][docs] fix incorrect auto sampler behavior description for beam search#14487fuergaosi233 wants to merge 1 commit into
fuergaosi233 wants to merge 1 commit into
Conversation
… search The sampling.md documentation incorrectly stated that the `auto` sampler mode uses TRTLLMSampler when beam search is enabled. In practice, the `auto` mode always selects TorchSampler, as reflected in the field description in llm_args.py and the actual sampler instantiation logic in _torch/pyexecutor/_util.py. TRTLLMSampler is only used when explicitly set via sampler_type="TRTLLMSampler", which is deprecated and scheduled for removal in release 1.4. Signed-off-by: holegots <fuergaosi@gmail.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR updates the sampling backend documentation to clarify that the default ChangesSampling Backend Documentation
🎯 1 (Trivial) | ⏱️ ~2 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/source/features/sampling.mdincorrectly stated that theautosampler mode usesTRTLLMSamplerwhen beam search is enabledautoalways selectsTorchSamplerfor all requests, including beam searchllm_args.py:3810("Defaults to auto, which will use TorchSampler") and the actual sampler instantiation logic in_torch/pyexecutor/_util.py:1821Root Cause
The
automode was historically documented as routing toTRTLLMSamplerfor beam search, but this behavior does not exist in the code.TRTLLMSampleris only used when explicitly set viasampler_type="TRTLLMSampler", which is deprecated and scheduled for removal in release 1.4.Code evidence:
tensorrt_llm/_torch/pyexecutor/_util.py:1821: Only checksif llm_args.sampler_type == SamplerType.TRTLLMSampler— no auto-to-TRTLLM path for beam searchtensorrt_llm/llmapi/llm_args.py:3810: Field description says "Defaults to auto, which will use TorchSampler"tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py:843-844:if sampler_type == SamplerType.auto: sampler_type = SamplerType.TorchSamplerTest plan
llm_args.pyfield descriptionauto+ beam search selectsTRTLLMSamplerSummary by CodeRabbit
automode defaults to using Torch Sampler for all requests, providing consistent behavior across sampling configurations.