[Bug]: Possible mismatch in `truncate_prompt_tokens` value validation for `-1`

[subhalingamd]

Hello,

I tried setting truncate_prompt_tokens=-1, but it looks like the value validation prevents this. This behavior seems inconsistent with the documentation (see below). Could you please clarify if -1 is still a supported value?

vllm/vllm/sampling_params.py

Lines 185 to 188 in ebf7605

	truncate_prompt_tokens: Optional[Annotated[int, msgspec.Meta(ge=1)]] = None
	"""If set to -1, will use the truncation size supported by the model. If
	set to an integer k, will use only the last k tokens from the prompt
	(i.e., left truncation). If set to `None`, truncation is disabled."""

vllm/vllm/sampling_params.py

Lines 413 to 416 in ebf7605

	if (self.truncate_prompt_tokens is not None
	and self.truncate_prompt_tokens < 1):
	raise ValueError(f"truncate_prompt_tokens must be >= 1, "
	f"got {self.truncate_prompt_tokens}")

[noooop]

cc @hmellor

[hmellor]

I recently moved the docstrings for the class members of SampingParams to be directly attached to each class member.

It seems as though the monolithic docstring we had before had gone out of date as the type hint and the error suggests that -1 is not a valid value.

[hmellor]

Hmm actually it looks like the addition of -1 is newer than the type hint

[hmellor]

@gmarinho2 as the author of #14776, was it intentional that some of the type hints for truncate_prompt_tokens remained as >=1?

[noooop]

cc @maxdebayser

[gmarinho2]

@gmarinho2 as the author of #14776, was it intentional that some of the type hints for truncate_prompt_tokens remained as >=1?

It was not intentional. This is a bug. I am looking at it in the above PR. I did not find any other wrong type hints, please let me know if there were any others found.

[subhalingamd]

BTW, I'm wondering if truncate_prompt_tokens even gets used for chat? I tried the following code (for testing purposes) in vllm=0.9.1 and still end up with the value error.

from vllm import LLM, SamplingParams

model = LLM(model="Qwen/Qwen2.5-3B-Instruct", max_model_len=32)
sampling_params = SamplingParams(
    max_tokens=2,
    temperature=0,
    truncate_prompt_tokens=16,
)

outputs = model.chat(
    [{"role": "user", "content": "This is a very long sentence. " * 8}],
    sampling_params,
    use_tqdm=False,
)
raw_outputs = [output.outputs[0].text for output in outputs]
print(raw_outputs)

Output:

ValueError: The decoder prompt (length 86) is longer than the maximum model length of 32. Make sure that `max_model_len` is no smaller than the number of text tokens.

[hmellor]

@subhalingamd in your example you've set the max_model_len=8 which is smaller than the truncation size of 16. Surely it makes more sense to have something like max_model_len=16 and truncate_prompt_tokens=8 so that the model length isn't exceeded and the truncation leaves some space for the model to respond?

[subhalingamd]

I realised that I did not paste the correct code after posting. I have edited the previous comment. Sorry about that!

Also, even otherwise, shouldn't the error message say decoder prompt length is 16 (instead of 86) is more than max_model_len of 8 if it has truncated the input?

[maxdebayser]

I agree that if the truncate_prompt_tokens is set to a value larger than max_model_len there should be a better error message. The PR that originated the truncate_prompt_tokens arg focused a lot on the pooling endpoints and there might be some blind spots in the generative endpoints. We'll verify these.

[maxdebayser]

@subhalingamd, I was able to reproduce the problem triggered by your example code. The problem is that the chat() function pre-tokenizes the input. So when it reaches the processor it is a TokensPrompt instead of a TextPrompt. I've opened this PR #23131 to make sure that TokensPrompts are also truncated.