Doing Batched Inference Using TokensPrompt

[wlm64]

Does anyone know how to enable batched inference using a tokensprompt as input instead of text?

Trying to declare a list of tokens like with a text input doesn't work.

For example trying:

tp = TokensPrompt({"prompt_token_ids": [[1,2],[1,2]]})

Results in a *** TypeError: '>' not supported between instances of 'list' and 'int' when using:

model.generate(tp, sampling_params)

However passing a list of strings as an input does batched inference correctly.

Thanks!

[DarkLight1337]

You should be passing a list of TokensPrompt, not have the list inside one prompt.