[Bug]: n>1 streamed requests are logged and billed as one merged completion

[nazsats]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

With n=2 and stream=True, both completions arrive correctly over the stream —
two distinct choices[].index values, each with its own text. Passing those same
chunks to stream_chunk_builder collapses them into a single choice whose
content is the two completions concatenated.

len(response.choices) drops from 2 to 1, and the surviving text is two
different answers stuck together. Nothing raises and nothing warns.

This matters beyond direct callers: the proxy runs stream_chunk_builder over
streamed responses for logging, spend tracking and guardrails
(proxy/common_request_processing.py:359, litellm_core_utils/logging_utils.py:238,
and all four guardrail hooks). So on an n>1 streamed request the object that
gets logged, costed and guardrail-scanned is this merged one, even though the
client read the stream correctly.

Reproduction

import litellm
from litellm import stream_chunk_builder

MESSAGES = [{"role": "user", "content": "Name one colour. One word only."}]
COMMON = dict(model="gpt-4o-mini", messages=MESSAGES, max_tokens=12, temperature=1.0)

# Control: unstreamed n=2 returns two choices.
unstreamed = litellm.completion(**COMMON, n=2)
print(len(unstreamed.choices))
print([c.message.content for c in unstreamed.choices])

# The stream carries both choices correctly.
chunks = list(litellm.completion(**COMMON, n=2, stream=True))
seen = {}
for chunk in chunks:
    for choice in chunk.choices:
        piece = getattr(choice.delta, "content", None)
        if piece:
            seen[choice.index] = seen.get(choice.index, "") + piece
print(sorted(seen), seen)

# Reassembling those same chunks loses one.
rebuilt = stream_chunk_builder(chunks, messages=MESSAGES)
print(len(rebuilt.choices))
print([c.message.content for c in rebuilt.choices])

Observed output:

2
['Blue.', 'Azure.']            <- unstreamed: two choices

[0, 1] {0: 'Azure.', 1: 'Blue.'}   <- the stream delivers both, correctly

1                              <- reassembled: one choice
['Azure.Blue.']                <- the two completions concatenated

Expected

stream_chunk_builder should return the same shape the unstreamed call does —
two choices, rebuilt independently:

2
['Azure.', 'Blue.']

Actual

1
['Azure.Blue.']

n=1 streaming is unaffected.

---

Root cause

litellm/litellm_core_utils/streaming_chunk_builder_utils.py,
ChunkProcessor.get_combined_content:

content_list: Final[list[str]] = []
for chunk in chunks:
    choices = chunk["choices"]
    for choice in choices:              # every choice, whatever its index
        delta = choice.get("delta", {})
        content = delta.get(delta_key, "")
        if content is None:
            continue
        content_list.append(content)

combined_content: Final = "".join(content_list)

choice["index"] is never read. Deltas are flattened across choices before being
joined, so there is no point at which the second completion could be kept
separate.

get_combined_reasoning_content delegates to the same function with
delta_key="reasoning_content", so reasoning content collapses identically —
{0: "A1", 1: "B1", 0: "A2", 1: "B2"} rebuilds as "A1B1A2B2".

Index handling is inconsistent within this one module: the tool-call path
does group by index — see #17425, and the recent
perf(streaming): group tool-call fragments once instead of rescanning per index
— while the content path ignores it.

User Flow

Before a (hypothetical) fix: a developer asking for two completions in one
streamed request sees both in their app, but their logs and spend record only
one — with the two answers stuck together as a single reply.

1. They send POST http://localhost:4000/v1/chat/completions with "n": 2,
   "stream": true and "stream_options": {"include_usage": true}, asking the
   model to name one colour.
2. The SSE stream arrives correctly: chunks carry "index": 0 and "index": 1,
   which reassemble on their side into two separate answers — Azure. and
   Blue. Their application shows both to the user, as expected.
3. The record the gateway keeps for that request holds one completion, not
   two, and its text is Azure.Blue. — the two answers concatenated into one
   string with no separator.
4. The usage attached to that single record is the usage for both completions
   (completion_tokens: 4 for what is now shown as one reply), so the cost is
   attributed to a completion that was never generated.
5. Anything reading that record downstream — spend reporting, a saved
   conversation, a guardrail scanning the reply — sees Azure.Blue. as the
   assistant's single answer.

After a (hypothetical) fix: the same request is logged as two completions,
matching what the client received and what the unstreamed call returns.

1. They send the same POST http://localhost:4000/v1/chat/completions with
   "n": 2, "stream": true and "stream_options": {"include_usage": true}.
2. The SSE stream arrives correctly: chunks carry "index": 0 and "index": 1,
   which reassemble into Azure. and Blue. Unchanged from before.
3. The record now holds two completions — Azure. at index 0 and Blue. at
   index 1 — the same shape the identical request returns with "stream": false.
4. Usage is attached to the request as a whole, and each completion is a separate
   entry rather than one merged string.
5. Downstream readers see two answers, and a guardrail scanning the reply sees
   each completion as its own text rather than a run-on of both.

n=1 streaming is identical before and after.

Proof the bug occurs

Config the proxy ran with

config.yaml:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  drop_params: true
  # Prints the record the proxy keeps for each request. Without a callback or a
  # database that record never surfaces anywhere a user can look, which is why
  # this is easy to miss.
  callbacks: record_probe.probe

general_settings:
  disable_spend_logs: false

record_probe.py, in the same directory:

from litellm.integrations.custom_logger import CustomLogger


class RecordProbe(CustomLogger):
    def _report(self, tag, response_obj):
        choices = getattr(response_obj, "choices", None)
        if not choices:
            return
        print(f"\n=== PROXY RECORDED ({tag}) ===", flush=True)
        print(f"choices in the logged record: {len(choices)}", flush=True)
        for choice in choices:
            message = getattr(choice, "message", None)
            content = getattr(message, "content", None) if message else None
            print(f"  index {getattr(choice, 'index', '?')}: {content!r}", flush=True)
        print(f"usage: {getattr(response_obj, 'usage', None)}", flush=True)
        print("=== END RECORD ===\n", flush=True)

    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        self._report("sync", response_obj)

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        self._report("async", response_obj)


probe = RecordProbe()

Env: OPENAI_API_KEY=<redacted>. No database, no virtual keys.

Started with:

litellm --config config.yaml --port 4000

Version

• litellm 1.97.0, commit e0de9cbab272edb0c087b280295810225836bb84
• Python 3.13.2, Windows
• Live calls to the real OpenAI API (gpt-4o-mini), not mocked

Control — the same request unstreamed returns two completions

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Name one colour. One word only."}],"n":2,"max_tokens":12,"temperature":1.0}'

choices: 2
  index 0 -> 'Azure.'
  index 1 -> 'Blue.'
usage: {'completion_tokens': 4, 'prompt_tokens': 15, 'total_tokens': 19, ...}

The bug — the same request streamed

curl -s -N http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Name one colour. One word only."}],"n":2,"max_tokens":10,"temperature":1.6,"stream":true,"stream_options":{"include_usage":true}}'

What the client received over SSE — correct, two separate completions:

CLIENT RECEIVED OVER SSE:
  choice index 0: 'Azure.'
  choice index 1: 'Blue.'

What the proxy recorded for the same request — one merged completion:

=== PROXY RECORDED (async) ===
choices in the logged record: 1
  index 0: 'Azure.Blue.'
usage: Usage(completion_tokens=4, prompt_tokens=15, total_tokens=19, ...)
=== END RECORD ===

Two completions went out over the wire. One went into the record, and its text
is both answers concatenated

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.97.0

Twitter / LinkedIn details

https://www.linkedin.com/in/nazrul-ansari-ai

[nazsats]

Adding an end-to-end reproduction through the proxy, since the consequence is on the logging and spend path rather than in the SDK.

config.yaml:

model_list:

• model_name: gpt-4o-mini
  litellm_params:
  model: openai/gpt-4o-mini
  api_key: os.environ/OPENAI_API_KEY
  litellm_settings:
  callbacks: record_probe.probe
  record_probe.py is a CustomLogger that prints len(response_obj.choices) and each choice's content on async_log_success_event — the record never surfaces otherwise without a database.

Started with litellm --config config.yaml --port 4000, litellm 1.97.0, live OpenAI calls.

curl -s -N http://localhost:4000/v1/chat/completions
-H "Content-Type: application/json" -H "Authorization: Bearer sk-1234"
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Name one colour. One word only."}],"n":2,"max_tokens":10,"temperature":1.6,"stream":true,"stream_options":{"include_usage":true}}'

[nazsats]

Client received over SSE — two separate completions:

choice index 0: 'Azure.'
choice index 1: 'Blue.'
What the proxy recorded for that same request:

=== PROXY RECORDED (async) ===
choices in the logged record: 1
index 0: 'Azure.Blue.'
usage: Usage(completion_tokens=4, prompt_tokens=15, total_tokens=19, ...)
Two completions went over the wire, one went into the record, and its text is both answers concatenated. n=1 is unaffected.