Structured output broken on Bedrock Claude: output_config.format: Extra inputs are not permitted

[ffma-glen-sampson]

Structured output is broken on Bedrock Claude: output_config.format: Extra inputs are not permitted

Component: hindsight-api 0.8.6 (self-hosted), engine/providers/litellm_llm.py
Provider: HINDSIGHT_API_LLM_PROVIDER=bedrock, au.anthropic.claude-haiku-4-5-20251001-v1:0, ap-southeast-2
litellm: 1.93.0
Severity: High — on Bedrock Claude, retain and consolidation are completely non-functional

Summary

With provider=bedrock and any Anthropic Claude model, every structured-output
call fails. LiteLLMLLM.call() passes an OpenAI-style response_format; litellm
translates it into a Converse outputConfig block; Bedrock's Anthropic layer
rewrites that internally to snake_case output_config.format and its own validator
rejects the key:

litellm.BadRequestError: BedrockException - {"message": "The model returned the
following errors: output_config.format: Extra inputs are not permitted"}

Consequences:

• POST /v1/{tenant}/banks/{bank}/memories → HTTP 500 whenever
  retain_extraction_mode needs the LLM (e.g. concise). No facts, no entity graph.
• Consolidation silently degrades — 3 retries then skipping batch, so
  observations and mental models are never built. This one produces no API error,
  only log lines, so it's easy to miss.
• recall and reflect are unaffected. reflect goes through
  call_with_tools(), which emits toolConfig — and Bedrock accepts that.

Same model, same credentials, same region for both paths. The only variable is
response_format vs tools.

This is not a litellm bug, and not a config problem

litellm 1.93.0 sends a well-formed request. Captured wire request to
https://bedrock-runtime.ap-southeast-2.amazonaws.com/model/au.anthropic.claude-haiku-4-5-20251001-v1%3A0/converse:

{"outputConfig": {"textFormat": {"type": "json_schema",
                                 "structure": {"jsonSchema": {"...": "..."}}}}}

That is correct camelCase Converse shape — the snake_case output_config.format
from the error never appears in litellm's payload. Bedrock's Anthropic layer
generates it during internal translation, then refuses it.

Ruled out:

• pop_bedrock_invoke_output_config_format is present in 1.93.0 (it covers the
  Invoke path; this is the Converse route, confirmed via
  BedrockModelInfo.get_bedrock_route). Bumping litellm does not help.
• HINDSIGHT_API_LLM_STRICT_SCHEMA already defaults to False (config.py), so
  strict mode is not the trigger.
• Patching litellm.model_cost[...]["supports_response_schema"] = False does not
  reach the Converse path. drop_params likewise doesn't strip it.
• No env-var-only workaround exists.

Reproduction (raw boto3, no litellm involved)

import boto3, json
c = boto3.client("bedrock-runtime", region_name="ap-southeast-2")
MODEL = "au.anthropic.claude-haiku-4-5-20251001-v1:0"
msgs = [{"role": "user", "content": [{"text": "hi"}]}]
schema = {"type": "object", "properties": {"a": {"type": "string"}}, "required": ["a"]}

# 1. plain
c.converse(modelId=MODEL, messages=msgs)                                    # OK

# 2. structured output via outputConfig
c.converse(modelId=MODEL, messages=msgs, outputConfig={
    "textFormat": {"type": "json_schema", "structure": {"jsonSchema": schema}}})
# ValidationException: output_config.format: Extra inputs are not permitted

# 3. structured output via toolConfig
c.converse(modelId=MODEL, messages=msgs, toolConfig={
    "tools": [{"toolSpec": {"name": "r", "inputSchema": {"json": schema}}}],
    "toolChoice": {"tool": {"name": "r"}}})                                 # OK

Results: PLAIN: OK / OUTPUTCONFIG: ValidationException … / TOOLCONFIG: OK.

Via the API:

curl -s -X POST "$HINDSIGHT_URL/v1/default/banks/$BANK/memories" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"items":[{"content":"The sky is blue."}]}'
# {"detail":"Fact extraction failed: 1/1 chunks failed. First failures: chunk 0:
#  BadRequestError: litellm.BadRequestError: BedrockException - {\"message\":
#  \"The model returned the following errors: output_config.format: Extra inputs
#  are not permitted\"}"}

POST .../memories/dry-run-extract fails identically.

Scope

• Not model-specific: au.anthropic.claude-sonnet-4-5-20250929-v1:0 fails identically.
• Not fixable by switching to Amazon models: au.amazon.nova-lite-v1:0 /
  apac.amazon.nova-lite-v1:0 don't support outputConfig in ap-southeast-2 at all.
• Affects every LiteLLMLLM.call() caller with a response_format — fact extraction,
  consolidation, and any other structured operation on Bedrock Claude.

Suggested fix

Use a forced tool call for structured output when the model routes through Bedrock —
exactly what providers/anthropic_llm.py already does for the native SDK
(use_forced_tool + tool_choice={"type":"tool","name":...}).

In LiteLLMLLM.call(), when self.model.startswith("bedrock/"), replace the
response_format kwarg with:

call_kwargs["tools"] = [{
    "type": "function",
    "function": {"name": "structured_response",
                 "description": f"Return the structured response ({schema_name}).",
                 "parameters": schema},
}]
call_kwargs["tool_choice"] = {"type": "function",
                              "function": {"name": "structured_response"}}

then substitute the tool call's arguments for content before the existing
parse/validate block, so retries, markdown-stripping, parse_llm_json repair,
usage accounting, and tracing all keep working unchanged. Fall back to the text
parse when the tool call is absent (a gateway may drop tool_choice).

Verified locally against real Bedrock — call() with a list[str] schema returns a
validated model instead of 400:

prefers_forced_tool = True
RESULT: facts=['The sky is blue', 'Grass is green']

End-to-end on an otherwise-pristine 0.8.6 image: retain in concise mode returns
success: true with real usage (input_tokens=3587, output_tokens=349), atomic facts
are extracted with entity annotations, and consolidation logs
created=2 updated=0 skipped=0 plus mental-model refreshes, with zero
output_config errors.

Happy to open a PR if the approach looks right.

Workaround for other operators, in the meantime

Set the bank's retain_extraction_mode to chunks, which bypasses the LLM on the
write path:

curl -s -X PATCH "$HINDSIGHT_URL/v1/default/banks/$BANK/config" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"updates":{"retain_extraction_mode":"chunks"}}'

Retain and recall then work, but there is no fact extraction, no entity/relationship
graph, and no consolidation/observations.

Minor follow-up

POST /v1/{tenant}/banks/{bank}/health/llm is gated behind
HINDSIGHT_API_ENABLE_BANK_LLM_HEALTH. Since it exercises exactly the path that
breaks here, enabling it by default (or documenting it prominently) would make this
class of problem much faster to triage — a structured-output probe would have caught
this immediately.

[nicoloboschi]

Fixed in #3382 (merged to main, ships in 0.9.1). Thanks for the exceptionally good report — the boto3 probe isolating outputConfig vs toolConfig is what made this quick to act on, and the fix is the one you proposed.

What to set in 0.9.1

HINDSIGHT_API_LLM_STRUCTURED_OUTPUT_FORCED_TOOL=true

Server-level env var (like HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS) — not per-bank, so it applies to every internal LLM call. With it on, the LiteLLM-backed providers (litellm, litellmrouter, bedrock) send the response schema as a single forced tool call instead of response_format:

{"tools": [{"type": "function",
            "function": {"name": "structured_response", "parameters": {"...": "the schema"}}}],
 "tool_choice": {"type": "function", "function": {"name": "structured_response"}}}

The tool call's arguments are substituted for the message content before the existing parse step, so markdown-stripping, JSON repair, retries, usage accounting and tracing all behave exactly as before. If a gateway drops tool_choice and the model replies with text, that text is parsed as it is today. You can then drop the retain_extraction_mode: chunks workaround and get fact extraction, the entity graph and consolidation back.

Default is false, for a reason worth knowing about ⬇️

It's region-dependent, which your report didn't have visibility into

We reproduced this on our own AWS account, and the same claude-haiku-4-5 behaves differently by region:

Probe	us-east-1 / us.*	ap-southeast-2 / au.*
converse, plain	OK	OK
converse + outputConfig	OK	rejected
converse + toolConfig	OK	OK
Hindsight, flag off	OK	fails as you describe
Hindsight, flag on	OK	OK

Those first rows are raw boto3 with no litellm in the path, so this is the Bedrock deployment, not the client stack. That's why it's opt-in rather than automatic for bedrock/* models: flipping it on for everyone would push us-east-1 deployments off a route that works fine for them.

One small correction for anyone who lands here from a search: on our account the rejected key comes back as model: Extra inputs are not permitted, not output_config.format:. Same validator, same signature, different key in the message.

Also worth checking on Bedrock

If you set max_observations_per_scope > 0, you'll want HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS=false as well — that's #2500, a separate Bedrock schema-keyword rejection with its own flag.

On your follow-up

Agreed that POST /banks/{bank}/health/llm being behind HINDSIGHT_API_ENABLE_BANK_LLM_HEALTH made this slower to triage than it needed to be — it probes exactly the path that breaks. Filing that separately rather than folding it in here.

Closing as fixed. Please reopen (or ping here) if 0.9.1 with the flag set doesn't clear it in ap-southeast-2 — we can reproduce that region on demand now.

[ffma-glen-sampson]

Fantastic thanks will remove my monkey patch and upgrade