fix: clamp dispatcher confidence score to valid range [0.0, 1.0]

[stuagano]

Problem

LLMs occasionally return confidence scores > 1.0 (e.g., 1.2) when highly confident in their intent classification. This causes Pydantic validation errors since the DispatcherResponse schema constrains confidence to [0, 1].

Error observed:

pydantic_core._pydantic_core.ValidationError: 1 validation error for DispatcherResponse
confidence
  Input should be less than or equal to 1 [type=less_than_equal, input_value=1.2, input_type=float]

This error crashes the dispatcher service during chat intent detection, preventing users from interacting with the chat interface.

Solution

Clamp the confidence value to the valid range [0.0, 1.0] before creating the DispatcherResponse object:

# Clamp confidence to [0.0, 1.0] range (LLM sometimes returns >1.0)
confidence = min(max(float(intent_result["confidence"]), 0.0), 1.0)

This fix:

• ✅ Prevents validation errors
• ✅ Preserves the intent that the model was highly confident
• ✅ Handles edge cases (negative values, values > 1.0)
• ✅ Maintains backward compatibility

Changes

• Added confidence clamping in dispatcher_service.py line 948
• Added explanatory comment about LLM behavior

Testing

• Verified fix resolves validation error
• Tested with confidence values: 0.0, 0.5, 1.0, 1.2 (clamped to 1.0)
• Confirmed chat interface works after fix
• Redeployed to Databricks Apps successfully

Impact

• Users: Can now use chat interface without crashes
• System: More robust handling of LLM responses
• Compatibility: No breaking changes

Fixes dispatcher crashes during chat intent detection.