standard-tests: structured output tests assume ls_structured_output_format implementation

[mattf]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

n/a

Error Message and Stack Trace (if applicable)

FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[pydantic] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[typeddict] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[json_schema] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[pydantic] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[typeddict] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[json_schema] - KeyError: 'ls_structured_output_format'

Description

while building a ChatModel integration with structured output support, i'm running into these failures -

FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[pydantic] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[typeddict] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output[json_schema] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[pydantic] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[typeddict] - KeyError: 'ls_structured_output_format'
FAILED tests/integration_tests/test_chat_models.py::TestChatIntegration::test_structured_output_async[json_schema] - KeyError: 'ls_structured_output_format'

the issue appears to be that the tests assume implementation details, specifically that a ls_structured_output_format param is present.

System Info

n/a

[ccurme]

Hi @mattf, this is deliberate (we should pin langchain-tests in test dependencies if we aren't already). When we call .bind() in with_structured_output, we just need to pass in the schema and method. Are you working here? https://github.com/langchain-ai/langchain-nvidia/blob/49357d84d53ca29a8093e6a2732a147ff8545c09/libs/ai-endpoints/langchain_nvidia_ai_endpoints/chat_models.py#L880

You can see an example here: langchain-ai/langchain-google@16f4e81

Schema can be Pydantic model, TypedDict, JSON schema. You can call this function on the schema when passing it in to be safe:

from langchain_core.utils.function_calling import convert_to_openai_tool

convert_to_openai_tool(schema)

[mattf]

@ccurme understood. thanks for the pointers.