max_retries Parameter in ChatMistralAI Class Is Ineffective Due to Commented Code

[amarion35]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

The following code:

import string
import time
import random
import dotenv
from httpx import ReadTimeout
from langchain_mistralai import ChatMistralAI

dotenv.load_dotenv()

mistral = ChatMistralAI(timeout=1, max_retries=10)
test_input = "".join(random.choices(string.ascii_letters, k=10000))

t0 = time.time()
try:
    mistral.invoke(test_input)
except ReadTimeout:
    print("Timeout after", time.time() - t0, "seconds")

returns:

Timeout after 1.0581834316253662 seconds

Error Message and Stack Trace (if applicable)

---------------------------------------------------------------------------
ReadTimeout                               Traceback (most recent call last)
File project/.venv/lib/python3.11/site-packages/httpx/_transports/default.py:101, in map_httpcore_exceptions()
    100 try:
--> 101     yield
    102 except Exception as exc:

File project/.venv/lib/python3.11/site-packages/httpx/_transports/default.py:250, in HTTPTransport.handle_request(self, request)
    249 with map_httpcore_exceptions():
--> 250     resp = self._pool.handle_request(req)
    252 assert isinstance(resp.stream, typing.Iterable)

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py:256, in ConnectionPool.handle_request(self, request)
    255     self._close_connections(closing)
--> 256     raise exc from None
    258 # Return the response. Note that in this case we still have to manage
    259 # the point at which the response is closed.

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py:236, in ConnectionPool.handle_request(self, request)
    234 try:
    235     # Send the request on the assigned connection.
--> 236     response = connection.handle_request(
    237         pool_request.request
    238     )
    239 except ConnectionNotAvailable:
    240     # In some cases a connection may initially be available to
    241     # handle a request, but then become unavailable.
    242     #
    243     # In this case we clear the connection and try again.

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/connection.py:103, in HTTPConnection.handle_request(self, request)
    101     raise exc
--> 103 return self._connection.handle_request(request)

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py:136, in HTTP11Connection.handle_request(self, request)
    135         self._response_closed()
--> 136 raise exc

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py:106, in HTTP11Connection.handle_request(self, request)
     97 with Trace(
     98     "receive_response_headers", logger, request, kwargs
     99 ) as trace:
    100     (
    101         http_version,
    102         status,
    103         reason_phrase,
    104         headers,
    105         trailing_data,
--> 106     ) = self._receive_response_headers(**kwargs)
    107     trace.return_value = (
    108         http_version,
    109         status,
    110         reason_phrase,
    111         headers,
    112     )

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py:177, in HTTP11Connection._receive_response_headers(self, request)
    176 while True:
--> 177     event = self._receive_event(timeout=timeout)
    178     if isinstance(event, h11.Response):

File project/.venv/lib/python3.11/site-packages/httpcore/_sync/http11.py:217, in HTTP11Connection._receive_event(self, timeout)
    216 if event is h11.NEED_DATA:
--> 217     data = self._network_stream.read(
    218         self.READ_NUM_BYTES, timeout=timeout
    219     )
    221     # If we feed this case through h11 we'll raise an exception like:
    222     #
    223     #     httpcore.RemoteProtocolError: can't handle event type
   (...)
    227     # perspective. Instead we handle this case distinctly and treat
    228     # it as a ConnectError.

File project/.venv/lib/python3.11/site-packages/httpcore/_backends/sync.py:126, in SyncStream.read(self, max_bytes, timeout)
    125 exc_map: ExceptionMapping = {socket.timeout: ReadTimeout, OSError: ReadError}
--> 126 with map_exceptions(exc_map):
    127     self._sock.settimeout(timeout)

File ~/.pyenv/versions/3.11.6/lib/python3.11/contextlib.py:155, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    154 try:
--> 155     self.gen.throw(typ, value, traceback)
    156 except StopIteration as exc:
    157     # Suppress StopIteration *unless* it's the same exception that
    158     # was passed to throw().  This prevents a StopIteration
    159     # raised inside the "with" statement from being suppressed.

File project/.venv/lib/python3.11/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)
     13     if isinstance(exc, from_exc):
---> 14         raise to_exc(exc) from exc
     15 raise

ReadTimeout: The read operation timed out

The above exception was the direct cause of the following exception:

ReadTimeout                               Traceback (most recent call last)
Cell In[21], line 1
----> 1 mistral.invoke(test_input)

File project/.venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:284, in BaseChatModel.invoke(self, input, config, stop, **kwargs)
    273 def invoke(
    274     self,
    275     input: LanguageModelInput,
   (...)
    279     **kwargs: Any,
    280 ) -> BaseMessage:
    281     config = ensure_config(config)
    282     return cast(
    283         ChatGeneration,
--> 284         self.generate_prompt(
    285             [self._convert_input(input)],
    286             stop=stop,
    287             callbacks=config.get("callbacks"),
    288             tags=config.get("tags"),
    289             metadata=config.get("metadata"),
    290             run_name=config.get("run_name"),
    291             run_id=config.pop("run_id", None),
    292             **kwargs,
    293         ).generations[0][0],
    294     ).message

File project/.venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:860, in BaseChatModel.generate_prompt(self, prompts, stop, callbacks, **kwargs)
    852 def generate_prompt(
    853     self,
    854     prompts: list[PromptValue],
   (...)
    857     **kwargs: Any,
    858 ) -> LLMResult:
    859     prompt_messages = [p.to_messages() for p in prompts]
--> 860     return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)

File project/.venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:690, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs)
    687 for i, m in enumerate(messages):
    688     try:
    689         results.append(
--> 690             self._generate_with_cache(
    691                 m,
    692                 stop=stop,
    693                 run_manager=run_managers[i] if run_managers else None,
    694                 **kwargs,
    695             )
    696         )
    697     except BaseException as e:
    698         if run_managers:

File project/.venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:925, in BaseChatModel._generate_with_cache(self, messages, stop, run_manager, **kwargs)
    923 else:
    924     if inspect.signature(self._generate).parameters.get("run_manager"):
--> 925         result = self._generate(
    926             messages, stop=stop, run_manager=run_manager, **kwargs
    927         )
    928     else:
    929         result = self._generate(messages, stop=stop, **kwargs)

File project/.venv/lib/python3.11/site-packages/langchain_mistralai/chat_models.py:545, in ChatMistralAI._generate(self, messages, stop, run_manager, stream, **kwargs)
    543 message_dicts, params = self._create_message_dicts(messages, stop)
    544 params = {**params, **kwargs}
--> 545 response = self.completion_with_retry(
    546     messages=message_dicts, run_manager=run_manager, **params
    547 )
    548 return self._create_chat_result(response)

File project/.venv/lib/python3.11/site-packages/langchain_mistralai/chat_models.py:464, in ChatMistralAI.completion_with_retry(self, run_manager, **kwargs)
    461         _raise_on_error(response)
    462         return response.json()
--> 464 rtn = _completion_with_retry(**kwargs)
    465 return rtn

File project/.venv/lib/python3.11/site-packages/langchain_mistralai/chat_models.py:460, in ChatMistralAI.completion_with_retry.<locals>._completion_with_retry(**kwargs)
    458     return iter_sse()
    459 else:
--> 460     response = self.client.post(url="/chat/completions", json=kwargs)
    461     _raise_on_error(response)
    462     return response.json()

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:1144, in Client.post(self, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1123 def post(
   1124     self,
   1125     url: URL | str,
   (...)
   1137     extensions: RequestExtensions | None = None,
   1138 ) -> Response:
   1139     """
   1140     Send a `POST` request.
   1141 
   1142     **Parameters**: See `httpx.request`.
   1143     """
-> 1144     return self.request(
   1145         "POST",
   1146         url,
   1147         content=content,
   1148         data=data,
   1149         files=files,
   1150         json=json,
   1151         params=params,
   1152         headers=headers,
   1153         cookies=cookies,
   1154         auth=auth,
   1155         follow_redirects=follow_redirects,
   1156         timeout=timeout,
   1157         extensions=extensions,
   1158     )

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:825, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
    810     warnings.warn(message, DeprecationWarning, stacklevel=2)
    812 request = self.build_request(
    813     method=method,
    814     url=url,
   (...)
    823     extensions=extensions,
    824 )
--> 825 return self.send(request, auth=auth, follow_redirects=follow_redirects)

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:914, in Client.send(self, request, stream, auth, follow_redirects)
    910 self._set_timeout(request)
    912 auth = self._build_request_auth(request, auth)
--> 914 response = self._send_handling_auth(
    915     request,
    916     auth=auth,
    917     follow_redirects=follow_redirects,
    918     history=[],
    919 )
    920 try:
    921     if not stream:

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:942, in Client._send_handling_auth(self, request, auth, follow_redirects, history)
    939 request = next(auth_flow)
    941 while True:
--> 942     response = self._send_handling_redirects(
    943         request,
    944         follow_redirects=follow_redirects,
    945         history=history,
    946     )
    947     try:
    948         try:

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:979, in Client._send_handling_redirects(self, request, follow_redirects, history)
    976 for hook in self._event_hooks["request"]:
    977     hook(request)
--> 979 response = self._send_single_request(request)
    980 try:
    981     for hook in self._event_hooks["response"]:

File project/.venv/lib/python3.11/site-packages/httpx/_client.py:1014, in Client._send_single_request(self, request)
   1009     raise RuntimeError(
   1010         "Attempted to send an async request with a sync Client instance."
   1011     )
   1013 with request_context(request=request):
-> 1014     response = transport.handle_request(request)
   1016 assert isinstance(response.stream, SyncByteStream)
   1018 response.request = request

File project/.venv/lib/python3.11/site-packages/httpx/_transports/default.py:249, in HTTPTransport.handle_request(self, request)
    235 import httpcore
    237 req = httpcore.Request(
    238     method=request.method,
    239     url=httpcore.URL(
   (...)
    247     extensions=request.extensions,
    248 )
--> 249 with map_httpcore_exceptions():
    250     resp = self._pool.handle_request(req)
    252 assert isinstance(resp.stream, typing.Iterable)

File ~/.pyenv/versions/3.11.6/lib/python3.11/contextlib.py:155, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    153     value = typ()
    154 try:
--> 155     self.gen.throw(typ, value, traceback)
    156 except StopIteration as exc:
    157     # Suppress StopIteration *unless* it's the same exception that
    158     # was passed to throw().  This prevents a StopIteration
    159     # raised inside the "with" statement from being suppressed.
    160     return exc is not value

File project/.venv/lib/python3.11/site-packages/httpx/_transports/default.py:118, in map_httpcore_exceptions()
    115     raise
    117 message = str(exc)
--> 118 raise mapped_exc(message) from exc

ReadTimeout: The read operation timed out

Description

The max_retries parameter of the ChatMistralAI class doesn't do anything because the two lines necessary for the retry decorator have been commented out from the completion_with_retry method here:

langchain/libs/partners/mistralai/langchain_mistralai/chat_models.py

Line 460 in 0ba03d8

# retry_decorator = _create_retry_decorator(self, run_manager=run_manager)

As a result, as you can see in my example, even though timeout is set to 1s and the max_retries to 10, the call fails after about 1s which suggest that it fails after only 1 try.

The lines were commented a year ago in this commit 53ac1eb in this PR #19420.

System Info

python -m langchain_core.sys_info

System Information
------------------
> OS:  Linux
> OS Version:  #1 SMP Tue Nov 5 00:21:55 UTC 2024
> Python Version:  3.11.6 (main, Nov 22 2023, 18:29:18) [GCC 9.4.0]

Package Information
-------------------
> langchain_core: 0.3.40
> langchain: 0.3.14
> langchain_community: 0.3.14
> langsmith: 0.2.11
> langchain_mistralai: 0.2.4
> langchain_ollama: 0.2.3
> langchain_openai: 0.3.0
> langchain_postgres: 0.0.13
> langchain_text_splitters: 0.3.5
> langgraph_sdk: 0.1.55

Optional packages not installed
-------------------------------
> langserve

Other Dependencies
------------------
> aiohttp: 3.11.11
> async-timeout: Installed. No version info available.
> dataclasses-json: 0.6.7
> httpx: 0.28.1
> httpx-sse: 0.4.0
> jsonpatch<2.0,>=1.33: Installed. No version info available.
> langsmith-pyo3: Installed. No version info available.
> langsmith<0.4,>=0.1.125: Installed. No version info available.
> numpy: 1.26.4
> ollama: 0.4.7
> openai: 1.59.7
> orjson: 3.10.14
> packaging<25,>=23.2: Installed. No version info available.
> pgvector: 0.3.6
> psycopg: 3.2.5
> psycopg-pool: 3.2.6
> pydantic: 2.10.5
> pydantic-settings: 2.7.1
> pydantic<3.0.0,>=2.5.2;: Installed. No version info available.
> pydantic<3.0.0,>=2.7.4;: Installed. No version info available.
> PyYAML: 6.0.2
> PyYAML>=5.3: Installed. No version info available.
> requests: 2.32.3
> requests-toolbelt: 1.0.0
> SQLAlchemy: 2.0.37
> sqlalchemy: 2.0.37
> tenacity: 9.0.0
> tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
> tiktoken: 0.8.0
> tokenizers: 0.21.0
> typing-extensions>=4.7: Installed. No version info available.
> zstandard: Installed. No version info available.

[andrasfe]

as per above, I un-commented code, validated that the retries work as expected, added unit test that validates the ChatMistralAI retry functionality, and added an integration test for the retries. I do not know why this was commented out a year ago - I couldn't find a reason for it. Will leave it to the maintainer but I don't see immediate harm in re-enabling it.