Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google VertexAI InvalidArgument: 400 Request contains an invalid argument #30428

Closed
5 tasks done
khteh opened this issue Mar 22, 2025 · 4 comments
Closed
5 tasks done

Google VertexAI InvalidArgument: 400 Request contains an invalid argument #30428

khteh opened this issue Mar 22, 2025 · 4 comments
Labels
🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature investigate Flagged for investigation.

Comments

@khteh
Copy link

khteh commented Mar 22, 2025

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangChain documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

    async for event in agent.astream(
        #{"messages": [{"role": "user", "content": messages}]},
        {"messages": [("user", messages)]},
        stream_mode="values",
        config=config, # This is needed by Checkpointer
    ):
        event["messages"][-1].pretty_print()

await email_parser_chain.ainvoke({"message": email})
await email_parser_chain.ainvoke({"message": [{"role": "system", "content": email}]}

Error Message and Stack Trace (if applicable)

2025-03-21 16:12:03 WARNING  Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
2025-03-21 16:12:07 WARNING  Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
2025-03-21 16:12:11 WARNING  Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
2025-03-21 16:12:16 WARNING  Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 8.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
2025-03-21 16:12:24 WARNING  Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 10.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Traceback (most recent call last):
  File "/home/khteh/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 77, in wait_for_connection
    await self._call.wait_for_connection()
  File "/home/khteh/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 659, in wait_for_connection
    await self._raise_for_status()
  File "/home/khteh/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 272, in _raise_for_status
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Request contains an invalid argument."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.68.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2025-03-21T16:12:34.945102813+08:00"}"
>

Description

Any call to Google VertexAI with ainvoke, astream, with_structured_output hits that error.

System Info

System Information
------------------
> OS:  Linux
> OS Version:  #19-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 21:43:43 UTC 2025
> Python Version:  3.12.7 (main, Feb  4 2025, 14:46:03) [GCC 14.2.0]

Package Information
-------------------
> langchain_core: 0.3.45
> langchain: 0.3.20
> langchain_community: 0.3.19
> langsmith: 0.3.15
> langchain_google_genai: 2.1.0
> langchain_google_vertexai: 2.0.9
> langchain_openai: 0.3.8
> langchain_text_splitters: 0.3.6
> langgraph_api: 0.0.28
> langgraph_cli: 0.1.75
> langgraph_license: Installed. No version info available.
> langgraph_sdk: 0.1.57
> langgraph_storage: Installed. No version info available.

Optional packages not installed
-------------------------------
> langserve

Other Dependencies
------------------
> aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
> anthropic[vertexai]: Installed. No version info available.
> async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
> click: 8.1.8
> cryptography: 43.0.3
> dataclasses-json<0.7,>=0.5.7: Installed. No version info available.
> filetype: 1.2.0
> google-ai-generativelanguage: 0.6.16
> google-cloud-aiplatform: 1.84.0
> google-cloud-storage: 2.19.0
> httpx: 0.27.2
> httpx-sse: 0.4.0
> httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
> jsonpatch<2.0,>=1.33: Installed. No version info available.
> jsonschema-rs: 0.20.0
> langchain-anthropic;: Installed. No version info available.
> langchain-aws;: Installed. No version info available.
> langchain-cohere;: Installed. No version info available.
> langchain-community;: Installed. No version info available.
> langchain-core<1.0.0,>=0.3.34: Installed. No version info available.
> langchain-core<1.0.0,>=0.3.41: Installed. No version info available.
> langchain-core<1.0.0,>=0.3.42: Installed. No version info available.
> langchain-deepseek;: Installed. No version info available.
> langchain-fireworks;: Installed. No version info available.
> langchain-google-genai;: Installed. No version info available.
> langchain-google-vertexai;: Installed. No version info available.
> langchain-groq;: Installed. No version info available.
> langchain-huggingface;: Installed. No version info available.
> langchain-mistralai: Installed. No version info available.
> langchain-mistralai;: Installed. No version info available.
> langchain-ollama;: Installed. No version info available.
> langchain-openai;: Installed. No version info available.
> langchain-text-splitters<1.0.0,>=0.3.6: Installed. No version info available.
> langchain-together;: Installed. No version info available.
> langchain-xai;: Installed. No version info available.
> langchain<1.0.0,>=0.3.20: Installed. No version info available.
> langgraph: 0.3.11
> langgraph-checkpoint: 2.0.20
> langsmith-pyo3: Installed. No version info available.
> langsmith<0.4,>=0.1.125: Installed. No version info available.
> langsmith<0.4,>=0.1.17: Installed. No version info available.
> numpy<3,>=1.26.2: Installed. No version info available.
> openai-agents: Installed. No version info available.
> openai<2.0.0,>=1.58.1: Installed. No version info available.
> orjson: 3.10.15
> packaging: 24.2
> packaging<25,>=23.2: Installed. No version info available.
> pydantic: 2.9.2
> pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available.
> pydantic<3.0.0,>=2.5.2;: Installed. No version info available.
> pydantic<3.0.0,>=2.7.4: Installed. No version info available.
> pydantic<3.0.0,>=2.7.4;: Installed. No version info available.
> pyjwt: 2.10.1
> pytest: 8.3.2
> python-dotenv: 1.0.1
> PyYAML>=5.3: Installed. No version info available.
> requests: 2.32.3
> requests-toolbelt: 1.0.0
> requests<3,>=2: Installed. No version info available.
> rich: 13.9.4
> SQLAlchemy<3,>=1.4: Installed. No version info available.
> sse-starlette: 2.1.3
> starlette: 0.46.1
> structlog: 25.1.0
> tenacity: 9.0.0
> tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
> tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
> tiktoken<1,>=0.7: Installed. No version info available.
> typing-extensions>=4.7: Installed. No version info available.
> uvicorn: 0.34.0
> watchfiles: 1.0.4
> zstandard: 0.23.0
@langcarl langcarl bot added the investigate Flagged for investigation. label Mar 22, 2025
@dosubot dosubot bot added the 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature label Mar 22, 2025
@ccurme
Copy link
Collaborator

ccurme commented Mar 22, 2025

Can you provide a minimal reproducible example? ainvoke seems to work fine:

from langchain_google_vertexai import ChatVertexAI

llm = ChatVertexAI(model="gemini-1.5-flash-001")
await llm.ainvoke("hi")

@khteh
Copy link
Author

khteh commented Mar 22, 2025

I think you have to use / go through CompiledGraph to reproduce the error.

@khteh
Copy link
Author

khteh commented Mar 22, 2025

This is the minimum I can put together:

import os, vertexai, asyncio, logging
from dotenv import load_dotenv
from datetime import datetime
from langchain.chat_models import init_chat_model
from langgraph.prebuilt import ToolNode, tools_condition, create_react_agent, InjectedStore
from langgraph.checkpoint.memory import MemorySaver
from langgraph.store.memory import InMemoryStore
from langchain_core.runnables import RunnableConfig
load_dotenv()
from src.rag_agent.Tools import TOOLS
async def main():
    llm = init_chat_model("gemini-2.0-flash", model_provider="google_vertexai", streaming=True).bind_tools(TOOLS)
    agent = create_react_agent(llm, store=InMemoryStore(), tools=TOOLS, checkpointer=MemorySaver(), name="RAG ReAct Agent")
    config = RunnableConfig(run_name="RAG ReAct Agent", thread_id=datetime.now())
    messages = ["What is the standard method for Task Decomposition?", "Once you get the answer, look up common extensions of that method."]
    async for event in agent.astream(
        {"messages": [{"role": "user", "content": messages}]},
        stream_mode="values", # Use this to stream all values in the state after each step.
        config=config, # This is needed by Checkpointer
    ):
        event["messages"][-1].pretty_print()

if __name__ == "__main__":
    vertexai.init(project=os.environ.get("GOOGLE_CLOUD_PROJECT"), location=os.environ.get("GOOGLE_CLOUD_LOCATION"))
    asyncio.run(main())

Running it:

$ python whatever.py 
================================ Human Message =================================

['What is the standard method for Task Decomposition?', 'Once you get the answer, look up common extensions of that method.']
Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 4.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 8.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Retrying langchain_google_vertexai.chat_models._acompletion_with_retry.<locals>._completion_with_retry_inner in 10.0 seconds as it raised InvalidArgument: 400 Request contains an invalid argument..
Traceback (most recent call last):
  File "/home/khteh/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 77, in wait_for_connection
    await self._call.wait_for_connection()
  File "/home/khteh/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 659, in wait_for_connection
    await self._raise_for_status()
  File "/home/khteh/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 272, in _raise_for_status
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Request contains an invalid argument."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.24.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2025-03-22T19:48:06.353774327+08:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/Python/rag-agent/whatever.py", line 25, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/src/Python/rag-agent/whatever.py", line 16, in main
    async for event in agent.astream(
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2313, in astream
    async for _ in runner.atick(
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
    await arun_with_retry(
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
    return await task.proc.ainvoke(task.input, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 583, in ainvoke
    input = await step.ainvoke(input, config, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 359, in ainvoke
    ret = await asyncio.create_task(coro, context=context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langgraph/prebuilt/chat_agent_executor.py", line 705, in acall_model
    response = cast(AIMessage, await model_runnable.ainvoke(state, config))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3071, in ainvoke
    input = await asyncio.create_task(part(), context=context)  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 5377, in ainvoke
    return await self.bound.ainvoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 328, in ainvoke
    llm_result = await self.agenerate_prompt(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 853, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 813, in agenerate
    raise exceptions[0]
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 981, in _agenerate_with_cache
    result = await self._agenerate(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_google_vertexai/chat_models.py", line 1527, in _agenerate
    return await agenerate_from_stream(stream_iter)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 118, in agenerate_from_stream
    chunks = [chunk async for chunk in stream]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_google_vertexai/chat_models.py", line 1669, in _astream
    async for response_chunk in await response_iter:
                                ^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_google_vertexai/chat_models.py", line 645, in _acompletion_with_retry
    return await _completion_with_retry_inner(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/khteh/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/langchain_google_vertexai/chat_models.py", line 638, in _completion_with_retry_inner
    return await generation_method(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/khteh/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 178, in error_remapped_callable
    await call.wait_for_connection()
  File "/home/khteh/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 79, in wait_for_connection
    raise exceptions.from_grpc_error(rpc_error) from rpc_error
google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.
During task with name 'agent' and id '5efef15b-dd88-dcb3-ac66-90a5111b0d75'

Full-blown code: https://github.com/khteh/rag-agent/blob/master/src/rag_agent/RAGAgent.py

@khteh
Copy link
Author

khteh commented Apr 4, 2025

googleapis/python-aiplatform#5083

@khteh khteh closed this as completed Apr 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature investigate Flagged for investigation.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@khteh @ccurme