Bulk request agents on the same runner

[stonkmate-lkh]

Hi there!

First of thanks for the community calls, great to see the people behind the scenes.

I have a question about how to bulk request a root agent within the same runner. Usually, I’ve had use cases where I read a CSV (or similar) and then make a request for each row in the file. Using REST endpoints, this was pretty straightforward; however, I’m unsure if this is supported in ADK, and if so, how you would approach it.

The reason I’m asking is that ADK seems tailored to chat applications with sessions. In this case, the use doesn’t require any context handling or content persistence, but I’d still like to use the framework to combine tools and other LLMs.

Currently, I’m doing it like this: I create one runner and have a process_row_task function that creates a unique session for each request to avoid context conflicts. Then it runs runner.run_async and returns the results.

Is there a smarter or better way to do this?

    # Limit concurrent in-flight generations
    concurrency_limit = 50
    semaphore = asyncio.Semaphore(concurrency_limit)
    runner = InMemoryRunner(agent=root_agent, app_name="pdp_writer")

    for csv_path in csv_paths:
        tasks: list[asyncio.Task] = []
        df_products = pd.read_csv(csv_path)
        for idx, row in df_products.iterrows():
            tasks.append(
                asyncio.create_task(
                    process_row_task(
                        str(csv_path), idx, row.to_dict(), user_id, semaphore, runner
                    )
                )
            )
        results = await asyncio.gather(*tasks)
        for result in results:
            if result["final_text"]:
                df_products.at[result["row_index"], "LongDescription"] = result["final_text"]
    return df_products

[LukasKH]

Could it make sense to just have it hosted seperately instead?

So have Cloud Run run the agents, tools etc and then have a simple Cloud function pull in csv data, then concurrently call the cloud run function so you basically call the agent like a rest endpoint?

[hangfei]

@adk-bot

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: Your current approach is a solid and recommended pattern for bulk processing of independent tasks in ADK. Creating a unique session for each task is the correct way to ensure context isolation.

---

Hi there!

Thanks for the great question and for sharing your approach. It's great to see how you're using ADK for different use cases.

Your method of handling bulk requests is not only a valid way to approach this problem but is also aligned with the core design principles of ADK. Here’s a breakdown of why your approach is effective:

1. Asynchronous by Design: The ADK is built on asyncio, and runner.run_async is the primary method for executing agents. [1] This is ideal for handling I/O-bound operations like making LLM calls concurrently, which is exactly what you are doing in your process_row_task.

2. Session for Context Isolation: You are correct that sessions are used to handle the context of a conversation or task. For independent tasks like processing rows from a CSV, creating a unique session for each is the right way to ensure isolation. This prevents context from one task from leaking into another, which is crucial for getting correct and deterministic results. [2]

3. Concurrency Control: Your use of asyncio.Semaphore is a best practice for controlling concurrency. This is important for managing resources and avoiding overwhelming the system or hitting API rate limits from your LLM provider.

4. Flexibility: Your current approach of creating asyncio.Task for each runner.run_async call is very flexible. It allows you to use any type of agent (including LlmAgent) as your root agent, giving you the full power of the framework for each task.

An alternative for some use cases is the ParallelAgent, which is a workflow agent designed to execute a list of sub-agents concurrently when the tasks are independent. [3] However, it is deterministic and does not use an LLM to decide what to do. Given that you want to combine tools and other LLMs for each row, your current approach of using runner.run_async with a unique session per task is more appropriate and powerful.

So, to answer your question directly: your current method is a "smarter or better way to do this" for your use case. It correctly leverages ADK's asynchronous capabilities and session management to handle bulk, independent tasks efficiently.

Keep up the great work!

---

[1] https://google.github.io/adk-docs/runtime/
[2] https://google.github.io/adk-docs/sessions/
[3] https://google.github.io/adk-docs/agents/workflow-agents/parallel-agents/

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: Your current approach for bulk processing by creating a new session for each row in your CSV is a great way to handle this in ADK. It's an idiomatic and effective pattern for ensuring stateless, parallel execution.

---

Hi there! Thanks for the great question and for sharing your implementation.

What you are doing is indeed a very effective and sound way to handle bulk, stateless requests with ADK. Here’s a bit more detail on why your approach is on the right track:

• Session Management for Statelessness: Sessions in ADK are designed to manage context and conversation history. Since your use case involves processing independent rows from a CSV with no shared context, creating a unique session for each runner.run_async call is the correct approach. This guarantees that each task is completely isolated and avoids any potential context "bleeding" between your different data rows.

• Asynchronous Execution: Using runner.run_async is the intended method for high-performance and production scenarios in ADK. The ADK runtime is built on asyncio to handle concurrent operations efficiently, so leveraging asyncio.create_task and asyncio.gather as you are doing is perfectly idiomatic.

• Concurrency Control: Your use of asyncio.Semaphore is an excellent practice. When running a large number of concurrent requests, it's crucial to control the level of parallelism to avoid overwhelming your system's resources or hitting external API rate limits. This shows a robust implementation.

The alternative suggestion of deploying the agent as a separate service (e.g., on Cloud Run) and calling it via REST API is also a valid architectural pattern, especially for scalability and separating concerns. However, for processing within a single application runner, your current approach is efficient and well-designed.

Keep up the great work!