Docs agent loops on grep/search variations within a single turn, exceeding its own 2-round cap#593
Draft
langsmith-engine[bot] wants to merge 1 commit into
Draft
Conversation
Adds a pre-flight checklist to the docs agent prompt and a new DedupeToolCallsMiddleware that short-circuits any (tool_name, args) signature already issued since the latest HumanMessage. Together this enforces the prompt's 2-round cap that was previously advisory only. - src/prompts/docs_agent_prompt.py: new mandatory pre-flight check step covering identical-args repeats, shell-syntax-variation retries on the same .mdx file, and synonym/case/plural restatements. - src/middleware/dedupe_tool_calls_middleware.py: new middleware that tracks (tool_name, canonical_json(args)) per turn (keyed on the last HumanMessage index) and returns a synthetic ToolMessage on dupes. - src/agent/docs_graph.py: register dedupe middleware before tool_retry_middleware so duplicate signatures are skipped before retries run. - src/middleware/__init__.py: export DedupeToolCallsMiddleware. - tests/unit/test_dedupe_tool_calls_middleware.py: cover the four contract points (dupe short-circuit, different args not deduped, arg key ordering, new HumanMessage resets the window).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The docs agent repeatedly violates the system prompt's hard cap of 2 search/read rounds per turn by retrying
query_docs_filesystem_docs_by_lang_chainwith alternate shell-syntax variations (rg→grep→head→cat) on the same.mdxfile when the first command returns no matches, and by firingsearch_docs_by_lang_chainwith synonym/case-only restatements of the same concept after ambiguous results. In one trace (trace 1) the agent fired the exact samesearch_docs_by_lang_chain({"query":"command"})+search_support_articles({...})pair twice within one turn separated only by a single fs-read round; in another (trace 2) it ran 9 tool rounds in one turn cycling throughtracing→callbacks→console logging→stdout callback→ConsoleCallbackHandler. Final answers are typically still correct, but each looping turn burns 5–10× the intended token budget and inflates latency well above the p99 of 35s.