Skip to content

fix(prompt): calibrate response length to question scope to reduce verbosity#549

Open
langsmith-engine-dev[bot] wants to merge 1 commit into
masterfrom
auto/agent-fix-response-terseness
Open

fix(prompt): calibrate response length to question scope to reduce verbosity#549
langsmith-engine-dev[bot] wants to merge 1 commit into
masterfrom
auto/agent-fix-response-terseness

Conversation

@langsmith-engine-dev
Copy link
Copy Markdown
Contributor

Summary

  • Root cause: docs_agent_prompt.py instructs the agent to always "show working examples — copy-paste ready code" with no guidance on calibrating response length for follow-up clarifications, incremental change requests, or simple yes/no questions. This drives a 90% failure rate on the properly-terse-answer metric (18+ failures in a 100-trace sample).
  • Fix: Added a "Calibrate Response Length to Question Scope" section in the Writing Rules and corresponding DO/DON'T bullets that distinguish between full how-to questions (comprehensive answer appropriate) vs. follow-ups/incremental changes (show only the delta, be brief).
  • Scope: Minimal targeted addition to src/prompts/docs_agent_prompt.py — no code changes, no test changes.

Evidence traces (production)

All traces from the Chat-LangChain project showing overly-verbose responses:

Test plan

  • Existing unit tests pass (uv run pytest tests/unit/ -q — 7 passed, pre-existing failures in test_broken_doc_urls.py and test_check_links_async.py are unrelated)
  • No existing open PR covers this root cause
  • Change is minimal and scoped to the verbosity problem only
  • Monitor properly-terse-answer metric on next production trace batch

🤖 Generated with Claude Code

…rbosity

- Root cause: prompt instructed agent to always show working examples and copy-paste code without calibrating for question type (follow-ups, incremental changes, simple yes/no)
- Change: added response-length calibration guidance in docs_agent_prompt.py
- Verified: 90% properly-terse-answer failure rate observed across 18+ production traces

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants