Skip to content

fix(wecombot): deliver sandbox outbox media through full pipeline chain#2328

Open
MIKAZE3 wants to merge 6 commits into
langbot-app:masterfrom
MIKAZE3:fix/wecombot-image-delivery
Open

fix(wecombot): deliver sandbox outbox media through full pipeline chain#2328
MIKAZE3 wants to merge 6 commits into
langbot-app:masterfrom
MIKAZE3:fix/wecombot-image-delivery

Conversation

@MIKAZE3

@MIKAZE3 MIKAZE3 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

概述 / Overview

修复企业微信智能机器人(WeComBot)无法发送沙箱(Box Sandbox)生成的图片/语音/文件的问题。完整链路涉及 6 个独立的修复点:

  1. 媒体上传协议对齐:将 WebSocket 上传命令名从 aibot_upload_init/chunk/finish 对齐为 aibot_upload_media_init/chunk/finish,字段名对齐协议规范(type/filename/total_size/md5/base64_data),支持 data: 前缀的 base64 剥离。

  2. _send_media 集成 + import base64 缺失_send_media 方法已定义但从未被 reply_message / reply_message_chunk 调用,且缺少 import base64 导致 NameErrorexcept Exception 静默吞掉。将 yiri2target 改为返回组件 dict 列表(text/image/voice/file),在 reply_messagereply_message_chunkis_final 时)中遍历媒体组件调用 _send_media

  3. _send_media 诊断日志:所有失败路径都是 except Exception: return False 无任何日志,完全无法诊断。改为实例方法,每个失败路径添加 warning 日志,成功添加 info 日志。

  4. 空内容 final chunk 跳过 outbox 收集:流式输出中最后一个 chunk(is_final=True)可能无文本内容(LLM 已在前面的 chunk 发完所有文字),if result.content: 为 False 时整个分支跳过,_append_outbound_attachments 从未执行。新增 elif _is_final_assistant_message 分支处理此场景。

  5. stdout 截断导致 outbox 数据丢失(根因)_read_outbox_via_exec 通过 execute_tool 读取沙箱文件,但返回的 stdout 被 _serialize_result 截断到 output_limit_chars=4000 字符。7KB JPEG 的 base64+JSON 约 9400 字符,截断后 json.loads 解析失败,静默返回空列表。改为直接调用 client.execute 获取原始未截断的 stdout。

  6. 全链路诊断日志:在 wrapper(chunk 元数据、outbox 收集结果、reply_chain 大小)、respback(链组件类型、is_final、流式支持)、wecombot 适配器(items 类型、媒体数量、文本长度、每个媒体的发送结果)添加 INFO 级别日志,覆盖每一个决策点。

更改前后对比截图 / Screenshots

修改前 / Before:

  • LLM 在沙箱中生成图片并写入 outbox,但图片从未发送给用户
  • 日志显示 LangBot Box result 成功读取了 base64 数据,但 outbox: collected 0 attachments: []
  • 无任何错误日志输出,问题完全静默
image

修改后 / After:

  • 沙箱生成的图片通过三步上传协议(init → chunk × N → finish)上传到企业微信 CDN,再通过 reply_image 发送给用户
  • 全链路日志可追踪:outbox: collected 1 attachments: ['Image']reply_message_chunk: media_count=1_send_media: sent image media_id=xxx
image

检查清单 / Checklist

PR 作者完成 / For PR author

请在方括号间写x以打勾 / Please tick the box with x

  • 阅读仓库贡献指引了吗? / Have you read the contribution guide?
  • 我已签署或将在机器人提示后签署 CLA。 / I have signed, or will sign when prompted by the bot, the CLA.
  • 与项目所有者沟通过了吗? / Have you communicated with the project maintainer?
  • 我确定已自行测试所作的更改,确保功能符合预期。 / I have tested the changes and ensured they work as expected.

项目维护者完成 / For project maintainer

  • 相关 issues 链接了吗? / Have you linked the related issues?
  • 配置项写好了吗?迁移写好了吗?生效了吗? / Have you written the configuration items? Have you written the migration? Has it taken effect?
  • 依赖加到 pyproject.toml 和 core/bootutils/deps.py 了吗 / Have you added the dependencies to pyproject.toml and core/bootutils/deps.py?
  • 文档编写了吗? / Have you written the documentation?

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. eh: Improve enhance: 现有功能的改进 / improve current features IM: wecom 企业微信 适配器相关 / WeCom and WeComCS adapter related m: Platform 机器人管理相关 / Bots management labels Jul 10, 2026

@dadachann dadachann left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里仍混入了 monitoring.py,与 WeCom 媒体投递无关,请移出。分支也已经和 master 冲突,请 rebase 后保留当前 HITL / synthetic-event 逻辑,并清掉临时 debug 日志。

@MIKAZE3
MIKAZE3 force-pushed the fix/wecombot-image-delivery branch from c7cece1 to 4497d95 Compare July 16, 2026 10:00
@MIKAZE3

MIKAZE3 commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

两个测试失败的原因和修复:

  1. test_wrapper.py::test_assistant_empty_content :Mock 对象的 is_final 返回 truthy Mock,触发了新增的 elif 分支。修复:将 elif 条件从 _is_final_assistant_message(result) 改为严格检查 isinstance(result, MessageChunk) and result.is_final and not result.tool_calls 。
  2. test_box_service.py::test_collect_outbound_reads_and_clears :测试 mock 了 execute_tool ,但实现改成了 client.execute 。修复:更新测试 mock client.execute ,返回 BoxExecutionResult 实例。

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.22930% with 75 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/langbot/pkg/platform/sources/wecombot.py 41.66% 49 Missing ⚠️
src/langbot/libs/wecom_ai_bot_api/ws_client.py 65.51% 20 Missing ⚠️
src/langbot/pkg/pipeline/wrapper/wrapper.py 20.00% 4 Missing ⚠️
src/langbot/pkg/box/service.py 80.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@MIKAZE3
MIKAZE3 requested a review from dadachann July 17, 2026 04:34
fdc310 and others added 6 commits July 17, 2026 12:52
- Integrate _send_media into reply_message and reply_message_chunk so
  sandbox outbox images/voices/files are uploaded and sent instead of
  being silently dropped.
- Add missing import base64 that caused _send_media to fail with a
  NameError swallowed by its except clause.
- Change yiri2target to return component dicts (text/image/voice/file)
  so callers can distinguish text from media.
- Fix _get_message_for_tool_context using result.first()/row[0] which
  returned a raw string instead of the ORM object, causing
  "'str' object has no attribute 'pipeline_id'" in tool call recording.
  Use result.scalars().first() per SQLAlchemy 2.0 convention.
All failure paths in _send_media silently returned False without any
log output, making it impossible to diagnose why images were not
delivered. Convert from @staticmethod to instance method and add
warning/info logging at every exit point.
…ontent

When the last streaming chunk has is_final=True but empty content
(e.g. the LLM sends all text in earlier chunks), the 'if result.content'
branch is skipped entirely, so _append_outbound_attachments never runs
and sandbox outbox images are silently dropped.

Add an elif branch for _is_final_assistant_message that creates an
empty MessageChain and still collects outbox attachments, so images
are delivered even when the final chunk carries no text.
_read_outbox_via_exec used execute_tool which returns _serialize_result
where stdout is truncated to output_limit_chars (4000). A 7KB JPEG
encodes to ~9400 base64 chars, so the JSON payload was truncated and
json.loads failed silently, returning an empty list.

Call client.execute directly to get the raw BoxExecutionResult with
untruncated stdout, so base64 file data is preserved.
… is_final check

- wrapper.py: restrict outbox collection on empty-content chunks to
  actual MessageChunk instances with is_final=True, not generic Mock
  objects that happen to have role='assistant'
- test_box_service.py: update _read_outbox_via_exec tests to mock
  client.execute (returning BoxExecutionResult) instead of
  execute_tool, matching the implementation change
@MIKAZE3
MIKAZE3 force-pushed the fix/wecombot-image-delivery branch from d5ab339 to 103b8fb Compare July 17, 2026 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eh: Improve enhance: 现有功能的改进 / improve current features IM: wecom 企业微信 适配器相关 / WeCom and WeComCS adapter related m: Platform 机器人管理相关 / Bots management size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants