fix(bedrock-kb-retrieval): preserve non-ASCII text in tool results - #3821
fix(bedrock-kb-retrieval): preserve non-ASCII text in tool results#3821kimnamu wants to merge 2 commits into
Conversation
Knowledge Base content containing CJK / accented / emoji characters was serialized with json.dumps(...) using the default ensure_ascii=True, so QueryKnowledgeBases and ListKnowledgeBases returned \uXXXX escape sequences instead of readable text. This bloats tokens and makes the content unreadable to both the model and the user. Serialize with ensure_ascii=False in the two model/user-visible json.dumps calls. Add regression tests asserting CJK content survives round-trip without \u escaping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Status update (no action needed — just keeping this current):
Happy to rebase or adjust anything (e.g. drop the CHANGELOG entry) if preferred — thanks for your time reviewing! |
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
|
This pull request is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label. |
|
Re-verified after the stale bump: rebased locally onto current Still happy to adjust anything (e.g. drop the CHANGELOG entry) if that unblocks a review — thanks for keeping this open. |
…luate json.dumps defaults to ensure_ascii=True, which escapes CJK characters, emoji, and accented characters to \uXXXX sequences. This makes the output unreadable for non-Latin scripts. Add ensure_ascii=False so the result string preserves the original Unicode characters, matching the fix already applied to bedrock-kb-retrieval (PR awslabs#3821). Fixes awslabs#3994
|
This pull request is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label. |
|
Still relevant as of today's
Two things that may help a decision:
If this is only waiting on review bandwidth, the |
Thank you for maintaining the AWS MCP servers.
Fixes #3820
Summary
QueryKnowledgeBasesandListKnowledgeBasesserialized results withjson.dumps(...)(defaultensure_ascii=True), so non-ASCII KB content (CJK, accented, emoji) was returned as\uXXXXescapes — token bloat + unreadable payloads. This addsensure_ascii=Falseto the two user/model-visible serialization points (the MCP transport is already UTF-8).Changes
knowledgebases/retrieval.py:113,server.py:128— addensure_ascii=Falsetest_retrieval.py+test_server.py## [Unreleased]/### FixedCHANGELOG entryBefore / After
東京は日本の首都です。in tool output東京...(escaped)東京は日本の首都です。한국어 데이터 소스한국...한국어 데이터 소스json.loadsTests
40 passed,ruff checkclean. Reverting the source change (keeping the tests) makes the 2 new tests FAIL withassert '東京...' in '...\\u6771\\u4eac...', proving they catch the bug; restoring it → all pass.Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.