Fix s3a multi-bucket S3 routing - #5240
Open
VectorPeak wants to merge 1 commit into
Open
Conversation
VectorPeak
force-pushed
the
codex-s3a-multi-bucket-routing
branch
from
July 7, 2026 02:35
4ef5894 to
36c1113
Compare
Route s3a:// paths through the same bucket/key parser used for s3:// paths so multi-bucket readers and writers do not treat them as default-bucket relative keys. Co-authored-by: chatgpt-codex-connector[bot] <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
VectorPeak
force-pushed
the
codex-s3a-multi-bucket-routing
branch
from
July 7, 2026 12:28
36c1113 to
e78ebca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
What Problem This Solves
MinerU's S3 path handling currently has two different ideas of what counts as an absolute S3 object-store URI.
parse_s3path()already treats boths3://bucket/keyands3a://bucket/keyas valid S3-style paths. It strips query-like range parameters, splits the URI at://, and returns the declared bucket and object key:Before this PR, the multi-bucket reader and writer only called that parser when the input started with
s3://:That means a valid S3A path such as:
was not handled as an absolute S3 URI. It fell into the relative-path branch, selected the configured default bucket, and could send the full
s3a://target-bucket/dir/file.pdfstring, optionally prefixed bydefault_prefix, as the object key.For a multi-bucket reader configured with:
The intended routing is:
The previous routing could become:
This is especially easy to hit when paths come from Hadoop, Spark, or data-lake workflows, where
s3a://is a common URI scheme for S3-compatible storage.Changes
This PR adds a shared
is_s3_path()helper backed by the same supported schemes used byparse_s3path():MultiBucketS3DataReader.read_at()andMultiBucketS3DataWriter.write()now use that helper before dispatching toparse_s3path(). As a result,s3://ands3a://paths go through the same bucket/key parser, while ordinary relative paths still use the configured default bucket and optional default prefix.The reader docstring is also updated to mention
s3a://as a supported input form, and focused unit tests cover the reader and writer S3A routing behavior without requiring live S3 credentials.Evidence
The bug is visible at the dispatch boundary:
After this change, the same
s3a://input satisfiesis_s3_path()and is routed throughparse_s3path(), so the declared bucket and key are preserved.Validation run locally on Windows with Python 3.12.13:
Possible call chain / impact
The affected multi-bucket read path is:
The affected write path is:
This PR only changes S3 URI dispatch for
s3a://paths. It does not change low-level S3 client behavior, credentials, endpoint selection, range parsing, relative-path behavior, default bucket/default prefix behavior, or existings3://routing.Modification
is_s3_path()helper using the same supported URI schemes asparse_s3path().MultiBucketS3DataReader.read_at()andMultiBucketS3DataWriter.write()sos3a://paths are parsed into their declared bucket/key.s3a://paths.s3a://routing.BC-breaking (Optional)
No backward compatibility break is expected. Existing
s3://paths and relative paths keep the same behavior. The only behavior change is that strings beginning withs3a://are now treated as S3 URIs, matching the existingparse_s3path()contract, instead of being treated as default-bucket relative object keys.Use cases (Optional)
This helps users who pass S3A-style paths from Hadoop, Spark, or data lake workflows into MinerU's multi-bucket S3 reader/writer:
After this change, that path is routed to
target-bucketwith keydocuments/input.pdfinstead of being interpreted under the configured default bucket.Checklist
Before PR:
After PR: