feat(admin): add bucket migration commands for on-demand migration - #366
Merged
Merged
Conversation
Closes #364. Add `rc admin bucket migration set|get|rm|status` and `backfill start|cancel|status` for the RustFS On-Demand Migration admin routes under /rustfs/admin/v3/on-demand-migration/{bucket}. - core: request/response types with every response field optional and defaulted, credential values dropped at the parsing boundary, local validation mirroring the server bounds, and the OnDemandMigrationApi trait. Parse tests read the wire fixtures vendored from rustfs/rustfs (crates/madmin/fixtures/on_demand_migration, commit 1a888708). - s3: SigV4 transport with the PUT body held in zeroizing storage, bounded responses, credential-scrubbed error bodies, and status mapping: 400 source-unreachable -> network (3), other 400 -> usage (2), 401/403 -> auth (4), known 404 codes -> not found (5), other 404 -> unsupported (7, "server does not support on-demand migration"), 409 -> conflict (6), 501 -> unsupported (7). - cli: secret taken from --secret-key, RC_ODM_SECRET_KEY, or a hidden prompt; the REDACTED placeholder is refused because set replaces the configuration wholesale. Human tables render a null served_by_source_ratio as an em dash; --watch refreshes a progress line until the backfill job reaches a terminal state. JSON output uses schema v3 family on_demand_migration. - docs, README, output schema v3, fixtures, help contract and binary tests updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
Closes #364.
Background
RustFS server ships On-Demand Migration: a bucket names an external S3-compatible source bucket, a GET that misses locally is served from the source and stored locally, and a background backfill job pulls the rest. The server side is complete;
rchad no command surface for the/rustfs/admin/v3/on-demand-migration/{bucket}route family, so operators had to drive it withawscurl.Solution
Adds
rc admin bucket migration set|get|rm|statusandrc admin bucket migration backfill start|cancel|status, following the existing core / s3 / cli layering.crates/core/src/admin/on_demand_migration.rs): request and response types, theOnDemandMigrationApitrait, and local validation mirroring the server bounds. Every response field is optional with the server default, so an older server that omits fields still parses and unknown fields from a newer server are ignored. Credential values are dropped at the parsing boundary; only their presence survives.crates/s3/src/admin/on_demand_migration.rs): SigV4 transport. ThePUTbody is held in zeroizing storage and handed to the HTTP client by ownership; responses are bounded and error bodies are credential-scrubbed. Status mapping: 400OnDemandMigrationSourceUnreachable→ network (3), other 400 → usage (2), 401/403 → auth (4), 404 with a known code → not found (5), any other 404 → unsupported (7) withserver does not support on-demand migration, 409 → conflict (6), 501 → unsupported (7).crates/cli/src/commands/admin/bucket.rs): the secret is taken from--secret-key, thenRC_ODM_SECRET_KEY, then a hidden prompt (human mode on a terminal only). TheREDACTEDplaceholder is refused becausesetreplaces the configuration wholesale.set --dry-runmaps toPUT ...?dry-run=true. Human output renders one key/value table per document; a nullserved_by_source_ratiorenders as an em dash, never zero.--watchrefreshes a progress line until the backfill job reaches a terminal state, or streams one JSON record per refresh with--json.rustfs/rustfscrates/madmin/fixtures/on_demand_migration/at commit1a88870809896c989465540b519afc74731d7f33, with provenance in a README. Parse and transport tests read those files rather than hand-written literals.on_demand_migrationfamily (data: {operation, bucket, result}) with success/empty/error fixtures.docs/reference/rc/admin.mdand the README document the workflow, secret handling, output, and exit codes.Deviation from the issue
The issue lists exit code 5 for a 409 backfill conflict.
crates/cli/src/exit_code.rsis a protected contract where 5 is not-found and 6 is conflict, so a 409 exits 6 here. Changing that would need the Breaking Change process.Tests
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warnings(zero warnings)cargo test --workspace(all 64 test binaries pass)New coverage: 11 core parse/validation tests against the vendored fixtures, 13 s3 transport tests against a local HTTP server (routes, query encoding, signed headers, plaintext body, status mapping, credential scrubbing), 11 cli unit tests (parsing, secret resolution, exit codes via a fake API, watch termination), 9 binary-level tests in
crates/cli/tests/admin_bucket_migration.rs, plus help-contract and schema-contract cases.