chore(deps): update rust crate fs4 to v1#1837
Conversation
This comment has been minimized.
This comment has been minimized.
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b8201d4b0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| papaya = { version = "0.2", default-features = false } | ||
| tempfile = { version = "3", default-features = false } | ||
| fs4 = { version = "0.13.1", default-features = false } | ||
| fs4 = { version = "1.0.0", default-features = false } |
There was a problem hiding this comment.
Preserve quota-based disk limits on Windows
This upgrade resolves to fs4 1.1.0, whose Windows total_space now reports the full volume capacity while available_space remains caller/quota-scoped. The retry queue still combines those two values in lib/saluki-io/src/net/util/retry/queue/persisted.rs::on_disk_bytes_limit as available_space - total_space * (1 - storage_max_disk_ratio), so on quota-enabled Windows volumes (for example a 10 GB quota on a 1 TB disk with the default 80% ratio) the reserved amount is based on 1 TB and the limit saturates to 0, preventing persistence even though the user has quota space available. The dependency bump needs either pinning before that behavior change or adjusting the limit calculation to use comparable quota/volume values.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Looking at this locally, I agree that using fs4 1.10 (really, 1.0.1 and later) would very likely break things on Windows when quotas were in use. Just to drive it home, laying it all out in sort of a "if this, then that" way with tabular data:
A 1 TB volume, with the agent's data directory on it. I'll use the default
storage_max_disk_ratio = 0.8(so the queue reserves the bottom 20% of "total" as headroom). Two physical situations:
- No quota: 512 GB actually free on the volume.
- Per-user quota on the agent's account: limit 160 GB, 32 GB already used → 128 GB free to the caller (and still 512 GB free volume-wide).
Raw Windows values in each situation:
(quoted field names abbreviated to better fit in a PR comment)
FreeBytesAvailCallerTotalNumBytesTotalNumFreeBytesVolume Size No quota 512 GB 1000 GB 512 GB 1000 GB Quota 128 GB 160 GB (= quota limit) 512 GB 1000 GB How it flows through
on_disk_bytes_limitWe calculate the following values:
total = TotalNumBytesavailable = FreeBytesAvailCallerreserved = total * 0.2available_adj = (available_space − reserved).ceil()limit = min(max_on_disk_bytes, available_adj)
Case totalavailablereservedavailable_adjlimitA. No quota, 0.13.1 1000 GB 512 GB 200 GB 312 GB 312 GB ✅ B. No quota, 1.1.0 1000 GB 512 GB 200 GB 312 GB 312 GB ✅ C. Quota, 0.13.1 160 GB 128 GB 32 GB 96 GB 96 GB ✅ D. Quota, 1.1.0 1000 GB 128 GB 200 GB −72 GB 0 ❌ Two takeaways from the table:
- No quota → no change. A vs. B are identical (312 GB). On a quota-free volume,
TotalNumBytesalready equals the full volume, so swapping it for cluster geometry changes nothing. This is why the bug is invisible on most Windows hosts and on all of Linux.- Quota → regression. C (96 GB, scales sensibly to the quota) becomes D (0). The breakage is that
reservedis now computed off the 1 TB physical disk (200 GB) whileavailableis the 128 GB quota free — the reservation alone is bigger than the caller's entire free quota, so the subtraction goes negative and clamps to 0.
Forgive the clanker output being pasted verbatim, but it did a reasonable job explaining this succinctly.
Binary Size Analysis (Agent Data Plane)Baseline: 06d0a65 · Comparison: 498fcde · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (35)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
Edited/Blocked NotificationRenovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR. You can manually request rebase by checking the rebase/retry box above. |
|
Based on my comment on Codex's review, my recommendation is that we do not merge this change as-is. It seems like we'll need to either try and submit an upstream change to expose the additional information we need to calculate quota-aware disk usage, or roll our own thin wrapper to do it. Looking at the Datadog Agent's approach for calculating this, it does use the quota-aware measurements for both available and "total" space, so this would be a regression both in terms of matching the Datadog Agent and in the general sense of what Codex identified. |
This PR contains the following updates:
0.13.1→1.0.0Warning
Some dependencies could not be looked up. Check the Dependency Dashboard for more information.
Cargo major update — Before merging, verify the workspace compiles and check upstream release notes for removed features or API changes.
Release Notes
al8n/fs4 (fs4)
v1.1.0Changes
FileExtandAsyncFileExtinto single crate-root traits(
fs4::FileExt,fs4::AsyncFileExt) instead of generating a distincttrait per backend module. The per-backend modules (
fs4::tokio,fs4::async_std,fs4::smol,fs4::fs_err2,fs4::fs_err3,fs4::fs_err2_tokio,fs4::fs_err3_tokio) now re-export the unifiedcrate-root trait. Method-call sites that import the trait via
usecontinue to compile unchanged; code that named two backend traits as
distinct types will see them unify.
impl<F: FileExt + ?Sized> FileExt for &Fandimpl<F: AsyncFileExt + ?Sized> AsyncFileExt for &F, so theextension methods are now callable through shared references.
FileExtandAsyncFileExtvia a privatesealed::Sealedsupertrait, so the set of implementing types is closed to the
concrete file types fs4 already supports (and references to them).
This locks in the freedom to add methods to either trait in future
minor releases without breaking downstream impls.
DynAsyncFileExt, an object-safe mirror ofAsyncFileExtwhoseasync methods return
BoxFuture<'_, T>(alias forPin<Box<dyn Future<Output = T> + Send + '_>>). Use it whenevertype erasure is needed (
Box<dyn DynAsyncFileExt>,&dyn DynAsyncFileExt); prefer the staticAsyncFileExtforgeneric code since it has no allocation or dynamic-dispatch
overhead. Every type implementing
AsyncFileExtalso implementsDynAsyncFileExt, and the trait is sealed.#[inline(always)](skipped undertarpaulincoverage builds).v1.0.1Fixes
allocate: short-circuit on allocated blocks(
metadata().blocks() * 512 >= len) instead of logical EOF. Theprevious
metadata().len() >= lencheck silently turnedallocateinto a no-op on sparse files (logical length large, zero blocks
reserved), violating the documented preallocation guarantee. The
new check still skips the macOS
F_PREALLOCATEre-allocate-ENOSPCpath from #15, since it asks the right question: "are the blocks
already reserved?" Applies to both the sync and async
implementations.
statvfs: route the threeGetDiskFreeSpaceExWoutputscorrectly.
free_spacenow comes fromlpTotalNumberOfFreeBytes(volume-wide, quota-independent),
available_spacefromlpFreeBytesAvailable(caller-scoped, honours per-user quotas),and
total_spaceis computed from cluster math(
sectors_per_cluster * bytes_per_sector * total_number_of_clusters)so it reports volume capacity rather than the caller's quota. On
quota-enabled volumes the three fields now carry distinct,
documented meanings; previously
free_spaceandavailable_spacewere identical and
total_spaceunder-reported capacity.target_os = "fuchsia"to the Unixallocatefallocatebranch (sync and async). Fuchsia is
cfg(unix)under rustc andrustixexposesfallocatethere, so the previous omission leftthe Fuchsia Unix build without an
allocatesymbol onceFileExtwas enabled. With this fix, every fs4 feature builds on Fuchsia
except
fs-err3andfs-err3-tokio; those remain blocked onfs-err v3.3.0referencingstd::os::unix::fs::chroot, whichrustc gates out on
target_os = "fuchsia". The fs4 code no longerhas a Fuchsia gap — the remaining one is upstream
(andrewhickman/fs-err#90).
Testing
allocate_reserves_blocks_on_sparse_file(sync and async, Linux-gated) creates a sparse file via
set_len,asserts
allocated_size == 0, callsallocate, and assertsblocks are reserved.
Documentation
that the crate's declared MSRV (
rust-version = "1.75.0") coversthe default
syncfeature, and thatasync-std/smolinherita higher effective MSRV (1.85) from their transitive dependencies
(
async-lock,async-signal).v1.0.0Breakage
FileExt::lock_exclusive/AsyncFileExt::lock_exclusivetolock, matching the stabilized [std::fs::File::lock] API.FileExt::try_lock_exclusive/AsyncFileExt::try_lock_exclusiveto
try_lock, matching [std::fs::File::try_lock].try_lockandtry_lock_sharedfromstd::io::Result<bool>toResult<(), TryLockError>.Ok(())stillindicates the lock was acquired;
Err(TryLockError::WouldBlock)nowindicates the lock is held by another handle. This matches the stable
[
std::fs::File::try_lock] signature (Ok(false)was the nightlyshape prior to 1.89).
lock_contended_error()helper. UseTryLockError::WouldBlockinstead.fs_stdmodule: theFileExttrait forstd::fs::Filenow lives at the crate root. Update imports fromuse fs4::fs_std::FileExt;touse fs4::FileExt;. All otherbackends (
fs_err2,fs_err3,tokio,smol,async_std,fs_err2_tokio,fs_err3_tokio) remain nested, since eachdefines its own
FileExt/AsyncFileExtover a different concreteFiletype.Additions
fs4::TryLockErrorenum, mirroring [std::fs::TryLockError]with
Error(io::Error)andWouldBlockvariants. ImplementsDebug,Display,std::error::Error(withsource()exposingthe inner
io::Error),From<io::Error>(kindWouldBlockcollapses into the
WouldBlockvariant; everything else wraps intoError), andFrom<TryLockError> for io::Error(WouldBlockbecomes
io::Error::from(io::ErrorKind::WouldBlock);Error(inner)passes through verbatim).
Fixes
fs-err2,fs-err3,fs-err2-tokio, orfs-err3-tokioenabled(without
sync/tokio).#[cfg(feature = "fs-err")]andfeature = "fs-err-tokio"both referenced feature names that do notexist in
Cargo.toml.set_lencall when the file's existingcluster-aligned allocation already covers
len. Avoids the Windowsbuffered-I/O quirks observed when the end-of-file pointer is moved
inside an already-allocated cluster (#13). The trait doc now carves
this behavior out from the general "file size is at least
lenbytes" contract.
allocateon Unix when the file is already at leastlenbytes long. Fixes macOSfallocatespuriously returningENOSPCwhen re-callingallocate(len)on an existing file (#15).cygwinto theallocatetarget_osset so builds stopfailing with a missing
sys::allocatesymbol on that target (#44).redoxandcygwinto the asyncallocatefallback branchso it matches the sync variant (#43 follow-up).
fallocatebranch to theset_lenfallback branch, matching the sync implementation.
automatically when the owning file handle is closed (#23).
tests in
lib.rs(#16).std methods): because 1.0 renames the trait methods to match std
exactly,
std::fs::File::lock/try_lock/unlock(stable inRust 1.89+) now win via inherent dispatch and
unstable_name_collisionsno longer fires for std users on recent rustc.
lock_impl!macro with#[allow(unused_macros)]socargo clippy --no-default-features(filesystem-stats-only build)does not fail under
-D warnings.cfg_fs2_err/cfg_fs3_errfromlib.rs(unified withcfg_fs_err2/cfg_fs_err3).html_root_urlto the 1.0.0 docs.rs path.Dependency updates
windows-sysfrom 0.59 to 0.61.Testing
#[bench]functions from the test harness. Theymeasured OS syscalls (
flock,LockFileEx,fallocate,statvfs)rather than fs4 code, and produced numbers dominated by the
underlying filesystem. Dropping them lets the crate build and test
on stable Rust (the bench harness was the only thing pinning
nightly via
#![cfg_attr(test, feature(test))]).rust-toolchain.tomlis nowstable.allocate_preserves_eof_within_cluster— exercises the #13precondition (cluster-aligned
AllocationSizewith EOF insidethe cluster) and asserts EOF is not moved when
allocate(len)isre-called with
len <= allocated_sizeon Windows.allocate_idempotent— exercises the #15 precondition (back-to-backallocate(len)calls on macOS) to prevent a regression in theshort-circuit.
TryLockErrorunit tests covering variants,Display,std::error::Error::source,From<io::Error>,From<TryLockError> for io::Error, and round-trip preservation ofErrorKind.FsStatsgetter + derive unit tests (previously every testdestructured the struct, so the four getter method bodies were
never executed and showed as uncovered).
filesystem_spacetest assertedavailable_space <= free_space,which does not hold on macOS APFS. APFS counts purgeable space
(snapshots, cached data reclaimable on demand) in
f_bavailbutnot in
f_bfree, so the POSIX invariant is violated on every run.The assertion is removed and the async variant now makes a single
statvfscall plus destructure instead of three separate callsracing with concurrent filesystem activity from other tests.
CI / tooling
.github/workflows/coverage.ymlworkflow: per-OS matrix(ubuntu-latest, macos-latest, windows-latest) running
cargo tarpaulin --engine llvm --all-features --run-types tests --ignore-testson each runner and uploading per-OS reports toCodecov with per-OS flags. Each runner
--exclude-filesthe otherOS's sources so they do not register as 0/N uncovered lines.
crossjob now installs the MinGW-w64 toolchain when targeting*-pc-windows-gnu.windows-sys0.61 invokesdlltoolduringbuild, and
ubuntu-latestships only the x86_64 MinGW toolchainby default.
fs4 = { version = "0.13", ... }to
fs4 = { version = "1", ... }..codecov.yml,rustfmt.toml(2-space indent,explains the session-wide reformat), and
.github/workflows/loc.yml; removed the obsoletetea.yaml.Configuration
📅 Schedule: (UTC)
* 0-3 * * 1)🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.