repo-info --stats: report repository size (and optionally dedup stats) read-only, with --json

[ioanalytica]

Problem

Borg 2 has no read-only way to learn how large a repository is.

• repo-info --json reports id, location, last_modified, encryption and cache paths, no size.
• info --json (archive level) reports nfiles and original_size only. Its help says: "for the repository-wide deduplicated size, use borg compact --stats."
• compact --stats does report it ("Repository size is 402 kB in 8 objects", plus source data size, deduplicated size, compression factor), but only while compacting: it needs the exclusive lock, rewrites packs, and compact --dry-run --stats answers "Ignoring --stats. It is not supported when using --dry-run." The numbers are also only printed as rounded INFO log lines, so a program has to parse "402 kB" out of a log_message.
• repo-list --format "{size}" is per-archive original size, not repository size.

For anything that wants to show or monitor repository size without changing the repository (a UI, a Prometheus exporter, a cron report), the only options today are store-level measurements (du, rclone size, walking the borgstore backend), which measure files, not Borg objects, and which cannot be expressed uniformly across file:, sftp:, rest:, rclone: and s3: stores.

This is the read-only half of #8796 (closed with "use compact --stats -v") and overlaps the monitoring wish in #2934.

Observation: the data is already there and cheap

Measured on 2.0.0b23 and 2.0.0b24 (identical):

1. Repository size needs nothing but the chunk index. Repository.list() yields (chunk_id, storage_size) from the index/ fragments under a shared lock, without loading a pack. Summing it reproduces compact --stats's "Repository size" byte for byte (402434 bytes = "402 kB in 8 objects"; du of packs/ is ~5 % higher through pack headers). On a small scratch repository this takes 15 ms.

   from borg.logger import setup_logging; setup_logging()
   from borg.repository import Repository
   from borg.helpers import Location
   with Repository(Location(url), exclusive=False) as r:
       sizes = [s for _, s in r.list()]
   print(len(sizes), sum(sizes))

2. Source data size, file count and deduplicated size are what CompactCmd.analyze_archives() computes, and since the archive reference caches landed (cache/referenced_by_archive_<id> in the store, get_archive_references(..., cached=True)), an unchanged archive is not even opened again. A read-only variant is exactly the dry_run path of that method (store=False, no cache cleanup).

So a read-only repo-info --stats is a re-use of two existing code paths, not new bookkeeping.

Proposal

Add --stats to repo-info:

borg repo-info --stats [--json]

Text output (same wording as compact --stats, so users recognise it):

Repository size: 402 kB in 8 objects
Source data size: 700 kB in 2 files (2 archives)
Deduplicated size: 400 kB
Deduplication factor: 0.57
Compression factor: 1.00

JSON: add a stats object with exact byte values, alongside the existing keys, so repo-info --json stays backwards compatible without the flag (values illustrative):

{
  "repository": {"id": "...", "location": "...", "last_modified": "..."},
  "encryption": {...},
  "cache": {...},
  "stats": {
    "repository_size": 402434,
    "object_count": 8,
    "source_size": 700436,
    "source_files": 2,
    "archive_count": 2,
    "deduplicated_size": 400304
  }
}

Semantics:

• repository_size / object_count: sum and count over the chunk index (what compact --stats prints after compaction). Always available, shared lock, no pack access.
• source_size, source_files, archive_count, deduplicated_size: from the read-only analysis of all archives, using the per-archive reference caches when present. Archives without a cache are scanned (as compact --dry-run does today) but nothing is written to the store.
• No compaction_saved, since nothing is compacted.
• Requires a shared lock only, so it can run while a backup is in progress, unlike compact.

If the archive analysis is considered too expensive for repo-info, a smaller first step would already help most callers: repository_size and object_count only (index sum), with the dedup fields behind a second flag (--stats --analyze) or left to compact --stats. Alternatively, make compact --dry-run --stats honour --stats instead of ignoring it; that gives the same numbers read-only, but still under the exclusive lock and without JSON.

Why not parse compact --stats -v

• It changes the repository, so a monitor cannot call it on its own schedule.
• It takes the exclusive lock.
• The values are rounded to three significant digits and only reachable through INFO log_message lines.
• Borg 1's info --json exposed cache.stats with exact unique_csize, unique_size, total_size; tools built on that lost the field with Borg 2 and currently have no replacement.

Environment

borg 2.0.0b23 and 2.0.0b24, borgstore 0.6.1, Python 3.12, Linux; measurements above on a file: store; the same repo-info runs against rest: stores in production.

[ThomasWaldmann]

Thanks for the feedback.

I think we can offer this via some non-default option(s), but we have to be careful:

• some kinds of repos have a lot of latency and no server-side agent (e.g. cloud, sftp)
• some people have lots of archives, so if we implement stuff that costs time like O(archives), that can get really slow
• the index is not cached (yet?) and downloading it is quite a bit of I/O (index size) plus a bit of latency (there is more than 1 index fragment usually)
• listing the packs (as an alternative to using the index) costs ~257 round-trips for repos that have been used for a while, little traffic, much overall latency

[ioanalytica]

Thanks — agreed on all four points, and I think they suggest a shape for this rather than argue against it.

O(archives): yes, that is the analyze_archives() half, and it should stay out of --stats. The issue already offers the split; to make it explicit: repo-info --stats = index sum only (repository_size, object_count), the dedup/source figures behind a second option (--analyze) or left to compact --stats.

Index download: repo-info does not load the chunk index today (Cache.chunks is lazy), so --stats would add exactly one index load. That is the same cost every create, prune and compact already pays per run — build_chunkindex_from_repo() fetches all fragments from the store each time, there is no local copy. So the size-only variant costs what an empty create costs, and a caller runs it once per backup cycle, not on a timer. Rough numbers at ~50 bytes per entry (32-byte key, 8-byte value, hash table overhead):

repository	chunks (~1.5 MiB)	index
400 GB	~270k	~13 MB
4 TB	~2.7 M	~130 MB

Index not cached (yet?): the fragments are already content-addressed (index/<sha256>) and the repack comment says large fragments stay "immutable (and cache-stable for other clients)". Adding index/ to the writethrough store cache next to packs/ would use the same verify-by-name mechanism, and it would speed up every command that loads the index, not just this one. That turns the table above into a one-time cost per fragment.

Listing packs: agreed, ~257 round-trips for a few numbers is not an option; the index is the right source.

One more variant that avoids the download entirely: the commands that already hold the full index in memory at the end of a run (create, prune, compact, check) could store a tiny stats record — repository size, object count, timestamp — in the store, e.g. next to the manifest. repo-info --stats would then read one small object: O(1), no index I/O, no O(archives). The figure is at most as old as the last write, which is the right freshness for "how big is this repository". This answers all four concerns at once, at the price of one small extra store write per run.