borg2 check: overview / tracking ticket

[ThomasWaldmann]

This ticket collects all cmd: check tickets (especially the ones in the 2.0.0b23 milestone)
into one place: a summary of how borg2 check is supposed to work, plus a checkbox list of the
concrete work items. Ticked items are already in master (commit linked); unticked items are
still to do (open PR linked where one exists).

Tracked tickets: #9696 #9825 #9898 #9901 #9218 #9443 #8009 #7893 #2397 #8466 #8476 (b23),
#8478 (b24), #6584 (2.1), plus the already-closed #9897 #9411 #8518 #8517 #8559 #8297 #1931 #7062.

How borg2 check is supposed to work

What changed vs. borg 1.x

borg 1.x check had a repository part (crc32-check all segment entries, always rebuild the
repository index, run server-side for ssh repos) and an archives part (read all archives, check
chunk presence, optionally --verify-data). Much of that design no longer applies:

• A borg2 repository is a borgstore store. There is no borg process on the server side
  (except for rest: served by borg serve --rest), so "run the check on the server" is
  replaced by something better: content-addressed verification.
• Pack files (packs/<sha256>) and chunk index fragments (index/<sha256>) are named by the
  sha256 of their content (fd177535).
  So "is this object intact?" is simply Store.hash(name) == name. The REST backend computes
  the hash server-side, so a full repository check of a remote repo transfers no pack data.
• The chunks index (chunk id → pack id, offset, size) lives in the repository and is the
  authoritative record of which chunks exist and where.
• There is no chunks_healthy anymore (borg2: reconsider chunks_healthy approach #8559): an item's chunks list always holds the
  correct chunk ids, even when chunks are missing. Missing chunks are handled dynamically at
  read time (zeros or error), and if a later borg create re-produces a missing chunk, all
  old archives referencing it are healed instantly — no repeated check --repair runs.

Phase 1: repository check (cheap, read-only, no key needed)

1. Verify all index/ fragments via Store.hash (small, always checked in full).
2. If (and only if) the index is intact, verify all packs/ the same way.
3. Partial checks (--max-duration) resume from a persisted set of checked packs
   (cache/checked-packs, borg2 check: move from marker to set-of-checked-packs #9897), with periodic checkpoints. Packs recorded as corrupt are
   always re-verified, never skipped.
4. The read-only check never rebuilds the index: reading every pack is far too slow and
   expensive for a routine check. A corrupt index is reported and left for repair.

This phase is designed to be a routine, cron-able job: no key, no prompt, bounded time,
meaningful exit code. Its findings (corrupt pack ids) should be persisted so a later repair
can consume them instead of re-scanning everything (PR #9925).

Phase 2: archives check (client-side, needs the key)

1. Ask for the passphrase up front, not hours into the run (ux: borg check asks for passphrase very late #1931).
2. Check the manifest (rebuild if missing/corrupt), the archives directory, each selected
   archive's metadata object and item metadata stream (robust resync on damage), and each
   item's file-content chunk presence against the (verified) chunks index.
3. --verify-data: additionally read, decrypt (AEAD-authenticate) and decompress every
   object — full cryptographic verification, detects malicious tampering too (for encrypted
   repos, against attackers without the key).
4. --find-lost-archives: full scan of all objects for archive metadata without an archives
   directory entry; with --repair, recreate the entry (possible until borg compact removes
   the data).

A normal (non-repair) archives check trusts the hash-verified index and does not rebuild it
(#8476); only repair rebuilds from the actual packs.

Repair philosophy

• Only repair what is known to be broken. A repair should be able to consume the corrupt
  pack list a previous cheap check produced, instead of re-scanning all packs (borg2: improve borg check #9696).
• Index repair rebuilds the index by scanning pack object headers only (seek over
  payloads, no full read) — but must not blindly trust the unauthenticated headers (borg2: index rebuild trusts pack headers #9901).
• Pack repair salvages objects out of a corrupt pack: every object that still decrypts
  (AEAD-authenticates) goes into a new pack, the corrupt pack is dropped, the index updated
  (borg2: avoid doing an unnecessary rebuild of the index #8476, borg2: improve borg check #9696, refs to pack or not to pack ... #8572).
• Defect chunks are only deleted after a second read attempt also fails (transient errors —
  network, RAM — must not destroy data).
• Command split is open for discussion: borg check strictly read-only, all
  repairing/rebuilding in borg repair (with e.g. --index, --packs PACKID...) (borg2: improve borg check #9696).

Design goals

• Data safety: never delete data on a single failed read; never let a rebuilt index claim
  objects that do not exist (fatal for borg create dedup, borg2: avoid doing an unnecessary rebuild of the index #8476); verify before trusting
  (borg2: index rebuild trusts pack headers #9901); a interrupted check/repair must leave a consistent state (Ctrl-C at safe
  boundaries, Ctrl-C does not interrupt borg check #7893).
• Usability for admins: routine checks need no key and no prompts; progress indication
  everywhere (Add progress indicator to repository check loop #9443, Progress of Remote: checking segment file /whatever/data/17/17078... #8009); a summary of findings and repairs at the end (check: summarize findings, --repair: summarize repairs #2397); reports
  organized around the question the admin actually has — "which files/archives are affected?"
  (Improve borg2 check reporting of missing chunks #9218); a documented "what do I do now?" workflow after corruption (borg2: improve dealing with repo corruption #9825); customizable
  output (borg2: check does not support custom output formatting #9411).
• Performance: server-side hashing (no transfer); no unnecessary index rebuilds (borg2: avoid doing an unnecessary rebuild of the index #8476);
  partial/time-boxed checks; reuse of still-fresh check results (--max-age, PR check: keep pack check results, add --max-age to reuse them #9925);
  headers-only scans where possible; combine full-repo passes instead of reading everything
  twice (borg2: check --verify-data --find-lost-archives can be made faster #8478, ex-borg2 check redesign? #8518).

Work items

Repository check (read-only)

• Content-addressed read-only check: verify index/ and packs/ objects via
  Store.hash, server-side hashing for rest: repos; index first, packs only if the
  index is intact; no index rebuild in the read-only path (borg2: improve borg check #9696, borg2: avoid doing an unnecessary rebuild of the index #8476, borg2 check redesign? #8518) —
  27d693c2 (PR repository: read-only check by hashing pack/index objects #9794, with PRs repository: make check and delete work at N=1 with sha256 pack ids #9783/repository: always name packs by sha256(pack) #9789 for sha256 pack naming)
• Progress indicators (percent) for the index and pack verification loops (Add progress indicator to repository check loop #9443, Progress of Remote: checking segment file /whatever/data/17/17078... #8009) —
  27d693c2 (PR repository: read-only check by hashing pack/index objects #9794)
• Partial checks: persist a set of checked packs instead of a "last pack" marker, so
  packs added mid-cycle cannot be skipped (borg2 check: move from marker to set-of-checked-packs #9897) —
  bc8b3c07
• Re-verify packs recorded as corrupt instead of skipping them on resume (borg2 check: move from marker to set-of-checked-packs #9897) —
  9543c141
• Checkpoint the checked-packs set periodically (30 min) during long runs (borg2 check: move from marker to set-of-checked-packs #9897) —
  a1b88e2d
• Keep pack check results after a completed check and add --max-age to skip packs with
  a fresh intact result — makes cheap periodic checks feed later repairs and avoids
  re-hashing everything on slow/cloud storage (borg2: improve borg check #9696, borg2 check: vanished packs? #9898) — PR check: keep pack check results, add --max-age to reuse them #9925 open
• Detect vanished packs: cross-check that every pack referenced from the chunks index is
  present in packs/ and has the expected size (borg2 check: vanished packs? #9898). Currently a completely missing
  pack is only noticed at read time
  (cbc94ae8 PackNotFound), by --verify-data, or by the --repair index rebuild.
• Cheap check mode for cloud repos: make full pack-sha256 verification optional; default
  (or an option) could be listing packs and checking presence + size against the index,
  avoiding the full data transfer (borg2 check: vanished packs? #9898)

Archives check

• Ask for the key passphrase early, before the (potentially very long) repository phase
  (ux: borg check asks for passphrase very late #1931) — 2340e605
• Archive selection filters --first/--last/-a/--sort-by (adapt borg check like in PR 1554 #1663) and
  --older/--newer/--oldest/--newest (borg check --archives-only --last-n-seconds #7062) —
  b2654bc1
• --format support for the "Analyzing archive ..." output, validated up front,
  BORG_CHECK_FORMAT env var (borg2: check does not support custom output formatting #9411) —
  d7ffe748
• Non-repair archives check trusts the hash-verified in-repo chunks index — no slow index
  rebuild for a routine check (borg2: avoid doing an unnecessary rebuild of the index #8476, borg2: check [--repair] improvements #8466) —
  27d693c2
  (earlier step: reuse the repo-part ChunkIndex, 7288f4f9, Check improvements #8468)
• Remove chunks_healthy; self-healing archives, defined read-time behaviour for missing
  chunks, no all-zero patch chunks (borg2: reconsider chunks_healthy approach #8559) —
  6357f2eb
• --find-lost-archives (né --undelete-archives): scan for archive metadata lacking an
  archives directory entry, recreate entries with --repair —
  682aedba,
  renamed in a48a8d2b
• Speed up --verify-data combined with --find-lost-archives: remember archive-metadata
  object ids during the verify pass and reuse them instead of a second full repo scan
  (borg2: check --verify-data --find-lost-archives can be made faster #8478, ex-check: optimization potential with --verify-data AND --undelete-archives #8517 — milestone b24)
• --verify-data reads chunks in index (hash) order, i.e. randomly across packs; read
  pack-by-pack instead so each pack is fetched once, sequentially ("read data only once",
  idea from borg2 check redesign? #8518, folded into borg2: improve borg check #9696)

Repair

• Repository-level repair is not implemented yet — Repository.check(repair=True)
  currently just logs "repository repair not implemented" (borg2: improve borg check #9696 "to be continued",
  borg2: improve dealing with repo corruption #9825). This is the main open block of work:
  • Index repair: when index/ objects are corrupt (or the index provably lies), rebuild
    the index by scanning pack object headers only (range reads, skip payloads), not by
    reading full packs (borg2: improve borg check #9696, borg2: avoid doing an unnecessary rebuild of the index #8476; code TODO at repository.py check(), refs to pack or not to pack ... #8572)
  • Pack repair (--repair-pack PACKID... or similar): salvage all AEAD-valid objects of
    a corrupt pack into a new pack, drop the corrupt pack, update the index — without a
    full scan of all other packs (borg2: avoid doing an unnecessary rebuild of the index #8476, borg2: improve borg check #9696)
  • Let repair consume the persisted corrupt-pack list from previous read-only checks, so
    an admin's --repair doesn't redo the cron job's scanning work (borg2: improve borg check #9696; groundwork in
    PR check: keep pack check results, add --max-age to reuse them #9925)
  • Decide the command split: keep everything under borg check --repair vs. a separate
    read-only borg check + borg repair --index/--packs/... (borg2: improve borg check #9696)
• Index rebuild must not blindly trust unauthenticated pack headers: a tampered header
  (wrong chunk id / sizes) currently goes straight into the rebuilt index (borg2: index rebuild trusts pack headers #9901).
  Mitigation ideas: authenticate on first real read (AEAD binds the id), or verify object
  ids cryptographically during rebuild for suspicious/mismatching entries.
• After check --repair, write the freshly built chunks index back to the repository
  instead of deleting it and forcing the next command into a slow rebuild (borg2: check [--repair] improvements #8466 — the
  read-only path no longer rebuilds at all, but ArchiveChecker.finish() still deletes
  the index after repair)
• Repair progress indication: the --repair paths (verify_data delete pass, archive
  rebuild, future index/pack repair) need progress output like the read-only check has
  (Add progress indicator to repository check loop #9443, Progress of Remote: checking segment file /whatever/data/17/17078... #8009)
• Repair using redundant related repos: fetch chunks that are missing/corrupt here from a
  related repo (same key material / same chunk ids), e.g.
  borg check --repair --other-repo=...; borgstore cache vs. primary self-healing is a
  variant of the same idea (borg check: doing a repair using redundant related repos #6584 — milestone 2.1)

Reporting & UX

• Summarize findings at the end of check, and repairs actually done with --repair
  ("N problems found, M repaired") (check: summarize findings, --repair: summarize repairs #2397, dup [1.2.7] borg check -repair lacks a summarizing information at the end when it actually repaired something(s) #8192) — PR check: summarize findings and repairs #9891 open
• Report missing chunks grouped as chunk → files → archives instead of one log line per
  (archive, file, chunk); avoids the hardlink-caused flood of repeated lines and answers
  "which files are affected?" directly (Improve borg2 check reporting of missing chunks #9218) — PR check: report missing chunks grouped as chunk -> files -> archives, #9218 #9965 open
• Ctrl-C handling: first Ctrl-C stops soon but cleanly at a safe boundary (saving
  checkpoint state, running finish()), second Ctrl-C aborts immediately (Ctrl-C does not interrupt borg check #7893) —
  PR check: handle Ctrl-C at safe boundaries (#7893) #9966 open
• Document the post-corruption workflow (borg2: improve dealing with repo corruption #9825): what --repair does and does not do,
  that re-running borg create heals reappearing chunks automatically (no second
  check --repair needed), how to get rid of permanently lost data
  (borg recreate --exclude ... the affected files), "considerations to fix your repo
  corruption" guide
• A way to acknowledge known-lost chunks so subsequent check/compact runs stop returning
  exit code 2 / error mails for damage the admin has already accepted (borg2: improve dealing with repo corruption #9825)

Done / resolved by the borg2 design (no action needed)

• No more borg check requirement before prune/compact, so the --repair YES prompt no
  longer haunts cron jobs ([Feature Request] Disable the YES prompt when BORG_CHECK_I_KNOW_WHAT_I_AM_DOING is enabled #8297; BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=YES still skips the
  prompt for actual repair runs)
• Segment-era tickets are obsolete: there are no segments, transactions or commits in a
  borgstore repo — Review check operation if no commit exists #2454 (check with no commit), Corrupted segment reference count - corrupted index or hints #8535/Frequent "Index object count mismatch." errors on borg (1.4.0) check runs #8580 (segment refcount / index
  object count mismatch), Possibility to run borg check for specific segment(s) #8070 (check a single segment; nearest borg2 equivalent is the
  planned per-pack repair above)
• "check --repair commit prompt" (check --repair commit prompt #2398) — obsolete: borg2 has no repo transaction to
  commit; the equivalent admin need (see what check finds, then repair exactly that) is
  covered by the persisted check-results + repair-consumes-them items above

Milestone view (2.0.0b23)

Ticket	Topic	State
#9696	check redesign umbrella	read-only part done; repair part is the open core work
#8466	check --repair improvements	open (index caching after repair)
#8476	avoid unnecessary index rebuild	read-only part done; repair-index/-pack open
#9898	vanished packs	open (PR #9925 related)
#9901	index rebuild trusts pack headers	open
#9825	dealing with repo corruption	open (docs + acknowledge mechanism)
#9218	missing-chunk reporting	PR #9965 open
#2397	summarize findings/repairs	PR #9891 open
#7893	Ctrl-C	PR #9966 open
#9443	repo check progress	done for read-only (27d693c2); repair progress open
#8009	progress counters	borg1 wontfix; borg2 covered by #9443 items
#9897	set-of-checked-packs	done (bc8b3c07)
#9411	check --format	done (d7ffe748)

---

made by Claude Fable 5

[ThomasWaldmann]

@mr-raj12 Please check this manually, maybe it is useful for your ongoing work.

[ThomasWaldmann]

--verify-data needs checking: does it process data in pack order? it must not process chunks in random index order.