fall back to network checkout when the mirror sticky disk stalls

[Trent-TSE]

Summary

On Aug 3 (and Jul 21/23), degraded Ceph reads on git-mirror sticky disks stalled git fetch/git checkout (alternates object reads) until customers' timeout-minutes killed the step — ~145 descriptinc jobs failed at exactly 300s, plus similar bursts for other orgs. Once the mirror was mounted there was no escape hatch: a slow disk meant a hard job failure instead of a slower network clone. This adds two "degrade, don't fail" layers:

1. Read-health probe after mount (blacksmith-cache.ts): right after mounting the sticky disk, read 8MB directly from the block device (dd iflag=direct, 10s deadline). On failure: unmount, release the disk (commitStickyDisk with shouldCommit: false, via new releaseStickyDisk()), and return CacheInfo.readProbeFailed = true, which the caller treats like hydrationInProgress — standard network checkout, no mirror.

2. Stall timeout on mirror-assisted fetch/checkout (git-command-manager.ts, git-source-provider.ts): when the mirror is attached, git fetch and git checkout run under timeout -k 10 120 (MIRROR_STALL_TIMEOUT_SECS). Exit code 124 raises the new GitStallTimeoutError, on which the provider:

catch GitStallTimeoutError:
  removeAlternates(workspace)      # detach mirror
  cacheInfo = null
  git fetch <same refspec>         # network-only; re-downloads objects the mirror was providing
  (checkout path also retries git checkout)

Mirror-assisted fetch skips the retryHelper loop (single attempt) so a stall doesn't burn 3×120s before falling back. The post step is unchanged: refresh/GC already have their own timeouts and skip the commit on failure.

dist/index.js rebuilt via npm run build.

Testing

• npx tsc --noEmit, npm run lint, npm run format-check, npm test (115 passed)

Context: descriptinc read-stall incident Aug 3 (ClickHouse-verified: ~831s of read wait on /dev/vdb during a 5-min step, zero TCP retransmits; same signature on Jul 21 us-west and Jul 23 eu-west bursts).

Link to Devin session: https://app.devin.ai/sessions/63dfb9784da34f42a26241763bbb2d28
Requested by: @Trent-TSE

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)}

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7e5bccd. Configure here.}