Skip to content

Concurrent requests during a first clone get 503 "no-mirror" even when the clone succeeds, because the commit-graph write runs inside the fetch flight and followers only wait FetchTimeout #106

Open

Description

@matt-edmondson

What's wrong

Since c0a5c73, MirrorFetcher.CloneAsync (GitBranchStateCache/Mirrors/MirrorFetcher.cs:~306) and FetchAsync (:~352) both run WriteCommitGraphAsync before the mirror is moved into place and marked fetched, and before the single-flight ticket completes. WriteCommitGraphAsync has its own timeout of FetchTimeout.

So the leader can now hold the flight for up to:

  • clone or fetch: FetchTimeout
  • git config (clone only): ProbeTimeout
  • commit-graph write: FetchTimeout

That adds up to about 2×FetchTimeout. Followers wait only one FetchTimeout in FollowAsync (ticket.WaitForLeaderAsync(options.Value.FetchTimeout, …), :~229). When that wait expires and mirrors.Exists(directory) is still false, because the clone is sitting in staging while the graph is written, they return MirrorFetchResult.Unavailable, which the handler answers as 503 "no-mirror".

Concrete scenario

Default FetchTimeout of 2 minutes, first request for a large repository:

  1. The leader's clone --bare --filter=blob:none takes ~100 s and succeeds.
  2. commit-graph write --reachable --split over the full history takes ~30 s.
  3. Followers that arrived during the clone reach their 120 s wait limit at about the time the graph write starts. They see no mirror directory and each get 503, although the clone succeeded and the mirror appears seconds later.

The same ordering affects every fetch in a smaller way:

  • Followers can fall back to "stale" while the leader is only writing the graph.
  • MarkFetched runs after the graph write, so refsAsOf overstates freshness by the duration of the graph write. That loosens the bound AdmissionTtl/RefsTtl are meant to give.

Suggested fix

The method's own remarks call the graph write a best-effort optimization ("a mirror without a graph answers the same questions more slowly"), so take it off the critical path:

  • After a successful clone or fetch: MoveIntoPlace (clone), then MarkFetched, then complete the ticket.
  • Then write the commit-graph best-effort, outside the fetch flight. For example, use a per-mirror single-flight so two graph writes never overlap, and make sure the idle-mirror sweep and the staging sweep cannot remove the directory mid-write.
  • Alternatively, keep the write inside the flight but make followers wait for the leader's real bound, not FetchTimeout alone. This is the weaker option, because the graph still delays every waiter.

Acceptance criteria

  • With a fake runner where the clone takes just under FetchTimeout and the graph write takes longer than the remaining margin, a concurrent follower gets the mirror (Current), not Unavailable.
  • RefsFetchedAt is recorded when the fetch completes, not after the graph write.
  • A graph-write failure or timeout still leaves a usable, marked mirror.

Activity

  1. matt-edmondson commented on Oct 30, 2026

    @matt-edmondson
    ContributorAuthor

    Triage


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions