Skip to content

The lazy-migration background upload tars a live repo while a push writes it, holding no lease or lock #343

Description

@beardthelion

Two paths touch the same repo directory and neither excludes the other.

Writer, api/repos.rs:1881-1946: POST .../git-receive-pack takes the per-repo lease
(state.repo_write_leases.acquire), then repo_store.acquire_write, which takes the advisory lock on
its own pinned connection, runs git receive-pack, then guard.release(true) uploads.

Background upload, git/repo_store.rs:85-115: any read handler calling acquire() spawns a detached
task that tars the live directory and uploads it. It takes no lease, no advisory lock, and no
snapshot. repo_store.rs never references repo_write_leases at all, and acquire() is the only
function in the file that spawns an upload without going through a guard.

tokio::spawn(async move {
    match tigris.exists(&slug, &name).await {
        ... Ok(false) => { ... tigris.upload(&slug, &name, &path).await ... }

compress_repo (git/tigris.rs:165-176) walks the live bare repo with tar.append_dir_all(".", repo_path).
No snapshot, no clone, no barrier. And put_object (tigris.rs:96-104) carries no if_match and no
generation, so the last PUT to land wins regardless of which snapshot is older.

Two bad outcomes, both persisted

  1. Stale overwrites newer: the migration snapshot predates the push, its PUT lands after the guard's
    PUT, and the bucket ends up holding the pre-push repo. Internally consistent, but it is the
    canonical copy, and the next download destroys the good local tree too, because
    tigris.rs:239-242 does remove_dir_all(local_path) then rename. That is silent loss of an
    acknowledged push.
  2. Torn: the tar walk straddles the ref update, so the archive's refs point at objects the pack does
    not contain. Also persisted, also later served.

init() (repo_store.rs:289-299) has the same shape: a detached upload of the just-created repo with
no exclusion against a push arriving before the PUT completes.

Reachability

Requires Tigris enabled and the repo not yet in the bucket, since exists() returning Ok(true)
skips the upload. That state is exactly what this code is for (the doc comment at repo_store.rs:75-76),
and also arises when an init() upload failed, which only warns (:296), or after a bucket-side
delete.

Given that, the trigger is an anonymous permissionless GET /:owner/:repo/info/refs?service=git-upload-pack,
an ordinary clone, landing while an authenticated push is in flight. The window is the whole
compress plus PUT, which for a large repo is seconds to minutes of zstd over the full tree.

Not verified by execution: driving it needs a stallable S3 endpoint plus a compress-timing seam.
TigrisClient::for_testing_with_endpoint exists but there is no upload hook. Established by reading,
the same standing accepted for #283.

Not a duplicate

#283 is the mirror image: extraction clobbering a repo under a reader or writer. This is upload
capturing a repo under a writer. Opposite direction, different fix. #279 and PR #285 are
writer-versus-writer advisory-lock correctness and do not reach this, because the background upload
never asks for the lock, so even a perfectly session-affine lock excludes nothing here. #300 shares the
exists() call but is a read-rendering bug.

Fix direction

Put the migration and init uploads behind the same write exclusion release_after_write already
uses, or make the PUT conditional.

One correction: the obvious-looking fix of not inserting into migrated until the upload completes is
already the behavior. repo_store.rs:102-113 returns early on upload error, so the insert is only
reached after a successful PUT. The exclusion is the load-bearing part.

Metadata

Metadata

Assignees

No one assigned

    Labels

    crate:nodegitlawb-node — the serving node and REST APIkind:bugDefect fix — wrong or unsafe behaviorsev:mediumDegraded but workaround existssubsystem:storageBlob/object store, Arweave, IPFS, archives

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions