fix: fall back to full index when pull/checkout targets mix .dvc files and paths - #11076
fix: fall back to full index when pull/checkout targets mix .dvc files and paths#11076adarshsm wants to merge 2 commits into
Conversation
…s and paths `index_from_targets` builds a merged per-target index when every target is a stage or `.dvc` file, otherwise it falls back to the full repo index. The fallback signal is `index is None`, but `index` is also the loop variable that holds each per-target index. When a target list mixes a `.dvc`-file target (which parses) with a granular path inside a tracked directory (which raises `StageFileDoesNotExistError`), the loop set `index` from the `.dvc` target before the granular target failed, so the `except` left a *partial* index in place. The `index is None` fallback was then skipped, and that partial index — which knows nothing about the directory — was used with the full target list. Consequences (both from treeverse#11075): - fresh-clone state: the granular target is silently skipped (never checked out); - if the tracked directory has drifted, `checkout`'s `_check_can_delete` looks up a key absent from the partial index's `storage_map` and dies with an uncaught `KeyError`/`StorageKeyError`. All-data-path target lists already worked, but only because the first target fails immediately and leaves `index is None`. Reset `index = None` in the `except` so a partial parse falls back to the full repo index with the original targets, matching that working path. Added a regression test covering both the silent-skip and the drift-crash cases. Fixes treeverse#11075
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11076 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41153 +1358
Branches 3141 3263 +122
==========================================
+ Hits 36087 37443 +1356
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| # than keeping the partial index built only from the targets parsed | ||
| # before the failure — using that partial index with the full targets | ||
| # list silently drops the unparsed targets and can crash checkout (#11075). | ||
| index = None |
There was a problem hiding this comment.
I think the bug here is that we are setting index above inside the loop. Could we rename it to something else, for example idx?
There was a problem hiding this comment.
Agreed, that is the cleaner framing — renamed to idx in c9d81a6.
With the per-target index named separately, index is only ever the fallback signal, so it stays None when a target fails to parse and the handler no longer has to reset it. The except goes back to pass.
I kept a short comment there to record why falling back is the correct response rather than an oversight: the partial index built before the failure is still paired with the full targets list, which silently drops the unparsed targets and can raise from checkout when the tracked directory has drifted.
Re-verified after the rename: the regression test still fails on main and passes here, tests/func/test_repo_index.py (19) and tests/func/test_data_cloud.py (36, plus 2 pre-existing xpass) are green, and ruff is clean.
index served as both the loop variable and the "no per-target index" signal, so a target that failed to parse left a partial index behind and skipped the fallback. Naming the per-target index idx keeps index as the fallback signal only, which removes the need to reset it in the handler.
Fixes #11075.
Problem
dvc pull(anddvc checkout) with a target list that mixes a.dvc-file target with a granular path inside a tracked directory misbehaves two ways, both reported in #11075:KeyError/StorageKeyError:Each target form works on its own, and an all-data-path target list works too (that's the documented workaround).
Root cause
index_from_targets(dvc/repo/index.py) builds a merged per-target index when all targets are stages/.dvcfiles, and otherwise falls back to the full repo index. The fallback is gated onindex is None— butindexis also the loop variable holding each per-target index:With a mixed list like
["single.csv.dvc", "datadir/f1.txt"], the.dvctarget setsindex, then the granular path raises. Theexceptdoespass, leaving the partial single-target index in place, so theindex is Nonefallback is skipped and that partial index — which knows nothing about the directory — is used with the full target list. The directory target is dropped (silent skip), and when the dir has drifted,checkout._check_can_deletelooks up a key that isn't in the partial index'sstorage_mapand raises.All-data-path lists already work only because their first target fails immediately, so
indexstaysNoneand the fallback fires.Fix
Reset
index = Nonein theexcept, so a partial parse falls back to the full repo index with the original targets — the same working path that all-data-path target lists already take. One line.Test
Added
test_pull_mixed_dvcfile_and_granular_targets(tests/func/test_data_cloud.py) covering both the silent-skip and the drift-crash cases with alocal_remote. It fails onmain(skip +KeyError) and passes with the fix. Verifiedtests/func/test_checkout.py(48),tests/func/test_repo_index.py(19), andtests/unit/repo/(109) still pass;ruff check/formatclean.