Skip to content

fix: fall back to full index when pull/checkout targets mix .dvc files and paths - #11076

Open
adarshsm wants to merge 2 commits into
treeverse:mainfrom
adarshsm:fix/mixed-dvcfile-granular-targets
Open

fix: fall back to full index when pull/checkout targets mix .dvc files and paths#11076
adarshsm wants to merge 2 commits into
treeverse:mainfrom
adarshsm:fix/mixed-dvcfile-granular-targets

Conversation

@adarshsm

@adarshsm adarshsm commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #11075.

Problem

dvc pull (and dvc checkout) with a target list that mixes a .dvc-file target with a granular path inside a tracked directory misbehaves two ways, both reported in #11075:

  • Fresh-clone state: the granular target is silently skipped — pull exits 0 but never checks it out.
  • Drifted directory (any untracked file inside the tracked dir): checkout dies with an uncaught KeyError / StorageKeyError:
    ERROR: unexpected error - ('datadir', 'f1.txt')
    

Each target form works on its own, and an all-data-path target list works too (that's the documented workaround).

Root cause

index_from_targets (dvc/repo/index.py) builds a merged per-target index when all targets are stages/.dvc files, and otherwise falls back to the full repo index. The fallback is gated on index is None — but index is also the loop variable holding each per-target index:

index: Optional[Index] = None
if targets and all(targets) and not with_deps and not recursive:
    indexes = []
    try:
        for target in targets:
            ...
            if file and not name:
                index = Index.from_file(repo, file)   # .dvc target: succeeds
            else:
                index = Index(repo, stages=list(repo.stage.collect(target)))
            indexes.append(index)
    except (StageFileDoesNotExistError, StageNotFound):
        pass                                          # <-- leaves partial `index`
    else:
        index = Index.from_indexes(repo, indexes)
        targets = None

if index is None:                                     # <-- skipped: index is partial
    index = repo.index
return index.targets_view(targets, ...)

With a mixed list like ["single.csv.dvc", "datadir/f1.txt"], the .dvc target sets index, then the granular path raises. The except does pass, leaving the partial single-target index in place, so the index is None fallback is skipped and that partial index — which knows nothing about the directory — is used with the full target list. The directory target is dropped (silent skip), and when the dir has drifted, checkout._check_can_delete looks up a key that isn't in the partial index's storage_map and raises.

All-data-path lists already work only because their first target fails immediately, so index stays None and the fallback fires.

Fix

Reset index = None in the except, so a partial parse falls back to the full repo index with the original targets — the same working path that all-data-path target lists already take. One line.

Test

Added test_pull_mixed_dvcfile_and_granular_targets (tests/func/test_data_cloud.py) covering both the silent-skip and the drift-crash cases with a local_remote. It fails on main (skip + KeyError) and passes with the fix. Verified tests/func/test_checkout.py (48), tests/func/test_repo_index.py (19), and tests/unit/repo/ (109) still pass; ruff check/format clean.

…s and paths

`index_from_targets` builds a merged per-target index when every target is a
stage or `.dvc` file, otherwise it falls back to the full repo index. The
fallback signal is `index is None`, but `index` is also the loop variable that
holds each per-target index. When a target list mixes a `.dvc`-file target
(which parses) with a granular path inside a tracked directory (which raises
`StageFileDoesNotExistError`), the loop set `index` from the `.dvc` target
before the granular target failed, so the `except` left a *partial* index in
place. The `index is None` fallback was then skipped, and that partial index —
which knows nothing about the directory — was used with the full target list.

Consequences (both from treeverse#11075):
- fresh-clone state: the granular target is silently skipped (never checked out);
- if the tracked directory has drifted, `checkout`'s `_check_can_delete` looks
  up a key absent from the partial index's `storage_map` and dies with an
  uncaught `KeyError`/`StorageKeyError`.

All-data-path target lists already worked, but only because the first target
fails immediately and leaves `index is None`. Reset `index = None` in the
`except` so a partial parse falls back to the full repo index with the original
targets, matching that working path.

Added a regression test covering both the silent-skip and the drift-crash cases.

Fixes treeverse#11075
@github-project-automation github-project-automation Bot moved this to Backlog in DVC Aug 3, 2026
@CLAassistant

CLAassistant commented Aug 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.98%. Comparing base (2431ec6) to head (c9d81a6).
⚠️ Report is 213 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11076      +/-   ##
==========================================
+ Coverage   90.68%   90.98%   +0.30%     
==========================================
  Files         504      505       +1     
  Lines       39795    41153    +1358     
  Branches     3141     3263     +122     
==========================================
+ Hits        36087    37443    +1356     
- Misses       3042     3071      +29     
+ Partials      666      639      -27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread dvc/repo/index.py Outdated
# than keeping the partial index built only from the targets parsed
# before the failure — using that partial index with the full targets
# list silently drops the unparsed targets and can crash checkout (#11075).
index = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the bug here is that we are setting index above inside the loop. Could we rename it to something else, for example idx?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that is the cleaner framing — renamed to idx in c9d81a6.

With the per-target index named separately, index is only ever the fallback signal, so it stays None when a target fails to parse and the handler no longer has to reset it. The except goes back to pass.

I kept a short comment there to record why falling back is the correct response rather than an oversight: the partial index built before the failure is still paired with the full targets list, which silently drops the unparsed targets and can raise from checkout when the tracked directory has drifted.

Re-verified after the rename: the regression test still fails on main and passes here, tests/func/test_repo_index.py (19) and tests/func/test_data_cloud.py (36, plus 2 pre-existing xpass) are green, and ruff is clean.

index served as both the loop variable and the "no per-target index"
signal, so a target that failed to parse left a partial index behind and
skipped the fallback. Naming the per-target index idx keeps index as the
fallback signal only, which removes the need to reset it in the handler.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pull: KeyError crash (or silently skipped target) when mixing .dvc-file targets with granular paths inside a tracked directory

3 participants