dvc fetch reports 'Everything is up to date' when .dir manifest is missing from local cache

[FloHu]

DVC version

3.67.1 (conda), Python 3.10.19, Linux

Description

When data.dvc references a .dir manifest hash that is not present in the local cache, dvc fetch reports "Everything is up to date" and does not download the manifest or its referenced files from the remote. The subsequent dvc checkout then fails with FileNotFoundError.

When this happens

This occurs in multi-machine DVC workflows when data.dvc is updated via git pull to reference a new .dir hash that was never fetched to the local machine. Specifically:

1. Machine A: runs dvc add data, dvc push, git push (new .dir hash pushed to S3)
2. Machine B: already has an older .dir in its local cache from a previous version
3. Machine B: git pull updates data.dvc to reference the new .dir hash
4. Machine B: dvc pull fails — fetch reports "up to date", checkout fails

The root cause appears to be that DVC's fetch logic relies on reading the .dir manifest to enumerate which individual files need to be transferred. When the .dir manifest itself is the missing file, DVC cannot enumerate anything, collects 0 entries, and concludes nothing needs to be fetched.

Verbose output (abbreviated)

DEBUG: Preparing to transfer data from 's3://<bucket>/files/md5' to '<project>/.dvc/cache/files/md5'
DEBUG: Preparing to collect status from '<project>/.dvc/cache/files/md5'
DEBUG: Collecting status from '<project>/.dvc/cache/files/md5'
DEBUG: failed to create '<project>/data/<file1>' from '<project>/.dvc/cache/files/md5/xx/xxxxxxxxx' - [Errno 2] No such file or directory
DEBUG: failed to create '<project>/data/<file2>' from '<project>/.dvc/cache/files/md5/yy/yyyyyyyyy' - [Errno 2] No such file or directory
Everything is up to date.
ERROR: failed to pull data from the cloud - Checkout failed for following targets: data

Evidence that files exist on remote

Verified via boto3 that both the .dir manifest and individual files are present in the S3 remote (uploaded ~2h before the pull attempt).

Expected behavior

dvc fetch should detect that the .dir manifest is missing from the local cache, download it first, then enumerate and download any missing individual files.

Workaround

Manually download the missing files from S3 into the correct cache paths, then run dvc checkout:

# Download .dir manifest and missing files directly
aws s3 cp s3://<bucket>/files/md5/xx/<hash>.dir .dvc/cache/files/md5/xx/<hash>.dir
aws s3 cp s3://<bucket>/files/md5/yy/<hash> .dvc/cache/files/md5/yy/<hash>
dvc checkout

[prince3235]

Hi! @FloHu! ,

I'd like to investigate this issue.

The root cause makes sense — fetch logic depends
on reading .dir manifest to enumerate files,
but doesn't check if the manifest itself is missing.

Is this available to work on?

[prince3235]

Hi! @FloHu, I investigated this issue.

The bug is in dvc-data package, specifically in:
dvc_data/index/collect.py → _collect_from_index()

When .dir manifest is missing from local cache,
entry.loaded remains None, so files inside the
directory are never enumerated for fetching.

This is in the iterative/dvc-data repo, not dvc itself.
Would a fix there be the right approach?

[FloHu]

Hi @prince3235, thanks for looking into this! Feel free to open a PR in iterative/dvc-data. Thanks.