pull: "No file hash info found" and checkout fails when pulling frozen imported url that is a directory that has subsequently changed

[ahasha]

Bug Report

Description

I have used dvc import-url to track a GCS directory that is updated daily with incremental new data. The documentation suggests that once the data is imported, it is frozen and DVC should not attempt to download changes from the tracked URL unless dvc update is run.
It also says that

During dvc push, DVC will upload the version of the data tracked by [the .dvc file] to the DVC remote so that it is backed up in case you need to recover it.

Instead, when I dvc push, the log messages suggest the files were uploaded, but nothing shows up in the remote. y imported data to remote, dvc pull from a fresh checkout downloads all the files from the remote url (including updated ones), and fails to check out the frozen version of the data as expected.

Reproduce

I created a minimal example using a public GCS bucket in my account:

1. mkdir dvc-import-issue & cd dvc-import-issue
2. git init
3. dvc init
4. dvc import-url gs://hasha-ds-public/test_dir/ test_dir
5. dvc remote add origin gs://hasha-ds-public/test_remote/
6. dvc push -r origin

Output from dvc push is

Collecting                                                                                                           |5.00 [00:00,  441entry/s]
Pushing                                                                                                              |0.00 [00:01,     ?file/s]
Collecting hasha-ds-public/test_remote/ on gs                                                                        |5.00 [00:01, 3.29entry/s]
4 files pushed                                                      

However I later noticed that no files had been pushed to the remote at all despite the misleading log message. I was able to correct this by running dvc import-url gs://hasha-ds-public/test_dir --to-remote, but this feels like another bug.

8. git add .dvc/config test_dir.dvc .gitignore
9. git commit -m "Committed and pushed"

Now set up a fresh clone of the project

10. cd ..
11. git clone dvc-import-url-pull-issue dvc-import-url-pull-issue2

Add another file to the tracked URL

12. echo "more stuff" > blah.txt & gsutil cp blah.txt gs://hasha-ds-public/test_dir/

Finally, try pulling the frozen data associated with the imported URL from remote into the fresh clone
13. cd dvc-import-issue2
14. dvc pull -r origin

Result

Collecting                                                                                                          |5.00 [00:00, 11.3entry/s]
Fetching
Building workspace index                                                                                            |0.00 [00:00,    ?entry/s]
Comparing indexes                                                                                                   |6.00 [00:00, 14.9entry/s]
WARNING: No file hash info found for '/Users/alexhasha/repos/dvc-import-issue2/test_dir/baz.txt'. It won't be created.                        
WARNING: No file hash info found for '/Users/alexhasha/repos/dvc-import-issue2/test_dir/blah.txt'. It won't be created.                       
WARNING: No file hash info found for '/Users/alexhasha/repos/dvc-import-issue2/test_dir/bar.txt'. It won't be created.                        
WARNING: No file hash info found for '/Users/alexhasha/repos/dvc-import-issue2/test_dir/foo.txt'. It won't be created.                        
Applying changes                                                                                                    |0.00 [00:00,     ?file/s]
4 files fetched
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
test_dir
Is your cache up to date?
<https://error.dvc.org/missing-files>

Note that it's trying to checkout blah.txt in the error message, though that file wasn't present when the directory was imported and test_dir.dvc suggests there should only be 3 files in the directory.*-

I have pushed this repository to https://github.com/ahasha/dvc-import-url-pull-issue . You should be able to reproduce the error message by cloning it and running dvc pull -r origin.

Expected

1. After running dvc import-url, dvc push should upload the version of the data tracked by the created .dvc file to the remote
2. In a fresh clone of the repository, dvc pull should retrieve the version of the data tracked by the .dvc file from the remote, and not download anything new from the tracked URL.

Environment information

Output of dvc doctor:

DVC version: 3.36.0 (pip)
-------------------------
Platform: Python 3.11.4 on macOS-14.1.2-x86_64-i386-64bit
Subprojects:
	dvc_data = 3.2.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3)
Config:
	Global: /Users/alexhasha/Library/Application Support/dvc
	System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s3s1
Caches: local
Remotes: gs
Workspace directory: apfs on /dev/disk1s3s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/a48422a1ed3a65e3efe064e5b18830a7

[dberenbaum]

Thanks for the report @ahasha. I can confirm at least the part about push not doing anything, and I can reproduce even on a local remote with this script:

set -eux
IMPORT=$(mktemp -d)
REPO=$(mktemp -d)
REMOTE=$(mktemp -d)
echo foo > $IMPORT/foo
tree $IMPORT
cd $REPO
git init -q
dvc init -q
dvc remote add -d default $REMOTE
dvc import-url $IMPORT dir
tree dir
dvc push
tree $REMOTE

[dberenbaum]

@efiop I'm marking as p0 for now because it looks severe and it's unclear if it impacts anything besides import-url.

[skshetry]

It was regressed from 3.27.1 after:

• push: use index push #9807).

Looks like a problem on how we collect storages/indexes.

[ahasha-ml]

The pushed data not showing up in remote was a surprising new observation and not the bug that initially led me to open this issue, so I want to make sure this issue isn't considered resolved if that's corrected.

The "No file has info found" error I'm reporting originally occurred in an environment running 3.27.0. In that instance, I saw that the import-url data was in the remote, but dvc pull wasn't successfully fetching it. FWIW, I had rolled that environment back to 3.27.0 because of #10124

Here's the dvc doctor output from that environment:

DVC version: 3.27.0 (pip)
-------------------------
Platform: Python 3.11.4 on Linux-5.15.0-1047-gcp-x86_64-with-glibc2.31
Subprojects:
	dvc_data = 2.18.2
	dvc_objects = 1.2.0
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.4.1
Supports:
	gs (gcsfs = 2023.9.2),
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	ssh (sshfs = 2023.7.0)
Config:
	Global: /home/alex_hasha/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: nfs on 10.186.1.106:/data
Caches: local
Remotes: gs, gs, gs, gs
Workspace directory: nfs on 10.186.1.106:/data
Repo: dvc (subdir), git
Repo.site_cache_dir: /var/tmp/dvc/repo/993bbff78ba8ef27bcf733cfb308e638

[efiop]

Ok, so just for the record: we set remote storage for imports as the one that the source data is coming from and that is enough for import (because we don't push or pull it to regular remotes) but not for import-url, hence this problem. I'm looking into fixing that.

[efiop]

@ahasha Could you give an upstream dvc a try, please?

[ahasha-ml]

Thanks! I'm using poetry, so I installed the upstream version using poetry add git+ssh://[email protected]:iterative/dvc.git
It installed fine, but I'm getting a surprising missing dependency error even though the dvc-gs package is installed. Any thoughts on what might be behind this?

$ dvc pull --allow-missing
WARNING: failed to collect 'workspace', skipping                      
ERROR: unexpected error - gs is supported, but requires 'dvc-gs' to be installed: No module named 'dvc_objects.fs.path'
$ poetry show | grep dvc
dvc                           3.37.1.dev4+gdad464285 dad4642
dvc-data                      3.6.0                         
dvc-gs                        2.22.1                        
dvc-http                      2.30.2                        
dvc-objects                   3.0.3                         
dvc-render                    1.0.0                         
dvc-ssh                       2.22.2                        
dvc-studio-client             0.18.0                        
dvc-task                      0.3.0                         

[ahasha-ml]

It looks like I just need dvc-gs>=3.0.0 to avoid this error, but that's not enforced in any of the dependency files that I can tell.

[ahasha-ml]

The upstream version is successfully pulling my tracked directory data from the remote! 🍾 🎆

Additionally, I ran the script to reproduce #10124 with this issue and I got the behavior I had requested. I don't know if this was intentional on your part.

[efiop]

@ahasha-ml Thanks for the feedback! Yes, #10124 had the same underlying problem, so got fixed as well. Glad it works for you now, thank you!