partial fetch is broken

[skshetry]

Partial fetch is broken in dvc.

Steps to Reproduce

#! /bin/bash

set -ex

gen() {
    mkdir -p "$1"
    for i in {00..99}; do echo "$1/$i" > "$1/${i}.txt"; done
}

setup() {
    pip install -q -e "."
    pushd "$1"
    dvc init -q --no-scm
    gen data/dir01
    gen data/dir02
    ls data
    find data/dir01 -type file | wc -l
    dvc remote add -q -d local "$(mktemp -d)"
    if ! dvc add data; then
        # fix umask imports
        pip install dvc-objects==1
        dvc add data
    fi
    dvc push
    command rm -rf .dvc/{cache,tmp} data
    popd
}

repo="$(mktemp -d)"
setup "$repo" || exit 125
pushd "$repo"
dvc pull data/dir01 || exit 125
# 100 files + .dir file
[ "$(find .dvc/cache -type file | wc -l)" -eq 101 ] || exit 1

This breaks an important scenario, see https://dvc.org/doc/user-guide/data-management/modifying-large-datasets#modifying-remote-datasets.

This regressed in 4c0bb8d (#9424) during the index migration.

git bisect start main 2342099bd876e4afe8da39d75578724de96f8346
git bisect run bash script.sh

[skshetry]

Also see https://discuss.dvc.org/t/why-dvc-pulls-full-dataset-instead-of-a-single-file/1897.

[skshetry]

This is very interesting. If I do len(idx), it tells me the correct number of files and only downloads a subset. If I don't do anything, it downloads everything.