Link dvc data folder to dvc algorithm folder

[jdso1988]

Hi, I have a repo that contains all my algorithm stuff (scripts, .cfg files, etc but NO data) on the other hand I got a different DVC repo that contains only data (images, annotations files). I'm wondering how I can link those two folders to prevent copying my data into the algorithm (I check over the soft-link funcion but still not sure if I can ). If yes, can you give me some example?... thanks

[shcheklein]

Hey, one way that comes to my mind:

• shared cache in both repos
• setup symlinks, hardlinks, or reflinks
• use dvc import in the second (algorithm) repo to get data from the first one. See this scenario: https://dvc.org/doc/use-cases/data-registry/tutorial

I hope that makes sense. Let us know if something is not working.

[shcheklein]

Could you please run dvc version and share it? (in both folderA and folderX). Also, when you do dvc import, could you run it with -v and share the result?

[jdso1988]

Hello

FOLDER A:

DVC version: 2.10.1 (exe)
---------------------------------
Platform: Python 3.8.10 on Windows-10-10.0.19044-SP0
Supports:
        azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
        gdrive (pydrive2 = 1.10.1),
        gs (gcsfs = 2022.3.0),
        hdfs (fsspec = 2022.3.0, pyarrow = 7.0.0),
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.3.0, boto3 = 1.21.21),
        ssh (sshfs = 2022.3.1),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: s3
Workspace directory: NTFS on C:\
Repo: dvc (subdir), git
----------------------------------------------------------------

FOLDER X

DVC version: 2.10.1 (exe)
---------------------------------
Platform: Python 3.8.10 on Windows-10-10.0.19044-SP0
Supports:
        azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
        gdrive (pydrive2 = 1.10.1),
        gs (gcsfs = 2022.3.0),
        hdfs (fsspec = 2022.3.0, pyarrow = 7.0.0),
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.3.0, boto3 = 1.21.21),
        ssh (sshfs = 2022.3.1),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc (no_scm)

dvc import xxxxx -v 

2022-11-22 23:04:26,560 DEBUG: Removing output 'my_data' of stage: 'my_data.dvc'.
2022-11-22 23:04:26,560 DEBUG: Removing 'C:\Users\LTI\Projects\folderX\my_data'
Importing 'my_data (C:\Users\LTI\Projects\folderA\dataA)' -> 'my_data'
2022-11-22 23:04:26,565 DEBUG: Computed stage: 'my_data.dvc' md5: '9e8acdf1fd61d38a7e6bb8da4ed71c7c'
2022-11-22 23:04:26,565 DEBUG: 'md5' of stage: 'my_data.dvc' changed.
2022-11-22 23:04:26,569 DEBUG: Creating external repo C:\Users\LTI\Projects\folderA\dataA@None
2022-11-22 23:04:26,571 DEBUG: erepo: git clone 'C:\Users\LTI\Projects\folderA\dataA' to a temporary dir
2022-11-22 23:04:26,734 ERROR: failed to import 'my_data' from 'C:\Users\LTI\Projects\folderA\dataA'. - Failed to clone repo 'C:\Users\LTI\Projects\folderA\dataA' to 'C:\Users\LTI\AppData\Local\Temp\tmphyh8wwzudvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "scmrepo\git\backend\dulwich\__init__.py", line 193, in clone
  File "dulwich\porcelain.py", line 443, in clone
  File "dulwich\client.py", line 535, in clone
  File "dulwich\client.py", line 1444, in fetch
  File "dulwich\client.py", line 1366, in _open_repo
  File "dulwich\repo.py", line 1070, in __init__
dulwich.errors.NotGitRepository: No git repository was found at C:\Users\LTI\Projects\folderA\dataA

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "dvc\scm.py", line 106, in clone
  File "scmrepo\git\__init__.py", line 143, in clone
  File "scmrepo\git\backend\dulwich\__init__.py", line 196, in clone
scmrepo.exceptions.CloneError: Failed to clone repo 'C:\Users\LTI\Projects\folderA\dataA' to 'C:\Users\LTI\AppData\Local\Temp\tmphyh8wwzudvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\commands\imp.py", line 15, in run
  File "dvc\repo\imp.py", line 6, in imp
  File "dvc\repo\__init__.py", line 48, in wrapper
  File "dvc\repo\scm_context.py", line 152, in run
  File "dvc\repo\imp_url.py", line 83, in imp_url
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 533, in run
  File "funcy\decorators.py", line 45, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\stage\__init__.py", line 557, in _sync_import
  File "dvc\stage\imports.py", line 47, in sync_import
  File "dvc\dependency\repo.py", line 66, in download
  File "dvc\dependency\repo.py", line 94, in get_used_objs
  File "dvc\dependency\repo.py", line 107, in _get_used_and_obj
  File "contextlib.py", line 113, in __enter__
  File "dvc\external_repo.py", line 36, in external_repo
  File "dvc\external_repo.py", line 162, in _cached_clone
  File "funcy\decorators.py", line 45, in wrapper
  File "funcy\flow.py", line 274, in wrap_with
  File "funcy\decorators.py", line 66, in __call__
  File "dvc\external_repo.py", line 232, in _clone_default_branch
  File "dvc\scm.py", line 108, in clone
dvc.scm.CloneError: Failed to clone repo 'C:\Users\LTI\Projects\folderA\dataA' to 'C:\Users\LTI\AppData\Local\Temp\tmphyh8wwzudvc-clone'
------------------------------------------------------------
2022-11-22 23:04:26,740 DEBUG: Analytics is enabled.
2022-11-22 23:04:26,743 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\LTI\\AppData\\Local\\Temp\\tmpmmtwbz5q']'
2022-11-22 23:04:26,819 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\LTI\\AppData\\Local\\Temp\\tmpmmtwbz5q']'

[shcheklein]

okay, so the issue is that symlinks are not enabled (only hardlinks are, Cache types: hardlink). Please try to look into https://dvc.org/doc/user-guide/how-to/run-dvc-on-windows#enable-symbolic-links (and other things on this page) since you are using Windows.

Hardlinks will work also fine if both project are located on the same volume (C: in this case).

[jdso1988]

Ok, the annotations already worked... the images too (at the end it says that it found an error but I see that it got the images with the created link icon on it). Thank you !

[jdso1988]

fsutil behavior set symlinkEvaluation L2L:1 L2R:0 R2R:0 R2L:0 and login as administrator were the ones that managed to unlock the stuff