add/commit : command doesn't make cache on S3

[DonggookKim]

Bug Report

Description

I want to make cache on my Amazon S3 storage, not locally. So, I follow this guide to setting external cache.
But, after run dvc add --remote "s3 url", cache is disappear. That means, it is generated during command runs, but after terminate the job, cache files are disappear.

Reproduce

dvc remote add s3cache s3://dk-dvc-test/cache
dvc config cache.s3 s3cache

dvc add --external s3://dk-dvc-test/data

Expected

The file d4/bba--- is copied manually by my hands. I think there should make at least 2 cache file and 1 dir file.

dir

[
    {
        "etag": "c99cafc152244af753f735de768cd75f", 
        "relpath": "data_batch_1"
    }, 
    {
        "etag": "d4bba439e000b95fd0a9bffe97cbabec", 
        "relpath": "data_batch_2"
    }
]

Environment information

Macbook M1 Pro, Use zsh on PyCharm

config

[cache]
    s3 = s3cache
[core]
    remote = awss3
['remote "s3cache"']
    url = s3://dk-dvc-test/cache
['remote "awss3"']
    url = s3://dk-dvc-test/archiving

generated data.dvc

outs:
- etag: 88e0be3aa316fcd8a2f7e5c4f0d651ed.dir
  size: 62071024
  nfiles: 2
  path: s3://dk-dvc-test/data

Output of dvc doctor:

(3.9.10/envs/env) AL01983601:cifar10ex (remote_cache_s3*) $ dvc doctor
DVC version: 2.51.0 (pip)
-------------------------
Platform: Python 3.9.10 on macOS-12.1-arm64-i386-64bit
Subprojects:
        dvc_data = 0.44.1
        dvc_objects = 0.21.1
        dvc_render = 0.3.1
        dvc_task = 0.2.0
        scmrepo = 0.1.17
Supports:
        http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.3.0, boto3 = 1.24.59)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local, s3
Remotes: s3, s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/82d184ca7a1299683fd32a2be95f773e

Additional Information (if any):

[efiop]

@DonggookKim dvc add --remote only works with --to-remote and doesn't work with --external. Seems like there is some confusion here. Could you elaborate on your scenario, please?

[DonggookKim]

@efiop , I'm sorry for late response.
Here is my scenario.

• Make local dvc repository.
• Do actions on below sequentially.

dvc remote add s3cache s3://mybucket/cache
dvc config cache.s3 s3cache
dvc add --external s3://mybuctet/existing-data

• And then commit my local dataset by dvc commit or dvc add.
• Then, it make caches on s3://mybucket/cache, but after command finished the caches are gone.

[efiop]

@DonggookKim By scenario I meant what's your high level scenario, what are you trying to accomplish with this? The reason I'm asking is that --external is often misused and we usually don't recommend it. So I want to make you really need it in your scenario.

[DonggookKim]

@efiop I see.
I'm testing dvc remote cache for my coworkers who make model using very big data. They need to use over 1TB data and there is not enought machine to store duplicate of this data in my company. So, I think dvc can be solve this problem by exporting all local data to external S3 storage.
I don't know how dvc work, so I tested dvc external to CIFAR-10 data.

[efiop]

@DonggookKim Is that data static or are there some pipelines that update it directly on s3 (if so, do you have object versioning enabled in the bucket?)?

[DonggookKim]

@efiop Not yet, but we will migrate data to S3. On this time, data in NFS.

[efiop]

@DonggookKim And what about how static that data is? Is it static or are there going to be some pipelines that update it directly on the storage?

[DonggookKim]

@efiop Yes. Not now, it will be. Currently, there is a pipeline that preprocess for data and store to NFS and some static data also exists. I want to modify this pipeline store data to S3, and migrate static data to S3, if I can use dvc with S3 environment.

[efiop]

@DonggookKim I suppose some kind of ETL pipeline? If so, maybe you don't need external outputs or dvc at all at that stage of working with your data. You could depend on it in dvc pipelines or dvc import-url it into your dvc project.

[DonggookKim]

@efiop Yes. I design a ETL pipeline.
Actually, I use apache spark as ETL pipeline, and there will be several versions of dataset to be made. So, I think dvc will be useful on my job.
Thank you. I will check that dvc import-url command, and ask you later.