Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Update files related to mim download dataset #2635

Merged
merged 2 commits into from
Sep 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions dataset-index.yml
Original file line number Diff line number Diff line change
@@ -1,39 +1,40 @@
openxlab: true
kinetics400:
dataset: Kinetics-400
dataset: OpenMMLab/Kinetics-400
download_root: data
data_root: data/kinetics400
script: tools/data/kinetics/k400_preprocess.sh
script: tools/data/kinetics/preprocess_k400.sh

kinetics600:
dataset: Kinetics600
dataset: OpenMMLab/Kinetics600
download_root: data
data_root: data/kinetics600
script: tools/data/kinetics/k600_preprocess.sh
script: tools/data/kinetics/preprocess_k600.sh

kinetics700:
dataset: Kinetics_700
dataset: OpenMMLab/Kinetics_700
download_root: data
data_root: data/kinetics700
script: tools/data/kinetics/k700_preprocess.sh
script: tools/data/kinetics/preprocess_k700.sh

sthv2:
dataset: sthv2
dataset: OpenDataLab/sthv2
download_root: data
data_root: data/sthv2
script: tools/data/sthv2/preprocess.sh

ucf-101:
dataset: UCF101
dataset: OpenDataLab/UCF101
download_root: data
data_root: data/ucf101

finegym:
dataset: FineGym
dataset: OpenDataLab/FineGym
download_root: data
data_root: data/gym

diving48:
dataset: diving48
dataset: OpenDataLab/diving48
download_root: data
data_root: data/diving48
script: tools/data/diving48/preprocess.sh
8 changes: 4 additions & 4 deletions tools/data/diving48/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ For basic dataset information, you can refer to the official dataset [website](h
````{group-tab} Download by MIM
MIM supports downloading from OpenDataLab and preprocessing Diving48 dataset with one command line.
```Bash
# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab
odl login
# install OpenXlab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login
# download and preprocess by MIM
mim download mmaction2 --dataset diving48
```
Expand Down
8 changes: 4 additions & 4 deletions tools/data/diving48/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@
````{group-tab} 使用 MIM 下载
# MIM 支持下载 Diving48 数据集。用户可以通过一行命令,从 OpenDataLab 进行下载,并进行预处理。
```Bash
# 安装 OpenDataLab CLI 工具
pip install -U opendatalab
# 登录 OpenDataLab
odl login
# 安装 OpenXLab CLI 工具
pip install -U openxlab
# 登录 OpenXLab
openxlab login
# 通过 MIM 进行数据集下载,预处理。注意这将花费较长时间
mim download mmaction2 --dataset diving48
```
Expand Down
2 changes: 1 addition & 1 deletion tools/data/diving48/preprocess.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/diving48/raw/*.tar.gz.* | tar -xvz -C $(dirname $DATA_ROOT)
cat $DOWNLOAD_DIR/OpenDataLab___diving48/raw/*.tar.gz.* | tar -xvz -C $(dirname $DATA_ROOT)
tar -xvf $DATA_ROOT/diving48.tar -C $(dirname $DATA_ROOT)
rm $DATA_ROOT/diving48.tar
8 changes: 4 additions & 4 deletions tools/data/kinetics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ All experiments on Kinetics in MMAction2 are based on this version, we recommend
MIM supports downloading from OpenDataLab and preprocessing Kinetics-400/600/700 dataset with one command line.

```Bash
# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab
odl login
# install OpenXlab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login
# download and preprocess Kinetics-400 by MIM. Note that this might take a long time.
mim download mmaction2 --dataset kinetics400
# download and preprocess Kinetics-600 by MIM. Note that this might take a long time.
Expand Down
8 changes: 4 additions & 4 deletions tools/data/kinetics/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ MMAction2 代码仓库中提供的 Kinetics 实验性能,都是基于这个版

# MIM 支持下载 Kinetics-400/600/700 数据集。用户可以通过一行命令,从 OpenDataLab 进行下载,并进行预处理。
```Bash
# 安装 OpenDataLab CLI 工具
pip install -U opendatalab
# 登录 OpenDataLab
odl login
# 安装 OpenXLab CLI 工具
pip install -U openxlab
# 登录 OpenXLab
openxlab login
# 通过 MIM 进行 Kinetics-400 数据集下载,预处理。注意这将花费较长时间
mim download mmaction2 --dataset kinetics400
# 通过 MIM 进行 Kinetics-600 数据集下载,预处理。注意这将花费较长时间
Expand Down
2 changes: 1 addition & 1 deletion tools/data/kinetics/preprocess_k400.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ set -x
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/Kinetics-400/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
cat $DOWNLOAD_DIR/OpenMMLab___Kinetics-400/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
mv $(dirname $DATA_ROOT)/Kinetics-400 $DATA_ROOT
2 changes: 1 addition & 1 deletion tools/data/kinetics/preprocess_k600.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ set -x
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/Kinetics600/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
cat $DOWNLOAD_DIR/OpenMMLab___Kinetics600/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
mv $(dirname $DATA_ROOT)/Kinetics600 $DATA_ROOT
2 changes: 1 addition & 1 deletion tools/data/kinetics/preprocess_k700.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ set -x
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/Kinetics_700/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
cat $DOWNLOAD_DIR/OpenMMLab___Kinetics_700/raw/*.tar.gz* | tar -xvz -C $(dirname $DATA_ROOT)
mv $(dirname $DATA_ROOT)/Kinetics_700 $DATA_ROOT
8 changes: 4 additions & 4 deletions tools/data/kinetics710/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ Before we start, please make sure that the directory is located at `$MMACTION2`.
Kinetics-710 is a video benchmark based on Kinetics-400/600/700, which merges the training set of these Kinetics datasets, and deletes the repeated videos according to Youtube IDs. MMAction2 provides an annotation file based on the Kinetics-400/600/700 on [OpenDataLab](https://opendatalab.com/). So we suggest you download Kinetics-400/600/700 first from OpenDataLab by [MIM](https://github.com/open-mmlab/mim).

```shell
# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab
odl login
# install OpenXlab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login
# download Kinetics-400/600/700, note that this might take a long time.
mim download mmaction2 --dataset kinetics400
mim download mmaction2 --dataset kinetics600
Expand Down
8 changes: 4 additions & 4 deletions tools/data/kinetics710/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@
Kinetics-710 是基于 Kinetics-400/600/700 的视频数据集,它合并了这些 Kinetics 数据集的训练集,并根据 Youtube ID 删除了重复的视频。MMAction2 提供了一个基于 Kinetics-400/600/700 的 OpenDataLab 版本的标注文件,你可以通过 [MIM](https://github.com/open-mmlab/mim) 从 OpenDataLab 下载。

```shell
# 安装 OpenDataLab CLI 工具
pip install -U opendatalab
# 登录 OpenDataLab
odl login
# 安装 OpenXLab CLI 工具
pip install -U openxlab
# 登录 OpenXLab
openxlab login
# 下载 Kinetics-400/600/700,注意这可能需要很长时间。
mim download mmaction2 --dataset kinetics400
mim download mmaction2 --dataset kinetics600
Expand Down
8 changes: 4 additions & 4 deletions tools/data/sthv2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ For basic dataset information, you can refer to the dataset [website](https://de
````{group-tab} Download by MIM
MIM supports downloading from OpenDataLab and preprocessing Something-Something V2 dataset with one command line.
```Bash
# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab
odl login
# install OpenXlab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login
# download and preprocess by MIM
mim download mmaction2 --dataset sthv2
```
Expand Down
8 changes: 4 additions & 4 deletions tools/data/sthv2/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@
````{group-tab} 使用 MIM 下载
# MIM 支持下载 Something-Something V2 数据集。用户可以通过一行命令,从 OpenDataLab 进行下载,并进行预处理。
```Bash
# 安装 OpenDataLab CLI 工具
pip install -U opendatalab
# 登录 OpenDataLab
odl login
# 安装 OpenXLab CLI 工具
pip install -U openxlab
# 登录 OpenXLab
openxlab login
# 通过 MIM 进行数据集下载,预处理。注意这将花费较长时间
mim download mmaction2 --dataset sthv2
```
Expand Down
2 changes: 1 addition & 1 deletion tools/data/sthv2/preprocss.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/sthv2/raw/*.tar.gz | tar -xvz -C $(dirname $DATA_ROOT)
cat $DOWNLOAD_DIR/OpenDataLab___sthv2/raw/*.tar.gz | tar -xvz -C $(dirname $DATA_ROOT)
tar -xvf $DATA_ROOT/sthv2.tar -C $(dirname $DATA_ROOT)
rm $DATA_ROOT/sthv2.tar