Install on EC2

[mason-apmc]

I'm trying to set up a training run on EC2 on a g5.2xlarge on x86 Ubuntu 22.04 LTS, and am hitting an issue with numpy during the first attempt at a training run as per the docs:

$ bash prepare.sh --stage -1 --stop-stage 3
We will download the LibriTTS dataset by default. If the downloading fails or you want to download the dataset yourself, see the comments in this script for steps.
2023-08-30 05:12:18 (prepare.sh:72:main) dl_dir: /data/valle/valle/examples/libritts/download
2023-08-30 05:12:18 (prepare.sh:73:main) Stage 0: Download data
/data/valle/valle/examples/libritts/download/dev-clean.tar.gz: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 1.29G/1.29G [01:23<00:00, 15.4MB/s]
/data/valle/valle/examples/libritts/download/dev-other.tar.gz: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 925M/925M [05:02<00:00, 3.06MB/s]
/data/valle/valle/examples/libritts/download/test-clean.tar.gz: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 1.23G/1.23G [00:40<00:00, 30.3MB/s]
/data/valle/valle/examples/libritts/download/test-other.tar.gz: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 965M/965M [00:35<00:00, 27.0MB/s]
/data/valle/valle/examples/libritts/download/train-clean-100.tar.gz: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7.72G/7.72G [03:53<00:00, 33.1MB/s]
/data/valle/valle/examples/libritts/download/train-clean-360.tar.gz: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 27.5G/27.5G [15:28<00:00, 29.6MB/s]
/data/valle/valle/examples/libritts/download/train-other-500.tar.gz: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 44.6G/44.6G [22:34<00:00, 32.9MB/s]
/data/valle/valle/examples/libritts/download/train-other-500.tar.gz: 100%|█████████████████████████████████████████████████████████████████████████████████████████▉| 44.6G/44.6G [22:34<00:00, 33.8MB/s]
Downloading LibriTTS parts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [1:16:24<00:00, 654.89s/it]
2023-08-30 06:28:44 (prepare.sh:87:main) Stage 1: Prepare LibriTTS manifest
Scanning audio files (*.wav): 5736it [00:00, 7018.93it/s]                                                                                                                          | 0/7 [00:00<?, ?it/s]
Scanning audio files (*.wav): 5057it [00:00, 6501.76it/s]                2023-08-30 06:28:48,502 WARNING [qa.py:200] Trimmed 576 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 4613it [00:00, 7907.25it/s]                                                                                                                  | 1/7 [00:02<00:13,  2.18s/it]
Scanning audio files (*.wav): 4142it [00:00, 7395.99it/s]                2023-08-30 06:28:50,464 WARNING [qa.py:200] Trimmed 500 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 4837it [00:00, 7479.02it/s]██████████████                                                                                                    | 2/7 [00:04<00:09,  1.99s/it]
Scanning audio files (*.wav): 3946it [00:00, 7347.71it/s]                2023-08-30 06:28:52,364 WARNING [qa.py:200] Trimmed 576 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 5120it [00:00, 6896.45it/s]██████████████████████████████████                                                                                | 3/7 [00:05<00:07,  1.96s/it]
Scanning audio files (*.wav): 4575it [00:00, 6357.55it/s]                2023-08-30 06:28:54,445 WARNING [qa.py:200] Trimmed 518 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 33236it [00:05, 6356.01it/s]█████████████████████████████████████████████████████                                                            | 4/7 [00:08<00:06,  2.01s/it]
Scanning audio files (*.wav): 33041it [00:05, 5078.98it/s]                2023-08-30 06:29:05,095 WARNING [qa.py:200] Trimmed 4230 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 116500it [00:18, 6249.08it/s]████████████████████████████████████████████████████████████████████████                                        | 5/7 [00:21<00:12,  6.08s/it]
Scanning audio files (*.wav): 116072it [00:18, 5053.65it/s]                2023-08-30 06:29:44,188 WARNING [qa.py:200] Trimmed 14977 supervisions exceeding the end of the recording.
Scanning audio files (*.wav): 205043it [00:35, 5748.69it/s]████████████████████████████████████████████████████████████████████████████████████████████                    | 6/7 [01:08<00:20, 20.20s/it]
Scanning audio files (*.wav): 204864it [00:35, 6510.34it/s]               2023-08-30 06:30:56,196 WARNING [libritts.py:249] No recording exists for utterance id 1092_134562_000013_000004, skipping (in /data/valle/valle/examples/libritts/download/LibriTTS/train-other-500/1092/134562/1092_134562.trans.tsv)
                                                                           2023-08-30 06:30:59,937 WARNING [qa.py:200] Trimmed 25486 supervisions exceeding the end of the recording.
Preparing LibriTTS parts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [02:31<00:00, 21.70s/it]
2023-08-30 06:31:19 (prepare.sh:99:main) Stage 2: Tokenize/Fbank LibriTTS
Traceback (most recent call last):
  File "/data/valle/valle/egs/libritts/bin/tokenizer.py", line 35, in <module>
    from valle.data import (
  File "/data/valle/valle/valle/__init__.py", line 1, in <module>
    from . import data, models, modules, utils
  File "/data/valle/valle/valle/data/__init__.py", line 1, in <module>
    from .datamodule import *
  File "/data/valle/valle/valle/data/datamodule.py", line 41, in <module>
    from valle.data.fbank import get_fbank_extractor
  File "/data/valle/valle/valle/data/fbank.py", line 25, in <module>
    from librosa.filters import mel as librosa_mel_fn
  File "/home/ubuntu/.local/lib/python3.10/site-packages/librosa/__init__.py", line 211, in <module>
    from . import core
  File "/home/ubuntu/.local/lib/python3.10/site-packages/librosa/core/__init__.py", line 9, in <module>
    from .constantq import *  # pylint: disable=wildcard-import
  File "/home/ubuntu/.local/lib/python3.10/site-packages/librosa/core/constantq.py", line 1059, in <module>
    dtype=np.complex,
  File "/home/ubuntu/.local/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'complex'.
`np.complex` was a deprecated alias for the builtin `complex`. To avoid this error in existing code, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'complex_'?

When I try to walk back the dependencies to an earlier version of numpy to resolve the above deprecation, it ends up being conflicts to no end, which I assume means that there's a different system-level conflict which is at play here.

Is there a recommended EC2 instance-type, or Ubuntu version, or numpy version, or all of the above, that has worked for others to set this project up on?

[guoaoo]

I recommend you to conduct experiments using the Docker environment, and you can refer to this Dockerfile.

vall-e/docker/Dockerfile

Lines 1 to 26 in a1b44ca

	FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-devel
	RUN apt-get update && \
	apt-get upgrade -y
	RUN apt-get install -y vim wget git libsndfile1 espeak-ng
	RUN pip install torchmetrics==0.11.1 librosa==0.8.1 phonemizer==3.2.1 pypinyin==0.48.0 lhotse matplotlib h5py
	RUN pip install https://huggingface.co/csukuangfj/k2/resolve/main/cuda/k2-1.23.4.dev20230224+cuda11.6.torch1.13.1-cp310-cp310-linux_x86_64.whl
	RUN pip install torchaudio==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
	WORKDIR /workspace
	RUN git clone https://github.com/k2-fsa/icefall && \
	cd icefall && \
	pip install -r requirements.txt
	ENV PYTHONPATH=/workspace/icefall:$PYTHONPATH
	RUN cp /opt/conda/lib/libpython3.10.so.1.0 /usr/lib/x86_64-linux-gnu/
	WORKDIR /workspace
	RUN git clone https://github.com/lifeiteng/vall-e.git && \
	cd vall-e && \
	pip install -e .
	WORKDIR /workspace/vall-e