update docs

[shaoxiongji]

• add scripts for translation
• enhance tutorial

[TimotheeMickus]

Minor changes suggested, but it's only a suggestion.

[TimotheeMickus]

Feels like we should have a tutorial page about distributed training.

This page should

1. mention that mammoth was written with distributed modular training on SLURM clusters in mind
2. explain world_size (total number of devices) / gpu_ranks (devices visible on the node) in the configs and the node_gpu (device on which a task is ran) in the task configs
3. provide a wrapper, and explain
   • why it's necessary (i.e., so that we can set variables specific to a SLURM node)
   • and how it works (arguments declared inside wrapper.sh are not evaluated until the wrapper is ran on a node, so they are node specific; arguments declared outside of the config are evaluated globally and first, so they are shared by all nodes).
4. provide examples that work in multi-gpu and multi-node setting.

We might also want a page about how to define a task, and how this links to the vocab, encoder_layers, decoder_layers and so on:

1. mention the config has a dict tasks, where keys are unique task identifiers and values are structured task definitions that will define what your model does (we have the (i) (ii) (iii) criteria from the demo paper for swahili to catalan one could include here)
2. explain the sharing groups and how their shapes are defined globally by the encoder_layers and decoder_layers
3. explain how the src_tgt task key links a task to the source and target vocabs defined globally, and mention how to do explicit vocab sharing. We probably want to stress that embeddings are defined per vocab at this stage
4. explain how the physicial compute device is decided with node_gpu and link back to the distributed training page
5. provide other keys, in particular path_valid_{src,tgt} to define validation loops and transforms

We probably also want to change the 101, quickstart, and sharing scheme pages such that

• the example training only expects 1 GPU (it shouldn't fail if you have only 1 GPU but will silently ignore some of the tasks)
• the wrapper not presented in the sharing schemes page
• there are some pointers to the task definition page

[TimotheeMickus]

The provided command seems to be a mix between the wrapper's internal and a default wrapper-free call. You probably want one if the two following:

either run with a wrapper so it can handle multiple nodes; with the wrapper provided in sharing_schemes.md

srun wrapper.sh \
    ${PATH_TO_MAMMOTH}/train.py \
    -config my_config.yaml \
    -master_port 9974 \
    -master_ip ${SLURM_NODENAME} \
    # and maybe -tensorboard -tensorboard_log_dir -save_model

or the default python call:

python3 -u \
    ${PATH_TO_MAMMOTH}/train.py \
    -config my_config.yaml \
    # and maybe -tensorboard -tensorboard_log_dir -save_model    

In the latter case -master_port 9974 and -master_ip ${SLURM_NODENAME} should no longer be required.

[shaoxiongji]

mostly fixed. Quickstart removes wrapper.sh (i guess for simplicity). I will leave the distributed training later maybe a tutorial in a new page.

[TimotheeMickus]

@stefanik12 does the call to train.py still work if people install it from pip?