lighteval support after checkpoint, UX refactor #222

eliebak · 2024-08-24T19:27:42Z

Info

Follow up of the s3upload PR220

Changes

Automated Lighteval Job Execution
- Triggers after each checkpoint save (locally or on S3)
- Implemented via:
  - self.s3_mover.post_upload_callback (S3 enabled)
  - self.post_checkpoint_callback (local)
Enhanced Wandb
- Store ID and project in general args
- Lof tokens in the x axis instead of steps
- Enables logging for both training and evaluation
New scripts for users: launcher.py and create_config.py
Fix model parameter counting
- New function: model_config.get_llama_param_count()
- Updated get_flops() method:
  - Corrected block_compute_costs:
    - Included layer norm and GQA calculation for FLOPS
Customizable Slurm Folder
- Separate modifications for training and evaluation jobs

Usage Instructions

`create_config.py`

Modify the Python file as needed
Run: python create_config.py --save_path <path_to_save>

`launcher.py`

Basic usage:
python launcher.py --logs_path <path_to_log> --run <name_of_the_run>

Creates timestamped folder in logs_path/run
Numbering: Starts at 001 for first run

Additional options:

Slurm: Add --slurm --nodes <number_of_node>
Override config arguments:
python nanotron/launcher.py [other_args] --override KEY1=VALUE1 KEY2=VALUE2 ...

Minor Changes

Changed 'G' to 'B' in human format function
Lighteval branch fixes:

Added evaluation to Wandb
Set trust_remote_code=True globally
Created repository for uploading details and results

Recommended Workflow

Create a new folder for each project
Copy and customize in the folder:

launcher.py
create_config.py
slurm folder (if using Slurm)

Fancy prints ✨

Command
python launcher.py --base-config "smollm-360M-4nodes" --run "smol-360M" --override "tokens.micro_batch_size=8" "optimizer.learning_rate_scheduler.learning_rate=1e-3" --slurm --nodes 4
Output

⇄ Applied overrides:
  tokens.micro_batch_size=8
  optimizer.learning_rate_scheduler.learning_rate=1e-3

🏋️  Model Parameters:
┌───────────────────────┬────────────────────────┐
│ Total Parameters      │                   362M │
│ Layers                │                     32 │
│ Attention Heads       │                     15 │
│ Hidden Size           │                    960 │
│ Intermediate Size     │                   2560 │
│ Context Length        │                   2048 │
│ Tokenizer             │ HuggingFaceTB/cosmo2-t │
│ Vocab Size            │                  49152 │
└───────────────────────┴────────────────────────┘


🎛️ Parallelism Configuration:
┌───────────────────────┬────────────────────────┐
│ Nodes                 │                      4 │
│ Total GPUs            │                     32 │
│ Data Parallel (DP)    │                     32 │
│ Pipeline Parallel (PP)│                      1 │
│ Tensor Parallel (TP)  │                      1 │ 
└───────────────────────┴────────────────────────┘


📙 Training Configuration:
┌───────────────────────┬────────────────────────┐
│ Total Tokens          │                   629B │
│ Global Batch Size     │              1,048,576 │
│ Batch Size (per GPU)  │                 32,768 │
└───────────────────────┴────────────────────────┘


📊 Learning Rate Schedule:
┌───────────────────────┬────────────────────────┐
│ Initial LR            │               1.00e-03 │
│ Warmup Style          │                 linear │
│ Warmup Steps          │                   5000 │
│ Decay Style           │                 1-sqrt │
│ Decay Start Step      │                 500000 │
│ Decay Steps           │                 100000 │
│ Final LR              │               0.00e+00 │
└───────────────────────┴────────────────────────┘


🔧 Optimization Configuration:
┌───────────────────────┬────────────────────────┐
│ Optimizer             │     AdamWOptimizerArgs │
│ Weight Decay          │               1.00e-02 │
│ Gradient Clipping     │                   1.00 │
│ Adam Epsilon          │               1.00e-08 │
│ Adam Beta1            │                   0.90 │
│ Adam Beta2            │                   0.95 │
│ ZeRO Stage            │                      0 │
│ FP32 Grad Accumulation│                   True │
└───────────────────────┴────────────────────────┘

🚀 Slurm job launched with id=8552705
    🤖 Slurm Configuration Details:
        qos: high
        gpus_per_node: 8
        cpus_per_task: 88
        
    📁 Log structure:
    logs/smol-360M/
    └── run001_2024-09-03_05-45-23/
        ├── config/
        ├── launch-script/
        ├── slurm-logs/
        └── evals/
            ├── launch-config/
            └── logs/

…h and eval

…ange this condition just to slurm in the future and support local lighteval)

…the vf

… may work

…er big changes

…lighteval and log the eval to wandb

…h, solve serializaton pb of xPath

…rent places

zzhhjjj · 2024-09-13T12:45:40Z

What's the reason we have to "Install lighteval from https://github.com/eliebak/lighteval/tree/current-nanotron otherwise it will not work"

eliebak · 2024-09-13T16:07:59Z

What's the reason we have to "Install lighteval from https://github.com/eliebak/lighteval/tree/current-nanotron otherwise it will not work"

I changed the installation to https://github.com/huggingface/lighteval/tree/nanotron-compatible in the .toml file. It's an older version of lighteval that work with the current code, at the time i made the change lighteval was changing a bit so used an older commit that was the one for the FW ablation. I (or other ppl) can update to the current version after :)

xrsrke · 2024-09-24T18:52:21Z

src/nanotron/trainer.py

@@ -287,16 +304,30 @@ def pre_training(self, *args, **kwargs):
            rank=0,
        )

-        current_time = datetime.datetime.now().strftime("%d/%m/%Y_%H:%M:%S")
-        if dist.get_rank(self.parallel_context.world_pg) == self.logger_ranks[0] and wandb is not None:
+        datetime.datetime.now().strftime("%d/%m/%Y_%H:%M:%S")


xrsrke · 2024-09-24T18:52:54Z

src/nanotron/trainer.py

+            # Log initial tokens to set the starting point
+            wandb.log({"Tokens": initial_tokens})
+
+            print(f"Initial Tokens: {initial_tokens}")


log_rank(message, logger=logger, level=logging.INFO, rank=0)

xrsrke · 2024-09-24T18:54:43Z

src/nanotron/trainer.py

+        if wandb is not None and dist.get_rank(self.parallel_context.dp_pg) == 0:
+            if self.config.general.wandb_id is None:
+                self.config.general.wandb_id = wandb.run.id
+                self.config.general.wandb_project = wandb.run.project


why don't we keep it as the default in the initial config?

was because we previously need to store the wandb.run.id (and project) to pass it to lighteval to save the eval on the same wandb run. But we don't save to wandb anymore so no need for this will remove it thanks for noticing it!

xrsrke · 2024-09-24T18:54:53Z

src/nanotron/trainer.py

+                    "Update the wandb run due too resume from checkpoint", logger=logger, level=logging.WARNING, rank=0
+                )
+                self.config.general.wandb_id = wandb.run.id
+                self.config.general.wandb_project = wandb.run.project


same as above

xrsrke · 2024-09-24T18:55:48Z

src/nanotron/utils.py

 from contextlib import ExitStack, contextmanager
 from typing import ContextManager, List, Optional
+import json


seems like we don't use it? could u rerun precommit please

xrsrke · 2024-09-24T18:55:58Z

pyproject.toml

@@ -60,6 +60,9 @@ s3 = [
    "s5cmd",
 ]

+lighteval = [
+    "lighteval[nanotron]@git+https://github.com/huggingface/lighteval.git",


xrsrke · 2024-09-24T18:56:23Z

create_config.py

+        tokens=tokens,
+        optimizer=optimizer,
+        data_stages=data_stages,
+        lighteval=lighteval,


not defined variable?

xrsrke · 2024-09-24T18:57:01Z

src/nanotron/serialize/main.py

@@ -1,7 +1,10 @@
 import os
 from pathlib import Path
 from typing import Optional, cast
+from datasets.download.streaming_download_manager import xPath


rerun precommit

xrsrke · 2024-09-24T18:59:17Z

src/nanotron/config/config.py

+class S3UploadArgs:
+    """Arguments related to uploading checkpoints on s3"""
+
+    remove_after_upload: bool


should be True by default?

xrsrke · 2024-09-24T19:00:26Z

src/nanotron/config/config.py

+        if hasattr(self, "_post_init_done"):
+            return
+        self._post_init_done = True


this seems uncessary

xrsrke · 2024-09-24T19:01:42Z

src/nanotron/config/config.py

+    eval_slurm_config: Optional[str] = None
+    eval_slurm_template: Optional[str] = None
+    lighteval_config_path: Optional[str] = None
+    is_s3_available: Optional[bool] = None


remove? because we set in it line 406 and and 409?

i think it's more clear to have it in the config and define it later no? don't know what's the standard practice here (update: i remove it i think it's better you're right)

xrsrke · 2024-09-24T19:02:11Z

src/nanotron/config/config.py

@@ -490,3 +536,14 @@ def get_config_from_file(
            )
        config.model.model_config = model_config_class(**config.model.model_config)
    return config
+
+
+def save_as_yaml(config, config_class, file_path: str):


add type hints

xrsrke

Looks good overall. I left a few requested changes and one question: When is the lighteval supposed to run? It doesn’t seem to launch any lighteval runs after each checkpoint is saved on slurm? Thanks

eliebak · 2024-09-24T20:50:46Z

Thanks for the review, made the changes! :) It launch a slurm job using run_slurm_one_job for the evaluation here if no s3 and here if you upload ckpt to s3 (same way as in brrr for the s3 case)

3outeille · 2024-09-25T15:57:01Z

src/nanotron/trainer.py

                # LogItem("consumed_samples", self.consumed_train_samples, "human_format"),  # , "12d"),
-                LogItem(
-                    "consumed_tokens",
-                    self.metadata.consumed_train_samples * self.config.tokens.sequence_length,
-                    "human_format",
-                ),  # , "12d"),
+                # LogItem(
+                #     "consumed_tokens",
+                #     self.metadata.consumed_train_samples * self.config.tokens.sequence_length,
+                #     "human_format",
+                # ),  # , "12d"),


Why is this comment out ?

3outeille · 2024-09-25T16:00:37Z

src/nanotron/config/utils_config.py

+        elif isinstance(value, xPath):
+            result[field.name] = str(value)


Why xPath ? I thought we removed it for Path

No, we still need it in s3upload to upload to s3 (with path it consider s3 path as local path)

xrsrke

Hi, I think you made some breaking changes, the checkpoint saving is broke. reproduce:

CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc-per-node 8 --master_port=25621 run_train.py --config /fsx/phuc/temp/env_for_review_pr/nanotron/pr_config_new.yaml

trainer.train(dataloader)  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train

  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/run_train.py", line 237, in <module>
    trainer.train(dataloader)
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train
    self.save_checkpoint()
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    self = cls._from_parts(args)
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
    trainer.train(dataloader)
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 489, in train
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
    a = os.fspath(a)
    self.save_checkpoint()
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
TypeError: expected str, bytes or os.PathLike object, not NoneType
    self.save_checkpoint()
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
    self.save_checkpoint()
          File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
self.save_checkpoint()self.save_checkpoint()

  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
    self.save_checkpoint()
      File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
self.save_checkpoint()
  File "/fsx/phuc/temp/env_for_review_pr/nanotron/src/nanotron/trainer.py", line 912, in save_checkpoint
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    checkpoint_path = Path(checkpoints_path) / f"{self.iteration_step}"
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 960, in __new__
    self = cls._from_parts(args)
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
    self = cls._from_parts(args)
      File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
    self = cls._from_parts(args)    self = cls._from_parts(args)
self = cls._from_parts(args)
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts

  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
        drv, root, parts = self._parse_args(args)drv, root, parts = self._parse_args(args)

      File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
self = cls._from_parts(args)
self = cls._from_parts(args)  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts

  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 594, in _from_parts
    drv, root, parts = self._parse_args(args)
      File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
drv, root, parts = self._parse_args(args)
drv, root, parts = self._parse_args(args)  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args

  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
        a = os.fspath(a)a = os.fspath(a)

TypeErrorTypeError    : :     drv, root, parts = self._parse_args(args)expected str, bytes or os.PathLike object, not NoneTypeexpected str, bytes or os.PathLike object, not NoneTypedrv, root, parts = self._parse_args(args)



      File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/pathlib.py", line 578, in _parse_args
a = os.fspath(a)
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
TypeError: expected str, bytes or os.PathLike object, not NoneType
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[2024-10-02 20:09:03,528] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2082316 closing signal SIGTERM
[2024-10-02 20:09:03,529] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2082319 closing signal SIGTERM
[2024-10-02 20:09:03,529] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 2082323 closing signal SIGTERM
[2024-10-02 20:09:03,706] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 2082317) of binary: /fsx/phuc/temp/env_for_review_pr/env/bin/python
Traceback (most recent call last):
  File "/fsx/phuc/temp/env_for_review_pr/env/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/fsx/phuc/temp/env_for_review_pr/env/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================

eliebak · 2024-10-02T20:20:07Z

in your config the checkpoint_path is set to null checkpoint_path: null that's why you've got this error

eliebak added 14 commits August 23, 2024 03:32

adding slurm config as an argument to better generate slurm for launc…

5eb2549

…h and eval

working version of lighteval after s3 on 1 node

3875c60

add first version of launcher (still ugly)

e609d1c

not yet functional, lighteval stuff to figure out

a5c0cc2

remove torch.compile() bc it's not working (might be a me pb)

6b58c25

update launcher.py

34b50a6

fancy launcher

7a105be

fancy++ launcher

28770a5

add the possibility to override config yeahhh

7652089

don't run lighteval runner if no s3 uploader AND slurm (might want ch…

4e2d7d9

…ange this condition just to slurm in the future and support local lighteval)

add CUDA__DEVICE_MAX_CONNECTIONS=1 in interactive mode

e7f0437

add create_config, moove log_path to general

bb45352

fix launcher and create_config file, still need some improvement for …

e9d4a2e

…the vf

lot of changes, working on 1 node with s3, will test the rest soon it…

79ae2cb

… may work

eliebak changed the title ~~[WIP] Add lighteval support after checkpoint (when s3 upload) + launcher.py~~ [WIP] Add lighteval support after checkpoint + launcher.py Aug 30, 2024

eliebak marked this pull request as draft August 30, 2024 03:43

3outeille self-assigned this Aug 30, 2024

eliebak added 12 commits August 31, 2024 03:16

delete the SlurmArgs and add config to be more cluster agnostic + oth…

cfcbd70

…er big changes

update wandb restart logic + logging the id and project to pass it to…

0d43a95

…lighteval and log the eval to wandb

better wandb loggin, s3upload only for dl ckpt, correct Path and xPat…

207797e

…h, solve serializaton pb of xPath

fix some bug with the slurm related stuff

8ce8b18

add back slurm

6dd81b2

fix some stuff + introduce --base-config

4750736

fix the computation calculation by adding GQA and layer norm at diffe…

90860f5

…rent places

change the localisation of get_llama_param_count()

28b3847

change G to B i think it's better

157c2ae

last fix

6daa717

create_config is smollm-135M toy example

17bfd5f

last fix

43728d5

eliebak changed the title ~~[WIP] Add lighteval support after checkpoint + launcher.py~~ lighteval support after checkpoint, UX refactor Sep 3, 2024

eliebak added 3 commits September 10, 2024 13:29

remove the base-configs args

a7804f5

fix bs and gbs

efce15b

fix the logs structure

73da086

eliebak and others added 5 commits September 13, 2024 18:20

remove layer norm flops

67115a5

forget comma

6249264

put the comma in the right place

11d60c8

adapt it to the current lighteval main

5e8361c

remove print

43c833f

xrsrke reviewed Sep 24, 2024

View reviewed changes

xrsrke requested changes Sep 24, 2024

View reviewed changes

change after review

3d7c98f

3outeille reviewed Sep 25, 2024

View reviewed changes

uncomment logging item

e74ffd1

xrsrke requested changes Oct 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lighteval support after checkpoint, UX refactor #222

lighteval support after checkpoint, UX refactor #222

eliebak commented Aug 24, 2024 •

edited

Loading

zzhhjjj commented Sep 13, 2024

eliebak commented Sep 13, 2024 •

edited

Loading

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

eliebak Sep 24, 2024 •

edited

Loading

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

xrsrke Sep 24, 2024

eliebak Sep 24, 2024 •

edited

Loading

xrsrke Sep 24, 2024

xrsrke left a comment

eliebak commented Sep 24, 2024 •

edited

Loading

3outeille Sep 25, 2024

3outeille Sep 25, 2024

eliebak Sep 26, 2024 •

edited

Loading

xrsrke left a comment

eliebak commented Oct 2, 2024 •

edited

Loading

		elif isinstance(value, xPath):
		result[field.name] = str(value)

lighteval support after checkpoint, UX refactor #222

Are you sure you want to change the base?

lighteval support after checkpoint, UX refactor #222

Conversation

eliebak commented Aug 24, 2024 • edited Loading

Info

Changes

Usage Instructions

create_config.py

launcher.py

Minor Changes

Recommended Workflow

Fancy prints ✨

zzhhjjj commented Sep 13, 2024

eliebak commented Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eliebak Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eliebak Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xrsrke left a comment

Choose a reason for hiding this comment

eliebak commented Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eliebak Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

xrsrke left a comment

Choose a reason for hiding this comment

eliebak commented Oct 2, 2024 • edited Loading

eliebak commented Aug 24, 2024 •

edited

Loading

`create_config.py`

`launcher.py`

eliebak commented Sep 13, 2024 •

edited

Loading

eliebak Sep 24, 2024 •

edited

Loading

eliebak Sep 24, 2024 •

edited

Loading

eliebak commented Sep 24, 2024 •

edited

Loading

eliebak Sep 26, 2024 •

edited

Loading

eliebak commented Oct 2, 2024 •

edited

Loading