v0.15.0: Pytorch 2.0 stack support

PyTorch 2.0 stack support

We are very excited by the newly announced PyTorch 2.0 stack and you can try it using Accelerate on any model by using the dynamo_backend argument of the Accelerator, or when filling your config with accelerate config.

Note that to get the best performance, we recommend:

• using an Ampere GPU (or more recent)
• sticking to fixed shaped for now

• Add support for torch dynamo by @sgugger in #829

New CLI commands

• Added two new commands, accelerate config update and accelerate config default. The first will update a config file to have the latest keys added from latter releases of Accelerate, and the second will create a default configuration file automatically mimicking write_default_config() introduced in #851 and #853 by @muellerzr
• Also introduced a filterable help for accelerate launch which will show options relevant to the choices shown, such as accelerate launch --multi_gpu will show launch parameters relevant to multi-gpu training.

What's new?

• fix 🐛 by @pacman100 in #836
• Deepspeed example should use gather_for_metrics by @HammadB in #821
• Highlight selection with pretty colors by @muellerzr in #839
• Add join_uneven_inputs context manager to Accelerator by @Chris-hughes10 in #820
• Introduce default-config command by @muellerzr in #840
• Fix log error and add log level to get_logger by @muellerzr in #842
• Fix if/else by @muellerzr in #849
• Fix complete_cv example by @muellerzr in #848
• Refactor Accelerate config and introduce a multi-argument CLI interface by @muellerzr in #851
• Clean up, add update command by @muellerzr in #853
• Revert "Update pr docs actions by @mishig25 in #827)"
• Switch default log to warn by @muellerzr in #859
• Remove mixed precision hook as part of the unwrap_model by @muellerzr in #860
• update deepspeed error message wrt batch_size by @pacman100 in #861
• fix failing deepspeed test by @pacman100 in #868
• Even more log level refined, leave alone if not explicitly set by @muellerzr in #871
• Solve pickling issues by @muellerzr in #872
• Spring cleaning by @muellerzr in #865
• fixing lr_scheduler prepare issue when using pytorch nightly by @pacman100 in #878
• fix fsdp state_dict_config because of PyTorch changes by @pacman100 in #877
• Update deprecated logging warn by @SHi-ON in #881
• fix a bug by @xiaohu2015 in #887
• Allow safetensors offload by @sgugger in #873
• fixing lr scheduler for pytorch nightly by @pacman100 in #884
• Prefix all accelerate env vars with ACCELERATE by @muellerzr in #890
• fix prefix issues in tests by @pacman100 in #891
• Fix windows cli selector by @muellerzr in #893
• Better description for improper kwargs by @muellerzr in #894
• Support bfloat16 in load_offloaded_weight by @sgugger in #892

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @Chris-hughes10
  • Add join_uneven_inputs context manager to Accelerator (#820)