Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump to Ubuntu 22.04 #3540

Closed
wants to merge 1 commit into from
Closed

Bump to Ubuntu 22.04 #3540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
ARG CUDA_VERSION=11.3.1

# Calculate the base image based on CUDA_VERSION
ARG BASE_IMAGE=${CUDA_VERSION:+"nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu20.04"}
ARG BASE_IMAGE=${BASE_IMAGE:-"ubuntu:20.04"}
ARG BASE_IMAGE=${CUDA_VERSION:+"nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu22.04"}
ARG BASE_IMAGE=${BASE_IMAGE:-"ubuntu:22.04"}

# The Python version to install
ARG PYTHON_VERSION=3.10
Expand Down Expand Up @@ -250,7 +250,7 @@ ARG MOFED_VERSION

RUN if [ -n "$MOFED_VERSION" ] ; then \
wget -qO - http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - && \
wget -P /etc/apt/sources.list.d/ http://linux.mellanox.com/public/repo/mlnx_ofed/$MOFED_VERSION/ubuntu20.04/mellanox_mlnx_ofed.list && \
wget -P /etc/apt/sources.list.d/ http://linux.mellanox.com/public/repo/mlnx_ofed/$MOFED_VERSION/ubuntu22.04/mellanox_mlnx_ofed.list && \
apt-get update && \
apt-get install -y mlnx-ofed-dpdk-upstream-libs-user-only ; \
fi
Expand Down
18 changes: 9 additions & 9 deletions docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ To install composer, once inside the image, run `pip install mosaicml`.
<!-- BEGIN_PYTORCH_BUILD_MATRIX -->
| Linux Distro | Flavor | PyTorch Version | CUDA Version | Python Version | Docker Tags |
|----------------|----------|-------------------|---------------------|------------------|------------------------------------------------------------------------------------------|
| Ubuntu 20.04 | Base | 2.3.1 | 12.1.1 (Infiniband) | 3.11 | `mosaicml/pytorch:latest`, `mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu20.04` |
| Ubuntu 20.04 | Base | 2.3.1 | 12.1.1 (EFA) | 3.11 | `mosaicml/pytorch:latest-aws`, `mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu20.04-aws` |
| Ubuntu 20.04 | Base | 2.3.1 | cpu | 3.11 | `mosaicml/pytorch:latest_cpu`, `mosaicml/pytorch:2.3.1_cpu-python3.11-ubuntu20.04` |
| Ubuntu 20.04 | Base | 2.2.2 | 12.1.1 (Infiniband) | 3.11 | `mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu20.04` |
| Ubuntu 20.04 | Base | 2.2.2 | 12.1.1 (EFA) | 3.11 | `mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu20.04-aws` |
| Ubuntu 20.04 | Base | 2.2.2 | cpu | 3.11 | `mosaicml/pytorch:2.2.2_cpu-python3.11-ubuntu20.04` |
| Ubuntu 20.04 | Base | 2.1.2 | 12.1.1 (Infiniband) | 3.10 | `mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu20.04` |
| Ubuntu 20.04 | Base | 2.1.2 | 12.1.1 (EFA) | 3.10 | `mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu20.04-aws` |
| Ubuntu 20.04 | Base | 2.1.2 | cpu | 3.10 | `mosaicml/pytorch:2.1.2_cpu-python3.10-ubuntu20.04` |
| Ubuntu 22.04 | Base | 2.3.1 | 12.1.1 (Infiniband) | 3.11 | `mosaicml/pytorch:latest`, `mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu22.04` |
| Ubuntu 22.04 | Base | 2.3.1 | 12.1.1 (EFA) | 3.11 | `mosaicml/pytorch:latest-aws`, `mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu22.04-aws` |
| Ubuntu 22.04 | Base | 2.3.1 | cpu | 3.11 | `mosaicml/pytorch:latest_cpu`, `mosaicml/pytorch:2.3.1_cpu-python3.11-ubuntu22.04` |
| Ubuntu 22.04 | Base | 2.2.2 | 12.1.1 (Infiniband) | 3.11 | `mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu22.04` |
| Ubuntu 22.04 | Base | 2.2.2 | 12.1.1 (EFA) | 3.11 | `mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu22.04-aws` |
| Ubuntu 22.04 | Base | 2.2.2 | cpu | 3.11 | `mosaicml/pytorch:2.2.2_cpu-python3.11-ubuntu22.04` |
| Ubuntu 22.04 | Base | 2.1.2 | 12.1.1 (Infiniband) | 3.10 | `mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu22.04` |
| Ubuntu 22.04 | Base | 2.1.2 | 12.1.1 (EFA) | 3.10 | `mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu22.04-aws` |
| Ubuntu 22.04 | Base | 2.1.2 | cpu | 3.10 | `mosaicml/pytorch:2.1.2_cpu-python3.10-ubuntu22.04` |
<!-- END_PYTORCH_BUILD_MATRIX -->

**Note**: The `mosaicml/pytorch:latest`, `mosaicml/pytorch:latest_cpu`, and `mosaicml/pytorch:latest-aws`
Expand Down
40 changes: 20 additions & 20 deletions docker/build_matrix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# This file is automatically generated by generate_build_matrix.py. DO NOT EDIT!
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-3-1-cu121
MOFED_VERSION: latest-23.10
Expand All @@ -23,12 +23,12 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.3.1
TAGS:
- mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu20.04
- mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu22.04
- mosaicml/pytorch:latest
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.18.1
- AWS_OFI_NCCL_VERSION: v1.9.1-aws
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-3-1-cu121-aws
MOFED_VERSION: ''
Expand All @@ -51,12 +51,12 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.3.1
TAGS:
- mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu20.04-aws
- mosaicml/pytorch:2.3.1_cu121-python3.11-ubuntu22.04-aws
- mosaicml/pytorch:latest-aws
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.18.1
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: ubuntu:20.04
BASE_IMAGE: ubuntu:22.04
CUDA_VERSION: ''
IMAGE_NAME: torch-2-3-1-cpu
MOFED_VERSION: ''
Expand All @@ -66,12 +66,12 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.3.1
TAGS:
- mosaicml/pytorch:2.3.1_cpu-python3.11-ubuntu20.04
- mosaicml/pytorch:2.3.1_cpu-python3.11-ubuntu22.04
- mosaicml/pytorch:latest_cpu
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.18.1
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-2-2-cu121
MOFED_VERSION: latest-23.10
Expand All @@ -94,11 +94,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.2.2
TAGS:
- mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu20.04
- mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu22.04
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.17.2
- AWS_OFI_NCCL_VERSION: v1.9.1-aws
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-2-2-cu121-aws
MOFED_VERSION: ''
Expand All @@ -121,11 +121,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.2.2
TAGS:
- mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu20.04-aws
- mosaicml/pytorch:2.2.2_cu121-python3.11-ubuntu22.04-aws
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.17.2
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: ubuntu:20.04
BASE_IMAGE: ubuntu:22.04
CUDA_VERSION: ''
IMAGE_NAME: torch-2-2-2-cpu
MOFED_VERSION: ''
Expand All @@ -135,11 +135,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.2.2
TAGS:
- mosaicml/pytorch:2.2.2_cpu-python3.11-ubuntu20.04
- mosaicml/pytorch:2.2.2_cpu-python3.11-ubuntu22.04
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.17.2
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-1-2-cu121
MOFED_VERSION: latest-23.10
Expand All @@ -162,11 +162,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.1.2
TAGS:
- mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu20.04
- mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu22.04
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.16.2
- AWS_OFI_NCCL_VERSION: v1.9.1-aws
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA_VERSION: 12.1.1
IMAGE_NAME: torch-2-1-2-cu121-aws
MOFED_VERSION: ''
Expand All @@ -189,11 +189,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.1.2
TAGS:
- mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu20.04-aws
- mosaicml/pytorch:2.1.2_cu121-python3.10-ubuntu22.04-aws
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.16.2
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: ubuntu:20.04
BASE_IMAGE: ubuntu:22.04
CUDA_VERSION: ''
IMAGE_NAME: torch-2-1-2-cpu
MOFED_VERSION: ''
Expand All @@ -203,11 +203,11 @@
PYTORCH_NIGHTLY_VERSION: ''
PYTORCH_VERSION: 2.1.2
TAGS:
- mosaicml/pytorch:2.1.2_cpu-python3.10-ubuntu20.04
- mosaicml/pytorch:2.1.2_cpu-python3.10-ubuntu22.04
TARGET: pytorch_stage
TORCHVISION_VERSION: 0.16.2
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu20.04
BASE_IMAGE: nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
COMPOSER_INSTALL_COMMAND: mosaicml[all]==0.23.5
CUDA_VERSION: 12.1.1
IMAGE_NAME: composer-0-23-5
Expand Down Expand Up @@ -236,7 +236,7 @@
TARGET: composer_stage
TORCHVISION_VERSION: 0.18.1
- AWS_OFI_NCCL_VERSION: ''
BASE_IMAGE: ubuntu:20.04
BASE_IMAGE: ubuntu:22.04
COMPOSER_INSTALL_COMMAND: mosaicml[all]==0.23.5
CUDA_VERSION: ''
IMAGE_NAME: composer-0-23-5-cpu
Expand Down
8 changes: 4 additions & 4 deletions docker/generate_build_matrix.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ def _get_torchvision_version(pytorch_version: str):

def _get_base_image(cuda_version: str):
if not cuda_version:
return 'ubuntu:20.04'
return f'nvidia/cuda:{cuda_version}-cudnn8-devel-ubuntu20.04'
return 'ubuntu:22.04'
return f'nvidia/cuda:{cuda_version}-cudnn8-devel-ubuntu22.04'


def _get_cuda_version(pytorch_version: str, use_cuda: bool):
Expand Down Expand Up @@ -106,7 +106,7 @@ def _get_pytorch_tags(python_version: str, pytorch_version: str, cuda_version: s
else:
raise ValueError(f'Invalid stage: {stage}')
cuda_version_tag = _get_cuda_version_tag(cuda_version)
tags = [f'{base_image_name}:{pytorch_version}_{cuda_version_tag}-python{python_version}-ubuntu20.04']
tags = [f'{base_image_name}:{pytorch_version}_{cuda_version_tag}-python{python_version}-ubuntu22.04']

if python_version == PRODUCTION_PYTHON_VERSION and pytorch_version == PRODUCTION_PYTORCH_VERSION:
if not cuda_version:
Expand Down Expand Up @@ -281,7 +281,7 @@ def _main():
interconnect = 'EFA'
cuda_version = f"{entry['CUDA_VERSION']} ({interconnect})" if entry['CUDA_VERSION'] else 'cpu'
table.append([
'Ubuntu 20.04', # Linux distro
'Ubuntu 22.04', # Linux distro
'Base', # Flavor
entry['PYTORCH_VERSION'], # Pytorch version
cuda_version, # Cuda version
Expand Down
Loading