[Azure] Revert Azure images to address NCCL issues

[romilbhardwaj]

Intermediate fix for #4448 by reverting to Azure's default images. We should fix our custom image to support NCCL + Azure accelerated networking before starting to use them again.

Tested:

• sky launch -c azure --cloud azure --gpus A100-80GB:1 -- nvidia-smi

[Michaelvll]

Can we use the old Azure image as the base image for the packer file, so we don't get performance regression for A10 GPU instances?

[romilbhardwaj]

Yes, that's likely the fix for #4448. For A10 this PR is still using skypilot:custom-gpu-ubuntu-v2-grid to avoid the perf regression.

[romilbhardwaj]

/smoke-test azure

[Michaelvll]

I am ok with this temp solution, but we should figure out a good way to support it with our images, otherwise, this is a huge performance degradation on Azure.

[romilbhardwaj]

I think a better fix for this is to update the catalog instead of our code. That way we can push an update the catalog when this issue is resolved without needing users to upgrade to the latest version/nightly.

[Michaelvll]

I think a better fix for this is to update the catalog instead of our code. That way we can push an update the catalog when this issue is resolved without needing users to upgrade to the latest version/nightly.

This may worth discussion. Directly updating the catalog can cause an implicit behavior change for the VM launched on Azure. This can lead to surprising behavior, such as the cluster launched today has different setup than yesterday, e.g., some specific packages disappear, etc.

I feel a more explicit tag change in this PR is a better way to do this.

[romilbhardwaj]

/smoke-test --azure

[zpoint]

You need to merge the latest master to make the --azure work. Now it triggers all smoke tests as before. Some smoke test failures have been fixed on master.
cc @romilbhardwaj

[romilbhardwaj]

/smoke-test --azure

[romilbhardwaj]

This PR seems to be having issues with tensorflow + CUDA, because of which test_cancel_azure --azure is failing. It seems to require the [and-cuda] extras for tensorflow for it to detect GPUs:

python3 -m pip install 'tensorflow[and-cuda]'
# Verify the installation:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Do not merge until this is resolved.