Add Nebius Cloud #4573

SalikovAlex · 2025-01-16T16:58:42Z

Tested (run the relevant ones):

Add Nebius Cloud

cblmemo

Thanks @SalikovAlex for this amazing work! I think the PR is in a good shape. Left some nits. In the same time, could you help test the basic functionality of this new cloud? include but not limited to:

Launch CPU only instance
Launch GPU instance
Stop & Re-launch, check if the disk is persistent (write some content before stop, and cat them after re-launch)
Autostop & Autodown
Launch on existing cluster
SSH to the cluster
Failover: make sure it can failover from lambda to other clouds and the exceptions are printed correctly
launch on other clouds without nebius dependencies installed (make sure it does not introduce unnecessary dependencies when vast is not enabled)

examples/minimal.yaml

sky/authentication.py

sky/clouds/nebius.py

sky/provision/nebius/utils.py

sky/setup_files/MANIFEST.in

sky/setup_files/dependencies.py

sky/templates/nebius-ray.yml.j2

cblmemo · 2025-01-22T01:00:31Z

Also reference to: https://docs.google.com/document/d/1oWox3qb3Kz3wXXSGg9ZJWwijoa99a3PIQUHBR8UgEGs/edit?tab=t.0#heading=h.5lsecy1rnvwf

Split instance launch configurations to use separate platform and preset values, enabling better resource distinction. Added support for ImageId configuration in templates and clarified GPU cluster fabric requirements in related functions. Enhanced unsupported feature descriptions for Nebius cloud.

Mark Nebius as unsupported for various test cases due to its lack of support for specific GPUs, features, or configurations. This ensures accurate test coverage and avoids unnecessary failures on incompatible platforms.

Replaced the specific Git-based Nebius SDK reference with a version constraint (`nebius>=0.2.0`). This simplifies dependency management and ensures compatibility with future updates of the SDK.

Implemented retry mechanisms for starting and stopping instances in Nebius. Updated file paths for IAM token and tenant ID to include the home directory for better user guidance. Adjusted dependency and testing configurations to disable autostop functionality.

Revised instance naming logic by appending UUIDs to ensure uniqueness and clarity. Adjusted maximum cluster name length limit to accommodate the updated naming convention. Updated smoke tests and added or revised `no_nebius` marks to reflect current Nebius capabilities.

Added a no-op `open_ports` method for Nebius instances, as all ports are open by default. Updated features and tests to reflect the support for open ports while maintaining Nebius's behavior. Removed restrictions on open ports in the supported features list.

SalikovAlex · 2025-01-26T14:06:51Z

Unresolved problems:

Nebius require grpcio>=1.56.2 so ray>=2.6.1. So, we need to fix dependencies.py>remote
Nebius require python>=3.10

Feature request:
Split autoterminate to autostop and autodown

Enclosed the Python version in quotes to ensure proper YAML parsing. This change prevents potential issues with version interpretation in the CI pipeline.

Added error handling for missing IAM token and tenant ID files in `get_iam_token` and `get_tenant_id` functions, returning `None` if the files are not found. Updated `check_credentials` to provide clearer error messages with setup instructions when credentials are incomplete.

Reorganized initialization steps to include configurable setup commands and handled potential apt lock issues. Disabled unnecessary unattended-upgrades to prevent conflicts and ensured smoother package management during setup.

Simplify instance filtering logic for Nebius by removing redundant name checks. Add consistent `@pytest.mark.no_nebius` annotations in tests to reflect unsupported Autodown and Autostop features. Clean up unused Nebius-specific test case.

The `no_nebius` marker was added to skip the test where Autodown and Autostop are not supported. This ensures the test suite runs correctly on environments where these features are unavailable.

SalikovAlex · 2025-01-28T20:31:11Z

All tests are marked, now pytest tests/test_smoke.py --nebius is green.

romilbhardwaj · 2025-01-29T22:33:31Z

Hey @SalikovAlex thanks for the amazing work!

Nebius require grpcio>=1.56.2 so ray>=2.6.1. So, we need to fix dependencies.py>remote

Perhaps we can pin grpcio>=1.56.2 as remote dependency requirement. From my very primitive manual testing, seems like it works without requiring us to bump the remote ray version. cc @Michaelvll @cblmemo wdyt?

Nebius require python>=3.10

This should be ok, since pip on py3.9 environments throws a clear error. Users wanting to use nebius can upgrade to py3.10, which is also supported in SkyPilot.

ERROR: Ignored the following versions that require a different python version: 0.2.0 Requires-Python >=3.10; 0.2.1 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement nebius>=0.2.0; extra == "nebius" (from skypilot[nebius]) (from versions: none)
ERROR: No matching distribution found for nebius>=0.2.0; extra == "nebius"

romilbhardwaj · 2025-01-29T22:35:35Z

BTW, we may need to fix the nebius adaptor to lazy import nebius only when required. E.g., on this branch if I install pip install -e .[aws] without nebius python package in my environment, I get this error:

>>> import sky
Traceback (most recent call last):
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/adaptors/common.py", line 37, in load_module
    self._module = importlib.import_module(self._module_name)
  File "/Users/romilb/tools/anaconda3/envs/py39/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'nebius'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/__init__.py", line 82, in <module>
    from sky import backends
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/__init__.py", line 4, in <module>
    from sky.backends.cloud_vm_ray_backend import CloudVmRayBackend
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/backends/cloud_vm_ray_backend.py", line 30, in <module>
    from sky import check as sky_check
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/check.py", line 11, in <module>
    from sky import clouds as sky_clouds
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/clouds/__init__.py", line 15, in <module>
    from sky.clouds.aws import AWS
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/clouds/aws.py", line 16, in <module>
    from sky import provision as provision_lib
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/provision/__init__.py", line 23, in <module>
    from sky.provision import nebius
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/provision/nebius/__init__.py", line 4, in <module>
    from sky.provision.nebius.instance import cleanup_ports
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/provision/nebius/instance.py", line 9, in <module>
    from sky.provision.nebius import utils
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/provision/nebius/utils.py", line 12, in <module>
    sdk = nebius.sdk(credentials=nebius.get_iam_token())
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/adaptors/nebius.py", line 73, in sdk
    return nebius.sdk.SDK(credentials=credentials)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/adaptors/common.py", line 52, in __getattr__
    return getattr(self.load_module(), name)
  File "/Users/romilb/Romil/Berkeley/Research/sky-experiments/sky/adaptors/common.py", line 42, in load_module
    raise ImportError(self._import_error_message) from e
ImportError: Failed to import dependencies for Nebius AI Cloud. Try running: pip install "skypilot[nebius]"

Centralize Nebius SDK initialization by removing global SDK variables and replacing them with direct calls to `nebius.sdk()`. Introduce caching for IAM tokens to reduce redundant file reads. Also, update GRPC logging settings to suppress unnecessary console output.

SalikovAlex · 2025-01-30T12:31:25Z

BTW, we may need to fix the nebius adaptor to lazy import nebius only when required. E.g., on this branch if I install pip install -e .[aws]

Fixed

Simplified grpcio dependency to version >=1.56.2 and removed redundant version constraints and outdated comments. Also removed unnecessary ray dependency under 'nebius', ensuring a cleaner dependency structure.

SalikovAlex · 2025-01-30T20:27:04Z

This should be ok, since pip on py3.9 environments throws a clear error. Users wanting to use nebius can upgrade to py3.10, which is also supported in SkyPilot.

Yes, it's ok. But we need to change the version in GitHub actions for pep, lint and etc

This fixes a syntax issue in the `remote` list by adding a missing comma after `grpcio>=1.56.2`. This ensures the dependencies are parsed correctly.

cblmemo · 2025-01-30T20:52:11Z

Hey @SalikovAlex thanks for the amazing work!

Nebius require grpcio>=1.56.2 so ray>=2.6.1. So, we need to fix dependencies.py>remote

Perhaps we can pin grpcio>=1.56.2 as remote dependency requirement. From my very primitive manual testing, seems like it works without requiring us to bump the remote ray version. cc @Michaelvll @cblmemo wdyt?

Nebius require python>=3.10

This should be ok, since pip on py3.9 environments throws a clear error. Users wanting to use nebius can upgrade to py3.10, which is also supported in SkyPilot.
ERROR: Ignored the following versions that require a different python version: 0.2.0 Requires-Python >=3.10; 0.2.1 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement nebius>=0.2.0; extra == "nebius" (from skypilot[nebius]) (from versions: none)
ERROR: No matching distribution found for nebius>=0.2.0; extra == "nebius"

This looks good to me!

cblmemo · 2025-01-30T20:54:01Z

This should be ok, since pip on py3.9 environments throws a clear error. Users wanting to use nebius can upgrade to py3.10, which is also supported in SkyPilot.

Yes, it's ok. But we need to change the version in GitHub actions for pep, lint and etc

I think in the GH actions it will not install real cloud dependencies. Every test is running without actually provision VMs on the cloud, so IIUC no nebius dependency needs to be installed?

Those test that will provision VM and run workloads is located in the smoke_test folder. In GH actions we only run the unittests.

Adjust GitHub Actions workflows to use Python 3.8 instead of 3.10 for consistency. Simplify formatting in `dependencies.py` by condensing the 'nebius' dependency list.

SalikovAlex · 2025-01-30T21:08:12Z

I think in the GH actions it will not install real cloud dependencies. Every test is running without actually provision VMs on the cloud, so IIUC no nebius dependency needs to be installed?

Those test that will provision VM and run workloads is located in the smoke_test folder. In GH actions we only run the unittests.

pylint, pytest, doc build run uv pip install ".[all]"

cblmemo · 2025-01-30T21:23:31Z

I think in the GH actions it will not install real cloud dependencies. Every test is running without actually provision VMs on the cloud, so IIUC no nebius dependency needs to be installed?
Those test that will provision VM and run workloads is located in the smoke_test folder. In GH actions we only run the unittests.

pylint, pytest, doc build run uv pip install ".[all]"

Humm, good point! cc @romilbhardwaj @Michaelvll for a look here

cblmemo

Thanks for the prompt fix @SalikovAlex! Mostly looks good to me. Left some discussions ;)

sky/adaptors/nebius.py

cblmemo · 2025-01-30T21:30:59Z

sky/clouds/nebius.py

+    # Nebius maximum instance name length defined as <= 63 as a hostname length
+    # 63 - 8 - 5 = 50 characters since
+    # we add 4 character from UUID to make uniq `-xxxx`
+    # our provisioner adds additional `-worker`.
+    _MAX_CLUSTER_NAME_LEN_LIMIT = 50


the character uuid should be included in this limit. check aws & gcp implementation for details

It's another UUID
#4573 (comment)

sky/clouds/nebius.py

sky/provision/nebius/instance.py

.github/workflows/pylint.yml

sky/setup_files/dependencies.py

romilbhardwaj · 2025-01-30T23:21:50Z

sky/setup_files/dependencies.py

-    # Original issue: https://github.com/ray-project/ray/issues/33833
-    'grpcio >= 1.32.0, <= 1.51.3, != 1.48.0; python_version < \'3.10\' and sys_platform != \'darwin\'',  # noqa:E501 pylint: disable=line-too-long
-    'grpcio >= 1.42.0, <= 1.51.3, != 1.48.0; python_version >= \'3.10\' and sys_platform != \'darwin\'',  # noqa:E501 pylint: disable=line-too-long
+    'grpcio>=1.56.2',


@cblmemo Could you help run smoke tests for this change for on other clouds as well? Also would be great if you can verify

Replaced duplicated logic with assertions, simplified filename management, and unified token/tenant ID handling. Fixed typos (e.g., "creat" to "create"), improved logging, and added Python version checks for dependencies. These changes enhance readability, maintainability, and robustness.

Simplified assertions and cleaned up parameter handling in provisioning logic. Improved variable naming and formatting in dependencies setup for clarity. Removed extra whitespace and adjusted line splitting for better readability.

Updated the loop to properly iterate over dictionary items when processing stopped instances. This ensures correct functionality when starting instances and aligns with standard dictionary iteration practices.

Updated the Python version in `pytest`, `pylint`, and `test-doc-build` workflows from 3.10 to 3.8. This ensures consistency across workflows and accommodates compatibility with Python 3.8.

Switched the Python version from 3.8 to 3.10 in the GitHub Actions workflow for building documentation. This ensures compatibility with newer Python features and aligns with the project's current supported versions.

Add checks to ensure cluster states align with instance statuses during the wait operation, raising errors on mismatches. Enhance comments to explain how the region is inferred from project IDs due to the lack of direct region information retrieval in Nebius.

SalikovAlex · 2025-01-31T16:33:57Z

Also, why is there project metadata id limitation? Is this project a per-user thing? IIUC if it is the case, different user would have different id?

Added this to the comments.

    # To find a project in a specific region, we rely on the project ID to
    # deduce the region, since there is currently no method to retrieve region
    # information directly from the project. Additionally, there is only one
    # project per region, and projects cannot be created at this time.
    # The region is determined from the project ID using a region-specific
    # identifier embedded in it.
    # https://docs.nebius.com/overview/regions

cblmemo · 2025-01-31T18:01:32Z

https://docs.nebius.com/overview/regions

Just to confirm, so if a new user is using nebius for skypilot, does that user need to replace the project id here?

SalikovAlex · 2025-01-31T20:59:58Z

Just to confirm, so if a new user is using nebius for skypilot, does that user need to replace the project id here?

No. I'm requesting them by API. The user's project id looks like "project-e00xxxxxxxxxxxxxx", where "e00" is - the code of the region.

Where do you find hardcoded project id?

cblmemo · 2025-02-01T00:31:59Z

Just to confirm, so if a new user is using nebius for skypilot, does that user need to replace the project id here?

No. I'm requesting them by API. The user's project id looks like "project-e00xxxxxxxxxxxxxx", where "e00" is - the code of the region.

Where do you find hardcoded project id?

Oh ic! that makes sense. I thought the e00 is some unique value related to your project id.

If that is the case, can we add a comment for what this 8-11 for (iiuc it is the length of project-?) and what is the e00 & e01 for? to reduce confusion :)) Thanks!

Clarified how project IDs correlate to regions and added a note to address improvements once region data is included in the project list. Also provided context on instance deletion flow, explaining the need to wait for disk unlock before retries.

Raise a RuntimeError if the total number of running and stopped instances exceeds the user's requested count. This helps prevent resource leaks and provides clear guidance to use "sky down" to clean up the cluster.

SalikovAlex · 2025-02-01T13:55:29Z

If that is the case, can we add a comment for what this 8-11 for (iiuc it is the length of project-?) and what is the e00 & e01 for? to reduce confusion :)) Thanks!

added

SalikovAlex added 2 commits January 16, 2025 16:57

Add Nebius Cloud

9520d28

Add Nebius Cloud

formatting

db73f1e

SalikovAlex mentioned this pull request Jan 17, 2025

Add Nebius Cloud skypilot-org/skypilot-catalog#109

Open

Michaelvll requested a review from cblmemo January 21, 2025 03:31

bump pysdk version

1cc9a5e

cblmemo reviewed Jan 22, 2025

View reviewed changes

SalikovAlex added 14 commits January 22, 2025 14:39

some review fixes

7cb7be0

some review fixes 2

52b5700

some review fixes

64589e5

formating and fixes

feb7942

rm whiles and replace project to region

09e1417

Add Nebius exclusions to smoke tests

9b7ecf3

Mark Nebius as unsupported for various test cases due to its lack of support for specific GPUs, features, or configurations. This ensures accurate test coverage and avoids unnecessary failures on incompatible platforms.

Update Nebius dependency to use version constraint

1a5fb65

Replaced the specific Git-based Nebius SDK reference with a version constraint (`nebius>=0.2.0`). This simplifies dependency management and ensures compatibility with future updates of the SDK.

revert dep file

f9283e0

SalikovAlex added 8 commits January 26, 2025 14:10

Try to run test on python3.10

b65e87e

Fix Python version formatting in pytest workflow

5102540

Enclosed the Python version in quotes to ensure proper YAML parsing. This change prevents potential issues with version interpretation in the CI pipeline.

Just for test CI

a147ed1

Formating

21330cc

Refactor setup commands in Nebius Ray template.

7c1c7f4

Reorganized initialization steps to include configurable setup commands and handled potential apt lock issues. Disabled unnecessary unattended-upgrades to prevent conflicts and ensured smoother package management during setup.

Refactor setup commands in Nebius Ray template.

fc9acef

Reorganized initialization steps to include configurable setup commands and handled potential apt lock issues. Disabled unnecessary unattended-upgrades to prevent conflicts and ensured smoother package management during setup.

SalikovAlex added 2 commits January 28, 2025 20:27

Add no_nebius marker for unsupported skyserve test

5c00d9f

The `no_nebius` marker was added to skip the test where Autodown and Autostop are not supported. This ensures the test suite runs correctly on environments where these features are unavailable.

Merge remote-tracking branch 'origin/nebiuscloud' into nebiuscloud

a8c5749

Update grpcio dependency and cleanup outdated comments

4083a5a

Simplified grpcio dependency to version >=1.56.2 and removed redundant version constraints and outdated comments. Also removed unnecessary ray dependency under 'nebius', ensuring a cleaner dependency structure.

Add missing comma in dependencies list

42ddf66

This fixes a syntax issue in the `remote` list by adding a missing comma after `grpcio>=1.56.2`. This ensures the dependencies are parsed correctly.

Update Python version in workflows and clean dependencies

aa0e0ba

Adjust GitHub Actions workflows to use Python 3.8 instead of 3.10 for consistency. Simplify formatting in `dependencies.py` by condensing the 'nebius' dependency list.

cblmemo reviewed Jan 30, 2025

View reviewed changes

romilbhardwaj reviewed Jan 30, 2025

View reviewed changes

SalikovAlex added 6 commits January 31, 2025 14:15

Fix incorrect iteration over stopped instances dictionary

2698a76

Updated the loop to properly iterate over dictionary items when processing stopped instances. This ensures correct functionality when starting instances and aligns with standard dictionary iteration practices.

Set Python version to 3.8 in GitHub workflows

781f485

Updated the Python version in `pytest`, `pylint`, and `test-doc-build` workflows from 3.10 to 3.8. This ensures consistency across workflows and accommodates compatibility with Python 3.8.

Update Python version to 3.10 in doc build workflow

0c25e4b

Switched the Python version from 3.8 to 3.10 in the GitHub Actions workflow for building documentation. This ensures compatibility with newer Python features and aligns with the project's current supported versions.

SalikovAlex added 2 commits February 1, 2025 13:45

Add error handling for excess stopped instances in cluster

0d0c082

Raise a RuntimeError if the total number of running and stopped instances exceeds the user's requested count. This helps prevent resource leaks and provides clear guidance to use "sky down" to clean up the cluster.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Nebius Cloud #4573

Add Nebius Cloud #4573

SalikovAlex commented Jan 16, 2025 •

edited

Loading

cblmemo left a comment

cblmemo commented Jan 22, 2025

SalikovAlex commented Jan 26, 2025 •

edited

Loading

SalikovAlex commented Jan 28, 2025

romilbhardwaj commented Jan 29, 2025

romilbhardwaj commented Jan 29, 2025

SalikovAlex commented Jan 30, 2025 •

edited

Loading

SalikovAlex commented Jan 30, 2025

cblmemo commented Jan 30, 2025

cblmemo commented Jan 30, 2025

SalikovAlex commented Jan 30, 2025 •

edited

Loading

cblmemo commented Jan 30, 2025

cblmemo left a comment

cblmemo Jan 30, 2025

SalikovAlex Jan 31, 2025

romilbhardwaj Jan 30, 2025

cblmemo Jan 31, 2025

SalikovAlex commented Jan 31, 2025 •

edited

Loading

cblmemo commented Jan 31, 2025

SalikovAlex commented Jan 31, 2025

cblmemo commented Feb 1, 2025

SalikovAlex commented Feb 1, 2025

Add Nebius Cloud #4573

Are you sure you want to change the base?

Add Nebius Cloud #4573

Conversation

SalikovAlex commented Jan 16, 2025 • edited Loading

cblmemo left a comment

Choose a reason for hiding this comment

cblmemo commented Jan 22, 2025

SalikovAlex commented Jan 26, 2025 • edited Loading

SalikovAlex commented Jan 28, 2025

romilbhardwaj commented Jan 29, 2025

romilbhardwaj commented Jan 29, 2025

SalikovAlex commented Jan 30, 2025 • edited Loading

SalikovAlex commented Jan 30, 2025

cblmemo commented Jan 30, 2025

cblmemo commented Jan 30, 2025

SalikovAlex commented Jan 30, 2025 • edited Loading

cblmemo commented Jan 30, 2025

cblmemo left a comment

Choose a reason for hiding this comment

cblmemo Jan 30, 2025

Choose a reason for hiding this comment

SalikovAlex Jan 31, 2025

Choose a reason for hiding this comment

romilbhardwaj Jan 30, 2025

Choose a reason for hiding this comment

cblmemo Jan 31, 2025

Choose a reason for hiding this comment

SalikovAlex commented Jan 31, 2025 • edited Loading

cblmemo commented Jan 31, 2025

SalikovAlex commented Jan 31, 2025

cblmemo commented Feb 1, 2025

SalikovAlex commented Feb 1, 2025

SalikovAlex commented Jan 16, 2025 •

edited

Loading

SalikovAlex commented Jan 26, 2025 •

edited

Loading

SalikovAlex commented Jan 30, 2025 •

edited

Loading

SalikovAlex commented Jan 30, 2025 •

edited

Loading

SalikovAlex commented Jan 31, 2025 •

edited

Loading