Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Nebius Cloud #4573

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
9520d28
Add Nebius Cloud
SalikovAlex Jan 16, 2025
db73f1e
formatting
SalikovAlex Jan 17, 2025
1cc9a5e
bump pysdk version
SalikovAlex Jan 21, 2025
7cb7be0
some review fixes
SalikovAlex Jan 22, 2025
52b5700
some review fixes 2
SalikovAlex Jan 22, 2025
64589e5
some review fixes
SalikovAlex Jan 22, 2025
feb7942
formating and fixes
SalikovAlex Jan 22, 2025
09e1417
rm whiles and replace project to region
SalikovAlex Jan 22, 2025
c75f9d7
Refactor Nebius instance launching and resource handling
SalikovAlex Jan 22, 2025
9b7ecf3
Add Nebius exclusions to smoke tests
SalikovAlex Jan 22, 2025
1a5fb65
Update Nebius dependency to use version constraint
SalikovAlex Jan 24, 2025
55e83cf
Add autostop support for Nebius cloud instances.
SalikovAlex Jan 24, 2025
6141aef
Add autostop support for Nebius cloud instances.
SalikovAlex Jan 24, 2025
f9283e0
revert dep file
SalikovAlex Jan 24, 2025
7d54f55
Refactor Nebius instance naming and update related tests.
SalikovAlex Jan 25, 2025
96127d6
Refactor Nebius instance naming and update related tests.
SalikovAlex Jan 25, 2025
22eadcd
Support open ports for Nebius cloud provider
SalikovAlex Jan 26, 2025
b65e87e
Try to run test on python3.10
SalikovAlex Jan 26, 2025
5102540
Fix Python version formatting in pytest workflow
SalikovAlex Jan 26, 2025
a147ed1
Just for test CI
SalikovAlex Jan 26, 2025
a7f34bb
Handle missing Nebius credential files gracefully
SalikovAlex Jan 26, 2025
21330cc
Formating
SalikovAlex Jan 26, 2025
7c1c7f4
Refactor setup commands in Nebius Ray template.
SalikovAlex Jan 26, 2025
fc9acef
Refactor setup commands in Nebius Ray template.
SalikovAlex Jan 26, 2025
c1c90e9
Refactor Nebius instance filtering and update test exclusions.
SalikovAlex Jan 27, 2025
308ac64
Update test exclusions.
SalikovAlex Jan 27, 2025
f6a8618
Update test exclusions.
SalikovAlex Jan 27, 2025
bf2227b
Merge branch 'master' into nebiuscloud
SalikovAlex Jan 27, 2025
75d7d4c
Merge branch 'skypilot-org:master' into nebiuscloud
SalikovAlex Jan 27, 2025
5c00d9f
Add no_nebius marker for unsupported skyserve test
SalikovAlex Jan 28, 2025
a8c5749
Merge remote-tracking branch 'origin/nebiuscloud' into nebiuscloud
SalikovAlex Jan 28, 2025
17514eb
Refactor Nebius SDK usage and optimize IAM token handling
SalikovAlex Jan 30, 2025
4083a5a
Update grpcio dependency and cleanup outdated comments
SalikovAlex Jan 30, 2025
42ddf66
Add missing comma in dependencies list
SalikovAlex Jan 30, 2025
aa0e0ba
Update Python version in workflows and clean dependencies
SalikovAlex Jan 30, 2025
16a38cb
Refactor Nebius provisioning and adaptors for clarity and consistency
SalikovAlex Jan 31, 2025
dfa754c
Refactor Nebius-related code and fix minor inconsistencies
SalikovAlex Jan 31, 2025
2698a76
Fix incorrect iteration over stopped instances dictionary
SalikovAlex Jan 31, 2025
781f485
Set Python version to 3.8 in GitHub workflows
SalikovAlex Jan 31, 2025
0c25e4b
Update Python version to 3.10 in doc build workflow
SalikovAlex Jan 31, 2025
43c6500
Validate instance states during wait and clarify region logic.
SalikovAlex Jan 31, 2025
f1f4f7e
Add comments for region mapping and instance deletion process
SalikovAlex Feb 1, 2025
0d0c082
Add error handling for excess stopped instances in cluster
SalikovAlex Feb 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/minimal.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
name: minimal

resources:
cloud: aws
cloud: nebius
SalikovAlex marked this conversation as resolved.
Show resolved Hide resolved

setup: |
echo "running setup"
Expand Down
2 changes: 2 additions & 0 deletions sky/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ def set_proxy_env_var(proxy_var: str, urllib_var: Optional[str]):
RunPod = clouds.RunPod
Vsphere = clouds.Vsphere
Fluidstack = clouds.Fluidstack
Nebius = clouds.Nebius
optimize = Optimizer.optimize

__all__ = [
Expand All @@ -153,6 +154,7 @@ def set_proxy_env_var(proxy_var: str, urllib_var: Optional[str]):
'SCP',
'Vsphere',
'Fluidstack',
'Nebius',
'Optimizer',
'OptimizeTarget',
'backends',
Expand Down
29 changes: 29 additions & 0 deletions sky/adaptors/nebius.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""Nebius cloud adaptor."""

import functools

_nebius_sdk = None


def import_package(func):

@functools.wraps(func)
def wrapper(*args, **kwargs):
global _nebius_sdk
if _nebius_sdk is None:
try:
import nebius as _nebius # pylint: disable=import-outside-toplevel
_nebius_sdk = _nebius
except ImportError:
raise ImportError(
'Fail to import dependencies for nebius.'
'Try pip install "skypilot[nebius]"') from None
return func(*args, **kwargs)

return wrapper


@import_package
def nebius():
"""Return the nebius package."""
return _nebius_sdk
12 changes: 12 additions & 0 deletions sky/authentication.py
Original file line number Diff line number Diff line change
Expand Up @@ -497,3 +497,15 @@ def setup_fluidstack_authentication(config: Dict[str, Any]) -> Dict[str, Any]:
client.get_or_add_ssh_key(public_key)
config['auth']['ssh_public_key'] = PUBLIC_SSH_KEY_PATH
return configure_ssh_info(config)


def setup_nebius_authentication(config: Dict[str, Any]) -> Dict[str, Any]:
get_or_generate_keys()

config['auth']['ssh_public_key'] = PUBLIC_SSH_KEY_PATH

file_mounts = config['file_mounts']
file_mounts[PUBLIC_SSH_KEY_PATH] = PUBLIC_SSH_KEY_PATH
config['file_mounts'] = file_mounts

return configure_ssh_info(config)
SalikovAlex marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions sky/backends/backend_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,8 @@ def _add_auth_to_cluster_config(cloud: clouds.Cloud, cluster_config_file: str):
config = auth.setup_runpod_authentication(config)
elif isinstance(cloud, clouds.Fluidstack):
config = auth.setup_fluidstack_authentication(config)
elif isinstance(cloud, clouds.Nebius):
config = auth.setup_nebius_authentication(config)
else:
assert False, cloud
common_utils.dump_yaml(cluster_config_file, config)
Expand Down
3 changes: 2 additions & 1 deletion sky/backends/cloud_vm_ray_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,8 @@ def _get_cluster_config_template(cloud):
clouds.RunPod: 'runpod-ray.yml.j2',
clouds.Kubernetes: 'kubernetes-ray.yml.j2',
clouds.Vsphere: 'vsphere-ray.yml.j2',
clouds.Fluidstack: 'fluidstack-ray.yml.j2'
clouds.Fluidstack: 'fluidstack-ray.yml.j2',
clouds.Nebius: 'nebius-ray.yml.j2'
}
return cloud_to_template[type(cloud)]

Expand Down
2 changes: 2 additions & 0 deletions sky/clouds/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
from sky.clouds.ibm import IBM
from sky.clouds.kubernetes import Kubernetes
from sky.clouds.lambda_cloud import Lambda
from sky.clouds.nebius import Nebius
from sky.clouds.oci import OCI
from sky.clouds.paperspace import Paperspace
from sky.clouds.runpod import RunPod
Expand Down Expand Up @@ -49,6 +50,7 @@
'ProvisionerVersion',
'StatusVersion',
'Fluidstack',
'Nebius',
# Utility functions
'cloud_in_iterable',
]
Loading
Loading