Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[UX/Jobs] No spinner after jobs launch, which feels like a hang #4627

Open
Michaelvll opened this issue Jan 30, 2025 · 0 comments
Open

[UX/Jobs] No spinner after jobs launch, which feels like a hang #4627

Michaelvll opened this issue Jan 30, 2025 · 0 comments

Comments

@Michaelvll
Copy link
Collaborator

When launching a managed job where the controller is on k8s, there is no spinner showing up after the job is submitted, which feels like the system is hanging. However, sky jobs logs --controller does show that the job cluster is being launched.

$ sky jobs launch test.yaml --cloud aws --cpus 2 -n test-mount-bucket
Task from YAML spec: test.yaml
Managed job 'test-mount-bucket' will be launched on (estimated):
Considered resources (1 node):
----------------------------------------------------------------------------------------
 CLOUD   INSTANCE    vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
----------------------------------------------------------------------------------------
 AWS     m6i.large   2       8         -              us-east-1     0.10          ✔     
----------------------------------------------------------------------------------------
Launching a managed job 'test-mount-bucket'. Proceed? [Y/n]: 
⚙︎ Translating workdir and file_mounts with local source paths to SkyPilot Storage...
  Workdir: 'examples' -> storage: 'skypilot-filemounts-vscode-47fbf511'.
  Folder : 'examples' -> storage: 'skypilot-filemounts-vscode-47fbf511'.
  Created S3 bucket 'skypilot-filemounts-vscode-47fbf511' in us-east-1
  Excluded files to sync to cluster based on .gitignore.
✓ Storage synced: examples -> s3://skypilot-filemounts-vscode-47fbf511/  View logs at: ~/sky_logs/sky-2025-01-30-23-32-05-747700/storage_sync.log
  Excluded files to sync to cluster based on .gitignore.
✓ Storage synced: examples -> s3://skypilot-filemounts-vscode-47fbf511/  View logs at: ~/sky_logs/sky-2025-01-30-23-32-12-937934/storage_sync.log
✓ Uploaded local files/folders.
Launching managed job 'test-mount-bucket' from jobs controller...
Warning: Credentials used for [GCP, AWS] may expire. Clusters may be leaked if the credentials expire while jobs are running. It is recommended to use credentials that never expire or a service account.
⚙︎ Mounting files.
  Syncing (to 1 node): /tmp/managed-dag-test-mount-bucket-ksc7nx8g -> ~/.sky/managed_jobs/test-mount-bucket-c2d4.yaml
  Syncing (to 1 node): /tmp/tmp3j5i1vru -> ~/.sky/managed_jobs/test-mount-bucket-c2d4.config_yaml
✓ Files synced.  View logs at: ~/sky_logs/sky-2025-01-30-23-32-19-231465/file_mounts.log
Auto-stop is not supported for Kubernetes and RunPod clusters. Skipping.
⚙︎ Job submitted, ID: 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant