Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeouts from startup script #18

Open
NorseGaud opened this issue Jul 21, 2023 · 2 comments
Open

Timeouts from startup script #18

NorseGaud opened this issue Jul 21, 2023 · 2 comments

Comments

@NorseGaud
Copy link
Member

NorseGaud commented Jul 21, 2023

I0721 14:45:41.673001   51096 intel_v1.go:107] script timed out!
I0721 14:45:41.674897   51096 intel_v1.go:141] Script exit code: -1
I0721 14:45:41.674920   51096 intel_v1.go:142] Script stdout:
I0721 14:45:41.674927   51096 intel_v1.go:143] Script stderr: + cd /Users/anka/actions-runner
+ ./config.sh --url https://github.com/veertuinc/anka-actions-tests --token XXXXXXX --labels c55a1b77-194f-4882-a3d1-7da20ead6d1e --runnergroup Default --name c55a1b77-194f-4882-a3d1-7da20ead6d1e --work _work
I0721 14:45:41.674935   51096 startup_script.go:42] startup script failed. keeping vm cbdb0aa8-d3a0-4746-91c8-923767b8e021 alive
##[debug][7/21/2023, 2:45:46 PM] ListVMResponse status: 200; body: {"status":"OK","message":"","body":{"instance_id":"a0e5a9b5-779f-4b87-581f-9be6b007ae3e","instance_state":"Error","message":"startup script timed out","anka_registry":"http://17XXX:8089/","vmid":"16a6d2ce-1d64-4212-815f-5a86ef8e3522","tag":"v3","vminfo":{"uuid":"cbdb0aa8-d3a0-4746-91c8-923767b8e021","name":"13.4.1-22F82-3.2.0.153-gen-use-arm64-1689950649996162000","cpu_cores":6,"ram":"14G","status":"running","node_id":"88243419-c108-4458-a62e-542f4294677c","host_ip":"17XXXX","ip":"19XXX","tag":"v3","vnc_port":10002,"vnc_connection_string":"vnc://17XXXXX:10002","port_forwarding":[{"guest_port":22,"host_port":10003,"protocol":"tcp","rule_name":""},{"guest_port":5900,"host_port":10002,"protocol":"tcp","rule_name":""}],"creation_date":"2023-07-20T12:08:51Z","stop_date":"0001-01-01T00:00:00Z","version":"","timestamp":"2023-07-21T14:45:42.639110365Z"},"node_id":"88243419-c108-4458-a62e-542f4294677c","inflight_reqid":"e2e4ab96-e1ad-43f9-6099-61b36f4404bc","ts":"2023-07-21T14:45:42.639110365Z","cr_time":"2023-07-21T14:37:02.9181003Z","progress":1,"external_id":"c55a1b77-194f-4882-a3d1-7da20ead6d1e","arch":"arm64","vlan":"","startup_script":{"return_code":-1,"did_timeout":true,"stdout":"","stderr":"+ cd /Users/anka/actions-runner\n+ ./config.sh --url https://github.com/veertuinc/anka-actions-tests --token XXXX --labels c55a1b77-194f-4882-a3d1-7da20ead6d1e --runnergroup Default --name c55a1b77-194f-4882-a3d1-7da20ead6d1e --work _work\n"}}}
@NorseGaud
Copy link
Member Author

Looks related to heavy CPU usage on the host.

@NorseGaud
Copy link
Member Author

Unable to reproduce when lowering the CPU for the VMs so that the host doesn't get so overwhelmed.
Also, the timeout is from script_timeout being set to 90s, which increasing to say 180s would eliminate this even when CPU is high.

https://veertu.atlassian.net/browse/CLOUD-1291

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant