Akash Provider Fails to Start #69

dougbebber · 2022-03-12T14:53:07Z

Everything appears to be working except that the Provider is offline and will not successfully start.
I keep seeing the following in the /var/log/handyhost.log:

AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address

error calling http://rpc-1.handyhost.computer:26659/akash1ghhkmp9c0zwynvktefh2uu0saam2dzw04pf52m Error: connect ECONNREFUSED 72.8.228.138:26659
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '72.8.228.138',
port: 26659
}

Any known solution?

alexsmith540 · 2022-03-12T15:14:16Z

Are you on the latest v0.5.2 build?

…

Sent from my iPhone

On Mar 12, 2022, at 7:53 AM, Douglas Bebber ***@***.***> wrote: Everything appears to be working except that the Provider is offline and will not successfully start. I keep seeing the following in the /var/log/handyhost.log: AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address error calling http://rpc-1.handyhost.computer:26659/akash1ghhkmp9c0zwynvktefh2uu0saam2dzw04pf52m Error: connect ECONNREFUSED 72.8.228.138:26659 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '72.8.228.138', port: 26659 } Any known solution? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

dougbebber · 2022-03-12T18:53:10Z

Yes, I installed handyhost_v0.5.2.1.deb on Ubuntu 20.04 LTS.

Handy Host shows:
HandyHost v0.5.2
Akash v0.14.1

Are you on the latest v0.5.2 build?
…
Sent from my iPhone
On Mar 12, 2022, at 7:53 AM, Douglas Bebber @.***> wrote: Everything appears to be working except that the Provider is offline and will not successfully start. I keep seeing the following in the /var/log/handyhost.log: AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address AKT: get aggregates query error Error: post failed: Post "http://rpc-1.handyhost.computer/": dial tcp: address rpc-1.handyhost.computer: missing port in address error calling http://rpc-1.handyhost.computer:26659/akash1ghhkmp9c0zwynvktefh2uu0saam2dzw04pf52m Error: connect ECONNREFUSED 72.8.228.138:26659 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '72.8.228.138', port: 26659 } Any known solution? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

alexsmith540 · 2022-03-13T17:14:33Z

i think the errors you were seeing in the logs were likely due to an rpc outage on our aggregates node yesterday morning.
If you try to run the provider now do you still see any errors?

dougbebber · 2022-03-13T20:31:08Z

I no longer receive that error. However, each time I attempt to start the Provider, I initially get a dialog that says started Provider successfully. Then the provider status indicator in the upper right goes dark and the Provider entry on the left sidebar has a yellow exclamation point.

/var/log/handyhost.log now contains:
akash exists?? true
akash dir exists? true
is positional [ 'api', 'getIP' ]
is positional [ 'api', 'akt', 'getClusterConfig' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'runProvider' ]
should akash run? true
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getProviderLogs' ]
is positional [ 'api', 'akt', 'getProviderLogs' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getClusterConfig' ]
is positional [ 'api', 'akt', 'getProviderParams' ]
is positional [ 'api', 'akt', 'getState' ]
akash exists?? true
akash dir exists? true
is positional [ 'api', 'getIP' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'runProvider' ]
should akash run? true
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'providerRegistrationGasEstimate' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'providerRegistrationGasEstimate' ]
is positional [ 'api', 'akt', 'providerRegistrationGasEstimate' ]
is positional [ 'api', 'akt', 'providerRegistrationGasEstimate' ]
is positional [ 'api', 'akt', 'updateProviderRegistration' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'runProvider' ]
should akash run? true
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]
is positional [ 'api', 'akt', 'getMarketplaceOrders' ]
get orders params { page: 1, limit: 10 }
is positional [ 'api', 'akt', 'runProvider' ]
should akash run? true
is positional [ 'api', 'akt', 'getClusterStats' ]
is positional [ 'api', 'akt', 'getMarketAggregates' ]

I tried a few times (with the same result). I even tried to re-register the Provider certificate.

I have noticed that when I click the "Provider Logs" link and the dialog comes up, I click Refresh logs and it just hangs (nothing comes back).

Any suggestions on how I can get the Provider to stay up?

dougbebber · 2022-03-13T20:51:02Z

Here's my Cluster Nodes Resource Utilization:

alexsmith540 · 2022-03-14T19:45:27Z

Any chance you've tried a system reboot (on the provider system)? Sometimes, like the case you ran into where the rpc node was down, a provider can get stuck in an endless restarting loop which i suspect may be happening here.

dougbebber · 2022-03-15T13:18:35Z

Rebooted several times. Actually the machine locks up after running for several hours and I am forced to reboot.
Same problem with the akash provider. Initially start up provider, message says provider started successfully, the provider status in the right hand corner goes yellow for a few seconds, then the provider is indicated offline and the yellow exclamation is next to the "Start Provider" menu option.

The provider is running on the handy host machine. Not any of the 3 akash cluster nodes.

dougbebber · 2022-03-15T15:35:03Z

Is there an akash provider specific log I can inspect to try to discover the details of the problem?

freakinfofa · 2022-03-17T01:12:52Z

not sure if its the same thing it happened to me at first when i build my first cluster

Try this. start your provider but uncheck the "start at boot".It will probably fail if you had attempted to start the provider multiple times so reboot your host after a minute. (this is so that the provider does not try to start on its own and you have a better point of reference on when you actually started it)

when your host is backonline try to start your provider once again (this time check the start provider at boot. the dot will go yellow and then gray.... but just give it time (go make some coffee or something and be back in a few minutes) it will go online as long as you keep seeing the "Provider Logs" menu on the left DO NOT TRY TO START IT AGAIN.

if you dont see the provider logs menu on the left.. then the provider is not running

when i kept trying to start it right after it goes gray the first time. it would just never come online. it might be something totally unrelated, but my two cents!

dougbebber · 2022-03-17T18:31:08Z

not sure if its the same thing it happened to me at first when i build my first cluster

Try this. start your provider but uncheck the "start at boot".It will probably fail if you had attempted to start the provider multiple times so reboot your host after a minute. (this is so that the provider does not try to start on its own and you have a better point of reference on when you actually started it)

when your host is backonline try to start your provider once again (this time check the start provider at boot. the dot will go yellow and then gray.... but just give it time (go make some coffee or something and be back in a few minutes) it will go online as long as you keep seeing the "Provider Logs" menu on the left DO NOT TRY TO START IT AGAIN.

if you dont see the provider logs menu on the left.. then the provider is not running

when i kept trying to start it right after it goes gray the first time. it would just never come online. it might be something totally unrelated, but my two cents!

I tried your suggestion. However the provider never started. I eventually clicked the "Provider logs" link and no information was presented. I'd like to be able to view some Akash provider log information to attempt to debug what's happening on my machine.

freakinfofa · 2022-03-17T22:03:17Z

what does your networking look like?
--are you able to see if any traffic is being blocked or dropped? (any firewalls or fancy routers you might be using?)
--ports 80 and 8443 are forwarded/NAT'd to your provider correctly? (in my case i have a reverse proxy in front of port 80 as i
host other services under that public IP)

I know its probably the obvious... just trying to help :)

dougbebber · 2022-03-21T16:16:19Z

Port 80 is forwarded to the Ingress Controller machine on one of my cluster nodes. Port 8443 is forwarded to my Provider node (machine running Handy Host).

I have tried a number of things with no success getting the Provider running.

freakinfofa · 2022-03-29T14:43:06Z

You must open & forward ports 80 and 30000-32767 to your Ingress Controller

avolon42x · 2022-03-31T20:37:58Z

What you could try that helped on my side sometimes: do a systemctl stop handyhost followed by killall akash (do it twice till there is now akash proc running). systemctl start handyhost. Then be patient and let handyhost start the provider by itself -> if you start the provider through the ui there is the checkbox to automatically start it. If you enable it, handyhost will try and re-try to start the provider. It just took a few minutes on my setup.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Akash Provider Fails to Start #69

Akash Provider Fails to Start #69

dougbebber commented Mar 12, 2022

alexsmith540 commented Mar 12, 2022 via email

dougbebber commented Mar 12, 2022

alexsmith540 commented Mar 13, 2022

dougbebber commented Mar 13, 2022 •

edited

Loading

dougbebber commented Mar 13, 2022

alexsmith540 commented Mar 14, 2022

dougbebber commented Mar 15, 2022 •

edited

Loading

dougbebber commented Mar 15, 2022

freakinfofa commented Mar 17, 2022

dougbebber commented Mar 17, 2022

freakinfofa commented Mar 17, 2022

dougbebber commented Mar 21, 2022

freakinfofa commented Mar 29, 2022

avolon42x commented Mar 31, 2022

Akash Provider Fails to Start #69

Akash Provider Fails to Start #69

Comments

dougbebber commented Mar 12, 2022

alexsmith540 commented Mar 12, 2022 via email

dougbebber commented Mar 12, 2022

alexsmith540 commented Mar 13, 2022

dougbebber commented Mar 13, 2022 • edited Loading

dougbebber commented Mar 13, 2022

alexsmith540 commented Mar 14, 2022

dougbebber commented Mar 15, 2022 • edited Loading

dougbebber commented Mar 15, 2022

freakinfofa commented Mar 17, 2022

dougbebber commented Mar 17, 2022

freakinfofa commented Mar 17, 2022

dougbebber commented Mar 21, 2022

freakinfofa commented Mar 29, 2022

avolon42x commented Mar 31, 2022

dougbebber commented Mar 13, 2022 •

edited

Loading

dougbebber commented Mar 15, 2022 •

edited

Loading