2.3.0.giga36 harvester disconnects, and stops working, needs to be manually restarted

[jszalkowski]

this is after upgrade to 2.3.0.giga36

2024-05-23T08:18:53.847 daemon chia.daemon.server : DEBUG About to ping: chia_harvester
2024-05-23T08:18:58.210 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452338}, initial False, last_sync_id 1716428777, plot_file_count 25609, harvesting_mode 1
2024-05-23T08:19:08.246 harvester chia.plot_sync.sender : INFO _send_next_message didn't receive response expected_message_type: plot_sync_start, expected_identifier: {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452338}, message None
2024-05-23T08:19:18.251 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452358}, initial False, last_sync_id 1716428777, plot_file_count 25609, harvesting_mode 1
2024-05-23T08:19:23.879 daemon chia.daemon.server : DEBUG About to ping: chia_harvester
2024-05-23T08:19:28.287 harvester chia.plot_sync.sender : INFO _send_next_message didn't receive response expected_message_type: plot_sync_start, expected_identifier: {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452358}, message None
2024-05-23T08:19:38.290 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452378}, initial False, last_sync_id 1716428777, plot_file_count 25609, harvesting_mode 1
2024-05-23T08:19:48.325 harvester chia.plot_sync.sender : INFO _send_next_message didn't receive response expected_message_type: plot_sync_start, expected_identifier: {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452378}, message None
2024-05-23T08:19:53.891 daemon chia.daemon.server : DEBUG About to ping: chia_harvester
2024-05-23T08:19:58.327 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452398}, initial False, last_sync_id 1716428777, plot_file_count 25609, harvesting_mode 1
2024-05-23T08:20:08.362 harvester chia.plot_sync.sender : INFO _send_next_message didn't receive response expected_message_type: plot_sync_start, expected_identifier: {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452398}, message None
2024-05-23T08:20:18.367 harvester chia.plot_sync.sender : DEBUG _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1716432380, 'timestamp': 1716452418}, initial False, last_sync_id 1716428777, plot_file_count 25609, harvesting_mode 1
2024-05-23T08:20:23.922 daemon chia.daemon.server : DEBUG About to ping: chia_harvester

[madMAx43v3r]

It happend just once? Or keeps happening?

[Hugh-Top]

harvsters always connected farmer failed by 2.3.0.giga36 :

[Hugh-Top]

Roll back to 2.2.1.giga35 and all havesters are normal. I have lost a lot of rewards since upgrading to version xxx 2.3.0.giga36

farmer: Ubuntu 22.04.3 LTS
haverster: docker

run.sh

[Hugh-Top]

New error:

[madMAx43v3r]

Are all machines running the same GH version and not official?

[Hugh-Top]

Are all machines running the same GH version and not official?

yes

[madMAx43v3r]

Can you double check? Because it seems you are mixing offical farmer with GH harvester or different versions?

./chia.bin version after reboot, to make sure no old stuff still running.

[madMAx43v3r]

If you update to 2.3.0.giga36, both farmer and harvester need to be running 2.3.0.giga36.

And before updating you need to stop daemon with ./chia.bin stop all -d

[Hugh-Top]

I added gpus to compute, now haveters is normal

[Hugh-Top]

I added gpus to compute, now haveters is normal

2.3.0.giga36 and 2.2.1.giga35 are normal after add gpus

[madMAx43v3r]

Well the filter halvening just happened yesterday, so maybe you were overloading GPUs.

[Hugh-Top]

I have a suggestion: if there is not enough GPU, can there be some log prompt instead of the connection failure like now?

[Hugh-Top]

how many time is ok? I used to think that anything below 2000ms was fine

[madMAx43v3r]

I have a suggestion: if there is not enough GPU, can there be some log prompt instead of the connection failure like now?

I don't know why there was a connection failure... it's not from overload.

how many time is ok? I used to think that anything below 2000ms was fine

Need to look at chia harvester logs, not recompute time.