daily HA crash at 12am with HA 2024.12

[jata1]

Hey Gary - hope all is well and you are having a nice holiday. Thanks as always for the great work on this integration.

I'm still investigating this issue and I am not 100% sure it is caused by icloud3 - but it might be worth looking into.

I am currently running 3.0.5.9 due to the issue with apple authentication.

I think HAOS 14 (so 2024.12.x) includes an updated version of python. Since updating to 2024.12 I have been experiencing an issue where my HA container crashes at 12am. It recovers/restarts but the crash is screwing with a number of my sensors that get reset at 12am. This is how I worked out I had an issue at all.

I have done a load of investigating and I have a feeling it could be related to the icloud3 integration. My clue is that I setup a test HA environment in a VM (HAOS) and have been running it in parallel with my prod system. I disabled icloud in my test environment and HA does not crash.

In my prod environment (where I am still experiencing this issue), I see in the HA log at the time it crashes the following entry.

2024-12-11 00:00:10.446 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140732429842848] Jago from 172.1.0.7: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1733835610.4462795,"c":"01JERAFPAENT5WWBKX6QX48CMP"}}}},"id":7}'

Happy to provide additional info as needed. Good if you could take a look when you get back.

Today I have reverted back to 2024.11.3 in prod as I want to confirm the issue is linked to 2024.12 (HAOS 14).

[gcobb321]

I am working on v3.1.4 fo get the Appple password login working again (it is using he old v3.0.5.9 method) and want to try changing the time the end-of-day tasks are done from midnight to 12:15 or another time to see if that solves your problem..

Edit icloud3/icloud3_main.py. Change line 604:

From :
if Gb.this_update_time == '00:00:00':

To:
if Gb.this_update_time == '0015:00':

Restart HA

If that still have problems, try another time

[jata1]

Thanks. I will and report back.

I'm also planning to htop my system at 12am (and 12.15am) to see if I get any clues.

[jata1]

changed the midnight setting to the following. Report back in the next couple of days.

if Gb.this_update_time == '00:15:00':

[jata1]

I can confirm that the cpu/load spike is related to icloud. Happened at 12:15am since I changed the config. Same error in the HA log relating to the following.

However, I have HA setup on test server with a copy of my prod config and it is not having the same issue so I am stumped.

My prod server is rpi5 and test is rpi4 both running exact same version of raspbian etc.

ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140732429842848] Jago from 172.1.0.7: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1733835610.4462795,"c":"01JERAFPAENT5WWBKX6QX48CMP"}}}},"id":7}'

[gcobb321]

One of the end of day tasks is doing some maintenance on the Waze History Database. If it needs to recalculate the time/distance, a message may be displayed in the first device’s info sensor. Check and disable the Waze History on the Configure Waze Screen and see if the problem occurs.

[jata1]

Thank you. Do you think it is strange that this only happens on my prod server as my test setup has the same icloud config?

For now, I have disabled waze in prod. Is this correct and what you meant for me to do?

[gcobb321]

You can keep using Waze. The db maintenance is controlled by the Waze history db option so that is the only one that really needs to be disabled

[jata1]

Thanks. Understood. Ill turn it all off and then re-enable as we work it through.

[jata1]

I am still getting these errors and a big load spike related to something that is happening with icloud3.

Anything else that you can suggest I look into? It's just this one server/container as I can't reproduce it on either of my test setups.

Very strange...

2024-12-17 00:15:18.697 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140732349590272] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.icloud3_wazehist_track":{"+":{"s":"-33.768506, 151.205397","lc":1734354918.6969175,"c":"01JF7SQQ9883C0MBQH2F0MFNYW","a":{"records":5937,"latitude":-33.768506,"longitude":151.205397}}}}},"id":7}'
2024-12-17 00:15:21.709 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140731563208864] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734354921.7095127,"c":"01JF7SQT7DT2RFVB7FPTX2VHSD"}}}},"id":7}'
2024-12-17 00:15:24.403 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140731763869584] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734354924.4030519,"c":"01JF7SQWVKAHV0S2BANZ2H1H4T"}}}},"id":7}'
2024-12-17 00:15:26.635 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140731540671072] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734354926.6352224,"c":"01JF7SQZ1BWG1SR9NDSCB39TA9"}}}},"id":7}'

[gcobb321]

I found a problem with the waze history maintenance. If there were duplicate or incorrect errors in the database, the maintenance would delete them. If there were a lot of them, the updates were not being 'committed' and that could lead to error notifications.

Unzip the file below into the icloud3/support directory and restart HA. Maybe the history database on your production system is fine but the one on your test system has these problems.

waze_history.zip

[jata1]

Thanks. I didn't have time to make this change yesterday so I will see what happens tonight after I update waze history.

Not sure if this helps but last night I had the issue and it caused the HA container to crash. So I have now reproduced the error that causes HA to crash and it happened at 12.15am and does seem to be related to waze/icloud.

log from HA at the time:

2024-12-18 00:15:18.544 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140731696901296] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734441318.5446994,"c":"01JFAC4E4GT4Y3BD8KHNX5T1MX"}}}},"id":10}'
2024-12-18 00:15:23.145 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140731528515296] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"s":"Waze History > Records-7955/11682, GPS--33.7538:151.2884","lc":1734441323.1448536,"c":"01JFAC4JM86V096GDKT60X8KDS"}}}},"id":7}'

icloud log at the time:

12-18 00:15:09 ------   Waze History Database > Compressed, Record Count-11682
12-18 00:15:09 ------   Waze History > Refreshing HA Map locations for
                            • Sensor-sensor.icloud3_wazehist_track
                            • RecordCnt-11682, OrderedBy-North-South

[jata1]

Sorry I forgot to say that I don't really need waze history so I can delete the database if that will help and is possible without causing other issues.

For now I will just use the new waze_history.py and report back,

[jata1]

Hey Gary - for some reason this update to waze_history causes issues with icloud services/actions. None are available after updating and this breaks a number of automations that rely on icloud3.action for my device trackers.

This wouldn't be a problem if this issue was on my test setup but the issue is on prod...

related log entries.

2024-12-18 07:11:29.848 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module with args ('custom_components.icloud3',) in /usr/src/homeassistant/homeassistant/loader.py, line 1074: ComponentProtocol, importlib.import_module(self.pkg_path) inside the event loop; This is causing stability issues. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#import_module
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 227, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 213, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 154, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 708, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 679, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2027, in _run_once
    handle._run()
  File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component
    result = await _async_setup_component(hass, domain, config)
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1026, in async_get_component
    comp = self._get_component()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1074, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)

2024-12-18 07:11:29.861 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration 'icloud3': Unable to import component: cannot import name '_evlog' from 'custom_components.icloud3.helpers.messaging' (/config/custom_components/icloud3/helpers/messaging.py)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1014, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self._get_component, True
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1074, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/icloud3/__init__.py", line 34, in <module>
    from .support.v2v3_config_migration import iCloud3_v2v3ConfigMigration
  File "/config/custom_components/icloud3/support/v2v3_config_migration.py", line 112, in <module>
    from .                      import config_file
  File "/config/custom_components/icloud3/support/config_file.py", line 39, in <module>
    from ..support              import start_ic3
  File "/config/custom_components/icloud3/support/start_ic3.py", line 69, in <module>
    from .waze         import Waze
  File "/config/custom_components/icloud3/support/waze.py", line 12, in <module>
    from ..support.waze_history         import WazeRouteHistory as WazeHist
  File "/config/custom_components/icloud3/support/waze_history.py", line 6, in <module>
    from ..helpers.messaging    import (broadcast_info_msg, format_filename,
    ...<3 lines>...
                                        _evlog, _log, )
ImportError: cannot import name '_evlog' from 'custom_components.icloud3.helpers.messaging' (/config/custom_components/icloud3/helpers/messaging.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1034, in async_get_component
    self._component_future.result()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1026, in async_get_component
    comp = self._get_component()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1074, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/icloud3/__init__.py", line 34, in <module>
    from .support.v2v3_config_migration import iCloud3_v2v3ConfigMigration
  File "/config/custom_components/icloud3/support/v2v3_config_migration.py", line 112, in <module>
    from .                      import config_file
  File "/config/custom_components/icloud3/support/config_file.py", line 39, in <module>
    from ..support              import start_ic3
  File "/config/custom_components/icloud3/support/start_ic3.py", line 69, in <module>
    from .waze         import Waze
  File "/config/custom_components/icloud3/support/waze.py", line 12, in <module>
    from ..support.waze_history         import WazeRouteHistory as WazeHist
  File "/config/custom_components/icloud3/support/waze_history.py", line 6, in <module>
    from ..helpers.messaging    import (broadcast_info_msg, format_filename,
    ...<3 lines>...
                                        _evlog, _log, )
ImportError: cannot import name '_evlog' from 'custom_components.icloud3.helpers.messaging' (/config/custom_components/icloud3/helpers/messaging.py)

[gcobb321]

My bad. I sent you a waze_history.py file for v3.1.4. That would create all kinds of problems and iCloud3 would not start due to version incompatibilities.

The v3.0.5.9 file is below.
waze_history v3.0.5.9.zip

[jata1]

sorted. thanks.

[jata1]

Not sure if this is related but my wife's iphone is no longer getting gps updates from icloud3. My iphone and my daughters is still working fine. All phones are all 'family sharing' to my icloud account and all worked previously.

Location working fine in find my app - just not with icloud3.

Note that this issue is not related to using the waze_history.py (as I noticed this issue yesterday).

This error originated from a custom integration.

Logger: custom_components.icloud3
Source: custom_components/icloud3/helpers/messaging.py:457
integration: iCloud3 v3: iDevice Tracker (documentation, issues)
First occurred: 8:46:49 AM (1 occurrences)
Last logged: 8:46:49 AM

iCloud3 > Magda iPhone/magda_iphone > No GPS data was returned from iCloud Location Service on the initial locate

[gcobb321]

A quick guess would be that HA has appended a '_2' to device_tracker and sensor entities of your wife. iCloud3 is using device_tracker and sensor entities (without the '_2'). Check on 'developer tools > states` to see what entities HA is using.

[jata1]

Nope. Her phone/setup does not use the HA app (just me but I don't have the xx_2 issue either)

[gcobb321]

Check the iCloud3 entities, not the HA app entities. HA may have changed the ic3 entity names and added a _2 on a restart. Then they would both exist and ic3 would update device_tracker.wife_phone_2 while the Dashboard and automation would be looking at device_tracker.wife_phone. Same for the sensors.

The best/easiest way to fix this is to disable the iCloud3 integration and restart hA 2 times (yes 2 times to clear it's cache).
Then go to the Settings > Entities screen and check the all entities box in the upper left.
Then select delete in the upper right to delete everything. HA will give you a warning that only the unused ones are deleted.
Do it.
Enable iCloud3 Integration and restart HA.

[jata1]

none of my icloud3 devices/entities have _2 at the end. They all look correct...

[gcobb321]

That's actually good. I assume you mean that iCloud3 is no longer getting location updates from her phone. If the FindMy app on your phone shows her location, then her Location Sharing is enabled. Check the d_t entity assigned to her phone in Stage 4 & 5 when iC3 starts to make sure that is correct. Stage 4 shows the phone names Apple is sending to iC3.
Verify the iC3 Configure > Update Device for her phone to make sure the Apple/iCloud/FamShr phone name for her phone is assigned to her iC3 phone

[jata1]

Yes - icloud no longer getting updates from her phone
Yes - find my app working correctly with location updating for her phone
Yes - device tracker assigned looks correct
Yes - stage 4 startup looks fine for all 3 phones that I track
Yes - device setup looks fine for all 3 phones

[jata1]

Maybe her phone needs a reboot. I will update it to latest iOS and see if this clears the issue

[jata1]

Some progress...

This morning at 12.15am the cpu/load spike was very short/quick. I am still seeing some messages relating to system load but not a message specific to waze. So I think there is still something weird going on. I will continue to monitor.

Let me know if you have any other thoughts/suggestions.

2024-12-19 00:15:24.608 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140735671683968] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734527724.6087534,"c":"01JFCYHB20XZBPH80Z4K656VCF"}}}},"id":7}'
2024-12-19 00:15:27.541 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140735371438704] Jago from 172.1.0.8: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.jago_iphone_info":{"+":{"lc":1734527727.5411427,"c":"01JFCYHDXN5K4MQ7B72WEHPZC8"}}}},"id":7}'

[jata1]

... and in other good news. Updating my wife's phone to iOS 18.2 and a restart seems to have cleared the issue with location updates to icloud3. So I think this issue was a glitch on the client (iphone).

[gcobb321]

I was mistaken about turning off the Waze history would stop the end of day maintenance running. It will run anyway on v3.0.5.9. I will change that on v3.1.4. Rename the .storage/icloud3/waze_location_history.db to something else and restart HA. If Waze history is disabled, it will not be recreated on a restart and that will prove that is the source.

When it runs at night, it will call Waze to update the location time and distance info. That will issue http calls to Waze to do this. It wits for one update to finish before starting the next one and it might be that your system can not handle the load. My RPi 4 and 5 chugs along without any problems.

The purpose of keeping the history is to eliminate Waze requests for the same locations as you drive. Especially when close to home with short location request intervals.

[jata1]

thanks. I will fully disable and delete the db. Report back tomorrow

[jata1]

More progress. After totally disabling waze and renaming the waze history db - no more cpu/load spike. So there does look like something has gone wrong with my waze db (but it only affects my prod system). My test systems both are direct copies of prod (including waze db) so it is still strange.

I will re-enable waze when you release the next update to icloud3 and see if the issue is resolved.

Will a new db for waze history be created if/when I reenable waze?

[jata1]

and I can see no spike at 12.15am on prod today. Small spikes as expected on my test systems.

and this is the same time range a day earlier - note cpu/load peaks at 15 but only on prod

[JtwoA]

@gcobb321 I am seeing this same behaviour - and it moved when you moved the db rebuild. I'm not going to troubleshoot further or press on the issue as I'm planning to move my setup off of my RPi to a MiniPC in the coming weeks. Just wanted you to be aware.

[gcobb321]

It would be interesting to see what happens if you copy the .storage/icloud3/waze_location_history.db file from the production to the test system.

In any case, I do not see any problems. It’s just cleaning up the database.

[DevHaydon]

Hey there, have been following this over the last couple of weeks have had my HomeAssistant go down a few times I have just been restarting it until I got the time to look into it.

So what I think is happening is that after the Database Clear up has been done then after it is getting a state and posting thousands of updates to the database.

As you see from the logs it seems its doing some updating of the info

But when you look at the database its getting hit alot

This is what I am seeing in the homeassistant logs before it crashes

2025-01-03 00:15:10.537 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546520473680] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.12.0 aiohttp/3.11.10 Python/3.12): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.aaron_haydon_iphone_info":{"+":{"lc":1735863310.5369067,"c":"01JGMR86685ZRVGJETVWQRGC17"}}}},"id":7}' 2025-01-03 00:15:10.538 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546434589616] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.12.0 aiohttp/3.11.10 Python/3.12): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.aaron_haydon_iphone_info":{"+":{"lc":1735863310.5369067,"c":"01JGMR86685ZRVGJETVWQRGC17"}}}},"id":7}' 2025-01-03 00:15:12.309 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [546435575104] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.12.0 aiohttp/3.11.10 Python/3.12): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"c":{"sensor.aaron_haydon_iphone_info":{"+":{"lc":1735863312.3088803,"c":"01JGMR87XMTBFRMNBG17KJEZBQ"}}}},"id":2}' 2025-01-03 00:15:12.309 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [545997019536] Aaron from fe80::11bb:3e14:9558:229 (Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/6.7.1 Chrome/118.0.5993.220 Safari/537.36): Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"state_changed","data":{"entity_id":"sensor.aaron_haydon_iphone_info","old_state":{"entity_id":"sensor.aaron_haydon_iphone_info","state":"Home-@6:53p","attributes":{"integration":"iCloud3","sensor_updated":"2025-01-03 00:15:12","icon":"mdi:information-outline","friendly_name":"Aaron Haydon iPhone Info"},"last_changed":"2025-01-03T00:15:12.308434+00:00","last_reported":"2025-01-03T00:15:12.308434+00:00","last_updated":"2025-01-03T00:15:12.308434+00:00","context":{"id":"01JGMR87XMRJMXRAMGGHBATAW6","parent_id":null,"user_id":null}},"new_state":{"entity_id":"sensor.aaron_haydon_iphone_info","state":"Home-@6:53p","attributes":{"integration":"iCloud3","sensor_updated":"2025-01-03 00:15:12","icon":"mdi:information-outline","friendly_name":"Aaron Haydon iPhone Info"},"last_changed":"2025-01-03T00:15:12.308880+00:00","last_reported":"2025-01-03T00:15:12.308880+00:00","last_updated":"2025-01-03T00:15:12.308880+00:00","context":{"id":"01JGMR87XMTBFRMNBG17KJEZBQ","parent_id":null,"user_id":null}}},"origin":"LOCAL","time_fired":"2025-01-03T00:15:12.308880+00:00","context":{"id":"01JGMR87XMTBFRMNBG17KJEZBQ","parent_id":null,"user_id":null}},"id":3}'

Icloud logs before crashes:
01-03 00:15:05 ------ ⡇ Waze History Database > Compressed, Record Count-6691 01-03 00:15:05 ------ ⡇ Waze History > Refreshing HA Map locations for • Sensor-sensor.icloud3_wazehist_track • RecordCnt-6691, OrderedBy-North-South 01-03 00:15:08 ------ ⡇⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂ ⡇ END-OF-DAY MAINTENANCE, COMPLETE,

My HomeAssistant
Core - 2024.12.5
Supervisor - 2024.12.0
Operating System - 14.1
I'm on the latest release of the iCloud integration

Hopefully this will help troubleshoot if others have the issue for now I have deleted the Waze database and Ill report back tomorrow and let you know if its still happening

One thing to mention is I only think it crashes on mine when I open homeassistant on a device but that I can't be sure about

[jata1]

Thanks for your detailed post and investigation.

I have waze turned off and the problem (core dump/system crash) has not returned. I am not getting nearly the same load spike or cpu load warnings on my server.

A service/integration that is doing maintenance (anything really) and it causes the container/service to crash - there is something going badly wrong.

I'm happy to keep monitoring and do any testing if that could help to get it resolved.

Happy New Year all.