Expose device configuration after hass is online

[urpylka]

Hi, I solved the problem with device resending device configuration after restarting Homeassistant

Short story long: It is about Zigbee devices unavailable (in Homeassistant 2023.12.3) after Home Assistant restarting.

I added calling the discover(entity, forced) when we get online message in homeassistant state topic.

I am not sure about using next section:

if (entity.isDevice() && this.discoveredTriggers[entity.ieeeAddr]) {
    for (const config of this.discoveredTriggers[entity.ieeeAddr]) {
        const key = config.substring(0, config.indexOf('_'));
        const value = config.substring(config.indexOf('_') + 1);
        this.publishDeviceTriggerDiscover(entity, key, value, true);
    }
}

Please let me know if some changes I should to add to the PR or edit my PR by yourself (I allow it).

Source: #18862

[urpylka]

As I see we need to cover by tests on homeassistant.ts 1688-1691.

But the problem that I have never wrote on TypeScript and covered smth by tests, as I see we need smth like that https://github.com/Koenkk/zigbee2mqtt/blob/master/test/homeassistant.test.js#L78 or just delete the section 1687-1694.

Please assist

[Koenkk]

With this PR, all discovery messages will be send again everytime HA comes online. We already have issues with this: #20648

Short story long: It is about Zigbee devices unavailable (in Homeassistant 2023.12.3) after Home Assistant restarting.

I also don't understand how this fixes this problem. MQTT messages should be retained so Z2M shouldn't have to send them again. I would expect that just resending the device availability would also fix this issue?

[urpylka]

Hi, let me investigate it this week. Will return with answer.

[urpylka]

The basis for understanding how Homeassistant works with mqtt is this instruction.

After configuring MQTT integration in Homeassistant besides the ability to use topics in manually described devices, there is the ability to automatically discover and configure new devices.

The most important information related to this case is described in How to use discovery messages. I will make some quotes from it:

When Home Assistant is restarting, discovered MQTT items with a unique ID will be unavailable until a discovery message is received.

There are two approaches to make sure the discovered items are set up at startup:

Using Birth and Will messages to trigger setup
Using retained messages

You were right – one way is using retained messages.

After that, I tested subscriptions of Homeassistant after restarting at webui of my broker:

As you see it has:

homeassistant.*.*.*.config
homeassistant.*.*.config

It means that any messages that devices are available will not be consumed by Homeassistant - there are no subscriptions to these topics (for ex zigbee2mqtt/0xa4c138bae2333cf1).

I don't know how next problems relate to another brokers, but I use RabbitMQ 3.8.19 and it has next ones:

• No support to send retained messages by a mask.
• The broker will flush retained messages after restarting.
• Finally there is a problem with working retained messages on HA setup it will only work on the same node.

Sources:

• Retained messages are only delivered to clients for which a queue currently exists
• MQTT: replicated message retention

After this investigation my opinion that we need to implement resending discover information when get Birth message (and probably can delete retain flag). Or second way - disable discovering entities by mqtt in Homeassistant and configure it manual.

[Koenkk]

@urpylka this sounds like a broker issue? Can you try with mosquitto?

[mundschenk-at]

Both Mosquitto and EMQX can persist retained messages over a broker restart.

[urpylka]

@Koenkk I tested as you asked on Mosquitto regarding subscription by mask to get a retained message, and it works.

> mosquitto_sub -t '#' -v
test/status online

But I am not sure that it will change smth. Because Mosquitto doesn't pretend to be a production solution (it doesn't have High Availability functions). And I still use RabbitMQ with the known issue that I can't get retained message by mask.

If you see reason to use current approach to discover devices, maybe we need to create a configuration parameter to allow user to choose which trigger is use for discover (by retain mes / by birth mes)?

[corporategoth]

The problem seems to be that the ONLY topic being persisted is the bridge state.
To wit:

 mosquitto_sub -u test -P '****' -t 'homeassistant.*.*.*.config' -t 'homeassistant.*.*.config' -t 'zigbee2mqtt/bridge/stat
e' -t 'zigbee2mqtt/Office Closet Light'
{"state":"online"}

I get no data, no results. Home Assistant and Z2M ((and mosquitto) are all up.
This means, if I restart HA right now, it would not autoconfig (because that is triggered by Z2M's restarting, not HA's) because all the config topics are not persisted.
It would also get the current status of any device until they happened to update in Z2M (ie. the device itself sends an update).

This means

1. HA might not have the correct list of devices, as it would just be configured with whatever it was previously (and Z2M config could have changed while HA was offline).
2. HA would not have the current state of devices it DOES know about, because those device states are not set to be persisted in MQTT.
3. It should know things are online, but that's about it.

If I leave the above command running, I get updates on my office closet light state when it sends updates. But NOT at HA startup. And if that was a new device added while HA was down, HA would never know about it. Or if that device was REMOVED while HA was disconnected, HA would always remember it / think it still exists.

A number of versions ago (I don't remember which), HA used to send some kind of message to Z2M that triggered autoconfig again. But that doesn't appear to happen anymore. Either because HA stopped sending it, or Z2M stopped handling it. So right now, if HA restarts, I DO have to restart Z2M too to ensure autoconfig happens again.

[mundschenk-at]

The problem seems to be that the ONLY topic being persisted is the bridge state. To wit:

 mosquitto_sub -u test -P '****' -t 'homeassistant.*.*.*.config' -t 'homeassistant.*.*.config' -t 'zigbee2mqtt/bridge/stat
e' -t 'zigbee2mqtt/Office Closet Light'
{"state":"online"}

I get no data, no results.

That is not valid MQTT wildcard syntax, so it's no surprise you don't get any results.

[Koenkk]

@corporategoth is anything published to homeassistant/status when HA starts?

[mundschenk-at]

@corporategoth try this:

 mosquitto_sub -u test -P '****' -t 'homeassistant/+/+/+/config' -t 'homeassistant/+/+/config' -t 'zigbee2mqtt/bridge/state' -t 'zigbee2mqtt/Office Closet Light'

[corporategoth]

@corporategoth try this:

 mosquitto_sub -u test -P '****' -t 'homeassistant/+/+/+/config' -t 'homeassistant/+/+/config' -t 'zigbee2mqtt/bridge/state' -t 'zigbee2mqtt/Office Closet Light'

Aha, that works. Then I get >10k entries.

@Koenkk
Yeah, when I just restarted my HA, I got

offline
online

So apparently it does publish there. Not a JSON blob, just the words themselves.

HOWEVER, if I look at most of the config options on the device in question above, MOST of them are unavailable in HA.
Zeroing in on 1 control, 'ActiveEnergyReports', I noticed I have 2 config messages returned by that wildcard above:

homeassistant/sensor/0x6c5cb1fffe5e7187/activeEnergyReports/config:
{"availability":[{"topic":"zigbee2mqtt/bridge/state","value_template":"{{ value_json.state }}"}],"device":{"identifiers":["zigbee2mqtt_0x6c5cb1fffe5e7187"],"manufacturer":"Inovelli","model":"Inovelli 2-in-1 switch + dimmer (VZM31-SN)","name":"Office Closet Light","sw_version":"2.15","via_device":"zigbee2mqtt_bridge_0xe0798dfffea88b0a"},"enabled_by_default":false,"name":"ActiveEnergyReports","object_id":"office_closet_light_activeEnergyReports","origin":{"name":"Zigbee2MQTT","sw":"1.35.3","url":"https://www.zigbee2mqtt.io"},"state_topic":"zigbee2mqtt/Office Closet Light","unique_id":"0x6c5cb1fffe5e7187_activeEnergyReports_zigbee2mqtt","value_template":"{{ value_json.activeEnergyReports }}"}

homeassistant/number/0x6c5cb1fffe5e7187/activeEnergyReports/config:
{"availability":[{"topic":"zigbee2mqtt/bridge/state","value_template":"{{ value_json.state }}"}],"command_topic":"zigbee2mqtt/Office Closet Light/set/activeEnergyReports","device":{"identifiers":["zigbee2mqtt_0x6c5cb1fffe5e7187"],"manufacturer":"Inovelli","model":"Inovelli 2-in-1 switch + dimmer (VZM31-SN)","name":"Office Closet Light","sw_version":"2.15","via_device":"zigbee2mqtt_bridge_0xe0798dfffea88b0a"},"max":32767,"min":0,"name":"ActiveEnergyReports","object_id":"office_closet_light_activeEnergyReports","origin":{"name":"Zigbee2MQTT","sw":"1.35.3","url":"https://www.zigbee2mqtt.io"},"state_topic":"zigbee2mqtt/Office Closet Light","unique_id":"0x6c5cb1fffe5e7187_activeEnergyReports_zigbee2mqtt","value_template":"{{ value_json.activeEnergyReports }}"}

I wonder if, in my case, something is hanging around, the topic not cleaned up by Z2M. And one is overriding the other?

Meaning:

1. I would have to clear out my MQTT state entirely and re-build it to fix this, and
2. The logic Z2M has to clean up topics that are not/no longer valid is not working

This would also be why, if I restart Z2M, it will re-send config stuff for the correct topic only, which will kick HA to use the correct info.

Not sure what's going on in my case

[urpylka]

@corporategoth it is not a wildcard syntax (homeassistant.*.*.config). It is a routing key of RabbitMQ, but we can assume which wildcard it is suppose to be (as @mundschenk-at has wrote).

[Koenkk]

I expect the only thing needed to fix this is adding this.eventBus.emitPublishAvailability(); to

zigbee2mqtt/lib/extension/homeassistant.ts

Line 1660 in f04ade6

[urpylka]

I expect the only thing needed to fix this is adding this.eventBus.emitPublishAvailability(); to

zigbee2mqtt/lib/extension/homeassistant.ts

Line 1660 in f04ade6

I will check next week and let you know help it or not.

[popy2k14]

@urpylka any news on this?
Does the provided fix from Koenkk #21500 (comment) work?

Thanks a lot for investigating the issue.

[urpylka]

Hi, I am sorry for the delay, I've just tested this.eventBus.emitPublishAvailability(); and it doesn't work: after restarting devices are unavailable.

@Koenkk I understand that can consider it MQTT issue, but I think that we should implement another discovery mechanism by birth message from Homeassistant. (If this problem is so widespread). That approach doesn't have any problems for that from my perspective.

[xekil]

Good morning,
Is a fix coming?
THANKS

[Koenkk]

@urpylka

Hi, I am sorry for the delay, I've just tested this.eventBus.emitPublishAvailability(); and it doesn't work: after restarting devices are unavailable.

After adding this, do you see availability info being published to Z2M when HA starts?

[urpylka]

@Koenkk Hi, I am not sure what do you mean by availability info so after restart HA I got next strings in logs of Z2M:

Zigbee2MQTT:info  2024-04-10 09:35:10: MQTT publish: topic 'zigbee2mqtt/<some_dev_id>', payload '{"linkquality":111,"power_on_behavior":"previous","state":"OFF","switch_type":"toggle"}'
Zigbee2MQTT:info  2024-04-10 09:35:10: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"online"}'

But for correct working Homeassistant we need to resend to it devices configs to topics: homeassistant/<dev_type>/<dev_id>/<type>/config

[Koenkk]

You should also see, e.g. zigbee2mqtt/bulb_color/availability with payload online. If that' is not the case, the this.eventBus.emitPublishAvailability(); was not used correctly.

[urpylka]

You should also see, e.g. zigbee2mqtt/bulb_color/availability with payload online. If that' is not the case, the this.eventBus.emitPublishAvailability(); was not used correctly.

Everything that I got after restart HA, I texted above

[Koenkk]

• Can you add, e.g. a console.log('POINT 1') just below

  zigbee2mqtt/lib/extension/availability.ts

  Line 133 in ffc2ff1

  @bind private publishAvailabilityForAllEntities(): void {

  to see if this function is called when adding this.eventBus.emitPublishAvailability();?
• With this.eventBus.emitPublishAvailability(); added, could you provide the z2m debug log when HA comes online?

See this on how to enable debug logging.

[mundschenk-at]

I followed some of the recent MQTT-related PRs on the Home Assistant GitHub. It appears that there were some issues with message buffers being to low to handle the startup congestion caused by inefficient processing on the HA MQTT client, causing messages to be dropped. That might be one of the reasons things why the issue with HA restarts does not appear for everyone and does not seem to happen with EMQX at all.

Since those fixes are scheduled to be in 2024.6.0, it might be worthwhile to retest the original issue with that version.

[github-actions]

This pull request is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days

[Karatekid2407]

I have still this problem and it seems that Z2M should resend the device config every time it connects to MQTT.
What I noticed is that I miss MQTT Homeassistant Discovery messages when I reset MQTT and of course if I reset afterwards Homeassistant, the Devices can not be found.

Meaning Z2M should send the devices every time it connects to MQTT or should make same retain.

[mundschenk-at]

The discovery messages are retained, @Karatekid2407

[Karatekid2407]

ahh I See the problem. After upgrading to new MQTT version the messages are not stored anymore and therefore also the retained messages are gone. Meaning I need to change the config in MQTT. But maybe this is the same problem for HA and the add-on (I'm not using it so just guessing).

[jenshenk]

The discovery messages are retained, @Karatekid2407

Happy to have stumbled upon this PR! My setup is HA core as docker, mosquitto as service and z2m as docker on another box. With this setup I only experience the issue if I fully restart the machine HA and mosquitto are running on. Interestingly, I also experience it with some openbeken flashed devices, always the same out of a handful. With those devices if I broadcast a new discovery message they instantly appear online. With Z2M the only way I found for a solid workaround became restarting the z2m docker container twice.

With the info about the retained discovery messages and my experience with the effected openbeken devices I start to wonder whether HA might think it already did something with the retained messages but too long ago. While the issue seems to be on HA's side, could a "fix" be to have an option in z2m that monitors whether HA wasn't running for a while and then renew all discovery broadcasts?

[jenshenk]

A number of versions ago (I don't remember which), HA used to send some kind of message to Z2M that triggered autoconfig again. But that doesn't appear to happen anymore. Either because HA stopped sending it, or Z2M stopped handling it. So right now, if HA restarts, I DO have to restart Z2M too to ensure autoconfig happens again.

Connected to the above comment from around a year ago, I might have found something interesting. I'm on z2m 2.0.0 and if I go to Settings/HA Integration there's an option for entering the Home Assistant status topic, the default is set to homeassistant/status. Checking with MQTT explorer I noticed this topic doesn't exist, if this is supposed to check the status of ha, z2m might not even know whether ha is running or not.

What does Z2M expect on homeassistant/status and is there a timeout triggering some action? Because if the whole issue is caused by this it might be easily fixable with an automation in ha that sends a status message for z2m.

[mundschenk-at]

HA should publish birth and last will messages. Z2M expexts the default payloads online and offline there.