Add the device.changed_api_heartbeat_state_on__date term to the model

[thgreasi]

Resolves: #644
Change-type: minor
See: https://balena.fibery.io/Work/Project/Start-recording-the-last-timestamp-the-device-heartbeat-changed-254

[thgreasi]

I originally added this as a hook, but at the end I preferred moving it here

• so that we can avoid the extra .get() request to confirm that the value has indeed changed, or blindly setting last_api_heartbeat_state_change_event whenever values.api_heartbeat_state != null.
• since this is the only place that we should be updating the heartbeat anyway.

Let me know though if you prefer to switch it back to hooks.

[fisehara]

Unfortunately I don't understand properly what this added property adds to the device state information?
Do we want to convey the last proper heartbeat time or do we want to surface all changes of heartbeat event states?
Eg. when it's set to online over and over again, it would be of interest when the last time the heartbeat was set to online, or am I mixing things up?

[fisehara]

I wonder if last_changed_api_heartbeat_state_on__date should only be updated when called via: await this.updateDeviceModel(deviceId, DeviceOnlineStates.Online);
and not when patching it to timeout or offline? As these events are internal events that do not show the device are heartbeating anymore?

[thgreasi]

The intention here is to track when the heartbeat was changes both for any reason (both from state-GETs and from heartbeat changes b/c of the heartbeat mechanism). Knowing then the device switched to Timeout/Offline is probably more useful than knowing since when the device originally started heartbeating.

[fisehara]

When we use the last_changed_api_heartbeat_state_on__date what do we want to convey? Do we want to convey when the device was last seen with a proper heartbeat or when the heartbeat status changes from online, to timeout, to offline?
How will we use the timeout and offline timestamps, eg as indicator in the UI?

[thgreasi]

We want to convey when the heartbeat status changed in general for all possible transitions.
Similarly to how UI atm uses the last vpn even to show

• online since
• offline since

with this field it can show

• heartbeating since
• heartbeat timeout since
• heartbeat offline since

On top of that this also allows us t compute the last heartbeat if needed on read time:

Given this field, with a simple computation using the device's poll interval, a client/computed term could easily compute even more useful information on read time, eg: compute the last heartbeat for a non-Online device.

[thgreasi]

Unfortunately I don't understand properly what this added property adds to the device state information?
Do we want to convey the last proper heartbeat time or do we want to surface all changes of heartbeat event states?
Eg. when it's set to online over and over again, it would be of interest when the last time the heartbeat was set to online, or am I mixing things up?

@fisehara in an ideal world we would add a device has last heartbeat on date, but that would tons of load on the writer, effectively making all state GETs do DB writes. This is exactly why we had to add the heartbeat update cache mechanism in the heartbeat implementation, so that we can reduce the writer load.

Given that we can't do the above, we started thinking what fact related to heartbeat we could store that would come with low DB load impact, and that's how we ended up with last_changed_api_heartbeat_state_on__date in that arch call since:

• it doesn't add extra DB load, since we do the write along with the operation that changes the heartbeat (and PG rewrites the whole row anyway)
• we can compute more interesting information on read based on this

Part of the reasoning in the arch call was that we atm have no additional info about the heartbeat, so adding any kind of free field is better than the no info that we atm have.

Even w/o extra computations, being able to surface when a device last switched to Online/Timeout/Offline does have some value, especially when compared w/ how atm we have no more info about the heartbeat (and we have to dive into Redis to get more).

Given this field, with a simple computation using the device's poll interval, a client/computed term could easily compute even more useful information on read time, eg: compute the last heartbeat for a non-Online device.

[thgreasi]

Thank you @fisehara I will make this part of the next cycle, so that we confirm the name again ❤️

[dfunckt]

@thgreasi I wonder if having a “state last modified date” instead would be sufficient, ie the date the device last switched between Online, Downloading, etc. states.

That would allow us to show eg. “Downloading” and “for 10 minutes”, or something similar.

It would also be useful to filter/order by this field in the device list and check if I, as a fleet operator, need to intervene—3 days in Downloading state clearly means something is off.

cc @shaunmulligan

[thgreasi]

@dfunckt the device.state is whatever the supervisor reports & is not changing based on the vpn.
The device.overall_status is computed term that depends on the heartbeat state on the other hand has different definitions between v6 & v7, so in order to capture the change date of the overall_status we would need to be computing & storing the value for both versions (or ignore v6 for example).

That's why we ended up deciding (years back) that storing the change date of the most primitive value is probably better, since the data is still usable even if we change how overall_status gets computed.