feature request: add or clarify GPS fix timestamp and timezone #286
Comments
|
The iCloud3 device_tracker entity contains the info you want. You can click on the device tracker entity on the dashboard screen, then select attributes or go to Developer Tools > States and select it. Mine are: It has the last located time as a date based value but I can easily add it as a full timestamp/secs without any problems if that will help. You do not want the HA device tracker info since iCloud3 already incorporates this. It monitors the MobileApp location on a 5-sec interval, reacts to MobileApp triggers and monitors the iCloud acccount location data. It will select the latest location timestamp and report that data. The update interval is normally determined by the distance from home to minimize the number of locate requests to the iCloud servers. This can be tailored on the Tracking Parameters config screen. You can also set a Fixed Interval on the Update Devices screen for each device being tracked. You might want to review the Mobile App docs for the triggers it generates. Pay close attentio to the Significant Location Change and Periodic triggers. Hope that helps, |
|
It really helps. Thanks. I am getting some very strange speed calculations using the iCloud data so I will pass this info onto the composite device tracker dev who is also helping me. How do you handle the fix/update timestamp when you use the data from the HA app as the integration does not have this attribute? and just checking that for either iCloud or HA app data, the last located attribute holds the fix time? thanks again and great work! |
|
I am finishing up v3.0.1 when I saw your note yesterday and have added some timestamp attributes to the device_tracker entity. They are: I have added and removed the actual timestamp seconds (local & utc) which can be derived from the located_local & located_utc. I'll send you v3.0.1b4 later today or tomorrow for you to work with. I'll also contact the other developer to see if there is anything specific that would make everything work together easier. The Mobile App location time comes from a last_update_time attribute from the device_tracker and the sensor.last_update_trigger entities. The code for getting the mobapp times is at the top of the /support/mobapp_data_handler.py and /helpers/entity_io.py modules. |
|
Any way you want to present the timestamp that corresponds to when the location values (latitude, etc.) were read/obtained on the device is fine. I just need to know how to interpret it, and that it won't be changing. I started implementing using the last_located attribute that already existed. Based on my reading of your code, it seemed that was the correct attribute to use, and that the time was always "local" in HA's configured time zone. If you want to change it to something else, that is fine, too. For me, the easiest would be to use a time zone aware format instead of a naive one. E.g., one of these: >>> from datetime import UTC, datetime
>>> utc_time_str = "2024-03-14 19:47:57"
>>> print(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC))
2024-03-14 19:47:57+00:00
>>> print(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC).astimezone())
2024-03-14 14:47:57-05:00Note that I'm in the US Central time zone, which is why the last one comes out that way. If the code ran in HA, it would be in HA's configured time zone (well, at least, the underlying OS's time zone, which should be the same.) Or, you could use something like: >>> from homeassistant.util import dt as dt_util
>>> # Simulate HA's time zone configuration.
>>> dt_util.set_default_time_zone(dt_util.get_time_zone("US/Eastern"))
>>> print(dt_util.as_local(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC)))
2024-03-14 15:47:57-04:00 |
|
So good you two guys are discussing this together. I am an idiot in the middle. haha Thanks so much to you both! |
|
Here is some data from this morning. I went for a short run - 5km. You can see the icloud GPS data is doing something weird.
Here is the same chart without icloud so we can compare the other data sources. By the way - this data below looks correct.
|
|
Let me get a beta out for the composite integration that uses the existing last_located attribute. I can always change it if the iCloud3 attribute changes. |
|
Ok, 3.4.0b0 ready for a try. |
|
Thanks. Will try that now and report back. |
|
Just to update this thread that we are making progress - latest chart below shows no more crazy speed spikes and overall data not too far away from what traccar app is reporting (based on a 2min update frequency)
|
|
I've changed the format of the located attribute to utc time. I've had to wrap the located attribute in parentheses _ (2024-03-15 05:37:58-04:00)_ so HA does not reformat it on the device_tracker > Attributes screen to March 15, 2024 at 7:11:17 AM where it reports the time and drops the time zone info. Here is the latest v3.0.1b4Unzip into the icloud3 directory and restart HA. Change Log : v3.0.1 - 3/15/2024
|
That is now a time zone aware time (as opposed to a naive time, which it was before), but it's not UTC. UTC would be "+00:00".
That is not necessary. The attribute will still have the time zone suffix. It shouldn't matter if the frontend chooses to display the time as local (no matter what time zone offset the attribute has.) That is normal. |
|
Understand, but was trying to avoid confusion by having the front end display the located time with the time zone instead of the utc time as a normal time which would confuse a lot of people. I have changed the local time to located and it is displayed under the gps location. The utc time you want is now named located (utc) and at the bottom. A new iCloud3 v3.0.1b4 with this change. Let me know if other adjustments will help. |
|
I think you're confusing the terms "UTC" and "time zone aware". "
As I stated previously, I don't care how the time is formatted, I just need to know how to interpret it, and that it doesn't change once decided on. A naive time (i.e., with no UTC offset suffix) is fine, as long as it is always in HA's local time. A time zone aware format (w/ a UTC offset suffix) is better, since it's not open to interpretation, but isn't absolutely necessary. The time zone aware format can be in any time zone, that doesn't matter, which is the whole point to a time zone aware format. 😄 Having said that, the HA core team generally prefers UTC. So, at this point, the "located (utc)" attribute is, at least, misleading. I'd recommend changing its value to UTC, or changing the name of the attribute. Also, attribute names in HA are typically snake case, so Putting that all together, instead of:
it should probably be:
|
|
Here you go. |
|
I'll update my integration accordingly, then @jata1 can test and let us know. |
|
Wait a second. |
|
If you are starting with a "POSIX timestamp" (i.e., seconds since the "Epoch", which is 1/1/1970, and which is what If you want a time zone aware time in UTC: >>> import time
>>> from datetime import datetime, UTC
>>> current_time = round(time.time())
>>> print(datetime.fromtimestamp(current_time, UTC))
2024-03-15 20:02:57+00:00If, however, you want a time zone aware time in the local time zone: >>> import time
>>> from datetime import datetime, UTC
>>> current_time = round(time.time())
>>> print(datetime.fromtimestamp(current_time).astimezone())
2024-03-15 15:02:57-05:00I've also shown above how to use an ISO formatted time string to get the same results. If you could point me to your code, I could give specific suggestions. I've given generic solutions so far for how you might do this. And, BTW, if you don't want to use a Python datetime object as the attribute value (which is another possibility), you can easily convert to a str, e.g.: >>> attribute_value = str(datetime.fromtimestamp(current_time, UTC))
>>> print(attribute_value)
2024-03-15 20:02:57+00:00 |
|
Thanks to you both I have my driver speeding tracking system working so much appreciated. @gcobb321 - I will try out the 3.0.1b4 and report back. The only other thing that would help me is if the integration requests gps data a few times (over a few minutes) when first entering a zone (especially home zone) - to confirm device not moving / speed zero - before switching to the defined inzone interval. Do you think this is possible? @pnbruckner - will me switching to icloud 3.0.1b4 break the version of your composite tracker integration? |
|
@jata1 It may be possible. When you enter the zone, set up an automation that calls the icloud3.update service call with a locate xxx seconds/minutes action. You may be able to set it to run several times. See docs Service Calls chapter for more info. @pnbruckner - I'm back home now so ping me if you have any questions about my timer handling. More info -- when iCloud3 starts, it determines the local-utc time zone offset secs and saves the tz offset string. That is in the time_util.py module in the def calculate_time_zone_offset(): function (line 542). The offset is used to convert timestamps from the iCloud account device locate data and the MobileApp data into local time for all reporting and display purposes. The time displayed in an earlier note as Correct me if I am wrong but I thought that 14:14:01+00:00 = 10:14:01-04:00. I have had it both ways and changed based on this comment: I do not understand why the HA Frontend displays Also, the attribute is already being posted as a string. secs is the epoch time in the local tz |
good idea - i will try that. Thanks |
Yes, I believe I saw that. But you're doing work the standard Python libraries and/or homeassistant.util.dt will do for you. How are the raw time values formatted? Are they the number of seconds since the epoch (i.e., a Unix/POSIX timestamp)? Or are they in some other format, say a string that represents time? If they are the former, then it's very easy (as I've explained) to convert that to any time zone aware time (UTC, the local time zone, or any other time zone.) If it's the latter, and it includes a UTC offset suffix, then again, it's very easy to convert to any time zone. If it's the latter, but does not contain any offset suffix, then you need to know what time zone it corresponds to, and then it's still very easy to convert to a time zone aware time in any time zone.
If it was for EDT, then the suffix is wrong. "+00:00" means UTC. "-04:00" would be correct for EDT. Also, there is no such thing as "local epoch secs". A Unix/POSIX timestamp is defined as the number of seconds since the epoch in UTC.
Yes, that is correct. Just like 2024-03-15 12:08:45+00:00 == 2024-03-15 08:08:45-04:00. My comment was concerning the name of the attribute. If the time is "xxx-04:00", then it is not UTC, so the name of the attribute should not imply it is.
Because 17:21:34+00:00 is the same as 13:21:34-04:00, assuming your local time zone is EDT. That is also equivalent to 1:21:34 PM. The frontend always displays times in local time, no matter what time zone they are reported in. The same goes for naive times (ones that don't have a UTC offset suffix), because naive times are assumed to be in local time. You say the iPhone was actually located at 5:21:34p. That is a naive time, so in what time zone? If local, then the attribute should not have been 2024-03-15 17:21:34+00:00; it should have been either 2024-03-15 17:21:34-04:00 or 2024-03-15 21:21:34+00:00.
Again, there is no such thing. From the Python docs:
|
Yes, it will. You'll have to wait until iCloud3 has an attribute with the correct data in an appropriate format. So far we haven't quite gotten there. |
|
Everything in iCloud3 is in the local time of the HA server which is where the Home zone is located. Distances and times are based on that zone since that is where most automations the user wants to manage are. It also supports other zones at other locations and can track devices using the distance and travel times to the other location. There is a feature in iCloud3 v3 that will let someone display a time in the local time it’s they are in another time zone (not released for general use yet) but all device tracking is still based on the time zone of the HA server, not the devices location. The epoch secs of the HA server Location is used for managing this. It does not matter that it is not a”true” epoch secs in a Python sense. I have never used that term before communicating with someone because I never had to. The way I have been handling the time has met my needs for a long time and not presented any problems. If I was starting over, I would probably do it differently but really have no need or desire to redesign it at this time since it works. Your comments are interesting in an educational sense and you have a better grasp of the terminology between utc, time zones, how they should be displayed, what the attributes field should be called than I do. I’ve never really needed to use local or utc times in anything so have not dig into it. I can make the field name anything you would like that I am not already using. I can make the field value in any format you want. It can be the:
My understanding of your app is to determine the devices speed between 2 locations and you need the gps of both locations and the time it was at each one so you can calculate the time it took to get from A to B to calculate the v=d/t value. It will take me only a few minutes to make the code changes to s set the field format to what you would like. Let me know its name and format. Gary |
The first is wrong, and the second is misleading and unconventional, so let's avoid those. FWIW, the composite integration already looks for an attribute with either of these names, in this order: And it accepts any of the following as values:
Please note it currently does not support a Also, a POSIX timestamp is what I believe you refer to as "utc secs." So, if you could create an attribute with one of the names composite already looks for, and whose value is any of those it already supports, then it would just work with the current release of the composite integration. But, it's no problem to add support for something different. |
|
Would you verify this info. The device was located at : Sat, Mar 16, 12:55:03p EDT Thanks |
|
According to https://www.unixtimestamp.com/, 1710608103 is "Sat Mar 16 2024 16:55:03 GMT+0000", which agrees with "Sat, Mar 16, 12:55:03p EDT", which also agrees with "2024-03-16 12:55:03" (assuming EDT, since it's naive.) Between the two, last_timestamp is unambiguous, whereas last_seen requires the assumption of HA's configured time zone, so I would recommend just including last_timestamp and not including last_seen. There's no need for both, at least not for the composite integration. If you want to include "2024-03-16 12:55:03", then please name the attribute anything else (otherwise composite will use it and not last_timestamp.) Of course, if the assumption always holds, then both are fine. Thanks! |
|
Done... Here you go with the last_timestamp attribute in the device_tracker entity. |
|
Thanks @gcobb321 - I will give this a try with the composite device_tracker and report back. Can you explain/confirm what is used for last_timestamp when GPS coordinates are from HA app rather than icloud? |
|
When the mobile app is used, the gps and time come from the mobile app device tracker and sensor entities. When the iCloud data is used, the gps and location time come from iCloud. |
|
Great thanks Gary. I was asking as there does not seem to be this info in the HA mobile app in the device tracker attributes. |
|
@pnbruckner I’ll mention that iCloud3 supports your app next time I get into the documentation |
|
The time comes from the last_changed our last_updated attribute which is not displayed I in the Developer tools >states screen it also comes from the sensor.last_update_trigger entity. Whichever is the latest. |
|
Hey Gary @gcobb321 - I wanted to thank you for your assistance with this. I think I have a workable solution now. FYI - linking to my recent post on the composite device tracker thread pnbruckner/ha-composite-tracker#66 (comment) |
@gcobb321 - big thanks for this work/integration. I have it working well overall but I am having some challenges getting reasonably accurate and consistent speed (for driving) using GPS data - I know this is challenging.
I have been chatting away with the guy who has created an integration that combines device trackers into one composite and uses this data to estimate speed. However, it all really depends on having the GPS fix time and timezone for all sensors pulling gps data - see link below.
pnbruckner/ha-composite-tracker#66
So what we need to move forward is:
I'm hoping you can help me take this forward.
The text was updated successfully, but these errors were encountered: