Crazy speed data - like 2,200km/h #66
Comments
|
It does have some minimal filtering. E.g., if two data points are too close together in time (3 seconds for the same input, and 9 seconds for different inputs), it will not update the speed. The problem may be the types of trackers you are using. Many do not have an attribute that specifies when the location data actually changed (e.g., Can you share the attributes of the two types of trackers? Obviously, obfuscate any sensitive data. |
|
Appreciate the help. I am getting so close to my overall project goal... Accurate speed is my last hurdle. icloud3 tracker: and the HA app device tracker |
|
Are those the state attributes of the The HA app tracker does not seem to have any sort of location timestamp. That is a shame, but not unexpected. I've tried to convince the powers-that-be how important such an attribute is, but my arguments have fallen on deaf ears. For the icloud3 input, the opposite seems to be the case. There are several timestamps. Any idea which one, if any, indicates when the GPS fix was taken? If I had to guess, I'd say Maybe you could run the following Jinja template for each of the trackers and post the results here: {% set state = states.device_tracker.XYZ %}
{{ '{:20}: '.format('last_updated') ~ state.last_updated }}
{% for attr, val in state.attributes.items() -%}
{{ '{:20}: '.format(attr) ~ val }}
{% endfor %} |
|
Thanks for this. I will report back later today. Maybe I could use another app/service with the nevessary gps attributes? the other thing is that icloud3 uses the gps data from the HA app somehow. So this might explain the issue |
|
I have run the Jinja code and it looks like both do have a last_updated timestamp. Not sure why this attribute was not showing in Developer tools for the HA app entity. Must be a hidden attribute or something.
Here is the output for the HA tracker using the jinja above, and for icloud3 |
|
I also have some new data points for us to look at to investigate. Here are the updates that were pushed from HA to my traccar server. Lots of data points look more or less correct but I have highlighted the rows that are rubbish. I can compare to the native traccar client that I am also using for testing but is not a good long term solution as it is a battery hog. Files uploaded are ready to assist! |
|
I have found one error in my work. Traccar is expecting data to be passed in kts and I am using kmh. I think this will partially fix the issue but It is still not right where I have flagged the data in the xls. Attaching the traccar server log - this is where / how I found the kn v. kmh issue |
Thanks for doing that! I guess I was surprised by those icloud3 attributes, and thought you were originally showing some sort of report or other info and not simply the entity's attributes. Very unexpectedly, that entity has three very unusual and strange attributes whose names and values are a bunch of "-" characters. Weird.
The
It is not directly set by the integration, but rather by the system when the integration updates the entity. It records when the entity is updated. It is not a direct indication of when the GPS values were obtained at the source (i.e., in the cell phone.) Those two times (when the GPS values are obtained in the phone, and when the HA entity is updated with those values) can be very different, depending on how those values are transferred to the HA entity. There could be seconds between them. I've even seen with some tracker entities where GPS values are received out of order due to network delays, etc. This is why it is very important (when trying to combine multiple GPS sources and/or trying to calculate speed) to have the data timestamped with when the data was actually obtained at the source. When this timestamp information is not available, the composite integration does the only thing it can and will use that So, again, the HA tracker has no timestamp for the GPS data. And I have no idea how much delay there may be between when the GPS values are obtained at the source and when the corresponding HA entity is updated, if it's consistent or if there's a lot of jitter, if data can be received out of order, etc. Here the only thing the composite integration can do is to use the And looking at the icloud3 entity, there are multiple time values, where If you could check with the icloud3 author as to which attribute, if any, records the actual time when the GPS fix was taken, and what time zone it corresponds to, then I could enhance composite to use that data. |
|
Thanks for the detailed and very helpful explanation. I will follow up with the icloud3 author and also make a request to HA core team to add the fix time info to the iOS companion app. Just thinking out loud but I wonder if the HA companion app for android has this attribute |
You can ask, but they'll likely not do it. Even if they add the data, they won't add it as an attribute. I tried to modify the gpslogger integration to do this and it was rejected. They wanted the data in a separate sensor, which is basically useless. |
|
Well I will try and ask them. What is the issue with a separate sensor? Is it related to having gps coordinates and fix time synchronous? I can’t believe how hard it is to use gps to estimate speed. I thought there would be plenty of integrations. How is life360 doing it? |
|
I think I have a possible workaround for my use case. I just need to identify rubbish speed data then not push this to my Traccar database/ststem. Just need to work out how to identify bogus data on the fly. I think the issue is related to when the composite integration gets twitchy gps data very close together |
|
This is data that i am pushing with an automation and REST API from HA to Traccar. While I am sleeping with phone on charge - so not moving. All I need to do is just identify and optionally not push these updates to traccar. Any ideas?
|
|
Yes, in your case, it seems you need to work around the lack of data which is generating incorrect results. The better solution would be to obtain the data you need. For the HA app, I think that is a non-starter. For icloud3, I suspect it's possible. I started looking at the custom integration code, but it's pretty big and will take a bit to get through it enough to try and figure this out myself. The alternative is to use better data sources. I myself use Google Maps (my own custom version of it), as well as gpslogger (yep, my own custom version of it, too.) You could use either or both of those if you like. The nice thing about Google Maps is, people often already have that on their phones, even Apple phones, and either already have location sharing set up, too, or can easily add it. |
|
righto. I will see what I can find out about about icloud3 and I will report back. I do have google maps on my phone so I will try that integration and also gpslogger too. I just went to take my son to the dentist then to school and I was driving at 290km/h at one point. LOL |
|
I wonder if the outliers are due to the interaction of the two trackers, or if it's mostly from just one. Might be interesting to add two more composite entities, where each uses just one of the two "input" trackers. Then you'd get speed readings for each individually. If those are reasonable, then it's the interaction (which is probably due to lack of proper timestamps for the location data.) |
|
I read the icloud3 code some more, and it's quite, um, different. It seems to like to do things the hard way. Anyway, as best as I can tell, the Do any of your phones that use the icloud3 integration ever go in a different time zone? It would be interesting to see if I'm reading the code right. |
|
Wait. Just realized, if the times are always in HA's time zone, then that should work fine. I'll add support for it. I'll try to check with the author to see if my interpretation is correct. |
|
Thanks for such awesome help with this. Above and beyond for sure. I will try out your google maps integration to see how that works for me. Although probably only for testing. My project goal is to be able to keep an eye on my son's driving as he is getting to the age where he will be driving and in cars with friends. The snag is the traccar client app is a total battery hog but the location and speed data is OK. Right now I want to use a combo of GPS device trackers that give me high location accuracy & frequency that do not drain the battery on my phone. Once I get everything working for me then I will setup my son in the same way. |
Nope. All local in here in NSW Australia at the moment. Only on my phone currently and I installed it a few days ago so its all quite new. |
I have been thinking something similar myself. One feature of the icloud3 tracker is that is already interacting with the HA tracker. I think it uses HA tracker together with icloud to handle location and maybe this is part of the issue. I will setup a composite just with HA tracker and see how that works but we already know that the HA tracker is missing an important date/time attribute. GPSlogger is android only so that's not an option. I have google setup but can't see any location data in the entity so something went wrong with my setup/cookie stuff i think. |
Are you using my custom version, and latest version? If so, you should enable debug for it. Then, on the log page, type "custom_components.google_maps" in the search box and click LOAD FULL LOGS. There should be some messages that can give some clues as to what's going on. |
|
I am using your google integration and installation / cookie went fine I think. below are the only logs I can find - with debug on. Looks OK to me. I was expecting to see GPS info in the attributes but there is nothing.
|
|
So just working on the HA side of things. This is data from my latest test drive in the car. Circled in yellow is actual driving. The rest is nonsense. What this is telling me (for this trip/drive) is overall not bad. no crazy spikes (might be lucky) So i'm hoping that with a bit of your help we can create a fairly good speed sensor with icloud and HA app. I'm going to create a sensor that uses the average for HA and icloud and use this to push into traccar server - even better would be to do this and drop crap data as this will skew the data.
|
|
Just one more update. Went for another drive and it is the icloud3 tracker that keeps generating crazy speed values so I think this is where the root of the issue is that I need to find/fix - or just stop using the icloud3 integration. After all, I am only using it as I thought it would be better and/or combine well with the HA tracker. It would be good to get the google tracker working then I have something else to compare with... |
|
I'm a bit confused about the Traccar data. You keep referring to pushing data to it, but you're also getting data from it??? Is it providing independent GPS data, or is it somehow data derived from the other trackers? Note that the speed sensor only updates when it gets valid input that represents a new position. If the input(s) aren't changing, or the inputs are being rejected (for various reasons, typically they contain older data than the composite has already seen), then the speed sensor output won't change. That could be why the iCloud spike lasts a while. Also, it won't "drop" due to lack of new data. Regarding the Google Maps integration, that's very strange. Which account did you add to the integration? Your personal account that is associated with your phone? Given the name of the tracker entity, it does appear you're using the "Account Holder" location. If you're using a different Google account (e.g., one you created just for use with this HA integration), then you should not use the "account holder" tracker, and you need to share your personal Google account's location with the account you add to the HA integration. Hopefully, that makes sense. |
There is the traccar server and also a client app for ios. In HA there are integrations for both the server and client. To help with this testing, I have the traccar iOS client app (a GPS tracker) --> Traccar Client in HA -this gives me good test/baseline data for speed. In addition, I am working with you on other options for GPS tracking for my iphone. In all cases (traccar client or HA/icloud/google) I need to push the data from HA to the traccar server. Pushing the data into the traccar server is all working fine. I have this part working great it can even test speed v. location based speed limit and notifications when speeding. Guess what? Driving at 2,000 km/h triggers a speeding notification. :-) |
|
I am just using my personal google account in 'account holder' mode. For testing this is all I need I think. Issue is I'm not getting any data through this integration... Yet... Not sure what I have done wrong. Your instructions were quite clear and everything seemed to work fine - just not getting data into HA. |
|
When you get no location data with the account holder tracker, that generally means it's not linked to an actual device. Do you have Google Maps installed on your iPhone and logged into the same Google account? If you open https://www.google.com/maps in a browser, logged into that same account, can you see your device location? |
|
set to query every minute and this is what I see in the logs every minute. |
and this is why icloud3 setup with HA together is providing the weird data. I think there is an option to setup icloud3 without linking it with HA mobile app. I will try that next |
|
So I have changed how icloud3 works so that it does not try to use HA mobile app data. So now I think we have two separate and hopefully clean data sources for location/speed (HA mobile app and icloud3) I will monitor both of them today and see what I get in terms of consistency etc. |
|
So I just tried the extra debug line in my test system and I'm seeing the same thing. Unfortunately, the raw data has line feeds and HA's log page only shows the first line of log messages. Give me a sec and I'll have you try something slightly different (after I try it myself.) |
|
all good. I'm so sorry to be causing you all this work / effort - and it is the weekend too. Is there a way I can say thanks somehow? |
|
Ok, try this instead (with the line you added removed, and a different line entered a bit further down): def get_new_data(self) -> None:
"""Get new data from Google server."""
try:
resp = self._session.get(_URL, params=_PARAMS, verify=True)
resp.raise_for_status()
except (RequestException, MaxRetryError) as err:
raise RequestFailed(f"{err.__class__.__name__}: {err}") from err
raw_data = resp.text
try:
self._data = json.loads(raw_data[5:])
except (IndexError, json.JSONDecodeError) as err:
raise InvalidData(f"Could not parse: {raw_data}") from err
_LOGGER.debug("%s: Parsed data: %s", self._account_email, self._data) # <<<<< ADD THIS LINE
if not isinstance(self._data, Sequence):
raise InvalidData(f"Expected a Sequence, got: {self._data}")
try:
if self._data[6] == "GgA=":
self._dump_cookies()
_LOGGER.debug("%s: Parsed data: %s", self._account_email, self._data)
raise InvalidCookies("Invalid session indicated")
except IndexError:
raise InvalidData(f"Unexpected parsed data: {self._data}") from NoneThe added line is actually a copy of the line just like it a few lines down. Just make sure it's indented correctly. |
|
so remove the other debug code and use this new one instead? |
|
No problem. I'm just sitting around today watching a few Blackhawks games I recorded earlier (ones that they actually won, which doesn't happen very often anymore! 😆 ) |
|
just to wind you up I am going to say what are the blackhawks :-) |
|
NHL hockey team in Chicago. |
|
here we go... |
|
Ok, thanks, that's more like it. Let me analyze the data... |
|
Well, the data looks good for the "account holder", and there is no location data from any shared accounts. What is the state of the device_tracker entity? |
|
Looks like last_seen is 2024-03-16 23:11:00.720000+11:00. I don't think there's anything wrong with the integration. For whatever reason, google is not being updated with new location data from your phone. Try opening Google Maps on your phone, then going to Location Sharing. Sometimes if things are messed up it will give you some sort of option to get it fixed. I've seen this happen a few times (mostly on other people's phones), but not very often, and it hasn't happened very recently, so I don't remember the details |
|
Entity details below. I have checked in location sharing for my google account and there was nothing that I could see. My google account is very old (pre google maps) so maybe there is something with this old account. I will investigate further. Maybe create a HA shared account and then share my personal with the HA account.
|
Probably worth a try. But if Google isn't getting new data from your phone, I doubt sharing it to another will make a difference. But who knows! |
|
so I have created a shared gmail account and shared my personal account with it. Now I am getting these in the logs. Are we onto something here? |
|
I'm not sure what you mean by "I have created a shared gmail account". In any case, there is still just data for the "account holder" (i.e., for the account that you added to the Google Maps integration), no data for any accounts shared with that account, but now last_seen is 2024-03-17 10:00:45.502000+11:00. |
|
I created a new gmail account for sharing. Then I shared my location from my personal account to the shared account. Then I added the shared gmail account to HA following the normal instructions to get the cookie file. Going for a drive to see if this does anything interesting now... |
|
FWIW, those both contain the same data (well, at least taking into account the "account holder" one is missing some attributes, as the docs explain), and last seen is 2024-03-17 10:41:24.510000+11:00, which was just before your post. 🤞 |
|
Hey Phil (@pnbruckner) - I have some good news and progress.
First chart is traccar client, Second is with the composite overlayed. Third is HA, icloud and google separately. Let me know what you think but I reckon we have a reasonable solution here. Just need to monitor battery drain on my iphone. Chart 1 - Traccar client app running at 2min interval (gold standard / baseline) Chart 2 - Composite overlayed on Traccar - (this composite is what I will use instead of traccar for my speeding detection) Chart 3 - the three parts to the composite (HA, icloud and google) |
That's surprising, but as long as it's working, then that's all that matters. Yay!
Sounds good. I'm going to back out adding last_located as an attribute composite looks for, but I'll keep the change that accepts a date/time string. I'm still considering adding an option for the user to specify the attribute name (in case some other entity has a useful timestamp in an attribute with another name.) FWIW, thinking about the template idea some more made me realize it's kind of problematic. I won't go into the details here, but if I do add this feature, it will only evaluate when the associated input changes state, and will provide that new state as a variable to use in the template, allowing it to extract the info from any attribute, as well as to convert it as necessary. I'm just not sure how much use it would get, if any, so not sure it's worth the effort of adding. |
|
I don't mean to jump in but have a question:
A couple of things with the Mobile App:
A clarification a couple of things in iCloud3:
Thanks for the info on the Mobile App last_update/_change times. That explained a few things I have noticed but never understood. It seems that the Mobile App update/change times are either fairly close to the FamShr location times or very far off. Or iCloud3 handles trigger changes before a FamShr locate is scheduled so it does not impact the overall tracking. |
|
Big thanks from me to you both for helping me with this. I will run my driving/speedlimit tracking project for a week or so and see how it works in real world conditions. I have created a feature request for HA mobile app to include the fix time as an attribute. If we had this (and icloud3 used this info) I think I could go back to including the HA mobile app GPS data integrated with icloud3. |
I'm not sure of the context of where this information would be used. In the HA integration? Or somewhere on the phone? FWIW, the entities created by the Google Maps integration are no different from any other entity in HA in that whenever they change, they cause a
Agreed, very inaccurate location updates can cause problems. FWIW, the Google Maps integration (the standard, built-in one, as well as my enhanced custom version of it) does the same thing in that it has a configurable "max accuracy" option, and any updates that are less accurate (i.e., a bigger value) than that will be rejected/ignored.
That is a good question. If it did send a timestamp for the location data to the HA integration, it would be very helpful if it put that information in an attribute of the device_tracker entity. Unfortunately, if there isn't one already (and there doesn't seem to be), if it was added, they would probably put it in a separate sensor entity instead of an attribute of the device_tracker, which, IMHO, is almost useless.
Sorry, I'm not following you at all here. Where exactly does the value of |
Google maps app might provide a data source for iPhone users not in the Family Sharing List and not using the Mobile App to be located again under the iCloud3 umbrella. As an aside, I have users were mom and dad have iPhones but their kids do not. The kids have Samsung phones or other Android phones the Mobile App installed. They are being tracked using only the Mobile App as a data source without any problems. |
I wasn't questioning the format of the data, or how to interpret it, but the source of the data. When you said, "The last_timestamp is still the HA server timestamp", I wasn't sure if you meant the source of the data was now the state object's last_updated field, or if it was still ultimately the time the location data was read on the device. If the latter, then no problem.
That is already possible with the Google Maps integration. It allows one or more Google accounts to be registered. For each, it can show location data for the phone associated with that account, and it can show location data for any other Google accounts that have shared their location with the registered account. So, I'm still not understanding what added value there would be for iCloud3 to use that data that is already available in HA (as device_tracker entities) when using the Google Maps integration. FWIW, it works with both Android and Apple devices. |
|
|
Closing since there hasn't been an update in a couple of months. Not really sure what composite could/should do to consistently & robustly filter/reject large spikes in speed calculation. Feel free to reopen with new data/info. |
Hey @pnbruckner - First of all thanks for this integration.
I have just installed this and I am using it to combine 2 GPS device trackers that both work well (HA companion app and icloud3 integration).
The main reason for using this is to have a speed sensor for my iphone. Generally it works ok but I am getting the occasional crazy speed - like over 2000 km/h :-)
Can you help me to trouble shoot this or is it something I will have to live with (or find a way to deal with the bogus data)?
I need accurate speed data as I am trying to create an open source and data privacy based system for monitoring my teenage son's driving. Basically life 360 but home grown.
The text was updated successfully, but these errors were encountered: