While restarting multiple times device may stop streaming

[bmegli]

Occasionally it's not possible to initialize communication with the device and only plugging/unplugging helps.

This generally happens when restarting communication with the device multiple times.

Cause to be investigated.

[Pyrojambo]

Has there been any update on this bug?
I've encountered it and in the setup I will be using it is very inconvenient to unplug the usb cable

[bmegli]

Hi @Pyrojambo,

Yes, the problem exists. I will look into it but I can't promise when.

The problem can be one of (in decreasing probability):

1. vmu931 library implementation (likely, new software)
2. vmu931 device firmware/hardware (likely, new device)
3. OS level (very unlikely)

Checking 1 and 2 is actually rather easy:

• run until the problem happens
• check with
  • OS tty tools (picocom, minicom, ...) if device responds/sends anything
  • or other software for vmu931
  • or oscilloscope

If device is still sending data it is the library problem, otherwise it is likely vmu firmware/hardware who is the villain. In later case we have to contact Variense.

It is one hour experiment, an hour I don't have currently ;-) But I will look into it (no promises when).

[frugaltech]

Hi!

I am also using - well trying to use - the VMU931 device but the problem you describe is making the thing almost useless in my case. I made a simple driver with an integrated test that demonstrates the problem nicely here: https://github.com/frugaltech/VMU931Driver

It seems to me the problem has to be in the firmware somwhere. I initially used PyVMU to test out the device and I thought the problems arose from that package, since there were a few things that seemed a little off to me in that driver. But - as it turns out, my driver (and yours) have the save issues. Pyserial is definitely not to blame - I use that all over the place with no problems at all ever. The OS is not the problem either - I have tried both Ubuntu, Raspian and Windows with the same results.

What are your thoughts on this? Have you had any luck contacting Variense support? I haven't had any response from them yet, unfortunately.

[bmegli]

Hi @frugaltech,

I made a simple driver with an integrated test that demonstrates the problem nicely here: https://github.com/frugaltech/VMU931Driver

Looks ok at first sight and generally should work.

Some minor quirks of vmu931 that I noticed:

• it doesn't always react to commands, sometimes one has to repeat the command until it works
• Variense vmu_utilities mentions that one should wait 1 ms between each character sent to VMU
• the reaction to the command is not always immediate (I am not entirely sure of that one, it has been some time since I wrote the library)

You might have a problem if vmu ignored the stream Euler command. But since vmu streams Euler by default this would not happen in your code. If vmu ignored toggle quaternion you might get repeatedly serial connect/disconnect in get_message on unexpected quaternion data. Nonetheless, even in this scenario it should work.

I initially used PyVMU to test out the device and I thought the problems arose from that package, since there were a few things that seemed a little off to me in that driver.

If I understand correctly, we have 3 independent implementations that suffer from the same problem:

• your VMU931Driver
• mine vmu931
• PyVMU

Did you experience the same problem in PyVMU? Do I understand correctly?

The OS is not the problem either - I have tried both Ubuntu, Raspian and Windows with the same results.

Adding mine to the list:

• Ubuntu
• Rasbian
• Windows
• Debian (on arm)
• Ubuntu (on x86)

It seems to me the problem has to be in the firmware somwhere

I agree.

Now, that we have:

• 3 (?) independent implementations
• tested on 4 OSes
• on 2 architectures

It seems very likely.

What are your thoughts on this? Have you had any luck contacting Variense support? I haven't had any response from them yet, unfortunately.

I contacted Variense support on different issue (calibration and drift) and got response in the past (with undocumented magnetometer calibration procedure).

Yes, we need to contact Variense.

[frugaltech]

Thanks for the reply - it's nice to play some ball with these issues. Wrt. the PyVMU driver, then yes: I had the same issues with that implementation. Also, the device does not seem to ignore commands I send to it. If I configure it to only stream gyro data, it does so and I can reliably capture data - until I can't. I also tried the other streaming modes and while I sometimes get spurious packets that don't match the configuration; that behaviour does not seem to cause too much trouble. The lock ups I see apparently happen spontaneously in the get_message function in my driver. I just wrote the support team at Variense pointing them to my driver so I guess we'll see...

[frugaltech]

Hi again! I also made a different driver using the USB CDC stuff directly rather than via the emulated serial port. It is available here: CDC driver - but ultimately, it makes no difference. The problems are the same. I did manage to get a response from Variense which was this:

Hi Lars,

We have received your support inquiry about the sensor VMU931.

We have used internally only our C API under windows and Linux and we haven't faced any issues like the ones you noted in your email. Have you tried the VMU utilities or the VMUReader to test if the issue is with your Python script? According to your description of the problem, it's may be the buffer which is getting overloaded.

Please don't hesitate to contact me if you need further information.

Best Regards,

Variense, Inc

I haven't had a chance to pursue this further but I will at some point (soonish). In the meantime, I have found a new device that works without any hiccups - Yocto 3D v2
It comes with apx. 700 pages of documentation! Leave it to Switzerland to (over)do things right.

[Pyrojambo]

I contacted Variense about this again recently and they said it might be OS input buffer overflow, and suggested clearing the buffer after every read command.
I've tried this using tcflush from termios.h and I've not came across the issue since but not been able to properly test it so I may have just been lucky so far

[bmegli]

Hi @frugaltech, @Pyrojambo,

[...] it's may be the buffer which is getting overloaded.

[... ]OS input buffer overflow, and suggested clearing the buffer after every read command.

Interesting. I don't think it should "hang" the device but if you say that it helps it's worth trying, at least for the sake of investigation.

Clearing the buffer after every read means that one may miss some packets from the device.
I see that I am using if(tcflush(v->fd, TCIOFLUSH) < 0) during initialization (not after every read).

Also if buffer overflow would be causing trouble, it should be reproducible - just leave the device streaming data.

[bmegli]

@frugaltech

I haven't had a chance to pursue this further but I will at some point (soonish)

Huh, I also thought so but no time so far.

In the meantime, I have found a new device that works without any hiccups - Yocto 3D v2

Thanks for the info. I like the idea that you may separate magnetometer from accelerometer and gyroscope. I have some trouble in my projects to find the right place for IMU.

Accelerometer should be near the expected center of rotation to minimize rotational acceleration misinterpreted as linear acceleration. This usually means middle of the device between engines and near high current wires.

Magnetometer should be away from engines, high current wires, etc.

I am using Ultimate Sensor Fusion Solution. It is I2C, not USB communication, rather for integration with microcontroller. One may precisely timestamp the readings on interrupt.
I haven't had a chance to put it into real-world tests yet but works promising in synthetic tests on desk.

Another thing I am considering is dropping magnetometer completely. This means drift over longer period of time but no noisy magnetometer readings over short period of time.

[JosephRedfern]

Hi Guys,

I'm the guy that wrote PyVMU -- Interesting to read of your experiences with the VMU931 and reliability issues. Also cool to see your Python implementation @frugaltech, I hadn't considered this approach. It's a shame that it doesn't help things :(

Was just wondering if either of you had had any further success with the VMU931, or have received any further information from Variense. Specifically, whether there's any kind of internal buffering on the device itself -- my current project needs the freshest possible data, so I flush input buffers in PySerial before reading the device. This often works, but is not reliable and still returns delayed data ~50% of the time.

Cheers,
Joe

[bmegli]

Hi @JosephRedfern,

Was just wondering if either of you had had any further success with the VMU931, or have received any further information from Variense

Not in my case, but I haven't contacted Variense on this issue.

Specifically, whether there's any kind of internal buffering on the device itself -- my current project needs the freshest possible data, so I flush input buffers in PySerial before reading the device. This often works, but is not reliable and still returns delayed data ~50% of the time.

I generally experienced some kind of lag sometimes but of different character. Something like a break and large number of buffered data. But I didn't/don't attribute it to VMU931, this was part of larger stack of software. Maybe, who knows.

What order of buffering do you mean (e.g. in ms, if you can assess).
What OS are you on?

In general USB bandwidth is managed by the host (your PC). The transmission is made when the host allows, typically this happens at 1000 Hz (each 1 ms). There is additional delay between the data arriving and your process being awaken. What OS are you on?

So typically this means:

• 0 to 1 ms delay just from the way USB works
• possible additional delay before your process is awaken

If you are streaming quaternions/euler, you will get data at 200 Hz (each 5 ms).

Some ideas

Packet by packet reading

Taking a look at PyVMU

You are reading on packet by packet basis

Now if 1 + time_to_awake_your_process + your_packet_processing_time > 5 ms this will already buffer some packet from VMU on host side.

Possible solution: instead of reading packet by packet grab as much data as there is and parse through it to the last packet.

Other USB devices

If you have other USB devices transferring at the same time they may have higher priority (like isochronous and input devices)

low_latency

If you are on Linux you may try your luck with setting low_latency flag on your device

Just some ideas.

Kind regards

[JosephRedfern]

Hi @bmegli,

Thanks for your prompt and detailed response.

I generally experienced some kind of lag sometimes but of different character. Something like a break and large number of buffered data. But I didn't/don't attribute it to VMU931, this was part of larger stack of software. Maybe, who knows.

This may well be the case here.

What order of buffering do you mean (e.g. in ms, if you can assess).

I'm capturing Data from two devices -- the ZED Mini (specifically, it's IMU) and the Variense VMU931. The ZED Mini IMU has a sample rate of 800Hz, and you can read from it asynchronously (i.e. if you sample at >800Hz you just get duplicate packets).

My experimental setup is as follows: place ZED and VMU931 on the same surface. Wait for new packet (with a fresh timestamp) from the ZED, then immediately flush buffer and retrieve accelerometer packet from VMU931. I then log each packet. So over 2 seconds, at 800Hz, I get 1600 packets from each device. While the data is recording, I whack the table to generate an impulse. I'm then using convolution to figure out the offset between each signal.

Sometimes (like above), there is minimal delay/lag between the two readings.

Other times (again, as above) there is a delay. I flush the serial buffer before every read, but the delay seems to be either 0 or 1 packets, or 40-50 packets -- over ~30 runs of this experiment, I've not seen anything in the middle. 40-50 packets corresponds to ~50-60ms delay.

I've also repeated this, plotting timestamp on X axis rather than packet number -- but the same thing is observed, which is particularly curious... I use the arrival time of the first packet as the epoch (for each respective device), so perhaps an incorrect offset could be introduced there?

What OS are you on?

I've observed this under Debian Unstable (my main OS) and Ubuntu 18.04.

In general USB bandwidth is managed by the host (your PC). The transmission is made when the host allows, typically this happens at 1000 Hz (each 1 ms). There is additional delay between the data arriving and your process being awaken. What OS are you on?

So typically this means:

• 0 to 1 ms delay just from the way USB works

0 to 1ms delay would be totally absolutely acceptable.

• possible additional delay before your process is awaken

This could be it....

If you are streaming quaternions/euler, you will get data at 200 Hz (each 5 ms).

I'm only streaming Accelerometer packets, so this should be 1000Hz.

Some ideas

Packet by packet reading

Taking a look at PyVMU

You are reading on packet by packet basis

Now if 1 + time_to_awake_your_process + your_packet_processing_time > 5 ms this will already buffer some packet from VMU on host side.

Yes -- but it seems strange that there is 0-1 packet delay ~50% of the time, and a 40-50 packet delay the other 50% of the time. It doesn't seem consistent!

Possible solution: instead of reading packet by packet grab as much data as there is and parse through it to the last packet.
This could be a good idea, I'll look into it. I could put the retrieval/parsing code into a separate thread... would allow me to sample asynchronously, like you can with the ZED SDK.

Other USB devices

If you have other USB devices transferring at the same time they may have higher priority (like isochronous and input devices)
That could be an issue, yes... will try experimenting. Do you know of any way to set the USB priority? Would ensuring they were on different USB Controllers help here?

low_latency

If you are on Linux you may try your luck with setting low_latency flag on your device
Have tried this -- no obvious impact!

Thanks again for the help.

Joe

[bmegli]

Some more random thoughs

If you have other USB devices transferring at the same time they may have higher priority (like isochronous and input devices)

That could be an issue, yes... will try experimenting. Do you know of any way to set the USB priority? Would ensuring they were on different USB Controllers help here?

I think that ensuring that they are on different USB controllers should help.
Also if you are using only fraction of your USB bandwidth this should probably not matter.

The priority is decided by USB device class. The higher priorities are for example:

• interrupt devices like your mouse
• isochronous devices like video camera and USB sound card

I have no idea what class is VMU, this could be checked with lsusb.

E.g.

lsusb # note the bus and device number
lsusb -s bus:device_number -v #v for verbose

Look for Transfer Type (Interrupt, Bulk, Isochronous, ....)

[bmegli]

possible additional delay before your process is awaken

This could be it....

After talking to embedded Linux expert in my company the time to wake your process sleeping on IO may be reduced nearly to 0 if you have preemptive kernel and high priority process

Before diving into that I would just check how much time the process spends sleeping.
E.g. measure time before and after VMU931 blocking read and store the duration in table.

If the sleep time exceeds occasionally 1000 ms/800 Hz this may already buffer some packet on the host side.

Another thing is that I have no idea how to explain 40-50 packet delay.

[frugaltech]

Hi guys.

I haven't really made any real progress using the Variense device. I really liked it because of the very nice finish and its features but practically speaking, it has not really been worth it. I am now using Yoctopuce devices instead - they are absolutely issue free, very precise and almost delay-free. I might do a C/C++ test to see if that helps somehow. That could then be the start of a Python integration point, but I am not very optimistic that it will work, so ...

Lars