Rosetta Drone and AI

[stefansommer]

In the rosettadrone_mini2 repository, we have had some discussions regarding interfacing machine learning/AI algorithms with rosetta drone m4xw#19 Still in an early stage, but we thought that it could make sense to continue the discussion in the upstream repo as people are looking into it here and it may be of general interest.

From my side: We (four cs students and I) are looking into monocular depth estimation using neural networks (convolutional and vision transformers). I view this as a first step in more actual autonomous control. Next could e.g. be direct prediction of flight paths from camera input similar to https://www.science.org/doi/10.1126/scirobotics.abg5810 We are currently investigating this from different perspectives: convolutional vs. ViT, supervising with simulated data and transfer learning (between data sets or simulated to real data), unsupervised learning (consistency of depth prediction when camera position/orientation changes, downscaled inference networks for running on device, etc.

We are currently using Rosetta Drone and dronekit to get telemetry data and then coupling this with saved video streams for training and evaluating models offline. It would be great to get this to work online on the UDP videostream, but so far we haven't managed to get the streaming to an external computer to work (it is a little while since we tested it, it may be that it works now). The depth estimation networks do work in real time when tested on e.g. a tello drone.

This was just to start the discussion. It would be great to hear what you are working on, other projects, or perhaps ideas to improve the above projects.

[kripper]

Hi @stefansommer,

Welcome to RossetaDrone. We are on the same page.

1. I just released for you my RawVideoSender plugin, which I use to send raw frames with no video re-encoding, no video buffer latencies and maximum quality (7693a41). You will need to enable this plugin (kripper@c234b1c). To receive the frames check this.
   But if you want to stream video, please check this.

2. Our friend Andre van Calster (@avncalst) did some nice AI scritps time ago.
   Check his videos here: https://www.youtube.com/watch?v=fEGiTKeYNMw
   And here is his work: https://github.com/avncalst/drone_cnn/tree/master/rosetta

3. I'm currently working on Vision Landing:
   https://github.com/kripper/vision-landing-2
   One problem we both will have to address is described here: EKF for vision landing #132

4. Also check in the issues section. There is a lot of information about using DroneKit with Rosetta.

[stefansommer]

Excellent, thank you! We will try both streaming methods. I agree about #132. In principle, the depth estimation could give a notion of height but how precise it will be is to be evaluated and point 2. will apply, thus the need for EKF

[OMAHDrone]

Dear Prof.Stefan,
May you help us how to setup dronekit + dji mini 2 with video streaming and control?
I tried to use rossetadrone3 app but l could not connect with dronekit nor l could control the dji mini 2 using rosettadrone3 app.
Thank you very much for your help
My pleasure

[kripper]

Excellent, thank you! We will try both streaming methods. I agree about #132. In principle, the depth estimation could give a notion of height but how precise it will be is to be evaluated and point 2. will apply, thus the need for EKF

Hi @stefansommer, I'm currently doing successful tests with vision based precision landing.
I just added the timestamp and the yaw to the RawVideoStreamer packets to deal with video latency.
For position estimation (lat/lng), the flight controller data is useless. Only yaw and altitude are usefull. For the rest, I'm using computer vision:

Please let me know about your progress.
If you have a github repo, I may also collaborate.

I have also been developing a framework for measuring the drone dynamics using the pose estimation as a reference. It's like a laboratory (an environment for testing algorithms and dynamics) that executes a series of tests to measures latencies, effective velocities, stability, generates graphs in real time, etc.

[kripper]

Please note that the EKF is needed mostly for dealing with the delay between the time you take the image and the time you finished computing and are able use these estimations (always too late).
So instead of using computer vision calculations and positioning sensors (which also have some a delay), you probably want to use a model to 1) estimate the current position (based on the EKF model) and 2) predict the future position (to be able to react on time especially when dealing with moving objects).

[kripper]

My first landing test:

[OMAHDrone]

Thank you very much kripper for the video and it is great
May you help us with the setup. We need only the minimum dronekit code that will receive the video feed and do some very basic control. We are working on deep reinforcement learning project using dji mini 2. May you help us with setup. We can collaborate with you and update you.
Thank you very much again for your help and time

[kripper]

Thank you very much kripper for the video and it is great May you help us with the setup. We need only the minimum dronekit code that will receive the video feed and do some very basic control. We are working on deep reinforcement learning project using dji mini 2. May you help us with setup. We can collaborate with you and update you. Thank you very much again for your help and time

Yes. I will help you both with your AI projects.
I gave the instructions in my first comment here, but I will create a detailed setup guide for you and release my latest versions.

[stefansommer]

very nice video! Great to hear about the progress, and the dynamics testing setup.
So far, we have mostly worked on the different neural network designs for depth estimation, trained on existing datasets and simulated data. No so much progress yet running it on live data.
@OMAHDrone , do you also have problems with the dronekit connection? For us, video has been an issue, but dronekit has worked relatively well.

[OMAHDrone]

Thank you very much kripper and Prof.Stefan
Yes l have problems with dronekit connection. I could not mange to communicate with the python dronekit and dji mini2
May you help us Prof.Stefan with the following regrading dji mini2:
1- We need the rosettadrone app you use to communicate with dji mini2.
2- We need the basic dronekit python code you used to receive the video -if possible - and do some basic motion control
We can collaborate with you Prof.Stefan and kripper and make our projects open source. I have to submit my project in June. That is why I am in hurry to quickly finish the setup and develop the AI network because of time constrains. Basically what l am trying to do is to develop an AI agent that will enable dji mini2 to track and land over a moving car using deep reinforcement learning.
The basic dronekit python code that we need is like this:
.....
takeoff
While true:
receive drone image
get drone states like speed and altitude (if possible)
send these data to AI agent
let AI agent generate control signals
send control signals back to drone
Repeat until mission is complete
then land
...
We are ready to collaborate with you kripper and Prof.Stefan and make our projects open source to help the community and researchers
We are waiting for any instructions form you to start
Thank you very much again for your help and time
My pleasure

[kripper]

Here are the instructions (draft):
https://docs.google.com/document/d/1wR56Mvd2OsT82kxMOD7f_3-c6yJGBtLaT8LuVrmC5wI/edit#heading=h.q9mp6yjsfxe5

[OMAHDrone]

Thank you very much kripper for the instructions
I am currently reading it and l will reply back after testing
Thank you very much for your help and support
My pleasure

[tr0043t]

Android MSDK 5.3.0 with Mini 3/Pro support is now up.

https://developer.dji.com/

[kripper]

the video is very slow

What is slow? It's configured for 15 fps and it's uncompressed. You can increase it, but how much fps is your algorithm going to process?

glitches, sometimes with color and sometimes gray

You are omitting 2048 bytes and we don't know what format you are receiving.
Or you didn't place a fan in front and it's overheating.

3- The yaw reading is always zero. But when l change to drone mode, it updates.

Test mode doesn't send real telemetry.

But I can not start the app in Drone mode. I have first to start the app in Test mode and then go back to the main menu and try several times until Drone mode is activated.

Maybe a problem with the cable or your smartphone or something that prevents proper detection.

in some cases - I open Drone mode, the phone screen starts to rotate as if it will open the Drone mode but it redo the rotation and returns back to the main menu.

Maybe the activity is finishing because the flight controller is null (not correctly detected).
Check the logs. Use breakpoints.

You should also first test and comment this:
#131

And for reporting Issues, create issues, one issue per issue.

[kripper]

But I can not start the app in Drone mode

Issue created here: #152

[kripper]

Ok. I'm hidding this discussion. Make sure to report issues correctly.