Snapshots for Facial Recognition

[bachdinhnhan]

Feature Request / Seeking advices

Description

One use case is create a facial recognition system to learn/recognize every family/relative members. From that the system can do many useful things:

1. Automatically disarm whenever a recognized person at home.
2. Send warning messages (through camera audio, doorbell) to strangers who approaching the home.
3. Send "says hi" voice audio to specific recognized people, for example: Welcome John arrives home.

Solution

1. Capture snapshots from video streaming whenever a motion detected.
2. Pipe snapshots through facial recognition models and list out all recognized faces to users/admins through a web portal.
3. Let users/admins label those faces.
4. Once recognized faces labeled, the system can decide what to do as above use-case.

Comment

I'm about to implement such system and stuck at step 1 in solution section. Please give me some advices how to extract snapshots from video streaming.

Code to extract images from stream

const ringRtpOptions = await sipSession.start({ output: [ '-f', 'image2', '-r', '1/3', path.join(outputDirectory,${now}-%02d.jpeg) ], }),

Thank you

[dgreif]

The streaming is set up to take any ffmpeg arguments that you would like to supply. I'm not an ffmpeg expert, so I can't say the best way to fetch snapshots from the live stream.

If you want to get the snapshots using the more traditional approach, you can use camera.getSnapshot(), although there are some limitations noted here: https://github.com/dgreif/ring/wiki/Snapshot-Limitations.

Sounds like a fun experiment you are working on and I wish you luck.

[kkuehlz]

Hi @bachdinhnhan,

Funny enough, I just did a similar project to this, albeit a bit simpler than what you want. I wanted to save videos from my ring only while a person is in the frame.

For facial recognition, you're going to need OpenCV. The essence of your ffmpeg command is correct. You either use image2 or image2pipe to transcode the HEVC stream to images that OpenCV models work with. image2 will write files to a directory, whereas image2pipe can write to any pipe such as stdout. I found image2 difficult for this task. You will need to watch for the files as ffmpeg creates, re-order them, only then sending those frames to OpenCV for processing. That's why I added #1386 so a JS program can hook into the output of image2pipe and get an in memory image stream.

I chose to transcode my stream to PNGs since the format is easy to work with. I wrote a little PNG unpacker that you can use to separate individual PNGs from a stream. Then, OpenCV can read in that raw data.

So, I have about 90% of the problem solved for you, with code. However, I only needed an object detection model. But you will need a facial recognition model fine-tuned on the members of your household. As a side note, you'll want a to choose a model than can infer in real-time. Because I noticed that if detection takes too long ffmpeg will fall behind the stream and start dropping packets. Everything will become a distorted mess.

Best of luck
-Kevin