Using your hand to control your connected devices and applications.
WIP. Some parts non essential to what I wanted to demonstrate in this project, are not yet fully implemented (service messages exchanged between client and service, logging, debugging option, starting the service as a linux service, ...). Also the documentation describing how to configure Universal Hand Control on the client side is missing (mainly describing gestures, airzones and their parameters).
For now, a brief description of the files in this repository :
Implements Universal Hand Control on the server side. Waiting to be contacting by a client app via MQTT, then analyzing frames from the sensor, detecting the events that the client is interesred in, and sending the events to the client via MQTT.
Currently, started by :
python UniversalHandControlService.py
Implements Universal Hand Control on the client side. This is the code to import in your application code:
from UniversalHandControl import *
Called by UniversalHandControlService. This is where almost the hard and complex job is done (communicating with the sensor, analyzing the frames). A part of the frame analysis is an adaptation in python of the code from the Mediapipe Hands solution.
Some common classes and definitions
A class that manages a Realsense D4xx camera. Used by HandPose via the methods:
- next_frames() : that returns synchronized frames (currently only the color frame is directly used by HandPose). Depth information is indirectly queried via the 2 follwing methods;
- deproject() : takes a pixel position (x,y) in the color frame as argument and returns its (X,Y,Z) position in the 3D camera coordinate system. Used to get the location of the hand in space;
- project() : takes a (X,Y,Z) position in the 3D camera coordinate system as argument and returns its projection in the color image. Used to draw airzones in the image (for debugging).
Equivalent of the previous file for the BW1092, with the same methods. Temporary version. Will modified when aligned color and depth frames will be available in depthai library.
Text-to-speech function accessible to the client app via UniversalHandControl method 'say(sentence)'. TTS_win.py is the Windows version but currently does not work well.
Playing a sound (a wav file) is accessible to the client app via UniversalHandControl method 'play(wav_file').
Module with a simple and a double exponential filters. Useful in a client app for instance when you want to control the computer mouse with a pad airzone. As the coordinates of the hand can be noisy, applying a filter makes the mouse less jittery.
A basic example that demonstrates the airzones.
Switching on/off a light bulb via a gesture. Uses yeelight python package.
Controlling devices (light, TV, music speaker) with gestures. Uses soco (controlling Sonos), yeelight (controlling light), RM3 (controlling TV via RM3 mini controller) python packages.
Controlling devices (light, music speaker) with airzones. Uses soco (controlling Sonos), yeelight (controlling light).
Controlling the keyboard with gestures to play Tetris.
Controlling the mouse with pad and gestures to play Angry Birds.
Virtual synth with pad, slider and gestures.
The 2 models, the palm detection model and the hand landmark model, are Openvino IR models generated by PINTO0309. PINTO0309 converted himself the models from their original Tensorflow Lite version.