Skip to content

Latest commit

 

History

History
327 lines (298 loc) · 14.3 KB

README_EN.md

File metadata and controls

327 lines (298 loc) · 14.3 KB
Raw

[Japanese/English]

NARUTO-HandSignDetection

This is a model and sample program that detects the Naruto' hand sign using object detection.

　
Right figure：© NARUTO Episode 9 『Kakashi, Sharingan Warrior!』Masashi Kishimoto/Shueisha/Studio Pierrot
※The bounding box is not overlaid in the figure on the right because it may correspond to "modification" in Article 20 "Identity Preservation Right" of the Japanese Copyright Law1.

Title

Deep写輪眼(Sharingan)：Development of NARUTO's Hand Sign Recognition System using Object Detection YOLOX

Abstract

This repository publishes trained models and sample programs for recognizing the Naruto’s2 hand sign.

With the exception of some ninjutsu, the activation of 忍術(ninjutsu) requires a hand-signs.
In addition, because the characteristics of the property change appear in the hand-sign (Fire Style → Tiger's mark, Earth Style → Boar's mark, etc.)
If you can recognize the hand-sign quickly, you can gain an advantage in the battle between Shinobi. By using YOLOX-Nano, one of the deep learning object detection models, for hand-sign recognition,
The speed has been greatly improved compared to the Deep写輪眼(using EfficientDet-D0) that was created on a basis in the past.

Requirements

  • onnxruntime 1.10.0 or Later
  • OpenCV 3.4.2 or Later
  • Pillow 6.1.0 or Later (Only when running Ninjutsu_demo.py)
  • Tensorflow 2.3.0 or Later (Only when running SSD or EfficientDet, or only when merging post-processing into ONNX)

DataSet

About the dataset

The dataset is private(trained models are public)
※Follow Article 47-7 of the Copyright Act of Japan "Transfer of reproductions made due to restrictions on reproduction rights"3

In addition to the images I took and the anime images, I use naruto-hand-sign-dataset4.

Request

Since the dataset consists of images collected on the Internet and images taken by ourselves,
Depending on the background color and clothing, the detection accuracy may drop or false detection may occur.
It would be helpful if you could tell us the conditions that were falsely detected in the Issue.
If possible, it would be greatly appreciated if you could provide an image of the conditions for false detection(Rat-Boar, Mizunoe, Hand Claps).
At that time, the received image will be added to the training data set and used for retraining the model.

Kind of hand-sign

It corresponds to 14 kinds of hand-signs(Rat-Boar, Mizunoe, Hand Claps).

子(Ne/Rat) 丑(Ushi/Ox) 寅(Tora/Tiger) 卯(U/Hare)
辰(Tatsu/Dragon) 巳(Mi/Snake) 午(Uma/Horse) 未(Hitsuji/Ram)
申(Saru/Monkey) 酉(Tori/Bird) 戌(Inu/Dog) 亥(I/Boar)
壬(Mizunoe) 合掌(Gassho/Hand Claps) - -

Number of datasets

Total number：10026 sheets(Anime image：2651sheets)
Number of tagged sheets：7098sheets
Number of untagged sheets：2928sheets
Number of annotation boxes：8941 boxes
　

Trained Model

The trained model is published under the 'model' directory. * Move the old version model to the "_legacy" directory.

  • YOLOX-Nano

Directory

│  simple_demo.py
│  Ninjutsu_demo.py
│
├─model
│  └─yolox
│      │ yolox_nano.onnx
│      └─yolox_onnx.py
│
├─post_process_gen_tools
│      │ convert_script.sh
│      │ make_box_gather_nd.py
│      │ make_boxes_scores.py
│      │ make_cxcywh_y1x1y2x2.py
│      │ make_final_batch_nums_final_class_nums_final_box_nums.py
│      │ make_grids.py
│      │ make_input_output_shape_update.py
│      │ make_nms_outputs_merge.py
│      └─make_score_gather_nd.py
│
├─setting─┬─labels.csv
│         └─jutsu.csv
│
├─utils
│
└─_legacy

simple_demo.py

　A simple detection demo.
　

Ninjutsu_demo.py

　This is a demonstration of the establishment of Ninjutsu.
　The Ninjutsu name that matches the Ninjutsu-data(jutsu.csv) from the history of the hand-sign is displayed.
　 　

model

　Contains trained models.

setting

　Contains label data(labels.csv) and Ninjutsu name data(jutsu.csv).

  • labels.csv
    The label name of the hand-sign is listed
    • Column A：English hand-sign name
    • Column B：Japanese hand-sign name
  • jutsu.csv
    The name of the Ninjutsu name and the required hand-sign are listed.
    • Column A：Japanese technique type name(Fire Style, etc)
    • Column B：English technique type name(Fire Style, etc)
    • Column C：Japanese Ninjutsu name
    • Column D：English Ninjutsu name
    • After column E：Hand-signs required to activate ninjutsu

utils

Contains the FPS measurement module(cvfpscalc.py) and the character string drawing module(cvdrawtext.py).
Used only in Ninjutsu_demo.py.

Usage

Here's how to run the demo.

python simple_demo.py
python simple_demo_without_post.py
python Ninjutsu_demo.py

In addition, the following options can be specified when running the demo.

Option specification
  • --device
    Camera device number
    Default：
    • simple_demo.py：0
    • Ninjutsu_demo.py：0
  • --file
    Video file name ※If specified, the video will be loaded in preference to the camera
    Default：
    • simple_demo.py：None
    • Ninjutsu_demo.py：None
  • --fps
    Processing FPS ※Valid only if the inference time is less than FPS
    Default：
    • simple_demo.py：10
    • Ninjutsu_demo.py：10
  • --width
    Width when shooting with a camera
    Default：
    • simple_demo.py：960
    • Ninjutsu_demo.py：960
  • --height
    Height when shooting with a camera
    Default：
    • simple_demo.py：540
    • Ninjutsu_demo.py：540
  • --skip_frame
    Whether to thin out when loading the camera or video
    Default：
    • simple_demo.py：0
    • Ninjutsu_demo.py：0
  • --model
    Storage path of the model to load
    Default：
    • simple_demo.py：model/yolox/yolox_nano.onnx
    • Ninjutsu_demo.py：model/yolox/yolox_nano.onnx
  • --input_shape
    Model input shape
    Default：
    • simple_demo.py：416,416
    • Ninjutsu_demo.py：416,416
  • --score_th
    Class discrimination threshold
    Default：
    • simple_demo.py：0.7
    • Ninjutsu_demo.py：0.7
  • --nms_th
    NMS threshold
    Default：
    • simple_demo.py：0.45
    • Ninjutsu_demo.py：0.45
  • --nms_score_th
    NMS score threshold
    Default：
    • simple_demo.py：0.1
    • Ninjutsu_demo.py：0.1
  • --sign_interval
    The hand-sign history is cleared when the specified time(seconds) has passed since the last mark was detected.
    Default：
    • Ninjutsu_demo.py：2.0
  • --jutsu_display_time
    Time to display the Ninjutsu name when the hand-sign procedure is completed(seconds)
    Default：
    • Ninjutsu_demo.py：5
  • --use_display_score
    Whether to display the hand-sign detection score
    Default：
    • Ninjutsu_demo.py：False
  • --erase_bbox
    Whether to clear the bounding box overlay display
    Default：
    • Ninjutsu_demo.py：False
  • --use_jutsu_lang_en
    Whether to use English notation for displaying the Ninjutsu name
    Default：
    • Ninjutsu_demo.py：False
  • --chattering_check
    Continuous detection is regarded as hand-sign detection
    Default：
    • Ninjutsu_demo.py：1
  • --use_fullscreen
    Whether to use full screen display(experimental function)
    Default：
    • Ninjutsu_demo.py：False

Application Example

Here are some application examples.

Acknowledgements

During the EfficientDet model training, I referred to karaage-san explanation article5.
Also, Karaage-san introduced Deep写輪眼 on his blog6.
Thank you very much.

I use YOLOX-Colaboratory-Training-Sample7 for YOLOX training.

References

  1. ^Japan：Copyright Law Article 20 "Right to maintain identity"
  2. ^『NARUTO』Masashi Kishimoto/Shueisha 1999-2014
  3. ^Japan：Copyright Act Article 47-7 "Transfer of reproductions made due to restrictions on reproduction rights"
  4. ^Kaggle Public dataset：naruto-hand-sign-dataset
  5. ^Karaage-san's blog：「Object Detection API」で物体検出の自前データを学習する方法（TensorFlow 2.x版）
  6. ^Karaage-san's blog：AIでNARUTO気分！「Deep写輪眼」で遊んでみよう
  7. ^Kazuhito00/YOLOX-Colaboratory-Training-Sample

Authors

Kazuhito Takahashi(https://twitter.com/KzhtTkhs)

License

NARUTO-HandSignDetection is under MIT license.

License(Font)

KouzanMouhitsu(衡山毛筆) Font(https://opentype.jp/kouzanmouhitufont.htm)