diff --git a/README.md b/README.md index 2f04cba6..e3674026 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This website contains projects built by the Edge Impulse Expert Network. -There are computer vision, audio classification, anomaly detection, gesture recognition, robotics, environmental and spacial awareness use cases and more, covering a wide variety of hardware, sensors, and devices. +There are computer vision, audio classification, anomaly detection, gesture recognition, robotics, environmental and spatial awareness use cases and more, covering a wide variety of hardware, sensors, and devices. As you browse the projects, be sure to make note that many of them could be adapted to use other hardware targets, sensors, or add-ons. If you have any questions, simply let us know over on the [forum](https://forum.edgeimpulse.com/). @@ -10,92 +10,92 @@ As you browse the projects, be sure to make note that many of them could be adap ### Featured Machine Learning Projects -* [Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1](image-projects/getting-started-nvidia-tao-renesas-ekra8d1.md) -* [Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano](image-projects/traffic-analysis-tao-jetson-orin.md) -* [ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D](image-projects/robotic-arm-sorting-arduino-braccio.md) -* [Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments](image-projects/fomo-ad-in-aws.md) -* [Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator](nvidia-projects/rooftop-ice-synthetic-data-omniverse.md) -* [Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator](image-projects/surgery-inventory-synthetic-data.md) -* [NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects](nvidia-omniverse-synthetic-data.md) -* [Community Guide – Using Edge Impulse with Nvidia DeepStream](nvidia-deepstream-community-guide.md) -* [Computer Vision Object Counting - Avnet RZBoard V2L](image-projects/avnet-rzboard-object-counting.md) -* [Gesture Appliances Control with Pose Detection - BrainChip AKD1000](accelerometer-and-activity-projects/gesture-appliances-control-brainchip.md) -* [Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)](image-projects/quality-control-jetson-nano.md) -* [High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)](image-projects/high-speed-counting-jetson-nano.md) +* [Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1](readme/featured-machine-learning-projects/getting-started-nvidia-tao-renesas-ekra8d1.md) +* [Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano](readme/featured-machine-learning-projects/traffic-analysis-tao-jetson-orin.md) +* [ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D](readme/featured-machine-learning-projects/robotic-arm-sorting-arduino-braccio.md) +* [Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments](readme/featured-machine-learning-projects/fomo-ad-in-aws.md) +* [Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator](readme/featured-machine-learning-projects/rooftop-ice-synthetic-data-omniverse.md) +* [Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator](readme/featured-machine-learning-projects/surgery-inventory-synthetic-data.md) +* [NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects](readme/featured-machine-learning-projects/nvidia-omniverse-synthetic-data.md) +* [Community Guide – Using Edge Impulse with Nvidia DeepStream](readme/featured-machine-learning-projects/nvidia-deepstream-community-guide.md) +* [Computer Vision Object Counting - Avnet RZBoard V2L](readme/featured-machine-learning-projects/avnet-rzboard-object-counting.md) +* [Gesture Appliances Control with Pose Detection - BrainChip AKD1000](readme/featured-machine-learning-projects/gesture-appliances-control-brainchip.md) +* [Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)](readme/featured-machine-learning-projects/quality-control-jetson-nano.md) +* [High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)](readme/featured-machine-learning-projects/high-speed-counting-jetson-nano.md) ### Prototype and Concept Projects -* [Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning](renesas-ra6m5-getting-started.md) -* [TI CC1352P Launchpad - Getting Started with Machine Learning](ti-cc1352p-getting-started.md) +* [Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning](readme/prototype-and-concept-projects/renesas-ra6m5-getting-started.md) +* [TI CC1352P Launchpad - Getting Started with Machine Learning](readme/prototype-and-concept-projects/ti-cc1352p-getting-started.md) * [OpenMV Cam RT1062 - Getting Started with Machine Learning](getting-started-openmv-rt1062.md) -* [Getting Started with Edge Impulse Experiments](experiments-getting-started.md) +* [Getting Started with Edge Impulse Experiments](readme/prototype-and-concept-projects/experiments-getting-started.md) -### Image Projects +### Computer Vision Projects Computer vision projects that make use of image classification, object detection, or other tasks performed by a camera and machine learning. -* [Workplace Organizer - Nvidia Jetson Nano](image-projects/workplace-organizer-nvidia-jetson-nano.md) -* [Recyclable Materials Sorter - Nvidia Jetson Nano](image-projects/recyclable-materials-sorter-nvidia-jetson-nano.md) -* [Analog Meter Reading - Arduino Nicla Vision](image-projects/analog-meter-reading-arduino-nicla-vision.md) -* [Creating Synthetic Data with Nvidia Omniverse Replicator](image-projects/nvidia-omniverse-replicator.md) -* [SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display](image-projects/particle-photon-2-sonicsight-ar.md) -* [Traffic Monitoring - Brainchip Akida](image-projects/brainchip-akida-traffic-monitoring.md) -* [Multi-camera Video Stream Inference - Brainchip Akida](image-projects/brainchip-akida-multi-camera-inference.md) -* [Industrial Inspection Line - Brainchip Akida](image-projects/brainchip-akida-industrial-inspection.md) -* [X-Ray Classification and Analysis - Brainchip Akida](image-projects/brainchip-akida-xray-classification.md) -* [Inventory Stock Tracker - FOMO - BrainChip Akida](image-projects/fomo-stock-tracker-brainchip.md) -* [Container Counting - Arduino Nicla Vision](image-projects/container-counting-arduino-nicla-vision.md) -* [Smart Smoke Alarm - Arduino Nano 33](image-projects/smart-smoke-alarm-arduino-nano-33.md) -* [Shield Bot Autonomous Security Robot](image-projects/shieldbot-autonomous-security-robot.md) -* [Cyclist Blind Spot Detection - Himax WE-I Plus](image-projects/cyclist-blind-spot-detection-himax-we-i-plus.md) -* [IV Drip Fluid-Level Monitoring - Arduino Portenta H7](image-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md) -* [Worker PPE Safety Monitoring - Nvidia Jetson Nano](image-projects/worker-safety-monitoring-nvidia-jetson-nano.md) -* [Delivered Package Detection - ESP-EYE](image-projects/delivered-package-detection-esp-eye.md) -* [Bean Leaf Disease Classification - Sony Spresense](image-projects/bean-leaf-disease-classification-sony-spresense.md) -* [Oil Tank Measurement Using Computer Vision - Sony Spresense](image-projects/oil-tank-gauge-monitoring-sony-spresense.md) -* [Object Counting for Smart Industries - Raspberry Pi](image-projects/object-counting-for-smart-industries-raspberry-pi.md) -* [Smart Cashier with FOMO - Raspberry Pi](image-projects/smart-cashier-with-fomo-raspberry-pi.md) -* [PCB Defect Detection with Computer Vision - Raspberry Pi](image-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md) -* [Bicycle Counting - Sony Spresense](image-projects/bicycle-counting-sony-spresense.md) -* [Counting Eggs with Computer Vision - OpenMV Cam H7](image-projects/counting-eggs-openmv-cam-h7.md) -* [Elevator Passenger Counting - Arduino Nicla Vision](image-projects/elevator-passenger-counting-arduino-nicla-vision.md) -* [ESD Protection using Computer Vision - Seeed ReComputer](image-projects/esd-protection-using-computer-vision-seeed-recomputer.md) -* [Solar Panel Defect Detection - Arduino Portenta H7](image-projects/solar-panel-defect-detection-arduino-portenta-h7.md) -* [Label Defect Detection - Raspberry Pi](image-projects/label-defect-detection-raspberry-pi.md) -* [Dials and Knob Monitoring with Computer Vision - Raspberry Pi](image-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md) -* [Digital Character Recognition on Electric Meter System - OpenMV Cam H7](image-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md) -* [Corrosion Detection with Computer Vision - Seeed reTerminal](image-projects/corrosion-detection-computer-vision-seeed-reterminal.md) -* [Inventory Management with Computer Vision - Raspberry Pi](image-projects/inventory-management-with-computer-vision-raspberry-pi.md) -* [Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L](image-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md) -* [Counting Retail Inventory with Computer Vision - Renesas RZ/V2L](image-projects/renesas-rzv2l-counting-retail-inventory.md) -* [Pose Detection - Renesas RZ/V2L](image-projects/renesas-rzv2l-pose-detection.md) -* [Product Quality Inspection - Renesas RZ/V2L](image-projects/renesas-rzv2l-product-quality-inspection.md) -* [Smart Grocery Cart Using Computer Vision - OpenMV Cam H7](image-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md) -* [Driver Drowsiness Detection With FOMO - Arduino Nicla Vision](image-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md) -* [Gastroscopic Image Processing - OpenMV Cam H7](image-projects/gastroscopic-image-processing-openmv-cam-h7.md) -* [Pharmaceutical Pill Quality Control and Defect Detection](image-projects/pharmaceutical-pill-defect-detection.md) -* [Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM](image-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md) -* [Smart Factory Prototype - Texas Instruments TDA4VM](image-projects/smart-factory-prototype-ti-tda4vm.md) -* [Correct Posture Detection and Enforcement - Texas Instruments TDA4VM](image-projects/ti-tda4vm-posture-detection.md) -* [Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM](image-projects/fomo-ad-ti-tda4vm.md) -* [Surface Crack Detection and Localization - Texas Instruments TDA4VM](image-projects/surface-crack-detection-ti-tda4vm.md) -* [Surface Crack Detection - Seeed reTerminal](image-projects/surface-crack-detection-seeed-reterminal.md) -* [Retail Image Classification - Nvidia Jetson Nano](image-projects/retail-image-classification-nvidia-jetson-nano.md) -* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1](image-projects/silabs-xg24-card-sorting-and-robotics-1.md) -* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2](image-projects/silabs-xg24-card-sorting-and-robotics-2.md) -* [Object Detection and Visualization - Seeed Grove Vision AI Module](image-projects/object-detection-ubidots-seeed-grove-ai.md) -* [Bike Rearview Radar - Raspberry Pi](image-projects/bike-rearview-radar-raspberry-pi.md) -* [Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision](image-projects/arduino-portenta-h7-self-driving-rc-car.md) -* ["Bring Your Own Model" Image Classifier for Wound Identification](image-projects/arduino-portenta-h7-byom-wound-classification.md) -* [Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano](image-projects/ai-leukemia-classifier-nvidia-jetson-nano.md) -* [Hardhat Detection in Industrial Settings - Alif Ensemble E7](image-projects/helmet-detection-alif-ensemble.md) -* [Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A](image-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md) -* [Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A](image-projects/asl-byom-ti-am62a.md) -* [Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense](image-projects/fomo-ad-product-inspection-spresense.md) -* [Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5](image-projects/textile-fabric-anomaly-detection.md) -* [Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit](image-projects/vehicle-detection-raspberry-pi-ai-kit.md) -* [Visual Anomaly Detection - Seeed Grove Vision AI Module V2](image-projects/visual-anomaly-detection-seeed-vision-ai-2.md) -* [Object Counting with FOMO - OpenMV Cam RT1062](image-projects/object-counting-fomo-openmv-rt1062.md) +* [Workplace Organizer - Nvidia Jetson Nano](computer-vision-projects/workplace-organizer-nvidia-jetson-nano.md) +* [Recyclable Materials Sorter - Nvidia Jetson Nano](computer-vision-projects/recyclable-materials-sorter-nvidia-jetson-nano.md) +* [Analog Meter Reading - Arduino Nicla Vision](computer-vision-projects/analog-meter-reading-arduino-nicla-vision.md) +* [Creating Synthetic Data with Nvidia Omniverse Replicator](computer-vision-projects/nvidia-omniverse-replicator.md) +* [SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display](computer-vision-projects/particle-photon-2-sonicsight-ar.md) +* [Traffic Monitoring - Brainchip Akida](computer-vision-projects/brainchip-akida-traffic-monitoring.md) +* [Multi-camera Video Stream Inference - Brainchip Akida](computer-vision-projects/brainchip-akida-multi-camera-inference.md) +* [Industrial Inspection Line - Brainchip Akida](computer-vision-projects/brainchip-akida-industrial-inspection.md) +* [X-Ray Classification and Analysis - Brainchip Akida](computer-vision-projects/brainchip-akida-xray-classification.md) +* [Inventory Stock Tracker - FOMO - BrainChip Akida](computer-vision-projects/fomo-stock-tracker-brainchip.md) +* [Container Counting - Arduino Nicla Vision](computer-vision-projects/container-counting-arduino-nicla-vision.md) +* [Smart Smoke Alarm - Arduino Nano 33](computer-vision-projects/smart-smoke-alarm-arduino-nano-33.md) +* [Shield Bot Autonomous Security Robot](computer-vision-projects/shieldbot-autonomous-security-robot.md) +* [Cyclist Blind Spot Detection - Himax WE-I Plus](computer-vision-projects/cyclist-blind-spot-detection-himax-we-i-plus.md) +* [IV Drip Fluid-Level Monitoring - Arduino Portenta H7](computer-vision-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md) +* [Worker PPE Safety Monitoring - Nvidia Jetson Nano](computer-vision-projects/worker-safety-monitoring-nvidia-jetson-nano.md) +* [Delivered Package Detection - ESP-EYE](computer-vision-projects/delivered-package-detection-esp-eye.md) +* [Bean Leaf Disease Classification - Sony Spresense](computer-vision-projects/bean-leaf-disease-classification-sony-spresense.md) +* [Oil Tank Measurement Using Computer Vision - Sony Spresense](computer-vision-projects/oil-tank-gauge-monitoring-sony-spresense.md) +* [Object Counting for Smart Industries - Raspberry Pi](computer-vision-projects/object-counting-for-smart-industries-raspberry-pi.md) +* [Smart Cashier with FOMO - Raspberry Pi](computer-vision-projects/smart-cashier-with-fomo-raspberry-pi.md) +* [PCB Defect Detection with Computer Vision - Raspberry Pi](computer-vision-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md) +* [Bicycle Counting - Sony Spresense](computer-vision-projects/bicycle-counting-sony-spresense.md) +* [Counting Eggs with Computer Vision - OpenMV Cam H7](computer-vision-projects/counting-eggs-openmv-cam-h7.md) +* [Elevator Passenger Counting - Arduino Nicla Vision](computer-vision-projects/elevator-passenger-counting-arduino-nicla-vision.md) +* [ESD Protection using Computer Vision - Seeed ReComputer](computer-vision-projects/esd-protection-using-computer-vision-seeed-recomputer.md) +* [Solar Panel Defect Detection - Arduino Portenta H7](computer-vision-projects/solar-panel-defect-detection-arduino-portenta-h7.md) +* [Label Defect Detection - Raspberry Pi](computer-vision-projects/label-defect-detection-raspberry-pi.md) +* [Dials and Knob Monitoring with Computer Vision - Raspberry Pi](computer-vision-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md) +* [Digital Character Recognition on Electric Meter System - OpenMV Cam H7](computer-vision-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md) +* [Corrosion Detection with Computer Vision - Seeed reTerminal](computer-vision-projects/corrosion-detection-computer-vision-seeed-reterminal.md) +* [Inventory Management with Computer Vision - Raspberry Pi](computer-vision-projects/inventory-management-with-computer-vision-raspberry-pi.md) +* [Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md) +* [Counting Retail Inventory with Computer Vision - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-counting-retail-inventory.md) +* [Pose Detection - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-pose-detection.md) +* [Product Quality Inspection - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-product-quality-inspection.md) +* [Smart Grocery Cart Using Computer Vision - OpenMV Cam H7](computer-vision-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md) +* [Driver Drowsiness Detection With FOMO - Arduino Nicla Vision](computer-vision-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md) +* [Gastroscopic Image Processing - OpenMV Cam H7](computer-vision-projects/gastroscopic-image-processing-openmv-cam-h7.md) +* [Pharmaceutical Pill Quality Control and Defect Detection](computer-vision-projects/pharmaceutical-pill-defect-detection.md) +* [Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM](computer-vision-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md) +* [Smart Factory Prototype - Texas Instruments TDA4VM](computer-vision-projects/smart-factory-prototype-ti-tda4vm.md) +* [Correct Posture Detection and Enforcement - Texas Instruments TDA4VM](computer-vision-projects/ti-tda4vm-posture-detection.md) +* [Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM](computer-vision-projects/fomo-ad-ti-tda4vm.md) +* [Surface Crack Detection and Localization - Texas Instruments TDA4VM](computer-vision-projects/surface-crack-detection-ti-tda4vm.md) +* [Surface Crack Detection - Seeed reTerminal](computer-vision-projects/surface-crack-detection-seeed-reterminal.md) +* [Retail Image Classification - Nvidia Jetson Nano](computer-vision-projects/retail-image-classification-nvidia-jetson-nano.md) +* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1](computer-vision-projects/silabs-xg24-card-sorting-and-robotics-1.md) +* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2](computer-vision-projects/silabs-xg24-card-sorting-and-robotics-2.md) +* [Object Detection and Visualization - Seeed Grove Vision AI Module](computer-vision-projects/object-detection-ubidots-seeed-grove-ai.md) +* [Bike Rearview Radar - Raspberry Pi](computer-vision-projects/bike-rearview-radar-raspberry-pi.md) +* [Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision](computer-vision-projects/arduino-portenta-h7-self-driving-rc-car.md) +* ["Bring Your Own Model" Image Classifier for Wound Identification](computer-vision-projects/arduino-portenta-h7-byom-wound-classification.md) +* [Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano](computer-vision-projects/ai-leukemia-classifier-nvidia-jetson-nano.md) +* [Hardhat Detection in Industrial Settings - Alif Ensemble E7](computer-vision-projects/helmet-detection-alif-ensemble.md) +* [Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A](computer-vision-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md) +* [Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A](computer-vision-projects/asl-byom-ti-am62a.md) +* [Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense](computer-vision-projects/fomo-ad-product-inspection-spresense.md) +* [Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5](computer-vision-projects/textile-fabric-anomaly-detection.md) +* [Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit](computer-vision-projects/vehicle-detection-raspberry-pi-ai-kit.md) +* [Visual Anomaly Detection - Seeed Grove Vision AI Module V2](computer-vision-projects/visual-anomaly-detection-seeed-vision-ai-2.md) +* [Object Counting with FOMO - OpenMV Cam RT1062](computer-vision-projects/object-counting-fomo-openmv-rt1062.md) ### Audio Projects @@ -130,27 +130,27 @@ Audio classification, keyword spotting, wakeword detection, or other machine lea * [Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline](audio-projects/synthetic-data-pipeline-keyword-spotting.md) * [Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier](audio-projects/synthetic-data-dog-bark-classifier.md) -### Predictive Maintenance & Fault Classification +### Predictive Maintenance and Fault Classification Projects Projects devoted to the use of sensors, audio, or image data specific to the predictive maintenance use-case. -* [Predictive Maintenance - Nordic Thingy:91](predictive-maintenance-and-fault-classification/predictive-maintenance-with-nordic-thingy91.md) -* [Brushless DC Motor Anomaly Detection](predictive-maintenance-and-fault-classification/brushless-dc-motor-anomaly-detection.md) -* [Industrial Compressor Predictive Maintenance - Nordic Thingy:53](predictive-maintenance-and-fault-classification/compressor-predictive-maintenance-nordic-thingy53.md) -* [Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/anticipate-power-outages-arduino-nano-33.md) -* [Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/faulty-lithium-ion-cell-identification-battery-packs.md) -* [Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/weight-scale-predictive-maintenance-arduino-nano-33.md) -* [Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md) -* [Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md) -* [Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/refrigerator-predictive-maintenance-arduino-nano-33.md) -* [Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549](predictive-maintenance-and-fault-classification/motor-pump-cypress-cn0549.md) -* [BrickML Demo Project - 3D Printer Anomaly Detection](predictive-maintenance-and-fault-classification/brickml-3d-printer-anomaly-detection.md) -* [Condition Monitoring - Syntiant TinyML Board](predictive-maintenance-and-fault-classification/condition-monitoring-syntiant-tinyml.md) -* [Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense](predictive-maintenance-and-fault-classification/predictive-maintenance-sony-spresense-commonsense.md) -* [Vibration Classification with BrainChip's Akida](predictive-maintenance-and-fault-classification/vibration-classification-with-brainchip-akida.md) -* [AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32](predictive-maintenance-and-fault-classification/multimodal-hvac-failure-anomaly-detection-esp32.md) - -### Accelerometer & Activity Projects +* [Predictive Maintenance - Nordic Thingy:91](predictive-maintenance-and-defect-detection-projects/predictive-maintenance-with-nordic-thingy91.md) +* [Brushless DC Motor Anomaly Detection](predictive-maintenance-and-defect-detection-projects/brushless-dc-motor-anomaly-detection.md) +* [Industrial Compressor Predictive Maintenance - Nordic Thingy:53](predictive-maintenance-and-defect-detection-projects/compressor-predictive-maintenance-nordic-thingy53.md) +* [Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/anticipate-power-outages-arduino-nano-33.md) +* [Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/faulty-lithium-ion-cell-identification-battery-packs.md) +* [Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/weight-scale-predictive-maintenance-arduino-nano-33.md) +* [Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md) +* [Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md) +* [Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/refrigerator-predictive-maintenance-arduino-nano-33.md) +* [Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549](predictive-maintenance-and-defect-detection-projects/motor-pump-cypress-cn0549.md) +* [BrickML Demo Project - 3D Printer Anomaly Detection](predictive-maintenance-and-defect-detection-projects/brickml-3d-printer-anomaly-detection.md) +* [Condition Monitoring - Syntiant TinyML Board](predictive-maintenance-and-defect-detection-projects/condition-monitoring-syntiant-tinyml.md) +* [Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense](predictive-maintenance-and-defect-detection-projects/predictive-maintenance-sony-spresense-commonsense.md) +* [Vibration Classification with BrainChip's Akida](predictive-maintenance-and-defect-detection-projects/vibration-classification-with-brainchip-akida.md) +* [AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32](predictive-maintenance-and-defect-detection-projects/multimodal-hvac-failure-anomaly-detection-esp32.md) + +### Accelerometer and Activity Projects Gestures, movements, and other motions classified by machine learning models to identify or take action when detected. @@ -173,7 +173,7 @@ Gestures, movements, and other motions classified by machine learning models to * [Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53](accelerometer-and-activity-projects/continuous-gait-monitor-nordic-thingy53.md) * [Classifying Exercise Activities on a BangleJS Smartwatch](accelerometer-and-activity-projects/classify-exercise-activities-banglejs-smartwatch.md) -### Air Quality & Environmental Projects +### Air Quality and Environmental Projects Using machine learning to predict, understand, or identify information in the air or environment. diff --git a/SUMMARY.md b/SUMMARY.md index 92c7611f..fe457b94 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,204 +1,186 @@ # Table of contents * [Welcome](README.md) - -## Featured Machine Learning Projects - -* [Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1](image-projects/getting-started-nvidia-tao-renesas-ekra8d1.md) -* [Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano](image-projects/traffic-analysis-tao-jetson-orin.md) -* [ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D](image-projects/robotic-arm-sorting-arduino-braccio.md) -* [Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments](image-projects/fomo-ad-in-aws.md) -* [Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator](nvidia-projects/rooftop-ice-synthetic-data-omniverse.md) -* [Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator](image-projects/surgery-inventory-synthetic-data.md) -* [NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects](nvidia-omniverse-synthetic-data.md) -* [Community Guide – Using Edge Impulse with Nvidia DeepStream](nvidia-deepstream-community-guide.md) -* [Computer Vision Object Counting - Avnet RZBoard V2L](image-projects/avnet-rzboard-object-counting.md) -* [Gesture Appliances Control with Pose Detection - BrainChip AKD1000](accelerometer-and-activity-projects/gesture-appliances-control-brainchip.md) -* [Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)](image-projects/quality-control-jetson-nano.md) -* [High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)](image-projects/high-speed-counting-jetson-nano.md) - -## Prototype and Concept Projects - -* [Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning](renesas-ra6m5-getting-started.md) -* [TI CC1352P Launchpad - Getting Started with Machine Learning](ti-cc1352p-getting-started.md) -* [OpenMV Cam RT1062 - Getting Started with Machine Learning](image-projects/getting-started-openmv-rt1062.md) -* [Getting Started with Edge Impulse Experiments](experiments-getting-started.md) - -## Image Projects - -* [Workplace Organizer - Nvidia Jetson Nano](image-projects/workplace-organizer-nvidia-jetson-nano.md) -* [Recyclable Materials Sorter - Nvidia Jetson Nano](image-projects/recyclable-materials-sorter-nvidia-jetson-nano.md) -* [Analog Meter Reading - Arduino Nicla Vision](image-projects/analog-meter-reading-arduino-nicla-vision.md) -* [Creating Synthetic Data with Nvidia Omniverse Replicator](image-projects/nvidia-omniverse-replicator.md) -* [SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display](image-projects/particle-photon-2-sonicsight-ar.md) -* [Traffic Monitoring - Brainchip Akida](image-projects/brainchip-akida-traffic-monitoring.md) -* [Multi-camera Video Stream Inference - Brainchip Akida](image-projects/brainchip-akida-multi-camera-inference.md) -* [Industrial Inspection Line - Brainchip Akida](image-projects/brainchip-akida-industrial-inspection.md) -* [X-Ray Classification and Analysis - Brainchip Akida](image-projects/brainchip-akida-xray-classification.md) -* [Inventory Stock Tracker - FOMO - BrainChip Akida](image-projects/fomo-stock-tracker-brainchip.md) -* [Container Counting - Arduino Nicla Vision](image-projects/container-counting-arduino-nicla-vision.md) -* [Smart Smoke Alarm - Arduino Nano 33](image-projects/smart-smoke-alarm-arduino-nano-33.md) -* [Shield Bot Autonomous Security Robot](image-projects/shieldbot-autonomous-security-robot.md) -* [Cyclist Blind Spot Detection - Himax WE-I Plus](image-projects/cyclist-blind-spot-detection-himax-we-i-plus.md) -* [IV Drip Fluid-Level Monitoring - Arduino Portenta H7](image-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md) -* [Worker PPE Safety Monitoring - Nvidia Jetson Nano](image-projects/worker-safety-monitoring-nvidia-jetson-nano.md) -* [Delivered Package Detection - ESP-EYE](image-projects/delivered-package-detection-esp-eye.md) -* [Bean Leaf Disease Classification - Sony Spresense](image-projects/bean-leaf-disease-classification-sony-spresense.md) -* [Oil Tank Measurement Using Computer Vision - Sony Spresense](image-projects/oil-tank-gauge-monitoring-sony-spresense.md) -* [Object Counting for Smart Industries - Raspberry Pi](image-projects/object-counting-for-smart-industries-raspberry-pi.md) -* [Smart Cashier with FOMO - Raspberry Pi](image-projects/smart-cashier-with-fomo-raspberry-pi.md) -* [PCB Defect Detection with Computer Vision - Raspberry Pi](image-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md) -* [Bicycle Counting - Sony Spresense](image-projects/bicycle-counting-sony-spresense.md) -* [Counting Eggs with Computer Vision - OpenMV Cam H7](image-projects/counting-eggs-openmv-cam-h7.md) -* [Elevator Passenger Counting - Arduino Nicla Vision](image-projects/elevator-passenger-counting-arduino-nicla-vision.md) -* [ESD Protection using Computer Vision - Seeed ReComputer](image-projects/esd-protection-using-computer-vision-seeed-recomputer.md) -* [Solar Panel Defect Detection - Arduino Portenta H7](image-projects/solar-panel-defect-detection-arduino-portenta-h7.md) -* [Label Defect Detection - Raspberry Pi](image-projects/label-defect-detection-raspberry-pi.md) -* [Dials and Knob Monitoring with Computer Vision - Raspberry Pi](image-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md) -* [Digital Character Recognition on Electric Meter System - OpenMV Cam H7](image-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md) -* [Corrosion Detection with Computer Vision - Seeed reTerminal](image-projects/corrosion-detection-computer-vision-seeed-reterminal.md) -* [Inventory Management with Computer Vision - Raspberry Pi](image-projects/inventory-management-with-computer-vision-raspberry-pi.md) -* [Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L](image-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md) -* [Counting Retail Inventory with Computer Vision - Renesas RZ/V2L](image-projects/renesas-rzv2l-counting-retail-inventory.md) -* [Pose Detection - Renesas RZ/V2L](image-projects/renesas-rzv2l-pose-detection.md) -* [Product Quality Inspection - Renesas RZ/V2L](image-projects/renesas-rzv2l-product-quality-inspection.md) -* [Smart Grocery Cart Using Computer Vision - OpenMV Cam H7](image-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md) -* [Driver Drowsiness Detection With FOMO - Arduino Nicla Vision](image-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md) -* [Gastroscopic Image Processing - OpenMV Cam H7](image-projects/gastroscopic-image-processing-openmv-cam-h7.md) -* [Pharmaceutical Pill Quality Control and Defect Detection](image-projects/pharmaceutical-pill-defect-detection.md) -* [Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM](image-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md) -* [Smart Factory Prototype - Texas Instruments TDA4VM](image-projects/smart-factory-prototype-ti-tda4vm.md) -* [Correct Posture Detection and Enforcement - Texas Instruments TDA4VM](image-projects/ti-tda4vm-posture-detection.md) -* [Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM](image-projects/fomo-ad-ti-tda4vm.md) -* [Surface Crack Detection and Localization - Texas Instruments TDA4VM](image-projects/surface-crack-detection-ti-tda4vm.md) -* [Surface Crack Detection - Seeed reTerminal](image-projects/surface-crack-detection-seeed-reterminal.md) -* [Retail Image Classification - Nvidia Jetson Nano](image-projects/retail-image-classification-nvidia-jetson-nano.md) -* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1](image-projects/silabs-xg24-card-sorting-and-robotics-1.md) -* [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2](image-projects/silabs-xg24-card-sorting-and-robotics-2.md) -* [Object Detection and Visualization - Seeed Grove Vision AI Module](image-projects/object-detection-ubidots-seeed-grove-ai.md) -* [Bike Rearview Radar - Raspberry Pi](image-projects/bike-rearview-radar-raspberry-pi.md) -* [Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision](image-projects/arduino-portenta-h7-self-driving-rc-car.md) -* ["Bring Your Own Model" Image Classifier for Wound Identification](image-projects/arduino-portenta-h7-byom-wound-classification.md) -* [Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano](image-projects/ai-leukemia-classifier-nvidia-jetson-nano.md) -* [Hardhat Detection in Industrial Settings - Alif Ensemble E7](image-projects/helmet-detection-alif-ensemble.md) -* [Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A](image-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md) -* [Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A](image-projects/asl-byom-ti-am62a.md) -* [Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense](image-projects/fomo-ad-product-inspection-spresense.md) -* [Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5](image-projects/textile-fabric-anomaly-detection.md) -* [Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit](image-projects/vehicle-detection-raspberry-pi-ai-kit.md) -* [Visual Anomaly Detection - Seeed Grove Vision AI Module V2](image-projects/visual-anomaly-detection-seeed-vision-ai-2.md) -* [Object Counting with FOMO - OpenMV Cam RT1062](image-projects/object-counting-fomo-openmv-rt1062.md) - -## Audio Projects - -* [Glass Window Break Detection - Nordic Thingy:53](audio-projects/glass-break-detection-nordic-thingy53.md) -* [Occupancy Sensing - SiLabs xG24](audio-projects/occupancy-sensing-silabs-xg24.md) -* [Smart Appliance Control Using Voice Commands - Nordic Thingy:53](audio-projects/smart-appliance-voice-commands-nordic-thingy53.md) -* [Illegal Logging Detection - Nordic Thingy:53](audio-projects/illegal-logging-detection-nordic-thingy53.md) -* [Illegal Logging Detection - Syntiant TinyML](audio-projects/illegal-logging-detection-syntiant-tinyml.md) -* [Wearable Cough Sensor and Monitoring - Arduino Nano 33 BLE Sense](audio-projects/wearable-cough-sensor-arduino-nano-33.md) -* [Collect Data for Keyword Spotting - Raspberry Pi Pico](audio-projects/collect-data-keyword-spotting-raspberry-pi-pico.md) -* [Voice-Activated LED Strip - Raspberry Pi Pico](audio-projects/voice-activated-led-controller-raspberry-pi-pico.md) -* [Snoring Detection on a Smart Phone](audio-projects/snoring-detection-on-smartphone.md) -* [Gunshot Audio Classification - Arduino Nano 33 + Portenta H7](audio-projects/gunshot-audio-classification-arduino.md) -* [AI-Powered Patient Assistance - Arduino Nano 33 BLE Sense](audio-projects/ai-patient-assistance-arduino-nano-33.md) -* [Acoustic Pipe Leakage Detection - Arduino Portenta H7](audio-projects/acoustic-pipe-leak-detection-arduino-portenta-h7.md) -* [Location Identification using Sound - Syntiant TinyML](audio-projects/location-identification-using-sound-syntiant-tinyml.md) -* [Environmental Noise Classification - Nordic Thingy:53](audio-projects/environmental-noise-classification-nordic-thingy53.md) -* [Running Faucet Detection - Seeed XIAO Sense + Blues Cellular](audio-projects/running-faucet-detection-seeed-xiao-blues-wireless.md) -* [Vandalism Detection via Audio Classification - Arduino Nano 33 BLE Sense](audio-projects/vandalism-detection-audio-classification-arduino-nano-33.md) -* [Predictive Maintenance Using Audio Classification - Arduino Nano 33 BLE Sense](audio-projects/predictive-maintenance-using-audio-classification-arduino-nano-33.md) -* [Porting an Audio Project from the SiLabs Thunderboard Sense 2 to xG24](audio-projects/audio-classification-silabs-xg24.md) -* [Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 1](audio-projects/environmental-audio-monitoring-wearable-syntiant-tinyml-part-1.md) -* [Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 2](audio-projects/environmental-audio-monitoring-wearable-syntiant-tinyml-part-2.md) -* [Keyword Spotting - Nordic Thingy:53](audio-projects/keyword-spotting-nordic-thingy53.md) -* [Detecting Worker Accidents with Audio Classification - Syntiant TinyML](audio-projects/detecting-worker-accidents-syntiant-tinyml.md) -* [Snoring Detection with Syntiant NDP120 Neural Decision Processor - Arduino Nicla Voice](audio-projects/arduino-nicla-voice-syntiant-snoring-detection.md) -* [Recognize Voice Commands with the Particle Photon 2](audio-projects/voice-commands-particle-photon-2.md) -* [Voice Controlled Power Plug with Syntiant NDP120 (Nicla Voice)](audio-projects/voice-controlled-power-plug-nicla-voice.md) -* [Determining Compressor State with Audio Classification - Avnet RaSynBoard](audio-projects/compressor-audio-classification-rasynboard.md) -* [Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline](audio-projects/synthetic-data-pipeline-keyword-spotting.md) -* [Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier](audio-projects/synthetic-data-dog-bark-classifier.md) - -## Predictive Maintenance & Fault Classification - -* [Predictive Maintenance - Nordic Thingy:91](predictive-maintenance-and-fault-classification/predictive-maintenance-with-nordic-thingy91.md) -* [Brushless DC Motor Anomaly Detection](predictive-maintenance-and-fault-classification/brushless-dc-motor-anomaly-detection.md) -* [Industrial Compressor Predictive Maintenance - Nordic Thingy:53](predictive-maintenance-and-fault-classification/compressor-predictive-maintenance-nordic-thingy53.md) -* [Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/anticipate-power-outages-arduino-nano-33.md) -* [Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/faulty-lithium-ion-cell-identification-battery-packs.md) -* [Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/weight-scale-predictive-maintenance-arduino-nano-33.md) -* [Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md) -* [Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-fault-classification/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md) -* [Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-fault-classification/refrigerator-predictive-maintenance-arduino-nano-33.md) -* [Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549](predictive-maintenance-and-fault-classification/motor-pump-cypress-cn0549.md) -* [BrickML Demo Project - 3D Printer Anomaly Detection](predictive-maintenance-and-fault-classification/brickml-3d-printer-anomaly-detection.md) -* [Condition Monitoring - Syntiant TinyML Board](predictive-maintenance-and-fault-classification/condition-monitoring-syntiant-tinyml.md) -* [Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense](predictive-maintenance-and-fault-classification/predictive-maintenance-sony-spresense-commonsense.md) -* [Vibration Classification with BrainChip's Akida](predictive-maintenance-and-fault-classification/vibration-classification-with-brainchip-akida.md) -* [AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32](predictive-maintenance-and-fault-classification/multimodal-hvac-failure-anomaly-detection-esp32.md) - -## Accelerometer & Activity Projects - -* [Arduino x K-Way - Outdoor Activity Tracker](accelerometer-and-activity-projects/arduino-kway-outdoor-activity-tracker.md) -* [Arduino x K-Way - Gesture Recognition for Hiking](accelerometer-and-activity-projects/arduino-kway-gesture-recognition-weather.md) -* [Arduino x K-Way - TinyML Fall Detection](accelerometer-and-activity-projects/arduino-kway-fall-detection.md) -* [Posture Detection for Worker Safety - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/posture-detection-for-worker-safety-silabs-thunderboard-sense-2.md) -* [Hand Gesture Recognition - OpenMV Cam H7](accelerometer-and-activity-projects/hand-gesture-recognition-openmv-cam-h7.md) -* [Arduin-Row, a TinyML Rowing Machine Coach - Arduino Nicla Sense ME](accelerometer-and-activity-projects/arduin-row-tinyml-rowing-machine-coach-arduino-nicla-sense.md) -* [Fall Detection using a Transformer Model – Arduino Giga R1 WiFi](accelerometer-and-activity-projects/fall-detection-with-transformer-model-arduino-giga-r1.md) -* [Bluetooth Fall Detection - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/bluetooth-fall-detection-arduino-nano-33.md) -* [Monitor Packages During Transit with AI - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/monitor-packages-during-transit-with-ai-arduino-nano-33.md) -* [Smart Baby Swing - Arduino Portenta H7](accelerometer-and-activity-projects/smart-baby-swing-arduino-portenta-h7.md) -* [Warehouse Shipment Monitoring - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/warehouse-shipment-monitoring-silabs-thunderboard-sense-2.md) -* [Gesture Recognition - Bangle.js Smartwatch](accelerometer-and-activity-projects/gesture-recognition-banglejs-smartwatch.md) -* [Gesture Recognition for Patient Communication - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/gesture-recognition-patient-communication-silabs-thunderboard-sense-2.md) -* [Hospital Bed Occupancy Detection - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/hospital-bed-occupancy-detection-arduino-nano-33.md) -* [Porting a Posture Detection Project from the SiLabs Thunderboard Sense 2 to xG24](accelerometer-and-activity-projects/silabs-xg24-posture-detection.md) -* [Porting a Gesture Recognition Project from the SiLabs Thunderboard Sense 2 to xG24](accelerometer-and-activity-projects/silabs-xg24-gesture-recognition.md) -* [Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53](accelerometer-and-activity-projects/continuous-gait-monitor-nordic-thingy53.md) -* [Classifying Exercise Activities on a BangleJS Smartwatch](accelerometer-and-activity-projects/classify-exercise-activities-banglejs-smartwatch.md) - -## Air Quality & Environmental Projects - -* [Arduino x K-Way - Environmental Asthma Risk Assessment](air-quality-and-environmental-projects/arduino-kway-environmental-asthma-risk-assessment.md) -* [Gas Detection in the Oil and Gas Industry - Nordic Thingy:91](air-quality-and-environmental-projects/gas-detection-nordic-thingy91.md) -* [Smart HVAC System with a Sony Spresense](air-quality-and-environmental-projects/sony-spresense-smart-hvac-system.md) -* [Smart HVAC System with an Arduino Nicla Vision](air-quality-and-environmental-projects/arduino-nicla-vision-smart-hvac.md) -* [Indoor CO2 Level Estimation - Arduino Portenta H7](air-quality-and-environmental-projects/indoor-co2-level-estimation-arduino-portenta-h7.md) -* [Harmful Gases Detection - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/harmful-gases-detection-arduino-nano-33.md) -* [Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/fire-detection-sensor-fusion-arduino-nano-33.md) -* [AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3](air-quality-and-environmental-projects/dairy-manufacturing-with-ai-seeed-xiao-esp32c3.md) -* [AI-Assisted Air Quality Monitoring - DFRobot Firebeetle ESP32](air-quality-and-environmental-projects/air-quality-monitoring-firebeetle-esp32.md) -* [Air Quality Monitoring with Sipeed Longan Nano - RISC-V Gigadevice](air-quality-and-environmental-projects/air-quality-monitoring-sipeed-longan-nano-riscv.md) -* [Methane Monitoring in Mines - Silabs xG24 Dev Kit](air-quality-and-environmental-projects/methane-monitoring-silabs-xg24.md) -* [Smart Building Ventilation with Environmental Sensor Fusion](air-quality-and-environmental-projects/smart-building-sensor-fusion.md) -* [Sensor Data Fusion with Spresense and CommonSense](air-quality-and-environmental-projects/environmental-sensor-fusion-commonsense.md) -* [Water Pollution Detection - Arduino Nano ESP32 + Ultrasonic Scan](air-quality-and-environmental-projects/water-pollution-detection-arduino-nano-esp32.md) -* [Fire Detection Using Sensor Fusion - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/fire-detection-sensor-fusion-arduino-nano.md) - -## Novel Sensor Projects - -* [8x8 ToF Gesture Classification - Arduino RP2040 Connect](novel-sensor-projects/tof-gesture-classification-arduino-rp2040-connect.md) -* [Food Irradiation Dose Detection - DFRobot Beetle ESP32C3](novel-sensor-projects/food-irradiation-dose-detection-dfrobot-beetle-esp32c3.md) -* [Applying EEG Data to Machine Learning, Part 1](novel-sensor-projects/eeg-data-machine-learning-part-1.md) -* [Applying EEG Data to Machine Learning, Part 2](novel-sensor-projects/eeg-data-machine-learning-part-2.md) -* [Applying EEG Data to Machine Learning, Part 3](novel-sensor-projects/eeg-data-machine-learning-part-3.md) -* [Liquid Classification with TinyML - Seeed Wio Terminal + TDS Sensor](novel-sensor-projects/liquid-classification-seeed-wio-terminal.md) -* [AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar](novel-sensor-projects/ai-pipeline-inspection-mmwave-radar.md) -* [Soil Quality Detection Using AI and LoRaWAN - Seeed Sensecap A1101](novel-sensor-projects/soil-quality-detection-lorawan-seeed-sensecap-a1101.md) -* [Smart Diaper Prototype - Arduino Nicla Sense ME](novel-sensor-projects/smart-diaper-arduino-nicla-sense-me.md) -* [DIY Smart Glove with Flex Sensors](novel-sensor-projects/flex-sensors-hci.md) -* [EdgeML Energy Monitoring - Particle Photon 2](novel-sensor-projects/energy-monitoring-particle-photon-2.md) - -## Software Integration Demos - -* [Azure Machine Learning with Kubernetes Compute and Edge Impulse](software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md) -* [ROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python](software-integration-demos/ros2-part1-pubsub-node.md) -* [ROS2 + Edge Impulse, Part 2: MicroROS](software-integration-demos/ros2-part2-microros.md) -* [Using Hugging Face Datasets in Edge Impulse](software-integration-demos/huggingface-dataset-with-edge-impulse.md) -* [Using Hugging Face Image Classification Datasets with Edge Impulse](software-integration-demos/hugging-face-image-classification-dataset.md) -* [Edge Impulse API Usage Sample Application - Jetson Nano Trainer](software-integration-demos/api-sample-application-jetson-nano.md) -* [MLOps with Edge Impulse and Azure IoT Edge](software-integration-demos/mlops-azure-iot-edge.md) -* [A Federated Approach to Train and Deploy Machine Learning Models](software-integration-demos/federated-learning-raspberry-pi.md) -* [DIY Model Weight Update for Continuous AI Deployments](software-integration-demos/diy-model-ota.md) + * [Featured Machine Learning Projects](readme/featured-machine-learning-projects/README.md) + * [Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1](readme/featured-machine-learning-projects/getting-started-nvidia-tao-renesas-ekra8d1.md) + * [Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano](readme/featured-machine-learning-projects/traffic-analysis-tao-jetson-orin.md) + * [ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D](readme/featured-machine-learning-projects/robotic-arm-sorting-arduino-braccio.md) + * [Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments](readme/featured-machine-learning-projects/fomo-ad-in-aws.md) + * [Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator](readme/featured-machine-learning-projects/rooftop-ice-synthetic-data-omniverse.md) + * [Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator](readme/featured-machine-learning-projects/surgery-inventory-synthetic-data.md) + * [NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects](readme/featured-machine-learning-projects/nvidia-omniverse-synthetic-data.md) + * [Community Guide – Using Edge Impulse with Nvidia DeepStream](readme/featured-machine-learning-projects/nvidia-deepstream-community-guide.md) + * [Computer Vision Object Counting - Avnet RZBoard V2L](readme/featured-machine-learning-projects/avnet-rzboard-object-counting.md) + * [Gesture Appliances Control with Pose Detection - BrainChip AKD1000](readme/featured-machine-learning-projects/gesture-appliances-control-brainchip.md) + * [Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)](readme/featured-machine-learning-projects/quality-control-jetson-nano.md) + * [High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)](readme/featured-machine-learning-projects/high-speed-counting-jetson-nano.md) + * [Prototype and Concept Projects](readme/prototype-and-concept-projects/README.md) + * [Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning](readme/prototype-and-concept-projects/renesas-ra6m5-getting-started.md) + * [TI CC1352P Launchpad - Getting Started with Machine Learning](readme/prototype-and-concept-projects/ti-cc1352p-getting-started.md) + * [OpenMV Cam RT1062 - Getting Started with Machine Learning](readme/prototype-and-concept-projects/getting-started-openmv-rt1062.md) + * [Getting Started with Edge Impulse Experiments](readme/prototype-and-concept-projects/experiments-getting-started.md) +* [Computer Vision Projects](computer-vision-projects/README.md) + * [Workplace Organizer - Nvidia Jetson Nano](computer-vision-projects/workplace-organizer-nvidia-jetson-nano.md) + * [Recyclable Materials Sorter - Nvidia Jetson Nano](computer-vision-projects/recyclable-materials-sorter-nvidia-jetson-nano.md) + * [Analog Meter Reading - Arduino Nicla Vision](computer-vision-projects/analog-meter-reading-arduino-nicla-vision.md) + * [Creating Synthetic Data with Nvidia Omniverse Replicator](computer-vision-projects/nvidia-omniverse-replicator.md) + * [SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display](computer-vision-projects/particle-photon-2-sonicsight-ar.md) + * [Traffic Monitoring - Brainchip Akida](computer-vision-projects/brainchip-akida-traffic-monitoring.md) + * [Multi-camera Video Stream Inference - Brainchip Akida](computer-vision-projects/brainchip-akida-multi-camera-inference.md) + * [Industrial Inspection Line - Brainchip Akida](computer-vision-projects/brainchip-akida-industrial-inspection.md) + * [X-Ray Classification and Analysis - Brainchip Akida](computer-vision-projects/brainchip-akida-xray-classification.md) + * [Inventory Stock Tracker - FOMO - BrainChip Akida](computer-vision-projects/fomo-stock-tracker-brainchip.md) + * [Container Counting - Arduino Nicla Vision](computer-vision-projects/container-counting-arduino-nicla-vision.md) + * [Smart Smoke Alarm - Arduino Nano 33](computer-vision-projects/smart-smoke-alarm-arduino-nano-33.md) + * [Shield Bot Autonomous Security Robot](computer-vision-projects/shieldbot-autonomous-security-robot.md) + * [Cyclist Blind Spot Detection - Himax WE-I Plus](computer-vision-projects/cyclist-blind-spot-detection-himax-we-i-plus.md) + * [IV Drip Fluid-Level Monitoring - Arduino Portenta H7](computer-vision-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md) + * [Worker PPE Safety Monitoring - Nvidia Jetson Nano](computer-vision-projects/worker-safety-monitoring-nvidia-jetson-nano.md) + * [Delivered Package Detection - ESP-EYE](computer-vision-projects/delivered-package-detection-esp-eye.md) + * [Bean Leaf Disease Classification - Sony Spresense](computer-vision-projects/bean-leaf-disease-classification-sony-spresense.md) + * [Oil Tank Measurement Using Computer Vision - Sony Spresense](computer-vision-projects/oil-tank-gauge-monitoring-sony-spresense.md) + * [Object Counting for Smart Industries - Raspberry Pi](computer-vision-projects/object-counting-for-smart-industries-raspberry-pi.md) + * [Smart Cashier with FOMO - Raspberry Pi](computer-vision-projects/smart-cashier-with-fomo-raspberry-pi.md) + * [PCB Defect Detection with Computer Vision - Raspberry Pi](computer-vision-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md) + * [Bicycle Counting - Sony Spresense](computer-vision-projects/bicycle-counting-sony-spresense.md) + * [Counting Eggs with Computer Vision - OpenMV Cam H7](computer-vision-projects/counting-eggs-openmv-cam-h7.md) + * [Elevator Passenger Counting - Arduino Nicla Vision](computer-vision-projects/elevator-passenger-counting-arduino-nicla-vision.md) + * [ESD Protection using Computer Vision - Seeed ReComputer](computer-vision-projects/esd-protection-using-computer-vision-seeed-recomputer.md) + * [Solar Panel Defect Detection - Arduino Portenta H7](computer-vision-projects/solar-panel-defect-detection-arduino-portenta-h7.md) + * [Label Defect Detection - Raspberry Pi](computer-vision-projects/label-defect-detection-raspberry-pi.md) + * [Dials and Knob Monitoring with Computer Vision - Raspberry Pi](computer-vision-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md) + * [Digital Character Recognition on Electric Meter System - OpenMV Cam H7](computer-vision-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md) + * [Corrosion Detection with Computer Vision - Seeed reTerminal](computer-vision-projects/corrosion-detection-computer-vision-seeed-reterminal.md) + * [Inventory Management with Computer Vision - Raspberry Pi](computer-vision-projects/inventory-management-with-computer-vision-raspberry-pi.md) + * [Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md) + * [Counting Retail Inventory with Computer Vision - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-counting-retail-inventory.md) + * [Pose Detection - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-pose-detection.md) + * [Product Quality Inspection - Renesas RZ/V2L](computer-vision-projects/renesas-rzv2l-product-quality-inspection.md) + * [Smart Grocery Cart Using Computer Vision - OpenMV Cam H7](computer-vision-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md) + * [Driver Drowsiness Detection With FOMO - Arduino Nicla Vision](computer-vision-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md) + * [Gastroscopic Image Processing - OpenMV Cam H7](computer-vision-projects/gastroscopic-image-processing-openmv-cam-h7.md) + * [Pharmaceutical Pill Quality Control and Defect Detection](computer-vision-projects/pharmaceutical-pill-defect-detection.md) + * [Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM](computer-vision-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md) + * [Smart Factory Prototype - Texas Instruments TDA4VM](computer-vision-projects/smart-factory-prototype-ti-tda4vm.md) + * [Correct Posture Detection and Enforcement - Texas Instruments TDA4VM](computer-vision-projects/ti-tda4vm-posture-detection.md) + * [Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM](computer-vision-projects/fomo-ad-ti-tda4vm.md) + * [Surface Crack Detection and Localization - Texas Instruments TDA4VM](computer-vision-projects/surface-crack-detection-ti-tda4vm.md) + * [Surface Crack Detection - Seeed reTerminal](computer-vision-projects/surface-crack-detection-seeed-reterminal.md) + * [Retail Image Classification - Nvidia Jetson Nano](computer-vision-projects/retail-image-classification-nvidia-jetson-nano.md) + * [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1](computer-vision-projects/silabs-xg24-card-sorting-and-robotics-1.md) + * [SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2](computer-vision-projects/silabs-xg24-card-sorting-and-robotics-2.md) + * [Object Detection and Visualization - Seeed Grove Vision AI Module](computer-vision-projects/object-detection-ubidots-seeed-grove-ai.md) + * [Bike Rearview Radar - Raspberry Pi](computer-vision-projects/bike-rearview-radar-raspberry-pi.md) + * [Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision](computer-vision-projects/arduino-portenta-h7-self-driving-rc-car.md) + * ["Bring Your Own Model" Image Classifier for Wound Identification](computer-vision-projects/arduino-portenta-h7-byom-wound-classification.md) + * [Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano](computer-vision-projects/ai-leukemia-classifier-nvidia-jetson-nano.md) + * [Hardhat Detection in Industrial Settings - Alif Ensemble E7](computer-vision-projects/helmet-detection-alif-ensemble.md) + * [Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A](computer-vision-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md) + * [Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A](computer-vision-projects/asl-byom-ti-am62a.md) + * [Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense](computer-vision-projects/fomo-ad-product-inspection-spresense.md) + * [Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5](computer-vision-projects/textile-fabric-anomaly-detection.md) + * [Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit](computer-vision-projects/vehicle-detection-raspberry-pi-ai-kit.md) + * [Visual Anomaly Detection - Seeed Grove Vision AI Module V2](computer-vision-projects/visual-anomaly-detection-seeed-vision-ai-2.md) + * [Object Counting with FOMO - OpenMV Cam RT1062](computer-vision-projects/object-counting-fomo-openmv-rt1062.md) +* [Audio Projects](audio-projects/README.md) + * [Occupancy Sensing - SiLabs xG24](audio-projects/occupancy-sensing-silabs-xg24.md) + * [Smart Appliance Control Using Voice Commands - Nordic Thingy:53](audio-projects/smart-appliance-voice-commands-nordic-thingy53.md) + * [Glass Window Break Detection - Nordic Thingy:53](audio-projects/glass-break-detection-nordic-thingy53.md) + * [Illegal Logging Detection - Nordic Thingy:53](audio-projects/illegal-logging-detection-nordic-thingy53.md) + * [Illegal Logging Detection - Syntiant TinyML](audio-projects/illegal-logging-detection-syntiant-tinyml.md) + * [Wearable Cough Sensor and Monitoring - Arduino Nano 33 BLE Sense](audio-projects/wearable-cough-sensor-arduino-nano-33.md) + * [Collect Data for Keyword Spotting - Raspberry Pi Pico](audio-projects/collect-data-keyword-spotting-raspberry-pi-pico.md) + * [Voice-Activated LED Strip - Raspberry Pi Pico](audio-projects/voice-activated-led-controller-raspberry-pi-pico.md) + * [Snoring Detection on a Smart Phone](audio-projects/snoring-detection-on-smartphone.md) + * [Gunshot Audio Classification - Arduino Nano 33 + Portenta H7](audio-projects/gunshot-audio-classification-arduino.md) + * [AI-Powered Patient Assistance - Arduino Nano 33 BLE Sense](audio-projects/ai-patient-assistance-arduino-nano-33.md) + * [Acoustic Pipe Leakage Detection - Arduino Portenta H7](audio-projects/acoustic-pipe-leak-detection-arduino-portenta-h7.md) + * [Location Identification using Sound - Syntiant TinyML](audio-projects/location-identification-using-sound-syntiant-tinyml.md) + * [Environmental Noise Classification - Nordic Thingy:53](audio-projects/environmental-noise-classification-nordic-thingy53.md) + * [Running Faucet Detection - Seeed XIAO Sense + Blues Cellular](audio-projects/running-faucet-detection-seeed-xiao-blues-wireless.md) + * [Vandalism Detection via Audio Classification - Arduino Nano 33 BLE Sense](audio-projects/vandalism-detection-audio-classification-arduino-nano-33.md) + * [Predictive Maintenance Using Audio Classification - Arduino Nano 33 BLE Sense](audio-projects/predictive-maintenance-using-audio-classification-arduino-nano-33.md) + * [Porting an Audio Project from the SiLabs Thunderboard Sense 2 to xG24](audio-projects/audio-classification-silabs-xg24.md) + * [Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 1](audio-projects/environmental-audio-monitoring-wearable-syntiant-tinyml-part-1.md) + * [Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 2](audio-projects/environmental-audio-monitoring-wearable-syntiant-tinyml-part-2.md) + * [Keyword Spotting - Nordic Thingy:53](audio-projects/keyword-spotting-nordic-thingy53.md) + * [Detecting Worker Accidents with Audio Classification - Syntiant TinyML](audio-projects/detecting-worker-accidents-syntiant-tinyml.md) + * [Snoring Detection with Syntiant NDP120 Neural Decision Processor - Arduino Nicla Voice](audio-projects/arduino-nicla-voice-syntiant-snoring-detection.md) + * [Recognize Voice Commands with the Particle Photon 2](audio-projects/voice-commands-particle-photon-2.md) + * [Voice Controlled Power Plug with Syntiant NDP120 (Nicla Voice)](audio-projects/voice-controlled-power-plug-nicla-voice.md) + * [Determining Compressor State with Audio Classification - Avnet RaSynBoard](audio-projects/compressor-audio-classification-rasynboard.md) + * [Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline](audio-projects/synthetic-data-pipeline-keyword-spotting.md) + * [Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier](audio-projects/synthetic-data-dog-bark-classifier.md) +* [Predictive Maintenance and Defect Detection Projects](predictive-maintenance-and-defect-detection-projects/README.md) + * [Predictive Maintenance - Nordic Thingy:91](predictive-maintenance-and-defect-detection-projects/predictive-maintenance-with-nordic-thingy91.md) + * [Brushless DC Motor Anomaly Detection](predictive-maintenance-and-defect-detection-projects/brushless-dc-motor-anomaly-detection.md) + * [Industrial Compressor Predictive Maintenance - Nordic Thingy:53](predictive-maintenance-and-defect-detection-projects/compressor-predictive-maintenance-nordic-thingy53.md) + * [Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/anticipate-power-outages-arduino-nano-33.md) + * [Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/faulty-lithium-ion-cell-identification-battery-packs.md) + * [Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/weight-scale-predictive-maintenance-arduino-nano-33.md) + * [Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md) + * [Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal](predictive-maintenance-and-defect-detection-projects/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md) + * [Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense](predictive-maintenance-and-defect-detection-projects/refrigerator-predictive-maintenance-arduino-nano-33.md) + * [Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549](predictive-maintenance-and-defect-detection-projects/motor-pump-cypress-cn0549.md) + * [BrickML Demo Project - 3D Printer Anomaly Detection](predictive-maintenance-and-defect-detection-projects/brickml-3d-printer-anomaly-detection.md) + * [Condition Monitoring - Syntiant TinyML Board](predictive-maintenance-and-defect-detection-projects/condition-monitoring-syntiant-tinyml.md) + * [Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense](predictive-maintenance-and-defect-detection-projects/predictive-maintenance-sony-spresense-commonsense.md) + * [Vibration Classification with BrainChip's Akida](predictive-maintenance-and-defect-detection-projects/vibration-classification-with-brainchip-akida.md) + * [AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32](predictive-maintenance-and-defect-detection-projects/multimodal-hvac-failure-anomaly-detection-esp32.md) +* [Accelerometer and Activity Projects](accelerometer-and-activity-projects/README.md) + * [Arduino x K-Way - Outdoor Activity Tracker](accelerometer-and-activity-projects/arduino-kway-outdoor-activity-tracker.md) + * [Arduino x K-Way - Gesture Recognition for Hiking](accelerometer-and-activity-projects/arduino-kway-gesture-recognition-weather.md) + * [Arduino x K-Way - TinyML Fall Detection](accelerometer-and-activity-projects/arduino-kway-fall-detection.md) + * [Posture Detection for Worker Safety - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/posture-detection-for-worker-safety-silabs-thunderboard-sense-2.md) + * [Hand Gesture Recognition - OpenMV Cam H7](accelerometer-and-activity-projects/hand-gesture-recognition-openmv-cam-h7.md) + * [Arduin-Row, a TinyML Rowing Machine Coach - Arduino Nicla Sense ME](accelerometer-and-activity-projects/arduin-row-tinyml-rowing-machine-coach-arduino-nicla-sense.md) + * [Fall Detection using a Transformer Model – Arduino Giga R1 WiFi](accelerometer-and-activity-projects/fall-detection-with-transformer-model-arduino-giga-r1.md) + * [Bluetooth Fall Detection - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/bluetooth-fall-detection-arduino-nano-33.md) + * [Monitor Packages During Transit with AI - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/monitor-packages-during-transit-with-ai-arduino-nano-33.md) + * [Smart Baby Swing - Arduino Portenta H7](accelerometer-and-activity-projects/smart-baby-swing-arduino-portenta-h7.md) + * [Warehouse Shipment Monitoring - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/warehouse-shipment-monitoring-silabs-thunderboard-sense-2.md) + * [Gesture Recognition - Bangle.js Smartwatch](accelerometer-and-activity-projects/gesture-recognition-banglejs-smartwatch.md) + * [Gesture Recognition for Patient Communication - SiLabs Thunderboard Sense 2](accelerometer-and-activity-projects/gesture-recognition-patient-communication-silabs-thunderboard-sense-2.md) + * [Hospital Bed Occupancy Detection - Arduino Nano 33 BLE Sense](accelerometer-and-activity-projects/hospital-bed-occupancy-detection-arduino-nano-33.md) + * [Porting a Posture Detection Project from the SiLabs Thunderboard Sense 2 to xG24](accelerometer-and-activity-projects/silabs-xg24-posture-detection.md) + * [Porting a Gesture Recognition Project from the SiLabs Thunderboard Sense 2 to xG24](accelerometer-and-activity-projects/silabs-xg24-gesture-recognition.md) + * [Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53](accelerometer-and-activity-projects/continuous-gait-monitor-nordic-thingy53.md) + * [Classifying Exercise Activities on a BangleJS Smartwatch](accelerometer-and-activity-projects/classify-exercise-activities-banglejs-smartwatch.md) +* [Air Quality and Environmental Projects](air-quality-and-environmental-projects/README.md) + * [Arduino x K-Way - Environmental Asthma Risk Assessment](air-quality-and-environmental-projects/arduino-kway-environmental-asthma-risk-assessment.md) + * [Gas Detection in the Oil and Gas Industry - Nordic Thingy:91](air-quality-and-environmental-projects/gas-detection-nordic-thingy91.md) + * [Smart HVAC System with a Sony Spresense](air-quality-and-environmental-projects/sony-spresense-smart-hvac-system.md) + * [Smart HVAC System with an Arduino Nicla Vision](air-quality-and-environmental-projects/arduino-nicla-vision-smart-hvac.md) + * [Indoor CO2 Level Estimation - Arduino Portenta H7](air-quality-and-environmental-projects/indoor-co2-level-estimation-arduino-portenta-h7.md) + * [Harmful Gases Detection - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/harmful-gases-detection-arduino-nano-33.md) + * [Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/fire-detection-sensor-fusion-arduino-nano-33.md) + * [AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3](air-quality-and-environmental-projects/dairy-manufacturing-with-ai-seeed-xiao-esp32c3.md) + * [AI-Assisted Air Quality Monitoring - DFRobot Firebeetle ESP32](air-quality-and-environmental-projects/air-quality-monitoring-firebeetle-esp32.md) + * [Air Quality Monitoring with Sipeed Longan Nano - RISC-V Gigadevice](air-quality-and-environmental-projects/air-quality-monitoring-sipeed-longan-nano-riscv.md) + * [Methane Monitoring in Mines - Silabs xG24 Dev Kit](air-quality-and-environmental-projects/methane-monitoring-silabs-xg24.md) + * [Smart Building Ventilation with Environmental Sensor Fusion](air-quality-and-environmental-projects/smart-building-sensor-fusion.md) + * [Sensor Data Fusion with Spresense and CommonSense](air-quality-and-environmental-projects/environmental-sensor-fusion-commonsense.md) + * [Water Pollution Detection - Arduino Nano ESP32 + Ultrasonic Scan](air-quality-and-environmental-projects/water-pollution-detection-arduino-nano-esp32.md) + * [Fire Detection Using Sensor Fusion - Arduino Nano 33 BLE Sense](air-quality-and-environmental-projects/fire-detection-sensor-fusion-arduino-nano.md) +* [Novel Sensor Projects](novel-sensor-projects/README.md) + * [8x8 ToF Gesture Classification - Arduino RP2040 Connect](novel-sensor-projects/tof-gesture-classification-arduino-rp2040-connect.md) + * [Food Irradiation Dose Detection - DFRobot Beetle ESP32C3](novel-sensor-projects/food-irradiation-dose-detection-dfrobot-beetle-esp32c3.md) + * [Applying EEG Data to Machine Learning, Part 1](novel-sensor-projects/eeg-data-machine-learning-part-1.md) + * [Applying EEG Data to Machine Learning, Part 2](novel-sensor-projects/eeg-data-machine-learning-part-2.md) + * [Applying EEG Data to Machine Learning, Part 3](novel-sensor-projects/eeg-data-machine-learning-part-3.md) + * [Liquid Classification with TinyML - Seeed Wio Terminal + TDS Sensor](novel-sensor-projects/liquid-classification-seeed-wio-terminal.md) + * [AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar](novel-sensor-projects/ai-pipeline-inspection-mmwave-radar.md) + * [Soil Quality Detection Using AI and LoRaWAN - Seeed Sensecap A1101](novel-sensor-projects/soil-quality-detection-lorawan-seeed-sensecap-a1101.md) + * [Smart Diaper Prototype - Arduino Nicla Sense ME](novel-sensor-projects/smart-diaper-arduino-nicla-sense-me.md) + * [DIY Smart Glove with Flex Sensors](novel-sensor-projects/flex-sensors-hci.md) + * [EdgeML Energy Monitoring - Particle Photon 2](novel-sensor-projects/energy-monitoring-particle-photon-2.md) +* [Software Integration Demos](software-integration-demos/README.md) + * [Azure Machine Learning with Kubernetes Compute and Edge Impulse](software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md) + * [ROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python](software-integration-demos/ros2-part1-pubsub-node.md) + * [ROS2 + Edge Impulse, Part 2: MicroROS](software-integration-demos/ros2-part2-microros.md) + * [Using Hugging Face Datasets in Edge Impulse](software-integration-demos/huggingface-dataset-with-edge-impulse.md) + * [Using Hugging Face Image Classification Datasets with Edge Impulse](software-integration-demos/hugging-face-image-classification-dataset.md) + * [Edge Impulse API Usage Sample Application - Jetson Nano Trainer](software-integration-demos/api-sample-application-jetson-nano.md) + * [MLOps with Edge Impulse and Azure IoT Edge](software-integration-demos/mlops-azure-iot-edge.md) + * [A Federated Approach to Train and Deploy Machine Learning Models](software-integration-demos/federated-learning-raspberry-pi.md) + * [DIY Model Weight Update for Continuous AI Deployments](software-integration-demos/diy-model-ota.md) diff --git a/accelerometer-and-activity-projects/README.md b/accelerometer-and-activity-projects/README.md new file mode 100644 index 00000000..bbfea365 --- /dev/null +++ b/accelerometer-and-activity-projects/README.md @@ -0,0 +1,2 @@ +# Accelerometer and Activity Projects + diff --git a/air-quality-and-environmental-projects/README.md b/air-quality-and-environmental-projects/README.md new file mode 100644 index 00000000..ca0fa4af --- /dev/null +++ b/air-quality-and-environmental-projects/README.md @@ -0,0 +1,2 @@ +# Air Quality and Environmental Projects + diff --git a/audio-projects/README.md b/audio-projects/README.md new file mode 100644 index 00000000..bd7211d3 --- /dev/null +++ b/audio-projects/README.md @@ -0,0 +1,2 @@ +# Audio Projects + diff --git a/audio-projects/synthetic-data-pipeline-keyword-spotting.md b/audio-projects/synthetic-data-pipeline-keyword-spotting.md index d3296d5b..0e69d1ca 100644 --- a/audio-projects/synthetic-data-pipeline-keyword-spotting.md +++ b/audio-projects/synthetic-data-pipeline-keyword-spotting.md @@ -1,6 +1,8 @@ --- description: >- - End-to-end synthetic data pipeline for the creation of a portable LED product equipped with keyword spotting capabilities. The project serves as a comprehensive guide for development of any KWS product, emphasizing the utilization of synthetic data for model training. + End-to-end synthetic data pipeline for the creation of a portable LED product + equipped with keyword spotting capabilities. The project serves as a + comprehensive guide for development of any KWS produc --- # Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline @@ -41,14 +43,14 @@ The Arduino Nicla Voice is an ideal choice for this project due to its use of th ## Hardware Requirements -- Arduino Nicla Voice (or other Edge Impulse supported MCU with mic) -- PCB and SMD components (parts breakdown explained later) +* Arduino Nicla Voice (or other Edge Impulse supported MCU with mic) +* PCB and SMD components (parts breakdown explained later) ## Software Requirements -- Edge Impulse CLI -- Arduino IDE -- OpenAI API account +* Edge Impulse CLI +* Arduino IDE +* OpenAI API account ## Dataset Collection @@ -64,9 +66,9 @@ Once you have your secret key, you can navigate to your Edge Impulse organizatio ### Generating TTS Synthetic Data -Now that we have the environment configured, and our OpenAI API saved in the Edge Impulse Studio, we are ready to start a new project and begin generating some synthetic voice data. +Now that we have the environment configured, and our OpenAI API saved in the Edge Impulse Studio, we are ready to start a new project and begin generating some synthetic voice data. -On your project's page select Data acquisition --> Data sources --> + Add new data source --> Transformation block --> Whisper Synthetic Voice Generator --> Fill out the details as follow: +On your project's page select Data acquisition --> Data sources --> + Add new data source --> Transformation block --> Whisper Synthetic Voice Generator --> Fill out the details as follow: ![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/generate-tts.jpg) @@ -104,14 +106,14 @@ Once satisfied with all the data generated, perform a Train / Test split into ap ![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/impulse-design.png) -The Impulse design values are chosen for optimal keyword spotting performance. The 968 ms window size captures enough audio for accurate detection, while the 500 ms window increase balances responsiveness and efficiency. The 16000 Hz frequency is standard for capturing human voice, ensuring quality without excessive data. Using the Audio (Syntiant) block leverages the NDP120's capabilities for efficient digital signal processing. The Classification block distinguishes between commands, with output classes "extinguish," "illuminate," and "z_openset" allowing for control of the lighting system and handling unknown inputs. +The Impulse design values are chosen for optimal keyword spotting performance. The 968 ms window size captures enough audio for accurate detection, while the 500 ms window increase balances responsiveness and efficiency. The 16000 Hz frequency is standard for capturing human voice, ensuring quality without excessive data. Using the Audio (Syntiant) block leverages the NDP120's capabilities for efficient digital signal processing. The Classification block distinguishes between commands, with output classes "extinguish," "illuminate," and "z\_openset" allowing for control of the lighting system and handling unknown inputs. -- Window size: 968 ms -- Window increase: 500 ms -- Frequency: 16000 Hz -- Audio (Syntiant) -- Classification -- Output: extinguish, illuminate, z_openset +* Window size: 968 ms +* Window increase: 500 ms +* Frequency: 16000 Hz +* Audio (Syntiant) +* Classification +* Output: extinguish, illuminate, z\_openset ![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/processing-feature.png) @@ -149,13 +151,13 @@ The schematic, pcb, and gerber (manufacturing) files are accessible in the proje ![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/pcb-order.png) -| LCSC Part Number | Manufacture Part Number | Manufacturer | Package | Description | Order Qty. | Unit Price($) | Order Price($) | -|--------------------|----------------------------------------------------------|--------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------|------------|---------------|----------------| -| C176224 | QR1206F5R10P05Z | Ever Ohms Tech | 1206 | 250mW Thick Film Resistors 200V ą1% ą400ppm/? 5.1? 1206 Chip Resistor - Surface Mount ROHS | 50 | 0.0156 | 0.78 | -| C516126 | HL-AM-2835H421W-S1-08-HR5(R9) (2800K-3100K)(SDCM<6,R9>50) | HONGLITRONIC(Hongli Zhihui (HONGLITRONIC)) | SMD2835 | 60mA 3000K Foggy yellow lens -40?~+85? Positive Stick White 120° 306mW 3.4V SMD2835 LED Indication - Discrete ROHS | 50 | 0.0144 | 0.72 | -| C2589 | IRLML2502TRPBF | Infineon Technologies | SOT-23 | 20V 4.2A 1.25W 45m?@4.5V,4.2A 1.2V@250uA 1 N-Channel SOT-23 MOSFETs ROHS | 5 | 0.1838 | 0.92 | -| C5440143 | CS3225X7R476K160NRL | Samwha Capacitor | 1210 | 16V 47uF X7R ą10% 1210 Multilayer Ceramic Capacitors MLCC - SMD/SMT ROHS | 5 | 0.0765 | 0.38 | -| C153338 | FCR1206J100RP05Z | Ever Ohms Tech | 1206 | 250mW Safety Resistor 200V ą5% 100? 1206 Chip Resistor - Surface Mount ROHS | 10 | 0.0541 | 0.54 | +| LCSC Part Number | Manufacture Part Number | Manufacturer | Package | Description | Order Qty. | Unit Price($) | Order Price($) | +| ---------------- | --------------------------------------------------------- | ------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------- | ---------- | ------------- | -------------- | +| C176224 | QR1206F5R10P05Z | Ever Ohms Tech | 1206 | 250mW Thick Film Resistors 200V ą1% ą400ppm/? 5.1? 1206 Chip Resistor - Surface Mount ROHS | 50 | 0.0156 | 0.78 | +| C516126 | HL-AM-2835H421W-S1-08-HR5(R9) (2800K-3100K)(SDCM<6,R9>50) | HONGLITRONIC(Hongli Zhihui (HONGLITRONIC)) | SMD2835 | 60mA 3000K Foggy yellow lens -40?\~+85? Positive Stick White 120° 306mW 3.4V SMD2835 LED Indication - Discrete ROHS | 50 | 0.0144 | 0.72 | +| C2589 | IRLML2502TRPBF | Infineon Technologies | SOT-23 | 20V 4.2A 1.25W 45m?@4.5V,4.2A 1.2V@250uA 1 N-Channel SOT-23 MOSFETs ROHS | 5 | 0.1838 | 0.92 | +| C5440143 | CS3225X7R476K160NRL | Samwha Capacitor | 1210 | 16V 47uF X7R ą10% 1210 Multilayer Ceramic Capacitors MLCC - SMD/SMT ROHS | 5 | 0.0765 | 0.38 | +| C153338 | FCR1206J100RP05Z | Ever Ohms Tech | 1206 | 250mW Safety Resistor 200V ą5% 100? 1206 Chip Resistor - Surface Mount ROHS | 10 | 0.0541 | 0.54 | ![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/3d-pcb.png) diff --git a/computer-vision-projects/README.md b/computer-vision-projects/README.md new file mode 100644 index 00000000..014804b4 --- /dev/null +++ b/computer-vision-projects/README.md @@ -0,0 +1,2 @@ +# Computer Vision Projects + diff --git a/image-projects/ai-leukemia-classifier-nvidia-jetson-nano.md b/computer-vision-projects/ai-leukemia-classifier-nvidia-jetson-nano.md similarity index 100% rename from image-projects/ai-leukemia-classifier-nvidia-jetson-nano.md rename to computer-vision-projects/ai-leukemia-classifier-nvidia-jetson-nano.md diff --git a/image-projects/analog-meter-reading-arduino-nicla-vision.md b/computer-vision-projects/analog-meter-reading-arduino-nicla-vision.md similarity index 100% rename from image-projects/analog-meter-reading-arduino-nicla-vision.md rename to computer-vision-projects/analog-meter-reading-arduino-nicla-vision.md diff --git a/image-projects/arduino-portenta-h7-byom-wound-classification.md b/computer-vision-projects/arduino-portenta-h7-byom-wound-classification.md similarity index 100% rename from image-projects/arduino-portenta-h7-byom-wound-classification.md rename to computer-vision-projects/arduino-portenta-h7-byom-wound-classification.md diff --git a/image-projects/arduino-portenta-h7-self-driving-rc-car.md b/computer-vision-projects/arduino-portenta-h7-self-driving-rc-car.md similarity index 100% rename from image-projects/arduino-portenta-h7-self-driving-rc-car.md rename to computer-vision-projects/arduino-portenta-h7-self-driving-rc-car.md diff --git a/image-projects/asl-byom-ti-am62a.md b/computer-vision-projects/asl-byom-ti-am62a.md similarity index 74% rename from image-projects/asl-byom-ti-am62a.md rename to computer-vision-projects/asl-byom-ti-am62a.md index 8ceb14b5..d465f647 100644 --- a/image-projects/asl-byom-ti-am62a.md +++ b/computer-vision-projects/asl-byom-ti-am62a.md @@ -1,6 +1,8 @@ --- description: >- - Use the "Bring Your Own Model" feature to upload an existing pre-trained TensorFlow Lite Sign Language model into Edge Impulse, then deploy it to the TI AM62A Development Kit. + Use the "Bring Your Own Model" feature to upload an existing pre-trained + TensorFlow Lite Sign Language model into Edge Impulse, then deploy it to the + TI AM62A Development Kit. --- # Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A @@ -25,13 +27,13 @@ I decided to try out this feature by importing an American Sign Language model a ![](../.gitbook/assets/asl-byom-ti-am62a/asl.jpg) -One thing I learned from this project is that sign language is not the same everywhere. In the US and most of Canada, they use American Sign Language, but in Argentina where I am located, they use a different version. Both versions have some signs that are static, and some that have movement, like the letter Z. For the moving signs, I only used the final position of the movement to train and recognize the letter, since I was using images and not videos. For motion activity, video classification would be needed. +One thing I learned from this project is that sign language is not the same everywhere. In the US and most of Canada, they use American Sign Language, but in Argentina where I am located, they use a different version. Both versions have some signs that are static, and some that have movement, like the letter Z. For the moving signs, I only used the final position of the movement to train and recognize the letter, since I was using images and not videos. For motion activity, video classification would be needed. Another thing to keep in mind is that some signs, like **A, M, N, S and T**, can look very similar from a distance. This makes data collection and model training more challenging. ## Model Import -From the original idea to the point where I actually started the project, the [ASL Hugging Face model]((https://huggingface.co/spaces/datasciencedojo/AmericanSignLanguage-Detection)) I wanted to use was no longer available. +From the original idea to the point where I actually started the project, the [ASL Hugging Face model](../image-projects/\(https:/huggingface.co/spaces/datasciencedojo/AmericanSignLanguage-Detection\)/) I wanted to use was no longer available. So at this point I decided to train a simple ASL model outside Edge Impulse, export the model as a TensorFlow Lite file, import to Edge Impulse, deploy the model to the Texas Instruments AM62A board, and develop a simple Python training game. @@ -51,7 +53,7 @@ Before importing to Edge Impulse, I tested the model with a live feed from the c For the exporting step in _Teachable Machine_, I used TensorFlow Lite, Floating Point. That generates a `.zip` file download, which I then unzipped. Inside, I found 2 files: the model with the `.tflite` extension, and a label .txt file. This label file has 2 columns: **order** and **label name**. To prepare everything for Edge Impulse BYOM, I removed the **order** column and compiled everything in one row, comma separated. -Example: +Example: ``` A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z @@ -91,16 +93,16 @@ The deployment procedure varies from board to board. In this case, the deploymen ## Texas Instruments AM62A Setup and Deployment - - Download this operating system image version: [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM62A/08.06.00.45](https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-D37Ls3JjkT/08.06.00.45/tisdk-edgeai-image-am62axx-evm.wic.xz) - - Flash the image to a 16gb or larger microSD card with Balena Etcher or any other similar software - - Connect the Power Supply, HDMI, USB Camera, and Ethernet Cable - - Check the board IP on the HDMI screen when board boots up and the default application loads - - Login to that IP using Putty or any other SSH client, using `root` as the user, and no password - - Run `npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm` - - Run `pip3 install art` (a library to display bigger letters) - - Run `edge-impulse-linux-runner`. The first time you run this, you will need to login to your Edge Impulse account and select the BYOM project. Once running, launch a web browser and navigate to your board's IP address, port 4912. For example, `http://192.168.1.66:4912` in my case. - - Download the `am62a_signlanguage.py` file [from the GitHub repository](https://github.com/ronibandini/ASLTrainer) and upload the script to the AM62A board using SFTP. The credentials are the same as logging in directly: You'll need your IP address, username is `root`, and there is no password. - - Run `python3 am62a_signlanguage.py` +* Download this operating system image version: [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM62A/08.06.00.45](https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-D37Ls3JjkT/08.06.00.45/tisdk-edgeai-image-am62axx-evm.wic.xz) +* Flash the image to a 16gb or larger microSD card with Balena Etcher or any other similar software +* Connect the Power Supply, HDMI, USB Camera, and Ethernet Cable +* Check the board IP on the HDMI screen when board boots up and the default application loads +* Login to that IP using Putty or any other SSH client, using `root` as the user, and no password +* Run `npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm` +* Run `pip3 install art` (a library to display bigger letters) +* Run `edge-impulse-linux-runner`. The first time you run this, you will need to login to your Edge Impulse account and select the BYOM project. Once running, launch a web browser and navigate to your board's IP address, port 4912. For example, `http://192.168.1.66:4912` in my case. +* Download the `am62a_signlanguage.py` file [from the GitHub repository](https://github.com/ronibandini/ASLTrainer) and upload the script to the AM62A board using SFTP. The credentials are the same as logging in directly: You'll need your IP address, username is `root`, and there is no password. +* Run `python3 am62a_signlanguage.py` ## Training Game @@ -130,9 +132,9 @@ It is important to note that Edge Impulse's visualizations and tuning for extern ### Files -- Source Code: [https://github.com/ronibandini/ASLTrainer](https://github.com/ronibandini/ASLTrainer) -- Edge Impulse Public Project: [https://studio.edgeimpulse.com/public/270046/latest](https://studio.edgeimpulse.com/public/270046/latest) -- [https://huggingface.co/ronibandini/AmericanSignLanguage](https://huggingface.co/ronibandini/AmericanSignLanguage) (this model has all the letters) +* Source Code: [https://github.com/ronibandini/ASLTrainer](https://github.com/ronibandini/ASLTrainer) +* Edge Impulse Public Project: [https://studio.edgeimpulse.com/public/270046/latest](https://studio.edgeimpulse.com/public/270046/latest) +* [https://huggingface.co/ronibandini/AmericanSignLanguage](https://huggingface.co/ronibandini/AmericanSignLanguage) (this model has all the letters) ### References @@ -140,11 +142,12 @@ It is important to note that Edge Impulse's visualizations and tuning for extern ### Contact -If you are interested in other Artificial Intelligence and Machine Learning projects: [https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP](https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP) +If you are interested in other Artificial Intelligence and Machine Learning projects: [https://www.youtube.com/playlist?list=PLIw\_UcVWFyBVYAQXp8S2pfe2frzGcyxlP](https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP) #### Social Media -[https://twitter.com/RoniBandini](https://twitter.com/RoniBandini) -[https://www.instagram.com/ronibandini](https://www.instagram.com/ronibandini) -#### Web +[https://twitter.com/RoniBandini](https://twitter.com/RoniBandini) [https://www.instagram.com/ronibandini](https://www.instagram.com/ronibandini) + +#### Web + [https://bandini.medium.com/](https://bandini.medium.com/) diff --git a/image-projects/bean-leaf-disease-classification-sony-spresense.md b/computer-vision-projects/bean-leaf-disease-classification-sony-spresense.md similarity index 100% rename from image-projects/bean-leaf-disease-classification-sony-spresense.md rename to computer-vision-projects/bean-leaf-disease-classification-sony-spresense.md diff --git a/image-projects/bicycle-counting-sony-spresense.md b/computer-vision-projects/bicycle-counting-sony-spresense.md similarity index 100% rename from image-projects/bicycle-counting-sony-spresense.md rename to computer-vision-projects/bicycle-counting-sony-spresense.md diff --git a/image-projects/bike-rearview-radar-raspberry-pi.md b/computer-vision-projects/bike-rearview-radar-raspberry-pi.md similarity index 100% rename from image-projects/bike-rearview-radar-raspberry-pi.md rename to computer-vision-projects/bike-rearview-radar-raspberry-pi.md diff --git a/image-projects/brainchip-akida-industrial-inspection.md b/computer-vision-projects/brainchip-akida-industrial-inspection.md similarity index 100% rename from image-projects/brainchip-akida-industrial-inspection.md rename to computer-vision-projects/brainchip-akida-industrial-inspection.md diff --git a/image-projects/brainchip-akida-multi-camera-inference.md b/computer-vision-projects/brainchip-akida-multi-camera-inference.md similarity index 100% rename from image-projects/brainchip-akida-multi-camera-inference.md rename to computer-vision-projects/brainchip-akida-multi-camera-inference.md diff --git a/image-projects/brainchip-akida-traffic-monitoring.md b/computer-vision-projects/brainchip-akida-traffic-monitoring.md similarity index 100% rename from image-projects/brainchip-akida-traffic-monitoring.md rename to computer-vision-projects/brainchip-akida-traffic-monitoring.md diff --git a/image-projects/brainchip-akida-xray-classification.md b/computer-vision-projects/brainchip-akida-xray-classification.md similarity index 100% rename from image-projects/brainchip-akida-xray-classification.md rename to computer-vision-projects/brainchip-akida-xray-classification.md diff --git a/image-projects/container-counting-arduino-nicla-vision.md b/computer-vision-projects/container-counting-arduino-nicla-vision.md similarity index 100% rename from image-projects/container-counting-arduino-nicla-vision.md rename to computer-vision-projects/container-counting-arduino-nicla-vision.md diff --git a/image-projects/corrosion-detection-computer-vision-seeed-reterminal.md b/computer-vision-projects/corrosion-detection-computer-vision-seeed-reterminal.md similarity index 100% rename from image-projects/corrosion-detection-computer-vision-seeed-reterminal.md rename to computer-vision-projects/corrosion-detection-computer-vision-seeed-reterminal.md diff --git a/image-projects/counting-eggs-openmv-cam-h7.md b/computer-vision-projects/counting-eggs-openmv-cam-h7.md similarity index 100% rename from image-projects/counting-eggs-openmv-cam-h7.md rename to computer-vision-projects/counting-eggs-openmv-cam-h7.md diff --git a/image-projects/cyclist-blind-spot-detection-himax-we-i-plus.md b/computer-vision-projects/cyclist-blind-spot-detection-himax-we-i-plus.md similarity index 100% rename from image-projects/cyclist-blind-spot-detection-himax-we-i-plus.md rename to computer-vision-projects/cyclist-blind-spot-detection-himax-we-i-plus.md diff --git a/image-projects/delivered-package-detection-esp-eye.md b/computer-vision-projects/delivered-package-detection-esp-eye.md similarity index 100% rename from image-projects/delivered-package-detection-esp-eye.md rename to computer-vision-projects/delivered-package-detection-esp-eye.md diff --git a/image-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md b/computer-vision-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md similarity index 100% rename from image-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md rename to computer-vision-projects/deter-shoplifting-with-computer-vision-ti-tda4vm.md diff --git a/image-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md b/computer-vision-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md similarity index 100% rename from image-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md rename to computer-vision-projects/dials-and-knob-monitoring-with-computer-vision-raspberry-pi.md diff --git a/image-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md b/computer-vision-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md similarity index 100% rename from image-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md rename to computer-vision-projects/digital-character-recognition-on-electric-meter-system-openmv-cam-h7.md diff --git a/image-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md b/computer-vision-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md similarity index 100% rename from image-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md rename to computer-vision-projects/driver-drowsiness-detection-with-computer-vision-arduino-nicla-vision.md diff --git a/image-projects/elevator-passenger-counting-arduino-nicla-vision.md b/computer-vision-projects/elevator-passenger-counting-arduino-nicla-vision.md similarity index 100% rename from image-projects/elevator-passenger-counting-arduino-nicla-vision.md rename to computer-vision-projects/elevator-passenger-counting-arduino-nicla-vision.md diff --git a/image-projects/esd-protection-using-computer-vision-seeed-recomputer.md b/computer-vision-projects/esd-protection-using-computer-vision-seeed-recomputer.md similarity index 100% rename from image-projects/esd-protection-using-computer-vision-seeed-recomputer.md rename to computer-vision-projects/esd-protection-using-computer-vision-seeed-recomputer.md diff --git a/image-projects/fomo-ad-product-inspection-spresense.md b/computer-vision-projects/fomo-ad-product-inspection-spresense.md similarity index 100% rename from image-projects/fomo-ad-product-inspection-spresense.md rename to computer-vision-projects/fomo-ad-product-inspection-spresense.md diff --git a/image-projects/fomo-ad-ti-tda4vm.md b/computer-vision-projects/fomo-ad-ti-tda4vm.md similarity index 88% rename from image-projects/fomo-ad-ti-tda4vm.md rename to computer-vision-projects/fomo-ad-ti-tda4vm.md index 2926da77..852cda05 100644 --- a/image-projects/fomo-ad-ti-tda4vm.md +++ b/computer-vision-projects/fomo-ad-ti-tda4vm.md @@ -1,6 +1,7 @@ --- description: >- - Use computer vision and a Texas Instruments TDA4VM to perform anomaly detection for quality assurance of parts. + Use computer vision and a Texas Instruments TDA4VM to perform anomaly + detection for quality assurance of parts. --- # Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM @@ -19,19 +20,19 @@ Quality control is a critical aspect of any production line or manufacturing pro Artificial intelligence and specifically a Machine Learning algorithm named FOMO-AD (Faster Objects, More Objects - Anomaly Detection) by Edge Impulse could be instrumental in addressing this challenge. -## How Does Visual Anomaly Detection Work? +## How Does Visual Anomaly Detection Work? ![](../.gitbook/assets/fomo-ad-ti-tda4vm/how-it-works.jpg) -Detecting anomalies visually can be a challenging task. While we understand what a standard item should look like, identifying faulty items that exhibit previously unseen anomalies poses a unique set of difficulties. +Detecting anomalies visually can be a challenging task. While we understand what a standard item should look like, identifying faulty items that exhibit previously unseen anomalies poses a unique set of difficulties. -Imagine having to collect samples for every conceivable anomaly that might occur. This is called dataset asymmetry: you end up with a huge dataset of valid items and a tiny dataset of faulty items. +Imagine having to collect samples for every conceivable anomaly that might occur. This is called dataset asymmetry: you end up with a huge dataset of valid items and a tiny dataset of faulty items. One solution is to make use of unsupervised learning techniques. These methods allow us to build models without relying on labeled data, that has been specifically identified and labeled as faulty. Instead, they learn from the overall distribution of data points, identifying patterns and deviations. Once we've trained our anomaly detection model, we can deploy it to small devices with limited hardware resources. These lightweight models consume minimal power, and can even operate without an internet connection. -Under the hood, a clustering technique named Gaussian Mixture Models is used. Data points with low probabilities are identified and if a data point has a significantly lower probability of being generated by the mixture model compared to most other data points, it is considered an anomaly and it will output a high anomaly score. +Under the hood, a clustering technique named Gaussian Mixture Models is used. Data points with low probabilities are identified and if a data point has a significantly lower probability of being generated by the mixture model compared to most other data points, it is considered an anomaly and it will output a high anomaly score. ![](../.gitbook/assets/fomo-ad-ti-tda4vm/cluster.jpg) @@ -43,14 +44,14 @@ One interesting thing about FOMO-AD is that it detects not only the presence of ## Hardware and Software Setup -FOMO-AD requires an Enterprise version of Edge Impulse. To explore the capability, you can sign up for a free Enterprise Trial at [https://studio.edgeimpulse.com/trial-signup](https://studio.edgeimpulse.com/trial-signup), then create a new project named _Visual Anomaly_. +FOMO-AD requires an Enterprise version of Edge Impulse. To explore the capability, you can sign up for a free Enterprise Trial at [https://studio.edgeimpulse.com/trial-signup](https://studio.edgeimpulse.com/trial-signup), then create a new project named _Visual Anomaly_. -Download the Python parser script at [https://github.com/ronibandini/visualAnomaly](https://github.com/ronibandini/visualAnomaly), or `git clone` the full repo. +Download the Python parser script at [https://github.com/ronibandini/visualAnomaly](https://github.com/ronibandini/visualAnomaly), or `git clone` the full repo. -I am using the [Texas Instruments TDA4VM development board](https://docs.edgeimpulse.com/docs/edge-ai-hardware/cpu-+-ai-accelerators/sk-tda4vm) in this project, so I will download the OS image from [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/08.06.00.11](https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/08.06.00.11). The TI AM62A and AM68A Development Kits could also be used in a similar fashion as described in this project. +I am using the [Texas Instruments TDA4VM development board](https://docs.edgeimpulse.com/docs/edge-ai-hardware/cpu-+-ai-accelerators/sk-tda4vm) in this project, so I will download the OS image from [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/08.06.00.11](https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/08.06.00.11). The TI AM62A and AM68A Development Kits could also be used in a similar fashion as described in this project. > Note: Download version 8 (08.06.00.11 as indicated by the URL), as the latest version of the image doesn't seem to detect the USB camera while executing the Edge Impulse Linux Runner later. - + Extract and flash the image with Etcher or any other similar application. ![](../.gitbook/assets/fomo-ad-ti-tda4vm/flash-sd.jpg) @@ -61,8 +62,8 @@ Place the microSD card into the TDA4VM. Connect a USB-C power supply with at lea Determine the IP address that your router assigned to the board, and then connect to the TDA4VM using SSH with the following credentials: -- User: root -- Password: (empty) +* User: root +* Password: (empty) ![](../.gitbook/assets/fomo-ad-ti-tda4vm/ssh.jpg) @@ -83,17 +84,17 @@ Then install Edge Impulse: npm install -g --unsafe-perm edge-impulse-linux ``` -Now you can check your camera focus by opening the stream at http://your-ip-address:4912. For example, mine was http://192.168.1.67:4912. +Now you can check your camera focus by opening the stream at http://your-ip-address:4912. For example, mine was http://192.168.1.67:4912. ## Data Collection ![](../.gitbook/assets/fomo-ad-ti-tda4vm/parts.jpg) -For this sample project the idea is to spot and alert about faulty electricity components in a production line. The dataset to be used is composed of hundreds of pictures of properly assembled, high quality items. +For this sample project the idea is to spot and alert about faulty electricity components in a production line. The dataset to be used is composed of hundreds of pictures of properly assembled, high quality items. -- Upload around 100 pictures of correct, quality products to Edge Impulse using **No Anomaly** as the label -- In Impusle Design, Select Image Data, 96x96 pixels, and **Squash** as the resize mode. -- Select an Image processing block, and choose **FOMO-AD**. +* Upload around 100 pictures of correct, quality products to Edge Impulse using **No Anomaly** as the label +* In Impusle Design, Select Image Data, 96x96 pixels, and **Squash** as the resize mode. +* Select an Image processing block, and choose **FOMO-AD**. ![](../.gitbook/assets/fomo-ad-ti-tda4vm/impulse.jpg) @@ -103,7 +104,7 @@ Proceed to **Generate features** and then configure the training settings. **Capacity setting**: the higher the capacity, the higher the number of Gaussian components, and the more adapted the model becomes to the original distribution. -For obvious reasons there should not be any anomalies in the training dataset and accuracy is not calculated here. +For obvious reasons there should not be any anomalies in the training dataset and accuracy is not calculated here. To calculate F1 score if necessary, you could later upload a set of anomaly pictures and label them as **Anomaly**. @@ -165,13 +166,13 @@ The JSON will be parsed and a text matrix will be displayed showing the anomaly ![](../.gitbook/assets/fomo-ad-ti-tda4vm/limitations.jpg) -One limitation for Visual Anomaly Detection is having a dataset of noisy images. Convolutions can have trouble with these kinds of images and predictions may not be reliable. +One limitation for Visual Anomaly Detection is having a dataset of noisy images. Convolutions can have trouble with these kinds of images and predictions may not be reliable. -Another limitation is trying to spot things that are larger than receptive fields. In this case 2 models should be deployed, with the second model backing up the first the model, to add trust to the first model. +Another limitation is trying to spot things that are larger than receptive fields. In this case 2 models should be deployed, with the second model backing up the first the model, to add trust to the first model. ## Final Notes -In conclusion, embracing FOMO-AD can revolutionize quality control, making it more efficient, accurate, and less burdensome for human workers.  +In conclusion, embracing FOMO-AD can revolutionize quality control, making it more efficient, accurate, and less burdensome for human workers. ![](../.gitbook/assets/fomo-ad-ti-tda4vm/conclusion.jpg) @@ -181,7 +182,7 @@ In conclusion, embracing FOMO-AD can revolutionize quality control, making it mo ## References -- [https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html](https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html) -- [https://www.youtube.com/watch?v=se9ZDBVKN1M](https://www.youtube.com/watch?v=se9ZDBVKN1M) -- [https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse](https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse) -- [https://cumulusds.com/unveiling-the-hidden-cost-of-poor-quality-copq-on-construction-and-maintenance-projects](https://cumulusds.com/unveiling-the-hidden-cost-of-poor-quality-copq-on-construction-and-maintenance-projects) +* [https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html](https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html) +* [https://www.youtube.com/watch?v=se9ZDBVKN1M](https://www.youtube.com/watch?v=se9ZDBVKN1M) +* [https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse](https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse) +* [https://cumulusds.com/unveiling-the-hidden-cost-of-poor-quality-copq-on-construction-and-maintenance-projects](https://cumulusds.com/unveiling-the-hidden-cost-of-poor-quality-copq-on-construction-and-maintenance-projects) diff --git a/image-projects/fomo-stock-tracker-brainchip.md b/computer-vision-projects/fomo-stock-tracker-brainchip.md similarity index 100% rename from image-projects/fomo-stock-tracker-brainchip.md rename to computer-vision-projects/fomo-stock-tracker-brainchip.md diff --git a/image-projects/gastroscopic-image-processing-openmv-cam-h7.md b/computer-vision-projects/gastroscopic-image-processing-openmv-cam-h7.md similarity index 100% rename from image-projects/gastroscopic-image-processing-openmv-cam-h7.md rename to computer-vision-projects/gastroscopic-image-processing-openmv-cam-h7.md diff --git a/image-projects/helmet-detection-alif-ensemble.md b/computer-vision-projects/helmet-detection-alif-ensemble.md similarity index 100% rename from image-projects/helmet-detection-alif-ensemble.md rename to computer-vision-projects/helmet-detection-alif-ensemble.md diff --git a/image-projects/inventory-management-with-computer-vision-raspberry-pi.md b/computer-vision-projects/inventory-management-with-computer-vision-raspberry-pi.md similarity index 100% rename from image-projects/inventory-management-with-computer-vision-raspberry-pi.md rename to computer-vision-projects/inventory-management-with-computer-vision-raspberry-pi.md diff --git a/image-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md b/computer-vision-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md similarity index 100% rename from image-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md rename to computer-vision-projects/iv-drip-fluid-level-monitoring-arduino-portenta-h7.md diff --git a/image-projects/label-defect-detection-raspberry-pi.md b/computer-vision-projects/label-defect-detection-raspberry-pi.md similarity index 100% rename from image-projects/label-defect-detection-raspberry-pi.md rename to computer-vision-projects/label-defect-detection-raspberry-pi.md diff --git a/image-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md b/computer-vision-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md similarity index 100% rename from image-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md rename to computer-vision-projects/motorcycle-helmet-detection-smart-light-ti-am62a.md diff --git a/image-projects/nvidia-omniverse-replicator.md b/computer-vision-projects/nvidia-omniverse-replicator.md similarity index 100% rename from image-projects/nvidia-omniverse-replicator.md rename to computer-vision-projects/nvidia-omniverse-replicator.md diff --git a/image-projects/object-counting-fomo-openmv-rt1062.md b/computer-vision-projects/object-counting-fomo-openmv-rt1062.md similarity index 100% rename from image-projects/object-counting-fomo-openmv-rt1062.md rename to computer-vision-projects/object-counting-fomo-openmv-rt1062.md diff --git a/image-projects/object-counting-for-smart-industries-raspberry-pi.md b/computer-vision-projects/object-counting-for-smart-industries-raspberry-pi.md similarity index 100% rename from image-projects/object-counting-for-smart-industries-raspberry-pi.md rename to computer-vision-projects/object-counting-for-smart-industries-raspberry-pi.md diff --git a/image-projects/object-detection-ubidots-seeed-grove-ai.md b/computer-vision-projects/object-detection-ubidots-seeed-grove-ai.md similarity index 100% rename from image-projects/object-detection-ubidots-seeed-grove-ai.md rename to computer-vision-projects/object-detection-ubidots-seeed-grove-ai.md diff --git a/image-projects/oil-tank-gauge-monitoring-sony-spresense.md b/computer-vision-projects/oil-tank-gauge-monitoring-sony-spresense.md similarity index 100% rename from image-projects/oil-tank-gauge-monitoring-sony-spresense.md rename to computer-vision-projects/oil-tank-gauge-monitoring-sony-spresense.md diff --git a/image-projects/particle-photon-2-sonicsight-ar.md b/computer-vision-projects/particle-photon-2-sonicsight-ar.md similarity index 100% rename from image-projects/particle-photon-2-sonicsight-ar.md rename to computer-vision-projects/particle-photon-2-sonicsight-ar.md diff --git a/image-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md b/computer-vision-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md similarity index 100% rename from image-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md rename to computer-vision-projects/pcb-defect-detection-with-computer-vision-raspberry-pi.md diff --git a/image-projects/pharmaceutical-pill-defect-detection.md b/computer-vision-projects/pharmaceutical-pill-defect-detection.md similarity index 100% rename from image-projects/pharmaceutical-pill-defect-detection.md rename to computer-vision-projects/pharmaceutical-pill-defect-detection.md diff --git a/image-projects/recyclable-materials-sorter-nvidia-jetson-nano.md b/computer-vision-projects/recyclable-materials-sorter-nvidia-jetson-nano.md similarity index 100% rename from image-projects/recyclable-materials-sorter-nvidia-jetson-nano.md rename to computer-vision-projects/recyclable-materials-sorter-nvidia-jetson-nano.md diff --git a/image-projects/renesas-rzv2l-counting-retail-inventory.md b/computer-vision-projects/renesas-rzv2l-counting-retail-inventory.md similarity index 100% rename from image-projects/renesas-rzv2l-counting-retail-inventory.md rename to computer-vision-projects/renesas-rzv2l-counting-retail-inventory.md diff --git a/image-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md b/computer-vision-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md similarity index 100% rename from image-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md rename to computer-vision-projects/renesas-rzv2l-monitoring-retail-checkout-lines.md diff --git a/image-projects/renesas-rzv2l-pose-detection.md b/computer-vision-projects/renesas-rzv2l-pose-detection.md similarity index 100% rename from image-projects/renesas-rzv2l-pose-detection.md rename to computer-vision-projects/renesas-rzv2l-pose-detection.md diff --git a/image-projects/renesas-rzv2l-product-quality-inspection.md b/computer-vision-projects/renesas-rzv2l-product-quality-inspection.md similarity index 100% rename from image-projects/renesas-rzv2l-product-quality-inspection.md rename to computer-vision-projects/renesas-rzv2l-product-quality-inspection.md diff --git a/image-projects/retail-image-classification-nvidia-jetson-nano.md b/computer-vision-projects/retail-image-classification-nvidia-jetson-nano.md similarity index 100% rename from image-projects/retail-image-classification-nvidia-jetson-nano.md rename to computer-vision-projects/retail-image-classification-nvidia-jetson-nano.md diff --git a/image-projects/shieldbot-autonomous-security-robot.md b/computer-vision-projects/shieldbot-autonomous-security-robot.md similarity index 100% rename from image-projects/shieldbot-autonomous-security-robot.md rename to computer-vision-projects/shieldbot-autonomous-security-robot.md diff --git a/image-projects/silabs-xg24-card-sorting-and-robotics-1.md b/computer-vision-projects/silabs-xg24-card-sorting-and-robotics-1.md similarity index 100% rename from image-projects/silabs-xg24-card-sorting-and-robotics-1.md rename to computer-vision-projects/silabs-xg24-card-sorting-and-robotics-1.md diff --git a/image-projects/silabs-xg24-card-sorting-and-robotics-2.md b/computer-vision-projects/silabs-xg24-card-sorting-and-robotics-2.md similarity index 100% rename from image-projects/silabs-xg24-card-sorting-and-robotics-2.md rename to computer-vision-projects/silabs-xg24-card-sorting-and-robotics-2.md diff --git a/image-projects/smart-cashier-with-fomo-raspberry-pi.md b/computer-vision-projects/smart-cashier-with-fomo-raspberry-pi.md similarity index 100% rename from image-projects/smart-cashier-with-fomo-raspberry-pi.md rename to computer-vision-projects/smart-cashier-with-fomo-raspberry-pi.md diff --git a/image-projects/smart-factory-prototype-ti-tda4vm.md b/computer-vision-projects/smart-factory-prototype-ti-tda4vm.md similarity index 100% rename from image-projects/smart-factory-prototype-ti-tda4vm.md rename to computer-vision-projects/smart-factory-prototype-ti-tda4vm.md diff --git a/image-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md b/computer-vision-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md similarity index 100% rename from image-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md rename to computer-vision-projects/smart-grocery-cart-with-computer-vision-openmv-cam-h7.md diff --git a/image-projects/smart-smoke-alarm-arduino-nano-33.md b/computer-vision-projects/smart-smoke-alarm-arduino-nano-33.md similarity index 100% rename from image-projects/smart-smoke-alarm-arduino-nano-33.md rename to computer-vision-projects/smart-smoke-alarm-arduino-nano-33.md diff --git a/image-projects/solar-panel-defect-detection-arduino-portenta-h7.md b/computer-vision-projects/solar-panel-defect-detection-arduino-portenta-h7.md similarity index 100% rename from image-projects/solar-panel-defect-detection-arduino-portenta-h7.md rename to computer-vision-projects/solar-panel-defect-detection-arduino-portenta-h7.md diff --git a/image-projects/surface-crack-detection-seeed-reterminal.md b/computer-vision-projects/surface-crack-detection-seeed-reterminal.md similarity index 100% rename from image-projects/surface-crack-detection-seeed-reterminal.md rename to computer-vision-projects/surface-crack-detection-seeed-reterminal.md diff --git a/image-projects/surface-crack-detection-ti-tda4vm.md b/computer-vision-projects/surface-crack-detection-ti-tda4vm.md similarity index 100% rename from image-projects/surface-crack-detection-ti-tda4vm.md rename to computer-vision-projects/surface-crack-detection-ti-tda4vm.md diff --git a/image-projects/textile-fabric-anomaly-detection.md b/computer-vision-projects/textile-fabric-anomaly-detection.md similarity index 100% rename from image-projects/textile-fabric-anomaly-detection.md rename to computer-vision-projects/textile-fabric-anomaly-detection.md diff --git a/image-projects/ti-tda4vm-posture-detection.md b/computer-vision-projects/ti-tda4vm-posture-detection.md similarity index 100% rename from image-projects/ti-tda4vm-posture-detection.md rename to computer-vision-projects/ti-tda4vm-posture-detection.md diff --git a/image-projects/vehicle-detection-raspberry-pi-ai-kit.md b/computer-vision-projects/vehicle-detection-raspberry-pi-ai-kit.md similarity index 100% rename from image-projects/vehicle-detection-raspberry-pi-ai-kit.md rename to computer-vision-projects/vehicle-detection-raspberry-pi-ai-kit.md diff --git a/image-projects/visual-anomaly-detection-seeed-vision-ai-2.md b/computer-vision-projects/visual-anomaly-detection-seeed-vision-ai-2.md similarity index 86% rename from image-projects/visual-anomaly-detection-seeed-vision-ai-2.md rename to computer-vision-projects/visual-anomaly-detection-seeed-vision-ai-2.md index 6d42e07c..f80ed5b2 100644 --- a/image-projects/visual-anomaly-detection-seeed-vision-ai-2.md +++ b/computer-vision-projects/visual-anomaly-detection-seeed-vision-ai-2.md @@ -1,6 +1,7 @@ --- description: >- - Perform visual anomaly detection using the Arm Ethos NPU found in the Seeed Grove Vision AI Module V2. + Perform visual anomaly detection using the Arm Ethos NPU found in the Seeed + Grove Vision AI Module V2. --- # Visual Anomaly Detection - Seeed Grove Vision AI Module V2 @@ -19,15 +20,15 @@ In the 1970s, a Russian engineer named Bikov trained pigeons for visual anomaly ## Parts Required -For this project I will use the [Seeed Grove Vision AI Module V2](https://www.seeedstudio.com/Grove-Vision-AI-V2-Kit-p-5852.html). This module has a WiseEye2 HX6538 processor with a dual-core Arm Cortex-M55 and integrated Arm Ethos-U55 neural processing unit (NPU). It also has 16mb flash memory, a PDM microphone, SD card slot, USB Type-C, and Grove interface. +For this project I will use the [Seeed Grove Vision AI Module V2](https://www.seeedstudio.com/Grove-Vision-AI-V2-Kit-p-5852.html). This module has a WiseEye2 HX6538 processor with a dual-core Arm Cortex-M55 and integrated Arm Ethos-U55 neural processing unit (NPU). It also has 16mb flash memory, a PDM microphone, SD card slot, USB Type-C, and Grove interface. For the inspection servo arm, I will use a separate module made with an Arduino Nano and an SG90 servo motor. -- Seeed Grove Vision AI Module V2 -- OV5647-62 Raspberry Pi Cam -- CSI Cable -- Arduino Nano -- Servo motor SG90 +* Seeed Grove Vision AI Module V2 +* OV5647-62 Raspberry Pi Cam +* CSI Cable +* Arduino Nano +* Servo motor SG90 ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/architecture.jpg) @@ -39,7 +40,7 @@ Connect the cam to the module using a CSI cable. ## Model Training -FOMO-AD requires an Enterprise version of Edge Impulse. To explore the capability, you can sign up for a free Enterprise Trial at [https://studio.edgeimpulse.com/trial-signup](https://studio.edgeimpulse.com/trial-signup), then create a new project named **Visual Anomaly**. +FOMO-AD requires an Enterprise version of Edge Impulse. To explore the capability, you can sign up for a free Enterprise Trial at [https://studio.edgeimpulse.com/trial-signup](https://studio.edgeimpulse.com/trial-signup), then create a new project named **Visual Anomaly**. The Seeed Grove Vision AI V2 board does not come with Edge Impulse firmware out-of-the-box, so it will need to be flashed. To update the firmware, [download this file](https://cdn.edgeimpulse.com/firmware/seeed-grove-vision-ai-module-v2.zip) and extract it. @@ -47,17 +48,17 @@ Connect the Grove Vision AI V2 to a PC/Mac/Linux via a USB Type-C cable. Within the contents of the extracted firmware `.zip` file, there are install scripts that will flash your device: -- For MacOS run `flash_mac.command` -- For Windows run `flash_windows.bat` -- For Linux run `flash_linux.sh` +* For MacOS run `flash_mac.command` +* For Windows run `flash_windows.bat` +* For Linux run `flash_linux.sh` Now install the Edge Impulse CLI: 1. For Windows, [install Python 3](https://www.python.org/) (it is required later for the parsing script) 2. Install [Node.js](https://nodejs.org/en/) -3. Open a terminal and execute `npm install -g edge-impulse-cli --force` +3. Open a terminal and execute `npm install -g edge-impulse-cli --force` -For other OS'es, visit [https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli/cli-installation](https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli/cli-installation) +For other OS'es, visit [https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli/cli-installation](https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli/cli-installation) 4. Run `edge-impulse-daemon`, login to Edge Impulse, and select your project to start collecting data. @@ -65,9 +66,9 @@ For other OS'es, visit [https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/daemon.jpg) -- Sample around 100 pictures using `no anomaly` as the label. -- Select Image Data, 96x96 pixels, "Squash" as the resize mode, and **Visual Anomaly Detection** as the Learn block. -- Select Image, RGB Image, Generate Features. +* Sample around 100 pictures using `no anomaly` as the label. +* Select Image Data, 96x96 pixels, "Squash" as the resize mode, and **Visual Anomaly Detection** as the Learn block. +* Select Image, RGB Image, Generate Features. ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/features.jpg) @@ -77,7 +78,7 @@ For other OS'es, visit [https://docs.edgeimpulse.com/docs/tools/edge-impulse-cli > Capacity setting: the higher the capacity, the higher the number of Gaussian components, and the more adapted the model becomes to the original distribution. -There should not be any anomalies in the training dataset, as we are only interested in "good" examples, thus accuracy is not calculated here. +There should not be any anomalies in the training dataset, as we are only interested in "good" examples, thus accuracy is not calculated here. To calculate an F1 score, after the training has completed, you can upload a set of **anomaly** pictures and label them as `anomaly`. @@ -89,19 +90,19 @@ Then you can click **Classify All** and analyze both the global results and spec ## Deployment -On the **Deployment** page, select **Seeed Grove Vision AI Module V2**. +On the **Deployment** page, select **Seeed Grove Vision AI Module V2**. > Note: that there is also a `Seeed Grove Vision AI` for the previous version of the hardware. That should **not** be selected. ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/deployment.jpg) -Download the `.zip` file by clicking **Build**, and then extract the contents. Connect the Grove Vision AI Module V2 to your computer using a USB-C cable. +Download the `.zip` file by clicking **Build**, and then extract the contents. Connect the Grove Vision AI Module V2 to your computer using a USB-C cable. -Like before, run the flashing script inside of the extracted `.zip` file that corresponds to your OS. If you get a security warning, you will need to Allow or Accept in order to continue with the flashing of the device. +Like before, run the flashing script inside of the extracted `.zip` file that corresponds to your OS. If you get a security warning, you will need to Allow or Accept in order to continue with the flashing of the device. ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/flashing.jpg) -Now the Grove Vision AI Module V2 is ready to take pictures and identify visual anomalies. +Now the Grove Vision AI Module V2 is ready to take pictures and identify visual anomalies. ## Results @@ -151,12 +152,12 @@ As we saw in this project, implementing automated visual anomaly detection for a ![](../.gitbook/assets/visual-anomaly-detection-seeed-vision-ai-2/results.jpg) -Besides using this firmware method, you can also deploy the Edge Impulse model as an Arduino Library, add a XIAO ESP32 to the Grove Module V2, and set up a tiny offline, standalone Visual Anomaly Detection device. +Besides using this firmware method, you can also deploy the Edge Impulse model as an Arduino Library, add a XIAO ESP32 to the Grove Module V2, and set up a tiny offline, standalone Visual Anomaly Detection device. ## Links -- Project: [https://studio.edgeimpulse.com/public/513864/live](https://studio.edgeimpulse.com/public/513864/live) -- Source Code: [https://github.com/ronibandini/visualAnomalyGroveV2](https://github.com/ronibandini/visualAnomalyGroveV2) +* Project: [https://studio.edgeimpulse.com/public/513864/live](https://studio.edgeimpulse.com/public/513864/live) +* Source Code: [https://github.com/ronibandini/visualAnomalyGroveV2](https://github.com/ronibandini/visualAnomalyGroveV2) Demo video: @@ -164,12 +165,6 @@ Demo video: ## References -- [https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse](https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse) -- [https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu-+-ai-accelerators/himax-seeed-grove-vision-ai-module-v2-wise-eye-2#deploy-model-to-seeed-grove-vision-ai-module-v2-himax-wiseeye2]([https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu-+-ai-accelerators/himax-seeed-grove-vision-ai-module-v2-wise-eye-2#deploy-model-to-seeed-grove-vision-ai-module-v2-himax-wiseeye2) -- [https://bandini.medium.com/ia-palomas-y-detecci%C3%B3n-de-anomal%C3%ADas-a9870850795a](https://bandini.medium.com/ia-palomas-y-detecci%C3%B3n-de-anomal%C3%ADas-a9870850795a) - - - - - - +* [https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse](https://www.edge-ai-vision.com/2023/06/visual-anomaly-detection-with-fomo-ad-a-presentation-from-edge-impulse) +* [https://docs.edgeimpulse.com/docs/edge-ai-hardware/mcu-+-ai-accelerators/himax-seeed-grove-vision-ai-module-v2-wise-eye-2#deploy-model-to-seeed-grove-vision-ai-module-v2-himax-wiseeye2](../image-projects/\[https:/docs.edgeimpulse.com/docs/edge-ai-hardware/mcu-+-ai-accelerators/himax-seeed-grove-vision-ai-module-v2-wise-eye-2/#deploy-model-to-seeed-grove-vision-ai-module-v2-himax-wiseeye2) +* [https://bandini.medium.com/ia-palomas-y-detecci%C3%B3n-de-anomal%C3%ADas-a9870850795a](https://bandini.medium.com/ia-palomas-y-detecci%C3%B3n-de-anomal%C3%ADas-a9870850795a) diff --git a/image-projects/worker-safety-monitoring-nvidia-jetson-nano.md b/computer-vision-projects/worker-safety-monitoring-nvidia-jetson-nano.md similarity index 100% rename from image-projects/worker-safety-monitoring-nvidia-jetson-nano.md rename to computer-vision-projects/worker-safety-monitoring-nvidia-jetson-nano.md diff --git a/image-projects/workplace-organizer-nvidia-jetson-nano.md b/computer-vision-projects/workplace-organizer-nvidia-jetson-nano.md similarity index 100% rename from image-projects/workplace-organizer-nvidia-jetson-nano.md rename to computer-vision-projects/workplace-organizer-nvidia-jetson-nano.md diff --git a/novel-sensor-projects/README.md b/novel-sensor-projects/README.md new file mode 100644 index 00000000..f767df4d --- /dev/null +++ b/novel-sensor-projects/README.md @@ -0,0 +1,2 @@ +# Novel Sensor Projects + diff --git a/predictive-maintenance-and-defect-detection-projects/README.md b/predictive-maintenance-and-defect-detection-projects/README.md new file mode 100644 index 00000000..78440c8f --- /dev/null +++ b/predictive-maintenance-and-defect-detection-projects/README.md @@ -0,0 +1,2 @@ +# Predictive Maintenance and Defect Detection Projects + diff --git a/predictive-maintenance-and-fault-classification/anticipate-power-outages-arduino-nano-33.md b/predictive-maintenance-and-defect-detection-projects/anticipate-power-outages-arduino-nano-33.md similarity index 100% rename from predictive-maintenance-and-fault-classification/anticipate-power-outages-arduino-nano-33.md rename to predictive-maintenance-and-defect-detection-projects/anticipate-power-outages-arduino-nano-33.md diff --git a/predictive-maintenance-and-fault-classification/brickml-3d-printer-anomaly-detection.md b/predictive-maintenance-and-defect-detection-projects/brickml-3d-printer-anomaly-detection.md similarity index 100% rename from predictive-maintenance-and-fault-classification/brickml-3d-printer-anomaly-detection.md rename to predictive-maintenance-and-defect-detection-projects/brickml-3d-printer-anomaly-detection.md diff --git a/predictive-maintenance-and-fault-classification/brushless-dc-motor-anomaly-detection.md b/predictive-maintenance-and-defect-detection-projects/brushless-dc-motor-anomaly-detection.md similarity index 100% rename from predictive-maintenance-and-fault-classification/brushless-dc-motor-anomaly-detection.md rename to predictive-maintenance-and-defect-detection-projects/brushless-dc-motor-anomaly-detection.md diff --git a/predictive-maintenance-and-fault-classification/compressor-predictive-maintenance-nordic-thingy53.md b/predictive-maintenance-and-defect-detection-projects/compressor-predictive-maintenance-nordic-thingy53.md similarity index 100% rename from predictive-maintenance-and-fault-classification/compressor-predictive-maintenance-nordic-thingy53.md rename to predictive-maintenance-and-defect-detection-projects/compressor-predictive-maintenance-nordic-thingy53.md diff --git a/predictive-maintenance-and-fault-classification/condition-monitoring-syntiant-tinyml.md b/predictive-maintenance-and-defect-detection-projects/condition-monitoring-syntiant-tinyml.md similarity index 100% rename from predictive-maintenance-and-fault-classification/condition-monitoring-syntiant-tinyml.md rename to predictive-maintenance-and-defect-detection-projects/condition-monitoring-syntiant-tinyml.md diff --git a/predictive-maintenance-and-fault-classification/faulty-lithium-ion-cell-identification-battery-packs.md b/predictive-maintenance-and-defect-detection-projects/faulty-lithium-ion-cell-identification-battery-packs.md similarity index 100% rename from predictive-maintenance-and-fault-classification/faulty-lithium-ion-cell-identification-battery-packs.md rename to predictive-maintenance-and-defect-detection-projects/faulty-lithium-ion-cell-identification-battery-packs.md diff --git a/predictive-maintenance-and-fault-classification/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md b/predictive-maintenance-and-defect-detection-projects/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md similarity index 100% rename from predictive-maintenance-and-fault-classification/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md rename to predictive-maintenance-and-defect-detection-projects/fluid-leak-detection-with-flowmeter-seeed-wio-terminal.md diff --git a/predictive-maintenance-and-fault-classification/motor-pump-cypress-cn0549.md b/predictive-maintenance-and-defect-detection-projects/motor-pump-cypress-cn0549.md similarity index 100% rename from predictive-maintenance-and-fault-classification/motor-pump-cypress-cn0549.md rename to predictive-maintenance-and-defect-detection-projects/motor-pump-cypress-cn0549.md diff --git a/predictive-maintenance-and-fault-classification/multimodal-hvac-failure-anomaly-detection-esp32.md b/predictive-maintenance-and-defect-detection-projects/multimodal-hvac-failure-anomaly-detection-esp32.md similarity index 100% rename from predictive-maintenance-and-fault-classification/multimodal-hvac-failure-anomaly-detection-esp32.md rename to predictive-maintenance-and-defect-detection-projects/multimodal-hvac-failure-anomaly-detection-esp32.md diff --git a/predictive-maintenance-and-fault-classification/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md b/predictive-maintenance-and-defect-detection-projects/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md similarity index 100% rename from predictive-maintenance-and-fault-classification/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md rename to predictive-maintenance-and-defect-detection-projects/pipeline-clog-detection-with-flowmeter-seeed-wio-terminal.md diff --git a/predictive-maintenance-and-fault-classification/predictive-maintenance-sony-spresense-commonsense.md b/predictive-maintenance-and-defect-detection-projects/predictive-maintenance-sony-spresense-commonsense.md similarity index 100% rename from predictive-maintenance-and-fault-classification/predictive-maintenance-sony-spresense-commonsense.md rename to predictive-maintenance-and-defect-detection-projects/predictive-maintenance-sony-spresense-commonsense.md diff --git a/predictive-maintenance-and-fault-classification/predictive-maintenance-with-nordic-thingy91.md b/predictive-maintenance-and-defect-detection-projects/predictive-maintenance-with-nordic-thingy91.md similarity index 100% rename from predictive-maintenance-and-fault-classification/predictive-maintenance-with-nordic-thingy91.md rename to predictive-maintenance-and-defect-detection-projects/predictive-maintenance-with-nordic-thingy91.md diff --git a/predictive-maintenance-and-fault-classification/refrigerator-predictive-maintenance-arduino-nano-33.md b/predictive-maintenance-and-defect-detection-projects/refrigerator-predictive-maintenance-arduino-nano-33.md similarity index 100% rename from predictive-maintenance-and-fault-classification/refrigerator-predictive-maintenance-arduino-nano-33.md rename to predictive-maintenance-and-defect-detection-projects/refrigerator-predictive-maintenance-arduino-nano-33.md diff --git a/predictive-maintenance-and-fault-classification/vibration-classification-with-brainchip-akida.md b/predictive-maintenance-and-defect-detection-projects/vibration-classification-with-brainchip-akida.md similarity index 100% rename from predictive-maintenance-and-fault-classification/vibration-classification-with-brainchip-akida.md rename to predictive-maintenance-and-defect-detection-projects/vibration-classification-with-brainchip-akida.md diff --git a/predictive-maintenance-and-fault-classification/weight-scale-predictive-maintenance-arduino-nano-33.md b/predictive-maintenance-and-defect-detection-projects/weight-scale-predictive-maintenance-arduino-nano-33.md similarity index 100% rename from predictive-maintenance-and-fault-classification/weight-scale-predictive-maintenance-arduino-nano-33.md rename to predictive-maintenance-and-defect-detection-projects/weight-scale-predictive-maintenance-arduino-nano-33.md diff --git a/readme/featured-machine-learning-projects/README.md b/readme/featured-machine-learning-projects/README.md new file mode 100644 index 00000000..7e55133d --- /dev/null +++ b/readme/featured-machine-learning-projects/README.md @@ -0,0 +1,2 @@ +# Featured Machine Learning Projects + diff --git a/image-projects/avnet-rzboard-object-counting.md b/readme/featured-machine-learning-projects/avnet-rzboard-object-counting.md similarity index 58% rename from image-projects/avnet-rzboard-object-counting.md rename to readme/featured-machine-learning-projects/avnet-rzboard-object-counting.md index 3335310c..c7ed85ea 100644 --- a/image-projects/avnet-rzboard-object-counting.md +++ b/readme/featured-machine-learning-projects/avnet-rzboard-object-counting.md @@ -1,6 +1,7 @@ --- description: >- - Counting objects with computer vision, using the Avnet RZBoard V2L and it's Renesas RZ/V2L with DRP-AI accelerator. + Counting objects with computer vision, using the Avnet RZBoard V2L and it's + Renesas RZ/V2L with DRP-AI accelerator. --- # Computer Vision Object Counting - Avnet RZBoard V2L @@ -9,90 +10,90 @@ Created By: David Tischler Public Project Link: [https://studio.edgeimpulse.com/public/315846/latest](https://studio.edgeimpulse.com/public/315846/latest) -![Image source: https://unsplash.com/photos/black-glass-bottles-on-stainless-steel-table-hgrFoK4bjao](../.gitbook/assets/avnet-rzboard-object-counting/cover.jpg) +![Image source: https://unsplash.com/photos/black-glass-bottles-on-stainless-steel-table-hgrFoK4bjao](../../.gitbook/assets/avnet-rzboard-object-counting/cover.jpg) ## Introduction -Large scale production and manufacturing operations rely on effective and accurate inventory and product counting, so that businesses have accurate and known quantities of products to fulfill orders, ship to retailers, and plan their finances accordingly. In typical scenarios, business have up-to-date counts of inputs such as supplies and raw materials, partially completed products that are currently being worked on, and finished goods ready for distribution. To alleviate the burden of counting the units in each stage by hand, which could possibly be very time-consuming, computer vision can be used to identify and quantify parts, supplies, or products instead. +Large scale production and manufacturing operations rely on effective and accurate inventory and product counting, so that businesses have accurate and known quantities of products to fulfill orders, ship to retailers, and plan their finances accordingly. In typical scenarios, business have up-to-date counts of inputs such as supplies and raw materials, partially completed products that are currently being worked on, and finished goods ready for distribution. To alleviate the burden of counting the units in each stage by hand, which could possibly be very time-consuming, computer vision can be used to identify and quantify parts, supplies, or products instead. -There are two distinct counting operations to consider. The first is a "total quantity" at any given time, such as "there are 8 objects on the assembly line at this exact moment". The second scenario is a value that includes a time factor, for example, "14 items moved from point A to point B along the conveyor belt since we began work this morning." Each of these counts are important, so we'll cover both of them here. First, we'll perform a count of items detected in a camera frame, then we will explore how to count the total number of objects that moved past a stationary camera placed above a conveyor belt. +There are two distinct counting operations to consider. The first is a "total quantity" at any given time, such as "there are 8 objects on the assembly line at this exact moment". The second scenario is a value that includes a time factor, for example, "14 items moved from point A to point B along the conveyor belt since we began work this morning." Each of these counts are important, so we'll cover both of them here. First, we'll perform a count of items detected in a camera frame, then we will explore how to count the total number of objects that moved past a stationary camera placed above a conveyor belt. A machine learning model that recognizes a distinct item will be needed, along with the camera and hardware. ## Solution -We'll use the [Avnet RZBoard V2L](https://www.avnet.com/wps/portal/us/products/avnet-boards/avnet-board-families/rzboard-v2l/) along with Edge Impulse to accomplish this task. The Avnet RZBoard V2L is a compact single board computer powered by a Renesas RZ/V2L SoC, running a Linux operating system. It has 2gb of RAM, 32gb of onboard eMMC storage, an SD Card slot, micro-HDMI display output, an ethernet port, built-in WiFi and Bluetooth connectivity, USB ports, and a 40-pin GPIO header for expansion. It's powered by a single 5V/3A USB-C power supply. +We'll use the [Avnet RZBoard V2L](https://www.avnet.com/wps/portal/us/products/avnet-boards/avnet-board-families/rzboard-v2l/) along with Edge Impulse to accomplish this task. The Avnet RZBoard V2L is a compact single board computer powered by a Renesas RZ/V2L SoC, running a Linux operating system. It has 2gb of RAM, 32gb of onboard eMMC storage, an SD Card slot, micro-HDMI display output, an ethernet port, built-in WiFi and Bluetooth connectivity, USB ports, and a 40-pin GPIO header for expansion. It's powered by a single 5V/3A USB-C power supply. -![](../.gitbook/assets/avnet-rzboard-object-counting/rzboard.jpg) +![](../../.gitbook/assets/avnet-rzboard-object-counting/rzboard.jpg) The Renesas RZ/V2L SoC contains two 1.2GHz Arm® Cortex®-A55 cores for running Linux, a 200MHz Cortex-M33 core for RTOS or other microprocessor applications, and very important for this use-case, a DRP-AI machine learning model accelerator. With all of the connectivity, memory, storage, and compute power the RZBoard V2L contains, it is a very capable and highly efficient platform for AI projects. -![](../.gitbook/assets/avnet-rzboard-object-counting/rzv2l.jpg) +![](../../.gitbook/assets/avnet-rzboard-object-counting/rzv2l.jpg) -For ease of testing, we'll use small conveyor belt to prototype the system so that objects pass into and out of the camera's field of view. This way we can test both scenarios as mentioned: the number of objects in view at a distinct moment, and the total count of objects that have moved past the camera. +For ease of testing, we'll use small conveyor belt to prototype the system so that objects pass into and out of the camera's field of view. This way we can test both scenarios as mentioned: the number of objects in view at a distinct moment, and the total count of objects that have moved past the camera. ## Hardware Requirements - - [Avnet RZBoard V2L](https://www.avnet.com/wps/portal/us/products/avnet-boards/avnet-board-families/rzboard-v2l/) - - USB Webcam - - HDMI monitor, keyboard, mouse - - Conveyor belt, motor, power supply - - M5 hex nuts (this is the object I will be detecting, but you can choose something else) - +* [Avnet RZBoard V2L](https://www.avnet.com/wps/portal/us/products/avnet-boards/avnet-board-families/rzboard-v2l/) +* USB Webcam +* HDMI monitor, keyboard, mouse +* Conveyor belt, motor, power supply +* M5 hex nuts (this is the object I will be detecting, but you can choose something else) + ## Software Requirements - - Edge Impulse - - Updated RZBoard OS - +* Edge Impulse +* Updated RZBoard OS + ## Data Collection -The first step in our machine learning workflow is data collection. In this example, we are going to identify, and count, some small M5 hex bolts traveling down a conveyor belt. I've used M5 hex nuts due to their convenient size, but you could use any object. To build a model that can identify a hex nut, we need to first take pictures of hex nuts and label them accordingly. Knowing that a USB camera is going to be hooked up to the RZBoard and placed above the conveyor belt, I have (for now) connected the same camera directly to my laptop in order to capture representative images. This allows me to gather pictures of the M5 nuts from the same angle, distance, and lighting as what we will experience once the model is deployed to the RZBoard. +The first step in our machine learning workflow is data collection. In this example, we are going to identify, and count, some small M5 hex bolts traveling down a conveyor belt. I've used M5 hex nuts due to their convenient size, but you could use any object. To build a model that can identify a hex nut, we need to first take pictures of hex nuts and label them accordingly. Knowing that a USB camera is going to be hooked up to the RZBoard and placed above the conveyor belt, I have (for now) connected the same camera directly to my laptop in order to capture representative images. This allows me to gather pictures of the M5 nuts from the same angle, distance, and lighting as what we will experience once the model is deployed to the RZBoard. -![](../.gitbook/assets/avnet-rzboard-object-counting/conveyor.jpg) +![](../../.gitbook/assets/avnet-rzboard-object-counting/conveyor.jpg) -Log in to Edge Impulse, click on Create Project, and provide a name for your project. Next, click "Data acquisition" on the left, and then click on "Connect to your computer". A new tab or window will open, with the ability to take pictures by clicking the "Capture" button. Images collected will be automatically added to your dataset. I was able to select the camera I wanted to use in my browser settings, and I re-positioned the M5 bolts, moved the conveyor a bit, rotated the bolts, and varied the lighting in order to build a robust collection of images. +Log in to Edge Impulse, click on Create Project, and provide a name for your project. Next, click "Data acquisition" on the left, and then click on "Connect to your computer". A new tab or window will open, with the ability to take pictures by clicking the "Capture" button. Images collected will be automatically added to your dataset. I was able to select the camera I wanted to use in my browser settings, and I re-positioned the M5 bolts, moved the conveyor a bit, rotated the bolts, and varied the lighting in order to build a robust collection of images. -![](../.gitbook/assets/avnet-rzboard-object-counting/data-collection.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/data-collection.png) -Next, we need to label the objects in each image. This locates where the objects of interest are in each picture, which will be used when training the model. Click on "Labeling Queue" at the top, and draw a bounding box around each bolt in the picture, and give it a label. I simply entered `m5_nut` on mine, though yours could vary. Click on "Save labels" to advance to the next image in the dataset, and you will notice that the bounding boxes will follow through to the next picture, making this process quick and easy. Once complete, you can click on "Dataset" to return to the summary list of data elements. You can click on them if you'd like to inspect them closer, but they should be ready for use at this point. +Next, we need to label the objects in each image. This locates where the objects of interest are in each picture, which will be used when training the model. Click on "Labeling Queue" at the top, and draw a bounding box around each bolt in the picture, and give it a label. I simply entered `m5_nut` on mine, though yours could vary. Click on "Save labels" to advance to the next image in the dataset, and you will notice that the bounding boxes will follow through to the next picture, making this process quick and easy. Once complete, you can click on "Dataset" to return to the summary list of data elements. You can click on them if you'd like to inspect them closer, but they should be ready for use at this point. -![](../.gitbook/assets/avnet-rzboard-object-counting/label-queue.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/label-queue.png) -![](../.gitbook/assets/avnet-rzboard-object-counting/data.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/data.png) ## Model Creation -After the images have all been labeled, it is time to move on to the machine learning model creation phase. Click on "Impulse design" on the left, and you will see 4 columns (2 of which are empty for the moment), that will make up the machine learning pipeline. The first column is the input, which should be pre-populated with "Image data". You can however, increase the image height and width to 320 by 320, as the RZBoard will have plenty of processing power available to make use of the larger image size (more on that in a bit). In the second column, click "Add a processing block", and choose "Image" by clicking "Add". In column 3, click "Add a learning block", and choose "YOLOv5 for Renesas DRP-AI" by clicking "Add". Finally, the fourth column should be pre-populated as well, with only one Output feature, the label we created earlier called `m5_nut`. Click "Save Impulse". +After the images have all been labeled, it is time to move on to the machine learning model creation phase. Click on "Impulse design" on the left, and you will see 4 columns (2 of which are empty for the moment), that will make up the machine learning pipeline. The first column is the input, which should be pre-populated with "Image data". You can however, increase the image height and width to 320 by 320, as the RZBoard will have plenty of processing power available to make use of the larger image size (more on that in a bit). In the second column, click "Add a processing block", and choose "Image" by clicking "Add". In column 3, click "Add a learning block", and choose "YOLOv5 for Renesas DRP-AI" by clicking "Add". Finally, the fourth column should be pre-populated as well, with only one Output feature, the label we created earlier called `m5_nut`. Click "Save Impulse". -![](../.gitbook/assets/avnet-rzboard-object-counting/impulse.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/impulse.png) -On the left, click "Image", and we'll configure the Image Processing Block. Here you can review the Raw features, switch to grayscale to save some memory on lower power devices or those with grayscale image sensors (not necessary in this case), and review the DSP results. We won't make any changes, so click "Save parameters". It will automatically move to the "Generate features" page, and here you can click the "Generate features" button to create a visualization of the analyzed features. With only one class in this project, there should be a nice cluster of data points represented, though the clustering is a bit easier to comprehend or represent visually when multiple labels / objects are used in a dataset. +On the left, click "Image", and we'll configure the Image Processing Block. Here you can review the Raw features, switch to grayscale to save some memory on lower power devices or those with grayscale image sensors (not necessary in this case), and review the DSP results. We won't make any changes, so click "Save parameters". It will automatically move to the "Generate features" page, and here you can click the "Generate features" button to create a visualization of the analyzed features. With only one class in this project, there should be a nice cluster of data points represented, though the clustering is a bit easier to comprehend or represent visually when multiple labels / objects are used in a dataset. -Next, click on "YOLOv5 for Renesas DRP-AI" on the left navigation to go to the Neural Network Settings page. You can leave the default selections alone, but do check to make sure that the Target is set to "Renesas RZ/V2L with DRP-AI accelerator" in the top-right corner, for more accurate inference time and memory usage estimations, and also double check that the Neural network architecture is correctly set to "Renesas / YOLOv5 for Renesas DRP-AI", then click "Start training". This will take a short while to iterate through each epoch, and at the end of the process you should get back a Precision score and estimated inference time and memory usage. +Next, click on "YOLOv5 for Renesas DRP-AI" on the left navigation to go to the Neural Network Settings page. You can leave the default selections alone, but do check to make sure that the Target is set to "Renesas RZ/V2L with DRP-AI accelerator" in the top-right corner, for more accurate inference time and memory usage estimations, and also double check that the Neural network architecture is correctly set to "Renesas / YOLOv5 for Renesas DRP-AI", then click "Start training". This will take a short while to iterate through each epoch, and at the end of the process you should get back a Precision score and estimated inference time and memory usage. -![](../.gitbook/assets/avnet-rzboard-object-counting/model-build.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/model-build.png) -![](../.gitbook/assets/avnet-rzboard-object-counting/model-complete.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/model-complete.png) ## Deployment -In order to get the model (and eventually our counting application) onto the RZBoard V2L, we have a bit of prep work to do. The RZBoard comes from the factory with an operating system and sample application installed on it's eMMC, which is nice for an immediate way to get started with the board and a great out-of-the-box experience, but won't work for our purposes here. Instead, we need a version of the Yocto OS that includes `nodejs` and `npm`, so that we can install the Edge Impulse Linux Runner. You _could_ go down the path of building Yocto yourself (I tested it, and it does work fine), but to save you the trouble Avnet has already gone ahead and built one, that you can find in their Sharepoint site here: +In order to get the model (and eventually our counting application) onto the RZBoard V2L, we have a bit of prep work to do. The RZBoard comes from the factory with an operating system and sample application installed on it's eMMC, which is nice for an immediate way to get started with the board and a great out-of-the-box experience, but won't work for our purposes here. Instead, we need a version of the Yocto OS that includes `nodejs` and `npm`, so that we can install the Edge Impulse Linux Runner. You _could_ go down the path of building Yocto yourself (I tested it, and it does work fine), but to save you the trouble Avnet has already gone ahead and built one, that you can find in their Sharepoint site here: -[https://avtinc.sharepoint.com/teams/ET-Downloads/Shared%20Documents/Forms/AllItems.aspx?id=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles%2FRZBoard%5FEdgeImpulse%5FeMMC%2Ezip&parent=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles&p=true&ga=1](https://avtinc.sharepoint.com/teams/ET-Downloads/Shared%20Documents/Forms/AllItems.aspx?id=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles%2FRZBoard%5FEdgeImpulse%5FeMMC%2Ezip&parent=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles&p=true&ga=1) +[https://avtinc.sharepoint.com/teams/ET-Downloads/Shared%20Documents/Forms/AllItems.aspx?id=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles%2FRZBoard%5FEdgeImpulse%5FeMMC%2Ezip\&parent=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles\&p=true\&ga=1](https://avtinc.sharepoint.com/teams/ET-Downloads/Shared%20Documents/Forms/AllItems.aspx?id=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles%2FRZBoard%5FEdgeImpulse%5FeMMC%2Ezip\&parent=%2Fteams%2FET%2DDownloads%2FShared%20Documents%2Fprojects%2FRenesas%20RzBoard%20V2L%2Fsoftware%5Fimage%5Ffiles\&p=true\&ga=1) -![](../.gitbook/assets/avnet-rzboard-object-counting/sharepoint.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/sharepoint.png) -The name of the file you need to download is `avnet-core-image-rzboard-20230124105646.rootfs.wic`. (If you enter the folder at the top-most driectory, navigate into the "images" folder to find it there). Download that file, and flash it directly to an SD Card. Now, on the RZBoard, you'll need to flip a small DIP-switch that tells the board to boot from SD Card instead of the eMMC storage. Look for two tiny switches near the headphone jack, and make sure they are both flipped *away* from the headphone jack, facing the silkscreened `1` and `2` markings on the switch. Here is an example of how mine looks: +The name of the file you need to download is `avnet-core-image-rzboard-20230124105646.rootfs.wic`. (If you enter the folder at the top-most driectory, navigate into the "images" folder to find it there). Download that file, and flash it directly to an SD Card. Now, on the RZBoard, you'll need to flip a small DIP-switch that tells the board to boot from SD Card instead of the eMMC storage. Look for two tiny switches near the headphone jack, and make sure they are both flipped _away_ from the headphone jack, facing the silkscreened `1` and `2` markings on the switch. Here is an example of how mine looks: -![](../.gitbook/assets/avnet-rzboard-object-counting/boot-switch.jpg) +![](../../.gitbook/assets/avnet-rzboard-object-counting/boot-switch.jpg) Once this is done, insert the SD Card, plug in USB-C power, attach an HDMI monitor, USB webcam, and USB keyboard/mouse, then power on the board by pressing the power button near the USB-C power supply. -![](../.gitbook/assets/avnet-rzboard-object-counting/terminal.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/terminal.png) -Once booted up, you can open a terminal session by clicking on the top left icon, and we'll need to connect the board to WiFi. To do that, enter: +Once booted up, you can open a terminal session by clicking on the top left icon, and we'll need to connect the board to WiFi. To do that, enter: ``` wpa_passphrase "SSID_Name_Goes_Here" "Password_Goes_Here" >> /etc/wpa_supplicant.conf @@ -100,7 +101,7 @@ wpa_supplicant -B -i mlan0 -c /etc/wpa_supplicant.conf udhcpc -i mlan0 -n -R ``` -> NOTE: You can also attach a serial console and use Putty or a similar terminal application, if that's easier for you. +> NOTE: You can also attach a serial console and use Putty or a similar terminal application, if that's easier for you. We'll also need to expand the available space on the SD Card, so enter: @@ -108,7 +109,7 @@ We'll also need to expand the available space on the SD Card, so enter: fdisk /dev/mmcblk0 ``` -Enter `p` to print the current partition information, and make note of the `mmcblk0p2` start address displayed on the screen. We'll need that in a moment (mine was `204832`). Follow the series of commands below to [p] print the partition info, [d] delete the current second [2] partition, make a [n] new [p] primary second [2] partition, and type in the start address you discovered a moment ago and press [enter]. Then press [enter] again on the next question to accept the default end address, [N] to *not* remove the signature, and [w] to write the changes to disk. The chain of commands is thus: +Enter `p` to print the current partition information, and make note of the `mmcblk0p2` start address displayed on the screen. We'll need that in a moment (mine was `204832`). Follow the series of commands below to \[p] print the partition info, \[d] delete the current second \[2] partition, make a \[n] new \[p] primary second \[2] partition, and type in the start address you discovered a moment ago and press \[enter]. Then press \[enter] again on the next question to accept the default end address, \[N] to _not_ remove the signature, and \[w] to write the changes to disk. The chain of commands is thus: ``` p -> d -> 2 -> n -> p -> 2 -> ,enter -> enter (to accept default) -> N -> w @@ -132,17 +133,17 @@ Once completed, we can test out that everything works thus far, by running: edge-impulse-linux-runner --debug ``` -You will be asked for your username and password, and the project name to connect to, then a download and build will run to get the model ready for use on the RZBoard. Once complete, local inferencing will start, and results will be printed to the command line. Make note of the final model location just before the inferencing starts, we'll use that later on (mine was `/root/.ei-linux-runner/models/315846/v15/model.eim`). You can also load http://:4912 in a browser on your development machine, to get a view from the camera with any detected objects outlined by bounding boxes. Before we move on to building our object counting application, let's highlight an important item here. My inference time as you can see below is approximately 8ms to 10ms, so roughly 100 inferences per second - incredible performance. The web view of the camera however, provides a slow frame rate: The reason is that the sample webserver sending the camera view is not really optimized, and the WiFi latency itself is also at play here. A compiled binary version of an application is much more responsive. +You will be asked for your username and password, and the project name to connect to, then a download and build will run to get the model ready for use on the RZBoard. Once complete, local inferencing will start, and results will be printed to the command line. Make note of the final model location just before the inferencing starts, we'll use that later on (mine was `/root/.ei-linux-runner/models/315846/v15/model.eim`). You can also load http://:4912 in a browser on your development machine, to get a view from the camera with any detected objects outlined by bounding boxes. Before we move on to building our object counting application, let's highlight an important item here. My inference time as you can see below is approximately 8ms to 10ms, so roughly 100 inferences per second - incredible performance. The web view of the camera however, provides a slow frame rate: The reason is that the sample webserver sending the camera view is not really optimized, and the WiFi latency itself is also at play here. A compiled binary version of an application is much more responsive. -![](../.gitbook/assets/avnet-rzboard-object-counting/cli-runner.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/cli-runner.png) -![](../.gitbook/assets/avnet-rzboard-object-counting/camera.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/camera.png) ## Object Counting Applications ### Unique Count at any Moment -For our first counting task, we'll quantify the number of objects detected within a camera frame. To do this, we'll put together a bit of python code to run on the RZBoard - but this will also require some more tooling and dependencies to get installed. In that same terminal session you already have running, enter the following series of commands to install `portaudio`, `pip`, a few dependencies, `edge_impulse_linux`, and set the Display variable. You could probably wrap this all up into a shell command to run in bulk, but here are the individual commands to run: +For our first counting task, we'll quantify the number of objects detected within a camera frame. To do this, we'll put together a bit of python code to run on the RZBoard - but this will also require some more tooling and dependencies to get installed. In that same terminal session you already have running, enter the following series of commands to install `portaudio`, `pip`, a few dependencies, `edge_impulse_linux`, and set the Display variable. You could probably wrap this all up into a shell command to run in bulk, but here are the individual commands to run: ``` curl https://files.portaudio.com/archives/pa_stable_v190700_20210406.tgz --output portaudio.tgz @@ -304,15 +305,15 @@ with ImageImpulseRunner(modelfile) as runner: runner.stop() ``` -Be sure to update line 8 with the location of your model file on the RZBoard. We determined that mine was `/root/.ei-linux-runner/models/315846/v15/model.eim` earlier when we ran the Linux Runner the first time. Finally, it is time to test out the counter, simply run `python3 unique_count.py`, and in the terminal you will see the number of detected M5 nuts that are in view of the camera printed out to the console. +Be sure to update line 8 with the location of your model file on the RZBoard. We determined that mine was `/root/.ei-linux-runner/models/315846/v15/model.eim` earlier when we ran the Linux Runner the first time. Finally, it is time to test out the counter, simply run `python3 unique_count.py`, and in the terminal you will see the number of detected M5 nuts that are in view of the camera printed out to the console. I was able to then use the conveyor belt and observe the quantity increase and decrease as the bolts moved down the line and entered / exited the field of view of the camera. -![](../.gitbook/assets/avnet-rzboard-object-counting/unique-count.png) +![](../../.gitbook/assets/avnet-rzboard-object-counting/unique-count.png) ### Sum of Total Count Over Time -Now we can move on to our second counting application, which totals up the number of M5 hex nuts that pass in front of the camera over a period of time. For that, we'll use a second python snippet, and this time we'll render the camera view on screen so we can have a look at what the camera is seeing. Create a new file with `nano total_count.py` and paste in the following snippet: +Now we can move on to our second counting application, which totals up the number of M5 hex nuts that pass in front of the camera over a period of time. For that, we'll use a second python snippet, and this time we'll render the camera view on screen so we can have a look at what the camera is seeing. Create a new file with `nano total_count.py` and paste in the following snippet: ``` #!/usr/bin/env python @@ -476,10 +477,10 @@ python3 total_count.py /path/to/model/file/goes/here In my case, this means I entered `python3 total_count.py /root/.ei-linux-runner/models/315846/v15/model.eim` -It will take a moment for the camera view to appear on screen, and it should be noted that once again the framerate here is not optimized, as we are running non-threaded single core python, and compiled code is much faster. But for purposes of demonstating how to acheive the counting, this will work. I once again turned on the conveyor belt, and as M5 hex nuts travel past the camera, the count increases by one in the rendered camera view window. My model could probably benefit from some extra images added to my dataset, as I did have a few sneak by undetected, so perhaps 60 images was not quite enough in the training dataset, but we have proven the concept works! +It will take a moment for the camera view to appear on screen, and it should be noted that once again the framerate here is not optimized, as we are running non-threaded single core python, and compiled code is much faster. But for purposes of demonstating how to acheive the counting, this will work. I once again turned on the conveyor belt, and as M5 hex nuts travel past the camera, the count increases by one in the rendered camera view window. My model could probably benefit from some extra images added to my dataset, as I did have a few sneak by undetected, so perhaps 60 images was not quite enough in the training dataset, but we have proven the concept works! -![](../.gitbook/assets/avnet-rzboard-object-counting/counting.gif) +![](../../.gitbook/assets/avnet-rzboard-object-counting/counting.gif) ## Conclusion -The Avnet RZBoard with it's Renesas RZ/V2L SoC and DRP-AI Accelerator made prototpying our computer vision counting applications quick and easy, and demonstrated excellent performance with inference times in the 9ms range! +The Avnet RZBoard with it's Renesas RZ/V2L SoC and DRP-AI Accelerator made prototpying our computer vision counting applications quick and easy, and demonstrated excellent performance with inference times in the 9ms range! diff --git a/image-projects/fomo-ad-in-aws.md b/readme/featured-machine-learning-projects/fomo-ad-in-aws.md similarity index 79% rename from image-projects/fomo-ad-in-aws.md rename to readme/featured-machine-learning-projects/fomo-ad-in-aws.md index e9da10fd..7293aa8f 100644 --- a/image-projects/fomo-ad-in-aws.md +++ b/readme/featured-machine-learning-projects/fomo-ad-in-aws.md @@ -1,9 +1,10 @@ --- description: >- - Advanced ML workflow with available Jupyter Notebook using computer vision, AWS SageMaker and MLFlow to benchmark industry visual anomaly models. + Advanced ML workflow with available Jupyter Notebook using computer vision, + AWS SageMaker and MLFlow to benchmark industry visual anomaly models. --- -# Optimize a Cloud-based Visual Anomaly Detection Model for Edge Deployments +# Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments Created By: Mathieu Lescaudron @@ -11,11 +12,11 @@ Public Project Link: [https://studio.edgeimpulse.com/public/376268/latest](https GitHub Repo: [https://github.com/emergy-official/anomaly.parf.ai](https://github.com/emergy-official/anomaly.parf.ai) -![](../.gitbook/assets/fomo-ad-in-aws/cover1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/cover1.png) -![](../.gitbook/assets/fomo-ad-in-aws/cover2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/cover2.png) -![](../.gitbook/assets/fomo-ad-in-aws/cover3.png) +![](../../.gitbook/assets/fomo-ad-in-aws/cover3.png) ## Introduction @@ -23,27 +24,27 @@ Let's explore the development and optimization of a cloud-based visual anomaly d We will cover the following topics: -- **Datasets**: Creation of our own datasets. -- **Models**: Development of three different models: - - A baseline model + usage of BYOM (Bring Your Own Model on Edge Impulse), - - Efficient AD model, - - FOMO AD model by Edge Impulse (automated). -- **Web App**: - - Setting up a real-time and serverless inference endpoint, - - Dataset explorer, - - Automating deployments with GitHub Actions and Terraform on AWS. +* **Datasets**: Creation of our own datasets. +* **Models**: Development of three different models: + * A baseline model + usage of BYOM (Bring Your Own Model on Edge Impulse), + * Efficient AD model, + * FOMO AD model by Edge Impulse (automated). +* **Web App**: + * Setting up a real-time and serverless inference endpoint, + * Dataset explorer, + * Automating deployments with GitHub Actions and Terraform on AWS. This is a demo project. All code is provided for you to implement any or all parts yourself. ## Software used -- [Edge Impulse Studio](https://studio.edgeimpulse.com/studio) -- [Edge Impulse Mobile client](https://github.com/edgeimpulse/mobile-client) -- [Visual Studio Code](https://code.visualstudio.com/) -- [Amazon Web Services](https://aws.amazon.com/) -- [Terraform](https://www.terraform.io/) -- [MLFlow](https://mlflow.org/) -- [Astro](https://astro.build/) +* [Edge Impulse Studio](https://studio.edgeimpulse.com/studio) +* [Edge Impulse Mobile client](https://github.com/edgeimpulse/mobile-client) +* [Visual Studio Code](https://code.visualstudio.com/) +* [Amazon Web Services](https://aws.amazon.com/) +* [Terraform](https://www.terraform.io/) +* [MLFlow](https://mlflow.org/) +* [Astro](https://astro.build/) ## Context @@ -51,31 +52,31 @@ Imagine we are a commercial baking company that produces cookies. Our goal is to We are developing a cloud-based proof-of-concept to understand the feasibility of this technique, before deploying it on edge devices. -Although this is only a hypothetical example and demonstration, this quality inspection process and computer vision workflow could absolutely be leveraged by large-scale food service providers, commercial kitches that make packaged retail food items, or any many other mass-produced retail products even beyond the food industry. +Although this is only a hypothetical example and demonstration, this quality inspection process and computer vision workflow could absolutely be leveraged by large-scale food service providers, commercial kitches that make packaged retail food items, or any many other mass-produced retail products even beyond the food industry. ## Step 1: Create the Datasets We assume we don't have access to [Onmiverse Replicator](https://docs.omniverse.nvidia.com/extensions/latest/index.html) to create a synthetic dataset. Instead, we manually create our own. The first step is to carefully review which cookies to ~~eat~~ use. -![](../.gitbook/assets/fomo-ad-in-aws/dataset1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/dataset1.png) We'll create three datasets using three different types of cookies: -- One with texture, -- One thicker cookie, -- One plain cookie. +* One with texture, +* One thicker cookie, +* One plain cookie. Each dataset will consist of 200 images, totaling 600 images: -- 100 without any anomalies -- 100 with anomalies - - 50 easy to recognize with a clear, strong separation down the middle, - - 25 medium difficulty with a separation that has no gap, - - 25 hard to detect, with small defects and knife marks. +* 100 without any anomalies +* 100 with anomalies + * 50 easy to recognize with a clear, strong separation down the middle, + * 25 medium difficulty with a separation that has no gap, + * 25 hard to detect, with small defects and knife marks. We take around five pictures of each cookie, making slight rotations each time. Here's the result: -![](../.gitbook/assets/fomo-ad-in-aws/dataset2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/dataset2.png) Each picture, taken from a mobile phone in a `1:1` ratio with an original size of 2992 x 2992 pixels, is resized to 1024 x 1024 pixels using [mogrify](https://imagemagick.org/script/mogrify.php) command from ImageMagick. It saves computing resources for both the training process and the inference endpoint: @@ -132,15 +133,15 @@ model.compile( Here's how the images are distributed for this model: -- Training: 144 images (72%) -- Validation: 16 images (8%) -- Test: 40 images (20%) +* Training: 144 images (72%) +* Validation: 16 images (8%) +* Test: 40 images (20%) Both "anomaly" and "no anomaly" images are used during training. The model is trained on a Mac using the CPU, running through 50 epochs. -You can find the results in the [Step 3: Benchmarking](#step-3-benchmarking) section. +You can find the results in the [Step 3: Benchmarking](fomo-ad-in-aws.md#step-3-benchmarking) section. ### Convert Baseline with BYOM @@ -181,8 +182,7 @@ if deploy_bytes_mac_os: f.write(deploy_bytes.getvalue()) ``` -You'll need to make it executable by using the command `chmod +x baseline.eim`. -And you're all set! Create an inference function to use it with this model: +You'll need to make it executable by using the command `chmod +x baseline.eim`. And you're all set! Create an inference function to use it with this model: ```python def ei_inference(img_path): @@ -206,13 +206,13 @@ def ei_inference(img_path): Let's use another method called **EfficientAD** ([detailed in a study from arXiv.org](https://arxiv.org/abs/2303.14535)). -EfficientAD employs an autoencoder paired with a student-teacher approach to quickly and effectively identify anomalies in images. +EfficientAD employs an autoencoder paired with a student-teacher approach to quickly and effectively identify anomalies in images. Take a look at their [video presentation](https://www.youtube.com/watch?v=yhpkHOdpyPU) for a brief overview. The network, named PDN (Patch Description Network), includes a design with 4 convolutional layers and 2 pooling layers. It examines each segment of the 33 x 33 pixel image and produces a feature vector of 384 values. -![](../.gitbook/assets/fomo-ad-in-aws/efficientad1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/efficientad1.png) Two models, `student` and `teacher` are trained on the same data. The teacher model guides the student model by providing a loss function which helps the student to improve their performance in detecting anomalies. @@ -238,7 +238,7 @@ mlflow server \ --gunicorn-opts --keep-alive=300 ``` -Here, we're using the `--artifacts-destination` argument to specify where to store our models. You can omit this argument if you're not using a S3 bucket on AWS, and it will default to storing the models on the disk. +Here, we're using the `--artifacts-destination` argument to specify where to store our models. You can omit this argument if you're not using a S3 bucket on AWS, and it will default to storing the models on the disk. In your code, you define an experiment like this: @@ -264,17 +264,17 @@ We primarily use MLFlow to track experiments and store artifacts, although it of You can find the full setup instructions for MLFlow for this demo [here](https://github.com/emergy-official/anomaly.parf.ai/blob/main/ai/AWS_ML_FLOW.md). -#### Training in the cloud +#### Training in the cloud Let's train our models in the cloud using our [notebook](https://github.com/emergy-official/anomaly.parf.ai/blob/main/ai/notebooks/2_efficientad.ipynb). We are using a Jupyter notebook, or you could also use a Python script. There are many different cloud providers that allow you to train a model. We will use an AWS instance that includes an [Nvidia Tesla 4 GPU](https://www.nvidia.com/en-us/data-center/tesla-t4/). -The specific instance type we use is `g4dn.xlarge`. To get access to this instance, you need to create a support ticket requesting access to the type G instance type in your region. It will cost us 0.526 USD per hour and we plan to use it for approximately 3h. +The specific instance type we use is `g4dn.xlarge`. To get access to this instance, you need to create a support ticket requesting access to the type G instance type in your region. It will cost us 0.526 USD per hour and we plan to use it for approximately 3h. For our setup, we'll use a pre-configured AMI with PyTorch named `Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.2.0`. -![](../.gitbook/assets/fomo-ad-in-aws/aws1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/aws1.png) Here is the CLI: @@ -311,71 +311,73 @@ Make sure you've enabled port forwarding so you can connect to the remote Jupyte ```sh ssh -N -f -L 8888:localhost:8888 ubuntu@44.200.180.25 # Change using your instance IP ``` + You can now access Jupyter Notebook on the remote instance from your local computer. -![](../.gitbook/assets/fomo-ad-in-aws/aws2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/aws2.png) For the training, we will only use the images without anomalies. Here's how the data is distributed: -- Training - - No anomaly: 72 images (36%) -- Validation - - No Anomaly: 8 images (4%) - - Anomaly: 20 images (10%) -- Testing - - No Anomaly: 20 images (10%) - - Anomaly: 80 images (40%) +* Training + * No anomaly: 72 images (36%) +* Validation + * No Anomaly: 8 images (4%) + * Anomaly: 20 images (10%) +* Testing + * No Anomaly: 20 images (10%) + * Anomaly: 80 images (40%) Once it is trained, you can see the different results in MLFlow: -![](../.gitbook/assets/fomo-ad-in-aws/mlflow1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/mlflow1.png) And you can create graphics to build reports: -![](../.gitbook/assets/fomo-ad-in-aws/mlflow2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/mlflow2.png) -For the cookies dataset three, the best model used 3,200 steps, pretrained weights, and the small network. In the study, they used 70,000 steps. We added `early stopping` based on the F1 score from the evaluation dataset. Modify this for your needs. +For the cookies dataset three, the best model used 3,200 steps, pretrained weights, and the small network. In the study, they used 70,000 steps. We added `early stopping` based on the F1 score from the evaluation dataset. Modify this for your needs. We use the same config for training datasets one and two. Here's an example of the inference results with EfficientAD. It localizes the anomaly within the image through a heatmap. -![](../.gitbook/assets/fomo-ad-in-aws/dataset3.png) +![](../../.gitbook/assets/fomo-ad-in-aws/dataset3.png) -Once you're finished, terminate the remote instance. You can find the results in the [Step 3: Benchmarking](#step-3-benchmarking) section. +Once you're finished, terminate the remote instance. You can find the results in the [Step 3: Benchmarking](fomo-ad-in-aws.md#step-3-benchmarking) section. ### FOMO-AD model (automated) The last model we will build is called FOMO-AD, a visual anomaly detection learning block developed by Edge Impulse. It's based on the FOMO architecture, specifically designed for constrained devices. -![](../.gitbook/assets/fomo-ad-in-aws/edgeimpulse1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/edgeimpulse1.png) Check the [FOMO-AD documentation](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/visual-anomaly-detection) for more information. Let's automate the entire process using the Edge Impulse API: -- Import the dataset, -- Create an impulse, -- Generate features, -- Train the model, -- Export the model. + +* Import the dataset, +* Create an impulse, +* Generate features, +* Train the model, +* Export the model. There's too much code to detail here, if you want to replicate it yourself **step by step**, check out [this notebook](https://github.com/emergy-official/anomaly.parf.ai/blob/main/ai/notebooks/3_fomoad.ipynb) We separate our dataset as follows: -- Training set - - No Anomaly: 80 images (40%) -- Testing set - - No Anomaly: 20 images (10%) - - Anomaly: 100 images (50%) +* Training set + * No Anomaly: 80 images (40%) +* Testing set + * No Anomaly: 20 images (10%) + * Anomaly: 100 images (50%) -The best part of the notebook is that it includes a pre-built pipeline in Edge Impulse that will `Find the best Visual AD Model` using our dataset. All you need to do is provide the dataset and run the pipeline. After that, you'll have the optimal model set up in your project, and you can find the best threshold to use in the logs (Refer to the `Option 2` section in the notebook for more details). +The best part of the notebook is that it includes a pre-built pipeline in Edge Impulse that will `Find the best Visual AD Model` using our dataset. All you need to do is provide the dataset and run the pipeline. After that, you'll have the optimal model set up in your project, and you can find the best threshold to use in the logs (Refer to the `Option 2` section in the notebook for more details). -![](../.gitbook/assets/fomo-ad-in-aws/edgeimpulse2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/edgeimpulse2.png) Edge Impulse lets you classify your entire dataset or just one image at a time: -![](../.gitbook/assets/fomo-ad-in-aws/edgeimpulse3.png) +![](../../.gitbook/assets/fomo-ad-in-aws/edgeimpulse3.png) Once the model is exported, you can create an inference function in Python to run it locally: @@ -403,7 +405,7 @@ Since each model was trained on different sets of data, we will use the test dat Here are the results, tested on a Macbook: -![](../.gitbook/assets/fomo-ad-in-aws/benchmark1.png) +![](../../.gitbook/assets/fomo-ad-in-aws/benchmark1.png) FOMO-AD performs the best in most datasets. Although EfficientAD could be improved to score higher, it would require more time. @@ -417,28 +419,27 @@ The models are trained and ready to be used, so let's build an app to showcase o We'll include two features: -- A serverless endpoint using [SageMaker Serverless Inference](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html) with EfficientAD, - -- A real-time inference using a compact version of the Edge Impulse [mobile client](https://github.com/edgeimpulse/mobile-client/tree/master/client) with FOMO-AD. +* A serverless endpoint using [SageMaker Serverless Inference](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html) with EfficientAD, +* A real-time inference using a compact version of the Edge Impulse [mobile client](https://github.com/edgeimpulse/mobile-client/tree/master/client) with FOMO-AD. In the public repository, you will find: -- The [API Code](https://github.com/emergy-official/anomaly.parf.ai/tree/main/api), -- The [Automated Infrastructure Code (using Terraform)](https://github.com/emergy-official/anomaly.parf.ai/tree/main/infrastructure), -- The [Website Code](https://github.com/emergy-official/anomaly.parf.ai/tree/main/website). +* The [API Code](https://github.com/emergy-official/anomaly.parf.ai/tree/main/api), +* The [Automated Infrastructure Code (using Terraform)](https://github.com/emergy-official/anomaly.parf.ai/tree/main/infrastructure), +* The [Website Code](https://github.com/emergy-official/anomaly.parf.ai/tree/main/website). ### SageMaker Serverless Inference This is the infrastructure of our serverless inference endpoint: -![](../.gitbook/assets/fomo-ad-in-aws/aws3.png) +![](../../.gitbook/assets/fomo-ad-in-aws/aws3.png) When a user uploads an image to get the anomaly result, it will go through: -- Cloudfront (which is also used by the front end. Users are redirected to the API Gateway when the request path matches `/api*`), -- An API Gateway (to communicate with Lambda and allows for future API expansions), -- A Lambda that communicate to the SageMaker endpoint securely, -- A Serverless SageMaker endpoint (executes the inference using a Docker container). +* Cloudfront (which is also used by the front end. Users are redirected to the API Gateway when the request path matches `/api*`), +* An API Gateway (to communicate with Lambda and allows for future API expansions), +* A Lambda that communicate to the SageMaker endpoint securely, +* A Serverless SageMaker endpoint (executes the inference using a Docker container). The SageMaker endpoint operates using a Docker image. You can build your dockerfile like this: @@ -498,7 +499,7 @@ The serverless inference is quite slow (12 sec per inference), you can speed thi If you've previously played with Edge Impulse, you might be familiar with the `Launch in browser` feature that lets you test your model in real-time. -![](../.gitbook/assets/fomo-ad-in-aws/edgeimpulse4.png) +![](../../.gitbook/assets/fomo-ad-in-aws/edgeimpulse4.png) Wouldn't it be great to include this feature directly in our web app? @@ -508,30 +509,30 @@ The way it work is that the client is downloading a **web assembly** .zip file o We're going to modify this a bit. -- We’ll no longer use the API KEY. -- We’ll include the web assembly zip file directly in the website’s assets. _(you can download this file manually from Edge Impulse, or it can be downloaded automatically using the API when building the website assets)_, -- We'll keep only the essential code and update what's needed to make it work the new way, -- We'll add a colormap function for fun to show the model's confidence. +* We’ll no longer use the API KEY. +* We’ll include the web assembly zip file directly in the website’s assets. _(you can download this file manually from Edge Impulse, or it can be downloaded automatically using the API when building the website assets)_, +* We'll keep only the essential code and update what's needed to make it work the new way, +* We'll add a colormap function for fun to show the model's confidence. This is what we obtain: -![](../.gitbook/assets/fomo-ad-in-aws/site2.png) +![](../../.gitbook/assets/fomo-ad-in-aws/site2.png) All the modifications are detailed [here](https://github.com/emergy-official/anomaly.parf.ai/tree/main/website#mobile-client-compressed-version-detail) in the `Mobile Client compressed version detail` section. ### Website -For the website, we're using [Astro](https://astro.build/) with React based on the [AstroWind](https://astrowind.vercel.app/) template. +For the website, we're using [Astro](https://astro.build/) with React based on the [AstroWind](https://astrowind.vercel.app/) template. -To automatically deploy the website, we use [this github action](https://github.com/emergy-official/anomaly.parf.ai/blob/main/.github/workflows/staticWebsite.yml). It triggers a deployment whenever the commit message includes `deploy:website`. +To automatically deploy the website, we use [this github action](https://github.com/emergy-official/anomaly.parf.ai/blob/main/.github/workflows/staticWebsite.yml). It triggers a deployment whenever the commit message includes `deploy:website`. The website is hosted on AWS within an S3 bucket and is behind a Cloudfront distribution. -![](../.gitbook/assets/fomo-ad-in-aws/site4.png) +![](../../.gitbook/assets/fomo-ad-in-aws/site4.png) It also features a dataset explorer that showcases the data used for benchmarking: -![](../.gitbook/assets/fomo-ad-in-aws/site3.png) +![](../../.gitbook/assets/fomo-ad-in-aws/site3.png) It includes all the images, scores, predictions, and timings for all the models and cookies. @@ -540,4 +541,3 @@ It includes all the images, scores, predictions, and timings for all the models One key improvement could be enhancing the dataset. We used a mobile phone with a combination of natural and artificial lighting. The model's performance might improve if you create a synthetic dataset using [Onmiverse Replicator](https://docs.omniverse.nvidia.com/extensions/latest/index.html) featuring different lighting conditions, backgrounds, and more. It will eliminate manual processing, and you won't need to run 10 km to burn off all the cookies you've eaten. - diff --git a/accelerometer-and-activity-projects/gesture-appliances-control-brainchip.md b/readme/featured-machine-learning-projects/gesture-appliances-control-brainchip.md similarity index 79% rename from accelerometer-and-activity-projects/gesture-appliances-control-brainchip.md rename to readme/featured-machine-learning-projects/gesture-appliances-control-brainchip.md index af3b5071..6c8b7282 100644 --- a/accelerometer-and-activity-projects/gesture-appliances-control-brainchip.md +++ b/readme/featured-machine-learning-projects/gesture-appliances-control-brainchip.md @@ -1,14 +1,15 @@ --- -description: Control your TV, Air Conditioner or Lightbulb by just pointing your finger at them, using the BrainChip AKD1000 achieving great accuracy and low power consumption. +description: >- + Control your TV, Air Conditioner or Lightbulb by just pointing your finger at + them, using the BrainChip AKD1000 achieving great accuracy and low power + consumption. --- -# Gesture Appliances Control with Pose Detection - BrainChip AKD1000 +# Gesture Appliances Control with Pose Detection - BrainChip AKD1000 -Created By: -Christopher Mendez +Created By: Christopher Mendez -Public Project Link: -[https://studio.edgeimpulse.com/public/319871/latest](https://studio.edgeimpulse.com/public/319871/latest) +Public Project Link: [https://studio.edgeimpulse.com/public/319871/latest](https://studio.edgeimpulse.com/public/319871/latest) ## Introduction @@ -20,29 +21,29 @@ This is why I thought "What if we could control the whole house with just gestur ## Hardware and Software Requirements -To develop this project we will use a __BrainChip Akida Development Kit__ and a __Logitech BRIO 4K Webcam__, together with an __Edge Impulse__ Machine Learning model for pose identification. +To develop this project we will use a **BrainChip Akida Development Kit** and a **Logitech BRIO 4K Webcam**, together with an **Edge Impulse** Machine Learning model for pose identification. -![Hardware required for the project](../.gitbook/assets/gesture-appliances-control-brainchip/hardware.png) +![Hardware required for the project](../../.gitbook/assets/gesture-appliances-control-brainchip/hardware.png) ### Akida Dev Kit -It should be noted that this kit is the main component of this project thanks to some interesting characteristics that make it ideal for this use case. This kit consists of a Raspberry Pi Compute Module 4 with Wi-Fi and 8 GB RAM, also its IO Board, which includes a PCIe interface to carry an **Akida PCIe board** with the **AKD1000 Neuromorphic Hardware Accelerator**. +It should be noted that this kit is the main component of this project thanks to some interesting characteristics that make it ideal for this use case. This kit consists of a Raspberry Pi Compute Module 4 with Wi-Fi and 8 GB RAM, also its IO Board, which includes a PCIe interface to carry an **Akida PCIe board** with the **AKD1000 Neuromorphic Hardware Accelerator**. Considering that our project will end up being one more smart device that we will have at home, it's crucial that it can do its job efficiently and with very low energy consumption. This is where BrainChip's technology makes sense. Akida™ neuromorphic processor mimics the human brain to analyze only essential sensor inputs at the point of acquisition—processing data with unparalleled performance, precision, and economy of energy. ### Software -The whole system will be running independently identifying poses, if a desired pose is detected it will send an HTTP request to the Google Assistant SDK being hosted by a Raspberry Pi with Home Assistant OS. +The whole system will be running independently identifying poses, if a desired pose is detected it will send an HTTP request to the Google Assistant SDK being hosted by a Raspberry Pi with Home Assistant OS. ## Setting up the Development Environment -The system comes with the basic requirements installed to run machine learning models using Akida processor acceleration. Once the system is powered up and connected to the internet (I used an ethernet cable), you can access it by an SSH connection: you will need to know the device's local IP address, in my case, I got it from the list of connected devices of my router. +The system comes with the basic requirements installed to run machine learning models using Akida processor acceleration. Once the system is powered up and connected to the internet (I used an ethernet cable), you can access it by an SSH connection: you will need to know the device's local IP address, in my case, I got it from the list of connected devices of my router. -![Device IP Address](../.gitbook/assets/gesture-appliances-control-brainchip/ip_show.png) +![Device IP Address](../../.gitbook/assets/gesture-appliances-control-brainchip/ip_show.png) -To verify the device is working properly, you can try an included demo by navigating to **http://**, in my case to http://10.0.0.150 and try some of the examples: +To verify the device is working properly, you can try an included demo by navigating to **http://\**, in my case to http://10.0.0.150 and try some of the examples: -![Built-in demo running](../.gitbook/assets/gesture-appliances-control-brainchip/demo.png) +![Built-in demo running](../../.gitbook/assets/gesture-appliances-control-brainchip/demo.png) To start setting up the device for a custom model deployment, let's verify we have installed all the packages we need. @@ -59,7 +60,7 @@ lspci | grep Co-processor # will check if the PCIe card is plugged in correctly. python3 --version # will check the installed Python version (3.8 is required). ``` -![Verifying packages](../.gitbook/assets/gesture-appliances-control-brainchip/verifications.png) +![Verifying packages](../../.gitbook/assets/gesture-appliances-control-brainchip/verifications.png) You will also need Node Js v14.x to be able to use the [Edge Impulse CLI](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-installation). Install it by running these commands: @@ -69,6 +70,7 @@ curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash - sudo apt-get install -y nodejs node -v ``` + The last command should return the node version, v14 or above. Finally, let's install the [Linux Python SDK](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk), you just need to run these commands: @@ -85,15 +87,15 @@ pip3 install edge_impulse_linux -i https://pypi.python.org/simple First, we need to create an [Edge Impulse Studio](https://studio.edgeimpulse.com) account if we haven't yet, and create a new project: -![New project creation](../.gitbook/assets/gesture-appliances-control-brainchip/new_project.png) +![New project creation](../../.gitbook/assets/gesture-appliances-control-brainchip/new_project.png) For the creation of the dataset of our model, we have two options, uploading the images from the BrainChip Development Kit or using our computer or phone. In this case, I chose to take them from the computer using the same webcam that we are finally going to use in the project. -![Dataset creating source](../.gitbook/assets/gesture-appliances-control-brainchip/pc_upload.png) +![Dataset creating source](../../.gitbook/assets/gesture-appliances-control-brainchip/pc_upload.png) The dataset consists of 3 classes in which we finger point each appliance and a last one of unknown cases. -![Raw image & PoseNet output](../.gitbook/assets/gesture-appliances-control-brainchip/classes.png) +![Raw image & PoseNet output](../../.gitbook/assets/gesture-appliances-control-brainchip/classes.png) > **Taking at least +50 pictures of each class will let you create a robust enough model** @@ -105,38 +107,38 @@ In the left side menu, we navigate to **Impulse design** > **Create impulse** an ### Input block (Image data): -- Image width: 192 -- Image height: 192 -- Resize mode: Fit longest +* Image width: 192 +* Image height: 192 +* Resize mode: Fit longest ### Processing block (PoseNet): Use this block to turn raw images into pose vectors, then pair it with an ML block to detect what a person is doing. -PoseNet processing block is just enabled for Enterprise projects, if we want to use it on a Developer one, we need to locally run the block, for this, you must clone the [PoseNet block repository](https://github.com/edgeimpulse/pose-estimation-processing-block) and follow the __README__ steps. +PoseNet processing block is just enabled for Enterprise projects, if we want to use it on a Developer one, we need to locally run the block, for this, you must clone the [PoseNet block repository](https://github.com/edgeimpulse/pose-estimation-processing-block) and follow the **README** steps. You will end up with an URL similar to "https://abe7-2001-1308-a2ca-4f00-e65f-1ff-fe27-d3aa.ngrok-free.app" hosting the processing block, click on **Add a processing block** > **Add custom block**, then paste the [**ngrok**](https://ngrok.com/) generated URL, and click on **Add block**. -![Adding a Custom Block](../.gitbook/assets/gesture-appliances-control-brainchip/custom_block.png) +![Adding a Custom Block](../../.gitbook/assets/gesture-appliances-control-brainchip/custom_block.png) ### Learning block (BrainChip Akida) To classify the features extracted from the different poses, we'll use a classification learn block specifically designed for the hardware we're using. -![Adding a Custom Block](../.gitbook/assets/gesture-appliances-control-brainchip/learning.png) +![Adding a Custom Block](../../.gitbook/assets/gesture-appliances-control-brainchip/learning.png) Finally, we save the **Impulse design**, it should end up looking like this: -![Adding a Custom Block](../.gitbook/assets/gesture-appliances-control-brainchip/impulse_design_2.png) +![Adding a Custom Block](../../.gitbook/assets/gesture-appliances-control-brainchip/impulse_design_2.png) ## Model Training -After having designed the impulse, it's time to set the processing and learning blocks. The **Pose estimation** block doesn't have any configurable parameters, so we just need to click on **Save parameters** and then **Generate features**. +After having designed the impulse, it's time to set the processing and learning blocks. The **Pose estimation** block doesn't have any configurable parameters, so we just need to click on **Save parameters** and then **Generate features**. In the _classifier block_ define the following settings: -- Number of training cycles: 100 -- Learning rate: 0.001 +* Number of training cycles: 100 +* Learning rate: 0.001 In the Neural network architecture, add 3 Dense layers with 35, 25 and 10 neurons respectively. @@ -254,11 +256,11 @@ akida_model = akida_perform_qat( callbacks=callbacks) ``` -Click on the __Start training__ button and wait for the model to be trained and the confusion matrix to show up. +Click on the **Start training** button and wait for the model to be trained and the confusion matrix to show up. -### Confusion Matrix +### Confusion Matrix -![Confusion matrix results](../.gitbook/assets/gesture-appliances-control-brainchip/confusion.png) +![Confusion matrix results](../../.gitbook/assets/gesture-appliances-control-brainchip/confusion.png) The results of the confusion matrix can be improved by adding more samples to the dataset. @@ -270,6 +272,7 @@ To be able to run the project, we need to go back to our SSH connection with the bash git clone https://github.com/edgeimpulse/pose-akida-classification.git ``` + Install all the project requirements with the following command, and wait for the process to be done. ``` @@ -288,7 +291,7 @@ apt-get update && apt-get install ffmpeg libsm6 libxext6 -y Once the project is cloned locally in the Akida Development Kit, you can download the project model from Edge Impulse Studio by navigating to the **Dashboard** section and downloading the **MetaTF** `.fbz` file. -![Downloading the project model](../.gitbook/assets/gesture-appliances-control-brainchip/model-down.png) +![Downloading the project model](../../.gitbook/assets/gesture-appliances-control-brainchip/model-down.png) Once downloaded, from the model path, open a new terminal and copy the model to the Dev Kit using `scp` command as follows: @@ -297,6 +300,7 @@ bash scp .fbz ubuntu@:~ # command format scp akida_model.fbz ubuntu@10.0.0.154:~ # actual command in my case ``` + > _You will be asked for your Linux machine login password._ Now, the model is on the Akida Dev Kit local storage `(/home/ubuntu)` and you can verify it by listing the directory content using `ls`. @@ -310,7 +314,7 @@ mv akida_model.fbz ./pose-akida-classification/ Here we have the model on the project directory, so now everything is ready to be run. -![Project directory](../.gitbook/assets/gesture-appliances-control-brainchip/model-copy.png) +![Project directory](../../.gitbook/assets/gesture-appliances-control-brainchip/model-copy.png) ## Run Inferencing @@ -321,13 +325,13 @@ bash python3 class-pose.py akida_model.fbz 0 ``` -- The first parameter `class-pose.py` is the project's main script to be run. -- `akida_model.fbz` is the Meta TF model name we downloaded from our Edge Impulse project. -- `0` force the script to use the first camera available. +* The first parameter `class-pose.py` is the project's main script to be run. +* `akida_model.fbz` is the Meta TF model name we downloaded from our Edge Impulse project. +* `0` force the script to use the first camera available. The project will start running and printing the inference results continuously in the terminal. -![Project running and printing the results](../.gitbook/assets/gesture-appliances-control-brainchip/running.png) +![Project running and printing the results](../../.gitbook/assets/gesture-appliances-control-brainchip/running.png) To watch a preview of the camera feed, you can do it by opening a new `ssh` session and running the `make-page.py` script from the project directory: @@ -336,25 +340,25 @@ bash python3 make-page.py ``` -![Preview Web Page script command](../.gitbook/assets/gesture-appliances-control-brainchip/preview-web.png) +![Preview Web Page script command](../../.gitbook/assets/gesture-appliances-control-brainchip/preview-web.png) Finally, you will be able to see the camera preview alongside the inference results organized in the following order: `AC`, `Light`, `Other` and `TV`. -![Project running | Inference results](../.gitbook/assets/gesture-appliances-control-brainchip/results-preview.png) +![Project running | Inference results](../../.gitbook/assets/gesture-appliances-control-brainchip/results-preview.png) ## Google Assistant Setup -For the actual appliance control, I used the __Google Assistant SDK__ integration for __Home Assistant__. Follow the [documentation](https://www.home-assistant.io/integrations/google_assistant_sdk) to configure it for your setup. +For the actual appliance control, I used the **Google Assistant SDK** integration for **Home Assistant**. Follow the [documentation](https://www.home-assistant.io/integrations/google_assistant_sdk) to configure it for your setup. > **The Home Assistant is running on a separate Raspberry PI.** Once the integration is set, we can send `HTTP` requests to it with the following format: -- URL: `http://:8123/api/services/google_assistant_sdk/send_text_command` -- Headers: - - Authorization: "Bearer " - - Content-Type: "application/json" -- Body: {"command":"turn on the light"} +* URL: `http://:8123/api/services/google_assistant_sdk/send_text_command` +* Headers: + * Authorization: "Bearer " + * Content-Type: "application/json" +* Body: {"command":"turn on the light"} You must edit the `url` and `auth` variables in the code with the respective ones of your setup. @@ -366,7 +370,7 @@ auth = 'Bearer ***************************************************************** ## Demo -![Final project deployment](../.gitbook/assets/gesture-appliances-control-brainchip/setup-on.png) +![Final project deployment](../../.gitbook/assets/gesture-appliances-control-brainchip/setup-on.png) Here I show you the whole project working and controlling appliances when they are pointed. diff --git a/image-projects/getting-started-nvidia-tao-renesas-ekra8d1.md b/readme/featured-machine-learning-projects/getting-started-nvidia-tao-renesas-ekra8d1.md similarity index 75% rename from image-projects/getting-started-nvidia-tao-renesas-ekra8d1.md rename to readme/featured-machine-learning-projects/getting-started-nvidia-tao-renesas-ekra8d1.md index fb2ea2b5..fa40b9aa 100644 --- a/image-projects/getting-started-nvidia-tao-renesas-ekra8d1.md +++ b/readme/featured-machine-learning-projects/getting-started-nvidia-tao-renesas-ekra8d1.md @@ -1,6 +1,7 @@ --- description: >- - A complete end-to-end sample project and guide to get started with Nvidia TAO for the Renesas RA8D1 MCU. + A complete end-to-end sample project and guide to get started with Nvidia TAO + for the Renesas RA8D1 MCU. --- # Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1 @@ -11,13 +12,13 @@ Public Project Link: [https://studio.edgeimpulse.com/public/568291/latest](https ## Introduction -The Renesas RA8 series is the first product to implement the Arm Cortex-M85, a high-performance MCU core tailored for advanced AI and machine learning at the edge. Featuring Arm Helium technology and enhanced ML instructions, it delivers up to 4x the ML performance of earlier M-series cores. With high clock speeds, energy efficiency, and TrustZone security, it's ideal for tasks like speech recognition, anomaly detection, and image classification on embedded devices. +The Renesas RA8 series is the first product to implement the Arm Cortex-M85, a high-performance MCU core tailored for advanced AI and machine learning at the edge. Featuring Arm Helium technology and enhanced ML instructions, it delivers up to 4x the ML performance of earlier M-series cores. With high clock speeds, energy efficiency, and TrustZone security, it's ideal for tasks like speech recognition, anomaly detection, and image classification on embedded devices. Edge Impulse includes support for Nvidia TAO transfer learning and deployment of Nvidia Model Zoo models to the Renesas RA8D1. This project provides a walkthrough of how to use the Renesas EK-RA8D1 Development kit with Edge Impulse using an Nvidia TAO-enabled backend to train Nvidia Model Zoo models for deployment onto the EK-RA8D1. By integrating the EK-RA8D1 with Edge Impulse's Nvidia TAO training pipeline, you can explore advanced machine learning applications and leverage the latest features in model experimentation and deployment. -## Hardware +## Hardware Renesas EK-RA8D1 - [Evaluation Kit for RA8D1 MCU Group](https://www.renesas.com/en/products/microcontrollers-microprocessors/ra-cortex-m-mcus/ek-ra8d1-evaluation-kit-ra8d1-mcu-group?srsltid=AfmBOoovp-039RtY9ng5rk2nFEVNOQuruTXKineI1JmVl9tDr64N7Ao2) @@ -27,21 +28,19 @@ Edge Impulse [Visit](https://edgeimpulse.com) ## Software -Edge Impulse CLI [Download](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-installation) -JLink Flashing Tools [Download](https://www.segger.com/downloads/jlink) -Edge Implulse Firmware for EK-RA8D1 [Download](https://cdn.edgeimpulse.com/firmware/renesas-ek-ra8d1.zip) +Edge Impulse CLI [Download](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-installation) JLink Flashing Tools [Download](https://www.segger.com/downloads/jlink) Edge Implulse Firmware for EK-RA8D1 [Download](https://cdn.edgeimpulse.com/firmware/renesas-ek-ra8d1.zip) ## Getting Started -### Renesas EK-RA8D1 +### Renesas EK-RA8D1 -Renesas supports developers building on the RA8 with various kits, including the EK-RA8D1, a comprehensive evaluation board that simplifies prototyping. +Renesas supports developers building on the RA8 with various kits, including the EK-RA8D1, a comprehensive evaluation board that simplifies prototyping. As part of the Renesas Advanced (RA) series of MCU evaluation kits, the EK-RA8D1 features the RA8 Cortex-M85 MCU which is the latest high-end MCU from Arm, superseding the Cortex M7. The Cortex M85 is a high-performance MCU core designed for advanced embedded and edge AI applications. It offers up to 4x the ML performance of earlier Cortex-M cores, powered by Arm Helium technology for accelerated DSP and ML tasks. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/ekra8d1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/ekra8d1.jpg) -The Renesas EK-RA8D1 evaluation kit is a versatile platform designed for embedded and AI application development. It features USB Full-Speed host and device support with 5V input via USB or external power supply, along with onboard debugging through Segger J-Link® and support for ETM, SWD, and JTAG interfaces. Developers can utilize 3 user LEDs, 2 buttons, and multiple connectivity options, including Seeed Grove® (I2C & analog), Digilent Pmod™ (SPI & UART), Arduino™ Uno R3 headers, MikroElektronika™ mikroBUS, and SparkFun® Qwiic® (I2C). An MCU boot configuration jumper further enhances flexibility, making the EK-RA8D1 ideal for rapid prototyping and testing. +The Renesas EK-RA8D1 evaluation kit is a versatile platform designed for embedded and AI application development. It features USB Full-Speed host and device support with 5V input via USB or external power supply, along with onboard debugging through Segger J-Link® and support for ETM, SWD, and JTAG interfaces. Developers can utilize 3 user LEDs, 2 buttons, and multiple connectivity options, including Seeed Grove® (I2C & analog), Digilent Pmod™ (SPI & UART), Arduino™ Uno R3 headers, MikroElektronika™ mikroBUS, and SparkFun® Qwiic® (I2C). An MCU boot configuration jumper further enhances flexibility, making the EK-RA8D1 ideal for rapid prototyping and testing. The kit also features a camera and full color LCD display, making it ideal for the development and deployment of edge AI solutions allowing on-device inference results to be rendered to the onboard LCD. @@ -51,95 +50,95 @@ The EK-RA8D1 is an officially supported target in Edge Impulse, which means it c ### Create Edge Impulse Project -To get started, create a project and be sure to use an Enterprise or Professional Plan as the Nvidia TAO training pipeline requires either a Professional or Enterprise subscription. For more info on the options, [see here](https://edgeimpulse.com/pricing). +To get started, create a project and be sure to use an Enterprise or Professional Plan as the Nvidia TAO training pipeline requires either a Professional or Enterprise subscription. For more info on the options, [see here](https://edgeimpulse.com/pricing). -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/new-project.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/new-project.jpg) ### Connect your Device There two ways to connect the board, either using the Edge Impulse CLI or directly from within the Studio UI. To access via the CLI run the command `edge-impulse-daemon` and provide login credentials, then select the appropriate Studio project to connect your board. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/cli-connect.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/cli-connect.jpg) -Alternatively, clicking the _Data acquisition_ menu item in the left navigation bar presents the data collection page. Select **320x240** to get the maximum resolution out of the camera on the EK-RA8D1 when capturing samples. +Alternatively, clicking the _Data acquisition_ menu item in the left navigation bar presents the data collection page. Select **320x240** to get the maximum resolution out of the camera on the EK-RA8D1 when capturing samples. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-acquition.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-acquition.jpg) Edge Impulse will ask you if the project is object detection project. Select 'No' to configure the project as an Image classification project when using image data. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/project-type.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/project-type.jpg) Alternatively, go the Dashboard page by clicking _Dashboard_ on the left navigation and select **One label per data item** from the Labeling method dropdown. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/project-type-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/project-type-2.jpg) Capture sample images by presenting objects to the camera that you wish to identify, and click the **Start sampling** button to capture a full color image from the board. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-1.jpg) Different types or classes of object can be captured, and these can be added by changing the label string in the **Label** text box. For example, a class called `needle_sealed` is created by setting the label to this name and then capturing pictures of sealed needles. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/data-2.jpg) Once all images are annotated you should balance your data so that you split your dataset between a Training and Test set. This is done by selecting _Dashboard_ from the navigation menu on the left and then scrolling down to find and click the **Perform train / test split** button. Edge Impulse will try to get as close to an 80/20 split as possible depending on the size of your dataset. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/split-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/split-1.jpg) The data split can be seen at the top of the _Data acquisition_ page where you can not only see the split of data items collected by label as a pie chart, but also the resulting split under the TRAIN / TEST SPLIT element. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/split-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/split-2.jpg) ### Create Impulse The next step is to create a new Impulse which is accessed from the Create Impulse menu. Select the **Renesas RA8D1 (Cortex M85 480Mhz)** as the target, doing so automatically targets the EK-RA8D1 which is the RA8D1 based board supported by Edge Impulse. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/target-device.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/target-device.jpg) Set the image width and height to 224px x 224px to match the pretrained backbone dimensions in Nvidia TAO Model Zoo: -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-1.jpg) ### Feature Generation Classification requires an Image processing block; this is added by clicking **Add a processing block** and then selecting **Image** from the options presented. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-2.jpg) -Once the Image processing block is added the Transfer Learning Block needs be added by selecting **Add a learning block** and then choosing the first option, **Transfer Learning (Images)**. Nvidia TAO is based on transfer learning so selecting this block is the first step towards activating the Nvidia TAO classification pipeline in the backend. +Once the Image processing block is added the Transfer Learning Block needs be added by selecting **Add a learning block** and then choosing the first option, **Transfer Learning (Images)**. Nvidia TAO is based on transfer learning so selecting this block is the first step towards activating the Nvidia TAO classification pipeline in the backend. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-3.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-3.jpg) The resulting Impulse should look as follows before proceeding. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-4.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/impulse-4.jpg) The next step is to generate the raw features that will be used to train the model. First click **Save Impulse** then select the _Image_ submenu from the Impulse Design menu in the left hand navigation to access the settings of the Image processing block. In the **Parameters** tab, leave the color depth as RGB as the TAO Models use 3 channel RGB models: -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/features-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/features-1.jpg) Under the **Generate features** tab simply click the **Generate features** button to create the scaled down 224x224 images that will be used by TAO to train and validate the model. The process will take a few seconds to minutes depending on the dataset size. Once done the results of the job are shown and the reduced images are stored in the backend as features to be passed to the model during training and validation. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/features-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/features-2.jpg) ### Nvidia TAO Classification Once the image features are done, a green dot appears next to Images in the Impulse design navigation. The Transfer Learning submenu is then activated, and can be accessed by clicking **Transfer learning** in the navigation pane under Impulse design, this takes you to the configuration area of the learning block. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-1.jpg) -To activate Nvidia TAO in the project the default MobileNetV2 model architecture needs to be deleted, by clicking the Delete model (trash can) icon on the lower right corner of the model. +To activate Nvidia TAO in the project the default MobileNetV2 model architecture needs to be deleted, by clicking the Delete model (trash can) icon on the lower right corner of the model. Once this is done you will see there is no model architecture activated for the project, and a button titled "Choose a different model" will be shown in place of the deleted MobileNet model. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-2.jpg) Clicking the "Choose a different model" button will present a list of model architectures available in Edge Impulse. Since the project is configured as Classification, only classification model architectures are available. To access the Nvidia TAO Classification Model scroll down to the bottom. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-3.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-3.jpg) The Nvidia TAO models are only available under Professional and Enterprise subscriptions as shown by the labels. For this project we are going to use **Nvidia TAO Image Classification**. Selecting any of the Nvidia TAO models like this activates the Nvidia TAO training environment automatically behind the scenes in the project. @@ -147,63 +146,62 @@ The Nvidia TAO models are only available under Professional and Enterprise subsc Once the Nvidia TAO Classification model is selected all the relevant hyperparameters are exposed by the GUI. The default training settings are under the _Training settings_ menu and the _Advanced training settings_ menu can be expanded to show the full set of parameters specific to TAO. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-4.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-4.jpg) All of the relevant settings available in TAO including Data augmentation and backbone selection are available from the GUI. The data augmentation features of TAO can be accessed by expanding the _Augmentation settings_ menu. Backbone selection is accessed from the _Backbone_ dropdown menu and for this project we will be using the **MobileNet v2 (800K params)** backbone. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-6.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-6.jpg) It's also essential to select **GPU** for training as TAO only trains on GPU's. Also set the number of training cycles (epochs) to a higher number than the default. Here we start with 300. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-7.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-7.jpg) All that's left to do is click the **Save and train** button to commence training. This can take from 1 to several hours depending upon the dataset size and other factors such as backbone, etc. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-8.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-8.jpg) Once training is completed, the results are shown: -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-9.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/training-9.jpg) -The accuracy and confusion matrix, latency and memory usage are shown for both Unoptmized (float32) and Quantized (int8) models, which can be used with the EK-RA8AD1. Take note of the **PEAK RAM USAGE** and **FLASH USAGE** statistics at the bottom. These indicate if the model will fit within RAM and ROM on the target. +The accuracy and confusion matrix, latency and memory usage are shown for both Unoptmized (float32) and Quantized (int8) models, which can be used with the EK-RA8AD1. Take note of the **PEAK RAM USAGE** and **FLASH USAGE** statistics at the bottom. These indicate if the model will fit within RAM and ROM on the target. ### Model Testing Before deploying the model to the development kit, the model can first be tested by accessing the _Live classification_ menu on the left navigation. Clicking the **Classify all** button runs the Test dataset through the model, and shows the results on the right: -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-1.jpg) The results are visible in the right side of the window, and can give a good indication of the model performance against the captured dataset. The Model testing page also allows you to perform realtime classification using uploaded files, by selecting a file from the **Classify existing test sample** dropdown menu and clicking the **Load sample** button. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-2.jpg) The results shown when doing this are from the classification being performed in Edge Impulse, not on the device. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-3.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-3.jpg) If you wish to test the camera on the EK-RA8D1 but still run the model in Edge Impulse, you can connect the camera using the `edge-impulse-daemon` CLI command to connect the camera just as you would when you perform data acquisition. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-4.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-4.jpg) You can iteratively improve the model by capturing more data and choosing the Retrain model sub menu item which takes you to the retrain page where you can simply click the Train model button to retrain the model with the existing hyperparameters. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-5.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/testing-5.jpg) ## Deployment To test the model directly on the EK-RA8D1, go to the _Deployment_ page by clicking the Deployment sub menu item in the left navigation. In the search box type _Renesas_. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-1.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-1.jpg) The drop down menu will filter out all the other supported boards and give you two options for the EK-RA8D1. The RA8D1 MCU itself has 2Mb of FLASH for code storage and 1Mb of RAM integrated. The EK-RA8D1 development kit adds 64Mb of external SDRAM and 64Mb of external QSPI FLASH to support bigger models. The **Quantized (int8)** model should be selected by default and the RAM and ROM usage is shown, which is what you would have seen in the training page when training completed. - - Renesas EK-RA8D1 target – This builds a binary for when RAM and ROM usage fit within the RA8D1 MCU's integrated RAM and FLASH memory. - - - Renesas EK-RA8D1 SDRAM target – This builds a binary that loads the model into the external SDRAM when the model is over 1Mb. (Note there is a slight performance penalty as the external RAM has to be accessed over a memory bus and is also SDRAM vs the internal SRAM) +* Renesas EK-RA8D1 target – This builds a binary for when RAM and ROM usage fit within the RA8D1 MCU's integrated RAM and FLASH memory. +* Renesas EK-RA8D1 SDRAM target – This builds a binary that loads the model into the external SDRAM when the model is over 1Mb. (Note there is a slight performance penalty as the external RAM has to be accessed over a memory bus and is also SDRAM vs the internal SRAM) When you click the **Build** button Edge Impulse builds the project and generates a `.zip` archive containing the prebuilt binary and supporting files, which downloads automatically when completed. @@ -211,23 +209,23 @@ This archive contains the same files as the Edge Impulse firmware you would have To flash the new firmware to your board, replace the contents of the folder where you have the firmware with the contents of the downloaded archive. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-2.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-2.jpg) Note, you need to make sure you have connected the USB cable to the JLink port (J10). -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-3.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-3.jpg) Run the appropriate command to flash the firmware to the board. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-4.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-4.jpg) To test the performance of the image classification on the board and see inference latency and DSP processing time, connect the USB cable to J11. -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-5.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/deployment-5.jpg) Then run the `edge-impulse-run-impulse` CLI command: -![](../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/results.jpg) +![](../../.gitbook/assets/getting-started-nvidia-tao-renesas-ekra8d1/results.jpg) The inference execution time and results are then shown in the CLI. diff --git a/image-projects/high-speed-counting-jetson-nano.md b/readme/featured-machine-learning-projects/high-speed-counting-jetson-nano.md similarity index 64% rename from image-projects/high-speed-counting-jetson-nano.md rename to readme/featured-machine-learning-projects/high-speed-counting-jetson-nano.md index b675d17f..c88ac2f9 100644 --- a/image-projects/high-speed-counting-jetson-nano.md +++ b/readme/featured-machine-learning-projects/high-speed-counting-jetson-nano.md @@ -1,27 +1,28 @@ --- description: >- - High speed object counting with computer vision and an Nvidia Jetson Nano Developer Kit. + High speed object counting with computer vision and an Nvidia Jetson Nano + Developer Kit. --- -# High-resolution, High-speed Object Counting - Nvidia Jetson Nano +# High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT) Created By: Jallson Suryo Public Project Link: [https://studio.edgeimpulse.com/public/207728/live](https://studio.edgeimpulse.com/public/207728/live) -GitHub Repo: [https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720](https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720) +GitHub Repo: [https://github.com/Jallson/High\_res\_hi\_speed\_object\_counting\_FOMO\_720x720](https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720) -![](../.gitbook/assets/high-speed-counting-jetson-nano/Photo01.png) +![](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo01.png) ## Problem Statement -The object counting systems in the manufacturing industry are essential to inventory management and supply chains. They mostly use proximity sensors or color sensors to detect objects for counting. Proximity sensors detect the presence or absence of an object based on its proximity to the sensor, while color sensors can distinguish objects based on their color or other visual characteristics. There are some limitations of these systems though; they typically have difficulty detecting small objects in large quantities, especially when they are not in a row or orderly manner. This can be compounded by a relatively fast conveyor belt. These conditions make the object counting inaccurate. +The object counting systems in the manufacturing industry are essential to inventory management and supply chains. They mostly use proximity sensors or color sensors to detect objects for counting. Proximity sensors detect the presence or absence of an object based on its proximity to the sensor, while color sensors can distinguish objects based on their color or other visual characteristics. There are some limitations of these systems though; they typically have difficulty detecting small objects in large quantities, especially when they are not in a row or orderly manner. This can be compounded by a relatively fast conveyor belt. These conditions make the object counting inaccurate. ## Our Solution After experimenting with computer vision on the Jetson Nano [in a previous project](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/quality-control-jetson-nano), I believe that a computer vision system with its object detection capabilities can explore its potential to accurately count small objects in large quantities and on fast-moving conveyor belts. Basically, we'll explore the capability of [Edge Impulse's FOMO models](https://edge-impulse.gitbook.io/docs/edge-impulse-studio/learning-blocks/object-detection/fomo-object-detection-for-constrained-devices) that have been optimized for the GPU in the Jetson Nano. In this project, the production line / conveyor belt will run quite fast, with lots of small objects in random positions, and the number of objects will be counted live and displayed on a 16x2 LCD display. Speed and accuracy are the goals of the project. -![Schematic diagram](../.gitbook/assets/high-speed-counting-jetson-nano/Photo02.png) +![Schematic diagram](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo02.png) ## How Does It Work? @@ -29,23 +30,23 @@ This project utilizes Edge Impulse's FOMO algorithm, which can quickly detect ob The machine learning model (named `model.eim`) will be deployed using the TensorRT library, configured with GPU optimizations and integrated through the Linux C++ SDK. Additionally, the Edge Impulse model will be seamlessly integrated into our Python codebase to facilitate cumulative object counting. Our proprietary algorithm compares current frame coordinates with those of previous frames to identify new objects and avoid duplicate counting. -![Jetson Nano, camera, and conveyor belt](../.gitbook/assets/high-speed-counting-jetson-nano/Photo03.png) +![Jetson Nano, camera, and conveyor belt](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo03.png) ### Hardware Requirements: -- NVIDIA Jetson Nano Developer Kit -- USB Camera (eg. Logitech C922) -- Mini conveyer belt system with camera stand -- Objects: eg. bolt -- Ethernet cable -- PC/Laptop to access Jetson Nano via SSH +* NVIDIA Jetson Nano Developer Kit +* USB Camera (eg. Logitech C922) +* Mini conveyer belt system with camera stand +* Objects: eg. bolt +* Ethernet cable +* PC/Laptop to access Jetson Nano via SSH ### Software & Online Services: -- Edge Impulse Studio -- Edge Impulse Linux, Python & C++ SDK -- NVIDIA Jetpack SDK -- Terminal +* Edge Impulse Studio +* Edge Impulse Linux, Python & C++ SDK +* NVIDIA Jetpack SDK +* Terminal ## Steps @@ -53,53 +54,52 @@ The machine learning model (named `model.eim`) will be deployed using the Tensor In this project we use a Logitech C922 USB camera capable of 720p at 60 fps connected to a PC/laptop to capture the images for data collection, for ease of use. Take pictures from above the parts, at slightly different angles and lighting conditions to ensure that the model can work under different conditions (to prevent overfitting). Object size is a crucial aspect when using FOMO, to ensure the performance of this model. You must keep the camera distance from the objects consistent, because significant difference in object sizes will confuse the algorithm and cause difficulty in the auto-labelling process. -![Data variation](../.gitbook/assets/high-speed-counting-jetson-nano/Photo04.png) +![Data variation](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo04.png) ### 2. Data Acquisition and Labeling Open studio.edgeimpulse.com, login or create an account then create a new project. -Choose the *Images* project option, then *Object detection*. In *Dashboard > Project Info*, choose *Bounding Boxes* for the labeling method and *NVIDIA Jetson Nano* for the target device. Then in *Data acquisition*, click on *Upload Data* tab, choose your photo files, automatically split them between Training and Testing, then click on *Begin upload*. - -Next, +Choose the _Images_ project option, then _Object detection_. In _Dashboard > Project Info_, choose _Bounding Boxes_ for the labeling method and _NVIDIA Jetson Nano_ for the target device. Then in _Data acquisition_, click on _Upload Data_ tab, choose your photo files, automatically split them between Training and Testing, then click on _Begin upload_. -- For Developer accounts: click on the *Labeling queue* tab then drag a bounding box around an object and label it, then click Save. Repeat this until all images labelled. It goes quickly though, as the bounding boxes will attempt to follow an object from image to image. +Next, - - For Enterprise accounts: click on *Auto-Labeler* in *Data Acquisition*. This auto-labeling segmentation / cluster process will save a lot of time over the manual process above. Set min/max object pixels and sim threshold (0.9 - 0.999) to adjust the sensitivity of cluster detection, then click *Run*. If something doesn't match or if there is additional data, labeling can be done manually as well. +* For Developer accounts: click on the _Labeling queue_ tab then drag a bounding box around an object and label it, then click Save. Repeat this until all images labelled. It goes quickly though, as the bounding boxes will attempt to follow an object from image to image. +* For Enterprise accounts: click on _Auto-Labeler_ in _Data Acquisition_. This auto-labeling segmentation / cluster process will save a lot of time over the manual process above. Set min/max object pixels and sim threshold (0.9 - 0.999) to adjust the sensitivity of cluster detection, then click _Run_. If something doesn't match or if there is additional data, labeling can be done manually as well. -![Upload data](../.gitbook/assets/high-speed-counting-jetson-nano/Photo05.png) +![Upload data](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo05.png) -![Auto-labeling](../.gitbook/assets/high-speed-counting-jetson-nano/Photo06.png) +![Auto-labeling](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo06.png) -![Label cluster](../.gitbook/assets/high-speed-counting-jetson-nano/Photo07.png) +![Label cluster](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo07.png) -![Manual labeling](../.gitbook/assets/high-speed-counting-jetson-nano/Photo08.png) +![Manual labeling](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo08.png) -![Balance ratio_80/20](../.gitbook/assets/high-speed-counting-jetson-nano/Photo09.png) +![Balance ratio\_80/20](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo09.png) ### 3. Train and Build Model Using FOMO Object Detection -Once you have the dataset ready, go to *Create Impulse* and set 720 x 720 as the image width and height. Then choose *Fit shortest axis*, and choose *Image* and *Object Detection* as Learning and Processing blocks. +Once you have the dataset ready, go to _Create Impulse_ and set 720 x 720 as the image width and height. Then choose _Fit shortest axis_, and choose _Image_ and _Object Detection_ as Learning and Processing blocks. -In the Image block configuration, select Grayscale as the color depth and click *Save parameters*. Then click on *Generate features* to get a visual representation of the features extracted from each image in the dataset. Navigate to the Object Detection block setup, and leave the default selections as-is for the Neural Network, but perhaps bump up the number of training epochs to 120. Then we choose *FOMO (MobileNet V2 0.35)*, and train the model by clicking the *Start training* button. You can see the progress on the right side of the page. +In the Image block configuration, select Grayscale as the color depth and click _Save parameters_. Then click on _Generate features_ to get a visual representation of the features extracted from each image in the dataset. Navigate to the Object Detection block setup, and leave the default selections as-is for the Neural Network, but perhaps bump up the number of training epochs to 120. Then we choose _FOMO (MobileNet V2 0.35)_, and train the model by clicking the _Start training_ button. You can see the progress on the right side of the page. -If everything is OK, then we can test the model, go to *Model Testing* on the left navigation and click *Classify all*. Our result is above 90%, so we can move on to the next step — Deployment. +If everything is OK, then we can test the model, go to _Model Testing_ on the left navigation and click _Classify all_. Our result is above 90%, so we can move on to the next step — Deployment. -![Blocks](../.gitbook/assets/high-speed-counting-jetson-nano/Photo10.png) +![Blocks](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo10.png) -![Save parameters](../.gitbook/assets/high-speed-counting-jetson-nano/Photo11.png) +![Save parameters](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo11.png) -![Generate features](../.gitbook/assets/high-speed-counting-jetson-nano/Photo12.png) +![Generate features](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo12.png) -![Result](../.gitbook/assets/high-speed-counting-jetson-nano/Photo13.png) +![Result](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo13.png) -![Test](../.gitbook/assets/high-speed-counting-jetson-nano/Photo14.png) +![Test](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo14.png) ### 4. Deploy Model Targeting Jetson Nano's GPU -Click on the *Deployment* navigation item, then search for *TensorRT*. Select *Float32* and click *Build*. This will build an NVIDIA TensorRT library for running inferencing targeting the Jetson Nano's GPU. After it has downloaded, open the `.zip` file and then we're ready for model deployment with the Edge Impulse C++ SDK directly on the NVIDIA Jetson Nano. +Click on the _Deployment_ navigation item, then search for _TensorRT_. Select _Float32_ and click _Build_. This will build an NVIDIA TensorRT library for running inferencing targeting the Jetson Nano's GPU. After it has downloaded, open the `.zip` file and then we're ready for model deployment with the Edge Impulse C++ SDK directly on the NVIDIA Jetson Nano. -![TensorRT build library](../.gitbook/assets/high-speed-counting-jetson-nano/Photo15.png) +![TensorRT build library](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo15.png) On the Jetson Nano, there are several things that need to be done to get ready for our project. Make sure the device is running it's native Ubuntu OS and JetPack which are usually pre-installed on the SD card. More information on [downloading and flashing the SD Card is available here](https://developer.nvidia.com/jetpack-sdk-463). Then `ssh` via a PC or laptop with Ethernet and setup Edge Impulse firmware in the terminal: @@ -144,15 +144,15 @@ Now the model is ready to run in a high-level language such as the Python progra edge-impulse-linux-runner --model-file /model.eim ``` -![Video stream from your browser](../.gitbook/assets/high-speed-counting-jetson-nano/Video01.gif) +![Video stream from your browser](../../.gitbook/assets/high-speed-counting-jetson-nano/Video01.gif) The inferencing time is around 15ms, which is an incredibly fast detection speed. To compare these results, I have also deployed with the standard CPU-based deployment option (Linux AARCH64 model), and run with the same command above. The inferencing time is around 151ms with a Linux model that targets the CPU. -![Deploy to CPU](../.gitbook/assets/high-speed-counting-jetson-nano/Photo16.png) +![Deploy to CPU](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo16.png) -![CPU vs GPU](../.gitbook/assets/high-speed-counting-jetson-nano/Photo17.png) +![CPU vs GPU](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo17.png) You can see the difference in inferencing time is about 10x faster when we target the GPU for the process. Impressive! @@ -160,9 +160,9 @@ You can see the difference in inferencing time is about 10x faster when we targe Before we start with Python, we need to install the Edge Impulse Python SDK and clone the repository from the previous Edge Impulse examples. Follow the steps here, [https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk). -With the impressive performance of live inferencing in the Runner, now we will create a Python program to be able to calculate the cumulative count of moving objects taken from camera capture. The program is a modification of Edge Impulse's `classify.py` in `examples/image` from the `linux-python-sdk directory`. We turned it into an object tracking program by solving a bipartite matching problem so the same object can be tracked across different frames to avoid double counting. For more detail, you can download and check the python program at this link, [https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720](https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720) +With the impressive performance of live inferencing in the Runner, now we will create a Python program to be able to calculate the cumulative count of moving objects taken from camera capture. The program is a modification of Edge Impulse's `classify.py` in `examples/image` from the `linux-python-sdk directory`. We turned it into an object tracking program by solving a bipartite matching problem so the same object can be tracked across different frames to avoid double counting. For more detail, you can download and check the python program at this link, [https://github.com/Jallson/High\_res\_hi\_speed\_object\_counting\_FOMO\_720x720](https://github.com/Jallson/High_res_hi_speed_object_counting_FOMO_720x720) -![count_moving_bolt.py](../.gitbook/assets/high-speed-counting-jetson-nano/Photo18.png) +![count\_moving\_bolt.py](../../.gitbook/assets/high-speed-counting-jetson-nano/Photo18.png) You can git clone the repo, or then run the program with the command pointing to the path where `model.eim` is located: @@ -170,7 +170,7 @@ You can git clone the repo, or then run the program with the command pointing to python3 count_moving_bolt.py /model.eim ``` -Here is a demo video of the results: +Here is a demo video of the results: {% embed url="https://youtu.be/ouqvACe48ts" %} diff --git a/nvidia-deepstream-community-guide.md b/readme/featured-machine-learning-projects/nvidia-deepstream-community-guide.md similarity index 78% rename from nvidia-deepstream-community-guide.md rename to readme/featured-machine-learning-projects/nvidia-deepstream-community-guide.md index 153d6fcb..0f50a178 100644 --- a/nvidia-deepstream-community-guide.md +++ b/readme/featured-machine-learning-projects/nvidia-deepstream-community-guide.md @@ -1,28 +1,29 @@ --- -description: How to create computer vision models with the Edge Impulse Studio, and then transform those models into a format that is usable by Nvidia DeepStream applications. +description: >- + How to create computer vision models with the Edge Impulse Studio, and then + transform those models into a format that is usable by Nvidia DeepStream + applications. --- -# Community Guide — Using Edge Impulse with Nvidia DeepStream +# Community Guide – Using Edge Impulse with Nvidia DeepStream -Created By: -Peter Ing +Created By: Peter Ing -GitHub Repo: -[https://github.com/peteing/DeepStream-EdgeImpulse](https://github.com/peteing/DeepStream-EdgeImpulse) +GitHub Repo: [https://github.com/peteing/DeepStream-EdgeImpulse](https://github.com/peteing/DeepStream-EdgeImpulse) ## Introduction -Nvidia DeepStream is an audio, video, and image analytics toolkit based on GStreamer for AI and computer vision processing. It can be used as a building block for creating applications and services that incorporate machine learning and computer vision components or capabilities, such as human-machine interfaces, self-driving and autonomous vehicles, security, and many other use-cases. Detailed information and documentation for DeepStream is located here: [https://developer.nvidia.com/deepstream-sdk](https://developer.nvidia.com/deepstream-sdk). +Nvidia DeepStream is an audio, video, and image analytics toolkit based on GStreamer for AI and computer vision processing. It can be used as a building block for creating applications and services that incorporate machine learning and computer vision components or capabilities, such as human-machine interfaces, self-driving and autonomous vehicles, security, and many other use-cases. Detailed information and documentation for DeepStream is located here: [https://developer.nvidia.com/deepstream-sdk](https://developer.nvidia.com/deepstream-sdk). -> Be sure to check out the official blog post over on the Nvidia Developer blog, as well: [https://developer.nvidia.com/blog/fast-track-computer-vision-deployments-with-nvidia-deepstream-and-edge-impulse/](https://developer.nvidia.com/blog/fast-track-computer-vision-deployments-with-nvidia-deepstream-and-edge-impulse/) +> Be sure to check out the official blog post over on the Nvidia Developer blog, as well: [https://developer.nvidia.com/blog/fast-track-computer-vision-deployments-with-nvidia-deepstream-and-edge-impulse/](https://developer.nvidia.com/blog/fast-track-computer-vision-deployments-with-nvidia-deepstream-and-edge-impulse/) ## Workflow Overview The following general workflow can be followed when using Edge Impulse with Nvidia DeepStream to produce the final ONNX model and Inference Plugin Configuration files for use with your preferred deployment option. -![](.gitbook/assets/nvidia-deepstream-community-guide/workflow.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/workflow.jpg) -The first step in the process is exactly the same as when you are using Edge Impulse to build TinyML models for embedded systems, up until the last stage of testing and validation. +The first step in the process is exactly the same as when you are using Edge Impulse to build TinyML models for embedded systems, up until the last stage of testing and validation. The process above varies depending on the type of model being used. One difference between using DeepStream vs the lower level TensorRT API is that DeepStream provides convenience for developers, with a higher level interface taking away the need to work directly with the more complex TensorRT. This convenience does come with a constraint on the model's input layer being used with DeepStream. @@ -31,16 +32,17 @@ With TensorRT you can effectively work more freely using the **Layers API**, whi At runtime, DeepStream's Inference plugin, **`Gst-nvinfer`**, automatically transforms input data to match the model's input layer, effectively performing preprocessing that are operations similar to the DSP block when using Edge Impulse's SDK on embedded systems. DeepStream reduces the development burden by implementing design choices and this convenience requires models to have a consistent input shape. Gst-nvinfer requires the input tensor to be in the NCHW format where: - - N: batch size – number of frames to pass to the inference Engine for batch inference - - C: channels – number of input channels - - H: height – input height dimension - - W: width – input width - + +* N: batch size – number of frames to pass to the inference Engine for batch inference +* C: channels – number of input channels +* H: height – input height dimension +* W: width – input width + Edge Impulses models are not in this format by default, and some steps are required to prepare the model for use with Gst-nvinfer. Gst-nvinfer makes use of TensorRT under the hood to do all the heavy lifting, with models needing to be converted to TensorRT engines. The simplest way to build a DeepStream TensorRT engine is to allow Gst-nvinfer to automatically build the engine which DeepStream does, if the model is in ONNX format. Preparing the model for DeepStream therefore requires the conversion of the model to ONNX format while also converting the input tensor to NCHW. -> Note: There is a TensorRT deployment option in Edge Impulse, however these models don't work directly with DeepStream because the input layer is not in NCHW format. The TensorRT deployment is better suited for when a developer is manually building applications from the ground up, directly on top of TensorRT where there is complete control of the Inference Engine and application. This requires more coding than using DeepStream. It is also used with Edge Impulse EIM Linux deployments. +> Note: There is a TensorRT deployment option in Edge Impulse, however these models don't work directly with DeepStream because the input layer is not in NCHW format. The TensorRT deployment is better suited for when a developer is manually building applications from the ground up, directly on top of TensorRT where there is complete control of the Inference Engine and application. This requires more coding than using DeepStream. It is also used with Edge Impulse EIM Linux deployments. > Note: Due to TensorRT being used at heart of Gst-nvinfer, it is possible to apply all the TensorRT development capabilities to override DeepStream if it is necessary. For example, manual Engine creation in C++ or Python as well as custom input and output layers through TensorRT's C++ plugin architecture could be reason's to override DeepStream. @@ -49,8 +51,7 @@ Since the goal of DeepStream is to make the development process easier and more The two primary no-code TensorRT engine creation approaches are: 1. Automatic Engine Creation: Gst-nvInfer builds the TensorRT Engine automatically, and produces a serialized Engine file for Inference. - -2. Manual Engine Creation: Using the `trtexec` command from the command line to manually produce serialized Engine files. +2. Manual Engine Creation: Using the `trtexec` command from the command line to manually produce serialized Engine files. Option 2 provides more flexibility but adds in an additional manual step. For the purpose of this guide, we going with the simplest approach using automatic Engine creation with a no-code deployment. If you wish to manually create an Engine, refer to the documentation for the `trtexec` command at [https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#trtexec](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#trtexec). @@ -60,16 +61,17 @@ Depending on the type of model you are building (Image Classification or Audio C ## Object Detection — Edge Impulse -Object detection is a category of computer vision that attempts to identify a specific item (or items) within an image or frame of video, and assigns a confidence value to the item. Items are generally shown with a bounding box around them, to indicate where in the image the object of interest is located. +Object detection is a category of computer vision that attempts to identify a specific item (or items) within an image or frame of video, and assigns a confidence value to the item. Items are generally shown with a bounding box around them, to indicate where in the image the object of interest is located. Edge Impulse currently supports 3 object detection model architectures: - - **MobileNetV2 SSD FPN-Lite 320 x 320**: From the TensorFlow Object Detection (TFOD) family of models - - **Faster Object More Objects (FOMO)**: Edge Impulse's custom object detection model designed run on constrained devices - - **YOLOv5**: Standard YOLOv5 from Ultralytics + +* **MobileNetV2 SSD FPN-Lite 320 x 320**: From the TensorFlow Object Detection (TFOD) family of models +* **Faster Object More Objects (FOMO)**: Edge Impulse's custom object detection model designed run on constrained devices +* **YOLOv5**: Standard YOLOv5 from Ultralytics These are available natively in the Studio, or if those don't meet your needs then it is possible for you to [bring your own models](https://docs.edgeimpulse.com/docs/edge-impulse-studio/bring-your-own-model-byom) using custom learning blocks, though that will require additional effort. -The TensorFlow Object Detection (TFOD) model contains some operations that are not natively supported by TensorRT. While it is possible to make TFOD work using some workarounds, the TFOD models are not easy to use out-of-the-box with DeepStream. +The TensorFlow Object Detection (TFOD) model contains some operations that are not natively supported by TensorRT. While it is possible to make TFOD work using some workarounds, the TFOD models are not easy to use out-of-the-box with DeepStream. FOMO, which is designed to run directly on MCU targets, has an output layer not directly compatible with what DeepStream expects, and requires the additional step of implementing and managing a custom TensorRT output layer. @@ -81,7 +83,7 @@ Image models, including object detection, are machine learning models focused on Image models built with Edge Impulse use raw pixels as input features. The input image is scaled down to reduce the model input layer size, in order to maintain processing throughput on lower-powered hardware. With DeepStream you are only limited by the power of the chosen Nvidia platform to run the model on. The resolution and input layer size can be made larger, and experimentation for each platform is useful to determine the best choice. -In addition to resolution, the model can be trained on RGB colour or Grayscale. Edge Impulse takes care of removing the alpha channel and allows you to select the input resolution and colour depth. Grayscale is ideal for tinyML applications due to the limited performance of most hardware, but on Nvidia hardware, color images can be utilized. +In addition to resolution, the model can be trained on RGB colour or Grayscale. Edge Impulse takes care of removing the alpha channel and allows you to select the input resolution and colour depth. Grayscale is ideal for tinyML applications due to the limited performance of most hardware, but on Nvidia hardware, color images can be utilized. The input features for full color images are generated by Edge Impulse in RGB format (DeepStream supports RGB or BGR). At runtime, DeepStream's Inference plugin automatically transforms input frames to match the model input resolution and colour depth, which eliminates the need to write custom preprocessing code to do this step manually. @@ -93,7 +95,7 @@ For object detection projects ultimately destined for use in DeepStream, the pro Edge Impulse provides a few versions of YOLOv5 that have been optimized for use with different accelerators. For Nvidia, you can select the standard YOLOv5 Community model. -![](.gitbook/assets/nvidia-deepstream-community-guide/choose-yolo5.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/choose-yolo5.jpg) Training of the model is then done in the same way as other object detection projects in Edge Impulse. Documentation is located here: [https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/object-detection](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/object-detection) @@ -101,7 +103,7 @@ Training of the model is then done in the same way as other object detection pro Once training is complete, Edge Impulse allows you to export the YOLOv5 model directly into ONNX format, which also has the correct NCHW input layer. This saves the conversion to ONNX step, as its already done for the YOLOv5 model. -![](.gitbook/assets/nvidia-deepstream-community-guide/export-onnx.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/export-onnx.jpg) ### 3. Deepstream Gst-Nvinfer Integration – YOLOv5 Model @@ -109,11 +111,11 @@ The Gst-nvinfer plugin outputs data in a standardized format that can be consume The output tensor format of YOLO varies depending on the specific implementation and model version, but I'll describe a common format used in YOLOv3 as an example. -In YOLOv3, the output tensor is a 3D array with dimensions **[batch_size, grid_size, grid_size * num_anchors, num_classes + 5]**. Since this is not compatible with DeepStream NvDsObectMeta structure, a custom output parser needs to be implemented. Fortunately Nvidia provides an SDK that allows you to create custom output parsers to support any kind of output format. Nvidia also provides an implementation for YOLOv2 and YOLOv3 and its variants, however Edge Impulse uses the later YOLOv5. +In YOLOv3, the output tensor is a 3D array with dimensions **\[batch\_size, grid\_size, grid\_size \* num\_anchors, num\_classes + 5]**. Since this is not compatible with DeepStream NvDsObectMeta structure, a custom output parser needs to be implemented. Fortunately Nvidia provides an SDK that allows you to create custom output parsers to support any kind of output format. Nvidia also provides an implementation for YOLOv2 and YOLOv3 and its variants, however Edge Impulse uses the later YOLOv5. The included repo contains an implementation of a custom output parser that works with Edge Impulse's YOLO, which is based on [https://github.com/marcoslucianops/DeepStream-Yolo](https://github.com/marcoslucianops/DeepStream-Yolo). -In order to use the custom output parser it needs to be built, which can be done from the command line on your Nvidia appliance. However, CUDA versions vary by Jetpack version, and this needs to be specified using the `CUDA_VER` environment variable. The output is a .so file that then needs to be added to the Gst-nvinfer plugin configuration file using the _custom-lib-path_ parameter. The custom bounding box parsing function also needs to be specified with the _parse-bbox-func-name_ parameter. In this case the repo provides this function, called _NvDsInferParseYolo_. The next section will cover the process of configuring the Gst-nvinfer plugin. +In order to use the custom output parser it needs to be built, which can be done from the command line on your Nvidia appliance. However, CUDA versions vary by Jetpack version, and this needs to be specified using the `CUDA_VER` environment variable. The output is a .so file that then needs to be added to the Gst-nvinfer plugin configuration file using the _custom-lib-path_ parameter. The custom bounding box parsing function also needs to be specified with the _parse-bbox-func-name_ parameter. In this case the repo provides this function, called _NvDsInferParseYolo_. The next section will cover the process of configuring the Gst-nvinfer plugin. The Gst-Nvinfer plugin is configured by means of a plain text file which specifies all the relevant parameters required to run a model with Deepstream and TensorRT behind the scenes. This file needs to be referenced from the Deepstream Application either as a Primary GPU Inference Engine (PGIE) where it is the first Inference to take place in the pipeline or as Secondary GPU Inference Engine (SGIE) where it’s the performing secondary inference on the output of a PGIE upstream. @@ -121,7 +123,7 @@ Object Detection (in this case the YOLOv5 model built in Edge Impulse) is usuall The minimal working version of a PGIE using the Edge Impulse YOLOv5 ONNX export is shown below: -![](.gitbook/assets/nvidia-deepstream-community-guide/minimal-pgie.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/minimal-pgie.jpg) All paths are relative to the configuration file which can be created in a text editor and placed where it can be referenced by your application, which could either be a custom Python or C++ application or a DeepStream Reference app and sample apps. @@ -135,31 +137,31 @@ Image classification is a more generalized use of computer vision, instead of tr ### 1. Build and Train Model -To create an Image Classification model, Select the Transfer Learning classification model, which is already pretrained on a larger dataset. This speeds up the training process, allowing you to quickly train your own model that will rapidly converge on your dataset. +To create an Image Classification model, Select the Transfer Learning classification model, which is already pretrained on a larger dataset. This speeds up the training process, allowing you to quickly train your own model that will rapidly converge on your dataset. -![](.gitbook/assets/nvidia-deepstream-community-guide/transfer-learning.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/transfer-learning.jpg) ### 2. Export Image Classification Model -Once the model has been trained and tested, it is ready for deployment and can be exported for use with DeepStream. +Once the model has been trained and tested, it is ready for deployment and can be exported for use with DeepStream. DeepStream supports ONNX model conversion, but the ONNX export options available such as the TensorRT Deployment are not suitable for DeepStream's specific requirements. Thus, a conversion process needs to be followed starting with the TFLite export option that is available in Edge Impulse. A TFLite **float32** model is required, and this can be obtained by exporting from the Download Output Block on your project's Dashboard: -![](.gitbook/assets/nvidia-deepstream-community-guide/export-tflite.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/export-tflite.jpg) -This is the simplest way to get the model in TFLite format, but requires that you have direct access to the project in the Edge Impulse Studio. +This is the simplest way to get the model in TFLite format, but requires that you have direct access to the project in the Edge Impulse Studio. Alternatively another approach is by using the C/C++ or Arduino Library Deployment options. This requires that the EON compiler is turned off, and that **Unoptimized (float32)** is selected prior to clicking **Build**. -![](.gitbook/assets/nvidia-deepstream-community-guide/build.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/build.jpg) This generates and downloads a `.zip` file that contains your TFLite model, stored as an array in a C header file called `tflite-trained.h` (located in the `tflite-model` folder). When working with an Arduino Library, this folder is located under the `src` directory. -![](.gitbook/assets/nvidia-deepstream-community-guide/unzip.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/unzip.jpg) -The next step requires the TFlite model to be in the TFLite Flatbuffers format (.lite) and this requires converting the array data into a binary file. To convert to a normal Flatbuffer TFLite model, run the following code. +The next step requires the TFlite model to be in the TFLite Flatbuffers format (.lite) and this requires converting the array data into a binary file. To convert to a normal Flatbuffer TFLite model, run the following code. ``` carray2bin.py @@ -167,17 +169,17 @@ carray2bin.py _carray2bin.py_ is a python script that allows you to convert binary data payloads of c arrays to a bin file, and is provided in the /utils folder of the provided repo. -Note that this method is useful if you only have access to the C/C++ or Arduino Library export, if the export was built as **Unoptimized (float32)** as described above. +Note that this method is useful if you only have access to the C/C++ or Arduino Library export, if the export was built as **Unoptimized (float32)** as described above. ### 3. Convert Model to ONNX The float32 model generated by Edge Impulse has an input tensor named `x_input` with the following layout with each dimension: -[N,H,W,C] +\[N,H,W,C] -For example, with a 160x160 pixel input image, the input tensor is float32[1,160,160,1] +For example, with a 160x160 pixel input image, the input tensor is float32\[1,160,160,1] -This requires a transpose to be added to the input tensor to accept input images as NCHW. +This requires a transpose to be added to the input tensor to accept input images as NCHW. To convert the model from TensorFlow Lite to ONNX with the correct input shape for DeepStream requires the use of **"tf2onnx"**: @@ -189,7 +191,7 @@ Its important to note the `--inputs-as-nchw serving_default_x:0` parameter that The result of the conversion should yield results similar to the following: -![](.gitbook/assets/nvidia-deepstream-community-guide/conversion.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/conversion.jpg) ### 4. DeepStream Gst-nvinfer Integration @@ -197,7 +199,7 @@ The second stage classifier instance of Gst-nvinfer (SGIE) typically receives th To run an Edge Impulse model from the ONNX file produced in the prior steps, the following configuration is required at a minimum for a SGIE. -![](.gitbook/assets/nvidia-deepstream-community-guide/minimal-sgie.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/minimal-sgie.jpg) To use as a SGIE to classify the output of a PGIE Object Detector set the `process-mode` to `2`: @@ -207,21 +209,21 @@ process-mode=2 #SGIE The label file needs to contain the list of text labels separated by semicolons. The labels should be in the same order as shown in the Edge Impulse Studio when viewing the Impulse configuration by looking in the output features. -![](.gitbook/assets/nvidia-deepstream-community-guide/impulse.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/impulse.jpg) Alternatively, the labels can be found in the Edge Impulse C++ SDK in the `ei_classifier_inferencing_categories` array in the `model_variables.h` header file. -![](.gitbook/assets/nvidia-deepstream-community-guide/labels.jpg) +![](../../.gitbook/assets/nvidia-deepstream-community-guide/labels.jpg) This approach requires the ONNX model be in an accessible path on your system, and automatically builds the TensorRT as an **Engine** file after running. This is the simplest approach, only requiring the `.onnx` file. -After the first run, the TensorRT is created as an `.engine` file. To prevent rebuilding on subsequent runs, the ONNX file can be commented out and the .engine file can instead be directly referenced. This will prevent rebuilding the engine on each run, saving time. +After the first run, the TensorRT is created as an `.engine` file. To prevent rebuilding on subsequent runs, the ONNX file can be commented out and the .engine file can instead be directly referenced. This will prevent rebuilding the engine on each run, saving time. The major limitation of automatic conversion of the model is that it works with implicit batching, where the batch size is 1, ignoring the batch dimension in the model. This may not be ideal when you need to perform inference on batches of images, to take advantage of the hardware batch inference capabilities. ## Dynamic Batching -The underlying TensorRT runtime does support Dynamic Batching though, which means that the batch size for inference can be determined at runtime and isn't fixed in the model. +The underlying TensorRT runtime does support Dynamic Batching though, which means that the batch size for inference can be determined at runtime and isn't fixed in the model. In order to make use of Dynamic Batch sizes the model will have to be manually converted from ONNX to a TensorRT runtime using the Explicit Batch. The `trtexec` command is part of TensorRT, and allows TensorRT Engines to be manually constructed. @@ -233,9 +235,7 @@ sudo ./trtexec --explicitBatch --onnx=edgeimpulse_classify.onnx --workspace=4000 Here the following parameters are used at a minimum to convert your Edge Impulse Model: - - **--explicitBatch**: used to specify that the model must be built for Dynamic Batching support with an explicit batch dimension in the input tensor. Required to use batch sizes larger than 1 with an Edge Impulse model. - - **--onnx=**: specify the input ONNX file - - **--workspace=4000**: specify the amount of memory allocated to the build process in MB, in this case 4GB. This gives `trtexec` enough working temporary memory to create the model engine. - - **--saveEngine=**: the output TensorRT runtime engine. - - +* **--explicitBatch**: used to specify that the model must be built for Dynamic Batching support with an explicit batch dimension in the input tensor. Required to use batch sizes larger than 1 with an Edge Impulse model. +* **--onnx=**: specify the input ONNX file +* **--workspace=4000**: specify the amount of memory allocated to the build process in MB, in this case 4GB. This gives `trtexec` enough working temporary memory to create the model engine. +* **--saveEngine=**: the output TensorRT runtime engine. diff --git a/nvidia-omniverse-synthetic-data.md b/readme/featured-machine-learning-projects/nvidia-omniverse-synthetic-data.md similarity index 81% rename from nvidia-omniverse-synthetic-data.md rename to readme/featured-machine-learning-projects/nvidia-omniverse-synthetic-data.md index ca39ad05..55190869 100644 --- a/nvidia-omniverse-synthetic-data.md +++ b/readme/featured-machine-learning-projects/nvidia-omniverse-synthetic-data.md @@ -1,18 +1,18 @@ --- -description: Create synthetic data to rapidly build object detection datasets with Nvidia Omniverse's Replicator API and Edge Impulse. +description: >- + Create synthetic data to rapidly build object detection datasets with Nvidia + Omniverse's Replicator API and Edge Impulse. --- # NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects -Created By: -[Adam Milton-Barker](https://www.AdamMiltonBarker.com) +Created By: [Adam Milton-Barker](https://www.adammiltonbarker.com) -Public Project Link: -[https://studio.edgeimpulse.com/public/246023/latest](https://studio.edgeimpulse.com/public/246023/latest) +Public Project Link: [https://studio.edgeimpulse.com/public/246023/latest](https://studio.edgeimpulse.com/public/246023/latest) GitHub Repo: [NVIDIA Omniverse™ Synthetic Data Generation For Edge Impulse Projects](https://github.com/AdamMiltonBarker/ominverse-replicator-edge-impulse) -![](.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-edge-impulse.jpg) +![](../../.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-edge-impulse.jpg) ## Introduction @@ -20,39 +20,39 @@ In the realm of machine learning, the availability of diverse and representative ## Synthetic Data -![Synthetic Data](.gitbook/assets/nvidia-omniverse-synthetic-data/synthetic-data.jpg) +![Synthetic Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/synthetic-data.jpg) Synthetic data refers to artificially generated data that emulates the statistical properties and patterns of real-world data. It is created through sophisticated algorithms and models that simulate the characteristics of the original data while maintaining control over its properties. Domain randomization, on the other hand, is a technique used in conjunction with synthetic data generation, where various parameters and attributes of the data are intentionally randomized within specified bounds. This randomized variation helps the model become more robust and adaptable to different environments. ## Omniverse™ -![NVIDIA Omniverse™](.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse.jpg) +![NVIDIA Omniverse™](../../.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse.jpg) -NVIDIA Omniverse™ represents a groundbreaking platform that is set to revolutionize the collaborative, design, and simulation processes within industries. This cutting-edge tool combines real-time rendering, physics simulation, and advanced AI capabilities to create a highly powerful and scalable solution. +NVIDIA Omniverse™ represents a groundbreaking platform that is set to revolutionize the collaborative, design, and simulation processes within industries. This cutting-edge tool combines real-time rendering, physics simulation, and advanced AI capabilities to create a highly powerful and scalable solution. ### Omniverse™ Replicator -![NVIDIA Omniverse™](.gitbook/assets/nvidia-omniverse-synthetic-data/nvidia-omniverse-enterprise-diagram.jpg) +![NVIDIA Omniverse™](../../.gitbook/assets/nvidia-omniverse-synthetic-data/nvidia-omniverse-enterprise-diagram.jpg) [NVIDIA Omniverse™ Replicator](https://developer.nvidia.com/omniverse/replicator) is a versatile collection of APIs designed to empower researchers and enterprise developers in generating synthetic data that closely resembles real-world scenarios. With its extensibility, Omniverse™ Replicator allows users to effortlessly construct custom synthetic data generation (SDG) tools, effectively expediting the training of computer vision networks. -## Edge Impulse +## Edge Impulse -![Edge Impulse](.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse.jpg) +![Edge Impulse](../../.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse.jpg) -The Edge Impulse platform, along with its integrated Edge Impulse Studio, is a comprehensive solution tailored for developing and deploying embedded machine learning models. It empowers developers to seamlessly gather, process, and analyze sensor data from various edge devices, such as microcontrollers and sensors. With Edge Impulse Studio, users can easily create and train machine learning models using a visual interface or code-based workflow. +The Edge Impulse platform, along with its integrated Edge Impulse Studio, is a comprehensive solution tailored for developing and deploying embedded machine learning models. It empowers developers to seamlessly gather, process, and analyze sensor data from various edge devices, such as microcontrollers and sensors. With Edge Impulse Studio, users can easily create and train machine learning models using a visual interface or code-based workflow. ## Project -![On-Device Testing Results](.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse-omniverse.gif "On-Device Testing Results") +![On-Device Testing Results](../../.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse-omniverse.gif) In this project we will use the Omniverse™ Replicator API inside of Omniverse™ Code to generate our synthetic dataset of fruits (apples, oranges, and limes). Once our dataset has been created we will import the dataset into Edge Impulse Studio, create and train an object detection model, and then deploy it an NVIDIA Jetson Nano. -### Hardware +### Hardware #### RTX-Enabled GPU -![GPU Requirements](.gitbook/assets/nvidia-omniverse-synthetic-data/gpu-requirements.jpg) +![GPU Requirements](../../.gitbook/assets/nvidia-omniverse-synthetic-data/gpu-requirements.jpg) For this project an [RTX-enabled GPU](https://www.nvidia.com/en-us/geforce/rtx/) is required. I was lucky enough to be given access by NVIDIA to a Windows 10 VM equipped with an [RTX A40](https://www.nvidia.com/en-us/data-center/a40/) (A very big thank you to Liz, Sunny, and all involved). This project can be run on an RTX 3060 and up, if you do not have access to your own RTX-enabled GPU, there are some well known cloud service providers that offer NVIDIA RTX GPUs in the cloud. @@ -62,35 +62,35 @@ We will deploy our machine learning model to an [NVIDIA Jetson Nano](https://dev ### Software -- [NVIDIA Omniverse™](https://www.nvidia.com/en-us/omniverse/) -- [NVIDIA Omniverse™ Replicator](https://developer.nvidia.com/omniverse/replicator) -- [NVIDIA Omniverse™ Code](https://developer.nvidia.com/omniverse/code-app) -- [Edge Impulse For Linux](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/edge-impulse-for-linux) -- [Visual Studio Code](https://code.visualstudio.com/download) +* [NVIDIA Omniverse™](https://www.nvidia.com/en-us/omniverse/) +* [NVIDIA Omniverse™ Replicator](https://developer.nvidia.com/omniverse/replicator) +* [NVIDIA Omniverse™ Code](https://developer.nvidia.com/omniverse/code-app) +* [Edge Impulse For Linux](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/edge-impulse-for-linux) +* [Visual Studio Code](https://code.visualstudio.com/download) -### Platform +### Platform -- [Edge Impulse](https://www.edgeimpulse.com) +* [Edge Impulse](https://www.edgeimpulse.com) -## Installation +## Installation -![NVIDIA Omniverse™ Launcher](.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-launcher.jpg) +![NVIDIA Omniverse™ Launcher](../../.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-launcher.jpg) To get started with NVIDIA Omniverse™, head over to the official [Omniverse™ download site](https://www.nvidia.com/en-us/omniverse/). Once you have signed in you will be able to download the Omniverse™ launcher for Windows or Linux. Once downloaded, run the launcher and go through the settings options. ### Omniverse™ Code -![NVIDIA Omniverse™ Code](.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-code.jpg) +![NVIDIA Omniverse™ Code](../../.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-code.jpg) We are going to use [Omniverse™ Code](https://developer.nvidia.com/omniverse/code-app) to create our dataset. -![Omniverse™ Code](.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-code-extension.jpg) +![Omniverse™ Code](../../.gitbook/assets/nvidia-omniverse-synthetic-data/omniverse-code-extension.jpg) You can think of Code as an IDE for building advanced 3D design and simulation tools. Head over to the `Extensions` tab and search for `Code`, then click on **Code** and install it. #### Script Editor -![Script Editor](.gitbook/assets/nvidia-omniverse-synthetic-data/script-editor.jpg) +![Script Editor](../../.gitbook/assets/nvidia-omniverse-synthetic-data/script-editor.jpg) Within Omniverse™ Code there is a feature called `Script Editor`. This editor allows us to load Python code into the IDE and execute it. This makes it very easy for us to set up our scenes and manipulate our assets. @@ -98,9 +98,9 @@ Within Omniverse™ Code there is a feature called `Script Editor`. This editor For simplicity, in this tutorial we will use assets that are readily available in Omniverse™ Code. Within the IDE you will find a tab called `NVIDIA Assets`, opening this tab will provide you with a selection of ready to use assets. The assets are of type `USD` which stands for `Universal Scene Description`. -## Project Code +## Project Code -For this tutorial, code has been provided that will work out of the box in the script editor, all you will have to do is modify the `basepath` variable and alternate the different datasets. +For this tutorial, code has been provided that will work out of the box in the script editor, all you will have to do is modify the `basepath` variable and alternate the different datasets. ### Clone The Repository @@ -133,10 +133,10 @@ FRUIT = { You should set the `basepath` variable to the path to the project root on your machine. If you are using Linux you will need to modify any path in the code as the paths have backslashes for directory separators. For the `dataset` variable you can use the following to generate your dataset: -- **All** Will generate a dataset that includes images of all the fruit types on the table. -- **Apple** Will generate a dataset that includes images of apples on the table. -- **Orange** Will generate a dataset that includes images of oranges on the table. -- **Lime** Will generate a dataset that includes images of limes on the table. +* **All** Will generate a dataset that includes images of all the fruit types on the table. +* **Apple** Will generate a dataset that includes images of apples on the table. +* **Orange** Will generate a dataset that includes images of oranges on the table. +* **Lime** Will generate a dataset that includes images of limes on the table. Together, these images will make up our entire dataset. @@ -163,7 +163,7 @@ rep.randomizer.register(table) For more information about using physics with Replicator, you can check out the [NVIDIA documentation](https://docs.omniverse.nvidia.com/prod_extensions/prod_extensions/ext_replicator/physics_example.html). -### Lighting +### Lighting Next, the code will take care of the lighting. @@ -197,7 +197,7 @@ rep.randomizer.register(dome_lights) For more information about using lights with Replicator, you can check out the [NVIDIA documentation](https://docs.omniverse.nvidia.com/materials-and-rendering/latest/lighting.html). -### Fruits +### Fruits The next function will take care of the fruits. Here you will notice we use a uniform distribution for the `position`, `rotation` and `scale`. This means that each number in the ranges has an equal chance of being chosen. Here we also define a class for the data. @@ -223,7 +223,7 @@ rep.randomizer.register(randomize_asset) For more information about using distributions with Replicator, you can check out the [NVIDIA documentation](https://docs.omniverse.nvidia.com/app_code/prod_extensions/ext_replicator/distribution_examples.html). -### Camera +### Camera Next we set up the camera and set the value for `focus distance`, `focal length`, `position`, `rotation`, and `f-stop`. @@ -286,15 +286,15 @@ rep.orchestrator.run() ## Creating Our Dataset -![Generate Data](.gitbook/assets/nvidia-omniverse-synthetic-data/generate-data.jpg) +![Generate Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/generate-data.jpg) Now we have explored the code and updated our settings, it is time to run the code and generate our dataset. Ensuring Omniverse™ Code is opened, copy the contents of `omniverse.py` and paste it into the script editor. Once you have done this press the `Run` button, or `ctrl + enter`. -Remember to change the `dataset` variable to the relevant class and run the script for each of the 3 classes. +Remember to change the `dataset` variable to the relevant class and run the script for each of the 3 classes. -![Generated Data](.gitbook/assets/nvidia-omniverse-synthetic-data/generated-data.jpg) +![Generated Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/generated-data.jpg) -Head over to the `data/rendered` directory and you will find all of your generated data. Navigate through the various folders to view the created datasets. +Head over to the `data/rendered` directory and you will find all of your generated data. Navigate through the various folders to view the created datasets. ## Visualize Our Dataset @@ -319,7 +319,7 @@ vis_out_dir = "C:\\Users\\adam\\Desktop\\Omniverse\\Src\\data\\visualize" file_number = "0000" ``` -The writer will save images with an incrementing number in the file name, such as `rgb_0000.png`, `rgb_0001.png` etc. To visualize your data simply increment the `file_number` variable. +The writer will save images with an incrementing number in the file name, such as `rgb_0000.png`, `rgb_0001.png` etc. To visualize your data simply increment the `file_number` variable. You can now run the following code, ensuring you are in the project root directory. @@ -329,24 +329,24 @@ python visualize.py You should see similar to the following: -![Generated Data](.gitbook/assets/nvidia-omniverse-synthetic-data/bbox2d_0000_tight.png) +![Generated Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/bbox2d_0000_tight.png) ## Creating Our Model -Now it is time to head over to [Edge Impulse](https://www.edgeimpulse.com) and create our machine learning pipeline. +Now it is time to head over to [Edge Impulse](https://www.edgeimpulse.com) and create our machine learning pipeline. -![Create EI Project](.gitbook/assets/nvidia-omniverse-synthetic-data/create-ei-project.jpg) +![Create EI Project](../../.gitbook/assets/nvidia-omniverse-synthetic-data/create-ei-project.jpg) Log in or create an account on Edge Impulse and then create a new project. Once created scroll down on the project home page to the `Project Info` area and make sure to change `Labeling method` to `Bounding Boxes (Object Detection)` and `Target Device` to `Jetson Nano`. Now scroll down to the `Performance Settings` and ensure that `Use GPU for training` and `Enterprise performance` are selected if you have those options. ### Connect Your Device -![Connect device](.gitbook/assets/nvidia-omniverse-synthetic-data/connect-device.jpg) +![Connect device](../../.gitbook/assets/nvidia-omniverse-synthetic-data/connect-device.jpg) You need to install the required dependencies that will allow you to connect your device to the Edge Impulse platform. This process is documented on the [Edge Impulse Website](https://docs.edgeimpulse.com/docs/development-platforms/officially-supported-cpu-gpu-targets/nvidia-jetson-nano) and includes: -- Running the Edge Impulse NVIDIA Jetson Nano setup script -- Connecting your device to the Edge Impulse platform +* Running the Edge Impulse NVIDIA Jetson Nano setup script +* Connecting your device to the Edge Impulse platform Once the firmware has been installed enter the following command: @@ -362,13 +362,13 @@ edge-impulse-linux --clean Follow the instructions to log in to your Edge Impulse account. -![Device connected to Edge Impulse](.gitbook/assets/nvidia-omniverse-synthetic-data/device-connected.jpg) +![Device connected to Edge Impulse](../../.gitbook/assets/nvidia-omniverse-synthetic-data/device-connected.jpg) Once complete head over to the devices tab of your project and you should see the connected device. ### Upload Data -![Upload Data](.gitbook/assets/nvidia-omniverse-synthetic-data/upload-data.jpg) +![Upload Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/upload-data.jpg) Unfortunately Omniverse does not generate bounding boxes in the format that Edge Impulse requires, so for this project we will upload the data and then label it in Edge Impulse Studio. @@ -376,53 +376,53 @@ We will start with the `Apple` class. Head over to the `Data Aquisition` page, s ### Labelling Data -![Labelling Data](.gitbook/assets/nvidia-omniverse-synthetic-data/labelling-data.jpg) +![Labelling Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/labelling-data.jpg) Next head over to the `Labelling Queue` page. Here you can draw boxes around your data and add labels to each fruit in each image, then repeat these steps for each of the classes. -Note that the EI platform will attempt to track objects across frames, in some cases it makes duplicates or adds incorrect bounding boxes, ensure that you delete/modify these incorrect bounding boxes to avoid problems further down the line. +Note that the EI platform will attempt to track objects across frames, in some cases it makes duplicates or adds incorrect bounding boxes, ensure that you delete/modify these incorrect bounding boxes to avoid problems further down the line. Once you have completed the `apples` data, repeat the steps for the `oranges` and `limes` images. -![Completed Data](.gitbook/assets/nvidia-omniverse-synthetic-data/completed-dataset.jpg) +![Completed Data](../../.gitbook/assets/nvidia-omniverse-synthetic-data/completed-dataset.jpg) -Once you have finished labelling the data you should have 150 images that each have around 15 pieces of fruit labelled, and a data split of 80/20. +Once you have finished labelling the data you should have 150 images that each have around 15 pieces of fruit labelled, and a data split of 80/20. -### Create Impulse +### Create Impulse -![Create Impulse](.gitbook/assets/nvidia-omniverse-synthetic-data/create-impulse.jpg) +![Create Impulse](../../.gitbook/assets/nvidia-omniverse-synthetic-data/create-impulse.jpg) Now it is time to create our Impulse. Head over to the `Impulse Design` tab and click on the `Create Impulse` tab. Here you should set the `Image Width` and `Image Height` to `512`. Next add an `Image` block in the `Processing Blocks` section, then select `Yolov5` in the `Learning Blocks` section, and finally click `Save Impulse`. -### Parameters & Features +### Parameters & Features -![Parameters & Features](.gitbook/assets/nvidia-omniverse-synthetic-data/generate-features.jpg) +![Parameters & Features](../../.gitbook/assets/nvidia-omniverse-synthetic-data/generate-features.jpg) Next click on the `Images` tab and click on `Save Parameters`, you will redirected to the features page. Once on the features page click `Generate Features`. You should see that your features are nicely grouped, this is what we are looking for to achieve satisfactory results. -### Training +### Training -![Training](.gitbook/assets/nvidia-omniverse-synthetic-data/training.jpg) +![Training](../../.gitbook/assets/nvidia-omniverse-synthetic-data/training.jpg) Now it is time to train our model, head over to the `Yolov5` tab, leave all the settings as they are aside from training cycles which I set to 750, then click `Start Training`. This while take a while so grab a coffee. -![Training Results](.gitbook/assets/nvidia-omniverse-synthetic-data/training-results.jpg) +![Training Results](../../.gitbook/assets/nvidia-omniverse-synthetic-data/training-results.jpg) Once training is finished we see we achieved an exceptional F1 Score of 97.2%. -### Testing +### Testing -Now it is time to test our model. There are a few ways we can test through Edge Impulse Studio before carrying out the ultimate test, on-device testing. +Now it is time to test our model. There are a few ways we can test through Edge Impulse Studio before carrying out the ultimate test, on-device testing. #### Platform Testing -![Platform Testing](.gitbook/assets/nvidia-omniverse-synthetic-data/testing.jpg) +![Platform Testing](../../.gitbook/assets/nvidia-omniverse-synthetic-data/testing.jpg) Platform testing went very well, and our model achieved 99.24% on the Test (unseen) data. #### Platform Live Testing -![Platform Testing](.gitbook/assets/nvidia-omniverse-synthetic-data/live-testing.jpg) +![Platform Testing](../../.gitbook/assets/nvidia-omniverse-synthetic-data/live-testing.jpg) To carry out live testing through the Edge Impulse Studio, connect to your Jetson Nano and enter the following command: @@ -436,7 +436,7 @@ In my case live testing through Edge Impulse Studio also did very well, classify #### On-Device Testing -![On-Device Testing Results](.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse-omniverse.gif) +![On-Device Testing Results](../../.gitbook/assets/nvidia-omniverse-synthetic-data/edge-impulse-omniverse.gif) The final test is the on-device testing. For this we need to download the model and build it on our Jetson Nano. Luckily, Edge Impulse makes this a very easy task. If you are still connected to the platform disconnect, and then enter the following command: @@ -444,9 +444,9 @@ The final test is the on-device testing. For this we need to download the model edge-impulse-linux-runner ``` -This will download the model, build and then start classifying, ready for you to introduce some fruit. +This will download the model, build and then start classifying, ready for you to introduce some fruit. -In my case the model performed extremely well, easily classifying each fruit correctly. +In my case the model performed extremely well, easily classifying each fruit correctly. ## Conclusion @@ -456,4 +456,4 @@ The outcomes clearly demonstrate the effectiveness of NVIDIA's Replicator as a r By combining Replicator with Edge Impulse Studio, we have harnessed a cutting-edge solution that empowers us to rapidly and efficiently build reliable object detection solutions. This powerful combination holds immense potential for addressing various challenges across different domains. -Once again, a big thank you to NVIDIA for their support in this project. It has been an amazing experience learning about how to use Omniverse in an Edge Impulse pipeline, keep an eye out for future projects. +Once again, a big thank you to NVIDIA for their support in this project. It has been an amazing experience learning about how to use Omniverse in an Edge Impulse pipeline, keep an eye out for future projects. diff --git a/image-projects/quality-control-jetson-nano.md b/readme/featured-machine-learning-projects/quality-control-jetson-nano.md similarity index 75% rename from image-projects/quality-control-jetson-nano.md rename to readme/featured-machine-learning-projects/quality-control-jetson-nano.md index 3e7e6d29..3cfb4acd 100644 --- a/image-projects/quality-control-jetson-nano.md +++ b/readme/featured-machine-learning-projects/quality-control-jetson-nano.md @@ -1,6 +1,7 @@ --- description: >- - An Nvidia Jetson Nano computer vision project with TensorRT acceleration, to perform counting for manufacturing quality control. + An Nvidia Jetson Nano computer vision project with TensorRT acceleration, to + perform counting for manufacturing quality control. --- # Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT) @@ -9,13 +10,13 @@ Created By: Jallson Suryo Public Project Link: [https://studio.edgeimpulse.com/public/320746/latest](https://studio.edgeimpulse.com/public/320746/latest) -GitHub Repo: [https://github.com/Jallson/PizzaQC_Conveyor_Belt](https://github.com/Jallson/PizzaQC_Conveyor_Belt) +GitHub Repo: [https://github.com/Jallson/PizzaQC\_Conveyor\_Belt](https://github.com/Jallson/PizzaQC_Conveyor_Belt) -![](../.gitbook/assets/quality-control-jetson-nano/Photo02.jpg) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo02.jpg) ## Problem Statement -The Quality Control process, especially those involving visual-based calculations and carried out repeatedly, is time consuming and prone to errors when performed by humans. Using sensors to count or calculate presence also does not provide a solution if the object you want to detect is an object made up of multiple components, each needing measurement. Food products, finished goods, or electronic manufacturing processes could be examples of this type of scenario. +The Quality Control process, especially those involving visual-based calculations and carried out repeatedly, is time consuming and prone to errors when performed by humans. Using sensors to count or calculate presence also does not provide a solution if the object you want to detect is an object made up of multiple components, each needing measurement. Food products, finished goods, or electronic manufacturing processes could be examples of this type of scenario. ## Our Solution @@ -25,24 +26,24 @@ A computer vision system for quality/quantity inspection of product manufacturin This project uses Edge Impulse's FOMO (Faster Objects, More Objects) algorithm, which can quickly detect objects and use them as a quality/quantity check for products on a running conveyor belt. FOMO's ability to know the number and position of coordinates of an object is the basis of this system. This project will explore the capability of the Nvidia Jetson Nano's GPU to handle color video (RGB) with a higher resolution (320x320) than some other TinyML projects, while still maintaining a high inference speed. The machine learning model (`model.eim`) will be deployed with the TensorRT library, which will be compiled with optimizations for the GPU and will be setup via the Linux C++ SDK. Once the model can identify different pizza toppings, an additional Python program will be added, to check each pizza for a standard quantity of pepperoni, mushrooms, and paprikas. This project is a proof-of-concept that can be widely applied in the product manufacturing and food production industries to perform quality checks based on a quantity requirement of part in a product. -![](../.gitbook/assets/quality-control-jetson-nano/Photo01.jpg) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo01.jpg) ### Hardware Components -- Nvidia Jetson Nano with dedicated power adapter -- USB camera/webcam (eg. Logitech C270) -- Mini conveyer belt system (10cm x 50cm or larger) -- Camera stand/holder -- Objects: For this example, mini pizza with toppings (dough or printed paper) -- Ethernet cable -- PC/Laptop to access Jetson Nano via SSH +* Nvidia Jetson Nano with dedicated power adapter +* USB camera/webcam (eg. Logitech C270) +* Mini conveyer belt system (10cm x 50cm or larger) +* Camera stand/holder +* Objects: For this example, mini pizza with toppings (dough or printed paper) +* Ethernet cable +* PC/Laptop to access Jetson Nano via SSH ### Software & Online Services -- Edge Impulse Studio -- Edge Impulse Linux, Python, C++ SDK -- Ubuntu OS/Nvidia Jetpack -- Terminal +* Edge Impulse Studio +* Edge Impulse Linux, Python, C++ SDK +* Ubuntu OS/Nvidia Jetpack +* Terminal ## Steps @@ -50,30 +51,29 @@ This project uses Edge Impulse's FOMO (Faster Objects, More Objects) algorithm, In this project we can use a camera (webcam) connected to a PC/laptop to capture the images for data collection for ease of use. Take pictures of your pizza components from above, with slightly different angles and lighting conditions to ensure that the model can work under different conditions (to prevent overfitting). While using FOMO, object size is a crucial aspect to ensure the performance of this model. You must keep the camera distance from objects consistent, because significant differences in object size will confuse the FOMO algorithm. -![](../.gitbook/assets/quality-control-jetson-nano/Photo03.jpg) - +![](../../.gitbook/assets/quality-control-jetson-nano/Photo03.jpg) ### 2. Data Acquisition and Labeling Go to [http://studio.edgeimpulse.com](http://studio.edgeimpulse.com), login or create an account, then create a new project. -Choose the **Images** project option, then **Classify Multiple Objects**. In **Dashboard > Project Info**, choose **Bounding Boxes** for the labeling method and **Nvidia Jetson Nano** for the target device. Then in **Data acquisition**, click on the **Upload Data** tab, choose your photo files that you captured from your webcam or phone, choose **Auto split**, then click **Begin upload**. +Choose the **Images** project option, then **Classify Multiple Objects**. In **Dashboard > Project Info**, choose **Bounding Boxes** for the labeling method and **Nvidia Jetson Nano** for the target device. Then in **Data acquisition**, click on the **Upload Data** tab, choose your photo files that you captured from your webcam or phone, choose **Auto split**, then click **Begin upload**. -For Developer (free) accounts, next click on the **Labelling queue** tab, then start dragging a box around each object and label it, then save it. Repeat until all images are labelled. More information on this process, is located here: [https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/labeling-queue](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/labeling-queue) +For Developer (free) accounts, next click on the **Labelling queue** tab, then start dragging a box around each object and label it, then save it. Repeat until all images are labelled. More information on this process, is located here: [https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/labeling-queue](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/labeling-queue) For Enterprise (paid) accounts, you will instead click on **Auto-Labeler** in _Data Acquisition_. This auto-labeling segmentation / cluster process will save a lot of time over the manual process above. Set the min/max object pixels and sim threshold (0.9 - 0.999) to adjust the sensitivity of cluster detection, then click **Run**. Next, you can label each cluster result as desired. If something doesn’t match or if there is additional data, labeling can be done manually as well. -![](../.gitbook/assets/quality-control-jetson-nano/Photo04.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo04.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo05.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo05.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo06.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo06.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo07.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo07.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo08.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo08.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo09.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo09.png) ### 3. Train and Build Model using FOMO Object Detection @@ -81,25 +81,25 @@ Once you have the dataset ready, go to **Create Impulse** and in the Image block In the _Image_ parameters section, set the color depth as **RGB** then press **Save parameters**. Then click on **Generate** and navigate to the **Object Detection** block setup using the left navigation. Leave the training setting for Neural Network as it is or check our settings — in our case everything is quite balanced, so we'll leave them alone and choose **FOMO (MobileNet V2 0.35)**. Train the model by pressing the **Start training** button. You can see the progress of the training in the log to the right. -If everything is OK, the training will job will finish in a short while, then we can test the model. Go to the **Model Testing** section and click **Classify all**. Our result is above 90%, so we can move on to the next step — Deployment. +If everything is OK, the training will job will finish in a short while, then we can test the model. Go to the **Model Testing** section and click **Classify all**. Our result is above 90%, so we can move on to the next step — Deployment. -![](../.gitbook/assets/quality-control-jetson-nano/Photo10.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo10.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo11.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo11.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo12.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo12.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo13.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo13.png) -![](../.gitbook/assets/quality-control-jetson-nano/Photo14.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo14.png) ### 4. Deploy Model to NVIDIA Jetson Nano GPU Click on the **Deployment** tab then search for _TensorRT_, then select _Float32_ and click **Build**. This will build an Nvidia TensorRT library for running inference, targeting the Jetson Nano's GPU. Once the build finishes and the file is downloaded, open the `.zip` file, then we're ready for model deployment with the Edge Impulse C++ SDK on the Jetson Nano side. -![](../.gitbook/assets/quality-control-jetson-nano/Photo15.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo15.png) -On the Jetson Nano, there are several things that need to be done. Flash the Nvidia-provided Ubuntu OS with JetPack, which can be downloaded from the Nvidia Jetson website, to an SD Card. Insert the SD Card and power on the board, go through the setup process to finish the OS configuration, and connect the board to your local network. Then `ssh` from your PC/laptop and install the Edge Impulse tooling via the terminal: +On the Jetson Nano, there are several things that need to be done. Flash the Nvidia-provided Ubuntu OS with JetPack, which can be downloaded from the Nvidia Jetson website, to an SD Card. Insert the SD Card and power on the board, go through the setup process to finish the OS configuration, and connect the board to your local network. Then `ssh` from your PC/laptop and install the Edge Impulse tooling via the terminal: ``` wget -q -O - https://cdn.edgeimpulse.com/firmware/linux/jetson.sh | bash @@ -150,11 +150,11 @@ edge-impulse-linux-runner --model-file ./build/model.eim The inferencing time is about 5ms, which is an incredibly fast detection speed. -![](../.gitbook/assets/quality-control-jetson-nano/5ms.gif) +![](../../.gitbook/assets/quality-control-jetson-nano/5ms.gif) To compare, I have also used the Linux Runner with the CPU version of the model, downloaded via `edge-impulse-linux-runner --download modelfile.eim` then running it with the same command as above. -![](../.gitbook/assets/quality-control-jetson-nano/29ms.gif) +![](../../.gitbook/assets/quality-control-jetson-nano/29ms.gif) You can see the difference in inferencing time, which is almost 6-times faster when we compile and run on the GPU. Impressive! @@ -162,23 +162,23 @@ You can see the difference in inferencing time, which is almost 6-times faster w With the impressive performance of live inferencing shown by the Linux Runner, now we will create a Python program to be able to calculate the number of toppings on a pizza compared to a desired amount, and that will provide an **OK** or **Bad** output if the number of toppings is incorrect. -Because we'll use Python, we need to install the Edge Impulse Python SDK and clone the repository with the Edge Impulse provided examples. Follow the steps here [https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk) to install the Python SDK. Once the SDK is installed, be sure to `git clone https://github.com/edgeimpulse/linux-sdk-python` as well, so that you have the samples locally. +Because we'll use Python, we need to install the Edge Impulse Python SDK and clone the repository with the Edge Impulse provided examples. Follow the steps here [https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk](https://docs.edgeimpulse.com/docs/edge-impulse-for-linux/linux-python-sdk) to install the Python SDK. Once the SDK is installed, be sure to `git clone https://github.com/edgeimpulse/linux-sdk-python` as well, so that you have the samples locally. The program I made (`topping.py`) is a modification of Edge Impulse's `classify.py` in the `examples/image` folder from the `linux-python-sdk` directory. -![](../.gitbook/assets/quality-control-jetson-nano/Photo16.png) +![](../../.gitbook/assets/quality-control-jetson-nano/Photo16.png) -My program will change the moving object detection input from the model file (`model.eim`), for example: `0 0 2 3 3 1 0 1 3 3 3 2 0 0 0 2 3 3 2 0 0 2 5 5 1 0 0 2 3 3 1 0 0 1 2 2 0 0` will record **0** as the sequence separator and record the peak value in each sequence. As an example, if the correct number of toppings on a pizza (per quality control standards) is **3**, and we know that a 0 is a seperator, and anything other than 3 is bad...then `0 3 0 3 0 3 0 5 0 3 0 2 0` is: `OK OK OK BAD OK BAD` +My program will change the moving object detection input from the model file (`model.eim`), for example: `0 0 2 3 3 1 0 1 3 3 3 2 0 0 0 2 3 3 2 0 0 2 5 5 1 0 0 2 3 3 1 0 0 1 2 2 0 0` will record **0** as the sequence separator and record the peak value in each sequence. As an example, if the correct number of toppings on a pizza (per quality control standards) is **3**, and we know that a 0 is a seperator, and anything other than 3 is bad...then `0 3 0 3 0 3 0 5 0 3 0 2 0` is: `OK OK OK BAD OK BAD` -My Python program (`topping.py`) can be downloaded at this link: [https://github.com/Jallson/PizzaQC_Conveyor_Belt/blob/main/topping.py](https://github.com/Jallson/PizzaQC_Conveyor_Belt/blob/main/topping.py) +My Python program (`topping.py`) can be downloaded at this link: [https://github.com/Jallson/PizzaQC\_Conveyor\_Belt/blob/main/topping.py](https://github.com/Jallson/PizzaQC_Conveyor_Belt/blob/main/topping.py) -To run the program, use the command along with the path where `model.eim` file is located. Be sure to use the one built for the GPU, in case you have both on the Nano still: +To run the program, use the command along with the path where `model.eim` file is located. Be sure to use the one built for the GPU, in case you have both on the Nano still: ``` python3 topping.py ~/build/model.eim ``` -To see the process in action, have a look at the demo video availaCheck our demo video: +To see the process in action, have a look at the demo video availaCheck our demo video: {% embed url="https://youtu.be/8hvMst3ykvY" %} diff --git a/image-projects/robotic-arm-sorting-arduino-braccio.md b/readme/featured-machine-learning-projects/robotic-arm-sorting-arduino-braccio.md similarity index 64% rename from image-projects/robotic-arm-sorting-arduino-braccio.md rename to readme/featured-machine-learning-projects/robotic-arm-sorting-arduino-braccio.md index fac769f9..d7531dd2 100644 --- a/image-projects/robotic-arm-sorting-arduino-braccio.md +++ b/readme/featured-machine-learning-projects/robotic-arm-sorting-arduino-braccio.md @@ -1,6 +1,7 @@ --- description: >- - A robotic system for efficient object sorting and placement in dynamic environments, using computer vision to guide the robotic arm. + A robotic system for efficient object sorting and placement in dynamic + environments, using computer vision to guide the robotic arm. --- # ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D @@ -9,46 +10,46 @@ Created By: Naveen Kumar Public Project Link: [https://studio.edgeimpulse.com/public/178900/live](https://studio.edgeimpulse.com/public/178900/live) -GitHub Repository: [https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/urdf](https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/urdf) +GitHub Repository: [https://github.com/metanav/EI\_Pick\_n\_Place/tree/main/pnp\_ws/src/braccio\_description/urdf](https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/urdf) -![](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/cover.gif) +![](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/cover.gif) ## Introduction -In this project, we will design and implement a system capable of performing pick-and-place tasks using a robot arm and a 3D depth camera. The system can recognize and locate objects in a cluttered and dynamic environment, and plan and execute grasping and placing actions. The system consists of the following components: +In this project, we will design and implement a system capable of performing pick-and-place tasks using a robot arm and a 3D depth camera. The system can recognize and locate objects in a cluttered and dynamic environment, and plan and execute grasping and placing actions. The system consists of the following components: -- A 3D camera that can capture images of the scene and provide 3D information about the objects and their poses. -- A robot arm that can move and orient its end-effector according to the desired position and orientation. -- A gripper that can attach and detach objects of various shapes and sizes. -- A control system that can process the 3D images, perform object recognition and localization, plan the grasping and placing strategies, and control the robot arm and the gripper. +* A 3D camera that can capture images of the scene and provide 3D information about the objects and their poses. +* A robot arm that can move and orient its end-effector according to the desired position and orientation. +* A gripper that can attach and detach objects of various shapes and sizes. +* A control system that can process the 3D images, perform object recognition and localization, plan the grasping and placing strategies, and control the robot arm and the gripper. -The system can be used for various pick-and-place applications, such as bin picking, assembly, sorting, or packaging. The system can also be adapted to different scenarios by changing the camera, the robot arm, the gripper, or the software. The system can provide flexibility, accuracy, and efficiency for industrial or domestic tasks. This project might seem simple at first glance, but is surprisingly complex. We will be utilizing plastic toys to sort them. Sorting is a crucial task, from manufacturing to logistics, and requires a great deal of precision and attention to detail. By using these plastic toys, we will be able to test and refine our sorting techniques in a safe and controlled environment. +The system can be used for various pick-and-place applications, such as bin picking, assembly, sorting, or packaging. The system can also be adapted to different scenarios by changing the camera, the robot arm, the gripper, or the software. The system can provide flexibility, accuracy, and efficiency for industrial or domestic tasks. This project might seem simple at first glance, but is surprisingly complex. We will be utilizing plastic toys to sort them. Sorting is a crucial task, from manufacturing to logistics, and requires a great deal of precision and attention to detail. By using these plastic toys, we will be able to test and refine our sorting techniques in a safe and controlled environment. ## Hardware Selection -We are using [Arduino Braccio ++](https://www.arduino.cc/education/braccio/) for the robotic manipulation. +We are using [Arduino Braccio ++](https://www.arduino.cc/education/braccio/) for the robotic manipulation. -![braccio_plus](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/braccio_plus.png) +![braccio\_plus](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/braccio_plus.png) For a depth camera, we will be utilizing the [Luxonis OAK-D](https://docs.luxonis.com/projects/hardware/en/latest/pages/BW1098OAK/), which will be doing object recognition and localization. An object detection model trained using the Edge Impulse Studio will be deployed directly on the OAK-D camera. -![oakd](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/oakd.png) +![oakd](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/oakd.png) A Raspberry Pi 5 will be used as a main controller, to host ROS 2 nodes and an interface between the robotic arm and the depth camera. -![rpi5](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/rpi5.png) +![rpi5](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/rpi5.png) -Instead of sticking with the same old boring color cubes 🧊 that you see everywhere online for a pick-and-place demo, we’re going to have some fun sorting through these colorful plastic toys, **Penguins** 🐧 and **Pigs** 🐷! +Instead of sticking with the same old boring color cubes 🧊 that you see everywhere online for a pick-and-place demo, we’re going to have some fun sorting through these colorful plastic toys, **Penguins** 🐧 and **Pigs** 🐷! -Toys +![Toys](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/toys.jpeg) ## Setting up the Development Environment We can use the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) to install the Raspberry Pi OS (64-bit, Bookworm) on an SD card. The Raspberry Pi Imager allows for easy setup of user accounts, Wi-Fi credentials, and SSH server. -![rpi_imager](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/rpi_imager.png) +![rpi\_imager](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/rpi_imager.png) -After the installation is completed, we can insert the SD card back into the kit and power it on. Once it boots up, we can log in via ssh. +After the installation is completed, we can insert the SD card back into the kit and power it on. Once it boots up, we can log in via ssh. ## Installing ROS 2 Humble @@ -78,19 +79,19 @@ LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 ``` -Otherwise, run the following command to open the Raspberry Pi Configuration CLI: +Otherwise, run the following command to open the Raspberry Pi Configuration CLI: ``` $ sudo raspi-config ``` -Under `Localisation Options` > `Local`, choose `en_US.UTF-8`. +Under `Localisation Options` > `Local`, choose `en_US.UTF-8`. -![locale_1](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/locale_1.png) +![locale\_1](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/locale_1.png) -![locale_2](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/locale_2.png) +![locale\_2](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/locale_2.png) -### Add the ROS 2 apt repository +### Add the ROS 2 apt repository ``` $ sudo apt install software-properties-common @@ -156,10 +157,10 @@ $ colcon build --mixin release **DepthAI ROS** is a ROS 2 package that allows us to: -- Use the OAK-D camera as an RGBD sensor for the 3D vision needs. -- Load Neural Networks and get the inference results straight from the camera. +* Use the OAK-D camera as an RGBD sensor for the 3D vision needs. +* Load Neural Networks and get the inference results straight from the camera. -The following script will install depthai-core, update USB rules, and install depthai device drivers. +The following script will install depthai-core, update USB rules, and install depthai device drivers. ``` $ sudo wget -qO- https://raw.githubusercontent.com/luxonis/depthai-ros/main/install_dependencies.sh | sudo bash @@ -178,9 +179,9 @@ $ MAKEFLAGS="-j1 -l1" colcon build ## micro-ROS -The **micro-ROS** stack integrates microcontrollers seamlessly with standard ROS 2 and brings all major ROS concepts such as nodes, publishers, subscriptions, parameters, and lifecycle onto embedded systems. We will use micro-ROS on the **Arduino Nano RP2040 Connect** mounted on the **Braccio Carrier** board. The Arduino Nano RP2040 will publish the joint states and subscribe to the arm manipulation commands. It will communicate to ROS 2 on the Raspberry Pi 5 over serial port transports. +The **micro-ROS** stack integrates microcontrollers seamlessly with standard ROS 2 and brings all major ROS concepts such as nodes, publishers, subscriptions, parameters, and lifecycle onto embedded systems. We will use micro-ROS on the **Arduino Nano RP2040 Connect** mounted on the **Braccio Carrier** board. The Arduino Nano RP2040 will publish the joint states and subscribe to the arm manipulation commands. It will communicate to ROS 2 on the Raspberry Pi 5 over serial port transports. -![braccio_carrier](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/braccio_carrier.jpeg) +![braccio\_carrier](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/braccio_carrier.jpeg) ### micro-ROS Agent Installation @@ -202,48 +203,48 @@ $ ros2 run micro_ros_setup build_agent.sh We captured 101 images of the pigs and penguins using the OAK-D camera and uploaded them to Edge Impulse Studio. -![upload_data](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/upload_data.png) +![upload\_data](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/upload_data.png) We can see the uploaded images on the **Data Acquisition** page. -![datasets](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/datasets.png) +![datasets](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/datasets.png) We can now label the data using bounding boxes in the **Labeling Queue** tab, as demonstrated in the GIF below. -![labelling](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/labelling.gif) +![labelling](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/labelling.gif) ## Model Training To create an Impulse, follow these steps: -- Go to the **Impulse Design** section, then select the **Create Impulse** page. We have opted for a 320x320 pixel image size in the "Image Data" form fields to achieve better accuracy. -- Click on "Add a processing block" and choose "Image". This step will pre-process and normalize the image data while also giving us the option to choose the color depth. -- Click on "Add a learning block" and choose "Object Detection (Images)". -- Finally, click on the "Save Impulse" button to complete the process. +* Go to the **Impulse Design** section, then select the **Create Impulse** page. We have opted for a 320x320 pixel image size in the "Image Data" form fields to achieve better accuracy. +* Click on "Add a processing block" and choose "Image". This step will pre-process and normalize the image data while also giving us the option to choose the color depth. +* Click on "Add a learning block" and choose "Object Detection (Images)". +* Finally, click on the "Save Impulse" button to complete the process. -![create_impulse](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/create_impulse.png) +![create\_impulse](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/create_impulse.png) -On the **Image** page, choose *RGB* as color depth and click on the **Save parameters** button. The page will be redirected to the **Generate Features** page. +On the **Image** page, choose _RGB_ as color depth and click on the **Save parameters** button. The page will be redirected to the **Generate Features** page. -![raw_features](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/raw_features.png) +![raw\_features](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/raw_features.png) Now we can initiate feature generation by clicking on the **Generate features** button. Once the feature generation is completed, the data visualization will be visible in the **Feature Explorer** panel. -![generate_features](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/generate_features.png) +![generate\_features](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/generate_features.png) Go to the **Object Detection** page, then click "Choose a different model" and select the **YOLOv5** model. There are 4 variations of the model size available, and we selected the **Nano** version with 1.9 million parameters. Afterward, click the "Start training" button. The training process will take a few minutes to complete. -Toys +![Toys](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/training_settings.png) Once the training is completed we can see the precision score and metrics as shown below. -Toys +![Toys](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/training_accuracy.png) ## Model Testing On the **Model testing** page, click on the "Classify All" button which will initiate model testing with the trained float32 model. The testing accuracy is **100%**. -![model_testing](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/testing_results.png) +![model\_testing](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/testing_results.png) ## Model Deployment @@ -255,7 +256,7 @@ $ sudo apt install -y gcc g++ make build-essential nodejs sox gstreamer1.0-tools $ sudo npm install edge-impulse-linux -g --unsafe-perm ``` -Execute the following commands to use the OAK-D as a USB webcam for the Edge Impulse Linux Runner. +Execute the following commands to use the OAK-D as a USB webcam for the Edge Impulse Linux Runner. ``` $ git clone https://github.com/luxonis/depthai-python.git @@ -269,7 +270,7 @@ and open a UVC viewer to check the camera stream. To close: Ctrl+C ``` -To download the `eim` model and start the inferencing, run the following command and follow the instructions. +To download the `eim` model and start the inferencing, run the following command and follow the instructions. ``` $ edge-impulse-linux-runner @@ -289,17 +290,17 @@ Edge Impulse Linux runner v1.5.1 Want to see a feed of the camera and live classification in your browser? Go to http://192.168.3.10:4912 ``` -We can see the inferencing output on the web browser. Also, we can monitor the terminal logs. +We can see the inferencing output on the web browser. Also, we can monitor the terminal logs. -![inferencing](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/inferencing.gif) +![inferencing](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/inferencing.gif) To allow DepthAI to use our custom-trained model, we need to convert them into a MyriadX blob file format so that they are optimized for the Movidius Myriad X processor on the OAK-D. -![model_compile](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/model_compile.png) +![model\_compile](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/model_compile.png) The Edge Impulse Studio helps us save a step by providing the ONNX format for the trained YOLOv5 model that we can download from the project's **Dashboard** page. -![download_block_output](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/download_block_output.png) +![download\_block\_output](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/download_block_output.png) We will utilize the **OpenVINO** model optimizer for conversion on an x86 Linux machine. OpenVINO is an open-source software toolkit for optimizing and deploying deep learning models. Execute the following commands to install all prerequisites for the conversion process. @@ -353,10 +354,10 @@ base, ext = os.path.splitext(onnx_path) onnx.save(onnx_model, f'{base}_prune{ext}') ``` -The ONNX model can be large and architecture-dependent. For the on-device inferencing, we need to convert the model to the OpenVINO Intermediate Representation (IR) format which is a proprietary model format of OpenVINO. The model conversion API translates the frequently used deep learning operations to their respective similar representation in OpenVINO and tunes them with the associated weights and biases from the trained model. The resulting IR contains two files: +The ONNX model can be large and architecture-dependent. For the on-device inferencing, we need to convert the model to the OpenVINO Intermediate Representation (IR) format which is a proprietary model format of OpenVINO. The model conversion API translates the frequently used deep learning operations to their respective similar representation in OpenVINO and tunes them with the associated weights and biases from the trained model. The resulting IR contains two files: -- `.xml` - Describes the model topology. -- `.bin` - Contains the weights and binary data. +* `.xml` - Describes the model topology. +* `.bin` - Contains the weights and binary data. Execute the following command to generate the IR files. @@ -393,7 +394,7 @@ blob_path = blobconverter.from_openvino( shutil.move(str(blob_path), blob_dir) ``` -This will create the *ei-pnp_yolov5n_320_openvino_2022.1_6shave.blob* file in the IR directory. We should copy this blob file to the `~/EI_Pick_n_Place/pnp_ws/src/ei_yolov5_detections/resources` folder on the Raspberry Pi 5. We can test the generated model using the depthai-python library: +This will create the _ei-pnp\_yolov5n\_320\_openvino\_2022.1\_6shave.blob_ file in the IR directory. We should copy this blob file to the `~/EI_Pick_n_Place/pnp_ws/src/ei_yolov5_detections/resources` folder on the Raspberry Pi 5. We can test the generated model using the depthai-python library: ``` $ pip3 install -U pip @@ -405,29 +406,29 @@ $ python3 ei_yolov5_spatial_stream.py The Python script can be found in the GitHub repository: -https://github.com/metanav/EI_Pick_n_Place/blob/main/pnp_ws/src/ei_yolov5_detections/src/ei_yolov5_spatial_stream.py +https://github.com/metanav/EI\_Pick\_n\_Place/blob/main/pnp\_ws/src/ei\_yolov5\_detections/src/ei\_yolov5\_spatial\_stream.py -Take a look at the GIF below, which displays the RGB and spatial depth detections side by side. The RGB detections indicate the 3D location (X, Y, Z) with bounding boxes, while the depth image shows the bounding boxes with a 25% scale factor for accurate object localization. For depth (Z), each pixel inside the scaled bounding box (ROI) is taken into account. This gives us a set of depth values, which are then averaged to get the final depth value. Also, the depth image is wider than the RGB image because they have different resolutions. +Take a look at the GIF below, which displays the RGB and spatial depth detections side by side. The RGB detections indicate the 3D location (X, Y, Z) with bounding boxes, while the depth image shows the bounding boxes with a 25% scale factor for accurate object localization. For depth (Z), each pixel inside the scaled bounding box (ROI) is taken into account. This gives us a set of depth values, which are then averaged to get the final depth value. Also, the depth image is wider than the RGB image because they have different resolutions. -![depth_inferencing](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/depth_inferencing.gif) +![depth\_inferencing](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/depth_inferencing.gif) ## Setup the Robotic Arm ### Build a visual robot model -First, we need to define a visual model of the Arduino Braccio ++ using the URDF (Unified Robot Description Format) which is a file format for specifying the geometry and organization of robots in ROS 2. We will be using the [publicly available](https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/stl) STL files for the parts of the robot. We can see one of the STL parts (shoulder) in the following GIF. +First, we need to define a visual model of the Arduino Braccio ++ using the URDF (Unified Robot Description Format) which is a file format for specifying the geometry and organization of robots in ROS 2. We will be using the [publicly available](https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/stl) STL files for the parts of the robot. We can see one of the STL parts (shoulder) in the following GIF. -![stl](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/stl.gif) +![stl](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/stl.gif) -We created a ROS 2 package `moveit_resources_braccio_description` to keep all STL files and URDF for reusability. The robot model URDF can be found in the GitHub repository for this project: +We created a ROS 2 package `moveit_resources_braccio_description` to keep all STL files and URDF for reusability. The robot model URDF can be found in the GitHub repository for this project: -https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/urdf +https://github.com/metanav/EI\_Pick\_n\_Place/tree/main/pnp\_ws/src/braccio\_description/urdf ### Verify the robot model We can verify if the URDF is functioning as expected by publishing simulated joint states and observing the changes in the robot model using the RViz 2 graphical interface. Execute the following commands to install the `urdf_launch` and `joint_state_publisher` packages and launch the visualization. -``` +``` $ cd ~/ros2_humble/src $ git clone https://github.com/ros/urdf_launch.git $ git clone -b ros2 https://github.com/ros/joint_state_publisher.git @@ -443,11 +444,11 @@ $ ros2 launch moveit_resources_braccio_description display.launch.py By adjusting the sliders for the joints, we can observe the corresponding changes in the robot model. -![robot_urdf_rviz](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/robot_urdf_rviz.gif) +![robot\_urdf\_rviz](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/robot_urdf_rviz.gif) ### Generate configuration using the MoveIt Setup Assistant 2.0 -The MoveIt Setup Assistant 2.0 is a GUI for configuring the manipulator for use with MoveIt 2. Its primary function is generating a Semantic Robot Description Format (SRDF) file for the manipulator, which specifies additional information required by MoveIt 2 such as planning groups, end effectors, and various kinematic parameters. Additionally, it generates other necessary configuration files for use with the MoveIt 2 pipeline. +The MoveIt Setup Assistant 2.0 is a GUI for configuring the manipulator for use with MoveIt 2. Its primary function is generating a Semantic Robot Description Format (SRDF) file for the manipulator, which specifies additional information required by MoveIt 2 such as planning groups, end effectors, and various kinematic parameters. Additionally, it generates other necessary configuration files for use with the MoveIt 2 pipeline. To start the MoveIt Setup Assistant 2.0, execute the commands below. @@ -459,41 +460,41 @@ $ ros2 launch moveit_setup_assistant setup_assistant.launch.py Click on the **Create New MoveIt Configuration Package** and provide the path of the `braccio.urdf` file from the `moveit_resources_braccio_description` package. -![moveit2_assistant_1](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_1.png) +![moveit2\_assistant\_1](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_1.png) To generate the collision matrix, select the **Self-Collisions** pane on the left-hand side of the MoveIt Setup Assistant and adjust the self-collision sampling density. Then, click on the **Generate Collision Matrix** button to initiate the computation. The Setup Assistant will take a few seconds to compute the self-collision matrix, which involves checking for pairs of links that can be safely disabled from collision checking. -![moveit2_assistant_2](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_2.png) +![moveit2\_assistant\_2](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_2.png) We will define a `fixed` virtual joint that attaches the `base_link` of the arm to the `world` frame. This virtual joint signifies that the base of the arm remains stationary in the world frame. -![moveit2_assistant_3](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_3.png) +![moveit2\_assistant\_3](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_3.png) Planning groups in MoveIt 2 semantically describe different parts of the robot, such as the arm or end effector, to facilitate motion planning. -![moveit2_assistant_4](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_4.png) +![moveit2\_assistant\_4](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_4.png) The Setup Assistant allows us to add predefined poses to the robot’s configuration, which can be useful for defining specific initial or ready poses. Later, the robot can be commanded to move to these poses using the MoveIt API. Click on the **Add Pose** and choose a name for the pose. -![moveit2_assistant_5](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_5.png) +![moveit2\_assistant\_5](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_5.png) The robot will be in the default pose, with all joints set to their zero values. Move the individual joints around until we find the intended pose and then **Save** the pose. -![moveit2_assistant_7](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_7.png) +![moveit2\_assistant\_7](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_7.png) -Now we can designate the `braccio_gripper` group as an end effector. The end effectors can be used for attaching objects to the arm while carrying out pick-and-place tasks. +Now we can designate the `braccio_gripper` group as an end effector. The end effectors can be used for attaching objects to the arm while carrying out pick-and-place tasks. -![moveit2_assistant_6](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_6.png) +![moveit2\_assistant\_6](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_assistant_6.png) ## Arduino Braccio++ Controller Firmware Please follow the instructions [here](https://www.arduino.cc/en/software) to download and install the Arduino IDE. After installation, open the Arduino IDE and install the board package for the **Arduino Mbed OS Nano Boards** by going to **Tools** > **Board** > **Boards Manager**. Search the board package as shown below and install it. -![board_manager](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/board_manager.png) +![board\_manager](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/board_manager.png) -After completing the board package installation, choose the **Arduino Nano RP2040 Connect** from **Tools** > **Board** > **Arduino Mbed OS Nano boards** menu. We must install **[Arduino_Braccio_plusplus](https://github.com/arduino-libraries/Arduino_Braccio_plusplus) (1.3.2)** and **[micro_ros_arduino]([micro_ros_arduino](https://github.com/micro-ROS/micro_ros_arduino)) (humble)** libraries. The firmware sketch can be found in the GitHub repository: +After completing the board package installation, choose the **Arduino Nano RP2040 Connect** from **Tools** > **Board** > **Arduino Mbed OS Nano boards** menu. We must install [**Arduino\_Braccio\_plusplus**](https://github.com/arduino-libraries/Arduino_Braccio_plusplus) **(1.3.2)** and [**micro\_ros\_arduino**](../../image-projects/\[micro_ros_arduino]\(https:/github.com/micro-ROS/micro_ros_arduino\)/) **(humble)** libraries. The firmware sketch can be found in the GitHub repository: -[https://github.com/metanav/EI_Pick_n_Place/blob/main/Arduino/braccio_plus_plus_controller_final_v3.1/braccio_plus_plus_controller_final_v3.1.ino](https://github.com/metanav/EI_Pick_n_Place/blob/main/Arduino/braccio_plus_plus_controller_final_v3.1/braccio_plus_plus_controller_final_v3.1.ino). +[https://github.com/metanav/EI\_Pick\_n\_Place/blob/main/Arduino/braccio\_plus\_plus\_controller\_final\_v3.1/braccio\_plus\_plus\_controller\_final\_v3.1.ino](https://github.com/metanav/EI_Pick_n_Place/blob/main/Arduino/braccio_plus_plus_controller_final_v3.1/braccio_plus_plus_controller_final_v3.1.ino). Now we should build and upload the firmware to the Arduino Nano RP2040 connect. During startup, the application attempts to connect to the micro-ROS agent on the Raspberry Pi 5 over serial port transports. It then initiates a node that publishes real-time states of the robotic arm joints to the `/joint_states` topic and subscribes to the `/gripper/gripper_cmd` and `/arm/follow_joint_trajectory` topics. @@ -501,95 +502,91 @@ Now we should build and upload the firmware to the Arduino Nano RP2040 connect. We should launch the ROS 2 nodes on separate terminals on the Raspberry Pi 5 by executing the following commands step-by-step. -1. ##### Launch micro-ROS agent - - The micro-ROS agent exposes the publishers and action server running on the Braccio ++ MCU to ROS 2. - - ``` - $ source ~/ros2_humble/install/setup.sh - $ source ~/microros_ws/install/setup.sh - $ ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyACM0 - ``` - -2. ##### Launch ei_yolov5_detections node - - The `ei_yolov5_detections` node detects the objects and publishes the detection results using the Edge Impulse trained model on the OAK-D depth camera. - - ``` - $ source ~/ros2_humble/install/setup.sh - $ source ~/dai_ws/install/setup.sh - $ source ~/pnp_ws/install/setup.sh - $ ros2 launch ei_yolov5_detections ei_yolov5_publisher.launch.py - ``` - - We can check the spatial detection message as follows. - - ``` - $ ros2 topic echo /ei_yolov5/spatial_detections - - --- - header: - stamp: - sec: 1708778065 - nanosec: 719748991 - frame_id: oak_rgb_camera_optical_frame - detections: - - results: - - class_id: '0' - score: 0.0 - bbox: - center: - position: - x: 163.0 - y: 232.5 - theta: 0.0 - size_x: 54.0 - size_y: 91.0 - position: - x: 0.0026015033945441246 - y: -0.05825277045369148 - z: 0.35132256150245667 - is_tracking: false - tracking_id: '' - ``` - - - -3. ##### Launch pick_n_place node - - The `pick_n_place` node plans a pick and place operation using [MoveIt Task Constructor](https://github.com/ros-planning/moveit_task_constructor/tree/ros2/). MoveIt Task Constructor provides a way to plan for tasks that consist of multiple different subtasks (known as stages as shown in the image below). - -![moveit2_task_stages](../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_task_stages.png) - - This node subscribes to the `/ei_yolov5/spatial_detections` topic and plans the pick and place operation. While bringing up this node, we need to provide command line parameters for the exact (X, Y, Z) position of the camera in meters from the base of the robot. - - ``` - $ source ~/ros2_humble/install/setup.sh - $ source ~/ws_moveit2/install/setup.sh - $ source ~/pnp_ws/install/setup.sh - $ ros2 launch pick_n_place pick_n_place.launch.py \ - cam_pos_x:=0.26 \ - cam_pos_y:=-0.425 \ - cam_pos_z:=0.09 \ - cam_roll:=0.0 \ - cam_pitch:=0.0 \ - cam_yaw:=1.5708 \ - parent_frame:=base_link - ``` - - The launch file also brings up the `robot_state_publisher` and `move_group` nodes to publish the robot model and provide MoveIt 2 actions and services respectively. - -4. ##### Launch RViz 2 - - We can see the real-time motion planning solution execution visualization using the **RViz 2**. - - ``` - $ source ~/ros2_humble/install/setup.sh - $ source ~/ws_moveit2/install/setup.sh - $ source ~/pnp_ws/install/setup.sh - $ export DISPLAY=:0 - $ ros2 launch pick_n_place rviz.launch.py - ``` +1. **Launch micro-ROS agent** + + The micro-ROS agent exposes the publishers and action server running on the Braccio ++ MCU to ROS 2. + + ``` + $ source ~/ros2_humble/install/setup.sh + $ source ~/microros_ws/install/setup.sh + $ ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyACM0 + ``` +2. **Launch ei\_yolov5\_detections node** + + The `ei_yolov5_detections` node detects the objects and publishes the detection results using the Edge Impulse trained model on the OAK-D depth camera. + + ``` + $ source ~/ros2_humble/install/setup.sh + $ source ~/dai_ws/install/setup.sh + $ source ~/pnp_ws/install/setup.sh + $ ros2 launch ei_yolov5_detections ei_yolov5_publisher.launch.py + ``` + + We can check the spatial detection message as follows. + + ``` + $ ros2 topic echo /ei_yolov5/spatial_detections + + --- + header: + stamp: + sec: 1708778065 + nanosec: 719748991 + frame_id: oak_rgb_camera_optical_frame + detections: + - results: + - class_id: '0' + score: 0.0 + bbox: + center: + position: + x: 163.0 + y: 232.5 + theta: 0.0 + size_x: 54.0 + size_y: 91.0 + position: + x: 0.0026015033945441246 + y: -0.05825277045369148 + z: 0.35132256150245667 + is_tracking: false + tracking_id: '' + ``` +3. **Launch pick\_n\_place node** + + The `pick_n_place` node plans a pick and place operation using [MoveIt Task Constructor](https://github.com/ros-planning/moveit_task_constructor/tree/ros2/). MoveIt Task Constructor provides a way to plan for tasks that consist of multiple different subtasks (known as stages as shown in the image below). + +![moveit2\_task\_stages](../../.gitbook/assets/robotic-arm-sorting-arduino-braccio/moveit2_task_stages.png) + +This node subscribes to the `/ei_yolov5/spatial_detections` topic and plans the pick and place operation. While bringing up this node, we need to provide command line parameters for the exact (X, Y, Z) position of the camera in meters from the base of the robot. + +``` +$ source ~/ros2_humble/install/setup.sh +$ source ~/ws_moveit2/install/setup.sh +$ source ~/pnp_ws/install/setup.sh +$ ros2 launch pick_n_place pick_n_place.launch.py \ + cam_pos_x:=0.26 \ + cam_pos_y:=-0.425 \ + cam_pos_z:=0.09 \ + cam_roll:=0.0 \ + cam_pitch:=0.0 \ + cam_yaw:=1.5708 \ + parent_frame:=base_link +``` + +The launch file also brings up the `robot_state_publisher` and `move_group` nodes to publish the robot model and provide MoveIt 2 actions and services respectively. + +4. **Launch RViz 2** + + We can see the real-time motion planning solution execution visualization using the **RViz 2**. + + ``` + $ source ~/ros2_humble/install/setup.sh + $ source ~/ws_moveit2/install/setup.sh + $ source ~/pnp_ws/install/setup.sh + $ export DISPLAY=:0 + $ ros2 launch pick_n_place rviz.launch.py + ``` ## Live Demo @@ -598,4 +595,3 @@ We should launch the ROS 2 nodes on separate terminals on the Raspberry Pi 5 by ## Conclusion This project successfully demonstrates the design and implementation of a sophisticated pick-and-place system using a robot arm equipped with a 3D depth camera. The system's ability to recognize and locate objects in a cluttered and dynamic environment, coupled with its precise grasping and placing actions, showcases its potential for various industrial and domestic applications. This project underscores the complexity and importance of sorting tasks in various sectors, from manufacturing to logistics, and demonstrates how advanced robotic systems can meet these challenges with high efficiency and accuracy. - diff --git a/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md b/readme/featured-machine-learning-projects/rooftop-ice-synthetic-data-omniverse.md similarity index 76% rename from nvidia-projects/rooftop-ice-synthetic-data-omniverse.md rename to readme/featured-machine-learning-projects/rooftop-ice-synthetic-data-omniverse.md index dd5b46be..6c939021 100644 --- a/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md +++ b/readme/featured-machine-learning-projects/rooftop-ice-synthetic-data-omniverse.md @@ -1,6 +1,8 @@ --- description: >- - Tracking Rooftop ice buildup detection using Edge Impulse and The Things Network, with synthetic data created in NVIDIA Omniverse Replicator and Sun Studies. + Tracking Rooftop ice buildup detection using Edge Impulse and The Things + Network, with synthetic data created in NVIDIA Omniverse Replicator and Sun + Studies. --- # Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator @@ -11,13 +13,13 @@ Public Project Link: [https://studio.edgeimpulse.com/public/332581/live](https:/ GitHub Repo: [https://github.com/eivholt/icicle-monitor](https://github.com/eivholt/icicle-monitor) -![](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/cover1.png) +![](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/cover1.png) ## Introduction The portable device created in this project monitors buildings and warns the responsible parties when potentially hazardous icicles are formed. In ideal conditions, icicles can form at a rate of [more than 1 cm (0.39 in) per minute](https://en.wikipedia.org/wiki/Icicle). In cold climates, many people are injured and killed each year by these solid projectiles, leading responsible building owners to often close sidewalks in the spring to minimize risk. This project demonstrates how an extra set of digital eyes can notify property owners icicles are forming and need to be removed before they can cause harm. -![Downtown, photo: Avisa Nordland](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/IMG_8710.jpg) +![Downtown, photo: Avisa Nordland](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/IMG_8710.jpg) ## Hardware used @@ -54,7 +56,7 @@ The main challenge of detecting forming icicles is the translucent nature of ice A powerful platform combined with a high resolution camera with fish-eye lens would increase the ability to detect icicles. However, by deploying the object detection model to a small, power-efficient, but highly constrained device, options for device installation increase. Properly protected against moisture this device can be mounted outdoors on walls or poles facing the roofs in question. LoRaWAN communication enables low battery consumption and long transmission range. -![Arduino Portenta H7](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/20240413_215105_.jpg) +![Arduino Portenta H7](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/20240413_215105_.jpg) ## Object detection using a neural network @@ -72,41 +74,42 @@ NVIDIA Omniverse Code is an IDE that allows us to compose 3D scenes and to write It's possible to create an empty scene in Omniverse and add content programmatically. However, composing initial objects by hand serves as a practical starting point. In this project [a royalty free 3D model of a house](https://www.cgtrader.com/free-3d-models/exterior/house/house-model-3d-dom-2) was used as a basis. -![3D house model](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/house.png) +![3D house model](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/house.png) ### Icicle models -To represent the icicle, a high quality model pack was purchased at [Turbo Squid](https://www.turbosquid.com/3d-models/). +To represent the icicle, a high quality model pack was purchased at [Turbo Squid](https://www.turbosquid.com/3d-models/). -![3D icicle models purchased at Turbo Squid](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/turbo-squid-icicle.png) +![3D icicle models purchased at Turbo Squid](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/turbo-squid-icicle.png) To be able to import the models into Omniverse and Isaac Sim, all models have to be converted to [OpenUSD-format](https://developer.nvidia.com/usd). While USD is a great emerging standard for describing, composing, simulating, and collaborting within 3D worlds, it is not yet commonly supported in asset marketplaces. [This article](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/surgery-inventory-synthetic-data) outlines considerations when performing conversion using Blender to USD. Note that it is advisable to export each individual model and to choose a suitable origin/pivot point. Blender change origin cheat sheet: -+ Select vertex on model (Edit Mode), Shift+S-> Cursor to selected -+ (Object Mode) Select Hierarchy, Object>Set Origin\Origin to 3D Cursor -+ (Object Mode) Shift+S\Cursor to World Origin + +* Select vertex on model (Edit Mode), Shift+S-> Cursor to selected +* (Object Mode) Select Hierarchy, Object>Set Origin\Origin to 3D Cursor +* (Object Mode) Shift+S\Cursor to World Origin Tip for export: -+ Selection only -+ Convert Orientation: - + Forward Axis: X - + Up Axis: Y -![3D icicle models exported from Blender](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Blender_select_vertex.png) +* Selection only +* Convert Orientation: + * Forward Axis: X + * Up Axis: Y + +![3D icicle models exported from Blender](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Blender_select_vertex.png) ### Setting semantic metadata on objects To be able to produce images for training and include labels, we can use a feature of Replicator toolbox found under menu Replicator > Semantics Schema Editor. -![Semantics Schema Editor](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/semantic-editor.png) +![Semantics Schema Editor](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/semantic-editor.png) Here we can select each top node representing an item for object detection and add a key-value pair. Choosing "class" as Semantic Type and "ice" as Semantic Data enables us to export this string as a label later. ### Creating a program for domain randomization -With a basic 3D stage created and objects of interest labeled, we can continue creating a program that will make sure we produce images with slight variations. Our program can be named anything, ending in `.py` and preferably placed close to the stage USD-file. Here is a sample of such a program: [replicator_init.py](https://github.com/eivholt/icicle-monitor/blob/main/omniverse-replicator/replicator_init.py): - +With a basic 3D stage created and objects of interest labeled, we can continue creating a program that will make sure we produce images with slight variations. Our program can be named anything, ending in `.py` and preferably placed close to the stage USD-file. Here is a sample of such a program: [replicator\_init.py](https://github.com/eivholt/icicle-monitor/blob/main/omniverse-replicator/replicator_init.py): To keep the items generated in our script separate from the manually created content, we start by creating a new layer in the 3D stage: @@ -141,7 +144,7 @@ with rep.trigger.on_frame(num_frames=10000, rt_subframes=50): rep.randomizer.randomize_camera(icicles) ``` -The parameter *num_frames* specifies the desired number of renders. The *rt_subframes* parameter allows the rendering process to advance a set number of frames before the result is captured and saved to disk. A higher setting enhances complex ray tracing effects like reflections and translucency by giving them more time to interact across surfaces, though it increases rendering time. Each randomization routine is invoked with the option to include specific parameters. +The parameter _num\_frames_ specifies the desired number of renders. The _rt\_subframes_ parameter allows the rendering process to advance a set number of frames before the result is captured and saved to disk. A higher setting enhances complex ray tracing effects like reflections and translucency by giving them more time to interact across surfaces, though it increases rendering time. Each randomization routine is invoked with the option to include specific parameters. To save each image and its corresponding semantic data, we utilize a designated API. While customizing the writer was considered, attempts to do so using Replicator version 1.9.8 on Windows led to errors. Therefore, we are employing the "BasicWriter" and will develop an independent script to generate a label format that is compatible with Edge Impulse. @@ -156,11 +159,11 @@ writer.attach([render_product]) asyncio.ensure_future(rep.orchestrator.step_async()) ``` -*rgb* indicates that we want to save images to disk as `.png` files. Note that labels are created setting *bounding_box_2d_loose*. This is used in this case instead of *bounding_box_2d_tight* as the latter in some cases would not include the tip of the icicles in the resulting bounding box. It also creates labels from the previously defined semantics. The code ends with running a single iteration of the process in Omniverse Code, so we can preview the results. +_rgb_ indicates that we want to save images to disk as `.png` files. Note that labels are created setting _bounding\_box\_2d\_loose_. This is used in this case instead of _bounding\_box\_2d\_tight_ as the latter in some cases would not include the tip of the icicles in the resulting bounding box. It also creates labels from the previously defined semantics. The code ends with running a single iteration of the process in Omniverse Code, so we can preview the results. The bounding boxes can be visualized by clicking the sensor widget, checking "BoundingBox2DLoose" and finally "Show Window". -![Omniverse bounding box](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/omniverse-bb.png) +![Omniverse bounding box](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/omniverse-bb.png) Now we can implement the randomization logic. First we'll use a method that flips and scatters the icicles on a defined plane. @@ -190,13 +193,13 @@ with camera: return camera.node ``` -We can define the methods in any order we like, but in *rep.trigger.on_frame* it is crucial that the icicles are placed before pointing the camera. +We can define the methods in any order we like, but in _rep.trigger.on\_frame_ it is crucial that the icicles are placed before pointing the camera. ### Running domain randomization With a basic randomization program in place, we could run it from the embedded script editor (Window > Script Editor), but more robust Python language support can be achieved by developing in Visual Studio Code instead. To connect VS Code with Omniverse we can use the Visual Studio Code extension [Embedded VS Code for NVIDIA Omniverse](https://marketplace.visualstudio.com/items?itemName=Toni-SM.embedded-vscode-for-nvidia-omniverse). See the [extension repo](https://github.com/Toni-SM/semu.misc.vscode) for setup. When ready to run go to Replicator > Start and check progress in the defined output folder. -![Produced images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output1.png) +![Produced images](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output1.png) ### Randomizing colors @@ -229,9 +232,9 @@ with rep.trigger.on_frame(num_frames=2000, rt_subframes=50): # rt_subframes=50 rep.randomizer.randomize_screen(screen) ``` -![Random background color](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_color.png) +![Random background color](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_color.png) -![Random background color](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output2.png) +![Random background color](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output2.png) Now each image will have a background with random (deterministic, same starting seed) RGB color. Replicator takes care of creating a material with a shader for us. As you might remember, in an effort to reduce RAM usage our neural network reduces RGB color channels to grayscale. In this project we could simplify the color randomization to only pick grayscale colors. The example has been included as it would benefit in projects where color information is not reduced. To only randomize in grayscale, we could change the code in the randomization function to use the same value for R, G and B as follows: @@ -245,7 +248,7 @@ def randomize_screen(screen): return screen.node ``` -![Random background grayscale](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_grayscale.png) +![Random background grayscale](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_grayscale.png) ### Randomizing textures @@ -274,11 +277,11 @@ with rep.trigger.on_frame(num_frames=2000, rt_subframes=50): rep.randomizer.randomize_screen(screen, texture_files) ``` -![Random background texture](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_texture.png) +![Random background texture](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_texture.png) -![Random background texture, camera perspective](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_texture_viewport.png) +![Random background texture, camera perspective](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/random_texture_viewport.png) -![Random background texture](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output3.png) +![Random background texture](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output3.png) We could instead generate textures with random shapes and colors. Either way, the resulting renders will look weird, but help the model training process weight features that are relevant for the icicles, not the background. @@ -286,7 +289,7 @@ These are rather unsophisticated approaches. More realistic results would be ach ### Creating realistic outdoor lighting conditions using Sun Study -In contrast to a controlled indoor environment, creating a robust object detection model intended for outdoor use needs training images with a wide range of realistic natural light. When generating synthetic images we can utilize an [extension that approximates real world sunlight](https://docs.omniverse.nvidia.com/extensions/latest/ext_sun-study.html) based on sun studies. +In contrast to a controlled indoor environment, creating a robust object detection model intended for outdoor use needs training images with a wide range of realistic natural light. When generating synthetic images we can utilize an [extension that approximates real world sunlight](https://docs.omniverse.nvidia.com/extensions/latest/ext_sun-study.html) based on sun studies. {% embed url="https://youtu.be/MRD-oAxaV8w" %} @@ -294,23 +297,23 @@ The extension let's us set world location, date and time. We can also mix this w {% embed url="https://youtu.be/qvDXRqBxECo" %} -![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output5.png) +![Sun Study](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output5.png) -![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output4.png) +![Sun Study](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output4.png) -![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output6.png) +![Sun Study](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output6.png) ### Creating label file for Edge Impulse Studio -Edge Impulse Studio supports a wide range of image labeling formats for object detection. The output from Replicator's BasicWriter needs to be transformed so it can be uploaded either through the web interface or via [the Ingestion API](https://docs.edgeimpulse.com/reference/ingestion-api#ingestion-api). +Edge Impulse Studio supports a wide range of image labeling formats for object detection. The output from Replicator's BasicWriter needs to be transformed so it can be uploaded either through the web interface or via [the Ingestion API](https://docs.edgeimpulse.com/reference/ingestion-api#ingestion-api). -Provided is a simple Python program, [basic_writer_to_pascal_voc.py](https://github.com/eivholt/icicle-monitor/blob/main/scripts/basic_writer_to_pascal_voc.py) to help get started. Documentation on the supported Edge Impulse [label formats is located here](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from a terminal with: +Provided is a simple Python program, [basic\_writer\_to\_pascal\_voc.py](https://github.com/eivholt/icicle-monitor/blob/main/scripts/basic_writer_to_pascal_voc.py) to help get started. Documentation on the supported Edge Impulse [label formats is located here](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from a terminal with: -``` +``` python basic_writer_to_pascal_voc.py ``` -or debug from Visual Studio Code by setting input folder in `launch.json` like this: +or debug from Visual Studio Code by setting input folder in `launch.json` like this: ``` "args": ["../out"] @@ -344,25 +347,25 @@ Since our synthetic training images are based on both individual and two differe 2,000 images: -![2000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/2000-images.png) +![2000 images](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/2000-images.png) 6,000 images: -![6000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/6000-images-120cycles.png) +![6000 images](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/6000-images-120cycles.png) 14,000 images: -![14000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/14000-images-120cycles_no-opt.png) +![14000 images](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/14000-images-120cycles_no-opt.png) 26,000 images: -![26000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/26000-images-light-5000coco-120cycles_no-opt.png) +![26000 images](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/26000-images-light-5000coco-120cycles_no-opt.png) Note that the final results include 5000 images from the [COCO 2017 dataset](https://cocodataset.org/#download). Adding this reduces F1 score a bit, but results in a model with significantly less overfitting, that shows almost no false positives when classifying random background scenes. If we look at results from model testing in Edge Impulse Studio, at first glance the numbers are less than impressive. -![Model testing](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/model-testing1.png) +![Model testing](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/model-testing1.png) However if we investigate individual samples where F1 score is less than 100%, we see that the model indeed has detected the icicles, but clustered differently than how the image was originally labeled. What we should look out for are samples that contain visible icicles where none were detected. @@ -372,77 +375,77 @@ In the end virtual and real-life testing tells us how well the model really perf We can get useful information about model performance with minimal effort by testing it in a virtual environment. Install [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac-sim) and the [Edge Impulse extension](https://github.com/edgeimpulse/edge-impulse-omniverse-ext). -![Edge Impulse extension](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-enable.png) +![Edge Impulse extension](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-enable.png) Install the Sun Study extension in Isaac Sim to be able to vary light conditions while testing. -![Sun Study in Isaac Sim](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sunstudy.png) +![Sun Study in Isaac Sim](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sunstudy.png) Paste your API key found in the Edge Impulse Studio > Dashboard > Keys > Add new API key into Omniverse Extension: -![Edge Impulse extension API key](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-api-key.png) +![Edge Impulse extension API key](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-api-key.png) To be able to classify any virtual camera capture we first need to build a version of the model that can run in a JavaScript environment. In Edge Impulse Studio, go to **Deployment**, find "WebAssembly" in the search box and click **Build**. We don't need to keep the resulting .zip package, the extension will find and download it by itself in a moment. -![Edge Impulse WebAssembly](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-webasm.png) +![Edge Impulse WebAssembly](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-webasm.png) Back in the Edge Impulse extension in Isaac Sim, when we expand the "Classification" group, a message will tell us everything is ready: "Your model is ready! You can now run inference on the current scene". Before we test it we will make some accommodations in the viewport. -Switch to "RTX - Interactive" to make sure the scene is rendered realistically. +Switch to "RTX - Interactive" to make sure the scene is rendered realistically. Set viewport resolution to square 1:1 with either the same resolution as our intended device inference (120x120 pixels), or (512x512 pixels). -![Isaac Sim viewport resolution](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-resolution.png) +![Isaac Sim viewport resolution](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-resolution.png) Display Isaac bounding boxes by selecting "BoundingBox2DLoose" under the icon that resembles a robotic sensor, then click "Show Window". Now we can compare the ground truth with model prediction. -![Isaac Sim sensors](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sensor.png) +![Isaac Sim sensors](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sensor.png) -![Isaac Sim model testing](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-1.png) +![Isaac Sim model testing](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-1.png) -![Isaac Sim model testing](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-2.png) +![Isaac Sim model testing](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-2.png) -![Isaac Sim model testing](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-3.png) +![Isaac Sim model testing](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-EI-3.png) ## Deployment to device and LoRaWAN ### Testing model on device using OpenMV -To get visual verification our model works as intended we can go to Deployment in Edge Impulse Studio, select **OpenMV Firmware** as target and build. +To get visual verification our model works as intended we can go to Deployment in Edge Impulse Studio, select **OpenMV Firmware** as target and build. -![Edge Impulse Studio Deployment OpenMV Firmware](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV_deployment.png) +![Edge Impulse Studio Deployment OpenMV Firmware](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV_deployment.png) Follow [the documentation](https://docs.edgeimpulse.com/docs/run-inference/running-your-impulse-openmv) on how to flash the device and to modify the `ei_object_detection.py` code. Remember to change: `sensor.set_pixformat(sensor.GRAYSCALE)`. The file `edge_impulse_firmware_arduino_portenta.bin` is our firmware for the Arduino Portenta H7 with Vision shield. -![Testing model on device with OpenMV](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV-testing.png) +![Testing model on device with OpenMV](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV-testing.png) ### Deploy model as Arduino compatible library and send inference results to The Things Network with LoRaWAN Start by selecting **Arduino library** as a Deployment target. -![Deploy model as Arduino compatible library](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library.png) +![Deploy model as Arduino compatible library](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library.png) -Once built and downloaded, open Arduino IDE, go to **Sketch > Include Library > Add .zip Library ...** and locate the downloaded library. Next go to **File > Examples > [name of project]_inferencing > portenta_h7 > portenta_h7_camera** to open a generic sketch template using our model. To test the model continuously and print the results to console this sketch is ready to go. The code might appear daunting, but we really only need to focus on the `loop()` function. +Once built and downloaded, open Arduino IDE, go to **Sketch > Include Library > Add .zip Library ...** and locate the downloaded library. Next go to **File > Examples > \[name of project]\_inferencing > portenta\_h7 > portenta\_h7\_camera** to open a generic sketch template using our model. To test the model continuously and print the results to console this sketch is ready to go. The code might appear daunting, but we really only need to focus on the `loop()` function. -![Arduino compatible library example sketch](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library-example.png) +![Arduino compatible library example sketch](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library-example.png) ### Transmit results to The Things Stack sandbox using LoRaWAN Using The Things Stack sandbox (formerly known as The Things Network) we can create a low-power sensor network that allows transmitting device data with minimal energy consumption, long range, and no network fees. Your area may already be covered by a crowd funded network, or you can [create your own](https://www.thethingsnetwork.org/community/) gateway. [Getting started with LoRaWAN](https://www.thethingsindustries.com/docs/getting-started/) is really fun! -![The Things Network](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-map.png) +![The Things Network](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-map.png) Following the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/) on the topic, we create an application in The Things Stack sandbox and register our first device. -![The Things Stack application](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-app.png) +![The Things Stack application](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-app.png) -![The Things Stack device](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-device.png) +![The Things Stack device](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-device.png) Next we will simplify things by merging an example Arduino sketch for transmitting a LoRaWAN message, with the Edge Impulse generated object detection model code. Open the example sketch called `LoraSendAndReceive` included with the MKRWAN(v2) library mentioned in the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/). There is an example of this for you in the [project code repository](https://github.com/eivholt/icicle-monitor/tree/main/portenta-h7/portenta_h7_camera_lora), where you can find an Arduino sketch with the merged code. -![Arduino transmitting inference results over LoRaWAN](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/arduino-lora.png) +![Arduino transmitting inference results over LoRaWAN](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/arduino-lora.png) In short, we perform inference every 10 seconds. If any icicles are detected we simply transmit a binary `1` to the The Things Stack application. It is probably obvious that the binary payload is redundant, the presence of a message is enough, but this could be extended to transmit other data, for example the prediction confidence, number of clusters, battery level, temperature or light level. @@ -456,28 +459,28 @@ if(bb_found) { } ``` -There are a few things to consider in the implementation: The device should enter deep sleep mode and disable/put to sleep all periferals between object detection runs. Default operation of the Portenta H7 with the Vision shield consumes a lot of energy and will drain a battery quickly. To find out how much energy is consumed we can use a device such as the [Otii Arc from Qoitech](https://www.qoitech.com/otii-arc-pro/). Hook up the positive power supply to **VIN**, negative to **GND**. Since VIN bypasses the Portenta power regulator we should provide 5V, however in my setup the Otii Arc is limited to 4.55V. Luckily it seems to be sufficient and we can take some measurements. By connecting the Otii Arc pin RX to the Portenta pin D14/PA9/UART1 TX, in code we can write debug messages to _Serial1_. This is incredibly helpful in determining what power consumption is associated with what part of the code. +There are a few things to consider in the implementation: The device should enter deep sleep mode and disable/put to sleep all periferals between object detection runs. Default operation of the Portenta H7 with the Vision shield consumes a lot of energy and will drain a battery quickly. To find out how much energy is consumed we can use a device such as the [Otii Arc from Qoitech](https://www.qoitech.com/otii-arc-pro/). Hook up the positive power supply to **VIN**, negative to **GND**. Since VIN bypasses the Portenta power regulator we should provide 5V, however in my setup the Otii Arc is limited to 4.55V. Luckily it seems to be sufficient and we can take some measurements. By connecting the Otii Arc pin RX to the Portenta pin D14/PA9/UART1 TX, in code we can write debug messages to _Serial1_. This is incredibly helpful in determining what power consumption is associated with what part of the code. -![Arduino Portenta H7 power specs](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/portenta_h7_power.png) +![Arduino Portenta H7 power specs](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/portenta_h7_power.png) -![Arduino Portenta H7 pin-out](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/portenta_h7_pinout.png) +![Arduino Portenta H7 pin-out](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/portenta_h7_pinout.png) -![Otii Arc hook-up](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/otii-arc-portenta.png) +![Otii Arc hook-up](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/otii-arc-portenta.png) -![Otii Arc power profile](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/otii-icicle-profile.png) +![Otii Arc power profile](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/otii-icicle-profile.png) As we can see the highlighted section should be optimized for minimal power consumption. This is a complicated subject, especially on a [complex board such as the Arduino Portenta H7](https://github.com/arduino/ArduinoCore-mbed/issues/619) but there are some examples for general guidance: - - [Snow monitor](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-power-profiling-16) - - [Mail box sensor](https://community.element14.com/challenges-projects/project14/rf/b/blog/posts/got-mail-lorawan-mail-box-sensor). +* [Snow monitor](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-power-profiling-16) +* [Mail box sensor](https://community.element14.com/challenges-projects/project14/rf/b/blog/posts/got-mail-lorawan-mail-box-sensor). The project code presented here runs inference on an image every 10 seconds. However, this is for demonstration purposes and in a deployment should be much less frequent, like once per hour during daylight. Have a look at this project for an example of how to [remotely control inference interval](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-lora-application-14) via LoRaWAN downlink message. This could be further controlled automatically via an application that has access to an [API for daylight data](https://developer.yr.no/doc/GettingStarted/). -![YR weather API](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/yr-sun.png) +![YR weather API](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/yr-sun.png) Next, in the The Things Stack application we need to define a function that will be used to decode the byte into a JSON structure that is easier to interpet when we pass the message further up the chain of services. The function can be found in the [project code repository](https://github.com/eivholt/icicle-monitor/blob/main/TheThingsStack/decoder.js). -![The Things Stack decoder](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-decoder.png) +![The Things Stack decoder](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-decoder.png) ```javascript function Decoder(bytes, port) { @@ -498,7 +501,7 @@ function Decoder(bytes, port) { Now we can observe messages being received and decoded in **Live data** in the TTS console. -![The Things Stack live data](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-data.png) +![The Things Stack live data](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-data.png) An integral part of The Things Stack is an MQTT message broker. At this point we can use [any MQTT client to subscribe to topics](https://www.thethingsindustries.com/docs/integrations/mqtt/mqtt-clients/) and create any suitable notification system for the end user. The following is an MQTT client written in Python to demonstrate the principle. Note that the library `paho-mqtt` has been used in a way so that it will block the program execution until two messages have been received. Then it will print the topic and payloads. In a real implementation, it would be better to register a callback and perform some action for each message received. @@ -522,7 +525,7 @@ v3/icicle-monitor@ttn/devices/portenta-h7-icicle-00/up {"end_device_ids":{"device_id":"portenta-h7-icicle-00","application_ids":{"application_id":"icicle-monitor"},"dev_eui":"3036363266398F0D","join_eui":"0000000000000000"},"correlation_ids":["as:up:01HSKMTN7F60CC3BQXE06B3Q4X","rpc:/ttn.lorawan.v3.AppAs/SimulateUplink:17b97b44-a5cd-45f0-9439-2de42e187300"],"received_at":"2024-03-22T17:55:05.070404295Z","uplink_message":{"f_port":1,"frm_payload":"AQ==","decoded_payload":{"detected":true},"rx_metadata":[{"gateway_ids":{"gateway_id":"test"},"rssi":42,"channel_rssi":42,"snr":4.2}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"frequency":"868000000"},"locations":{"user":{"latitude":67.2951772015745,"longitude":14.43232297897339,"altitude":13,"source":"SOURCE_REGISTRY"}}},"simulated":true}' ``` -Observe the difference in the real uplink (first) and simulated uplink (second). In both we find "decoded_payload":{"detected":true}. +Observe the difference in the real uplink (first) and simulated uplink (second). In both we find "decoded\_payload":{"detected":true}. TTS has a range of [integration options](https://www.thethingsindustries.com/docs/integrations/) for specific platforms, or you could set up a [custom webhook using a standard HTTP/REST](https://www.thethingsindustries.com/docs/integrations/webhooks/) mechanism. @@ -536,29 +539,29 @@ For permanent outdoor installation the device requires a properly sealed enclosu The project has no safe-guard against false negatives. The device will not report if it's view is blocked. This could be resolved by placing static markers on both sides of an area to monitor and included in synthetic training data. Absence of at least one marker could trigger a notification that the view is obscured. -![Markers to avoid false negatives](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/marker.png) +![Markers to avoid false negatives](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/marker.png) ### Object scale Due to optimization techniques in Faster Objects - More Objects (FoMo) determining relative sizes of the icicles is not feasible. As even icicles with small mass can be harmful at moderate elevation this is not a crucial feature. -![Object scale](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/object-scale.png) +![Object scale](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/object-scale.png) ### Exact number of icicles The object detection model has not been trained to give an exact number of icicles in view. This has no practical implication other than the model verification results appearing worse than practical performance. -![Icicle grouping](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/grouping.png) +![Icicle grouping](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/grouping.png) ### Non-vertical icicles and snow Icicles can appear bent or angled either due to wind or more commonly due to ice and snow masses slowly dropping over roof edges. The dataset generated in this project does not cover this, but it would not take a lot of effort to extend the domain randomization to rotate or warp the icicles. -![AULSSON_EBBA](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/AULSSON_EBBA.png) +![AULSSON\_EBBA](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/AULSSON_EBBA.png) -![Martin Cathrae](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Martin-Cathrae.png) +![Martin Cathrae](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Martin-Cathrae.png) -The training images could benefit from simulating snow with particle effects in Omniverse. The project could also be extended to detect build-up of snow on roofs. For inspiration check out this demo of simulated snow dynamic made in 2014 by Walt Disney Animation Studios for the movie Frozen: +The training images could benefit from simulating snow with particle effects in Omniverse. The project could also be extended to detect build-up of snow on roofs. For inspiration check out this demo of simulated snow dynamic made in 2014 by Walt Disney Animation Studios for the movie Frozen: {% embed url="https://youtu.be/9H1gRQ6S7gg" %} @@ -566,9 +569,8 @@ The training images could benefit from simulating snow with particle effects in To be able to compile a representation of our neural network and have it run on the severely limited amount of RAM available on the Arduino Portenta H7, pixel representation has been limited to a single channel - grayscale. Colors are not needed to detect icicles so this does not affect the results. -![Grayscale](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/grayscale1.png) +![Grayscale](../../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/grayscale1.png) ## Further reading Insights into [how icicles are formed](https://www.insidescience.org/news/riddles-rippled-icicle). - diff --git a/image-projects/surgery-inventory-synthetic-data.md b/readme/featured-machine-learning-projects/surgery-inventory-synthetic-data.md similarity index 80% rename from image-projects/surgery-inventory-synthetic-data.md rename to readme/featured-machine-learning-projects/surgery-inventory-synthetic-data.md index 361af37f..a2194668 100644 --- a/image-projects/surgery-inventory-synthetic-data.md +++ b/readme/featured-machine-learning-projects/surgery-inventory-synthetic-data.md @@ -1,6 +1,7 @@ --- description: >- - A wearable surgery inventory object detection sensor, trained with synthetic data created using NVIDIA Omniverse Replicator. + A wearable surgery inventory object detection sensor, trained with synthetic + data created using NVIDIA Omniverse Replicator. --- # Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator @@ -11,7 +12,7 @@ Public Project Link: [https://studio.edgeimpulse.com/public/322153/latest](https GitHub Repo: [https://github.com/eivholt/surgery-inventory-synthetic-data](https://github.com/eivholt/surgery-inventory-synthetic-data) -![](../.gitbook/assets/surgery-inventory-synthetic-data/render2.png) +![](../../.gitbook/assets/surgery-inventory-synthetic-data/render2.png) ## Introduction @@ -23,13 +24,13 @@ Here is a demo of chrome surgical instrument detection running on an Arduino Nic {% embed url="https://www.youtube.com/watch?v=1k0pfPwzTw4" %} -In the following drawing we see how equipment and disposable materials are typically organized during surgery. Tools are pre-packaged in sets for the appropriate type of surgery and noted when organized on trays or tables. Swabs are packaged in numbers and contain tags that are noted and kept safe. When swabs are used they are displayed individually in transparent pockets on a stand so they can be counted and checked with the tags from the originating package. Extensive routines are in place to continuously count all equipment used; still errors occur [an estimated rate between 0.3 and 1 per 1000 abdominal operations](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5320916/#:~:text=The%20incidence%20of%20RSB%20is,the%20abdomen%2C%20retroperitoneum%20and%20pelvis.). +In the following drawing we see how equipment and disposable materials are typically organized during surgery. Tools are pre-packaged in sets for the appropriate type of surgery and noted when organized on trays or tables. Swabs are packaged in numbers and contain tags that are noted and kept safe. When swabs are used they are displayed individually in transparent pockets on a stand so they can be counted and checked with the tags from the originating package. Extensive routines are in place to continuously count all equipment used; still errors occur [an estimated rate between 0.3 and 1 per 1000 abdominal operations](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5320916/). -![Operation room, sketch Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/surgery_inventory_concept.jpg) +![Operation room, sketch Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/surgery_inventory_concept.jpg) Existing solutions are mainly based on either x-ray or RFID. With x-ray, the patient needs to be scanned using a scanner on wheels. Metal objects obviously will be visible, while other items such as swabs needs to have metal strips woven to be detected and the surgery team has to wear lead aprons. Some items have passive RFID-circuits embedded and can be detected by a handheld scanner. -![Swabs with metallic strip, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/swabxray.jpg) +![Swabs with metallic strip, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/swabxray.jpg) ## Hardware used: @@ -60,7 +61,7 @@ Many operating rooms (OR) are equiped with adjustable lights with a camera embed As if detecting objects on highly constrained devices wasn't challenging enough, this use case poses a potential show stopper. Most of the tools used in surgery have a chrome surface. Due to the reflective properties of chrome, especially the high specular reflection and highlights, a given item's features will vary highly judged by its composition of pixels, in this context known as features. Humans are pretty good at interpreting highly reflective objects, but there are many examples where even we may get confused. -![Optical illusion, photo acmedoge](../.gitbook/assets/surgery-inventory-synthetic-data/perspective-reflections-optical-illusions-11.jpg) +![Optical illusion, photo acmedoge](../../.gitbook/assets/surgery-inventory-synthetic-data/perspective-reflections-optical-illusions-11.jpg) ### Number of objects and classes @@ -78,13 +79,13 @@ The model trained on this data was quickly deemed useless in detecting reflectiv To isolate the chrome surfaces as a problematic issue, a number of chrome instruments were spray painted matte and a few plastic and cloth based items were used to make a new manually captured and labeled dataset of the same size. For each image the items were scattered and the camera angle varied. -![Matte vs. reflective surfaces, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/surgery_tools_matte_chrome.jpg) +![Matte vs. reflective surfaces, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/surgery_tools_matte_chrome.jpg) -![Matte objects capture, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/matte_label.png) +![Matte objects capture, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/matte_label.png) This model worked great and can be inspected [here](https://studio.edgeimpulse.com/public/94601/latest). -![Matte objects performance, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/matte_performance.png) +![Matte objects performance, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/matte_performance.png) A video demonstrating live inference from the device camera can be seen here. Only trained objects are marked. Flickering can be mitigated by averaging. @@ -125,17 +126,17 @@ We will be walking through the following steps to create and run an object detec NVIDIA Omniverse * Install [Omniverse from NVIDIA](https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/getting_started.html). -* Install Code: Open Omniverse Launcher, go to Exchange, install Code. +* Install Code: Open Omniverse Launcher, go to Exchange, install Code. -![NVIDIA Omniverse Code](../.gitbook/assets/surgery-inventory-synthetic-data/omni-code.png) +![NVIDIA Omniverse Code](../../.gitbook/assets/surgery-inventory-synthetic-data/omni-code.png) -* Launch Code from NVIDIA Omniverse Launcher. +* Launch Code from NVIDIA Omniverse Launcher. -![NVIDIA Omniverse Launcher](../.gitbook/assets/surgery-inventory-synthetic-data/omni-code-launch.png) +![NVIDIA Omniverse Launcher](../../.gitbook/assets/surgery-inventory-synthetic-data/omni-code-launch.png) * Go to Window->Extensions and install Replicator -![NVIDIA Omniverse Replicator](../.gitbook/assets/surgery-inventory-synthetic-data/replicator_exten.png) +![NVIDIA Omniverse Replicator](../../.gitbook/assets/surgery-inventory-synthetic-data/replicator_exten.png) * Install [Embedded VS Code for NVIDIA Omniverse](https://github.com/Toni-SM/semu.misc.vscode) @@ -146,21 +147,21 @@ NVIDIA Omniverse * Create a larger textured plane to fill the background * Add some lights -![Create stage, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/omniverse_stage.png) +![Create stage, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/omniverse_stage.png) -If you have a hefty heat producing GPU next to you, you might prefer to reduce the FPS limit in the viewports of Code. It may default to 120 FPS, generating a lot of heat when the viewport is in the highest quality rendering modes. Set "UI FPS Limit" and "Present thread FPS Limit" to 60. This setting unfortunately does not persist between sessions, so we have to repeat this everytime projects open. +If you have a hefty heat producing GPU next to you, you might prefer to reduce the FPS limit in the viewports of Code. It may default to 120 FPS, generating a lot of heat when the viewport is in the highest quality rendering modes. Set "UI FPS Limit" and "Present thread FPS Limit" to 60. This setting unfortunately does not persist between sessions, so we have to repeat this everytime projects open. -![FPS Limit](../.gitbook/assets/surgery-inventory-synthetic-data/Code-fps-limit.png) +![FPS Limit](../../.gitbook/assets/surgery-inventory-synthetic-data/Code-fps-limit.png) ## Working with 3D models in Blender The objects we want to be able to detect need to be represented with a 3D model and a surface (material). Omniverse provides a library of ready-to-import assets, further models can be created using editors such as Blender or purchased on sites such as [Turbo Squid](https://www.turbosquid.com/). -![Exporting model in Blender, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/blender.png) +![Exporting model in Blender, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/blender.png) A scene containing multiple geometric models should be exported on an individual model basis, with USD-format as output. -![Exporting model in Blender, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/blender_export.png) +![Exporting model in Blender, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/blender_export.png) Omniverse has recently received limited support in importing BSDF material compositions, but this is still experimental. In this project materials or textures were not imported directly. @@ -168,37 +169,37 @@ Omniverse has recently received limited support in importing BSDF material compo To avoid overwriting any custom scaling or other transformations set on exported models it is advisable to add a top node of type Xform on each model hierarchy. Later we can move the object around without loosing adjustments. -![Importing 3D model in Omniverse, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/scalpel.png) +![Importing 3D model in Omniverse, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/scalpel.png) The replicator toolbox has a function for scattering objects on a surface in it's API. To (mostly) avoid object intersection a few improvements can be made. In the screenshot a basic shape has been added as a bounding box to allow some clearance between objects and to make sure thin objects are more appropriately handled while scattering. The bounding box can be set as invisible. As of Replicator 1.9.8 some object overlap seems to be unavoidable. -![Bounding box, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/bounding_box.png) +![Bounding box, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/bounding_box.png) For the chrome surfaces a material from one of the models from the library provided through Omniverse was reused, look for [http://omniverse-content-production.s3-us-west-2.amazonaws.com/Materials/Base/Metals/Chrome/](http://omniverse-content-production.s3-us-west-2.amazonaws.com/Materials/Base/Metals/Chrome/) in the Omniverse Asset Store. Remember to switch to "RTX - Interactive" rendering mode to see representative ray-tracing results, "RTX - Real-Time" is a simplified rendering pipeline. -![Chrome material, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/material_chrome.png) +![Chrome material, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/material_chrome.png) For the cloth based materials some of the textures from the original models were used, more effort in setting up the shaders with appropriate texture maps could improve the results. -![Cloth material, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/material_swab.png) +![Cloth material, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/material_swab.png) -![Synthetic image generation](../.gitbook/assets/surgery-inventory-synthetic-data/render1.png) +![Synthetic image generation](../../.gitbook/assets/surgery-inventory-synthetic-data/render1.png) ## Setting metadata on objects To be able to produce images for training and include labels we can use a feature of Replicator toolbox found under menu Replicator->Semantics Schema Editor. -![Semantics Schema Editor, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/semantics_schema_editor.png) +![Semantics Schema Editor, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/semantics_schema_editor.png) Here we can select each top node representing an item for object detection and adding a key-value pair. Choosing "class" as Semantic Type and e.g. "tweezers" as Semantic Data enables us to export these strings as labels later. The UI could benefit from a bit more exploration in intuitive design, as it is easy to misinterpret what fields shows the actual semantic data set on an item, an what fields carry over intended to make labeling many consecutive items easier. -![Semantics Schema Editor suggestion, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/semantics_schema_editor_suggestion.png) +![Semantics Schema Editor suggestion, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/semantics_schema_editor_suggestion.png) Semantics Schema Editor may also be used with multiple items selected. It also has handy features to use the names of the nodes for automatic naming. ## Creating script for domain randomization -This part describes how to write a script in Python for randomizing the images we will produce. We could choose to start with an empty stage and programatically load models (from USD-files), lights, cameras and such. With a limited number of models and lights we will proceed with adding most items to the stage manually as described earlier. Our script can be named anything, ending in .py and preferably placed close to the stage USD-file. The following is a description of [such a script replicator_init.py](https://github.com/eivholt/surgery-inventory-synthetic-data/blob/main/omniverse-replicator/replicator_init.py): +This part describes how to write a script in Python for randomizing the images we will produce. We could choose to start with an empty stage and programatically load models (from USD-files), lights, cameras and such. With a limited number of models and lights we will proceed with adding most items to the stage manually as described earlier. Our script can be named anything, ending in .py and preferably placed close to the stage USD-file. The following is a description of [such a script replicator\_init.py](https://github.com/eivholt/surgery-inventory-synthetic-data/blob/main/omniverse-replicator/replicator_init.py): To keep the items generated in our script separate from the manually created content we start by creating a new layer in the 3D stage: @@ -258,9 +259,9 @@ Here `rgb` tells the API that we want the images to be written to disk as png-fi The bounding boxes can be visualized by clicking the sensor widget, checking "BoundingBox2DTight" and finally "Show Window". -![Bounding Boxes, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/omni_sensor.png) +![Bounding Boxes, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/omni_sensor.png) -![Bounding Boxes, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/omniverse_2dbb.png) +![Bounding Boxes, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/omniverse_2dbb.png) Only thing missing is defining the randomization logic: @@ -316,12 +317,13 @@ Whether the best approach for training data is to keep objects of interest and b Edge Impulse Studio supports a wide range of image labeling formats for object detection. Unfortunately the output from Replicator's BasicWriter needs to be transformed so it can be uploaded either through the web interface or via [web-API](https://docs.edgeimpulse.com/reference/ingestion-api#ingestion-api). -Provided is a simple Python program, [basic_writer_to_pascal_voc.py](https://github.com/eivholt/surgery-inventory-synthetic-data/blob/main/omniverse-replicator/basic_writer_to_pascal_voc.py). A simple prompt was written for ChatGPT describing the output from Replicator and the [desired results described at EI](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from shell with +Provided is a simple Python program, [basic\_writer\_to\_pascal\_voc.py](https://github.com/eivholt/surgery-inventory-synthetic-data/blob/main/omniverse-replicator/basic_writer_to_pascal_voc.py). A simple prompt was written for ChatGPT describing the output from Replicator and the [desired results described at EI](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from shell with -``` +``` python basic_writer_to_pascal_voc.py ``` -or debug from Visual Studio Code by setting input folder in `launch.json` like this: + +or debug from Visual Studio Code by setting input folder in `launch.json` like this: ``` "args": ["../out"] @@ -333,56 +335,57 @@ This will create a file `bounding_boxes.labels` that contains all labels and bou Look at the [provided object detection Edge Impulse project](https://studio.edgeimpulse.com/public/322153/latest) or [follow a guide to create a new object detection project](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/object-detection/fomo-object-detection-for-constrained-devices#how-to-get-started). -For a project intended to detect objects with reflective surfaces a large number of images is needed for training, but the exact number depends on a lot of factors and some experimentation should be expected. It is advisable to start relatively small, say 1,000 images of the objects to be detected. For this project over **30,000** images were generated; this is much more than needed. A number of images of random background items are also needed to produce results that will work in the real world. This project uses other surgery equipment for convenience, they do not need to be individually labeled. Still Edge Impulse Studio will create a labeling queue for each image for which it has not received labeling data. To avoid having to click through each image to confirm they contain no labels, the program described will produce a bounding_boxes.labels with empty labels for items tagged with semantic class "background". The factor between images of items to detect and background noise also relies on experimentation, but 1-2% background ratio seems to be a good starting point. +For a project intended to detect objects with reflective surfaces a large number of images is needed for training, but the exact number depends on a lot of factors and some experimentation should be expected. It is advisable to start relatively small, say 1,000 images of the objects to be detected. For this project over **30,000** images were generated; this is much more than needed. A number of images of random background items are also needed to produce results that will work in the real world. This project uses other surgery equipment for convenience, they do not need to be individually labeled. Still Edge Impulse Studio will create a labeling queue for each image for which it has not received labeling data. To avoid having to click through each image to confirm they contain no labels, the program described will produce a bounding\_boxes.labels with empty labels for items tagged with semantic class "background". The factor between images of items to detect and background noise also relies on experimentation, but 1-2% background ratio seems to be a good starting point. EI creates unique identifiers per image, so you can run multiple iterations to create and upload new datasets, even with the same file names. Just upload all the images from a batch together with the `bounding_boxes.labels` file. This way we can effortlessly produce thousands of labeled images and witness how performance on detecting reflective objects increases. Keep in mind to try to balance the number of labels for each class. -![Data acquisition, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/dataset_1.png) +![Data acquisition, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/dataset_1.png) Note that EI has created a nice [GUI-based extension](https://github.com/edgeimpulse/edge-impulse-omniverse-ext/tree/main) for uploading images from Replicator directly to your project. As of the time of writing this extension only uploads images, but this might include label data in the near future. -![Data acquisition](../.gitbook/assets/surgery-inventory-synthetic-data/ei_extension.png) +![Data acquisition](../../.gitbook/assets/surgery-inventory-synthetic-data/ei_extension.png) ## Training and deploying model to device Finally we can [design and train our object detection model](https://docs.edgeimpulse.com/docs/tutorials/end-to-end-tutorials/object-detection/detect-objects-using-fomo#3.-designing-an-impulse). Target device should be set and we need to remember that in the case of Arduino Nicla Vision we only have enough RAM for 96x96 pixels. Any type of "early stop" feature would be nice, but for now we need to experiment with the number of training cycles. Data augmentation should be avoided in the case where we generate thousands of images, it will not improve our results. -![Model performance, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/performance_synth.png) +![Model performance, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/performance_synth.png) ## 3D printing a protective housing for the device + To protect the device and make a simple way to wear it a housing was designed in CAD and 3D printed. It is a good practice to start by making basic 3D representations of all the components, this vastly reduces iterations due to surprises when it comes to assembly. -![CAD components, render Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/cad-components.png) +![CAD components, render Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/cad-components.png) -![CAD housing, render Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/cad-housing.png) +![CAD housing, render Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/cad-housing.png) -![Device housing, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/final3.jpg) +![Device housing, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/final3.jpg) -![Device housing, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/final2.jpg) +![Device housing, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/final2.jpg) -![Device housing, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/final1.jpg) +![Device housing, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/final1.jpg) ## Using object detection model in an application A trained model can be compiled into an Arduino-compatible library. Events can trigger inference and the Arduino Nicla Vision can broadcast any number of detected objects via a Bluetooth LE service. A BLE dongle or smart phone can listen for events and route them to a web-API for further integration with other systems. For instance this application can log the detected items in an Electronic Medical Record system. The e-health standard [HL7 FHIR allows for defining devices and materials used during procedures](https://build.fhir.org/device.html). Sandbox environments such as [Open DIPS](https://open.dips.no/) are great places to start experimenting with integrations with hospital systems. -![Open DIPS](../.gitbook/assets/surgery-inventory-synthetic-data/opendips.png) +![Open DIPS](../../.gitbook/assets/surgery-inventory-synthetic-data/opendips.png) ## Results The results of this project show that training and testing data for object detection models can be synthesized using 3D models, reducing manual labor in capturing images and annotation. Even more impressive is being able to detect unpredictable reflective surfaces on heavily constrained hardware by creating a large number of images. -![Chrome objects detection, photo Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/chrome_vid.png) +![Chrome objects detection, photo Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/chrome_vid.png) ## Conclusion The domain of visual object detection is currently experiencing a thrilling phase of evolution, thanks to the convergence of numerous significant advancements. Envision a service capable of accepting 3D models as input and generating a diverse array of images for training purposes. With the continuous improvements in generative diffusion models, particularly in the realm of text-to-3D conversion, we are on the cusp of unlocking even more potent capabilities for creating synthetic training data. This progression is not just a technological leap; it's set to revolutionize the way we approach object detection, paving the way for a new generation of highly innovative and effective object detection solutions. The implications of these advancements are vast, opening doors to unprecedented levels of accuracy and efficiency in various applications. -![Text-to-3D](../.gitbook/assets/surgery-inventory-synthetic-data/diagram.jpg) +![Text-to-3D](../../.gitbook/assets/surgery-inventory-synthetic-data/diagram.jpg) -Further reading: [How to Train an Object Detection Model for Visual Inspection with Synthetic Data](https://developer.nvidia.com/blog/how-to-train-an-object-detection-model-for-visual-inspection-with-synthetic-data/#) +Further reading: [How to Train an Object Detection Model for Visual Inspection with Synthetic Data](https://developer.nvidia.com/blog/how-to-train-an-object-detection-model-for-visual-inspection-with-synthetic-data/) ## Appendix @@ -391,45 +394,45 @@ I highly recommend learning how to debug Omniverse extension code. It requires a * To enable Python debugging via Visual Studio Code, in Omniverse Code, go to Extensions. * Search for "debug" and enable "Kit debug vscode" and "A debugger for Python". -![NVIDIA Omniverse Code debug python](../.gitbook/assets/surgery-inventory-synthetic-data/omni-code-debug-extensions.png) +![NVIDIA Omniverse Code debug python](../../.gitbook/assets/surgery-inventory-synthetic-data/omni-code-debug-extensions.png) -* In Code, the window "VS Code Link" should read "VS Code Debugger Unattached". +* In Code, the window "VS Code Link" should read "VS Code Debugger Unattached". -![VS Code Link](../.gitbook/assets/surgery-inventory-synthetic-data/Code-debug-ready.png) +![VS Code Link](../../.gitbook/assets/surgery-inventory-synthetic-data/Code-debug-ready.png) -* After activating the project extension, go to the extension details and click "Open in VSCode" icon. +* After activating the project extension, go to the extension details and click "Open in VSCode" icon. -![Open in VSCode](../.gitbook/assets/surgery-inventory-synthetic-data/Code-open-VSCode.png) +![Open in VSCode](../../.gitbook/assets/surgery-inventory-synthetic-data/Code-open-VSCode.png) -* In Visual Studio Code, make sure in `.vscode\launch.json` the two settings corresponds to what you see in the "VS Code Link" window, e.g. "host": "localhost", and "port": 3000. +* In Visual Studio Code, make sure in `.vscode\launch.json` the two settings corresponds to what you see in the "VS Code Link" window, e.g. "host": "localhost", and "port": 3000. -![VSCode launch settings](../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-launch.png) +![VSCode launch settings](../../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-launch.png) -* Go to the Run and Debug pane in VSCode, make sure "Python: Attach .." is selected and press the play button. +* Go to the Run and Debug pane in VSCode, make sure "Python: Attach .." is selected and press the play button. -![VSCode Run and Debug Attach](../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-Attach.png) +![VSCode Run and Debug Attach](../../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-Attach.png) -* Back in Omniverse Code, VS Code Link should read "VS Code Debugger Attached". +* Back in Omniverse Code, VS Code Link should read "VS Code Debugger Attached". -![NVIDIA Omniverse Code Debugger Attached](../.gitbook/assets/surgery-inventory-synthetic-data/Code-debug-attached.png) +![NVIDIA Omniverse Code Debugger Attached](../../.gitbook/assets/surgery-inventory-synthetic-data/Code-debug-attached.png) -* To test, in VSCode set a breakpoint in `exts\eivholt\extension.py`, e.g. inside the function "run_replicator". +* To test, in VSCode set a breakpoint in `exts\eivholt\extension.py`, e.g. inside the function "run\_replicator". -![VSCode Breakpoint 1](../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-debug-extension1.png) +![VSCode Breakpoint 1](../../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-debug-extension1.png) -* Back in Omniverse Code, find the project extension UI and click "Initialize Replicator". +* Back in Omniverse Code, find the project extension UI and click "Initialize Replicator". -![Omniverse Code Extension](../.gitbook/assets/surgery-inventory-synthetic-data/Code-extension.png) +![Omniverse Code Extension](../../.gitbook/assets/surgery-inventory-synthetic-data/Code-extension.png) -* In VSCode, you should now have hit the breakpoint. +* In VSCode, you should now have hit the breakpoint. -![VSCode Breakpoint 2](../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-debug-extension2.png) +![VSCode Breakpoint 2](../../.gitbook/assets/surgery-inventory-synthetic-data/VSCode-debug-extension2.png) ## Edge detection Another interesting approach to the challenge of detecting reflective surfaces is using edge detection. This would still benefit from synthetic data generation. -![Edge detection, render Eivind Holt](../.gitbook/assets/surgery-inventory-synthetic-data/edgedetection.png) +![Edge detection, render Eivind Holt](../../.gitbook/assets/surgery-inventory-synthetic-data/edgedetection.png) ## Disclosure diff --git a/image-projects/traffic-analysis-tao-jetson-orin.md b/readme/featured-machine-learning-projects/traffic-analysis-tao-jetson-orin.md similarity index 84% rename from image-projects/traffic-analysis-tao-jetson-orin.md rename to readme/featured-machine-learning-projects/traffic-analysis-tao-jetson-orin.md index a11bc55e..491fbbba 100644 --- a/image-projects/traffic-analysis-tao-jetson-orin.md +++ b/readme/featured-machine-learning-projects/traffic-analysis-tao-jetson-orin.md @@ -1,6 +1,7 @@ --- description: >- - Perform traffic analysis for smart city and vehicle detection projects with an NVIDIA TAO model and a Jetson Orin Nano. + Perform traffic analysis for smart city and vehicle detection projects with an + NVIDIA TAO model and a Jetson Orin Nano. --- # Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano @@ -9,9 +10,9 @@ Created By: Jallson Suryo Public Project Link: [https://studio.edgeimpulse.com/public/310628/live](https://studio.edgeimpulse.com/public/310628/live) -GitHub Repo: [https://github.com/Jallson/Traffic_Analysis_Orin_Nano/](https://github.com/Jallson/Traffic_Analysis_Orin_Nano) +GitHub Repo: [https://github.com/Jallson/Traffic\_Analysis\_Orin\_Nano/](https://github.com/Jallson/Traffic_Analysis_Orin_Nano) -![](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo01.png) +![](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo01.png) ## Problem Statement @@ -21,26 +22,26 @@ In a smart-city system, analyzing vehicle and traffic flow patterns is crucial f An object detection model from Edge Impulse is one way of addressing this problem, as model inference output will contains data labels, object coordinates, and timestamps. From this data, we will derive the object's speed and direction, as well as count objects entering or exiting. To simplify the process, we will use an NVIDIA TAO - YOLOv4 pre-trained neural network to build our model, then deploy on to an NVIDIA Jetson Orin Nano. This method grants access to a wide range of pre-trained models, enabling you to leverage existing neural network architectures and weights for your specific tasks. Therefore, the amount of data we need to collect is less than what's typically required when training and building an object detection model from scratch. The Edge Impulse model, combined with NVIDIA TAO, are optimized for efficient performance, achieving faster inference speeds through the Tensor RT library embedded in Orin Nano, which is essential for real-time applications. Overall, this approach can greatly accelerate the development cycle, enhance model performance, and streamline the process for Edge AI applications. -![](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo02.png) +![](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo02.png) ### Hardware Requirements -- NVIDIA Jetson Orin Nano Developer Kit (8GB) -- USB Camera/webcam (eg. Logitech C270/ C920) -- DisplayPort to HDMI cable -- Display/monitor -- Tripod -- Keyboard, mouse or PC/Laptop via ssh -- Orin Nano case ( 3D print file available at [https://www.thingiverse.com/thing:6068997](https://www.thingiverse.com/thing:6068997) ) +* NVIDIA Jetson Orin Nano Developer Kit (8GB) +* USB Camera/webcam (eg. Logitech C270/ C920) +* DisplayPort to HDMI cable +* Display/monitor +* Tripod +* Keyboard, mouse or PC/Laptop via ssh +* Orin Nano case ( 3D print file available at [https://www.thingiverse.com/thing:6068997](https://www.thingiverse.com/thing:6068997) ) -![Hardware](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo03.png) +![Hardware](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo03.png) ### Software & Online Services -- NVIDIA Jetpack (5.1.2) -- Edge Impulse Studio -- Edge Impulse Linux CLI & Python SDK -- Terminal +* NVIDIA Jetpack (5.1.2) +* Edge Impulse Studio +* Edge Impulse Linux CLI & Python SDK +* Terminal ## Steps @@ -50,11 +51,11 @@ In the initial stage of building a model in Edge Impulse Studio, we need to prep > Note: When collecting data samples, it's important to remember that the images of vehicles (trucks or cars) to be labeled should not be too small, as the model we're building can only recognize objects with a minimum size of 32x32 pixels. -![Collect_data](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo04.png) +![Collect\_data](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo04.png) -![Upload_COCO-json](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo05.png) +![Upload\_COCO-json](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo05.png) -![Upload_video](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo06.png) +![Upload\_video](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo06.png) ### 2. Labeling @@ -64,11 +65,11 @@ Once the images are ready, you'll see a labeling queue, and you can begin the pr After labeling, it's recommended to split the data into Training and Testing sets, using around an 80/20 ratio. If you haven't done this yet, you can go back to the Dashboard, and click on _Train / Test Split_ and proceed. As shown here, I only used 150 images, as we'll be training the model with the help of pre-trained NVIDIA TAO-YOLO based models. -![Split_into_image](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo07.png) +![Split\_into\_image](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo07.png) -![Labeling_with_Yolo](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo08.png) +![Labeling\_with\_Yolo](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo08.png) -![Train_and_Test](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo09.png) +![Train\_and\_Test](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo09.png) ### 3. Train and Build Model @@ -78,17 +79,17 @@ Now, move to the _Object Detection_ navigation on the left, and configure the tr If everything goes well and the precision result is around 80%, proceed to the next step. Go to the _Model Testing_ section, click _Classify all_, and if the result is around 90%, you can move on to the final step — Deployment. -![Learning_blocks](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo10.png) +![Learning\_blocks](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo10.png) -![Save_parameters](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo11.png) +![Save\_parameters](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo11.png) -![Generate_features](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo12.png) +![Generate\_features](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo12.png) -![NN_setting_and_result](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo13.png) +![NN\_setting\_and\_result](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo13.png) -![Live_classification](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo13a.png) +![Live\_classification](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo13a.png) -![Model_test](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo14.png) +![Model\_test](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo14.png) ### 4. Deploy Model Targeting Jetson Orin Nano GPU @@ -96,7 +97,7 @@ Click on the _Deployment_ tab, then search for **TensorRT**, select _(Unoptimize Alternatively, there's an easier method: simply ensure that the model has been built in Edge Impulse Studio. From there, you can test, download the model, and run everything directly from the Orin Nano. -![TensorRT_build_library](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo15.png) +![TensorRT\_build\_library](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo15.png) On the Orin Nano side, there are several things that need to be done. Make sure the unit uses JetPack — we use Jetpack v5.1.2 — which is usually pre-installed on the SD card. Then open a Terminal on the Orin Nano, or ssh to the Orin via your PC/laptop and setup Edge Impulse tooling in the terminal. @@ -136,11 +137,11 @@ Open a terminal on the Orin Nano or ssh from your PC/laptop then run `edge-impul For convenience, you can copy this file to the same directory as the Python program you'll be creating in the next steps. For instance, you can use the following command to copy it to the home directory: `cp -v model.eim /home/orin` -![Check_progress](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo16.png) +![Check\_progress](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo16.png) Now the model is ready to run in a high-level language such as the Python program used in the next step. To ensure this model works, we can run the Edge Impulse Linux Runner with a camera attached to the Orin Nano. You can see a view from the camera via your browser (the IP address location is provided when the Edge Impulse Linux Runner is started). Run this command to start it now: `edge-impulse-linux-runner --model-file /model.eim` -![Live_stream](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Video01.gif) +![Live\_stream](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Video01.gif) The inferencing time is around 6ms, which is incredibly fast for object detection projects. @@ -164,13 +165,13 @@ python3 traffic2.py /model.eim /video.mp4 > Note: For video/camera capture display, you cannot use the headless method from a PC/laptop. Instead, connect a monitor directly to the Orin Nano to view the visuals, including the lines, labeled bounding boxes, IN and OUT counts, and vehicle speeds. -![Python_code](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo17.png) +![Python\_code](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo17.png) -![Camera_feed](../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo18.png) +![Camera\_feed](../../.gitbook/assets/traffic-analysis-tao-jetson-orin/Photo18.png) -The Python code and the tracking library is avaialable and can be accessed at [https://github.com/Jallson/Traffic_Analysis_Orin_Nano](https://github.com/Jallson/Traffic_Analysis_Orin_Nano) +The Python code and the tracking library is avaialable and can be accessed at [https://github.com/Jallson/Traffic\_Analysis\_Orin\_Nano](https://github.com/Jallson/Traffic_Analysis_Orin_Nano) -Here are two demo videos, showing the results: +Here are two demo videos, showing the results: {% embed url="https://youtu.be/rRZKyNIsXXA" %} diff --git a/readme/prototype-and-concept-projects/README.md b/readme/prototype-and-concept-projects/README.md new file mode 100644 index 00000000..eb1c64ac --- /dev/null +++ b/readme/prototype-and-concept-projects/README.md @@ -0,0 +1,2 @@ +# Prototype and Concept Projects + diff --git a/experiments-getting-started.md b/readme/prototype-and-concept-projects/experiments-getting-started.md similarity index 66% rename from experiments-getting-started.md rename to readme/prototype-and-concept-projects/experiments-getting-started.md index 1c1d0481..637dad38 100644 --- a/experiments-getting-started.md +++ b/readme/prototype-and-concept-projects/experiments-getting-started.md @@ -1,16 +1,16 @@ --- -description: Discover how to use the Experiments feature to test and improve machine learning model accuracy. +description: >- + Discover how to use the Experiments feature to test and improve machine + learning model accuracy. --- # Getting Started with Edge Impulse Experiments -Created By: -[Adam Milton-Barker](https://www.AdamMiltonBarker.com) +Created By: [Adam Milton-Barker](https://www.adammiltonbarker.com) -Public Project Link: -[https://studio.edgeimpulse.com/public/521263/latest](https://studio.edgeimpulse.com/public/521263/latest) +Public Project Link: [https://studio.edgeimpulse.com/public/521263/latest](https://studio.edgeimpulse.com/public/521263/latest) -![](.gitbook/assets/experiments-getting-started/edge-impulse-experiments.jpg) +![](../../.gitbook/assets/experiments-getting-started/edge-impulse-experiments.jpg) ## Introduction @@ -22,27 +22,27 @@ This project provides a walk through of how to use Experiments, along with a tut ## Hardware -- Arduino Nano RPI2040 Connect [More Info](https://store.arduino.cc/products/arduino-nano-rp2040-connect) +* Arduino Nano RPI2040 Connect [More Info](https://store.arduino.cc/products/arduino-nano-rp2040-connect) ## Platform -- Edge Impulse [Visit](https://www.edgeimpulse.com) +* Edge Impulse [Visit](https://www.edgeimpulse.com) ## Software -- Edge Impulse CLI [Download](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-installation) -- Arduino CLI [Download](https://arduino.github.io/arduino-cli/latest/) -- Arduino IDE 2.2.1 [Download](https://www.arduino.cc/en/software) +* Edge Impulse CLI [Download](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-installation) +* Arduino CLI [Download](https://arduino.github.io/arduino-cli/latest/) +* Arduino IDE 2.2.1 [Download](https://www.arduino.cc/en/software) ## Getting Started ### Arduino Nano RPI2040 Connect -![Arduino Nano RPI2040 Connect](.gitbook/assets/experiments-getting-started/arduino-nano-rpi2040-connect.jpg "Arduino Nano RPI2040 Connect") +![Arduino Nano RPI2040 Connect](../../.gitbook/assets/experiments-getting-started/arduino-nano-rpi2040-connect.jpg) The Arduino Nano RP2040 Connect is a highly versatile development board, bringing the power of the Raspberry Pi RP2040 microcontroller to the compact Nano form factor. Equipped with dual-core 32-bit Arm Cortex-M0+ processors, it enables seamless creation of IoT projects with built-in Wi-Fi and Bluetooth support via the U-blox Nina W102 module. The board includes an accelerometer, gyroscope, RGB LED, and omnidirectional microphone, making it ideal for real-time data collection and embedded AI applications. -![Arduino Nano RPI2040 Connect Pins](.gitbook/assets/experiments-getting-started/arduino-nano-rpi2040-connect-pins.jpg "rduino Nano RPI2040 Connect Pins") +![Arduino Nano RPI2040 Connect Pins](../../.gitbook/assets/experiments-getting-started/arduino-nano-rpi2040-connect-pins.jpg) The Nano RP2040 Connect is fully compatible with the Arduino Cloud platform, allowing users to rapidly prototype IoT solutions. It also supports MicroPython for those who prefer Python for programming. With a clock speed of 133 MHz, the board is well-suited for machine learning tasks, offering support for frameworks like TinyML and TensorFlow Lite. Additionally, its 6-axis IMU and temperature sensor expand the board's capability for advanced real-world applications. @@ -52,13 +52,13 @@ To begin working with the Edge Impulse platform and the Nano RPI2040 Connect, fo ### Create Edge Impulse Project -![Create Edge Impulse Project](.gitbook/assets/experiments-getting-started/edge-impulse-project.jpg "Create Edge Impulse Project") +![Create Edge Impulse Project](../../.gitbook/assets/experiments-getting-started/edge-impulse-project.jpg) Now it's time to create your Edge Impulse project. Head over to [Edge Impulse](https://studio.edgeimpulse.com/), log in, and create your new project. Edge Impulse offers Experiments to all users, with the Community tier allowing up to three simultaneous Experiments. Users on the Professional Plan and Enterprise tiers enjoy unlimited access to Experiments. You can explore all the platform's advanced features by signing up for an [Enterprise Trial](https://studio.edgeimpulse.com/trial-signup). -![Create Edge Impulse Project Dashboard](.gitbook/assets/experiments-getting-started/edge-impulse-project-dashboard.jpg "Create Edge Impulse Project Dashboard") +![Create Edge Impulse Project Dashboard](../../.gitbook/assets/experiments-getting-started/edge-impulse-project-dashboard.jpg) Once your project is created, you will see the project dashboard which will show you new additions to the platform. @@ -70,49 +70,49 @@ Next you need to connect your device to the Edge Impulse platform. Ensuring you edge-impulse-daemon ``` -![Edge Impulse Project Device Connection](.gitbook/assets/experiments-getting-started/edge-impulse-devices-cmd.jpg "Edge Impulse Project Device Connection") +![Edge Impulse Project Device Connection](../../.gitbook/assets/experiments-getting-started/edge-impulse-devices-cmd.jpg) You will be prompted for your Edge Impulse login details to proceed. Once authenticated you will need to choose the COM port that your device is connected to, and then select the Edge Impulse project you want to connect your device to. -![Edge Impulse Project Device Connected](.gitbook/assets/experiments-getting-started/edge-impulse-devices-rp2040-connected.jpg "Edge Impulse Project Device Connected") +![Edge Impulse Project Device Connected](../../.gitbook/assets/experiments-getting-started/edge-impulse-devices-rp2040-connected.jpg) If you now head over to your project and go the `Devices` tab, you will see your device is now connected. ### Collect Data -![Edge Impulse Project Collect Data](.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data.jpg "Edge Impulse Project Collect Data") +![Edge Impulse Project Collect Data](../../.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data.jpg) Now that your device is connected to Edge Impulse, it is time to collect some data. Head over to the `Data aquisition` tab and select the RPI2040. -![Edge Impulse Project Collect Normal Data](.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data-normal.jpg "Edge Impulse Project Collect Normal Data") +![Edge Impulse Project Collect Normal Data](../../.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data-normal.jpg) First we will create the `normal` data. This data will represent when a machine is running normally with no abnormal vibrations. Select the `Intertial` sensor and use `Normal` as the label. Next record about 3 minutes data, collected in 10 second samples from the device. -![Edge Impulse Project Collect Vibrations Data](.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data-vibrations.jpg "Edge Impulse Project Collect Vibrations Data") +![Edge Impulse Project Collect Vibrations Data](../../.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collect-data-vibrations.jpg) Next we will collect some `Vibrations` data. Change the label to `Vibrations` and record 3 minutes more of samples, but this time shake the Arduino around while the samples are being recorded. -![Edge Impulse Project Collected Data](.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collected-data.jpg "Edge Impulse Project Collected Data") +![Edge Impulse Project Collected Data](../../.gitbook/assets/experiments-getting-started/edge-impulse-rp2040-collected-data.jpg) You should now have about 6 minutes of data. Note that at this point the data is not split into Training and Testing groups. -![Edge Impulse Project Collected Data Split](.gitbook/assets/experiments-getting-started/edge-impulse-data-train-test-split.jpg "Edge Impulse Project Collected Data Split") +![Edge Impulse Project Collected Data Split](../../.gitbook/assets/experiments-getting-started/edge-impulse-data-train-test-split.jpg) Head to the project dashboard and scroll to the `Danger Zone` at the bottom. Click on the `Perform train/test split` button to split the data. -![Edge Impulse Project Collected Data Split](.gitbook/assets/experiments-getting-started/edge-impulse-data-split.jpg "Edge Impulse Project Collected Data Split") +![Edge Impulse Project Collected Data Split](../../.gitbook/assets/experiments-getting-started/edge-impulse-data-split.jpg) Back on the `Data aquisition` tab, you will now see that the data has been split. ### Create Impulse -![Edge Impulse Project Create Impulse](.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse.jpg "Edge Impulse Project Create Impulse") +![Edge Impulse Project Create Impulse](../../.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse.jpg) Now it is time to create your Impulse. Head over to the `Create Impulse` tab and you should see the configuration for your Nano RPI2040. You can accept the defaults here. ### Spectral Analysis -![Edge Impulse Project Create Spectral Analysis Impulse](.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse-spectral.jpg "Edge Impulse Project Create Spectral Analysis Impulse") +![Edge Impulse Project Create Spectral Analysis Impulse](../../.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse-spectral.jpg) First we will use the `Spectral Analysis` Processing block. Spectral Analysis is ideal for examining repetitive movements, particularly using accelerometer data. Tthis tool breaks down signals to reveal their frequency and power patterns over time. @@ -120,7 +120,7 @@ Click `Add` to add the Spectral Analysis Processing block to your Impulse. ### Classification -![Edge Impulse Project Create Spectral Analysis Classification Impulse](.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse-spectral-classification.jpg "Edge Impulse Project Create Spectral Analysis Classification Impulse") +![Edge Impulse Project Create Spectral Analysis Classification Impulse](../../.gitbook/assets/experiments-getting-started/edge-impulse-create-impulse-spectral-classification.jpg) For the Learning block, we will use `Classification` to classify between `Normal` and `Vibrations`. Click `Add` to add the Classification block to your Impulse. @@ -128,21 +128,21 @@ Next click `Save Impulse`. ### Feature Generation -![Edge Impulse Project Spectral Analysis Classification Features](.gitbook/assets/experiments-getting-started/edge-impulse-generate-features.jpg "Edge Impulse Project Spectral Analysis Classification Features") +![Edge Impulse Project Spectral Analysis Classification Features](../../.gitbook/assets/experiments-getting-started/edge-impulse-generate-features.jpg) Now we will generate the features that the AI model will use to learn. Head over to the `Spectral Features` tab and click on `Autotune parameters`. An autotune job will start and you will see the output on the right hand side of the UI. -![Edge Impulse Project Spectral Analysis Classification Features](.gitbook/assets/experiments-getting-started/edge-impulse-save-parameters.jpg "Edge Impulse Project Spectral Analysis Classification Features") +![Edge Impulse Project Spectral Analysis Classification Features](../../.gitbook/assets/experiments-getting-started/edge-impulse-save-parameters.jpg) Once the job is complete click `Save parameters`. You will be redirected to the `Generate features` tab. -![Edge Impulse Project Spectral Analysis Classification Features](.gitbook/assets/experiments-getting-started/edge-impulse-generated-features.jpg "Edge Impulse Project Spectral Analysis Classification Features") +![Edge Impulse Project Spectral Analysis Classification Features](../../.gitbook/assets/experiments-getting-started/edge-impulse-generated-features.jpg) A feature generation job will start, and once finished you will see the features on the right hand side. The features should be nicely clustered, if you notice features that are not clustered correctly you can click on them, review the samples and update your dataset or settings to fix. ### Training -![Edge Impulse Project Spectral Analysis Classification Training](.gitbook/assets/experiments-getting-started/edge-impulse-train-spectral-classifier.jpg "Edge Impulse Project Spectral Analysis Classification Training") +![Edge Impulse Project Spectral Analysis Classification Training](../../.gitbook/assets/experiments-getting-started/edge-impulse-train-spectral-classifier.jpg) Now it is time to train our model. Head over to the `Classifier` tab, leave the default settings intact, and click on `Save and train`. @@ -150,19 +150,19 @@ A training job will start, and once completed you will see the results on the ri ### Testing -![Edge Impulse Project Spectral Analysis Classification Testing](.gitbook/assets/experiments-getting-started/edge-impulse-train-spectral-classifier-testing.jpg "Edge Impulse Project Spectral Analysis Classification Testing") +![Edge Impulse Project Spectral Analysis Classification Testing](../../.gitbook/assets/experiments-getting-started/edge-impulse-train-spectral-classifier-testing.jpg) If you now head over to the `Model testing` tab, you will be able to use your newly trained model on the Test data that was set aside. The Test data was not shown to the model during training, so this will help to evaluate how well the model performs on unseen data. The testing process will start and you will see the results once complete. -![Edge Impulse Project First Experiment](.gitbook/assets/experiments-getting-started/edge-impulse-first-experiment.jpg "Edge Impulse Project First Experiment") +![Edge Impulse Project First Experiment](../../.gitbook/assets/experiments-getting-started/edge-impulse-first-experiment.jpg) If you head to the `Experiments` tab, you will see that you now have your first Experiment listed. ### Deployment -![Edge Impulse Project Deployment](.gitbook/assets/experiments-getting-started/edge-impulse-deployment.jpg "Edge Impulse Project Deployment") +![Edge Impulse Project Deployment](../../.gitbook/assets/experiments-getting-started/edge-impulse-deployment.jpg) You are now able to deploy your model to your Arduino. Head over to the `Deployment` tab and search for Arduino, then follow the steps provided to you to deploy the model to your device. @@ -170,11 +170,11 @@ As this tutorial is specifically related to Experiments, we will continue straig ### EON Tuner -![Edge Impulse Project EON Tuner](.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner.jpg "Edge Impulse Project EON Tuner") +![Edge Impulse Project EON Tuner](../../.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner.jpg) The EON™ Tuner simultaneously tests multiple model architectures, chosen based on your device and latency needs, to identify the best one for your application. The tuning process can take a while, but you can monitor its progress at any point during the search. -![Edge Impulse Project EON Tuner](.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-run.jpg "Edge Impulse Project EON Tuner") +![Edge Impulse Project EON Tuner](../../.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-run.jpg) On the `Experiments` tab, select `EON Tuner`. For the `Search space configuration` select `Classification` in the `Usecase templates` drop down, then click `Start tuning` to run. @@ -182,15 +182,15 @@ At this point, it is time to grab a coffee and put your feet up, as this will ta ## Experiment 2 -![Edge Impulse Project EON Tuner](.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-add.jpg "Edge Impulse Project EON Tuner") +![Edge Impulse Project EON Tuner](../../.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-add.jpg) If at any time during the EON tuning process, you see a configuration you would like to try, you can simply click the `Add` button for that configuration. -![Edge Impulse Project EON Tuner](.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-choose.jpg "Edge Impulse Project EON Tuner") +![Edge Impulse Project EON Tuner](../../.gitbook/assets/experiments-getting-started/edge-impulse-eon-tuner-choose.jpg) Here we see a configuration that has a considerable reduction for latency, RAM, and ROM, so we will use this configuration for our next Experiment. -![Edge Impulse Project Experiment 2](.gitbook/assets/experiments-getting-started/edge-impulse-experiment-2.jpg "Edge Impulse Project Experiment 2") +![Edge Impulse Project Experiment 2](../../.gitbook/assets/experiments-getting-started/edge-impulse-experiment-2.jpg) The platform will create the blocks for your new Impulse and add the features automatically for you. If you head back to the `Experiments` tab you will now see your new model waiting for you to test or deploy. @@ -203,4 +203,3 @@ At this point, once you have trained yet a third model and tested it's results, ## Conclusion In this tutorial, we demonstrated how to build a defect detection system with Edge Impulse and the Arduino Nano RP2040, and how to leverage the EON Tuner to optimize your model. From there, we built a second and third model with the new Experiments feature in Edge Impulse to allow us to evaluate different options and results. With this, you can easily refine and enhance your models, showcasing the power and simplicity of Edge Impulse's new Experiments feature for continuous improvement in machine learning projects. - diff --git a/image-projects/getting-started-openmv-rt1062.md b/readme/prototype-and-concept-projects/getting-started-openmv-rt1062.md similarity index 67% rename from image-projects/getting-started-openmv-rt1062.md rename to readme/prototype-and-concept-projects/getting-started-openmv-rt1062.md index 0a6eb960..2d012c45 100644 --- a/image-projects/getting-started-openmv-rt1062.md +++ b/readme/prototype-and-concept-projects/getting-started-openmv-rt1062.md @@ -1,49 +1,50 @@ --- description: >- - Getting Started with image classification on the OpenMV RT1062 Camera and Edge Impulse. + Getting Started with image classification on the OpenMV RT1062 Camera and Edge + Impulse. --- -# Getting Started with the OpenMV RT1062 +# OpenMV Cam RT1062 - Getting Started with Machine Learning Created By: Roni Bandini -GitHub Repository: [https://github.com/ronibandini/openMVClassificationPerformance ](https://github.com/ronibandini/openMVClassificationPerformance ) +GitHub Repository: [https://github.com/ronibandini/openMVClassificationPerformance](https://github.com/ronibandini/openMVClassificationPerformance) -![](../.gitbook/assets/getting-started-openmv-rt1062/cover.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/cover.jpg) ## Introduction -The open source, MCU-based OpenMV cameras and shields are popular devices for computer vision projects due to their low cost, user-friendly documentation, and excellent community. They are also easy to program, with Python and the OpenMV IDE. The new OpenMV RT1062 is the most powerful version yet; its features are truly outstanding: +The open source, MCU-based OpenMV cameras and shields are popular devices for computer vision projects due to their low cost, user-friendly documentation, and excellent community. They are also easy to program, with Python and the OpenMV IDE. The new OpenMV RT1062 is the most powerful version yet; its features are truly outstanding: -- ARM Cortex M7 processor running at 600 MHz -- OV5640 image sensor capable of taking 2592x1944 (5MP) images -- 32MBs SDRAM, 1MB of SRAM and 16 MB of program/storage flash -- I/O pins -- microSD slot capable of 25MB/s reads/writes -- A SPI bus that can run up to 60Mb/s -- An onboard RTC which keeps running when the system is in low-power mode -- A 12-bit X/Y/Z accelerometer (2/4/8g) -- Onboard WiFi (a/b/g/n - 11/54/65 Mb/s) and Bluetooth (v5.1 – BR/EDR/BLE) -- Onboard 10/100 Mb/s Ethernet -- A LiPo battery connector +* ARM Cortex M7 processor running at 600 MHz +* OV5640 image sensor capable of taking 2592x1944 (5MP) images +* 32MBs SDRAM, 1MB of SRAM and 16 MB of program/storage flash +* I/O pins +* microSD slot capable of 25MB/s reads/writes +* A SPI bus that can run up to 60Mb/s +* An onboard RTC which keeps running when the system is in low-power mode +* A 12-bit X/Y/Z accelerometer (2/4/8g) +* Onboard WiFi (a/b/g/n - 11/54/65 Mb/s) and Bluetooth (v5.1 – BR/EDR/BLE) +* Onboard 10/100 Mb/s Ethernet +* A LiPo battery connector -But how easy is it to work with this OpenMV RT1062? And how well does it perform in terms of inference time and power consumption when running machine learning vision projects built with Edge Impulse? Let’s find out. +But how easy is it to work with this OpenMV RT1062? And how well does it perform in terms of inference time and power consumption when running machine learning vision projects built with Edge Impulse? Let’s find out. ## Classification Models -For this project I have trained a simple classification model using Edge Impulse. I took 30 pictures of a Lego figure and 30 pictures of a small blue ball. I have created an Impulse with 96x96 px images and a Classification Learning Block. These items are very distinct from one another so the classification is quite easy from a machine learning perspective, but that is because we are more interested in evaluating the performance and ease of use of the OpenMV RT1062 here. In the future we could always explore more difficult ML tasks. +For this project I have trained a simple classification model using Edge Impulse. I took 30 pictures of a Lego figure and 30 pictures of a small blue ball. I have created an Impulse with 96x96 px images and a Classification Learning Block. These items are very distinct from one another so the classification is quite easy from a machine learning perspective, but that is because we are more interested in evaluating the performance and ease of use of the OpenMV RT1062 here. In the future we could always explore more difficult ML tasks. -![](../.gitbook/assets/getting-started-openmv-rt1062/dataset.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/dataset.jpg) I have trained the model with 10 cycles at 0.0005 learning rate, which was enough again due to the disparity between the chosen objects. After testing I chose “OpenMV Library” on the Deployment page. -![](../.gitbook/assets/getting-started-openmv-rt1062/model.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/model.jpg) -![](../.gitbook/assets/getting-started-openmv-rt1062/deploy.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/deploy.jpg) Next I extracted the `.zip` file that was generated and downloaded. I connected the OpenMV camera to my computer with a USB-C cable, and copied the `labels.txt` and trained `.tflite` files to the camera, which conveniently exposes itself as a USB drive when connected. -![](../.gitbook/assets/getting-started-openmv-rt1062/file-copy.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/file-copy.jpg) ## OpenMV Setup @@ -51,7 +52,7 @@ I have downloaded the OpenMV IDE from [https://openmv.io/pages/download](https:/ I have created a MicroPython script based on the example classification script included in the .zip file, but with small modifications to measure inference time. -![](../.gitbook/assets/getting-started-openmv-rt1062/inference-240.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/inference-240.jpg) I have clicked the Connection icon, and then executed the script with “Ctrl+R”. In the upper right window, a live view of the camera was presented, and I was also able to view console messages like inference time and FPS. @@ -59,53 +60,53 @@ The default script has a configured resolution of 240x240. In order to change th After this change I was able to increase the resolution up to 1024x1024. -![](../.gitbook/assets/getting-started-openmv-rt1062/inference-1024.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/inference-1024.jpg) ## Measuring Power Consumption The OpenMV FAQs state that the OpenMV Cam consumes about 100 mA while idle, and 140 mA when processing images. However, it’s important to note that there are several OpenMV Cam models (such as the RT1062, H7 Plus, and H7 R2) and ML inferences could increase the power consumption, so I soldered two headers to VIN and GND pins and I put a Multimeter in series with the power lines to make my own tests. -![](../.gitbook/assets/getting-started-openmv-rt1062/power-test-1.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/power-test-1.jpg) Given that I couldn't use the USB cable for this step, I needed some way to automatically execute the MicroPython script at start. I clicked **Tools, Save Open Script to OpenMV cam**, then I selected **Tools, Reset OpenMV Cam**. -![](../.gitbook/assets/getting-started-openmv-rt1062/auto-start.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/auto-start.jpg) At this point the script was automatically run when powering on the device, and I was able to measure IDLE and Inference power consumption for every resolution. -![](../.gitbook/assets/getting-started-openmv-rt1062/power-test-2.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/power-test-2.jpg) -As expected, Inference time varies while using a higher frame size, but there is not much difference in regards to power consumption. +As expected, Inference time varies while using a higher frame size, but there is not much difference in regards to power consumption. > Note: Because I utilized the low resolution ammeter feature of a general-purpose multimeter, the measurement of power consumption may not be entirely accurate and precision equipment should be used for highly detailed investigation if needed. -![](../.gitbook/assets/getting-started-openmv-rt1062/results.jpg) +![](../../.gitbook/assets/getting-started-openmv-rt1062/results.jpg) ## OpenMV Status The OpenMV Cam has an on-board LED with blinking patterns related to different board states. -- Green: Bootloader is running -- Blue: The board is executing `main.py` script -- White: Firmware is panicking from a hardware failure +* Green: Bootloader is running +* Blue: The board is executing `main.py` script +* White: Firmware is panicking from a hardware failure If for any reason your OpenMV RT1062 becomes bricked during firmware updates, you have to make a small bridge between the SBL pin and 3.3v pin. Then you will be able to re-flash (functional) firmware via the OpenMV IDE. ## Visual Anomaly and OpenMV -Edge Impulse recently included Visual Anomaly detection but, but for now this is not supported on the OpenMV RT1062. Perhaps in the future this functionality can be added. - +Edge Impulse recently included Visual Anomaly detection but, but for now this is not supported on the OpenMV RT1062. Perhaps in the future this functionality can be added. + ## Final Notes The new OpenMV RT1062 camera is an excellent choice for edge AI projects, because the 5MP resolution allows you not only to make inferences but also to capture high-quality snapshots for your dataset, and store them on a microSD card or send them to your Edge Impulse project via WiFi. Additionally, its power configuration can be fine-tuned for efficiency (with low consumption to start) thanks to the board’s RTC/sleep features. And of course, using a high-level programming language like MicroPython simplifies any development considerably. -The MicroPython used for this project is available in this GitHub repository: [https://github.com/ronibandini/openMVClassificationPerformance](https://github.com/ronibandini/openMVClassificationPerformance ) +The MicroPython used for this project is available in this GitHub repository: [https://github.com/ronibandini/openMVClassificationPerformance](https://github.com/ronibandini/openMVClassificationPerformance) ## References [https://docs.openmv.io/](https://docs.openmv.io/) -[https://openmv.io/pages/faq#:~:text=The%20OpenMV%20Cam%20consumes%20about,140%20mA%20when%20processing%20images](https://openmv.io/pages/faq#:~:text=The%20OpenMV%20Cam%20consumes%20about,140%20mA%20when%20processing%20images) +[https://openmv.io/pages/faq#:\~:text=The%20OpenMV%20Cam%20consumes%20about,140%20mA%20when%20processing%20images](https://openmv.io/pages/faq) [https://www.edgeimpulse.com/blog/automated-manufacturing-inspection-with-fomo-ad/](https://www.edgeimpulse.com/blog/automated-manufacturing-inspection-with-fomo-ad/) diff --git a/renesas-ra6m5-getting-started.md b/readme/prototype-and-concept-projects/renesas-ra6m5-getting-started.md similarity index 63% rename from renesas-ra6m5-getting-started.md rename to readme/prototype-and-concept-projects/renesas-ra6m5-getting-started.md index 6e2d7f59..8fb373dd 100644 --- a/renesas-ra6m5-getting-started.md +++ b/readme/prototype-and-concept-projects/renesas-ra6m5-getting-started.md @@ -1,14 +1,14 @@ --- -description: Getting Started with machine learning on the Renesas CK-RA6M5 Cloud Kit and Edge Impulse. +description: >- + Getting Started with machine learning on the Renesas CK-RA6M5 Cloud Kit and + Edge Impulse. --- -# Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning +# Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning -Created By: -Swapnil Verma +Created By: Swapnil Verma -Public Project Link: -[https://studio.edgeimpulse.com/public/233106/latest](https://studio.edgeimpulse.com/public/233106/latest) +Public Project Link: [https://studio.edgeimpulse.com/public/233106/latest](https://studio.edgeimpulse.com/public/233106/latest) ## Introduction @@ -20,40 +20,40 @@ This is a Getting Started Guide for the Renesas CK-RA6M5 board with Edge Impulse The Cloud Kit comes with the following items in the box: -- The CK-RA6M5 development board -- RYZ014A PMOD (CAT-M1 Cellular Module) -- SIM card -- Antenna -- 2 Micro USB to A cables -- Micro USB A/B to A adapter cable -- Documentation +* The CK-RA6M5 development board +* RYZ014A PMOD (CAT-M1 Cellular Module) +* SIM card +* Antenna +* 2 Micro USB to A cables +* Micro USB A/B to A adapter cable +* Documentation -![Renesas CK-RA6M5 Cloud Kit](.gitbook/assets/renesas-ra6m5-getting-started/ra6m5-kit.jpg) +![Renesas CK-RA6M5 Cloud Kit](../../.gitbook/assets/renesas-ra6m5-getting-started/ra6m5-kit.jpg) ## Quick Start Project Each CK-RA6M5 board comes preinstalled with a quick-start project. Let's run that quick-start project to verify our board is working properly. -- Make sure that (a) J22 is set to link pins 2-3 (b) J21 link is closed and (c) J16 link is open. -- Connect J14 and J20 on the CK-RA6M5 board to USB ports on the host PC using the 2 micro USB cables supplied. -- The power LED (LED6) on the CK-RA6M5 board lights up white, indicating that the CK-RA6M5 board is powered on. +* Make sure that (a) J22 is set to link pins 2-3 (b) J21 link is closed and (c) J16 link is open. +* Connect J14 and J20 on the CK-RA6M5 board to USB ports on the host PC using the 2 micro USB cables supplied. +* The power LED (LED6) on the CK-RA6M5 board lights up white, indicating that the CK-RA6M5 board is powered on. -![Board Connection](.gitbook/assets/renesas-ra6m5-getting-started/ra6m5-connection.jpg) +![Board Connection](../../.gitbook/assets/renesas-ra6m5-getting-started/ra6m5-connection.jpg) Immediately after the power on, the four user LEDs will take on the following states: -![LED Status](.gitbook/assets/renesas-ra6m5-getting-started/led-status.gif) +![LED Status](../../.gitbook/assets/renesas-ra6m5-getting-started/led-status.gif) -- LED1 Red – Off -- LED2 RGB – Off -- LED3 Green – Steady, full intensity -- LED4 Blue – Blinking at 1hz frequency +* LED1 Red – Off +* LED2 RGB – Off +* LED3 Green – Steady, full intensity +* LED4 Blue – Blinking at 1hz frequency -![LED4 blinking at 1hz](.gitbook/assets/renesas-ra6m5-getting-started/led-status-2.gif) +![LED4 blinking at 1hz](../../.gitbook/assets/renesas-ra6m5-getting-started/led-status-2.gif) Press the user button (S2) on the board to change the blinking frequency of the user LED4 (blue). With every press of the first user button (S2), the frequency will switch from 1 Hz to 5 Hz to 10 Hz and cycle back. -![LED4 blinking at 5hz](.gitbook/assets/renesas-ra6m5-getting-started/led-status-3.gif) +![LED4 blinking at 5hz](../../.gitbook/assets/renesas-ra6m5-getting-started/led-status-3.gif) ## Updating the Firmware @@ -67,35 +67,35 @@ Once the board is flashed with Edge Impulse firmware, the real magic starts. To begin, you'll need to create an Edge Impulse account and a project in the Edge Impulse Studio. Please follow the below steps to do so: -- Navigate to the [Edge Impulse Studio](https://studio.edgeimpulse.com/login) and create an account. If you already have an account then please login using your credentials. +* Navigate to the [Edge Impulse Studio](https://studio.edgeimpulse.com/login) and create an account. If you already have an account then please login using your credentials. -![Edge Impulse Studio](.gitbook/assets/renesas-ra6m5-getting-started/studio.jpg) +![Edge Impulse Studio](../../.gitbook/assets/renesas-ra6m5-getting-started/studio.jpg) -- After login, please create a new project, give it a suitable name, and select an appropriate _Project type_. +* After login, please create a new project, give it a suitable name, and select an appropriate _Project type_. -![Project Type](.gitbook/assets/renesas-ra6m5-getting-started/project-type.jpg) +![Project Type](../../.gitbook/assets/renesas-ra6m5-getting-started/project-type.jpg) -- After creating a new project, navigate to the _Devices_ Tab. +* After creating a new project, navigate to the _Devices_ Tab. -![Device Tab](.gitbook/assets/renesas-ra6m5-getting-started/devices.jpg) +![Device Tab](../../.gitbook/assets/renesas-ra6m5-getting-started/devices.jpg) ## Connecting Renesas CK-RA6M5 to Edge Impulse The next step is connecting our Renesas CK-RA6M5 board to the Edge Impulse Studio, so we can ingest sensor data for the machine learning model. Please follow the below steps to do so: -- Connect the Renesas CK-RA6M5 board to the computer by following the steps mentioned in the _Quick Start_ section. -- Open a terminal or command prompt and type `edge-impulse-daemon`. The [Edge Impulse daemon](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-daemon) will start and prompt for user credentials. -- After providing user credentials, it will prompt you to select an Edge Impulse project. Please navigate and select the project created in the previous steps, using the arrow keys. +* Connect the Renesas CK-RA6M5 board to the computer by following the steps mentioned in the _Quick Start_ section. +* Open a terminal or command prompt and type `edge-impulse-daemon`. The [Edge Impulse daemon](https://docs.edgeimpulse.com/docs/edge-impulse-cli/cli-daemon) will start and prompt for user credentials. +* After providing user credentials, it will prompt you to select an Edge Impulse project. Please navigate and select the project created in the previous steps, using the arrow keys. -![Daemon](.gitbook/assets/renesas-ra6m5-getting-started/daemon.jpg) +![Daemon](../../.gitbook/assets/renesas-ra6m5-getting-started/daemon.jpg) -- After selecting the project, it will ask you to give the connected board a name. It is useful when you want to connect multiple boards to the same project. +* After selecting the project, it will ask you to give the connected board a name. It is useful when you want to connect multiple boards to the same project. -![Device Naming](.gitbook/assets/renesas-ra6m5-getting-started/naming.jpg) +![Device Naming](../../.gitbook/assets/renesas-ra6m5-getting-started/naming.jpg) -- Now the board should be connected to the selected project. The `edge-impulse-daemon` will tell you which project the board is connected to. We can also verify by checking the **Devices** tab of that project. +* Now the board should be connected to the selected project. The `edge-impulse-daemon` will tell you which project the board is connected to. We can also verify by checking the **Devices** tab of that project. -![Device Connected](.gitbook/assets/renesas-ra6m5-getting-started/connected.jpg) +![Device Connected](../../.gitbook/assets/renesas-ra6m5-getting-started/connected.jpg) It will also list all the sensors available for data gathering. @@ -103,43 +103,43 @@ It will also list all the sensors available for data gathering. Edge Impulse provides multiple options for [data acquisition](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition). In this Getting Started Guide, we will look at the direct data ingestion from the board using `edge-impulse-daemon`. Please follow the below steps for data acquisition: -- Navigate to the _Data Acquisition_ tab in the Edge Impulse Studio. +* Navigate to the _Data Acquisition_ tab in the Edge Impulse Studio. -![Data Acquisition](.gitbook/assets/renesas-ra6m5-getting-started/data-acquisition.jpg) +![Data Acquisition](../../.gitbook/assets/renesas-ra6m5-getting-started/data-acquisition.jpg) -- Here you will find the _Device_ we connected in the previous step and the sensor list. Please select the suitable sensor from the drop-down menu. For this project, I have selected the _Microphone_ sensor and used default parameters. -- Add a _Label name_ for the sample you are about to collect. I am collecting clap and whistle sounds therefore I will use _clap_ and _whistle_ as labels. -- Clicking _Start Sampling_ will start the sample collection process. Once the sample is collected, it will be automatically uploaded to the Edge Impulse Studio. +* Here you will find the _Device_ we connected in the previous step and the sensor list. Please select the suitable sensor from the drop-down menu. For this project, I have selected the _Microphone_ sensor and used default parameters. +* Add a _Label name_ for the sample you are about to collect. I am collecting clap and whistle sounds therefore I will use _clap_ and _whistle_ as labels. +* Clicking _Start Sampling_ will start the sample collection process. Once the sample is collected, it will be automatically uploaded to the Edge Impulse Studio. -![Data Collection Process](.gitbook/assets/renesas-ra6m5-getting-started/data-collection.gif) +![Data Collection Process](../../.gitbook/assets/renesas-ra6m5-getting-started/data-collection.gif) When enough samples are collected, [balance the data](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition#dataset-train-test-split-ratio) and if required [clean the data](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition#cropping-samples) as well. -![Dataset Train/Test Split](.gitbook/assets/renesas-ra6m5-getting-started/split.jpg) +![Dataset Train/Test Split](../../.gitbook/assets/renesas-ra6m5-getting-started/split.jpg) ## Machine Learning Model Preparation After data collection, the next step is machine learning model preparation. To do so, please navigate to the [_Impulse design_ tab](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design) and add relevant [preprocessing](https://docs.edgeimpulse.com/docs/edge-impulse-studio/processing-blocks) and [learning blocks](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks) to the pipeline. -- Edge Impulse Studio will automatically add an [input block](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design#input-block) and it will recommend a suitable preprocessing and a learning block based on the data type. I have used the recommended ones in this project with the default arguments. +* Edge Impulse Studio will automatically add an [input block](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design#input-block) and it will recommend a suitable preprocessing and a learning block based on the data type. I have used the recommended ones in this project with the default arguments. -![Impulse Design](.gitbook/assets/renesas-ra6m5-getting-started/impulse-design.jpg) +![Impulse Design](../../.gitbook/assets/renesas-ra6m5-getting-started/impulse-design.jpg) -- After Impulse design is complete, save the design and navigate to the preprocessing tab (MFE in this case) for the feature generation. +* After Impulse design is complete, save the design and navigate to the preprocessing tab (MFE in this case) for the feature generation. -![Preprocessing Block](.gitbook/assets/renesas-ra6m5-getting-started/feature-generation.jpg) +![Preprocessing Block](../../.gitbook/assets/renesas-ra6m5-getting-started/feature-generation.jpg) Click on the _Save parameters_ button, then navigate to the _Generate features_ tab and click _Generate features_ button for data preprocessing. -![Feature Generation](.gitbook/assets/renesas-ra6m5-getting-started/feature-generation-2.jpg) +![Feature Generation](../../.gitbook/assets/renesas-ra6m5-getting-started/feature-generation-2.jpg) -- After feature generation, please navigate to the _Learning Tab_ ([Classifier](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/classification) in this case) to design the neural network architecture. I have used the default architecture and parameters recommended by the Edge Impulse Studio. After selecting a suitable configuration, click on the _Start training_ button. +* After feature generation, please navigate to the _Learning Tab_ ([Classifier](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/classification) in this case) to design the neural network architecture. I have used the default architecture and parameters recommended by the Edge Impulse Studio. After selecting a suitable configuration, click on the _Start training_ button. -![Classifier Design](.gitbook/assets/renesas-ra6m5-getting-started/classifier-design.jpg) +![Classifier Design](../../.gitbook/assets/renesas-ra6m5-getting-started/classifier-design.jpg) -- Once the training is complete, please navigate to the [Model testing](https://docs.edgeimpulse.com/docs/edge-impulse-studio/model-testing) tab, and click _Classify all_ button. +* Once the training is complete, please navigate to the [Model testing](https://docs.edgeimpulse.com/docs/edge-impulse-studio/model-testing) tab, and click _Classify all_ button. -![Model Testing](.gitbook/assets/renesas-ra6m5-getting-started/model-testing.jpg) +![Model Testing](../../.gitbook/assets/renesas-ra6m5-getting-started/model-testing.jpg) After testing is finished, the Edge Impulse Studio will show the model accuracy, and other parameters. @@ -149,17 +149,17 @@ After testing is finished, the Edge Impulse Studio will show the model accuracy, In this step, we will deploy our prepared model to the Renesas CK-RA6M5 board, so we can perform inference locally on the board. -- Please navigate to the [Deployment](https://docs.edgeimpulse.com/docs/edge-impulse-studio/deployment) tab, select the Renesas CK-RA6M5 board using the search bar, and click on the _Build_ button. +* Please navigate to the [Deployment](https://docs.edgeimpulse.com/docs/edge-impulse-studio/deployment) tab, select the Renesas CK-RA6M5 board using the search bar, and click on the _Build_ button. -![Deployment Tab](.gitbook/assets/renesas-ra6m5-getting-started/deployment.jpg) +![Deployment Tab](../../.gitbook/assets/renesas-ra6m5-getting-started/deployment.jpg) -- After the _build_ is finished, the new firmware will be downloaded automatically to your computer, and the Edge Impulse Studio will provide next-step instructions. +* After the _build_ is finished, the new firmware will be downloaded automatically to your computer, and the Edge Impulse Studio will provide next-step instructions. -![Next Steps](.gitbook/assets/renesas-ra6m5-getting-started/deployment-2.jpg) +![Next Steps](../../.gitbook/assets/renesas-ra6m5-getting-started/deployment-2.jpg) -- Please extract the folder and double-click the `flash_` file. This will flash the newly created firmware on the CK-RA6M5 board. This firmware contains the machine learning model we prepared in the above steps. +* Please extract the folder and double-click the `flash_` file. This will flash the newly created firmware on the CK-RA6M5 board. This firmware contains the machine learning model we prepared in the above steps. -![Flashing Firmware](.gitbook/assets/renesas-ra6m5-getting-started/flashing.gif) +![Flashing Firmware](../../.gitbook/assets/renesas-ra6m5-getting-started/flashing.gif) ## Inferencing @@ -170,5 +170,3 @@ The next step is testing!! Let's see how well our model performs when run locall And, that's it. I hope this Getting Started Guide will be useful for you when using the Renesas CK-RA6M5 with Edge Impulse. If you have any questions, please check out the [Edge Impulse Forum](https://forum.edgeimpulse.com/). - - diff --git a/ti-cc1352p-getting-started.md b/readme/prototype-and-concept-projects/ti-cc1352p-getting-started.md similarity index 59% rename from ti-cc1352p-getting-started.md rename to readme/prototype-and-concept-projects/ti-cc1352p-getting-started.md index c17327ed..13db0b3d 100644 --- a/ti-cc1352p-getting-started.md +++ b/readme/prototype-and-concept-projects/ti-cc1352p-getting-started.md @@ -1,16 +1,16 @@ --- -description: Getting Started with machine learning on the TI CC1352P Launchpad and Edge Impulse. +description: >- + Getting Started with machine learning on the TI CC1352P Launchpad and Edge + Impulse. --- -# TI CC1352P Launchpad - Getting Started with Machine Learning +# TI CC1352P Launchpad - Getting Started with Machine Learning -Created By: -Swapnil Verma +Created By: Swapnil Verma -Public Project Link: -[https://studio.edgeimpulse.com/public/238204/latest](https://studio.edgeimpulse.com/public/238204/latest) +Public Project Link: [https://studio.edgeimpulse.com/public/238204/latest](https://studio.edgeimpulse.com/public/238204/latest) -![](.gitbook/assets/ti-cc1352p-getting-started/header.jpg) +![](../../.gitbook/assets/ti-cc1352p-getting-started/header.jpg) ## Introduction @@ -22,30 +22,30 @@ This is a Getting Started Guide for the TI LAUNCHXL-CC1352P development board wi The Launchpad Kit comes with the following items in the box: -- The LAUNCHXL-CC1352P development board -- Micro-USB to USB-A Cable -- Documentation +* The LAUNCHXL-CC1352P development board +* Micro-USB to USB-A Cable +* Documentation The BOOSTXL sensor comes with: -- BOOSTXL sensor board -- Documentation +* BOOSTXL sensor board +* Documentation ## Quick Start Project Each LAUNCHXL-CC1352P board comes preinstalled with a quick-start project called Project Zero. Let's run the quick-start project to verify our board is working properly. -- Connect the board to the computer using the provided micro-USB cable. -- Download the SimpleLink Starter smartphone app on your smartphone. This app lets you control the LEDs, see the state of the buttons and send data to the UART. -- Open the app, select Project Zero from the list of available devices, and click on Sensor View to get the GUI. +* Connect the board to the computer using the provided micro-USB cable. +* Download the SimpleLink Starter smartphone app on your smartphone. This app lets you control the LEDs, see the state of the buttons and send data to the UART. +* Open the app, select Project Zero from the list of available devices, and click on Sensor View to get the GUI. -![SimpleLink Starter Application](.gitbook/assets/ti-cc1352p-getting-started/simplelink.jpg) +![SimpleLink Starter Application](../../.gitbook/assets/ti-cc1352p-getting-started/simplelink.jpg) In the Project Zero GUI, tap on the lights to turn them On/Off on the board. Press the user buttons on the board to see their status on the app.In the Project Zero GUI, tap on the lights to turn them On/Off on the board. Press the user buttons on the board to see their status on the app. {% embed url="https://www.youtube.com/watch?v=vSlQIOXhFPc" %} -To learn more about Project Zero or user guide please follow [this link](https://dev.ti.com/tirex/explore/node?devtools=LAUNCHXL-CC1352P-2&node=A__AP29fH4CgXFx6NzMPibGgg__cc13xx_devtools__FUz-xrs__LATEST). +To learn more about Project Zero or user guide please follow [this link](https://dev.ti.com/tirex/explore/node?devtools=LAUNCHXL-CC1352P-2\&node=A__AP29fH4CgXFx6NzMPibGgg__cc13xx_devtools__FUz-xrs__LATEST). ## Updating the Firmware @@ -59,74 +59,74 @@ Please follow this official guide to update the firmware: To begin, you'll need to create an Edge Impulse account and a project in the Edge Impulse Studio. Please follow the below steps to do so: -- Navigate to the [Edge Impulse Studio](https://studio.edgeimpulse.com/login) and create an account. If you already have an account then please login using your credentials. +* Navigate to the [Edge Impulse Studio](https://studio.edgeimpulse.com/login) and create an account. If you already have an account then please login using your credentials. -![Edge Impulse Studio](.gitbook/assets/ti-cc1352p-getting-started/studio.jpg) +![Edge Impulse Studio](../../.gitbook/assets/ti-cc1352p-getting-started/studio.jpg) -- After login, please create a new project, give it a suitable name, and select an appropriate _Project type_. +* After login, please create a new project, give it a suitable name, and select an appropriate _Project type_. -![Project Type](.gitbook/assets/ti-cc1352p-getting-started/project-type.jpg) +![Project Type](../../.gitbook/assets/ti-cc1352p-getting-started/project-type.jpg) -- After creating a new project, let's connect the development board to the Studio. +* After creating a new project, let's connect the development board to the Studio. ## Connecting TI Launchpad Kit to Edge Impulse The next step is connecting our TI LAUNCHXL board to the Edge Impulse Studio, so we can ingest sensor data for the machine learning model. Please follow the below steps to do so: -- Open a terminal or command prompt and type `edge-impulse-daemon`. The [Edge Impulse daemon](https://docs.edgeimpulse.com/docs/Edge Impulse-cli/cli-daemon) will start and prompt for user credentials. If you have not installed Edge Impulse CLI as part of the **Update the Firmware** section, then please install it now. -- After providing user credentials, it will prompt you to select an Edge Impulse project. Please navigate and select the project created in the previous steps, using the arrow keys. +* Open a terminal or command prompt and type `edge-impulse-daemon`. The \[Edge Impulse daemon]\(https://docs.edgeimpulse.com/docs/Edge Impulse-cli/cli-daemon) will start and prompt for user credentials. If you have not installed Edge Impulse CLI as part of the **Update the Firmware** section, then please install it now. +* After providing user credentials, it will prompt you to select an Edge Impulse project. Please navigate and select the project created in the previous steps, using the arrow keys. -![Daemon](.gitbook/assets/ti-cc1352p-getting-started/daemon.jpg) +![Daemon](../../.gitbook/assets/ti-cc1352p-getting-started/daemon.jpg) -- After selecting the project, it will ask you to give the connected board a name. It is useful when you want to connect multiple boards to the same project. +* After selecting the project, it will ask you to give the connected board a name. It is useful when you want to connect multiple boards to the same project. -![Device Naming](.gitbook/assets/ti-cc1352p-getting-started/naming.jpg) +![Device Naming](../../.gitbook/assets/ti-cc1352p-getting-started/naming.jpg) -- Now the board should be connected to the selected project. The `edge-impulse-daemon` will tell you which project the board is connected to. We can also verify by checking the devices tab of that project. +* Now the board should be connected to the selected project. The `edge-impulse-daemon` will tell you which project the board is connected to. We can also verify by checking the devices tab of that project. -![Device Connected](.gitbook/assets/ti-cc1352p-getting-started/connected.jpg) +![Device Connected](../../.gitbook/assets/ti-cc1352p-getting-started/connected.jpg) ## Data Gathering Edge Impulse provides multiple options for [data acquisition](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition). In this Getting Started Guide, we will look at the direct data ingestion from the board using `edge-impulse-daemon`. Please follow the below steps for data acquisition: -- Navigate to the _Data Acquisition_ tab in the Edge Impulse Studio. +* Navigate to the _Data Acquisition_ tab in the Edge Impulse Studio. -![Data Acquisition](.gitbook/assets/ti-cc1352p-getting-started/data-acquisition.jpg) +![Data Acquisition](../../.gitbook/assets/ti-cc1352p-getting-started/data-acquisition.jpg) -- Here you will find the _Device_ we connected in the previous step and the sensor list. Please select the suitable sensor from the drop-down menu. For this project, I have selected the _Accelerometer_ sensor and used default parameters. -- Add a _Label name_ for the sample you are about to collect. I am collecting up-down, side-to-side and circular motion data therefore I will use up_down, side_to_side and circle as labels. As a default motion, I will also collect stationary data. -- Clicking _Start Sampling_ will start the sample collection process. Once the sample is collected, it will be automatically uploaded to the Edge Impulse Studio. +* Here you will find the _Device_ we connected in the previous step and the sensor list. Please select the suitable sensor from the drop-down menu. For this project, I have selected the _Accelerometer_ sensor and used default parameters. +* Add a _Label name_ for the sample you are about to collect. I am collecting up-down, side-to-side and circular motion data therefore I will use up\_down, side\_to\_side and circle as labels. As a default motion, I will also collect stationary data. +* Clicking _Start Sampling_ will start the sample collection process. Once the sample is collected, it will be automatically uploaded to the Edge Impulse Studio. -![Data Collection Process](.gitbook/assets/ti-cc1352p-getting-started/data-collection.gif) +![Data Collection Process](../../.gitbook/assets/ti-cc1352p-getting-started/data-collection.gif) -When enough samples are collected, [balance the data](https://docs.edgeimpulse.com/docs/Edge Impulse-studio/data-acquisition#dataset-train-test-split-ratio) and if required [clean the data](https://docs.edgeimpulse.com/docs/Edge Impulse-studio/data-acquisition#cropping-samples) as well. +When enough samples are collected, \[balance the data]\(https://docs.edgeimpulse.com/docs/Edge Impulse-studio/data-acquisition#dataset-train-test-split-ratio) and if required \[clean the data]\(https://docs.edgeimpulse.com/docs/Edge Impulse-studio/data-acquisition#cropping-samples) as well. -![Dataset Train/Test Split](.gitbook/assets/ti-cc1352p-getting-started/split.jpg) +![Dataset Train/Test Split](../../.gitbook/assets/ti-cc1352p-getting-started/split.jpg) ## Machine Learning Model Preparation After data collection, the next step is machine learning model preparation. To do so, please navigate to the [_Impulse design_ tab](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design) and add relevant [preprocessing](https://docs.edgeimpulse.com/docs/edge-impulse-studio/processing-blocks) and [learning blocks](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks) to the pipeline. -- Edge Impulse Studio will automatically add an [input block](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design#input-block) and it will recommend a suitable preprocessing and a learning block based on the data type. I have used the recommended ones in this project with the default arguments. +* Edge Impulse Studio will automatically add an [input block](https://docs.edgeimpulse.com/docs/edge-impulse-studio/impulse-design#input-block) and it will recommend a suitable preprocessing and a learning block based on the data type. I have used the recommended ones in this project with the default arguments. -![Impulse Design](.gitbook/assets/ti-cc1352p-getting-started/impulse-design.jpg) +![Impulse Design](../../.gitbook/assets/ti-cc1352p-getting-started/impulse-design.jpg) -- After Impulse design is complete, save the design and navigate to the preprocessing tab (Spectral features in this case) for the feature generation. +* After Impulse design is complete, save the design and navigate to the preprocessing tab (Spectral features in this case) for the feature generation. -![Preprocessing Block](.gitbook/assets/ti-cc1352p-getting-started/feature-generation.jpg) +![Preprocessing Block](../../.gitbook/assets/ti-cc1352p-getting-started/feature-generation.jpg) Click on the _Save parameters_ button, then navigate to the _Generate features_ tab and click _Generate features_ button for data preprocessing. -![Feature Generation](.gitbook/assets/ti-cc1352p-getting-started/feature-generation-2.jpg) +![Feature Generation](../../.gitbook/assets/ti-cc1352p-getting-started/feature-generation-2.jpg) -- After feature generation, please navigate to the _Learning Tab_ ([Classifier](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/classification) in this case) to design the neural network architecture. I have used the default architecture and parameters recommended by the Edge Impulse Studio. After selecting a suitable configuration, click on the _Start training_ button. +* After feature generation, please navigate to the _Learning Tab_ ([Classifier](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/classification) in this case) to design the neural network architecture. I have used the default architecture and parameters recommended by the Edge Impulse Studio. After selecting a suitable configuration, click on the _Start training_ button. -![Train a Classifier Model](.gitbook/assets/ti-cc1352p-getting-started/classifier-design.jpg) +![Train a Classifier Model](../../.gitbook/assets/ti-cc1352p-getting-started/classifier-design.jpg) -- Once the training is complete, please navigate to the [Model testing](https://docs.edgeimpulse.com/docs/Edge Impulse-studio/model-testing) tab, and click _Classify all_ button. +* Once the training is complete, please navigate to the \[Model testing]\(https://docs.edgeimpulse.com/docs/Edge Impulse-studio/model-testing) tab, and click _Classify all_ button. -![Model Testing](.gitbook/assets/ti-cc1352p-getting-started/model-testing.jpg) +![Model Testing](../../.gitbook/assets/ti-cc1352p-getting-started/model-testing.jpg) After testing is finished, the Edge Impulse Studio will show the model accuracy, and other parameters. @@ -134,17 +134,16 @@ After testing is finished, the Edge Impulse Studio will show the model accuracy, ## Deployment -In this step, we will deploy our prepared model to the TI LAUNCHXL-CC1352P development board, so we can perform inference locally on the board. +In this step, we will deploy our prepared model to the TI LAUNCHXL-CC1352P development board, so we can perform inference locally on the board. -- Please navigate to the [Deployment](https://docs.edgeimpulse.com/docs/Edge Impulse-studio/deployment) tab, select the TI LAUNCHXL-CC1352P board using the search bar, and click on the _Build_ button. +* Please navigate to the \[Deployment]\(https://docs.edgeimpulse.com/docs/Edge Impulse-studio/deployment) tab, select the TI LAUNCHXL-CC1352P board using the search bar, and click on the _Build_ button. -![Deployment Tab](.gitbook/assets/ti-cc1352p-getting-started/deployment.jpg) +![Deployment Tab](../../.gitbook/assets/ti-cc1352p-getting-started/deployment.jpg) -- After the _build_ is finished, the new firmware will be downloaded automatically to your computer, and the Edge Impulse Studio will provide next-step instructions. +* After the _build_ is finished, the new firmware will be downloaded automatically to your computer, and the Edge Impulse Studio will provide next-step instructions. +* Please extract the folder and double-click the `flash_` file. This will flash the newly created firmware on the TI LAUNCHXL-CC1352P board. This firmware contains the machine learning model we prepared in the above steps. -- Please extract the folder and double-click the `flash_` file. This will flash the newly created firmware on the TI LAUNCHXL-CC1352P board. This firmware contains the machine learning model we prepared in the above steps. - -![Flashing Firmware](.gitbook/assets/ti-cc1352p-getting-started/flashing.gif) +![Flashing Firmware](../../.gitbook/assets/ti-cc1352p-getting-started/flashing.gif) ## Inferencing @@ -155,5 +154,3 @@ The next step is testing!! Let's see how well our model performs when run locall And, that's it. I hope this Getting Started Guide will be useful for you when using the TI LAUNCHXL-CC1352P with Edge Impulse. If you have any questions, please check out the [Edge Impulse Forum](https://forum.edgeimpulse.com/). - - diff --git a/software-integration-demos/README.md b/software-integration-demos/README.md new file mode 100644 index 00000000..d257b89f --- /dev/null +++ b/software-integration-demos/README.md @@ -0,0 +1,2 @@ +# Software Integration Demos + diff --git a/software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md b/software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md index d3cefcd8..d990ed4a 100644 --- a/software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md +++ b/software-integration-demos/azure-machine-learning-kubernetes-compute-ei.md @@ -41,9 +41,9 @@ In this section I will show how to implement a voice-to-text translation endpoin The machine learning model we will use for voice-to-text transformation is the [**Wav2vec 2.0**](https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/) NLP framework, more specifically a pre-trained version of it. Wav2vec 2.0 is the second version of a speech recognition model developed by Facebook / Meta researchers: > _**Wav2vec 2.0: Learning the structure of speech from raw audio**_\ -> _https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/_ +> &#xNAN;_https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/_ -A [pre-trained version](https://huggingface.co/docs/transformers/model\_doc/wav2vec2) of Wav2vec 2.0 is available through the [🤗 Transformers](https://huggingface.co/docs/transformers/index) library. The pre-trained model supports both PyTorch and TensorFlow libraries. We will use it with PyTorch. +A [pre-trained version](https://huggingface.co/docs/transformers/model_doc/wav2vec2) of Wav2vec 2.0 is available through the [🤗 Transformers](https://huggingface.co/docs/transformers/index) library. The pre-trained model supports both PyTorch and TensorFlow libraries. We will use it with PyTorch. ### Getting Started with Azure ML @@ -254,7 +254,7 @@ Using a `kubectl` CLI tool we can also see what resource Azure ML deployed in ou Additionally, Azure ML also has a **CLI**, and **Python SDK** for direct interaction from a console and code: > _What is Azure Machine Learning CLI & Python SDK v2?_\ -> _https://docs.microsoft.com/en-us/azure/machine-learning/concept-v2_ +> &#xNAN;_https://docs.microsoft.com/en-us/azure/machine-learning/concept-v2_ The Azure ML CLI and Python SDK enable engineers the use [**MLOps**](https://en.wikipedia.org/wiki/MLOps) techniques. Similar to DevOps, MLOps is a set of practices that allows the reliable and efficient management of AI / ML application lifecycle. It enables processes like: @@ -280,7 +280,7 @@ The **Raspberry Pi 4** is a relatively low power single board computer, popular The **Raspberry Pi 4** can be set up the standard way. The Raspberry Pi OS is flashed to an SD Card, then we set up network connectivity / Wifi and SSH access. The official documentation describes in great details how to do this: > _**Setting up your Raspberry Pi**_\ -> _https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/0_ +> &#xNAN;_https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/0_ Next, there are couple steps to be done in order to connect the device to EdgeImpulse. The goal is to install the `edge-impulse-linux` utility, which can be done as follows: @@ -380,7 +380,7 @@ At this point our dataset is complete, and we can start building and training an ![](../.gitbook/assets/azure-machine-learning-EI/3-10-edgimpl-impulse.png) -The [**MFCC **_**(Mel Frequency Cepstral Coefficients)**_](https://docs.edgeimpulse.com/docs/edge-impulse-studio/processing-blocks/audio-mfcc) block extracts coefficients from an audio signal. For keyword spotting, training it with the default parameters usually works: +The [**MFCC \_(Mel Frequency Cepstral Coefficients)**\_](https://docs.edgeimpulse.com/docs/edge-impulse-studio/processing-blocks/audio-mfcc) block extracts coefficients from an audio signal. For keyword spotting, training it with the default parameters usually works: ![](../.gitbook/assets/azure-machine-learning-EI/3-11-edgimpl-mfcc.png)