Skip to content

Commit

Permalink
GITBOOK-18: David Dec 18 reorg
Browse files Browse the repository at this point in the history
  • Loading branch information
dtischler authored and gitbook-bot committed Dec 19, 2024
1 parent cfc15d5 commit 58b0823
Show file tree
Hide file tree
Showing 106 changed files with 1,390 additions and 1,389 deletions.
196 changes: 98 additions & 98 deletions README.md

Large diffs are not rendered by default.

384 changes: 183 additions & 201 deletions SUMMARY.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions accelerometer-and-activity-projects/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Accelerometer and Activity Projects

2 changes: 2 additions & 0 deletions air-quality-and-environmental-projects/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Air Quality and Environmental Projects

2 changes: 2 additions & 0 deletions audio-projects/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Audio Projects

46 changes: 24 additions & 22 deletions audio-projects/synthetic-data-pipeline-keyword-spotting.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
description: >-
End-to-end synthetic data pipeline for the creation of a portable LED product equipped with keyword spotting capabilities. The project serves as a comprehensive guide for development of any KWS product, emphasizing the utilization of synthetic data for model training.
End-to-end synthetic data pipeline for the creation of a portable LED product
equipped with keyword spotting capabilities. The project serves as a
comprehensive guide for development of any KWS produc
---

# Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline
Expand Down Expand Up @@ -41,14 +43,14 @@ The Arduino Nicla Voice is an ideal choice for this project due to its use of th

## Hardware Requirements

- Arduino Nicla Voice (or other Edge Impulse supported MCU with mic)
- PCB and SMD components (parts breakdown explained later)
* Arduino Nicla Voice (or other Edge Impulse supported MCU with mic)
* PCB and SMD components (parts breakdown explained later)

## Software Requirements

- Edge Impulse CLI
- Arduino IDE
- OpenAI API account
* Edge Impulse CLI
* Arduino IDE
* OpenAI API account

## Dataset Collection

Expand All @@ -64,9 +66,9 @@ Once you have your secret key, you can navigate to your Edge Impulse organizatio

### Generating TTS Synthetic Data

Now that we have the environment configured, and our OpenAI API saved in the Edge Impulse Studio, we are ready to start a new project and begin generating some synthetic voice data.
Now that we have the environment configured, and our OpenAI API saved in the Edge Impulse Studio, we are ready to start a new project and begin generating some synthetic voice data.

On your project's page select Data acquisition --> Data sources --> + Add new data source --> Transformation block --> Whisper Synthetic Voice Generator --> Fill out the details as follow:
On your project's page select Data acquisition --> Data sources --> + Add new data source --> Transformation block --> Whisper Synthetic Voice Generator --> Fill out the details as follow:

![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/generate-tts.jpg)

Expand Down Expand Up @@ -104,14 +106,14 @@ Once satisfied with all the data generated, perform a Train / Test split into ap

![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/impulse-design.png)

The Impulse design values are chosen for optimal keyword spotting performance. The 968 ms window size captures enough audio for accurate detection, while the 500 ms window increase balances responsiveness and efficiency. The 16000 Hz frequency is standard for capturing human voice, ensuring quality without excessive data. Using the Audio (Syntiant) block leverages the NDP120's capabilities for efficient digital signal processing. The Classification block distinguishes between commands, with output classes "extinguish," "illuminate," and "z_openset" allowing for control of the lighting system and handling unknown inputs.
The Impulse design values are chosen for optimal keyword spotting performance. The 968 ms window size captures enough audio for accurate detection, while the 500 ms window increase balances responsiveness and efficiency. The 16000 Hz frequency is standard for capturing human voice, ensuring quality without excessive data. Using the Audio (Syntiant) block leverages the NDP120's capabilities for efficient digital signal processing. The Classification block distinguishes between commands, with output classes "extinguish," "illuminate," and "z\_openset" allowing for control of the lighting system and handling unknown inputs.

- Window size: 968 ms
- Window increase: 500 ms
- Frequency: 16000 Hz
- Audio (Syntiant)
- Classification
- Output: extinguish, illuminate, z_openset
* Window size: 968 ms
* Window increase: 500 ms
* Frequency: 16000 Hz
* Audio (Syntiant)
* Classification
* Output: extinguish, illuminate, z\_openset

![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/processing-feature.png)

Expand Down Expand Up @@ -149,13 +151,13 @@ The schematic, pcb, and gerber (manufacturing) files are accessible in the proje

![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/pcb-order.png)

| LCSC Part Number | Manufacture Part Number | Manufacturer | Package | Description | Order Qty. | Unit Price($) | Order Price($) |
|--------------------|----------------------------------------------------------|--------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------|------------|---------------|----------------|
| C176224 | QR1206F5R10P05Z | Ever Ohms Tech | 1206 | 250mW Thick Film Resistors 200V ą1% ą400ppm/? 5.1? 1206 Chip Resistor - Surface Mount ROHS | 50 | 0.0156 | 0.78 |
| C516126 | HL-AM-2835H421W-S1-08-HR5(R9) (2800K-3100K)(SDCM<6,R9>50) | HONGLITRONIC(Hongli Zhihui (HONGLITRONIC)) | SMD2835 | 60mA 3000K Foggy yellow lens -40?~+85? Positive Stick White 120° 306mW 3.4V SMD2835 LED Indication - Discrete ROHS | 50 | 0.0144 | 0.72 |
| C2589 | IRLML2502TRPBF | Infineon Technologies | SOT-23 | 20V 4.2A 1.25W 45m?@4.5V,4.2A 1.2V@250uA 1 N-Channel SOT-23 MOSFETs ROHS | 5 | 0.1838 | 0.92 |
| C5440143 | CS3225X7R476K160NRL | Samwha Capacitor | 1210 | 16V 47uF X7R ą10% 1210 Multilayer Ceramic Capacitors MLCC - SMD/SMT ROHS | 5 | 0.0765 | 0.38 |
| C153338 | FCR1206J100RP05Z | Ever Ohms Tech | 1206 | 250mW Safety Resistor 200V ą5% 100? 1206 Chip Resistor - Surface Mount ROHS | 10 | 0.0541 | 0.54 |
| LCSC Part Number | Manufacture Part Number | Manufacturer | Package | Description | Order Qty. | Unit Price($) | Order Price($) |
| ---------------- | --------------------------------------------------------- | ------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------- | ---------- | ------------- | -------------- |
| C176224 | QR1206F5R10P05Z | Ever Ohms Tech | 1206 | 250mW Thick Film Resistors 200V ą1% ą400ppm/? 5.1? 1206 Chip Resistor - Surface Mount ROHS | 50 | 0.0156 | 0.78 |
| C516126 | HL-AM-2835H421W-S1-08-HR5(R9) (2800K-3100K)(SDCM<6,R9>50) | HONGLITRONIC(Hongli Zhihui (HONGLITRONIC)) | SMD2835 | 60mA 3000K Foggy yellow lens -40?\~+85? Positive Stick White 120° 306mW 3.4V SMD2835 LED Indication - Discrete ROHS | 50 | 0.0144 | 0.72 |
| C2589 | IRLML2502TRPBF | Infineon Technologies | SOT-23 | 20V 4.2A 1.25W 45m?@4.5V,4.2A 1.2V@250uA 1 N-Channel SOT-23 MOSFETs ROHS | 5 | 0.1838 | 0.92 |
| C5440143 | CS3225X7R476K160NRL | Samwha Capacitor | 1210 | 16V 47uF X7R ą10% 1210 Multilayer Ceramic Capacitors MLCC - SMD/SMT ROHS | 5 | 0.0765 | 0.38 |
| C153338 | FCR1206J100RP05Z | Ever Ohms Tech | 1206 | 250mW Safety Resistor 200V ą5% 100? 1206 Chip Resistor - Surface Mount ROHS | 10 | 0.0541 | 0.54 |

![](../.gitbook/assets/synthetic-data-pipeline-keyword-spotting/3d-pcb.png)

Expand Down
2 changes: 2 additions & 0 deletions computer-vision-projects/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Computer Vision Projects

Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
---
description: >-
Use the "Bring Your Own Model" feature to upload an existing pre-trained TensorFlow Lite Sign Language model into Edge Impulse, then deploy it to the TI AM62A Development Kit.
Use the "Bring Your Own Model" feature to upload an existing pre-trained
TensorFlow Lite Sign Language model into Edge Impulse, then deploy it to the
TI AM62A Development Kit.
---

# Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A
Expand All @@ -25,13 +27,13 @@ I decided to try out this feature by importing an American Sign Language model a

![](../.gitbook/assets/asl-byom-ti-am62a/asl.jpg)

One thing I learned from this project is that sign language is not the same everywhere. In the US and most of Canada, they use American Sign Language, but in Argentina where I am located, they use a different version. Both versions have some signs that are static, and some that have movement, like the letter Z. For the moving signs, I only used the final position of the movement to train and recognize the letter, since I was using images and not videos. For motion activity, video classification would be needed.
One thing I learned from this project is that sign language is not the same everywhere. In the US and most of Canada, they use American Sign Language, but in Argentina where I am located, they use a different version. Both versions have some signs that are static, and some that have movement, like the letter Z. For the moving signs, I only used the final position of the movement to train and recognize the letter, since I was using images and not videos. For motion activity, video classification would be needed.

Another thing to keep in mind is that some signs, like **A, M, N, S and T**, can look very similar from a distance. This makes data collection and model training more challenging.

## Model Import

From the original idea to the point where I actually started the project, the [ASL Hugging Face model]((https://huggingface.co/spaces/datasciencedojo/AmericanSignLanguage-Detection)) I wanted to use was no longer available.
From the original idea to the point where I actually started the project, the [ASL Hugging Face model](../image-projects/\(https:/huggingface.co/spaces/datasciencedojo/AmericanSignLanguage-Detection\)/) I wanted to use was no longer available.

So at this point I decided to train a simple ASL model outside Edge Impulse, export the model as a TensorFlow Lite file, import to Edge Impulse, deploy the model to the Texas Instruments AM62A board, and develop a simple Python training game.

Expand All @@ -51,7 +53,7 @@ Before importing to Edge Impulse, I tested the model with a live feed from the c

For the exporting step in _Teachable Machine_, I used TensorFlow Lite, Floating Point. That generates a `.zip` file download, which I then unzipped. Inside, I found 2 files: the model with the `.tflite` extension, and a label .txt file. This label file has 2 columns: **order** and **label name**. To prepare everything for Edge Impulse BYOM, I removed the **order** column and compiled everything in one row, comma separated.

Example:
Example:

```
A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z
Expand Down Expand Up @@ -91,16 +93,16 @@ The deployment procedure varies from board to board. In this case, the deploymen

## Texas Instruments AM62A Setup and Deployment

- Download this operating system image version: [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM62A/08.06.00.45](https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-D37Ls3JjkT/08.06.00.45/tisdk-edgeai-image-am62axx-evm.wic.xz)
- Flash the image to a 16gb or larger microSD card with Balena Etcher or any other similar software
- Connect the Power Supply, HDMI, USB Camera, and Ethernet Cable
- Check the board IP on the HDMI screen when board boots up and the default application loads
- Login to that IP using Putty or any other SSH client, using `root` as the user, and no password
- Run `npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm`
- Run `pip3 install art` (a library to display bigger letters)
- Run `edge-impulse-linux-runner`. The first time you run this, you will need to login to your Edge Impulse account and select the BYOM project. Once running, launch a web browser and navigate to your board's IP address, port 4912. For example, `http://192.168.1.66:4912` in my case.
- Download the `am62a_signlanguage.py` file [from the GitHub repository](https://github.com/ronibandini/ASLTrainer) and upload the script to the AM62A board using SFTP. The credentials are the same as logging in directly: You'll need your IP address, username is `root`, and there is no password.
- Run `python3 am62a_signlanguage.py`
* Download this operating system image version: [https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM62A/08.06.00.45](https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-D37Ls3JjkT/08.06.00.45/tisdk-edgeai-image-am62axx-evm.wic.xz)
* Flash the image to a 16gb or larger microSD card with Balena Etcher or any other similar software
* Connect the Power Supply, HDMI, USB Camera, and Ethernet Cable
* Check the board IP on the HDMI screen when board boots up and the default application loads
* Login to that IP using Putty or any other SSH client, using `root` as the user, and no password
* Run `npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm`
* Run `pip3 install art` (a library to display bigger letters)
* Run `edge-impulse-linux-runner`. The first time you run this, you will need to login to your Edge Impulse account and select the BYOM project. Once running, launch a web browser and navigate to your board's IP address, port 4912. For example, `http://192.168.1.66:4912` in my case.
* Download the `am62a_signlanguage.py` file [from the GitHub repository](https://github.com/ronibandini/ASLTrainer) and upload the script to the AM62A board using SFTP. The credentials are the same as logging in directly: You'll need your IP address, username is `root`, and there is no password.
* Run `python3 am62a_signlanguage.py`

## Training Game

Expand Down Expand Up @@ -130,21 +132,22 @@ It is important to note that Edge Impulse's visualizations and tuning for extern

### Files

- Source Code: [https://github.com/ronibandini/ASLTrainer](https://github.com/ronibandini/ASLTrainer)
- Edge Impulse Public Project: [https://studio.edgeimpulse.com/public/270046/latest](https://studio.edgeimpulse.com/public/270046/latest)
- [https://huggingface.co/ronibandini/AmericanSignLanguage](https://huggingface.co/ronibandini/AmericanSignLanguage) (this model has all the letters)
* Source Code: [https://github.com/ronibandini/ASLTrainer](https://github.com/ronibandini/ASLTrainer)
* Edge Impulse Public Project: [https://studio.edgeimpulse.com/public/270046/latest](https://studio.edgeimpulse.com/public/270046/latest)
* [https://huggingface.co/ronibandini/AmericanSignLanguage](https://huggingface.co/ronibandini/AmericanSignLanguage) (this model has all the letters)

### References

[https://teachablemachine.withgoogle.com](https://teachablemachine.withgoogle.com)

### Contact

If you are interested in other Artificial Intelligence and Machine Learning projects: [https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP](https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP)
If you are interested in other Artificial Intelligence and Machine Learning projects: [https://www.youtube.com/playlist?list=PLIw\_UcVWFyBVYAQXp8S2pfe2frzGcyxlP](https://www.youtube.com/playlist?list=PLIw_UcVWFyBVYAQXp8S2pfe2frzGcyxlP)

#### Social Media
[https://twitter.com/RoniBandini](https://twitter.com/RoniBandini)
[https://www.instagram.com/ronibandini](https://www.instagram.com/ronibandini)

#### Web
[https://twitter.com/RoniBandini](https://twitter.com/RoniBandini) [https://www.instagram.com/ronibandini](https://www.instagram.com/ronibandini)

#### Web

[https://bandini.medium.com/](https://bandini.medium.com/)
Loading

0 comments on commit 58b0823

Please sign in to comment.