From 4b4077b3b74001e608fb9e829b57bbe7b05ce9c6 Mon Sep 17 00:00:00 2001 From: Marcelo Rovai Date: Fri, 6 Oct 2023 10:33:15 -0300 Subject: [PATCH] Delete embedded_sys_exercise.qmd --- embedded_sys_exercise.qmd | 496 -------------------------------------- 1 file changed, 496 deletions(-) delete mode 100644 embedded_sys_exercise.qmd diff --git a/embedded_sys_exercise.qmd b/embedded_sys_exercise.qmd deleted file mode 100644 index 882eebb7..00000000 --- a/embedded_sys_exercise.qmd +++ /dev/null @@ -1,496 +0,0 @@ ---- -title: "[]{#_6ugxbtjss6rg .anchor}2 Embedded Systems" ---- - -# Exercise - The Nicla Vision - -**Introduction** - -The [Arduino Nicla -Vision](https://docs.arduino.cc/hardware/nicla-vision) (sometimes called -*NiclaV*) is a development board that includes two processors that can -run tasks in parallel. It is part of a family of development boards with -the same form factor but designed for specific tasks, such as the [Nicla -Sense -ME](https://www.bosch-sensortec.com/software-tools/tools/arduino-nicla-sense-me/) -and the [Nicla -Voice](https://store-usa.arduino.cc/products/nicla-voice?_gl=1*l3abc6*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY5NjM0Mzk1My4xMDIuMS4xNjk2MzQ0MjQ1LjAuMC4w). -The *Niclas* can efficiently run processes created with TensorFlow™ -Lite. For example, one of the cores of the NiclaV computing a computer -vision algorithm on the fly (inference), while the other leads with -low-level operations like controlling a motor and communicating or -acting as a user interface. - -> *The onboard wireless module allows the management of WiFi and -> Bluetooth Low Energy (BLE) connectivity simultaneously.* - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image29.jpg){width="6.5in" -height="3.861111111111111in"} - -### **Two Parallel Cores** - -The central processor is the dual-core -[STM32H747,](https://content.arduino.cc/assets/Arduino-Portenta-H7_Datasheet_stm32h747xi.pdf?_gl=1*6quciu*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY0NzQ0NTg1My4xMS4xLjE2NDc0NDYzMzkuMA..) -including a Cortex® M7 at 480 MHz and a Cortex® M4 at 240 MHz. The two -cores communicate via a Remote Procedure Call mechanism that seamlessly -allows calling functions on the other processor. Both processors share -all the on-chip peripherals and can run: - -- Arduino sketches on top of the Arm® Mbed™ OS - -- Native Mbed™ applications - -- MicroPython / JavaScript via an interpreter - -- TensorFlow™ Lite - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image22.jpg){width="5.78125in" -height="5.78125in"} - -### **Memory** - -Memory is crucial for embedded machine learning projects. The NiclaV -board can host up to 16 MB of QSPI Flash for storage. However, it is -essential to consider that the MCU SRAM is the one to be used with -machine learning inferences; the STM32H747 is only 1MB, shared by both -processors. This MCU also has incorporated 2MB of FLASH, mainly for code -storage. - -### **Sensors** - -- **Camera**: A GC2145 2 MP Color CMOS Camera. - -- **Microphone**: A - > [MP34DT05,](https://content.arduino.cc/assets/Nano_BLE_Sense_mp34dt05-a.pdf?_gl=1*12fxus9*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY0NzQ0NTg1My4xMS4xLjE2NDc0NDc3NzMuMA..) - > an ultra-compact, low-power, omnidirectional, digital MEMS - > microphone built with a capacitive sensing element and an IC - > interface. - -- **6-Axis IMU**: 3D gyroscope and 3D accelerometer data from the - > LSM6DSOX 6-axis IMU. - -- **Time of Flight Sensor**: The VL53L1CBV0FY Time-of-Flight sensor - > adds accurate and low power-ranging capabilities to the Nicla - > Vision. The invisible near-infrared VCSEL laser (including the - > analog driver) is encapsulated with receiving optics in an - > all-in-one small module below the camera. - -### **HW Installation (Arduino IDE)** - -Start connecting the board (USB-C) to your computer : - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image14.jpg){width="6.5in" -height="3.0833333333333335in"} - -Install the Mbed OS core for Nicla boards in the Arduino IDE. Having the -IDE open, navigate to Tools \> Board \> Board Manager, look for Arduino -Nicla Vision on the search window, and install the board. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image2.jpg){width="6.5in" -height="2.7083333333333335in"} - -Next, go to Tools \> Board \> Arduino Mbed OS Nicla Boards and select -Arduino Nicla Vision. Having your board connected to the USB, you should -see the Nicla on Port and select it. - -> *Open the Blink sketch on Examples/Basic and run it using the IDE -> Upload button. You should see the Built-in LED (green RGB) blinking, -> which means the Nicla board is correctly installed and functional!* - -### **Testing the Microphone** - -On Arduino IDE, go to Examples \> PDM \> PDMSerialPlotter, open and run -the sketch. Open the Plotter and see the audio representation from the -microphone: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image9.png){width="6.5in" -height="4.361111111111111in"} - -> *Vary the frequency of the sound you generate and confirm that the mic -> is working correctly.* - -### **Testing the IMU** - -Before testing the IMU, it will be necessary to install the LSM6DSOX -library. For that, go to Library Manager and look for LSM6DSOX. Install -the library provided by Arduino: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image19.jpg){width="6.5in" -height="2.4027777777777777in"} - -Next, go to Examples \> Arduino_LSM6DSOX \> SimpleAccelerometer and run -the accelerometer test (you can also run Gyro and board temperature): - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image28.png){width="6.5in" -height="4.361111111111111in"} - -### **Testing the ToF (Time of Flight) Sensor** - -As we did with IMU, installing the ToF library, the VL53L1X is -necessary. For that, go to Library Manager and look for VL53L1X. Install -the library provided by Pololu: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image15.jpg){width="6.5in" -height="2.4583333333333335in"} - -Next, run the sketch -[proximity_detection.ino](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/distance_image_meter.py): - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image12.png){width="4.947916666666667in" -height="4.635416666666667in"} - -On the Serial Monitor, you will see the distance from the camera and an -object in front of it (max of 4m). - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image13.jpg){width="6.5in" -height="4.847222222222222in"} - -### **Testing the Camera** - -We can also test the camera using, for example, the code provided on -Examples \> Camera \> CameraCaptureRawBytes. We can not see the image -directly, but it is possible to get the raw image data generated by the -camera. - -Anyway, the best test with the camera is to see a live image. For that, -we will use another IDE, the OpenMV. - -### **Installing the OpenMV IDE** - -OpenMV IDE is the premier integrated development environment for use -with OpenMV Cameras and the one on the Portenta. It features a powerful -text editor, debug terminal, and frame buffer viewer with a histogram -display. We will use MicroPython to program the camera. - -Go to the [OpenMV IDE page](https://openmv.io/pages/download), download -the correct version for your Operating System, and follow the -instructions for its installation on your computer. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image21.png){width="6.5in" -height="4.791666666666667in"} - -The IDE should open, defaulting the helloworld_1.py code on its Code -Area. If not, you can open it from Files \> Examples \> HelloWord \> -helloword.py - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image7.png){width="6.5in" -height="4.444444444444445in"} - -Any messages sent through a serial connection (using print() or error -messages) will be displayed on the **Serial Terminal** during run time. -The image captured by a camera will be displayed in the **Camera -Viewer** Area (or Frame Buffer) and in the Histogram area, immediately -below the Camera Viewer. - -OpenMV IDE is the premier integrated development environment with OpenMV -Cameras and the Arduino Pro boards. It features a powerful text editor, -debug terminal, and frame buffer viewer with a histogram display. We -will use MicroPython to program the Nicla Vision. - -> *Before connecting the Nicla to the OpenMV IDE, ensure you have the -> latest bootloader version. To that, go to your Arduino IDE, select the -> Nicla board, and open the sketch on Examples \> STM_32H747_System -> STM_32H747_updateBootloader. Upload the code to your board. The Serial -> Monitor will guide you.* - -After updating the bootloader, put the Nicla Vision in bootloader mode -by double-pressing the reset button on the board. The built-in green LED -will start fading in and out. Now return to the OpenMV IDE and click on -the connect icon (Left ToolBar): - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image23.jpg){width="4.010416666666667in" -height="1.0520833333333333in"} - -A pop-up will tell you that a board in DFU mode was detected and ask you -how you would like to proceed. First, select \"Install the latest -release firmware.\" This action will install the latest OpenMV firmware -on the Nicla Vision. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image10.png){width="6.5in" -height="2.6805555555555554in"} - -You can leave the option of erasing the internal file system unselected -and click \[OK\]. - -Nicla\'s green LED will start flashing while the OpenMV firmware is -uploaded to the board, and a terminal window will then open, showing the -flashing progress. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image5.png){width="4.854166666666667in" -height="3.5416666666666665in"} - -Wait until the green LED stops flashing and fading. When the process -ends, you will see a message saying, \"DFU firmware update complete!\". -Press \[OK\]. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image1.png){width="3.875in" -height="5.708333333333333in"} - -A green play button appears when the Nicla Vison connects to the Tool -Bar. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image18.jpg){width="4.791666666666667in" -height="1.4791666666666667in"} - -Also, note that a drive named "NO NAME" will appear on your computer.: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image3.png){width="6.447916666666667in" -height="2.4166666666666665in"} - -Every time you press the \[RESET\] button on the board, it automatically -executes the main.py script stored on it. You can load the -[main.py](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/main.py) -code on the IDE (File \> Open File\...). - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image16.png){width="4.239583333333333in" -height="3.8229166666666665in"} - -> *This code is the \"Blink\" code, confirming that the HW is OK.* - -For testing the camera, let\'s run helloword_1.py. For that, select the -script on File \> Examples \> HelloWorld \> helloword.py, - -When clicking the green play button, the MicroPython script -(hellowolrd.py) on the Code Area will be uploaded and run on the Nicla -Vision. On-Camera Viewer, you will start to see the video streaming. The -Serial Monitor will show us the FPS (Frames per second), which should be -around 14fps. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image6.png){width="6.5in" -height="3.9722222222222223in"} - -Let\'s go through the [helloworld.py](http://helloworld.py/) script: - - ----------------------------------------------------------------------- - \# Hello World Example\ - \#\ - \# Welcome to the OpenMV IDE! Click on the green run arrow button below - to run the script!\ - \ - import sensor, image, time\ - \ - sensor.reset() \# Reset and initialize the sensor.\ - sensor.set_pixformat(sensor.RGB565) \# Set pixel format to RGB565 (or - GRAYSCALE)\ - sensor.set_framesize(sensor.QVGA) \# Set frame size to QVGA (320x240)\ - sensor.skip_frames(time = 2000) \# Wait for settings take effect.\ - clock = time.clock() \# Create a clock object to track the FPS.\ - \ - while(True):\ - clock.tick() \# Update the FPS clock.\ - img = sensor.snapshot() \# Take a picture and return the image.\ - print(clock.fps()) - ----------------------------------------------------------------------- - - ----------------------------------------------------------------------- - -In GitHub, you can find the Python scripts used here. - -The code can be split into two parts: - -- **Setup**: Where the libraries are imported and initialized, and the - > variables are defined and initiated. - -- **Loop**: (while loop) part of the code that runs continually. The - > image (img variable) is captured (a frame). Each of those frames - > can be used for inference in Machine Learning Applications. - -To interrupt the program execution, press the red \[X\] button. - -> *Note: OpenMV Cam runs about half as fast when connected to the IDE. -> The FPS should increase once disconnected.* - -In [[the GitHub, You can find other Python -scripts]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main/Micropython). -Try to test the onboard sensors. - -### **Connecting the Nicla Vision to Edge Impulse Studio** - -We will use the Edge Impulse Studio later in other exercises. [Edge -Impulse I](https://www.edgeimpulse.com/)s a leading development platform -for machine learning on edge devices. - -Edge Impulse officially supports the Nicla Vision. So, for starting, -please create a new project on the Studio and connect the Nicla to it. -For that, follow the steps: - -- Download the [last EI - > Firmware](https://cdn.edgeimpulse.com/firmware/arduino-nicla-vision.zip) - > and unzip it. - -- Open the zip file on your computer and select the uploader related - > to your OS: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image17.png){width="4.416666666666667in" -height="1.5520833333333333in"} - -- Put the Nicla-Vision on Boot Mode, pressing the reset button twice. - -- Execute the specific batch code for your OS for uploading the binary - > (arduino-nicla-vision.bin) to your board. - -Go to your project on the Studio, and on the Data Acquisition tab, -select WebUSB (1). A window will appear; choose the option that shows -that the Nicla is pared (2) and press \[Connect\] (3). - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image27.png){width="6.5in" -height="4.319444444444445in"} - -In the Collect Data section on the Data Acquisition tab, you can choose -what sensor data you will pick. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image25.png){width="6.5in" -height="4.319444444444445in"} - -For example. IMU data: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image8.png){width="6.5in" -height="4.319444444444445in"} - -Or Image: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image4.png){width="6.5in" -height="4.319444444444445in"} - -And so on. You can also test an external sensor connected to the Nicla -ADC (pin 0) and the other onboard sensors, such as the microphone and -the ToF. - -### **Expanding the Nicla Vision Board (optional)** - -A last item to be explored is that sometimes, during prototyping, it is -essential to experiment with external sensors and devices, and an -excellent expansion to the Nicla is the [Arduino MKR Connector Carrier -(Grove -compatible)](https://store-usa.arduino.cc/products/arduino-mkr-connector-carrier-grove-compatible). - -The shield has 14 Grove connectors: five single analog inputs, one -single analog input, five single digital I/Os, one double digital I/O, -one I2C, and one UART. All connectors are 5V compatible. - -> *Note that besides all 17 Nicla Vision pins that will be connected to -> the Shield Groves, some Grove connections are disconnected.* - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image20.jpg){width="6.5in" -height="4.875in"} - -This shield is MKR compatible and can be used with the Nicla Vision and -the Portenta. - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image26.jpg){width="4.34375in" -height="5.78125in"} - -For example, suppose that on a TinyML project, you want to send -inference results using a LoRaWan device and add information about local -luminosity. Besides, with offline operations, a local low-power display -as an OLED display is advised. This setup can be seen here: - -![image.png](vertopal_b438acb27fb24e6a9820998b0b314503/media/image11.jpg){width="6.5in" -height="4.708333333333333in"} - -The [Grove Light -Sensor](https://wiki.seeedstudio.com/Grove-Light_Sensor/) would be -connected to one of the single Analog pins (A0/PC4), the [LoRaWan -device](https://wiki.seeedstudio.com/Grove_LoRa_E5_New_Version/) to the -UART, and the [OLED](https://arduino.cl/producto/display-oled-grove/) to -the I2C connector. - -The Nicla Pins 3 (Tx) and 4 (Rx) are connected with the Shield Serial -connector. The UART communication is used with the LoRaWan device. Here -is a simple code to use the UART.: - - ----------------------------------------------------------------------- - \# UART Test - By: marcelo_rovai - Sat Sep 23 2023\ - \ - import time\ - from pyb import UART\ - from pyb import LED\ - \ - redLED = LED(1) \# built-in red LED\ - \ - \# Init UART object.\ - \# Nicla Vision\'s UART (TX/RX pins) is on \"LP1\"\ - uart = UART(\"LP1\", 9600)\ - \ - while(True):\ - uart.write(\"Hello World!\\r\\n\")\ - redLED.toggle()\ - time.sleep_ms(1000) - ----------------------------------------------------------------------- - - ----------------------------------------------------------------------- - -To verify if the UART is working, you should, for example, connect -another device as an [Arduino -UNO](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Arduino-IDE/teste_uart_UNO/teste_uart_UNO.ino), -displaying the Hello Word. - -![B3D78F51-83F9-413D-8BF9-ED46FAA82F49.GIF](vertopal_b438acb27fb24e6a9820998b0b314503/media/image24.gif){width="2.8125in" -height="3.75in"} - -Here is a Hello World code to be used with the I2C OLED. The MicroPython -SSD1306 OLED driver (ssd1306.py), created by Adafruit, should also be -uploaded to the Nicla (the -[[ssd1306.py]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/ssd1306.py) -can be found in GitHub). - - ----------------------------------------------------------------------- - \# Nicla_OLED_Hello_World - By: marcelo_rovai - Sat Sep 30 2023\ - \ - #Save on device: MicroPython SSD1306 OLED driver, I2C and SPI - interfaces created by Adafruit\ - import ssd1306\ - \ - from machine import I2C\ - i2c = I2C(1)\ - \ - oled_width = 128\ - oled_height = 64\ - oled = ssd1306.SSD1306_I2C(oled_width, oled_height, i2c)\ - \ - oled.text(\'Hello, World\', 10, 10)\ - oled.show() - ----------------------------------------------------------------------- - - ----------------------------------------------------------------------- - -Finally, here is a simple script to read the ADC value on pin \"PC4\" -(Nicla pin A0): - - ----------------------------------------------------------------------- - \# Light Sensor (A0) - By: marcelo_rovai - Wed Oct 4 2023\ - \ - import pyb\ - from time import sleep\ - \ - adc = pyb.ADC(pyb.Pin(\"PC4\")) \# create an analog object from a pin\ - val = adc.read() \# read an analog value\ - \ - while (True):\ - \ - val = adc.read()\ - print (\"Light={}\".format (val))\ - sleep (1) - ----------------------------------------------------------------------- - - ----------------------------------------------------------------------- - -The ADC can be used for other valuable sensors, such as -[Temperature](https://wiki.seeedstudio.com/Grove-Temperature_Sensor_V1.2/). - -> *Note that the above scripts ([[downloaded from -> Github]{.underline}](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main/Micropython)) -> only introduce how to connect external devices with the Nicla Vision -> board using MicroPython.* - -### **Conclusion** - -The Arduino Nicla Vision is an excellent *tiny device* for industrial -and professional uses! However, it is powerful, trustworthy, low power, -and has suitable sensors for the most common embedded machine learning -applications such as vision, movement, sensor fusion, and sound. - -> *On the* *[GitHub -> repository,](https://github.com/Mjrovai/Arduino_Nicla_Vision/tree/main) -> you will find the last version of all the codes used or commented on -> in this exercise.*