diff --git a/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md b/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md index a2e9dcc..dd5b46b 100644 --- a/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md +++ b/nvidia-projects/rooftop-ice-synthetic-data-omniverse.md @@ -1,6 +1,6 @@ --- description: >- - Rooftop ice buildup detection using Edge Impulse, with synthetic data created with NVIDIA Omniverse Replicator and sun studies. + Tracking Rooftop ice buildup detection using Edge Impulse and The Things Network, with synthetic data created in NVIDIA Omniverse Replicator and Sun Studies. --- # Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator @@ -15,7 +15,7 @@ GitHub Repo: [https://github.com/eivholt/icicle-monitor](https://github.com/eivh ## Introduction -This portable device monitors buildings and warns the responsible parties when potentially hazardous icicles are formed. In ideal conditions icicles can form at a rate of [more than 1 cm (0.39 in) per minute](https://en.wikipedia.org/wiki/Icicle). Each year, many people are injured and killed by these solid projectiles, leading responsible building owners to often close sidewalks in the spring to minimize risk. This project demonstrates how an extra set of digital eyes can notify property owners icicles are forming and need to be removed before they can cause harm. +The portable device created in this project monitors buildings and warns the responsible parties when potentially hazardous icicles are formed. In ideal conditions, icicles can form at a rate of [more than 1 cm (0.39 in) per minute](https://en.wikipedia.org/wiki/Icicle). In cold climates, many people are injured and killed each year by these solid projectiles, leading responsible building owners to often close sidewalks in the spring to minimize risk. This project demonstrates how an extra set of digital eyes can notify property owners icicles are forming and need to be removed before they can cause harm. ![Downtown, photo: Avisa Nordland](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/IMG_8710.jpg) @@ -40,7 +40,7 @@ Project [Impulse](https://studio.edgeimpulse.com/public/332581/live) and [Github ## Working principle -Icicle formation is detected using a neural network (NN) designed to identify objects in images from the onboard camera. The NN is trained and tested exclusively on synthesized images. The images are generated with realistic simulated lighting conditions. A small amount of real images are used to verify the model. +Icicle formation is detected using a neural network (NN) designed to identify objects in images from the onboard camera. The NN is trained and tested **exclusively** on synthesized images. The images are generated with realistic simulated lighting conditions. A small amount of real images are used to later verify the model. {% embed url="https://youtube.com/shorts/aIkj3uZ_MSE" %} @@ -56,19 +56,17 @@ A powerful platform combined with a high resolution camera with fish-eye lens wo ![Arduino Portenta H7](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/20240413_215105_.jpg) -## Object detection using neural network +## Object detection using a neural network [FOMO (Faster Objects, More Objects)](https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/object-detection/fomo-object-detection-for-constrained-devices) is a novel machine learning algorithm that allows for visual object detection on highly constrained devices through training of a neural network with a number of convolutional layers. -![Edge Impulse](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EILogo.svg) - ### Capturing training data and labeling objects -One of the most labor intensive aspects of building any machine learning model is gathering the training data and to label it. For an object detection model this requires taking hundreds or thousands of images of the objects to detect, drawing rectangles around them and choosing the correct label for each class. Recently generating pre-labeled images has become feasible and has proven great results. This is referred to as synthetic data generation with domain randomization. In this project a model will be trained exclusively on synthetic data and we will see how it can detect the real life counterparts. +One of the most labor intensive aspects of building any machine learning model is gathering the training data and labeling it. For an object detection model this requires taking hundreds or thousands of images of the objects to detect, drawing rectangles around them, and choosing the correct label for each class. Recently generating pre-labeled images has become feasible and has proven to have great results. This is referred to as **synthetic data generation with domain randomization**. In this project a model will be trained exclusively on synthetic data, and we will see how it can detect the real life counterparts. ### Domain randomization using NVIDIA Omniverse Replicator -NVIDIA Omniverse Code is an IDE that allows us to compose 3D scenes and to write simple Python code to capture images. Further, the extension Replicator is a toolkit that allows us to label the objects in the images and to simplify common domain randomization tasks, such as scattering objects between images. For an in-depth walkthrough on getting started with Omniverse and Replicator [see this article](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/surgery-inventory-synthetic-data). +NVIDIA Omniverse Code is an IDE that allows us to compose 3D scenes and to write simple Python code to capture images. Further, the Replicator extension is a toolkit that allows us to label the objects in the images and to simplify common domain randomization tasks, such as scattering objects between images. For an in-depth walkthrough on getting started with Omniverse and Replicator, [see this associated article](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/surgery-inventory-synthetic-data). ### Making a scene @@ -78,11 +76,11 @@ It's possible to create an empty scene in Omniverse and add content programmatic ### Icicle models -To represent the icicle a high quality model pack was purchased at [Turbo Squid](https://www.turbosquid.com/3d-models/). +To represent the icicle, a high quality model pack was purchased at [Turbo Squid](https://www.turbosquid.com/3d-models/). ![3D icicle models purchased at Turbo Squid](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/turbo-squid-icicle.png) -To be able to import the models into Omniverse and Isaac Sim all models have to be converted to [OpenUSD-format](https://developer.nvidia.com/usd). While USD is a great emerging standard for describing, composing, simulating and collaborting within 3D-worlds, it is not yet commonly supported in asset marketplaces. [This article](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/surgery-inventory-synthetic-data) outlines considerations when performing conversion using Blender to USD. Note that it is advisable to export each individual model and to choose a suitable origin/pivot point. +To be able to import the models into Omniverse and Isaac Sim, all models have to be converted to [OpenUSD-format](https://developer.nvidia.com/usd). While USD is a great emerging standard for describing, composing, simulating, and collaborting within 3D worlds, it is not yet commonly supported in asset marketplaces. [This article](https://docs.edgeimpulse.com/experts/featured-machine-learning-projects/surgery-inventory-synthetic-data) outlines considerations when performing conversion using Blender to USD. Note that it is advisable to export each individual model and to choose a suitable origin/pivot point. Blender change origin cheat sheet: + Select vertex on model (Edit Mode), Shift+S-> Cursor to selected @@ -99,18 +97,18 @@ Tip for export: ### Setting semantic metadata on objects -To be able to produce images for training and include labels we can use a feature of Replicator toolbox found under menu Replicator> Semantics Schema Editor. +To be able to produce images for training and include labels, we can use a feature of Replicator toolbox found under menu Replicator > Semantics Schema Editor. ![Semantics Schema Editor](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/semantic-editor.png) -Here we can select each top node representing an item for object detection and adding a key-value pair. Choosing "class" as Semantic Type and "ice" as Semantic Data enables us to export this string as label later. +Here we can select each top node representing an item for object detection and add a key-value pair. Choosing "class" as Semantic Type and "ice" as Semantic Data enables us to export this string as a label later. ### Creating a program for domain randomization -With a basic 3D stage created and objects of interest labeled we can continue creating a program that will make sure we produce images with slight variations. Our program can be named anything, ending in .py and preferably placed close to the stage USD-file. The following is a description of such a program [replicator_init.py](https://github.com/eivholt/icicle-monitor/blob/main/omniverse-replicator/replicator_init.py): +With a basic 3D stage created and objects of interest labeled, we can continue creating a program that will make sure we produce images with slight variations. Our program can be named anything, ending in `.py` and preferably placed close to the stage USD-file. Here is a sample of such a program: [replicator_init.py](https://github.com/eivholt/icicle-monitor/blob/main/omniverse-replicator/replicator_init.py): -To keep the items generated in our script separate from the manually created content we start by creating a new layer in the 3D stage: +To keep the items generated in our script separate from the manually created content, we start by creating a new layer in the 3D stage: ```python with rep.new_layer(): @@ -145,7 +143,7 @@ with rep.trigger.on_frame(num_frames=10000, rt_subframes=50): The parameter *num_frames* specifies the desired number of renders. The *rt_subframes* parameter allows the rendering process to advance a set number of frames before the result is captured and saved to disk. A higher setting enhances complex ray tracing effects like reflections and translucency by giving them more time to interact across surfaces, though it increases rendering time. Each randomization routine is invoked with the option to include specific parameters. -To save each image and its corresponding semantic data, we utilize a designated API. While customizing the writer was considered, attempts to do so using Replicator version 1.9.8 on Windows led to errors. Therefore, we are employing the "BasicWriter" and will develop an independent script to generate a label format that is compatible with the Edge Impulse. +To save each image and its corresponding semantic data, we utilize a designated API. While customizing the writer was considered, attempts to do so using Replicator version 1.9.8 on Windows led to errors. Therefore, we are employing the "BasicWriter" and will develop an independent script to generate a label format that is compatible with Edge Impulse. ```python writer = rep.WriterRegistry.get("BasicWriter") @@ -158,13 +156,13 @@ writer.attach([render_product]) asyncio.ensure_future(rep.orchestrator.step_async()) ``` -*rgb* indicates that we want to save images to disk as png-files. Note that labels are created setting *bounding_box_2d_loose*. This is used in this case instead of *bounding_box_2d_tight* as the latter in some cases would not include the tip of the icicles in the resulting bounding box. It also creates labels from previously defined semantics. The code ends with running a single iteration of the process in Omniverse Code, so we can preview the results. +*rgb* indicates that we want to save images to disk as `.png` files. Note that labels are created setting *bounding_box_2d_loose*. This is used in this case instead of *bounding_box_2d_tight* as the latter in some cases would not include the tip of the icicles in the resulting bounding box. It also creates labels from the previously defined semantics. The code ends with running a single iteration of the process in Omniverse Code, so we can preview the results. The bounding boxes can be visualized by clicking the sensor widget, checking "BoundingBox2DLoose" and finally "Show Window". ![Omniverse bounding box](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/omniverse-bb.png) -Now we can implement the randomization logic. First a method that flips and scatters the icicles on a defined plane. +Now we can implement the randomization logic. First we'll use a method that flips and scatters the icicles on a defined plane. ```python def scatter_ice(icicles): @@ -181,7 +179,7 @@ with icicles: return icicles.node ``` -Next a method that randomly places the camera on an other defined plane and makes sure the camera is pointing at the group of icicles and randomizes focus. +Next a method that randomly places the camera on another defined plane, and makes sure the camera is pointing at the group of icicles and randomizes focus. ```python def randomize_camera(targets): @@ -196,17 +194,17 @@ We can define the methods in any order we like, but in *rep.trigger.on_frame* it ### Running domain randomization -With a basic randomization program in place, we could run it from the embedded script editor (Window> Script Editor), but more robust Python language support can be achieved by developing in Visual Studio Code instead. To connect VS Code with Omniverse we can use the Visual Studio Code extension [Embedded VS Code for NVIDIA Omniverse](https://marketplace.visualstudio.com/items?itemName=Toni-SM.embedded-vscode-for-nvidia-omniverse). See [extension repo](https://github.com/Toni-SM/semu.misc.vscode) for setup. When ready to run go to Replicator> Start and check progress in the defined output folder. +With a basic randomization program in place, we could run it from the embedded script editor (Window > Script Editor), but more robust Python language support can be achieved by developing in Visual Studio Code instead. To connect VS Code with Omniverse we can use the Visual Studio Code extension [Embedded VS Code for NVIDIA Omniverse](https://marketplace.visualstudio.com/items?itemName=Toni-SM.embedded-vscode-for-nvidia-omniverse). See the [extension repo](https://github.com/Toni-SM/semu.misc.vscode) for setup. When ready to run go to Replicator > Start and check progress in the defined output folder. ![Produced images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output1.png) ### Randomizing colors -The surface behind the icicles may vary greatly, both in color and texture. Using Replicator randomizing the color of an objects material is easy. +The surface behind the icicles may vary greatly, both in color and texture. Using Replicator randomizing the color of an object's material is easy. -In the scene in Omniverse either manually create a plane behind the icicles, or create one programmatically. +In the scene in Omniverse, either manually create a plane behind the icicles, or create one programmatically. -In code, define a function that takes in a reference to the plane we want to randomize the color of and use one of the distribution functions with min and max value span: +In Code, define a function that takes in a reference to the plane we want to randomize, the color of the distribution functions with min and max value span: ```python def randomize_screen(screen): @@ -286,27 +284,27 @@ We could instead generate textures with random shapes and colors. Either way, th These are rather unsophisticated approaches. More realistic results would be achieved by changing the [materials](https://docs.omniverse.nvidia.com/materials-and-rendering/latest/materials.html) of the actual walls of the house used as background. Omniverse has a large selection of available materials available in the NVIDIA Assets browser, allowing us to randomize a [much wider range of aspects](https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/randomizer_details.html) of the rendered results. -### Creating realistic outdoor lighting conditions using sun studies +### Creating realistic outdoor lighting conditions using Sun Study In contrast to a controlled indoor environment, creating a robust object detection model intended for outdoor use needs training images with a wide range of realistic natural light. When generating synthetic images we can utilize an [extension that approximates real world sunlight](https://docs.omniverse.nvidia.com/extensions/latest/ext_sun-study.html) based on sun studies. {% embed url="https://youtu.be/MRD-oAxaV8w" %} -The extension let's us set world location, date and time. We can also mix this with the Environment setting in Omniverse, allowing for a wide range of simulation of clouds, proper [Koyaanisqatsi](https://www.youtube.com/watch?v=tDW-1JIa2gI). As of March 2024 it is not easy to randomize these parameters in script, but this [is likely to change](https://forums.developer.nvidia.com/t/randomize-time-of-day-in-dynamic-sky/273833/9). In the mean time we can set the parameters, generate a few thousand images, change time of day, generate more images and so on. +The extension let's us set world location, date and time. We can also mix this with the Environment setting in Omniverse, allowing for a wide range of simulation of clouds. As of March 2024 it is not easy to randomize these parameters in script, but this [is likely to change](https://forums.developer.nvidia.com/t/randomize-time-of-day-in-dynamic-sky/273833/9). In the mean time we can set the parameters, generate a few thousand images, change time of day, generate more images and so on. {% embed url="https://youtu.be/qvDXRqBxECo" %} -![Sun study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output5.png) +![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output5.png) -![Sun study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output4.png) +![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output4.png) -![Sun study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output6.png) +![Sun Study](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/output6.png) ### Creating label file for Edge Impulse Studio -Edge Impulse Studio supports a wide range of image labeling formats for object detection. The output from Replicator's BasicWriter needs to be transformed so it can be uploaded either through the web interface or via [web-API](https://docs.edgeimpulse.com/reference/ingestion-api#ingestion-api). +Edge Impulse Studio supports a wide range of image labeling formats for object detection. The output from Replicator's BasicWriter needs to be transformed so it can be uploaded either through the web interface or via [the Ingestion API](https://docs.edgeimpulse.com/reference/ingestion-api#ingestion-api). -Provided is a simple Python program, [basic_writer_to_pascal_voc.py](https://github.com/eivholt/icicle-monitor/blob/main/scripts/basic_writer_to_pascal_voc.py). [Documentation on EI label formats](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from shell with +Provided is a simple Python program, [basic_writer_to_pascal_voc.py](https://github.com/eivholt/icicle-monitor/blob/main/scripts/basic_writer_to_pascal_voc.py) to help get started. Documentation on the supported Edge Impulse [label formats is located here](https://docs.edgeimpulse.com/docs/edge-impulse-studio/data-acquisition/uploader#understanding-image-dataset-annotation-formats). Run the program from a terminal with: ``` python basic_writer_to_pascal_voc.py @@ -332,37 +330,37 @@ Since we have generated both synthetic images and labels, we can use the [CLI to edge-impulse-uploader --category split --directory [folder] ``` -to connect to account and project and upload image files and labels in `bounding_boxes.labels`. To switch project first do: +to connect to your account and project, and upload the image files and labels in `bounding_boxes.labels`. To switch project if necessary, first run: ``` edge-impulse-uploader --clean ``` -At any time we can find "Perform train/test split" under "Danger zone" in project dashboard to distribute images between training/testing in a 80/20 split. +At any time we can find "Perform train/test split" under "Danger zone" in project dashboard, to distribute images between training/testing in a 80/20 split. ### Model training and performance Since our synthetic training images are based on both individual and two different sized clusters of icicles, we can't trust the model performance numbers too much. Greater F1 scores are better, but we will never achieve 100%. Still, we can upload increasing numbers of labeled images and observe how performance numbers increase. -2000 images: +2,000 images: ![2000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/2000-images.png) -6000 images: +6,000 images: ![6000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/6000-images-120cycles.png) -14000 images: +14,000 images: ![14000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/14000-images-120cycles_no-opt.png) -26000 images: +26,000 images: ![26000 images](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/26000-images-light-5000coco-120cycles_no-opt.png) Note that the final results include 5000 images from the [COCO 2017 dataset](https://cocodataset.org/#download). Adding this reduces F1 score a bit, but results in a model with significantly less overfitting, that shows almost no false positives when classifying random background scenes. -If we look at results from model testing in Edge Impulse Studio at first glance the numbers are less than impressive. +If we look at results from model testing in Edge Impulse Studio, at first glance the numbers are less than impressive. ![Model testing](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/model-testing1.png) @@ -372,33 +370,33 @@ In the end virtual and real-life testing tells us how well the model really perf ### Testing model in simulated environment with NVIDIA Isaac Sim and Edge Impulse extension -We can get useful information about model performance with minimal effort by testing it in a virtual environment. Install [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac-sim) and [Edge Impulse extension](https://github.com/edgeimpulse/edge-impulse-omniverse-ext). +We can get useful information about model performance with minimal effort by testing it in a virtual environment. Install [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac-sim) and the [Edge Impulse extension](https://github.com/edgeimpulse/edge-impulse-omniverse-ext). ![Edge Impulse extension](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-enable.png) -Install Sun study extension in Isaac Sim to be able to vary light conditions while testing. +Install the Sun Study extension in Isaac Sim to be able to vary light conditions while testing. -![Sun study in Isaac Sim](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sunstudy.png) +![Sun Study in Isaac Sim](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sunstudy.png) -Paste API key found under Edge Impulse Studio> Dashboard> Keys> Add new API key: +Paste your API key found in the Edge Impulse Studio > Dashboard > Keys > Add new API key into Omniverse Extension: ![Edge Impulse extension API key](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-ext-api-key.png) -To be able to classify any virtual camera capture we first need to build a version of the model that can run in a JavaScript environment. In Edge Impulse Studio, go to Deployment, find "WebAssembly" in the search box and hit Build. We don't need to keep the resulting .zip package, the extension will find and download it by itself. +To be able to classify any virtual camera capture we first need to build a version of the model that can run in a JavaScript environment. In Edge Impulse Studio, go to **Deployment**, find "WebAssembly" in the search box and click **Build**. We don't need to keep the resulting .zip package, the extension will find and download it by itself in a moment. ![Edge Impulse WebAssembly](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-webasm.png) -Back in the Edge Impulse extension in Isaac, when we expand the "Classification" group, a message will tell us everything is ready: "Your model is ready! You can now run inference on the current scene". +Back in the Edge Impulse extension in Isaac Sim, when we expand the "Classification" group, a message will tell us everything is ready: "Your model is ready! You can now run inference on the current scene". Before we test it we will make some accommodations in the viewport. Switch to "RTX - Interactive" to make sure the scene is rendered realistically. -Set viewport resolution to square 1:1 with either the same resolution as our intended device inference (120x120 pixels), or close (512x512 pixels). +Set viewport resolution to square 1:1 with either the same resolution as our intended device inference (120x120 pixels), or (512x512 pixels). ![Isaac Sim viewport resolution](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-resolution.png) -Display Isaac bounding boxes by selecting "BoundingBox2DLoose" under the icon that resembles a robotic sensor, the hit "Show Window". Now we can compare the ground truth with model prediction. +Display Isaac bounding boxes by selecting "BoundingBox2DLoose" under the icon that resembles a robotic sensor, then click "Show Window". Now we can compare the ground truth with model prediction. ![Isaac Sim sensors](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/Isaac-sensor.png) @@ -416,36 +414,37 @@ To get visual verification our model works as intended we can go to Deployment i ![Edge Impulse Studio Deployment OpenMV Firmware](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV_deployment.png) -Follow the [documentation](https://docs.edgeimpulse.com/docs/run-inference/running-your-impulse-openmv) on how to flash the device and to modify the `ei_object_detection.py` code. Remember to change: `sensor.set_pixformat(sensor.GRAYSCALE)`! The file `edge_impulse_firmware_arduino_portenta.bin` is our firmware for the Arduino Portenta H7 with Vision shield. +Follow [the documentation](https://docs.edgeimpulse.com/docs/run-inference/running-your-impulse-openmv) on how to flash the device and to modify the `ei_object_detection.py` code. Remember to change: `sensor.set_pixformat(sensor.GRAYSCALE)`. The file `edge_impulse_firmware_arduino_portenta.bin` is our firmware for the Arduino Portenta H7 with Vision shield. ![Testing model on device with OpenMV](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/OpenMV-testing.png) ### Deploy model as Arduino compatible library and send inference results to The Things Network with LoRaWAN -Start by selecting Arduino library as Deployment target. +Start by selecting **Arduino library** as a Deployment target. ![Deploy model as Arduino compatible library](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library.png) -Once built and downloaded, open Arduino IDE, go to **Sketch> Include Library> Add .zip Library ...** and locate the downloaded library. Next go to **File> Examples> [name of project]_inferencing> portenta_h7> portenta_h7_camera** to open a generic sketch template using our model. To test the model continuously and print the results to console this sketch is ready to go. The code might appear daunting, but we really only need to focus on the loop() function. +Once built and downloaded, open Arduino IDE, go to **Sketch > Include Library > Add .zip Library ...** and locate the downloaded library. Next go to **File > Examples > [name of project]_inferencing > portenta_h7 > portenta_h7_camera** to open a generic sketch template using our model. To test the model continuously and print the results to console this sketch is ready to go. The code might appear daunting, but we really only need to focus on the `loop()` function. ![Arduino compatible library example sketch](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/EI-arduino-library-example.png) ### Transmit results to The Things Stack sandbox using LoRaWAN -Using The Things Stack sandbox (formely known as The Things Network) we can create a low-power sensor network that allows transmitting device data with minimal energy consumption, long range without network fees. Your area might already be covered by a crowd funded network, or you can [initiate your own](https://www.thethingsnetwork.org/community/bodo/). [Getting started with LoRaWAN](https://www.thethingsindustries.com/docs/getting-started/) is really fun! + +Using The Things Stack sandbox (formerly known as The Things Network) we can create a low-power sensor network that allows transmitting device data with minimal energy consumption, long range, and no network fees. Your area may already be covered by a crowd funded network, or you can [create your own](https://www.thethingsnetwork.org/community/) gateway. [Getting started with LoRaWAN](https://www.thethingsindustries.com/docs/getting-started/) is really fun! ![The Things Network](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-map.png) -Following the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/) we create an application in The Things Stack sandbox and register our first device. +Following the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/) on the topic, we create an application in The Things Stack sandbox and register our first device. ![The Things Stack application](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-app.png) ![The Things Stack device](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-device.png) -Next we will simplify things by merging an example Arduino sketch for transmitting a LoRaWAN-message with the Edge Impulse generated object detection model code. Open the example sketch called LoraSendAndReceive included with the MKRWAN(v2) library mentioned in the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/). In the [project code repository](https://github.com/eivholt/icicle-monitor/tree/main/portenta-h7/portenta_h7_camera_lora) we can find an Arduino sketch witht the merged code. +Next we will simplify things by merging an example Arduino sketch for transmitting a LoRaWAN message, with the Edge Impulse generated object detection model code. Open the example sketch called `LoraSendAndReceive` included with the MKRWAN(v2) library mentioned in the [Arduino guide](https://docs.arduino.cc/tutorials/portenta-vision-shield/connecting-to-ttn/). There is an example of this for you in the [project code repository](https://github.com/eivholt/icicle-monitor/tree/main/portenta-h7/portenta_h7_camera_lora), where you can find an Arduino sketch with the merged code. ![Arduino transmitting inference results over LoRaWAN](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/arduino-lora.png) -In short we perform inference every 10 seconds. If any icicles are detected we simply transmit a binary 1 to the The Things Stack application. It is probably obvious that the binary payload is redundant, the presence of a message is enough, but this could be extended to transmit e.g. prediction confidence, number of clusters, battery level, temperature or light level. +In short, we perform inference every 10 seconds. If any icicles are detected we simply transmit a binary `1` to the The Things Stack application. It is probably obvious that the binary payload is redundant, the presence of a message is enough, but this could be extended to transmit other data, for example the prediction confidence, number of clusters, battery level, temperature or light level. ```python if(bb_found) { @@ -457,8 +456,7 @@ if(bb_found) { } ``` -A few things to consider in the implementation: -The device should enter deep sleep mode and disable/put to sleep all periferals between object detection. Default operation of the Portenta H7 with the Vision shield consumes a lot of energy and will drain battery quickly. To find out how much energy is consumed we can use a device such as the [Otii Arc from Qoitech](https://www.qoitech.com/otii-arc-pro/). Hook up positive power supply to VIN, negative to GND. Since VIN bypasses the Portenta power regulator we should provide 5V, however in my setup the Otii Arc is limited to 4.55V. Luckily it seems to be sufficient and we can take some measurements. By connecting the Otii Arc pin RX to the Portenta pin D14/PA9/UART1 TX, in code we can write debug messages to Serial1. This is incredibly helpful in establishing what power consumption is associated with what part of the code. +There are a few things to consider in the implementation: The device should enter deep sleep mode and disable/put to sleep all periferals between object detection runs. Default operation of the Portenta H7 with the Vision shield consumes a lot of energy and will drain a battery quickly. To find out how much energy is consumed we can use a device such as the [Otii Arc from Qoitech](https://www.qoitech.com/otii-arc-pro/). Hook up the positive power supply to **VIN**, negative to **GND**. Since VIN bypasses the Portenta power regulator we should provide 5V, however in my setup the Otii Arc is limited to 4.55V. Luckily it seems to be sufficient and we can take some measurements. By connecting the Otii Arc pin RX to the Portenta pin D14/PA9/UART1 TX, in code we can write debug messages to _Serial1_. This is incredibly helpful in determining what power consumption is associated with what part of the code. ![Arduino Portenta H7 power specs](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/portenta_h7_power.png) @@ -468,13 +466,16 @@ The device should enter deep sleep mode and disable/put to sleep all periferals ![Otii Arc power profile](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/otii-icicle-profile.png) -As we can see the highlighted section should be optimized for minimal power consumption. This is a complicated subject, especially on a [complex board such as the Arduino Portenta H7](https://github.com/arduino/ArduinoCore-mbed/issues/619) and out of scope for this article. Provided are some examples for general guidance: [snow monitor](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-power-profiling-16), [mail box sensor](https://community.element14.com/challenges-projects/project14/rf/b/blog/posts/got-mail-lorawan-mail-box-sensor). +As we can see the highlighted section should be optimized for minimal power consumption. This is a complicated subject, especially on a [complex board such as the Arduino Portenta H7](https://github.com/arduino/ArduinoCore-mbed/issues/619) but there are some examples for general guidance: + + - [Snow monitor](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-power-profiling-16) + - [Mail box sensor](https://community.element14.com/challenges-projects/project14/rf/b/blog/posts/got-mail-lorawan-mail-box-sensor). -The project code runs inference on an image every 10 seconds. This is for demonstration purposes and should be much less frequent, like once per hour during daylight. Have a look at this project for an example of how to [remotely control inference interval](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-lora-application-14) via LoRaWAN downlink message. This could be further controlled automatically via an application that has access to an [API for daylight data](https://developer.yr.no/doc/GettingStarted/). +The project code presented here runs inference on an image every 10 seconds. However, this is for demonstration purposes and in a deployment should be much less frequent, like once per hour during daylight. Have a look at this project for an example of how to [remotely control inference interval](https://www.hackster.io/eivholt/low-power-snow-depth-sensor-using-lora-e5-b8e7b8#toc-lora-application-14) via LoRaWAN downlink message. This could be further controlled automatically via an application that has access to an [API for daylight data](https://developer.yr.no/doc/GettingStarted/). ![YR weather API](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/yr-sun.png) -In the The Things Stack application we need to define a function that will be used to decode the byte into a JSON structure that is easier to interpet when we pass the message further up the chain of services. The function can be found in the [project code repository](https://github.com/eivholt/icicle-monitor/blob/main/TheThingsStack/decoder.js). +Next, in the The Things Stack application we need to define a function that will be used to decode the byte into a JSON structure that is easier to interpet when we pass the message further up the chain of services. The function can be found in the [project code repository](https://github.com/eivholt/icicle-monitor/blob/main/TheThingsStack/decoder.js). ![The Things Stack decoder](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-decoder.png) @@ -495,11 +496,11 @@ function Decoder(bytes, port) { } ``` -Now we can observe messages being received and decoded in **Live data** in TTS console. +Now we can observe messages being received and decoded in **Live data** in the TTS console. ![The Things Stack live data](../.gitbook/assets/rooftop-ice-synthetic-data-omniverse/ttn-data.png) -An integral part of TTS is a MQTT message broker. At this point we can use [any MQTT client to subscribe to topics](https://www.thethingsindustries.com/docs/integrations/mqtt/mqtt-clients/) and create any suitable notification system for the end user. The following is a MQTT client written in Python to demonstrate the principle. Note that the library paho-mqtt has been used in a way so that it will block the program execution until two messages have been received. Then it will print the topic and payloads. A real-life implementation would rather register a callback and perform some action for each message received. +An integral part of The Things Stack is an MQTT message broker. At this point we can use [any MQTT client to subscribe to topics](https://www.thethingsindustries.com/docs/integrations/mqtt/mqtt-clients/) and create any suitable notification system for the end user. The following is an MQTT client written in Python to demonstrate the principle. Note that the library `paho-mqtt` has been used in a way so that it will block the program execution until two messages have been received. Then it will print the topic and payloads. In a real implementation, it would be better to register a callback and perform some action for each message received. ``` python @@ -521,9 +522,9 @@ v3/icicle-monitor@ttn/devices/portenta-h7-icicle-00/up {"end_device_ids":{"device_id":"portenta-h7-icicle-00","application_ids":{"application_id":"icicle-monitor"},"dev_eui":"3036363266398F0D","join_eui":"0000000000000000"},"correlation_ids":["as:up:01HSKMTN7F60CC3BQXE06B3Q4X","rpc:/ttn.lorawan.v3.AppAs/SimulateUplink:17b97b44-a5cd-45f0-9439-2de42e187300"],"received_at":"2024-03-22T17:55:05.070404295Z","uplink_message":{"f_port":1,"frm_payload":"AQ==","decoded_payload":{"detected":true},"rx_metadata":[{"gateway_ids":{"gateway_id":"test"},"rssi":42,"channel_rssi":42,"snr":4.2}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"frequency":"868000000"},"locations":{"user":{"latitude":67.2951772015745,"longitude":14.43232297897339,"altitude":13,"source":"SOURCE_REGISTRY"}}},"simulated":true}' ``` -Observe the difference in the real uplink (first) and simulated uplink (last). In both we find "decoded_payload":{"detected":true}. +Observe the difference in the real uplink (first) and simulated uplink (second). In both we find "decoded_payload":{"detected":true}. -TTS has a range of [integration options](https://www.thethingsindustries.com/docs/integrations/) for specific platforms, or you could set up a [custom webhook using standard HTTP/REST](https://www.thethingsindustries.com/docs/integrations/webhooks/) mechanism. +TTS has a range of [integration options](https://www.thethingsindustries.com/docs/integrations/) for specific platforms, or you could set up a [custom webhook using a standard HTTP/REST](https://www.thethingsindustries.com/docs/integrations/webhooks/) mechanism. ## Limitations