The camera-traps application is both a simulator and an edge device application for classifying images, with the first deployment specializing in wildlife images. The simulation environment will be implemented first and serve as a test bed for protocols and techniques that optimize storage, execution time, power and accuracy. The ultimate goal is to deploy a version of this application on camera-trap devices in the wild.
This application uses the event-engine library to implement its plugin architecture and event-driven communication. The engine uses zmq sockets to deliver events between senders and the subscribers interested in specific events.
The event-engine supports internal and external plugins. Internal plugins are Rust plugins delivered with camera-traps and run in the camera-traps process. External plugins are configured by camera-traps to run outside the camera-traps process and use a TCP port to send and receive events. By using TCP, external plugins can be written in any language that supports the flatbuffers wire protocol.
To quickly start the application under Docker using docker-compose, follow these steps:
- ./installer/install.sh $PWD ./installer/example_input.yml
- cd test
- docker-compose up
- docker-compose down
The camera-traps application requires configuration through environment variables or configuration files. When launching the application from a releases subdirectory, the specific release's config directory will contain the default configuration files for running a short simulation test.
In general, plugins can also depend on their own environment variables and/or configuration files, and the same is true of test programs. The releases directory contains docker-compose files that use default configurations, which can serve as a template for production environment configuration.
| Target | Environment Variable | Default File | Notes |
|---|---|---|---|
| camera-traps application | TRAPS_CONFIG_FILE | ~/traps.toml | Can be 1st command line parameter |
| image_gen_plugin | /input.json | ||
| image_store_plugin | TRAPS_IMAGE_STORE_FILE | ~/traps-image-store.toml | |
| power_measure_plugin | TRAPS_POWER_LOG_PATH | ~/logs | |
| oracle_monitor_plugin | TRAPS_ORACLE_OUTPUT_PATH | ~/output | |
| integration tests | TRAPS_INTEGRATION_CONFIG_FILE | ~/traps-integration.toml | |
| logger | TRAPS_LOG4RS_CONFIG_FILE | resources/log4rs.yml | Packaged with application |
The external python plugins run in their own processes and do not currently use environment variables.
The camera-traps application uses log4rs as its log manager. The log settings in resources/log4rs.yml source code will be used unless overridden by assigning a log4rs.yml configuration filepath to the TRAPS_LOG4RS_CONFIG_FILE environment variable. To maximize logging, set root level to trace in the effective log4rs.yml file. Also, include the observer_plugin in the internal plugins list in the effective traps.toml file.
Camera-traps uses a TOML file to configure the internal and external plugins it loads. Internal plugins are registered with the event-engine by simply specfying their names since their runtime characteristics are compiled into the application. External plugins, on the other hand, require more detailed information in order to be registered. Here is the example resources/traps.toml file content:
# This is the camera-traps application configuration file for versions 0.x.y of the application.
# It assumes the use of containers and docker-compose as the deployment mechanism.
title = "Camera-Traps Application Configuration v0.3.2"# The event engine's publish and subscribe port used to create the event_engine::App instance.
publish_port = 5559 subscribe_port = 5560# An absolute path to the image directory is required but a file name prefix is optional.
# If present the prefix is preprended to generated image file names. This is the directory
# into which the image_recv_plugin writes incoming images and the image_store_plugin may
# delete images or output the scores for images.
images_output_dir = "/root/camera-traps/images"
# image_file_prefix = ""# The container for both internal and external plugins. Internal plugins are written in rust
# and compiled into the camera-traps application. External plugins are usually written in
# python but can be written in any language. External plugins run in their own processes
# and communicate via tcp or ipc.
[plugins] # Uncomment the internal plugins loaded when the camera-traps application starts.
internal = [
# "image_gen_plugin",
"image_recv_plugin",
# "image_score_plugin",
"image_store_plugin",
# "observer_plugin"
]# Configure each of the active internal plugins with the image processing action they should
# take when new work is received. If no action is specified for a plugin, its no-op action
# is used by default.
internal_actions = [
"image_recv_write_file_action",
"image_store_file_action"
]# External plugins require more configuration information than internal plugins.
# Each plugin must subscribe to PluginTerminateEvent.
#
# Note that each plugin must specify the external port to use in TWO PLACES: here as well as
# in the docker-compose.yml file. If external_port changes here, it must ALSO be changed in the
# docker-compose.yml file.
[[plugins.external]]
plugin_name = "ext_image_gen_plugin"
id = "d3266646-41ec-11ed-a96f-5391348bab46"
external_port = 6000
subscriptions = [
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_image_score_plugin"
id = "d6e8e42a-41ec-11ed-a36f-a3dcc1cc761a"
external_port = 6001
subscriptions = [
"ImageReceivedEvent",
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_power_monitor_plugin"
id = "4a0fca25-1935-472a-8674-58f22c3a32b3"
external_port = 6010
subscriptions = [
"MonitorPowerStartEvent",
"MonitorPowerStopEvent",
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_power_control_plugin"
id = "a59621f2-4db6-4892-bda1-59ecb7ff24ae"
external_port = 6011
subscriptions = [
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_oracle_monitor_plugin"
id = "6e153711-9823-4ee6-b608-58e2e801db51"
external_port = 6011
subscriptions = [
"ImageScoredEvent",
"ImageStoredEvent",
"ImageDeletedEvent",
"PluginTerminateEvent"
]
Every plugin must subscribe to the PluginTerminateEvent, which upon receipt causes the plugin to stop. Subscriptions are statically defined in internal plugin code and explicitly configured for external plugins. External plugins also provide their predetermined UUIDs and external TCP ports.
Camera-traps looks for its configuration file using these methods in the order shown:
- The environment variable $TRAPS_CONFIG_FILE.
- The first command line argument.
- $HOME/traps.toml
The first file it finds it uses. If no configuration file is found the program aborts.
The names listed in the internal list are the rust plugin file names. These plugins run as separate threads in the camera-traps process. The internal_actions list contains the file names that implement the different algorithms or actions associated with each internal plugin.
A naming convention is used to associate actions with their plugins: An action name starts with its plugin name minus the trailing "plugin" part, followed by an action identifier part, and ends with "_action". Each plugin has a no-op action that causes it to take no action other than, possibly, generating the next event in the pipeline. For example, image_gen_noop_action is associated with the image_gen_plugin.
Internal plugins for which no corresponding action is specified are assigned their no-op plugin by default.
When image_recv_write_file_action is specifed, the image_recv_plugin uses the image_dir and image_file_prefix parameters to manage files. The image_dir is the directory into which image files are placed. Image file names are constructed from the information received in a NewImageEvent and have this format:
<image_file_prefix><image_uuid>.<image_format> The image_uuid and image_format are from the NewImageEvent. The image_file_prefix can be the empty string and the image_format is always lowercased when used in the file name.
The Image Scoring plugin can make use of NVIDIA GPUs to improve the performance of object detection and classification with some ML models. In order to make use of NVIDIA GPUs in the Camera Traps application, the following steps must be taken:
- Ensure the NVIDIA drivers are installed natively on the machine. For example, on Ubuntu LTS, follow the instructions in Section 3.1 here. Be sure to reboot your machine after adding the keyring and installing the drivers. You can check to see if the drivers are installed properly and communicating with the hardware by running the following command:
nvidia-smi
- Install the NVIDIA Container Toolkit and configure the Docker Runtime. See the instructions here. Make sure to restart Docker after installing and configuring the toolkit. To check if the toolkit and Docker are installed and configured correctly, run the following:
docker run --gpus=all --rm -it ubuntu nvidia-smi
The output should be similar to the output from Step 1.
- Update the Camera Traps Compose File to Use GPUs. Starting with release 0.4, the installer includes options for making NVIDIA GPUs available to both the Image Scoring and Power Monitoring plugins. See the Installer README for more details.
In-memory representations of events are translated into flatbuffer binary streams plus a leading two byte sequence that identifies the event type. These statically defined byte sequences are specified in the events.rs source file and repeated here for convenience.
// Each event is assigned a binary prefix that zqm uses to route incoming binary streams to all of the event's subscribers.
pub const NEW_IMAGE_PREFIX: [u8; 2] = [0x01, 0x00];
pub const IMAGE_RECEIVED_PREFIX: [u8; 2] = [0x02, 0x00];
pub const IMAGE_SCORED_PREFIX: [u8; 2] = [0x03, 0x00];
pub const IMAGE_STORED_PREFIX: [u8; 2] = [0x04, 0x00];
pub const IMAGE_DELETED_PREFIX: [u8; 2] = [0x05, 0x00];
pub const PLUGIN_STARTED_PREFIX: [u8; 2] = [0x10, 0x00];
pub const PLUGIN_TERMINATING_PREFIX: [u8; 2] = [0x11, 0x00];
pub const PLUGIN_TERMINATE_PREFIX: [u8; 2] = [0x12, 0x00];
pub const MONITOR_POWER_START_PREFIX: [u8; 2] = [0x20, 0x00];
pub const MONITOR_POWER_STOP_PREFIX: [u8; 2] = [0x21, 0x00];
Each event sent or received begins with its two byte prefix followed by its serialized form as defined in the camera-traps flatbuffer definition file (events.fbs). The following section describes how to generate Rust source code from this definition file, a similar process can be used for any language supported by flatbuffers.
Flatbuffers info: https://google.github.io/flatbuffers/
The flatbuffers messages schema is defined in the resources/events.fsb file. To change the message formats do the following:
- Edit the
resources/events.fsbfile with your changes. - From the camera-traps directory, regenerate the
events_generated.rscode with the command:
$ flatc --rust -o src resources/events.fbs
- (Optional) Add the following line to the top of the
src/events_generated.rsfile so that clippy warnings are suppressed:
// this line added to keep clippy happy
#![allow(clippy::all)]
Each plugin is required to conform to the following conventions:
- Register for the PluginTerminateEvent.
- Send a PluginStartedEvent when it begins executing.
- Send a PluginTerminatingEvent when it shuts down.
The PluginStartedEvent advertises a plugin's name and uuid when it starts. When a plugin receives a PluginTerminateEvent, it checks if the event's target_plugin_name matches its name or the wildcard name (*). If either is true, then the plugin is expected to gracefully terminate. The plugin is also expected to gracefully terminate if the event's target_plugin_uuid matches the plugin's uuid. Part of plugin termination is for it to send a PluginTerminatingEvent to advertise that it's shutting down, whether in response to a PluginTerminateEvent or for any other reason.
The instructions in this section assume Docker (and docker-compose) are installed, as well as Rust, cargo and make.
From the top-level camera-traps directory, issue the following command to build the application's Docker images:
make build See Makefile for details. Use the installer install script to create a run directory. See the installer README for more details. Then, navigate to the new run directory. Issue the following command to run the application, including the external plugins for which it's configured:
docker-compose up See docker-compose.yaml for details. From the same release directory, issue the following command to stop the application:
docker-compose down
If you're just interested in building the Rust, issue cargo build from the top-level camera-traps directory. Alternatively, issue cargo run to build and run it. External plugins are not started using this approach. The internal plugins and their actions are configured using a traps.toml file, as discussed above.
The camera-traps/tests directory contains integration_tests.rs program. The integration test program runs as an external plugin configured via a traps.toml file as shown above. See the top-level comments in the source code for details.
This section addresses two questions:
- Why would I want to create a plugin?
- What kind of plugin should I create?
One would want to create their own plugin if they wanted to read or write events and perform some new action that isn't currently implemented. If an existing plugin doesn't do what you want, you have the option of modifying that plugin or creating another plugin that acts on the same events and does what you need.
For example, the image_gen_plugin injects new images into the event stream, the image_recv_plugin writes new images to file, etc. The observer_plugin is one that subscribes to all events and logs them for debugging purposes. Most of the time we don't run the observer_plugin, but if we want extended logging we just include it to run in the traps.toml file. In this case, having a separate plugin from which we can customize the logging of all events is more convenient then adding that logging capability to each existing plugin.
Another reason for introducing a new plugin would be to also service new events. As the application evolves new capabilities might require new events. This occurred as we develop support for power monitoring, which introduces 2 new events and a plugin to handle them.
When implementing a plugin the choice between internal and external is often technology driven. Do we want to write a plugin in Rust and compile it into the application (internal) or do we want to write it in some other language and start it up in its own container (external)? Considerations as to which approach to take include performance, resource usage, and availability of domain-specific libraries.
When development on a new release begins, create a new branch. If you would to test your changes, merge into the dev branch. This will trigger the building of docker images with the latest tag and a suite of tests. When development completes and the final version of the release's images are pushed to docker hub, we tag those images with the release number.
To be able to rebuild a release at anytime, we also tag the release's source code in github. The tag is the same as the release version number. Once confident that the tagged code is stable, release tags can be protected using github tag protection.
This work has been funded by grants from the National Science Foundation, including the ICICLE AI Institute (OAC 2112606) and Tapis (OAC 1931439).