Initial Commit

Initial commit of Python bindings Apps
Harsh-Vavaiya · Nov 5, 2019 · c6eca50 · c6eca50
commit c6eca50
Show file tree

Hide file tree

Showing 35 changed files with 3,149 additions and 0 deletions.
diff --git a/.python-app-pipeline.png b/.python-app-pipeline.png
diff --git a/.test3-app.png b/.test3-app.png
diff --git a/FAQ.md b/FAQ.md
@@ -0,0 +1,85 @@
+# Frequently Asked Questions and Troubleshooting Guide
+
+[Ctrl-C does not stop the app during engine file generation](#faq1)  
+[Application fails to create gst elements](#faq2)  
+[GStreamer debugging](#faq3)  
+[Application stuck with no playback](#faq4)  
+[Error on setting string field](#faq5)  
+[Pipeline unable to perform at real time](#faq6)  
+
+<a name="faq1"></a>
+### Ctrl-C does not stop the app during engine file generation
+This is a limitation of Python signal handling:  
+https://docs.python.org/3/library/signal.html  
+"A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes."  
+
+To work around this:  
+1. Use ctrl-z to bg the process  
+2. Optionally run "jobs" if there are potentially multiple bg processes:  
+    $ jobs  
+    [1]-  Stopped                 python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264  (wd: /opt/nvidia/deepstream/deepstream-4.0/sources/apps/python/deepstream-test1)  
+    [2]+  Stopped                 python3 deepstream_test_2.py /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264  
+3. Kill the bg job:  
+    $ kill %<job number, 1 or 2 from above. e.g. kill %1>  
+
+
+<a name="faq2"></a>
+### Application fails to create gst elements 
+As with DeepStream SDK, if the application runs into errors, cannot create gst elements, try again after removing gstreamer cache:  
+   rm ${HOME}/.cache/gstreamer-1.0/registry.x86_64.bin
+
+
+<a name="faq3"></a>
+### GStreamer debugging
+General GStreamer debugging: set debug level on the command line:  
+  ```
+  GST_DEBUG=<level> python3 <app> [options]
+  ```
+
+<a name="faq4"></a>
+### Application stuck with no playback
+The application appears to be stuck without any playback activity.  
+  One possible cause is incompatible input type. Some of the sample apps only support H264 elementary streams.  
+
+
+<a name="faq5"></a>
+### Error on setting string field  
+  ```
+  terminate called after throwing an instance of 'pybind11::error_already_set'  
+    what():  TypeError: (): incompatible function arguments. The following argument types are supported:  
+      1. (arg0: pyds.<struct, e.g. NvDsEventMsgMeta>, arg1: str) -> None  
+
+  Invoked with: <pyds.<struct, e.g. NvDsEventMsgMeta> object at 0x7ffa93d2fce0>, <int, e.g. 140710457366960>  
+  ```
+
+  This can happen if the string field is being set with another string field, e.g.:  
+  ```
+  dstmeta.sensorStr = srcmeta.sensorStr
+  ```
+
+  Remember from the String Access section above that reading ```srcmeta.sensorStr``` directly returns an int, not string.  
+  The proper way to copy the string field in this case is:  
+  ```
+  dstmeta.sensorStr = pyds.get_string(srcmeta.sensorStr)
+  ```
+
+<a name="faq6"></a>
+### Pipeline unable to perform at real time
+  WARNING: A lot of buffers are being dropped. (13): gstbasesink.c(2902):   
+  gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:  
+  There may be a timestamping problem, or this computer is too slow.  
+
+  Answer:  
+  This could be thrown from any GStreamer app when the pipeline through-put is low  
+  resulting in late buffers at the sink plugin.  
+  This could be a hardware capability limitation or expensive software calls  
+  that hinder pipeline buffer flow.  
+  With python, there is a possibility for such an induced delay in software.  
+  This is with regards to the probe() callbacks an user could leverage  
+  for metadata extraction and other use-cases as demonstrated in our test-apps.  
+
+  Please NOTE:  
+  a) probe() callbacks are synchronous and thus holds the buffer  
+     (info.get_buffer()) from traversing the pipeline until user return.  
+  b) loops inside probe() callback could be costly in python.  
+
diff --git a/HOWTO.md b/HOWTO.md
@@ -0,0 +1,195 @@
+# How To --
+## Run the Sample Applications 
+## Write a DeepStream Application in Python
+
+This guide provides resources for DeepStream application development in Python.
+
+* [Prerequisites](#prereqs)
+* [Running Sample Applications](#run_samples)
+* [Pipeline Construction](#pipeline_construction)
+* [MetaData Access](#metadata_access)
+* [FAQ and Troubleshooting](FAQ.md)
+
+<a name="prereqs"></a>
+## Prerequisites
+
+* Ubuntu 18.04
+* DeepStream SDK 4.0.1
+* Python 3
+* [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.14.5
+
+Gst python should be already installed on Jetson.  
+If missing, install with the following steps:
+```
+   $ sudo apt-get install python-gi-dev
+   $ export GST_LIBS="-lgstreamer-1.0 -lgobject-2.0 -lglib-2.0"
+   $ export GST_CFLAGS="-pthread -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include"
+   $ git clone https://github.com/GStreamer/gst-python.git
+   $ cd gst-python
+   $ git checkout 1a8f48a
+   $ ./autogen.sh PYTHON=python3
+   $ ./configure PYTHON=python3
+   $ make
+   $ sudo make install
+```
+
+<a name="run_samples"></a>
+## Running Sample Applications
+
+Download the release package and unpack it under DS 4.0.1 installation:  
+```tar xf ds_pybind_0.5.tbz2 -C <DeepStream 4.0.1 ROOT>/sources```  
+
+This will create the following directory:  
+```<DeepStream 4.0.1 ROOT>/sources/python```  
+
+The Python apps and bindings are under the "python" directory.  
+Go into each app directory and follow instructions in the README.  
+
+NOTE: The app configuration files contain relative paths for models.  
+
+<a name="pipeline_construction"></a>
+## Pipeline Construction
+
+DeepStream pipelines can be constructed using Gst Python, the GStreamer framework's Python bindings.  
+See [sample applications](apps/) main functions for pipeline construction examples.  
+
+<a name="metadata_access"></a>
+## MetaData Access
+
+DeepStream MetaData contains inference results and other information used in analytics. The MetaData is attached to the Gst Buffer received by each pipeline component. The metadata format is described in detail in the [SDK MetaData documentation](https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_metadata.03.1.html) and [API Guide](https://docs.nvidia.com/metropolis/deepstream/dev-guide/DeepStream_Development_Guide/baggage/group__NvDsMetaApi.html).  
+
+The SDK MetaData library is developed in C/C++. Python bindings provide access to the MetaData from Python applications. The bindings are provided in a compiled module, available for x86_64 and Jetson platforms. Find them in the release package with the following layout:
+```
+   bindings
+      |- x86_64
+      |     |- pyds.so
+      |- jetson
+            |- pyds.so
+```
+
+Applications can import the module thus:
+```
+import sys
+sys.path.append('../') # Add path to the bindings directory
+# The common/is_aarch64.py module adds the platform-specific path to pyds module
+from common.is_aarch64 import is_aarch64
+import pyds
+```
+
+The bindings generally follow the same API as the underlying C/C++ library, with a few exceptions detailed in sections below.  
+
+#### Memory Management
+
+Memory for MetaData is shared by the Python and C/C++ code paths. For example, a MetaData item may be added by a probe function written in Python, and needs to be accessed by a downstream plugin written in C/C++. The deepstream-test4 app contains such usage. The Python garbage collector does not have visibility into memory references in C/C++, and therefore cannot safely manage the lifetime of such shared memory. Because of this complication, Python access to MetaData memory is typically achieved via references without claiming ownership.
+
+#### Allocations
+
+When MetaData objects are allocated in Python, an allocation function is provided by the bindings to ensure proper memory ownership of the object. If the constructor is used, the the object will be claimed by the garbage collector when its Python references terminate. However, the object will still need to be accessed by C/C++ code downstream, and therefore must persist beyond those Python references.  
+
+Example:
+To allocate an NvDsEventMsgMeta instance, use this:  
+```
+msg_meta = pyds.alloc_nvds_event_msg_meta() # get reference to allocated instance without claiming memory ownership  
+```
+NOT this:  
+```
+msg_meta = NvDsEventMsgMeta() # memory will be freed by the garbage collector when msg_meta goes out of scope in Python  
+```
+
+Allocators are available for the following structs:  
+```
+NvDsVehicleObject: alloc_nvds_vehicle_object()  
+NvDsPersonObject:  alloc_nvds_person_object()  
+NvDsFaceObject:    alloc_nvds_face_object()  
+NvDsEventMsgMeta:  alloc_nvds_event_msg_meta()  
+NvDsEvent:         alloc_nvds_event()  
+NvDsPayload:       alloc_nvds_payload()  
+Generic buffer:    alloc_buffer(size)  
+```
+
+#### String Access
+
+Some MetaData structures contain string fields. These are accessable in the following manner:  
+
+##### Setting String Fields
+Setting a string field results in the allocation of a string buffer in the underlying C++ code.  
+```obj.type = "Type"```  
+This will cause a memory buffer to be allocated, and the string "TYPE" will be copied into it.  
+This memory is owned by the C code and will be freed later. To free the buffer in Python code, use:  
+```pyds.free_buffer(obj.type)```  
+
+##### Reading String Fields
+Directly reading a string field returns C address of the field in the form of an int, e.g.:  
+```
+obj = pyds.glist_get_nvds_vehicle_object(data);
+print(obj.type)
+```
+This will print an int representing the address of obj.type in C (which is a char*).  
+
+To retrieve the string value of this field, use ```pyds.get_string()```, e.g.:  
+```
+print(pyds.get_string(obj.type))
+```
+
+#### Casting
+
+Some MetaData instances are stored in GList form. To access the data in a GList node, the data field needs to be cast to the appropriate structure. This casting is done via binding functions:  
+```
+glist_get_nvds_batch_meta
+glist_get_nvds_frame_meta
+glist_get_nvds_object_meta
+glist_get_nvds_user_met
+glist_get_nvds_classifier_meta
+glist_get_nvds_display_meta
+glist_get_nvds_label_info
+glist_get_nvds_event_msg_meta
+glist_get_nvds_event_msg_meta
+glist_get_nvds_vehicle_object
+glist_get_nvds_person_object
+```
+
+Example:
+```
+l_frame = batch_meta.frame_meta_list
+frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)
+```
+
+#### Callback Function Registration
+
+Custom MetaData added to NvDsUserMeta require custom copy and release functions. The MetaData library relies on these custom functions to perform deep-copy of the custom structure, and free allocated resources. These functions are registered as callback function pointers in the NvDsUserMeta structure.
+
+Callback functions are registered using these functions:  
+```
+pyds.set_user_copyfunc(NvDsUserMeta_instance, copy_function)
+pyds.set_user_releasefunc(NvDsUserMeta_instance, free_func)
+```
+
+*NOTE*: Callbacks need to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
+```pyds.unset_callback_funcs()```  
+
+See the deepstream-test4 sample application for an example of callback registration and unregistration.  
+
+Limitation: the bindings library currently only supports a single set of callback functions for each application. The last registered function will be used.  
+
+#### Optimizations and Utilities
+
+Python interpretation is generally slower than running compiled C/C++ code. To provide better performance, some operations are implemented in C and exposed via the bindings interface. This is currently experimental and will expand over time.
+
+The following optimized functions are available:
+
+##### pyds.NvOSD_ColorParams.set(double red, double green, double blue, double alpha)
+
+This is a simple function that performs the same operations as the following:  
+```
+txt_params.text_bg_clr.red = red
+txt_params.text_bg_clr.green = green
+txt_params.text_bg_clr.blue = blue
+txt_params.text_bg_clr.alpha = alpha
+```
+
+These are performend on each object in deepstream_test_4.py, causing the aggregate processing time to slow down the pipeline. Pushing this function into the C layer helps to increase performance.
+
+##### generate_ts_rfc3339 (buffer, buffer_size)
+
+This function populates the input buffer with a timestamp generated according to RFC3339:  
+```%Y-%m-%dT%H:%M:%S.nnnZ\0```
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 NVIDIA CORPORATION. All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/README.md b/README.md
@@ -0,0 +1,46 @@
+# DeepStream Python Apps
+
+This repository contains Python bindings and sample applications for the [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk).  
+
+The bindings and apps are currently in *Alpha* at v0.5. API changes are expected in future releases.  
+SDK version supported: 4.0.1  
+
+Download the latest release package complete with bindings and sample applications from the release section.  
+
+* [Sample Applications](#sample_applications)
+* [MetaData Bindings](#metadata_bindings)
+
+<a name="sample_applications"></a>
+## Sample Applications
+
+Sample applications provided here demonstrate how to work with DeepStream pipelines using Python.  
+The sample applications require [MetaData Bindings](#metadata_bindings) to work.  
+
+To run the sample applications or write your own, please consult the [HOW-TO Guide](HOWTO.md)  
+
+<p align="center">
+<img src=".test3-app.png" alt="deepstream python app screenshot" height="400px"/>
+</p>
+
+We currently provide four sample applications:
+* [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline
+* [deepstream-test2](apps/deepstream-test2) -- 4-class object detection, tracking and attribute classification pipeline
+* [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection
+* [deepstream-test4](apps/deepstream-test4) -- msgbroker for sending analytics results to the cloud
+
+These are Python versions of the first four test applications included in the DeepStream SDK.  
+
+Detailed application information is provided in each application's subdirectory under [apps](apps).  
+
+<a name="metadata_bindings"></a>
+## Metadata Bindings
+
+Python bindings are provided for DeepStream MetaData in the form of a compiled module. This module is generated using [Pybind11](https://github.com/pybind/pybind11). 
+
+<p align="center">
+<img src=".python-app-pipeline.png" alt="bindings pipeline" height="400px"/>
+</p>
+
+These bindings support a Python interface to the MetaData structures and functions. Usage of this interface is documented in the [HOW-TO Guide](HOWTO.md) and demonstrated in the sample applications.  
+The current bindings are limited to the [NvDsBatchMeta](https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_metadata.03.2.html%23) hierarchy. Image data access is currently not included.  
+