This project focuses on demonstrating the following two points:
- MapR Streams (MapR-ES) makes it easier to distribute fast data streams (such as video feeds) to a group of stream consumers. It simplifies the buffering and connection management that's often complicated when you have a dynamic number of concurrent stream consumers. MapR-ES also allows you to partition streams so you can split up time consuming tasks like face detection across multiple consumers without redundant processing on duplicated data.
- MapR client containers (MapR PACC) for Docker enable workloads to elastically scale in the cloud. This makes it possible to utilize hardware more efficiently and meet SLAs more effectively.
The data that can be generated by detecting faces in video feeds is providing new quantitative paths for a wide range of applications including smart spaces, ad analytics, surveillance, and even political science. However, implementing real-time face detection is challenging on two fronts:
- Video feeds generate high throughput data streams
- Real-time face detection requires expensive GPU processors
To make matters worse, many applications need to simultaneously process multiple videos (e.g. webcam feeds) at near real-time frame rates. Until recently, the assumption has been that a dedicated GPU must be allocated for each video feed. With the advent of containerization, distributed pub/sub messaging services, and elastic GPUs in the cloud, we can architect applications that more efficiently utilize hardware to process multiple high speed video feeds within containerized processes that are easy to scale.
In this project we demonstrate an application designed to detect faces in multiple video feeds with an efficient use of GPU hardware. This architecture is generally applicable to any application that involves transporting and processing fast data with linear scalablity.
Docker makes it easy to run multiple video producers and consumers (face detectors), and when they're packaged with the MapR PACC (Persistent Application Client Container), they can maintain access to a MapR cluster and the video streams hosted on that cluster even though the containers themselves are ephemeral.
In other words, by dockerizing the video consumers, face detection processes can be provisioned on-the-fly when new video feeds demand faster processing, and discovering those streams will never be a problem since MapR’s global namespace ensures stream locations never change.
Distributed pub/sub messaging services are typically used for communication between services. When messages are produced faster than they are consumed, these services store the unconsumed messages in memory. As such, they provide a convenient decoupling that allows producers to send data to consumers without the burden of maintaining connections across a variable number of consumers.
Real-time face detection requires high speed processors. Video processing on traditional CPUs can only be done by significantly compromising the fidelity and timeliness of video frame processing.
To see how fast you can classify a video without a GPU, change ctx = mx.gpu(args.gpuid) to ctx = mx.cpu(0) in consumer/deploy/mapr_consumer.py.
Pub/sub streaming systems like MapR Streams have traditionally been thought of as a solution for inter-service communication, rather than a distribution mechanism for fast data streams such as video. However, this demo proves that video can be distributed through pub/sub systems like MapR Streams, and in fact doing so offers two distinct advantages:
- they buffer data between fast producers and slower consumers
- they simplify the processes of broadcasting fast data across a group of concurrent stream processors.
Provision a 3 node MapR cluster, and an ubuntu machine with a GPU. This will be our mapr client and docker host.
maprcli stream delete -path /tmp/rawvideostream
maprcli stream delete -path /tmp/identifiedstream
maprcli stream delete -path /tmp/processedvideostream
maprcli stream create -path /tmp/rawvideostream
maprcli stream edit -path /tmp/rawvideostream -produceperm p -consumeperm p -topicperm p
maprcli stream topic create -path /tmp/rawvideostream -topic topic1 -partitions 6
maprcli stream create -path /tmp/processedvideostream
maprcli stream edit -path /tmp/processedvideostream -produceperm p -consumeperm p -topicperm p
maprcli stream topic create -path /tmp/processedvideostream -topic topic1 -partitions 1
maprcli stream create -path /tmp/identifiedstream
maprcli stream edit -path /tmp/identifiedstream -produceperm p -consumeperm p -topicperm p
maprcli stream topic create -path /tmp/identifiedstream -topic all -partitions 1
maprcli stream topic list -path /tmp/rawvideostream -json
maprcli stream topic list -path /tmp/identifiedstream -json
maprcli stream topic list -path /tmp/processedvideostream -json
echo -e "\nSTREAM MONITORING:"; UPLINE=$(tput cuu1); ERASELINE=$(tput el); echo -e "\n\n\n"; while true; do x=`maprcli stream topic list -path /tmp/rawvideostream -json | grep physicalsize | sed -s 's/.*://' | tr -d ','`; y=`maprcli stream topic list -path /tmp/processedvideostream -json | grep physicalsize | sed -s 's/.*://' | tr -d ','`; z=`maprcli stream topic list -path /tmp/identifiedstream -json | grep physicalsize | sed -s 's/.*://' | tr -d ','`; echo -e "$UPLINE$ERASELINE$UPLINE$ERASELINE$UPLINE$ERASELINE\c"; echo -e "/tmp/rawvideostream size: $x\n/tmp/processedvideostream size: $y\n/tmp/identifiedstream size: $z"; done
Create a MapR PACC image, as described here:
wget http://package.mapr.com/releases/installer/mapr-setup.sh -P /tmp
chmod 700 ./mapr-setup.sh
./mapr-setup.sh docker client
Make sure to create PACC with ubuntu16.
Open ./docker_images/client/Dockerfile and replace the "FROM" command to following:
FROM nvidia/cuda:9.0-base-ubuntu16.04
Also, add the apt-get and pip install commands shown in the following Dockerfile example:
FROM pacc_nvidia:cuda-9.0-base
RUN apt-get update -y && \
apt-get install -y build-essential libopenblas-dev liblapack-dev \
libopencv-dev cuda-command-line-tools-9-0 \
cuda-cublas-dev-9-0 \
cuda-cudart-dev-9-0 \
cuda-cufft-dev-9-0 \
cuda-curand-dev-9-0 \
cuda-cusolver-dev-9-0 \
cuda-cusparse-dev-9-0 \
python-dev python-setuptools python-numpy python-pip graphviz && \
pip install opencv-python mxnet-cu90 flask graphviz easydict scipy tensorflow sklearn scikit-image
RUN echo 'LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/"' >> /etc/environment
RUN pip install --global-option=build_ext --global-option="--library-dirs=/opt/mapr/lib" --global-option="--include-dirs=/opt/mapr/include/" http://package.mapr.com/releases/MEP/MEP-4.0.0/mac/mapr-streams-python-0.9.2.tar.gz
RUN export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/mapr/lib:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/
Now build the new PACC image with cuda included:
docker build -t pacc_nvidia:cuda-9.0-base .
Copy the docker image to the VM where you have a GPU:
docker save -o pacc_nvidia_v1 pacc_nvidia
rsync -vapr --progress --partial --stats pacc_nvidia_v3 mapr@gcloud-mapr-client:~/
ssh to the GPU enabled mapr-client and load said docker image
docker load -i pacc_nvidia_v1
Run the steps in this section on the VM where you have a GPU. This is the VM that will be trying to detect faces, so here we'll be setting it up with the face detection model.
git clone https://github.com/mapr-demos/mapr-streams-mxnet-face
cd mapr-streams-mxnet-face
Download mxnet-face-fr50-0000.params from
dropbox
Copy it to consumer/deploy/
cp mxnet-face-fr50-0000.params consumer/deploy
Download model-0000.params model file from google drive
Copy it to consumer/models/
unzip model-r50-am-lfw.zip
cp model-r50-am-lfw/model-0000.params consumer/models/
Now we're ready to run the face detector. Run the following command on the VM with the GPU:
docker run -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_REQUIRE_CUDA="cuda>=8.0" --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse --memory 0 -e MAPR_CLUSTER=gcloud.cluster.com -e MAPR_MEMORY=0 -e MAPR_MOUNT_PATH=/mapr -e MAPR_TZ=America/Los_Angeles -e MAPR_CONTAINER_USER=mapr -e MAPR_CONTAINER_UID=500 -e MAPR_CONTAINER_GROUP=mapr -e MAPR_CONTAINER_GID=500 -e MAPR_CONTAINER_PASSWORD=mapr -e MAPR_CLDB_HOSTS=gcloudnodea.c.mapr-demos.internal -v /sys/fs/cgroup:/sys/fs/cgroup:ro --security-opt apparmor:unconfined -p 5000:5000 -p 5901:5901 -v /home/mapr/mapr-streams-mxnet-face:/tmp/mapr-streams-mxnet-face:ro --name pacc_nvidia pacc_nvidia
If you get the following error, that probably means you haven't attached a GPU to your VM, or your GPU is not compatible with Nvidia CUDA libraries. We recommend attaching an Nvidia Tesla K80 GPU.
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --compute --utility --require=cuda>=8.0 --pid=1375 /var/lib/docker/devicemapper/mnt/c5a646ba12b4c4406373b78189294321fd262b932fbbdfc7eea31046a3c133c4/rootfs]\\\\nnvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\"": unknown.
sudo apt-get update
sudo apt-get install tightvncserver aptitude tasksel -y
# this will take 4 minutes. Specify 29 and 1 for US keyboard:
sudo apt-get install xfce4 xfce4-goodies -y
vncserver
export DISPLAY=347d53c747ce:1 <-- use whatever string outputs after you start vncserver
Alias for starting the face detector on the VM with a GPU:
alias detect_faces='docker run -d -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_REQUIRE_CUDA="cuda>=8.0" --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse --memory 0 -e MAPR_CLUSTER=gcloud.cluster.com -e MAPR_MEMORY=0 -e MAPR_MOUNT_PATH=/mapr -e MAPR_TZ=America/Los_Angeles -e MAPR_CONTAINER_USER=mapr -e MAPR_CONTAINER_UID=500 -e MAPR_CONTAINER_GROUP=mapr -e MAPR_CONTAINER_GID=500 -e MAPR_CONTAINER_PASSWORD=mapr -e MAPR_CLDB_HOSTS=gcloudnodea.c.mapr-demos.internal -v /sys/fs/cgroup:/sys/fs/cgroup:ro --security-opt apparmor:unconfined -v /home/mapr/mapr-streams-mxnet-face:/tmp/mapr-streams-mxnet-face:ro --name pacc_nvidia2 pacc_nvidia & (sleep 15; docker exec -it pacc_nvidia2 su mapr -c "cd /tmp/mapr-streams-mxnet-face/consumer/deploy; LD_PRELOAD=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so python /tmp/mapr-streams-mxnet-face/consumer/deploy/mapr_consumer.py")'
Alias for starting the video producer on your laptop:
alias run_producer="docker run --rm -it --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse --memory 0 -e MAPR_CLUSTER=gcloud.cluster.com -e MAPR_MEMORY=0 -e MAPR_MOUNT_PATH=/mapr -e MAPR_TZ=America/Los_Angeles -e MAPR_CONTAINER_USER=mapr -e MAPR_CONTAINER_UID=500 -e MAPR_CONTAINER_GROUP=mapr -e MAPR_CONTAINER_GID=500 -e MAPR_CONTAINER_PASSWORD=mapr -e MAPR_CLDB_HOSTS=gcloudnodea.c.mapr-demos.internal -e DISPLAY=192.168.0.38:0 -v /sys/fs/cgroup:/sys/fs/cgroup:ro --security-opt apparmor:unconfined -v /Users/idownard/development/mapr-streams-mxnet-face:/tmp/mapr-streams-mxnet-face:ro --name pacc_nvidia pacc_nvidia"
Alias for starting the video producer on a mapr node:
alias monitor_streams='echo -e "\nSTREAM MONITORING:"; UPLINE=$(tput cuu1); ERASELINE=$(tput el); echo -e "\n\n\n"; while true; do x=maprcli stream topic list -path /tmp/rawvideostream -json | grep physicalsize | sed -s 's/.://' | tr -d ','; y=maprcli stream topic list -path /tmp/processedvideostream -json | grep physicalsize | sed -s 's/.://' | tr -d ','; z=maprcli stream topic list -path /tmp/identifiedstream -json | grep physicalsize | sed -s 's/.*://' | tr -d ','; echo -e "$UPLINE$ERASELINE$UPLINE$ERASELINE$UPLINE$ERASELINE\c"; echo -e "/tmp/rawvideostream size: $x\n/tmp/processedvideostream size: $y\n/tmp/identifiedstream size: $z"; done'
cd /tmp/mapr-streams-mxnet-face/consumer/deploy
LD_PRELOAD=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so python /tmp/mapr-streams-mxnet-face/consumer/deploy/mapr_consumer.py
nvidia-smi -l 1
or
pip install gpustat
watch --color -n1.0 gpustat --color
You can run the producer on any docker host as long as it can connect to the MapR cluster. I like to run it on my macbook where I can map a docker volume to ~/development/mapr-streams-mxnet-face where I keep the video file I want to produce.
Download the pacc_nvidia docker image we built earlier to your macbook and load it into Docker CE for Mac with the command docker load -i pacc_nvidia.
The producer will open a video player. Since I run the producer in a linux container, I first need to open xquartz on my mac and run xhost + so I can redirect the container's X11 display.
Now, start the producer with this command:
docker run -it --rm --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse --memory 0 -e MAPR_CLUSTER=gcloud.cluster.com -e MAPR_MEMORY=0 -e MAPR_MOUNT_PATH=/mapr -e MAPR_TZ=America/Los_Angeles -e MAPR_CONTAINER_USER=mapr -e MAPR_CONTAINER_UID=500 -e MAPR_CONTAINER_GROUP=mapr -e MAPR_CONTAINER_GID=500 -e MAPR_CONTAINER_PASSWORD=mapr -e MAPR_CLDB_HOSTS=gcloudnodea.c.mapr-demos.internal -e DOCKER_HOST_IP=`ifconfig en0 | awk '$1 == "inet" {print $2}'` -v /sys/fs/cgroup:/sys/fs/cgroup:ro --security-opt apparmor:unconfined -v ~/development/mapr-streams-mxnet-face:/tmp/mapr-streams-mxnet-face:ro --name pacc_nvidia pacc_nvidia
ln -s /opt/mapr/lib/libMapRClient.so.1 /opt/mapr/lib/libMapRClient_c.so
cp -R /tmp/mapr-streams-mxnet-face/producer ~
cd ~mapr/producer/
vi mapr-producer-video.py
update DLcluster to gcloud.cluster.com
:%s/DLcluster/gcloud.cluster.com/g
export DISPLAY=$DOCKER_HOST_IP:0
LD_PRELOAD=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so python mapr-producer-video-ian.py videos/PeopleInFrankfurt-small.mp4
(you can run this on your macbook):
docker run -it --rm --privileged --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse -e MAPR_CLUSTER=gcloud.cluster.com -e MAPR_CLDB_HOSTS=gcloudnodea -e MAPR_CONTAINER_USER=mapr -e MAPR_CONTAINER_UID=5000 -e MAPR_CONTAINER_GROUP=mapr -e MAPR_CONTAINER_GID=5000 -e MAPR_MOUNT_PATH=/mapr -e GROUPID=500 -e STREAM=/tmp/identifiedstream -e TOPIC=all -e TIMEOUT=0.035 -e PORT=5010 -p 5010:5010 --add-host "gcloudnodea":10.138.0.5 --name flask_client mengdong/mapr-pacc-mxnet:5.2.2_3.0.1_ubuntu16_yarn_fuse_hbase_streams_flask_client_arguments
The video will show up at the port you chose (go to 'http://localhost:5010').
The following Face Detection models were used in this project:
Here is a good video to use for this demo, since it shows many faces:
Dong Meng originally created this demo. Here is his original repository: