Jetpack 3.1 Results

[Alro10]

Hi @gustavz !

I tested run.py (changed video_input) using TF 1.3, and got this problem:

InvalidArgumentError (see above for traceback): NodeDef mentions attr 'dilations' not in Op<name=Conv2D; signature=input:T, filter:T -> output:T; attr=T:type,allowed=[DT_HALF, DT_FLOAT]; attr=strides:list(int); attr=use_cudnn_on_gpu:bool,default=true; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW"]>; NodeDef: MobilenetV2/Conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](sub_7, MobilenetV2/Conv/weights/read). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[Node: MobilenetV2/Conv/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](sub_7, MobilenetV2/Conv/weights/read)]]

I think you tested on Jetpack 3.2, sure?

Any configuration is necessary for deeplabv3 (installation)?

[gustavz]

Hey @AlexanderRobles21, deeplab models generally don’t work with tf versions below 1.5

If you upgrade you tf, please let me know if you get good results. I face strange visualizations on the jetson.

[Alro10]

Hi @gustavz . Recently, I tested on Jetpack 3.2 using TF 1.6 (video_input: /dev/video1) and found this issue:

[[Node: image_pooling/weights = Constdtype=DT_FLOAT, value=Tensor<type: float shape: [1,1,320,256] values: [[[0.0296224877 0.0163402725 0.00512141641]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

I am not sure if the problem is due on the Jetson memory capacity.

[gustavz]

@AlexanderRobles21 when/where do you face this error?
For me it executes normally on the jetson, but the results are completely random masks /only noise over the whole output image.

[Alro10]

Hey @gustavz , I face this error when I try to compile run.py
I realized what was the issue: Insufficiency GPU memory. if the variable bbox : False, it is possible to show segmentation but with poorly results.

Configuration:
Jetpack 3.2
TF 1.6 rc1
OpenCV 3.4

[gustavz]

@AlexanderRobles21 Can you post a sample screenshot of your results?
I guess its normal that the BBOX-PostProcessing Algorithm breaks the program if segmentation is garbage.

Anyways, have you ever been able to run the original deeplab repo on the jetson? I don't, so i guess it is tensroflow related. i opened an issue about this months ago, but it did not seem to be of any interest for the tensorflow guys...
tensorflow/models#3882

[Alro10]

@gustavz , this is a sample screenshot when demo.py runs.

This result is the same issue that you opened in tensorflow models. when I try to compile run.py using TF1.5:

[gustavz]

@AlexanderRobles21 Thats the same crap i see on my Jetson.
As far as i know its ARM-architecture related. I tried it with TF 1.5,1.6,1.7 on the jetson, but they all give the same garbage.

I don't think this is easy solvable, but needs to be done internally by tensorflow :/

[naisy]

It worked in my environment.
JetPack 3.1/Python 3.6.3/OpenCV 3.4.1/Tensorflow 1.6.0

but, it failed in JetPack 3.2/Python 3.6.3/OpenCV 3.4.1/Tensorflow 1.6.0.
This seems to be a problem specific to JetPack 3.2.

[Alro10]

HI! @naisy, this is really good!
Can you share your Python 3.6 .whl files for installing TensorFlow 1.6?

[naisy]

Hi, @AlexanderRobles21

My enviromnent is here.
https://github.com/naisy/JetsonTX2

1. Install JetPack 3.1
2. sudo apt-get install git
3. https://github.com/naisy/JetsonTX2
   (There is a reboot between setup.sh and install.sh.)
4. pip install cython
5. pip install scikit-image

[naisy]

I uninstalled CUDA 9 from JetPack 3.2 and installed CUDA 8 instead.
I rebuild OpenCV 3.4.1 and Tensorflow 1.6.1 and confirmed that they are displayed correctly.
JetPack 3.2 / CUDA 8 / Python 3.6.3 / OpenCV 3.4.1 / Tensorflow 1.6.1

It seems to be probably related to CUDA 9.
I do not know if it is a problem of Tensorflow implementation or CUDA 9 problem.

[gustavz]

@naisy thanks for investigating this!

How do you install CUDA8 on the Jetson, does it work the same way how you do it on a x86_64 architecture with the deb files?
Also which cuDNN do you use?

[naisy]

Hi @gustavz,

1. uninstall cuda

apt-get remove cuda*
apt-get remove libcudnn*

2. install JetPack 3.1's packages

dpkg -i cuda-repo-l4t-8-0-local_8.0.84-1_arm64.deb
dpkg -i libcudnn6_6.0.21-1+cuda8.0_arm64.deb
dpkg -i libcudnn6-dev_6.0.21-1+cuda8.0_arm64.deb
dpkg -i libcudnn6-doc_6.0.21-1+cuda8.0_arm64.deb

3. disable cuda9's repository

vi /etc/apt/sources.list.d/cuda-9-0-local.list

#deb file:///var/cuda-repo-9-0-local /

4. install cuda8

apt-get update
apt-get install cuda-toolkit-8-0
ldconfig

• Information

/usr/local/cuda/extras/demo_suite/deviceQuery

/usr/local/cuda/extras/demo_suite/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Tegra X2"
CUDA Driver Version / Runtime Version 9.0 / 8.0
CUDA Capability Major/Minor version number: 6.2
Total amount of global memory: 7846 MBytes (8227401728 bytes)
( 2) Multiprocessors, (128) CUDA Cores/MP: 256 CUDA Cores
GPU Max Clock rate: 1301 MHz (1.30 GHz)
Memory Clock rate: 1600 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = NVIDIA Tegra X2
Result = PASS

cat /usr/local/cuda/version.txt

CUDA Version 8.0.84

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_17:07:33_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

grep "#define CUDNN_MAJOR" -A2 /usr/include/cudnn.h

#define CUDNN_MAJOR 6
#define CUDNN_MINOR 0
#define CUDNN_PATCHLEVEL 21

I have not confirmed it in x86_64 architecture.

[gustavz]

you installed cuda 8 with apt get? And with that everything works for you?

Do you have JetPack 3.2, CUDA8 Tensorflow Wheelfiles? Version higher than 1.5 would be great!

I had a look at your enivronment, do you plan a JetPack3.2 / py2.7 Version?

[naisy]

Hi @gustavz,

JetPack 3.2/CUDA8 had been broken window manager.
Tensorflow/OpenCV could run on without window manager, but that was inconvenient.

[gustavz]

Do you mean with "broken window manager" That visualizing the ubuntu OS on a desktop fails?
So if i install CUDA8 on JetPack3.2 it breaks the window manager? So you only can use it over ssh. do i understand that correctly?

[naisy]

Hi GustavZ,

Session was alive. So, I could launch desktop. But "broken window manager".
Normally, GUI application (xterm, browser etc.) has window bar. The bar has close button.
When the window manager crashes, the bar disappears.
Xterm can launch but it has no window bar. It cannot move.

I can not recommend that environment.

By the way, in run.py
before:
with tf.Session(graph=detection_graph) as sess:
after:
with tf.Session(graph=detection_graph, config=config) as sess:
When I add config in tf.Session(), the noise vanished with JetPack3.2/CUDA9/Python3.6.3/OpenCV3.4.1.
But, with no detection class.
I tried on JetPack 3.2.1, but that was the same result, no detection.

[gustavz]

@naisy yes i saw similar with the config.
But i was not able to ever run deeplab on the jetson :/ and i dont want to try out the broken system you were talking about.

Do you still use it? I would be very interested how it looks, could you make some screenshots with the broken window manager?

[naisy]

Hi @gustavz,

Sorry, I deleted the broken environment.
I recommend JetPack 3.1 for deeplab environment.

The environment is here if you are OK with Python 3.6 with JetPack 3.1.
https://github.com/naisy/JetsonTX2
There is also Python 2.7 branch, but because I do not usually use it may fail to setup.

[vineetkr137]

HI , @gustavz
are you able to run deeplab on jetson without thrash results?

[gustavz]

I did not try it. But as naisy states, it should be possible with JetPack3.1

[sankim90]

HI @gustavz
Is there still noise problem at jetsonTX2?