Classify having trouble finding python

[whilton9]

I'm trying to get this package to work with just the example images and example code, but I get an error about not being able to find python even though it's installed in the exact location I've specified. Here's the code and the error:

classify(path_prefix = "/home/pmcmahon/Documents/MLWIC2_Package_Images",
data_info = "/home/pmcmahon/Documents/image_labels.csv",
model_dir = "/home/pmcmahon/Documents/MLWIC2_helper_files",
python_loc = "/anaconda3/bin/",
save_predictions = "model_predictions.txt",
make_output = TRUE,
output_name = "MLWIC2_output.csv",
num_cores = 4
)

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv.
Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images.
You are not using a Windows computer.
sh: 1: /anaconda3/bin/python: not found
The classify function did not run properly.
Warning message:
In system(eval_py) : error in running command

[mikeyEcology]

I don't understand why it's not finding python. Can you open your terminal and type /anaconda3/bin/python? This will open python. Then type print(2+2). See what happens

[mikeyEcology]

If you're able to run python from the command line like this, then let's try adding print_cmd=TRUE to your classify call, so it should look like this:

classify(path_prefix = "/home/pmcmahon/Documents/MLWIC2_Package_Images",
data_info = "/home/pmcmahon/Documents/image_labels.csv",
model_dir = "/home/pmcmahon/Documents/MLWIC2_helper_files",
python_loc = "/anaconda3/bin/",
save_predictions = "model_predictions.txt",
make_output = TRUE,
output_name = "MLWIC2_output.csv",
num_cores = 4,
print_cmd=TRUE # this is the changed line
)

This will print the command that is being passed to python.

Then in your terminal window, exit python by typing exit(), navigate to your MLWIC2_helper_files folder in the terminal (by typing cd /home/pmcmahon/Documents/MLWIC2_helper_files, and open Python again by typing python. Then copy the output from the R console and paste it into python in your terminal.

[whilton9]

I was able to open python and typing print(2+2) yielded an answer of 4, so it appears to be working. I then did everything you suggested in the last comment. However, when I remove print_cmd=TRUE, I still get the message saying that the classify function did not run properly.

[mikeyEcology]

Yes-it's not running properly because it's printing something instead of running, but it should have printed a system call that will look like python run.py eval --num_threads .... Do you see this?

[whilton9]

Yes it printed this: "/anaconda3/bin/python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000\n"

[mikeyEcology]

Ok. so copy that and paste it into your python, which you can get to by typing /anaconda3/bin/python into a terminal window.

[whilton9]

I did that, but nothing has changed.

[mikeyEcology]

Is there an error? Otherwise it will be running in the background. Does this file exist /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt?
I didn't mean to close this issue, keep posting here and I think it will reopen

[whilton9]

I wasn't sure if I should remove print_cmd=TRUE or not. If I leave it in the command, this is the output:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv.
Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images.
You are not using a Windows computer.
[1] "/anaconda3/bin/python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000\n"

If I remove it, this is the output:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv.
Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images.
You are not using a Windows computer.
sh: 1: /anaconda3/bin/python: not found
The classify function did not run properly.
Warning message:
In system(eval_py) : error in running command

So there was an error and that file doesn't exist.

[mikeyEcology]

What happens if in the terminal you navigate to /anaconda3/bin/ and then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

[whilton9]

It output this:
(null): can't open file 'run.py': [Errno 2] No such file or directory

[mikeyEcology]

Sorry. In type cd /home/pmcmahon/Documents/MLWIC2_helper_files, then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

[whilton9]

Ok I changed to that directory and typed the command, but got the same error message

[mikeyEcology]

what is in your MLWIC2_helper_files folder? Can you type ls in the terminal while in this folder?

[whilton9]

data_info_train.csv __MACOSX MLWIC2_helper_files MLWIC2_test_tf.py

[mikeyEcology]

Ok. Something went wrong when you downloaded the helper files folder. Can you try downloading it from the readme and unzipping it. It should contain:

ConfMatrixID.py pycache/ architectures/ empty_animal/ run.py utils.py
arch.py checkpoint data_loader.py species_model/

[whilton9]

Ok I downloaded it again and it contains all those things. However, I'm getting the same errors when running the classify command

[mikeyEcology]

Ok. Try this again: in terminal type cd /home/pmcmahon/Documents/MLWIC2_helper_files, then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

[whilton9]

Here's the output after doing that:

Traceback (most recent call last):
File "run.py", line 21, in
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

[mikeyEcology]

This means that tensorflow is not installed properly. I recently updated the setup function so that it should install tensorflow properly on your machine. You can update MLWIC2 and run setup again, or you can do it by yourself in conda. In terminal type conda activate r-reticulate then pip install tensorflow

[whilton9]

I ran setup again and got this output when I ran classify:

Your data_info file exists: /home/pmcmahon/Documents/image_labels.csv.
Your `path_prefix exists: /home/pmcmahon/Documents/MLWIC2_Package_Images.
You are not using a Windows computer.
sh: 1: /anaconda3/bin/python: not found
The classify function did not run properly.
Warning message:
In system(eval_py) : error in running command

I then ran those commands in the terminal and got the same output when I tried running classify again

[mikeyEcology]

What was your output when you ran it in the terminal?

[whilton9]

Collecting tensorflow
Downloading tensorflow-2.3.0-cp38-cp38-manylinux2010_x86_64.whl (320.5 MB)
|████████████████████████████████| 320.5 MB 732 bytes/s
Collecting tensorboard<3,>=2.3.0
Downloading tensorboard-2.3.0-py3-none-any.whl (6.8 MB)
|████████████████████████████████| 6.8 MB 7.2 MB/s
Collecting h5py<2.11.0,>=2.10.0
Downloading h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)
|████████████████████████████████| 2.9 MB 6.7 MB/s
Collecting grpcio>=1.8.6
Downloading grpcio-1.32.0-cp38-cp38-manylinux2014_x86_64.whl (3.8 MB)
|████████████████████████████████| 3.8 MB 9.4 MB/s
Collecting wrapt>=1.11.1
Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting gast==0.3.3
Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Requirement already satisfied: wheel>=0.26 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from tensorflow) (0.35.1)
Collecting numpy<1.19.0,>=1.16.0
Downloading numpy-1.18.5-cp38-cp38-manylinux1_x86_64.whl (20.6 MB)
|████████████████████████████████| 20.6 MB 7.7 MB/s
Collecting astunparse==1.6.3
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting google-pasta>=0.1.8
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
|████████████████████████████████| 57 kB 3.5 MB/s
Collecting tensorflow-estimator<2.4.0,>=2.3.0
Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
|████████████████████████████████| 459 kB 20.3 MB/s
Collecting absl-py>=0.7.0
Downloading absl_py-0.10.0-py3-none-any.whl (127 kB)
|████████████████████████████████| 127 kB 7.2 MB/s
Collecting six>=1.12.0
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting scipy==1.4.1
Downloading scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB)
|████████████████████████████████| 26.0 MB 5.6 MB/s
Collecting termcolor>=1.1.0
Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Collecting protobuf>=3.9.2
Downloading protobuf-3.13.0-cp38-cp38-manylinux1_x86_64.whl (1.3 MB)
|████████████████████████████████| 1.3 MB 9.1 MB/s
Collecting keras-preprocessing<1.2,>=1.1.1
Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
|████████████████████████████████| 42 kB 1.1 MB/s
Collecting opt-einsum>=2.3.2
Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
|████████████████████████████████| 65 kB 2.9 MB/s
Collecting tensorboard-plugin-wit>=1.6.0
Downloading tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
|████████████████████████████████| 779 kB 12.8 MB/s
Collecting google-auth-oauthlib<0.5,>=0.4.1
Downloading google_auth_oauthlib-0.4.1-py2.py3-none-any.whl (18 kB)
Collecting requests<3,>=2.21.0
Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 410 kB/s
Collecting werkzeug>=0.11.15
Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
|████████████████████████████████| 298 kB 10.6 MB/s
Requirement already satisfied: setuptools>=41.0.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (49.6.0.post20200814)
Collecting google-auth<2,>=1.6.3
Downloading google_auth-1.21.1-py2.py3-none-any.whl (93 kB)
|████████████████████████████████| 93 kB 600 kB/s
Collecting markdown>=2.6.8
Downloading Markdown-3.2.2-py3-none-any.whl (88 kB)
|████████████████████████████████| 88 kB 2.9 MB/s
Collecting requests-oauthlib>=0.7.0
Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting chardet<4,>=3.0.2
Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
|████████████████████████████████| 133 kB 10.5 MB/s
Requirement already satisfied: certifi>=2017.4.17 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (2020.6.20)
Collecting idna<3,>=2.5
Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
|████████████████████████████████| 58 kB 4.3 MB/s
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
|████████████████████████████████| 127 kB 12.9 MB/s
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
|████████████████████████████████| 155 kB 6.2 MB/s
Collecting cachetools<5.0,>=2.0.0
Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.5"
Downloading rsa-4.6-py3-none-any.whl (47 kB)
|████████████████████████████████| 47 kB 4.1 MB/s
Collecting oauthlib>=3.0.0
Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
|████████████████████████████████| 147 kB 20.8 MB/s
Collecting pyasn1<0.5.0,>=0.4.6
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
|████████████████████████████████| 77 kB 4.3 MB/s
Building wheels for collected packages: wrapt, termcolor
Building wheel for wrapt (setup.py) ... done
Created wheel for wrapt: filename=wrapt-1.12.1-cp38-cp38-linux_x86_64.whl size=78072 sha256=573b8e05638979fb768a30d9d098636a87d20beb1f90328fb72489b6260a61fb
Stored in directory: /home/pmcmahon/.cache/pip/wheels/5f/fd/9e/b6cf5890494cb8ef0b5eaff72e5d55a70fb56316007d6dfe73
Building wheel for termcolor (setup.py) ... done
Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=409e8ae29881363c2d813e7e199e890585d6e3522dd9eb09885aa7ae704a7d81
Stored in directory: /home/pmcmahon/.cache/pip/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501
Successfully built wrapt termcolor
Installing collected packages: six, absl-py, tensorboard-plugin-wit, numpy, oauthlib, chardet, idna, urllib3, requests, requests-oauthlib, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, google-auth-oauthlib, werkzeug, protobuf, markdown, grpcio, tensorboard, h5py, wrapt, gast, astunparse, google-pasta, tensorflow-estimator, scipy, termcolor, keras-preprocessing, opt-einsum, tensorflow
Successfully installed absl-py-0.10.0 astunparse-1.6.3 cachetools-4.1.1 chardet-3.0.4 gast-0.3.3 google-auth-1.21.1 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 grpcio-1.32.0 h5py-2.10.0 idna-2.10 keras-preprocessing-1.1.2 markdown-3.2.2 numpy-1.18.5 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.13.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.24.0 requests-oauthlib-1.3.0 rsa-4.6 scipy-1.4.1 six-1.15.0 tensorboard-2.3.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.3.0 tensorflow-estimator-2.3.0 termcolor-1.1.0 urllib3-1.25.10 werkzeug-1.0.1 wrapt-1.12.1

[mikeyEcology]

Shoot, I forgot you need the right version of tensorflow. In conda run pip uninstall tensorflow and if necessary, keep running it until you get the error "Cannot uninstall requirement tensorflow, not installed" (This is a very finicky package). Then run pip install tensorflow==1.14

[whilton9]

After running the uninstall command twice, I got this:
WARNING: Skipping tensorflow as it is not installed.

But when I ran the install command I got:
ERROR: Could not find a version that satisfies the requirement tensorflow==1.14 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0)
ERROR: No matching distribution found for tensorflow==1.14

[mikeyEcology]

try pip install tensorflow==1.14.0?

[whilton9]

Nah, I got the same error

[mikeyEcology]

The problem is you're using the wrong python version. You can change it in conda by using conda install python==3.7

[whilton9]

Ok I installed python3.7 and tensorflow1.14.0, but classify gave me the same error

[mikeyEcology]

Did you try running it from the terminal?

[whilton9]

I'm not sure what you mean. What should I run in the terminal?

[mikeyEcology]

type cd /home/pmcmahon/Documents/MLWIC2_helper_files, then type python run.py eval --num_threads 4 --architecture resnet --depth 18 --log_dir species_model --snapshot_prefix species_model --path_prefix /home/pmcmahon/Documents/MLWIC2_Package_Images --batch_size 128 --val_info /home/pmcmahon/Documents/image_labels.csv --delimiter , --save_predictions /home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt --top_n 5 --num_gpus 2 --num_classes 1000

[whilton9]

I ran those commands, but I am not seeing any differences when I run classify in R. Do I need to make any changes to the R command?

[mikeyEcology]

What do you mean you don't see any differences? Did you run this from the terminal?

[whilton9]

I mean that when I run classify in R, I still get the same error that I've been getting the whole time. I did run that command in the terminal, but it didn't do anything. Am I supposed to be running classify through the terminal as well? R is really the only programming language I have any meaningful experience with, so I have no idea what these commands I'm running in the terminal are supposed to be doing.

[mikeyEcology]

What do you mean it didn't do anything in the terminal? Nothing showed up when you ran this command? We're only using the terminal to troubleshoot.

[whilton9]

I ran that in the terminal and this was the output:
Traceback (most recent call last):
File "run.py", line 21, in
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

[mikeyEcology]

You need to install tensorflow. Type conda activate r-reticulate
pip install tensorflow==1.14

[whilton9]

Ok I did that. This was the output:

Requirement already satisfied: tensorflow==1.14 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (1.14.0)
Requirement already satisfied: astor>=0.6.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (0.8.1)
Requirement already satisfied: absl-py>=0.7.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (0.10.0)
Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.14.0)
Requirement already satisfied: grpcio>=1.8.6 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.32.0)
Requirement already satisfied: wrapt>=1.11.1 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.12.1)
Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.14.0)
Requirement already satisfied: wheel>=0.26 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (0.35.1)
Requirement already satisfied: termcolor>=1.1.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.1.0)
Requirement already satisfied: keras-applications>=1.0.6 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.0.8)
Requirement already satisfied: six>=1.10.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.15.0)
Requirement already satisfied: gast>=0.2.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (0.4.0)
Requirement already satisfied: numpy<2.0,>=1.14.5 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.19.1)
Requirement already satisfied: protobuf>=3.6.1 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (3.13.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (1.1.2)
Requirement already satisfied: google-pasta>=0.1.6 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorflow==1.14) (0.2.0)
Requirement already satisfied: setuptools>=41.0.0 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (49.6.0.post20200814)
Requirement already satisfied: werkzeug>=0.11.15 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (3.2.2)
Requirement already satisfied: h5py in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from keras-applications>=1.0.6->tensorflow==1.14) (2.10.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (1.7.0)
Requirement already satisfied: zipp>=0.5 in /home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (3.1.0)

Since it said, requirement already satisfied, I tried the previous command again and this was the output:

2020-09-29 13:42:30.294548: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2691265000 Hz
2020-09-29 13:42:30.296011: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55dff91e5380 executing computations on platform Host. Devices:
2020-09-29 13:42:30.296068: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
Namespace(LR_details='19, 30, 44, 53, 0.01, 0.005, 0.001, 0.0005, 0.0001', LR_policy='piecewise_linear', WD_details='30, 0.0005, 0.0', WD_policy='piecewise_linear', architecture='resnet', batch_size=128, chunked_batch_size=64, command='eval', delimiter=',', depth=18, log_debug_info=False, log_device_placement=False, log_dir='species_model', max_to_keep=5, num_batches=-1, num_classes=1000, num_epochs=55, num_gpus=2, num_prefetch=2000, num_threads=4, optimizer='momentum', path_prefix='/home/pmcmahon/Documents/MLWIC2_Package_Images', processed_size=[224, 224, 3], raw_size=[256, 256, 3], retrain_from=None, run_metadata=None, run_name='Run-29-09-2020_13-42-30', run_options=None, save_predictions='/home/pmcmahon/Documents/MLWIC2_helper_files/model_predictions.txt', shuffle=True, snapshot_prefix='species_model', top_n=5, train_info=None, transfer_mode=[0], val_info='/home/pmcmahon/Documents/image_labels.csv')
found 2 classes
{0: 0, 1: 1}
Filling queue with 2000 images before starting to train. This may take some time.
2020-09-29 13:42:32.524308: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2020-09-29 13:42:32.940708: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/0.jpg; No such file or directory
2020-09-29 13:42:32.983918: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/1.jpg; No such file or directory
2020-09-29 13:42:33.025684: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/2.jpg; No such file or directory
2020-09-29 13:42:33.068496: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/4.jpg; No such file or directory
Traceback (most recent call last):
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_batch/fifo_queue' is closed and has insufficient elements (requested 100, current size 0)
[[{{node batch}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run.py", line 412, in
main()
File "run.py", line 394, in main
do_evaluate(sess, args)
File "run.py", line 253, in do_evaluate
val_img, val_lbl, val_inf = sess.run([val_images, val_labels, val_info],feed_dict={batch_size_tf: args.num_val_samples%args.batch_size} if step==args.num_val_batches-1 else None)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_batch/fifo_queue' is closed and has insufficient elements (requested 100, current size 0)
[[node batch (defined at /home/pmcmahon/Documents/MLWIC2_helper_files/data_loader.py:161) ]]

Errors may have originated from an input operation.
Input Source operations connected to node batch:
PlaceholderWithDefault (defined at run.py:216)

Original stack trace for 'batch':
File "run.py", line 412, in
main()
File "run.py", line 394, in main
do_evaluate(sess, args)
File "run.py", line 224, in do_evaluate
val_images, val_labels, val_info = val_loader.load()
File "/home/pmcmahon/Documents/MLWIC2_helper_files/data_loader.py", line 161, in load
allow_smaller_final_batch=True if not self.is_training else False)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/training/input.py", line 1021, in batch
name=name)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/training/input.py", line 788, in _batch
dequeued = queue.dequeue_up_to(batch_size, name=name)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 532, in dequeue_up_to
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4042, in queue_dequeue_up_to_v2
timeout_ms=timeout_ms, name=name)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/pmcmahon/anaconda3/envs/r-reticulate/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

[mikeyEcology]

Ok. Now everything is working except you have some mistake in your paths. It's looking for images at /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/ and can't find them. Have you added C:/Users/winst/Documents/USDA/MLWIC2 Package Images/ to your image label file or something?

[whilton9]

I'm not entirely sure what you mean. /home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/ doesn't exist and I'm not sure why it'd be looking for the images there. The images are stored at /home/pmcmahon/Documents/MLWIC2_Package_Images

[mikeyEcology]

Right. The problem is that it doesn't exist, but in your output, it is looking for these images

/home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/4.jpg
/home/pmcmahon/Documents/MLWIC2_Package_Images/C:/Users/winst/Documents/USDA/MLWIC2 Package Images/2.jpg
...

So you must have the C:/.. in your code or image label file somewhere.

[whilton9]

I don't see that in the code anywhere. Both the code I used in R and the code I used in the terminal have the correct path. I also don't know how the image label file could be wrong because I thought that's just where the images are saved on this computer. So I'm not sure what's causing that or how to fix it.

[whilton9]

Ok I think I got that issue fixed