Run Imagenet validate dataset

The imagenet validation dataset contains 50K images for 1K categories, with 50 images per category. The total size is about 7GB, which may not fit in the flash drive of some of the phones/embedded devices. However, we do want to benchmark the performance as well as accuracy on phones for the entire validation dataset. FAI-PEP provides an easy way for people to achieve that goal.

FAI-PEP uses a batch mechanism to reduce the required dataset size. Obviously the benchmark runtime cannot process 50K images in one iteration. Thus, we can randomly divide the images to many mini-batches (e.g. 500 mini-batches), and the benchmark runtime only requires the data for that mini-batch, which consumes significantly less space.

An example of the the json specification is here. An example of the command to run the benchmark is:

./benchmarking/run_bench.py -b specifications/models/caffe2/squeezenet/squeezenet_accuracy_imagenet.json --string_map '{"imagenet_dir": "<local imagenet directory>"}' --platforms android/with_host

Please note the OpenCV needs to be installed on the host server. Below some key steps are explained.

Fields in model

The repeat field indicates the number of iterations the test would be performed. To separate out different iterations in the test, you can use placeholder {INDEX}. Details of the placeholder strings can be found here.

The preprocess field contains commands that set up some temporary directory and randomly generates 100 images from the total 50K images. If you want to process the the entire dataset, change the {total_images} to 50000 and remove arguments --limit-files {total_images} --limit-per-category 1 from the command line. You may also change repeat to modify the number of iterations.

The postprocess field contains a command that runs a python script aggregate_classification_results.py to aggregate all outputs from all iterations of the test.

Fields in tests

The preprocess field is executed on the host machine. It contains an awk script that extracts the images and labels to process for the current iteration. It also converts the images to the tensor format that caffe2 can immediately consume. The binary {convert_image_to_tensor} is a host binary that is built using script android/with_host.

The input_files field contains the generated tensor by {convert_image_to_tensor}. The tensor is placed on the host location. It is then referenced in the benchmark runtime in commands. FAI-PEP is smart enough to copy the file from host to target.

The commands field specifies all the commands running on the target platform. The main benchmark runtime is executed in this field. If, we need to measure the runtime of the preprocess and postprocess steps, we also need to put those steps here instead of in the preprocess and postprocess fields, which runs on the host machine.

Theoutput_files field contains the output files generated by the main benchmark runtime. FAI-PEP copies all the files to the host to compare with the golden labels.

The postprocess field is executed on the host machine. The script classification_compare.py compares the output with the golden label and saves the result to a file, which is used by the postprocess in the model field.