Run FAI PEP for the first time

When you use FAI-PEP to benchmark a model for the first time, you may need to setup some default flags. Those flags are not performance related and seldom change. The default flags are saved under directory ~/.aibench/git/config.txt. In later runs, the file is read and the default flags parsed. However, those flags can be overwritten in the command line.

A sample answers to the questions are:

./benchmarking/run_bench.py -b specifications/models/caffe2/squeezenet/squeezenet.json
Please enter the directory the framework repo resides [None]: ~/caffe2/pytorch
Please enter the remote reporter [None]:
Please enter the remote access token [None]:
Please enter the root model dir if needed [None]:
Do you want to print report to screen? [None]: Yes

The important question is the first one, which asks for the github repo for the framework you plan to use in the benchmark (i.e. pytorch, tensorflow).

After this, a config.txt file is generated:

{
  "--commit": "master",
  "--commit_file": "/Users/feisun/.aibench/git/processed_commit",
  "--exec_dir": "/Users/feisun/.aibench/git/exec",
  "--framework": "caffe2",
  "--local_reporter": "/Users/feisun/.aibench/git/reporter",
  "--model_cache": "/Users/feisun/.aibench/git/model_cache",
  "--platforms": "android",
  "--remote_repository": "origin",
  "--repo": "git",
  "--repo_dir": "~/caffe2/pytorch",
  "--screen_reporter": null,
  "--status_file": "/Users/feisun/.aibench/git/status",
  "--timeout": 300
}

If you plan to change the defaults, you can edit the file directly. Probably you may want to change the framework or platforms if you don't work on caffe2 and android.

Please be sure the framework repo can build the framework runtime before running FAI-PEP. If the build fails, FAI-PEP reports a build error. On some framework, e.g. Caffe2, if some libraries are not installed but the framework build desires, the framework may silently ignores the library.