computer_vision

Computer Vision

Photo Credit: Hannah O'Leary, Oregon State University

Right now most things are limited to a single car. Multi-car experiments are a work in progress.

End-to-End Learning

The current implementation is based on NVIDIA's DAVE-II Model.

Data Collection:

In three seperate terminals run the following:

Terminal 1:

$ roslaunch race f1_tenth_devel.launch

Terminal 2:

$ rosrun race disparity_extender_vanderbilt.py

Terminal 3:

$ rosrun computer_vision synchronize_img_command.py

The synchronize_img_command performs the data collection. It collects the steering angles and logs the data into a directory which can be found in the data directory, categorized based on the degree of the turn. This categorization is used to train the classification model in the next section. The classes are left, right, weak_left, weak_right, straight. You can change the classes if you wish. Simply add more classes in the following file.

To access and view the data run:

$ roscd computer_vision

I purposesly ignored the data directory for git tracking (It's very large). If you would like the data we used to train our models, I'm happy to provide it. Send me an email to obtain it.

Training

You can tweak the hyperparameters in the following file.

$ roscd computer_vision/training 
$ python train_daev_model.py -d data/ -o models/{name_of_your_model}.hdf5

Evaluation

In two seperate terminals run the following:

To run an experiment where the model controls the car.

Terminal 1:

$ roslaunch race f1_tenth_devel.launch

Terminal 2:

$ roscd computer_vision 
$ rosrun computer_vision ros_daev.py /racecar models/{name_of_your_model}.hdf5

To analyze the accuracy of your model. Instead of running the above in the second terminal. Run the following

Terminal 2:

$ roscd computer_vision/ 
$ rosrun computer_vision analyze_e2e.py /racecar models/{name_of_your_model}.hdf5 /vesc

Terminal 3:

$ rosrun race disparity_extender_vanderbilt.py

This will plot the error between the prediction from the neural network and ground truth (disparity extender).

You can also run the evaluation using the following launch. It assumes that your model is placed in the models directory within the computer vision package. Simply specify your model name in the following launch file and run the following after launching f1_tenth_devel.launch:

  $ roslaunch computer_vision end_to_end.launch

Classification Based Discrete Control

Data Collection: The data collection process is identical to the end-to-end scenario above.

Training

You can select the architechture and hyperparameters in the following file. Simply add your architechture in the nn/conv directory appropriately.

Models that are currently available:

• Mini VGGNET
• ShallowNet

To train a model run the following:

$ roscd computer_vision/training 
$ python train_classification_model.py -d data/ -o models/{name_of_your_model}.hdf5

Evalutation

Similarly to the end-to-end driving scenario, there are two ways to evaluate the model. The first methods maps the classifications to discrete actions in order to control the car. The second method simply runs the classification model online and identifies misclassifications art runtime. Since the disparity extender was used to generate the training data, we can evaulate its performance with respect its operation. (Very open to suggestions on how to improve this)

To run Discrete Control experiments:

Terminal 1:

$ roslaunch race f1_tenth_devel.launch

Terminal 2:

$ roscd computer_vision 
$ rosrun computer_vision ros_classifier.py /racecar models/{name_of_your_model}.hdf5

The discrete control actions are defined in the following file. Feel free to tweak them for your experiments.

In a similar vein to the end-to-end learning experiments. You can analyze how well your classification system adheres to the ground truth classifications generated by either the disparity extender or the potential field controller.

To analyze the accuracy of your model. Instead of running the above in the second terminal. Run the following:

Terminal 2:

$ roscd computer_vision 
$ rosrun computer_vision analyze_classification_model.py /racecar models/{name_of_your_model}.hdf5 /vesc

Terminal 3:

$ rosrun race disparity_extender_vanderbilt.py

This will plot a box plot with the number of detections labeled as misclassifications.

To run the end-to-end simulation:

  roscd && cd ..
$ docker-compose -f end_to_end.yml up

To teleoperate the car or run experiments run the following:

$ docker container exec -it keyboard bash 

Then run:

$ source devel/setup.bash && rosrun race keyboard.py

To run the end-to-end simulation:

$ roscd && cd ..
$ docker-compose -f end_to_end.yml up

End-to End Driving via Ensemble Based Classification

The training process of neural networks is challenging and in some cases it can fail. This can mean that the model at the end of training may not be stable or the best-peforming set of weights. One way to deal with this challenge is to use ensemble methods which refers to training a "large number of modles and then combining their output predictions via voting or averaging. The other method of combining several methds is to use an average of the weights of multiple models to create a new model, this form of combining methods is called Polyak-Ruppert Averaging. A discussion of these techniques can be found in the following tutorial by Dr. Jason Brownlee.

Training

Before we can use an ensemble of neural networks to make predictions. We must first make the ensemble. The file to train the ensemble can be found here. The syntax to train ensembles is given by:

python train_ensemble_classifier.py [-h] -d DATASET -o OUTPUT -l LOGS [-n NUM_MODELS]
	-h, --help show this help message and exit
	-d DATASET, --dataset DATASET path to input dataset
	-o OUTPUT, --output OUTPUT directory where we will output the model
	-l LOGS, --logs LOGS directory where we will output the plots and accuracy reports
	-n NUM_MODELS, --num-models NUM_MODELS # of models to train  

Example:

$ roscd computer_vision/training  
$ python train_ensemble_classifer.py -d data/ -o output/ -l logs/

The default is to train 5 models but you can change that with -n flag.

Combining Predictions Using Averaging

The first method I investigate for evaluating ensemble methods on the F1Tenth platform is inspired by Dr. Adrian Rosebrock. The idea here is that we create several ros nodes that make predictions on images received from the camera mounted on the car. Each model makes a prediction based on that image into 5 classes (left, weak left, straight, weak right, right) and then that prediction is sent to a ensemble manager. The ensemble manager then takes each of the prediction logits and averages them. This can be done in multiple ways but the default is to use a simple average. Based on that average the ensemble manager assigns the prediction to the highest value in the average logit. The architechture is shown below.

To run this demonstration run the following:

Terminal 1:

$ roslaunch race f1_tenth_devel.launch

Each node must be named in order for the ensemble manager to properly synchronize the messages. The name is the last parameter passed to the following commands. Each node gets its own terminal. Alternatively, you can create a launch file. The first parameter is the name of the racecar. This specifies which camera to subscribe to, if there are multiple.

Terminal 2-N:

$ roscd computer_vision 
$ rosrun race computer_vision ros_ensemble.py /racecar models/minivgg_3.hdf5 model_name

The names of each model are then passed to the ensemble manager.

Terminal N+1:

$ roscd computer_vision 
$ rosrun race computer_vision ensemble_manager.py /racecar model_1 .... model_n

Creating a new model using Polyak-Ruppert Averaging

Polyak-Ruppert Averaging is the idea that you can use an average of the weights from mulitple models seen toward the end of a training run to improve performance. It can be further improved by using a linearly or exponentially weighted average of the model weights. At runtime its considerably less computationally expensive than the ensemble approach presented in the previous section.

To run an experiment demonstrating the approach run the following:

$ roscd computer_vision
$ python python polyak_averaging_test.py -m ../models -s model_*.hdf5 -d ../data/

This will load five network models based on the MiniVGGNet architecture by selecting them using the model*.hdf5 search pattern. These models are then loaded into memory and the weights in each layer are combined using a simple average. These weights are then used to create the new model with the same architechture. Using the new model, the script makes predictions on the data contained in the data directory. In our experiments, using only five models resulted in a 25% accuracy. One explanation is that the average deviates largely from the actual weights exhibited by a single model. I did some experiments with a simple feedforward network and found that to be the case. The plots can be seen below. Once the accuracy improves, we can test these models on the F1Tenth Platform (Stay Tuned).

Feedforward Network

MiniVGGNet

Transfer Learning

In practice transfer learning is used widely as a means of acheiving state of the art performance. As mentioned in the following course notes by Andrej Karpathy, in practice very few people train an entire neural network from scratch. One often doesn't have a sufficient amount of data in order to do the training effectively. There are several different strategies for transfer learning and the following experiments use a strategy referred to as "fine tuning." The crux of the "fine tuning" strategy is replacing the final layers of a pre-trained convolutional neural network such as VGGNet, ResNet, or Inception with a new set of randomly initialized fully connected layers. Using this new architechture, all the layers from the pre-trained model are frozen, and a series of training epochs is conducted in order to allow the new set of FC layers to learn patterns from the pre-trained layers. Once that is done and depending on what your performance, you can unfreeze the pre-trained layers and continue training to further improve performance. These experiments were also inspired by Dr. Adrian Rosebrock.

Training

The pre-trained architechture, that I played with in these experiments is VGG16. The final layers were added using the following model definition. The architechture is visualized here. To train the model run the following commands:

$ python transfer_classifier.py [-h] -d DATASET -m MODEL
	-h, --help show this help message and exit
	-d DATASET, --dataset DATASET path to input dataset
	-m MODEL, --model MODEL path to save the output model

Example:

 $ python transfer_classifier.py -d ../data -m ../models/VGG16_transfer.hdf5