Add config for running condda 18

README.rst

@@ -8,7 +8,13 @@ Protocols for using new Tinker or Hydra with current SAIL-ON implementation
 * Free software: Apache Software License 2.0
 * Documentation: Compile Seperately... still in progress
 
+Month 18 Evaluation Instructions
+--------------------------------
 
+1. `Image Classification <ic_m18.md>`_
+
+.. warning::
+    The documentation underneath this section is out of date please ignore it.
 
 Step 1: Install
 --------

configs/problem/ic_conda18.yaml

@@ -0,0 +1,53 @@
+# @package _group_
+protocol: 'condda'
+workdir: ''
+harness: 'local'
+domain: image_classification
+test_ids:
+- OND.54011215.0000.1236
+hints:
+- red_light
+dataset_root: ""
+detectors:
+  has_reaction_baseline: 'False'
+  has_baseline: 'False'
+  cores: 6
+  detection_threshold: 0.1
+  detector_configs:
+    CONDDA_18_With_Redlight:
+      cnn_params:
+          model_path: ""
+          known_classes: 413
+          known_unknown_classes: 51
+          image_size: 320
+      evm_params:
+         model_path: ""
+         feature_known_path: ""
+         tail_size: 40000
+         cover_threshold: 0.8
+         distance_multiplier: 0.65
+         unknown_distance_multiplier: 0.80
+         distance_function: "cosine"
+         chunk_size: 200
+         gpu_list:
+           - 0
+           - 1
+         mu_train: 1.0
+         sigma_train: 0.028588828
+         KL_threshold: 62.855897282606946
+         threshold_evm: 0.5
+         number_of_unknown_to_crate_evm: 10
+      characterization_param:
+         clustering_type: "FINCH"
+         number_of_unknown_to_strat_clustering: 100
+         minimum_number_of_cluster_to_be_accepted: 4
+      dataloader_params:
+          batch_size: 100
+          num_workers: 6
+  csv_folder: 'condda_results'
+save_dir: "{workdir.id}"
+harness_config:
+  api_url: 'http://3.32.8.161:5001/'
+  data_dir: ''
+  gt_dir: ''
+  gt_config: ''

configs/problem/ic_conda18_wo_redlight.yaml

@@ -0,0 +1,53 @@
+# @package _group_
+protocol: 'condda'
+workdir: ''
+harness: 'local'
+domain: image_classification
+test_ids:
+- OND.54012315.0900.1239
+is_eval_enabled: False
+dataset_root: ""
+detectors:
+  has_reaction_baseline: 'False'
+  has_baseline: 'False'
+  cores: 6
+  detection_threshold: 0.1
+  detector_configs:
+    CONDDA_18_Without_Redlight:
+      cnn_params:
+          model_path: ""
+          known_classes: 413
+          known_unknown_classes: 51
+          image_size: 320
+      evm_params:
+         model_path: ""
+         feature_known_path: ""
+         tail_size: 40000
+         cover_threshold: 0.8
+         distance_multiplier: 0.65
+         unknown_distance_multiplier: 0.80
+         distance_function: "cosine"
+         chunk_size: 200
+         gpu_list:
+           - 0
+           - 1
+         mu_train: 1.0
+         sigma_train: 0.028588828
+         KL_threshold: 62.855897282606946
+         threshold_evm: 0.5
+         number_of_unknown_to_crate_evm: 10
+      characterization_param:
+         clustering_type: "FINCH"
+         number_of_unknown_to_strat_clustering: 100
+         minimum_number_of_cluster_to_be_accepted: 4
+      dataloader_params:
+          batch_size: 100
+          num_workers: 6
+  csv_folder: 'condda_results'
+save_attributes: "False"
+save_dir: "{workdir.id}"
+harness_config:
+  api_url: 'http://3.32.8.161:5001/'
+  data_dir: ''
+  gt_dir: ''
+  gt_config: ''

configs/problem/ic_ond18.yaml

@@ -0,0 +1,53 @@
+# @package _group_
+protocol: 'ond'
+workdir: ''
+harness: 'local'
+domain: "image_classification"
+test_ids:
+- "OND.54011215.0000.1236"
+hints:
+- "red_light"
+use_feedback: True
+is_eval_enabled: False
+dataset_root: ""
+detectors:
+  has_reaction_baseline: False
+  has_baseline: False
+  csv_folder: image_classification
+  detector_configs:
+    OND_18_With_Redlight:
+      cnn_params:
+          model_path: ""
+          known_classes: 413
+          known_unknown_classes: 51
+          image_size: 320
+      evm_params:
+         model_path: ""
+         tail_size: 40000
+         cover_threshold: 0.8
+         distance_multiplier: 0.65
+         distance_function: "cosine"
+         chunk_size: 200
+         gpu_list:
+           - 0
+           - 1
+         mu_train: 1.0
+         sigma_train: 0.028588828
+         KL_threshold: 62.855897282606946
+         threshold_evm: 0.5
+      dataloader_params:
+          batch_size: 100
+          num_workers: 6
+      feedback_params:
+          first_budget: 10
+          income_per_batch: 10
+          maximum_budget: 10
+  save_elementwise: False
+  save_attributes: False
+  use_saved_attributes: False
+save_dir: "{workdir.id}"
+harness_config:
+  api_url: 'http://3.32.8.161:5001/'
+  data_dir: ''
+  gt_dir: ''
+  gt_config: ''

configs/problem/ic_ond18_reaction.yaml

@@ -0,0 +1,64 @@
+# @package _group_
+protocol: 'ond'
+workdir: ''
+harness: 'local'
+domain: "image_classification"
+test_ids:
+- "OND.54011215.0000.1236"
+hints:
+- "red_light"
+use_feedback: True
+is_eval_enabled: True
+is_eval_roundwise_enabled: True
+dataset_root: ""
+detectors:
+  has_reaction_baseline: True
+  has_baseline: False
+  csv_folder: image_classification
+  baseline_class: "BaselineImageClassifier"
+  detector_configs:
+    BaselineImageClassifier:
+      model_name: "resnet50"
+      checkpoint_path: ""
+      image_size: 256
+      crop_size: 224
+      batch_size: 100
+      num_workers: 6
+      num_classes: 413
+      use_gpu: True
+    OND_18_With_Redlight:
+      cnn_params:
+          model_path: ""
+          known_classes: 413
+          known_unknown_classes: 51
+          image_size: 320
+      evm_params:
+         model_path: ""
+         tail_size: 40000
+         cover_threshold: 0.8
+         distance_multiplier: 0.65
+         distance_function: "cosine"
+         chunk_size: 200
+         gpu_list:
+           - 0
+           - 1
+         mu_train: 1.0
+         sigma_train: 0.028588828
+         KL_threshold: 62.855897282606946
+         threshold_evm: 0.5
+      dataloader_params:
+          batch_size: 100
+          num_workers: 6
+      feedback_params:
+          first_budget: 10
+          income_per_batch: 10
+          maximum_budget: 10
+  save_elementwise: False
+  save_attributes: False
+  use_saved_attributes: False
+save_dir: "{workdir.id}"
+harness_config:
+  url: 'http://3.32.8.161:5001/'
+  data_dir: ''
+  gt_dir: ''
+  gt_config: ''

configs/problem/ic_without_rd_ond18.yaml

@@ -0,0 +1,51 @@
+# @package _group_
+protocol: 'ond'
+workdir: ''
+harness: 'local'
+domain: "image_classification"
+test_ids:
+- "OND.54011215.0000.1236"
+use_feedback: True
+is_eval_enabled: False
+dataset_root: ""
+detectors:
+  has_reaction_baseline: Flase
+  has_baseline: False
+  csv_folder: image_classification
+  detector_configs:
+    OND_18_Without_Redlight:
+      cnn_params:
+          model_path: ""
+          known_classes: 413
+          known_unknown_classes: 51
+          image_size: 320
+      evm_params:
+         model_path: ""
+         tail_size: 40000
+         cover_threshold: 0.8
+         distance_multiplier: 0.65
+         distance_function: "cosine"
+         chunk_size: 200
+         gpu_list:
+           - 0
+           - 1
+         mu_train: 1.0
+         sigma_train: 0.028588828
+         KL_threshold: 62.855897282606946
+         threshold_evm: 0.5
+      dataloader_params:
+          batch_size: 100
+          num_workers: 0
+      feedback_params:
+          first_budget: 10
+          income_per_batch: 10
+          maximum_budget: 10
+  save_elementwise: False
+  save_attributes: False
+  use_saved_attributes: False
+save_dir: "{workdir.id}"
+harness_config:
+  api_url: 'http://3.32.8.161:5001/'
+  data_dir: ''
+  gt_dir: ''
+  gt_config: ''

configs/problem/image_classification_condda_config.yaml

@@ -14,7 +14,7 @@ detectors:
   cores: 6
   detection_threshold: 0.1
   detector_configs:
-    CONDDA_12_With_Redlight
+    CONDDA_12_With_Redlight:
       efficientnet_params:
         model_path: "/mnt/3bcbbdb5-8833-4d40-a51a-7fca7f39ec24/data2/sailon_data/models/OND_12/trained_efficientnet_b3_CC.pth.tar"
         known_classes: 413

ic_m18.md

@@ -0,0 +1,88 @@
+# Instructions for Image Classification
+
+## Installation
+1. Clone the following repositories
+  ```
+      git clone https://github.com/tinker-engine/tinker-engine.git
+      git clone https://github.com/darpa-sail-on/sail-on-api.git
+      git clone https://github.com/darpa-sail-on/evm_based_novelty_detector.git
+      git clone https://github.com/darpa-sail-on/sail-on-client.git
+      git clone https://github.com/darpa-sail-on/Sail_On_Evaluate.git
+      git clone https://github.com/darpa-sail-on/sailon_tinker_launcher.git
+  ```
+   This would create tinker-engine, sail-on-api, evm_based_novelty_detector,
+   sailon_tinker_launcher, Sail_On_Evaluate and sail-on-client directories
+   in your working directory.
+
+2. Checkout branches from different repositories
+  ```
+    cd sail-on-client
+    git checkout update-condda
+    cd ../evm_based_novelty_detector
+    git checkout add-eval-18-adapters
+    cd ../sailon_tinker_launcher
+    git checkout add-condda-18-configs
+  ```
+
+3. Create virtual environment for sail-on-client
+  ```
+    cd ../sail-on-client
+    pipenv install
+    pipenv shell
+  ```
+
+4. Install evm_based_novelty_detector
+  ```
+    cd ../evm_based_novelty_detector
+    pip install -r requirements.txt
+    pip install -e timm
+    pip install -e .
+  ```
+
+5. Install sailon_tinker_launcher
+  ```
+    cd ../sailon_tinker_launcher
+    pip install -r requirements.txt
+    pip install -e .
+  ```
+
+## Running Algorithms with Multirun and Submitit
+
+1. Download the feature extractor, evm model and known features using
+  ```
+  wget https://vast.uccs.edu/~mjafarzadeh/trained_f1_umd_464.pth
+  wget https://vast.uccs.edu/~mjafarzadeh/evm_cosine_F1_umd_464_tail_40000_ct_0.8_dm_0.65.pkl
+  wget https://vast.uccs.edu/~mjafarzadeh/feature_F1_umd_464_known_train.npy
+  ```
+
+2. Modify problem configs
+  Set the following variables in sailon_tinker_launcher/configs/problem/ic_condda18.yaml
+  and sailon_tinker_launcher/configs/problem/ic_condda18_wo_redlight.yaml
+    1. workdir: path to directory where all artifacts for the run are stored
+    2. harness: Harness used for experiment (choices: local/par)
+    3. test_ids: List of tests
+    4. dataset_root: Root directory for images
+    5. model_path (cnn_params): Path to the feature extractor model
+    6. model_path (evm_params): Path to the evm model
+    7. feature_known_path (evm_params): Path to the known features downloaded in previous step
+
+3. Without hydra
+  ```
+  tinker -c configs/problem/ic_conda18.yaml sailon_tinker_launcher/tinker_launcher.py
+  tinker -c configs/problem/ic_conda18_wo_redlight.yaml sailon_tinker_launcher/tinker_launcher.py
+  ```
+
+3. Setup cluster config
+  Create a configuration for the cluster that the job would execute on in
+  sailon_tinker_launcher/configs/hydra/launcher. Refer to
+  sailon_tinker_launcher/configs/hydra/launcher/veydrus.yaml for example.
+
+4. Launch the configs using multirun
+  ```
+    python hydra_launcher.py --multirun problem=ic_conda18_wo_redlight,ic_conda18
+  ```
+
+5. Launch the configs using SLURM
+  ```
+  python hydra_launcher.py --multirun problem=ic_conda18_wo_redlight,ic_condda18 hydra/launcher=submitit_slurm
+  ```