start e2e before all clusters are ready (#127)

Currently watcher will wait for all clusters to move to ready state
and once it happens then notifies all threads to run e2e testing.
If the user wants the created threads to start e2e before all
clusters are moved to ready state, he can create a file named "e2e"
dynamically in test directory. Watcher will check for this file and
notify all threads to start running e2e tests.

With this new change, e2e testing will start in either
1. all clusters moved to ready state
2. or user created a file name "e2e" in test directory i.e
   /tmp/<uuid>/e2e

rosa-hypershift/rosa-hosted-wrapper.py

@@ -880,6 +880,14 @@ def _watcher(rosa_cmnd, my_path, cluster_name_seed, cluster_count, delay, my_uui
     time.sleep(60)
     logging.info('Watcher thread started')
     logging.info('Getting status every %d seconds' % int(delay))
+    # watcher will stop iterating and notify to run e2e if one of the below conditions met
+    # 1) look if all the clusters move to ready state or
+    # 2) if the user created e2e file in the test directory
+    file_path = os.path.join(my_path, "e2e")
+    if os.path.exists(file_path):
+        os.remove(file_path)
+        time.sleep(60)
+
     cmd = [rosa_cmnd, "list", "clusters", "-o", "json"]
     while not force_terminate:
         logging.debug(cmd)
@@ -934,6 +942,11 @@ def _watcher(rosa_cmnd, my_path, cluster_name_seed, cluster_count, delay, my_uui
             else:
                 logging.info("Waiting %d seconds for next watcher run" % delay)
                 time.sleep(delay)
+        elif os.path.isfile(file_path):
+            with all_clusters_installed:
+                logging.info("User requested the wrapper to start e2e testing by creating e2e file in the test directory")
+                all_clusters_installed.notify_all()
+            break
         else:
             logging.info("Waiting %d seconds for next watcher run" % delay)
             time.sleep(delay)