allowed umpire to make new actions made in the app editor available i…

…mmediately

.dockerignore

@@ -2,7 +2,11 @@
 .idea
 api_gateway/client/node_modules
 docs
-data
+data/registry/
+data/postgres/
+data/redis/
+data/portainer/
+data/minio/
 testing
 __pycache__
 **/__pycache__

.gitignore

@@ -37,7 +37,8 @@ data/registry/
 data/postgres/
 data/redis/
 data/portainer/
+data/minio/
 app_sdk/walkoff_app_sdk.egg-info
 venv/
 *.unison.*
-tmp-compose.yml
+tmp-compose.yml

CHANGELOG.md

@@ -2,6 +2,40 @@
 <!-- Use the tags Added, Changed, Deprecated, Removed, Fixed, Security, and
      Contributor to describe changes -->
 
+## [1.0.0-beta.1]
+
+This update introduces a number of new features, including an App Editor in the UI, more granular role-based permissions, 
+and a bootloader for automating deployment of WALKOFF. 
+
+### Added
+* App Editor for editing app files and building Docker images from said apps.
+
+### Changed
+* App version no longer required in app_name in api.tyaml
+* CRUD endpoints now accept resource names as keys when applicable
+* Globals can now be arbitrary JSON (UI support pending)
+* Builtins build location moved to Umpire and is only built once on startup
+
+### Removed
+
+### Security
+* Implemented AES-256 encryption/decryption for Global Variables. Exclusive-access decryption based on account level standing still needs to be implemented in the future. Currently, any GET request to the API gateway will return a decrypted Global Variable, regardless of account.  
+
+### Fixed
+* Workflow import/export
+* Workflow validation (still needs work); workflows can be saved in an incomplete state again
+* Validate workfow name uniqueness when creating workflows
+* Testing suite (still needs expansion)
+* Uniqueness constraints on CRUD operations
+* Dereferencing Global and Workflow variables in workflows
+* Ability to override starting parameters in a workflow execution
+* Ability to update/delete encrypted Global Variables
+* Hide global values by default on Globals tab
+* Default boolean parameters to false
+* Copying and pasting of nodes in workflow editor
+* Accessing action results before conditionals in parameters that follow it
+
+
 ## [1.0.0-alpha.2]
 
 This update includes numerous bugfixes and a number of reintroduced features. 
@@ -68,7 +102,6 @@ server in preparation for a future move to an asynchronous framework. The follow
 * Triggers have been temporarily removed, but are targeted for a near-future 1.1 release.
 
 
-
 ## [0.9.4]
 ###### 2018-12-11
 

ROADMAP.md

@@ -1,11 +1,10 @@
 WALKOFF 1.0.0 is intended to present a more robust and scalable implementation of Apps and Workers.
 
-This alpha is **not feature complete and is not production-ready**, but is intended for interested parties to test the 
-new architecture and provide feedback.
+The aim of this beta is to finish building out features intended to make running WALKOFF in production smoother.  
 
 ### 1.0 Intentions
 
-- Designed to run inside Docker Compose or Docker Swarm.
+- Designed to run with Docker Swarm.
     - This eliminates all installation dependencies other than Docker and Docker Compose 
     - Each component can also be run outside of a Docker container for development or debug, as long as it has 
     connectivity to the other pieces. However this is not the supported method of installation.
@@ -21,32 +20,37 @@ new architecture and provide feedback.
 
 ### Testing & Feedback Areas of Interest
 
-For the alpha release, here are some of the areas of interest we would like feedback on:
+For the beta release, here are some of the areas of interest we would like feedback on:
 
-- First-time app development.
-- Stress-testing of large/branching workflows.
-- Anything you can manage to break.
+- App development through the App Editor page.
+- Stress-testing of large/branching workflows and large numbers of concurrent workflows.
+- Testing of scaling WALKOFF across multiple Docker Swarm nodes.
 
 ### Roadmap:
 
-The following section gives a rough approximation of what we are looking to accomplish in upcoming releases.
+The following section gives a rough approximation of what we are looking to accomplish in the leadup to a full 1.0.0 release.
 
 #### 1.0.0-alpha.1
 
 - Core components functioning under new architecture.
-- Stable enough to begin writing usable apps and workflows
+- Stable enough to begin writing usable apps and workflows.
+- "Essentials" app suite, e.g. SSH, PowerShell, other common utilities.
+- All workflow node types (Actions, Parallel Actions, Conditions, Transforms, Triggers).
 
 #### 1.0.0-beta.1
 
-- Functional feature parity with pre-1.0 WALKOFF intentions
-    - Event-driven actions (removed in 0.5.0)
-    - Triggers (removed in 1.0.0-alpha.1)
-- Support for more complex workflow composition
-    - Nested workflows (running a workflow as a node in another workflow)
-    - Parallelized actions (dividing up work for a node into n subnodes)
+- App Editor for modifying apps on the UI.
+- Bootloader for automating deployment of WALKOFF.
+- Further stability improvements.
 
 #### 1.0.0-rc.1:
 
-- Completed unit testing of components and end-to-end testing of a running cluster.
 - Security hardening.
-- "Essentials" app suite, e.g. SSH, PowerShell, other common utilities.
+- Complete unit testing of components and end-to-end testing of a running cluster.
+- Validation of WALKOFF running and scaling to large production workloads on multiple Docker Swarm nodes.
+- Dashboards for monitoring and working with WALKOFF outside of Docker CLI
+- Resource-aware scaling of apps and workflows.
+
+#### 1.1.0:
+
+- Future plans include features such as building App SDKs for other languages, building out more ways of executing workflows (e.g. pollers and listeners)

api_gateway/api/objects/global_variables.yaml

@@ -12,9 +12,6 @@ GlobalVariable:
     name:
       type: string
       description: The name of the global variable
-    _walkoff_type:
-      type: string
-      description: Workflow type for json decoder
     permissions:
       type: array
       items:
@@ -47,8 +44,6 @@ GlobalVariableTemplate:
     name:
       type: string
       description: The name of the global template
-    _walkoff_type:
-      type: string
     schema:
       type: object
       description: The schema for the global template

api_gateway/api/objects/workflows.yaml

@@ -63,6 +63,7 @@ WorkflowJSON:
       description: Workflow type for json decoder
     permissions:
       type: array
+      nullable: true
       items:
         type: object
 

api_gateway/api/umpire.yaml

@@ -31,7 +31,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/file/{app_name}/{app_version}:
   get:
@@ -71,7 +70,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/file_upload:
   post:
@@ -99,7 +97,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/build:
   get:
@@ -121,7 +118,7 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
+
 /umpire/build/{app_name}/{app_version}:
   post:
     tags:
@@ -155,7 +152,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/build/{build_id}:
   post:
@@ -184,7 +180,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/save/{app_name}/{app_version}:
   post:
@@ -219,5 +214,4 @@
             application/json:
               schema:
                 $ref: '#/components/schemas/Error'
-      security: []
 

api_gateway/client/dist/walkoff/index.html

@@ -14,5 +14,5 @@
 <link rel="stylesheet" href="client/dist/walkoff/styles.a7145bf960e458a8099b.css"></head>
 <body>
     <main-component>Loading...</main-component>
-<script src="client/dist/walkoff/runtime.5978eebdd8411421365f.js"></script><script src="client/dist/walkoff/polyfills-es5.38ed56f75e7d87c23835.js" nomodule></script><script src="client/dist/walkoff/polyfills.425a354b82ebc224c491.js"></script><script src="client/dist/walkoff/scripts.dad7fdaf9a0a03a091ee.js"></script><script src="client/dist/walkoff/main.9deae89733d616319ddf.js"></script></body>
-</html>
+<script src="client/dist/walkoff/runtime.5978eebdd8411421365f.js"></script><script src="client/dist/walkoff/polyfills-es5.38ed56f75e7d87c23835.js" nomodule></script><script src="client/dist/walkoff/polyfills.425a354b82ebc224c491.js"></script><script src="client/dist/walkoff/scripts.dad7fdaf9a0a03a091ee.js"></script><script src="client/dist/walkoff/main.ed6e481dffde331533b4.js"></script></body>
+</html>

api_gateway/executiondb/__init__.py

@@ -54,7 +54,8 @@ def __init__(self):
             self.engine = create_engine(
                 format_db_path(config.DB_TYPE, config.EXECUTION_DB_NAME, config.DB_USERNAME, config.DB_PASSWORD,
                                config.DB_HOST),
-                poolclass=NullPool)
+                poolclass=NullPool, isolation_level="AUTOCOMMIT")
+
             if not database_exists(self.engine.url):
                 try:
                     create_database(self.engine.url)

api_gateway/server/blueprints/root.py

@@ -28,7 +28,7 @@ def client_app_folder(filename):
 
 # Default route to angular application
 @root_page.route('/', defaults={'path': ''})
-@root_page.route('/walkoff', defaults={'path': ''})
+@root_page.route('/walkoff/', defaults={'path': ''})
 @root_page.route('/walkoff/<path:path>')
 def default(path):
     return send_from_directory(os.path.abspath(static.CLIENT_PATH), "dist/walkoff/index.html")

api_gateway/server/endpoints/umpire.py

@@ -32,7 +32,7 @@ def list_all_files(app_name, app_version):
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required
@@ -48,7 +48,7 @@ def get_file_contents(app_name, app_version, file_path):
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required
@@ -68,7 +68,7 @@ def update_file():
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required
@@ -80,7 +80,7 @@ def get_build_status():
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required
@@ -93,7 +93,7 @@ def build_image(app_name, app_version):
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required
@@ -105,7 +105,7 @@ def build_status_from_id(build_id):
     if r.status_code == 200:
         return r.json(), HTTPStatus.OK
     else:
-        return "Failed", HTTPStatus.BAD_REQUEST
+        return r.text, HTTPStatus.BAD_REQUEST
 
 
 # @jwt_required

bootloader/base-compose.yml

@@ -85,7 +85,4 @@ secrets:
 
 networks:
   walkoff_default:
-#    driver: overlay
-#    name: walkoff_default
-#    attachable: true
     external: true

bootloader/bootloader.py

@@ -376,11 +376,8 @@ async def down(self):
                             help="Removes the walkoff_encryption_key secret.")
         parser.add_argument("-r", "--registry", action="store_true",
                             help="Clears the registry bind mount directory.")
-        parser.add_argument("-s", "--skipnetwork", action="store_true",
-                            help="Skip network removal check. Use this if you have attached external services to it.")
         parser.add_argument("-d", "--debug", action="store_true",
                             help="Set log level to debug.")
-        parser.add_argument("-n")
 
         # Parse out the command
         args = parser.parse_args(sys.argv[2:])

bootloader/walkoff-compose.yml

@@ -103,7 +103,4 @@ secrets:
 
 networks:
   walkoff_default:
-#    driver: overlay
-#    name: walkoff_default
-#    attachable: true
     external: true

common/helpers.py

@@ -34,11 +34,11 @@ def sfloat(value, default):
 @retry(stop=stop_after_attempt(5), wait=wait_exponential(min=1, max=10))
 async def get_walkoff_auth_header(session, token=None, timeout=5*60):
     url = config.API_GATEWAY_URI.rstrip('/') + '/walkoff/api'
-
+    logger.debug("Attempting to refresh WALKOFF JWT")
     if token is None:
         with open(config.INTERNAL_KEY_PATH, 'rb') as f:
             key = f.read()
-            key = str(key)
+            key = str(key.strip())
         async with session.post(url + "/auth", json={"username": config.WALKOFF_USERNAME,
                                                      "password": key}, timeout=timeout) as resp:
             resp_json = await resp.json()

common/minio_helper.py

@@ -61,9 +61,9 @@ async def build_image(app_name, version):
                 logger.info("Sending image to be built")
                 dockerfile = "./Dockerfile"
                 log_stream = await docker_client.images.build(fileobj=context, tag=repo, rm=True,
-                                                          forcerm=True, pull=True, stream=True,
-                                                          path_dockerfile=dockerfile,
-                                                          encoding="application/x-tar")
+                                                              forcerm=True, pull=True, stream=True,
+                                                              path_dockerfile=dockerfile,
+                                                              encoding="application/x-tar")
                 logger.info("Docker image building")
                 await stream_docker_log(log_stream)
                 logger.info("Docker image Built")
@@ -97,14 +97,17 @@ async def update_file(app_name, version, path, file_data, file_size):
             minio_client.stat_object("apps-bucket", abs_path)
             found = True
         except Exception as e:
-            pass
+            logger.info("File does not exist, creating a new one.")
 
         if found is True:
             minio_client.remove_object("apps-bucket", abs_path)
+            logger.info("File exists, removing it before creating a new one.")
+
         file_data = io.BytesIO(file_data)
         try:
             minio_client.put_object("apps-bucket", abs_path, file_data, file_size)
-            return True, "Successfully placed file in Minio"
+            r = minio_client.stat_object("apps-bucket", abs_path)
+            return True, str(r)
         except Exception as e:
             return False, str(e)
 
@@ -129,5 +132,3 @@ async def save_file(app_name, version):
                 group_id = stat(f"apps/{app_name}/{version}/requirements.txt").st_uid
                 os.chown(p_dst, owner_id, group_id)
         return True
-
-