Merge pull request #720 from MetaCell/feature/CH-37

Allow to clone an external repo before the build

application-templates/base/.dockerignore

@@ -0,0 +1,73 @@
+.travis.yaml
+.openapi-generator-ignore
+README.md
+tox.ini
+git_push.sh
+test-requirements.txt
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+venv/
+.python-version
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints
+
+.git

application-templates/django-app/.dockerignore

@@ -1,2 +1,3 @@
 node_modules
 dist
+.git

application-templates/flask-server/backend/.dockerignore

@@ -69,3 +69,5 @@ target/
 
 #Ipython Notebook
 .ipynb_checkpoints
+
+.git

application-templates/webapp/.dockerignore

@@ -1,2 +1,73 @@
-node_modules
-dist
+.travis.yaml
+.openapi-generator-ignore
+README.md
+tox.ini
+git_push.sh
+test-requirements.txt
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+venv/
+.python-version
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints
+
+.git

docs/applications/README.md

@@ -102,6 +102,7 @@ The most important configuration entries are the following:
     - `hard`: hard dependencies mean that they are required for this application to work properly
     - `soft`: the application will function for most of its functionality without this dependency
     - `build`: the images declared as build dependencies can be referred as dependency in the Dockerfile 
+    - `git`: specify repos to be cloned before the container build
   - `database`: automatically generates a preconfigured database deployment for this application
     - `auto`: if true, turns on the database deployment functionality
     - `type`: one from `postgres` (default), `mongo`, `neo4j`

docs/applications/dependencies.md

@@ -0,0 +1,91 @@
+# Application dependencies
+
+Application dependencies can be specified in the main application configuration
+file (deploy/values.yaml), in the `harness` section.
+
+Example:
+```yaml
+harness:
+  dependencies:
+    build:
+    - cloudharness-base
+    soft:
+    - app1
+    hard:
+    - accounts
+    git:
+    - url: https://github.com/a/b.git
+      branch_tag: master
+```
+
+## Build dependencies
+
+Build dependencies specify which images must be built before the current one.
+Currently only base images and common images can be used as a build dependency.
+
+See also [base and common images documentation](../base-common-images.md).
+
+## Soft dependencies
+
+Soft dependencies specify other applications (from your app or cloudharness) that
+must be included in the deployment together with your application,
+but are not a prerequisite for the application to bootstrap and serve basic functionality.
+
+Soft dependencies are implicitly chained: if *A1* depends on *A2* and *A2* depends on *A3*,
+all *A1*, *A2*, *A3* are included in the deployment if A1 is requested (say with
+`harness-deployment ... -i A1`).
+
+## Hard dependencies
+
+Hard dependencies work similarly to soft dependencies but they are required for the 
+application declaring the dependency to start and provide even basic functionality.
+
+With a hard dependency, we are allowed to assume that the other application exists in the
+configuration and during the runtime.
+
+Note that Cloud Harness does not guarantee the the other application starts before the
+application declaring the dependency because that's how Kubernetes works. The application
+is supposed to crash in the absence of its dependency and Kubernetes will start the crash
+loop until both applications are settled.
+
+## Git (repository) dependencies
+
+Git dependencies allow us to build and deploy applications that are defined in another repository.
+This functionality is an alternative to the otherwise monorepo-centric view of CloudHarness-based
+applications.
+
+The repository is cloned before the build within skaffold build and the CI/CD inside the 
+`dependencies` directory at the same level of the Dockerfile.
+
+Hence, in the Dockerfile we are allowed to `COPY` or `ADD` the repository.
+
+For instance, given the following configuration:
+```yaml
+harness:
+  dependencies:
+    git:
+    - url: https://github.com/a/b.git
+      branch_tag: master
+    - url: https://github.com/c/d.git
+      branch_tag: v1.0.0
+      path: myrepo
+```
+
+The directory structure will be as following:
+```
+Dockerfile
+dependencies
+  b.git
+  myrepo
+  .dockerignore
+```
+
+Hence, inside the Dockerfile we expect to see something like
+
+```dockerfile
+COPY dependencies .
+COPY dependencies/b.git/src .
+COPY dependencies/myrepo .
+```
+
+> Note that Cloud Harness does not add the COPY/ADD statements to the Dockerfile

docs/applications/development/backend-development.md

@@ -55,7 +55,9 @@ harness:
 Every image defined as a base image or a common image can be used as a
 build dependency.
 
-For more details about how to define your custom image and the available images, see [here](../../base-common-images.md)
+For more details about how to define your custom image and the available images, see [here](../../base-common-images.md).
+
+For more info about dependencies, see [here](../dependencies.md)
 
 ## Use the CloudHarness runtime Python library
 

docs/base-common-images.md

@@ -15,9 +15,9 @@ a specific purpose (e.g. enable widely used application stacks to inherit from).
 
 After generating the codeChange the Dockerfile in order to inherit from the main Docker image need to:
 
-1. Add the image as a build dependency to the values.yaml file of your application. The name of the image corresponds to the directory name where the Dockerfile is located 
+1. Add the image as a [build dependency](applications/dependencies.md) to the values.yaml file of your application. The name of the image corresponds to the directory name where the Dockerfile is located 
 
-```
+```yaml
 harness:
   dependencies:
     build:

docs/model/ApplicationDependenciesConfig.md

@@ -11,6 +11,7 @@ Key | Input Type | Accessed Type | Description | Notes
 **[hard](#hard)** | list, tuple,  | tuple,  | Hard dependencies indicate that the application may not start without these other applications. | [optional] 
 **[soft](#soft)** | list, tuple,  | tuple,  | Soft dependencies indicate that the application will work partially without these other applications. | [optional] 
 **[build](#build)** | list, tuple,  | tuple,  | Hard dependencies indicate that the application Docker image build requires these base/common images | [optional] 
+**[git](#git)** | list, tuple,  | tuple,  |  | [optional] 
 **any_string_name** | dict, frozendict.frozendict, str, date, datetime, int, float, bool, decimal.Decimal, None, list, tuple, bytes, io.FileIO, io.BufferedReader | frozendict.frozendict, str, BoolClass, decimal.Decimal, NoneClass, tuple, bytes, FileIO | any string name can be used but the value must be the correct type | [optional]
 
 # hard
@@ -55,5 +56,17 @@ Class Name | Input Type | Accessed Type | Description | Notes
 ------------- | ------------- | ------------- | ------------- | -------------
 items | str,  | str,  |  | 
 
+# git
+
+## Model Type Info
+Input Type | Accessed Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+list, tuple,  | tuple,  |  | 
+
+### Tuple Items
+Class Name | Input Type | Accessed Type | Description | Notes
+------------- | ------------- | ------------- | ------------- | -------------
+[**GitDependencyConfig**](GitDependencyConfig.md) | [**GitDependencyConfig**](GitDependencyConfig.md) | [**GitDependencyConfig**](GitDependencyConfig.md) |  | 
+
 [[Back to Model list]](../../README.md#documentation-for-models) [[Back to API list]](../../README.md#documentation-for-api-endpoints) [[Back to README]](../../README.md)
 

docs/model/GitDependencyConfig.md

@@ -0,0 +1,19 @@
+# cloudharness_model.model.git_dependency_config.GitDependencyConfig
+
+Defines a git repo to be cloned inside the application path
+
+## Model Type Info
+Input Type | Accessed Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+dict, frozendict.frozendict,  | frozendict.frozendict,  | Defines a git repo to be cloned inside the application path | 
+
+### Dictionary Keys
+Key | Input Type | Accessed Type | Description | Notes
+------------ | ------------- | ------------- | ------------- | -------------
+**branch_tag** | str,  | str,  |  | 
+**url** | str,  | str,  |  | 
+**path** | str,  | str,  | Defines the path where the repo is cloned. default: /git | [optional] 
+**any_string_name** | dict, frozendict.frozendict, str, date, datetime, int, float, bool, decimal.Decimal, None, list, tuple, bytes, io.FileIO, io.BufferedReader | frozendict.frozendict, str, BoolClass, decimal.Decimal, NoneClass, tuple, bytes, FileIO | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../../README.md#documentation-for-models) [[Back to API list]](../../README.md#documentation-for-api-endpoints) [[Back to README]](../../README.md)
+

libraries/cloudharness-utils/cloudharness_utils/constants.py

@@ -43,7 +43,7 @@
 CD_E2E_TEST_STEP = 'tests_e2e'
 CD_STEP_PUBLISH = 'publish'
 BUILD_FILENAMES = ('node_modules',)
-
+CD_BUILD_STEP_DEPENDENCIES = 'post_main_clone'
 
 E2E_TESTS_DIRNAME = 'e2e'
 API_TESTS_DIRNAME = 'api'

libraries/models/api/openapi.yaml

@@ -68,6 +68,11 @@ components:
                     type: array
                     items:
                         type: string
+                git:
+                    description: ''
+                    type: array
+                    items:
+                        $ref: '#/components/schemas/GitDependencyConfig'
         DeploymentResourcesConf:
             description: ''
             type: object
@@ -849,3 +854,22 @@ components:
             example:
                 quota-ws-max: 5
                 quota-storage-max: 1G
+        GitDependencyConfig:
+            title: Root Type for GitDependencyConfig
+            description: Defines a git repo to be cloned inside the application path
+            required:
+                - branch_tag
+                - url
+            type: object
+            properties:
+                url:
+                    type: string
+                branch_tag:
+                    type: string
+                path:
+                    description: 'Defines the path where the repo is cloned. default: /git'
+                    type: string
+            example:
+                url: 'https://github.com/MetaCell/nwb-explorer.git'
+                branch_tag: master
+                path: /git

libraries/models/cloudharness_model/models/__init__.py

@@ -25,6 +25,7 @@
 from cloudharness_model.models.deployment_volume_spec_all_of import DeploymentVolumeSpecAllOf
 from cloudharness_model.models.e2_e_tests_config import E2ETestsConfig
 from cloudharness_model.models.file_resources_config import FileResourcesConfig
+from cloudharness_model.models.git_dependency_config import GitDependencyConfig
 from cloudharness_model.models.harness_main_config import HarnessMainConfig
 from cloudharness_model.models.ingress_config import IngressConfig
 from cloudharness_model.models.ingress_config_all_of import IngressConfigAllOf