Release 5.2.0 to master (#3794)

* removed unused variables * feat: SB-30654 generalizing plugins upload Signed-off-by: Keshav Prasad <[email protected]> * feat: upload plugins using existing roles Signed-off-by: Keshav Prasad <[email protected]> * grouping aws specific tasks into a single role (#3573) * fix: remove polling to save time * fix: updated comments Signed-off-by: Keshav Prasad <[email protected]> * fix: added a var to control container creation Signed-off-by: Keshav Prasad <[email protected]> * env changes for project certificate story (#3599) * fix: remove unnecessary become Signed-off-by: Keshav Prasad <[email protected]> * fix: poll incorrectly set to 0 Signed-off-by: Keshav Prasad <[email protected]> * fix: renamed task (#3601) * Certificate env changes- RC- internal call used instead env variable (#3603) * Update config.j2 * Update main.yml * Update config.j2 * Update main.yml * LR-262 cloud base path change * Issue #KN-427 feat: Knowledge service cloud-agnostic * Revert "Issue #KN-9 feat: Content Publish API refactor." This reverts commit e37ca72. * LR-262 cname template variable * LR-262 variable name updated * LR-262 variable name change * Update config.j2 * Update main.yml * Update main.yml * Update main.yml * Update main.yml * Update main.yml * Update main.yml * Update main.yml * Update main.yml * Issue #ED-521 fix: Updated CSP variables with Finalized vars * Issue #ED-521 fix: Updated CSP variables with Finalized vars * Issue #ED-521 fix: Updated CSP variables with Finalized vars * variables changes for sunbird cloud agnostic tool * variables changes - private vars * Make roles as anonymous for project certificate * Fix syntax (#3621) * Change condition to match boolean (#3623) * Update main.yml * updated CACHE_CONTEXT_URLS var (#3625) * csp migration variables update * added config.json file in certificatesign and updated templates * updated certificatesign helm_chart * fix: adding verbosity for desktop deploy (#3633) * Set authentication_enabled as false by default Add authentication_enabled as a new env key for the registry service and set it false by default. * Release 5.1.0 (#3637) * Update deploy plugins for GCP * Fix looping issue * LR-262 CSP related variable - Cloud storage base path added (#3641) * Add environment keys to enable async certificate Add environment keys to enable async certificate creation * LR-126:cloud agnostic variable changes * Update config.j2 (#3644) * Add kafka_bootstrap_address env key Add kafka_bootstrap_address env key * Issue #ED-536 feat: Added fetch all API for form * Issue #ED-536 fix: fetch all api added * LR-126 cloud variable name changes * Update main.yml (#3649) * Release 5.1.0 vars change (#3650) * updated knowlgbb service changes * updated knowlgbb service changes * Issue #KN-439 chore: Mering the changes from release-5.2.0-knowlg to release-5.2.0 * Issue #ED-354 chore: Update aab firebase upload logic. * Release 5.1.0 vars change (#3658) * Issue #IQ-193 feat: updated config of assessment * Add a new variable (#3661) * Update main.yml * Update shell_script_config.j2 * Update Ingestion Spec * Remove SAS token * Missing vars update (#3666) * LR-110 removed un-necessary prefix * Release 5.1.0 - csp changes (#3667) * Add logging level configuration for registry service (#3669) Add logging level configuration for registry service * LR-110 added default context * Update config.j2 (#3673) * fix: ED-573 jenkins job to kill spark jobs Signed-off-by: Keshav Prasad <[email protected]> * Update ingestion specs (#3677) * updated jenkins version and plugins (#3679) * Issue #ED-592 fix: Pdata version updated (#3678) * csp migration variables update * csp migration variables update * Updated template value for dial ansible group (#3681) * csp migration variables update * csp migration variables update * csp migration variables update (#3671) * csp migration variables update * Ansible-postgres-patroni cluster (#3684) * Ansible-postgres-patroni cluster * README.md updated * Prashnavali reminder query added * added new jenkins adhoc job for ml-analytics #ED-569 (#3675) * fix: https:// already part of another url Signed-off-by: Keshav Prasad <[email protected]> * fix: updated split logic based on url value Signed-off-by: Keshav Prasad <[email protected]> * fix: removed https:// url string Signed-off-by: Keshav Prasad <[email protected]> * fix: update url array index Signed-off-by: Keshav Prasad <[email protected]> * fix: remove hardcoded https from proxy_pass value (#3688) * fix: remove https as its included in the var Signed-off-by: Keshav Prasad <[email protected]> * Project certificate download uris edited (#3691) typo correction * Release 5.1.0 - CSP changes (#3692) * Update management bucketname for ES * Update ES snapshot roles * ED-621: added placeholder variable * removed variable * ED-621: added placeholder variable * Issue #KN-439 chore: Added the transcripts in cloudstorage_metadata_list. * Add gcp vars for service accounts (#3696) * Fix the command to get deployed image (#3698) * Fix gcp deploy issues (#3699) * Fix typo * Fix the command to get deployed image * Added new variables (#3701) * ED-621:updated configurations for release-5.1.0 * Updated private_repo template with CSP changes (#3689) * fix: adding graylog related changes * Release 5.1.0 (#3697) * Add Lern inventory template * Install openjdk-11.0.2 in jenkins * Add Lern BB vars to template * Remove duplicate flink jobs * Update template vars * Update lern specific vars * feat: ED-699 adding data to enable opa Signed-off-by: Keshav Prasad <[email protected]> * fear: adding required consumers for sessions Signed-off-by: Keshav Prasad <[email protected]> * fix: adding additional set of vars for sessions and graylog Signed-off-by: Keshav Prasad <[email protected]> * LR-278 added lern specific variables * LR-278 added placeholder as configurable value * #000 | LERN and ED Dataproducts jenkins changes added * #000 | LERN and ED Dataproducts jenkins changes added * fix: adding adminutil_learner_api_key consumer * fix: adding adminutls to learner api token Signed-off-by: Keshav Prasad <[email protected]> * Update youtube api key for content service (#3714) * fix: ED-1164 adding missing UCI job Signed-off-by: Keshav Prasad <[email protected]> * adding oci-cli in bootstrap and Jenkins job template (#3715) * ED-1173 OfflineInstaller deploy job fix (#3721) * updated electronuserland/builder tag form wine to 16-wine * added npm update command for electronuserland/builder:16-wine * Issue #LR-338 chore: Onboarding the knowlg APIs to lern kong-api * Issue #LR-338 chore: Fix the formatting. * Issue #LR-338 chore: Added the dial_service_url config * Update config.j2 * ED-700: Update Monitoring Stack Helm Chart (#3723) * Update CRDs Api version and definitions Remove depricated webhook crd-install * Let helm takecare of creating CRDs * Update admission webhook and rbac API version * Update admissionwebhook patch repo * Update apiVersions * Issue #LR-339 chore: Updated the latest config for content-service, dial-service and taxonomy-service from knowlg 5.2.0 * added UCI vars in private_repo template * added UCI vars in private_repo template * Issue #LR-339 chore: Fixed formatting * Correcting the Spelling Mistake (#3730) * Updated the spark_venv permissions * Changing the steps (#3732) * Issue #KN-828 chore: Added gcp related config * Issue #KN-828 chore: Added gcp related config * added jenkins-mobile-slave-setup.sh (#3734) * Issue #LR-307 chore: Onboarded the Device APIs from Obsrv * Issue #LR-307 chore: Onboarded the Device APIs from Obsrv * Issue #LR-307 chore: Onboarded the telemetry API from Obsrv * Issue #LR-307 chore: Fixed the content create API issue * Issue #LR-307 chore: Onboarded getDataExhaustRequest and listDataExhaustRequest Obsrv APIs * Issue #LR-307 chore: Updated the lock APIs * Issue #LR-325 feat: Refactoring of SB Lern Batch Service - DialCode Dependency * Fix: Release-5.1.0 installation issues (#3747) * Fix azcopy cmd execute issue * Update Learner service storage vars * Add new jenkins vars for KP and DP repo * Add adminutil_learner_api_key consumer api permissions * Sunbird installation issue fixes (#3746) * removed questionset-publish from KP flink job list * updated keycloak provision role * removed enc-service upstream * changed the order of mail server vars * updated the order of graylog and log-es hosts * UCI-2 - env variables for doubtnut integration (#3754) * env variables for doubtnut integration * updated env variables --------- Co-authored-by: Surabhi Mahawar <[email protected]> * Fix typo (#3755) * Fix typo * Fix: remove hardcoded https * Addition of ML Program UserInfoExhaustJob * Issue #ED-1429 fix: Updated pData version to 5.2.0 * Addition of ml-jobs * Update config.xml * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Updated the ansible script for Lern cassandra migration * Issue #LR-101 feat: Removed unused keyspace sunbirdplugin * LR-307 Merging 5.2.0 to 5.2.0-lern branch * LR-307 Merging 5.2.0 to 5.2.0-lern branch * Issue #LR-101 feat: Added the delete folder command. * Issue #LR-307 feat: Onboarded inQuiry APIs * UCI-2 - env variables for doubtnut integration (#3768) * UCI-2 - remove default values from env file for doubtnut integration * UCI-2 - change default values in env file for doubtnut integration * UCI-2 - env variables for doubtnut integration (#3769) * UCI-2 - remove default values from env file for doubtnut integration * UCI-2 - change default values in env file for doubtnut integration * UCI-2 - added nlapp variables in env file * UCI-2 - added nlapp variables in env file --------- Co-authored-by: Santhosh Gandham <[email protected]> * ED-539 Added proxy end point for android deeplink support * ED-539 Added proxy end point for android deeplink support * LR-101-Merging from lern_5.2.0 to devops_5.2.0 * Update sunbird_learner-service.env --------- Signed-off-by: Keshav Prasad <[email protected]> Co-authored-by: Surabhi <[email protected]> Co-authored-by: Keshav Prasad <[email protected]> Co-authored-by: G33tha <[email protected]> Co-authored-by: VISHNUDAS <[email protected]> Co-authored-by: Raghupathi <[email protected]> Co-authored-by: PrasadMoka <[email protected]> Co-authored-by: Jayaprakash8887 <[email protected]> Co-authored-by: Rajesh Kumaravel <[email protected]> Co-authored-by: Akash Shah <[email protected]> Co-authored-by: Akhil <[email protected]> Co-authored-by: SadanandGowda <[email protected]> Co-authored-by: saiakhil46 <[email protected]> Co-authored-by: Reshmi V Nair <[email protected]> Co-authored-by: G33tha <[email protected]> Co-authored-by: anilgupta <[email protected]> Co-authored-by: swayangjit <[email protected]> Co-authored-by: Kumar Gauraw <[email protected]> Co-authored-by: Ashwiniev95 <[email protected]> Co-authored-by: Santhosh Gandham <[email protected]> Co-authored-by: Sadanand <[email protected]> Co-authored-by: NIKHIL VARMA M <[email protected]> Co-authored-by: Rekha <[email protected]> Co-authored-by: kumarks1122 <[email protected]> Co-authored-by: Kenneth Heung <[email protected]> Co-authored-by: Surabhi Mahawar <[email protected]> Co-authored-by: Shakthieshwari <[email protected]> Co-authored-by: Reshmi Nair <[email protected]> Co-authored-by: Pankaj Jangid <[email protected]> Co-authored-by: Santhosh Gandham <[email protected]> Co-authored-by: BharathwajShankar <[email protected]>
project-sunbird · Apr 26, 2023 · b5797a4 · b5797a4
1 parent a5626ed
commit b5797a4
Show file tree

Hide file tree

Showing 314 changed files with 22,125 additions and 16,676 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,4 @@
 .DS_Store
 *.retry
 *.pyc
-.idea
+.idea
diff --git a/ansible/artifacts-download.yml b/ansible/artifacts-download.yml
@@ -3,8 +3,40 @@
   become: yes
   vars_files:
     - "{{inventory_dir}}/secrets.yml"
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{ sunbird_artifact_storage_account_name }}"
-    AZURE_STORAGE_SAS_TOKEN: "{{ sunbird_artifact_storage_account_sas }}"
-  roles:
-    - artifacts-download-azure
+  tasks:
+    - name: download artifact from azure storage
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: blob-download.yml
+      vars:
+        blob_container_name: "{{ cloud_storage_artifacts_bucketname }}"
+        blob_file_name: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+        storage_account_name: "{{ cloud_artifact_storage_accountname }}"
+        storage_account_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "azure"
+
+    - name: download artifact from gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: download.yml
+      vars:
+        gcp_storage_service_account_name: "{{ cloud_artifact_storage_accountname }}"
+        gcp_storage_key_file: "{{ cloud_artifact_storage_secret }}"
+        gcp_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        gcp_path: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+      when: cloud_service_provider == "gcloud"
+
+    - name: download artifact from aws s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: download.yml
+      vars:
+        local_file_or_folder_path: "{{ artifact_path }}"
+        s3_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        s3_path: "{{ artifact }}"
+        aws_default_region: "{{ cloud_public_storage_region }}"
+        aws_access_key_id: "{{ cloud_artifact_storage_accountname }}"
+        aws_secret_access_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "aws"  
diff --git a/ansible/artifacts-upload.yml b/ansible/artifacts-upload.yml
@@ -3,8 +3,41 @@
   become: yes
   vars_files:
     - "{{inventory_dir}}/secrets.yml"
-  environment:
-    AZURE_STORAGE_ACCOUNT: "{{ sunbird_artifact_storage_account_name }}"
-    AZURE_STORAGE_SAS_TOKEN: "{{ sunbird_artifact_storage_account_sas }}"
-  roles:
-    - artifacts-upload-azure
+  tasks:
+    - name: upload artifact to azure storage
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: blob-upload.yml
+      vars:
+        blob_container_name: "{{ cloud_storage_artifacts_bucketname }}"
+        container_public_access: "off"
+        blob_file_name: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+        storage_account_name: "{{ cloud_artifact_storage_accountname }}"
+        storage_account_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "azure"
+
+    - name: upload artifact to gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: upload.yml
+      vars:
+        gcp_storage_service_account_name: "{{ cloud_artifact_storage_accountname }}"
+        gcp_storage_key_file: "{{ cloud_artifact_storage_secret }}"
+        gcp_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        gcp_path: "{{ artifact }}"
+        local_file_or_folder_path: "{{ artifact_path }}"
+      when: cloud_service_provider == "gcloud"
+
+    - name: upload artifact to aws s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: upload.yml
+      vars:
+        local_file_or_folder_path: "{{ artifact_path }}"
+        s3_bucket_name: "{{ cloud_storage_artifacts_bucketname }}"
+        s3_path: "{{ artifact }}"
+        aws_default_region: "{{ cloud_public_storage_region }}"
+        aws_access_key_id: "{{ cloud_artifact_storage_accountname }}"
+        aws_secret_access_key: "{{ cloud_artifact_storage_secret }}"
+      when: cloud_service_provider == "aws"
diff --git a/ansible/artifacts/sunbird/login/resources/js/telemetry_service.js b/ansible/artifacts/sunbird/login/resources/js/telemetry_service.js
@@ -2506,7 +2506,7 @@ if(client_id.toLowerCase() === 'android'){
     "telemetry": {
       "pdata": {
         "id": pdataId,
-        "ver": "5.0.0",
+        "ver": "5.2.0",
         "pid": "sunbird-portal"
       }
     }
@@ -2687,7 +2687,6 @@ if(client_id.toLowerCase() === 'android'){
   function stringToHTML(str) {
     let parser = new DOMParser();
     let doc = parser.parseFromString(str, 'text/html');
-    console.log('Doc parse => ', doc); // TODO: log!
     return doc?.body?.innerText || document.createElement('body');
   }
 

diff --git a/ansible/assets-upload.yml b/ansible/assets-upload.yml
@@ -1,8 +1,77 @@
 ---
 - hosts: localhost
-  vars:
-    ansible_connection: local
   vars_files:
     - ['{{inventory_dir}}/secrets.yml', 'secrets/{{env}}.yml']
-  roles:
-    - assets-upload-azure
+  # Azure
+  tasks:
+  - name: this block consists of tasks related to azure storage
+    block:
+    - name: set common azure variables
+      set_fact:
+        blob_container_name: "{{ cloud_storage_playercdn_bucketname }}"
+        container_public_access: "container"
+        blob_container_folder_path: ""
+        storage_account_name: "{{ cloud_public_storage_accountname }}"
+        storage_account_key: "{{ cloud_public_storage_secret }}"
+
+    - name: delete files and folders from azure storage using azcopy
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: delete-using-azcopy.yml
+
+    - name: upload batch of files to azure storage
+      include_role:
+        name: azure-cloud-storage
+        tasks_from: blob-upload-batch.yml
+      vars:
+        local_file_or_folder_path: "{{ assets }}"
+    when: cloud_service_provider == "azure"
+
+##### AWS
+  - name: this block consists of tasks related to aws storage
+    block:
+    - name: set common aws variables
+      set_fact:
+        s3_bucket_name: "{{ cloud_storage_playercdn_bucketname }}"
+        s3_path: ""
+        aws_default_region: "{{ cloud_public_storage_region }}"
+        aws_access_key_id: "{{ cloud_public_storage_accountname }}"
+        aws_secret_access_key: "{{ cloud_public_storage_secret }}"
+
+    - name: delete files and folders from s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: delete-folder.yml
+
+    - name: upload batch of files to s3
+      include_role:
+        name: aws-cloud-storage
+        tasks_from: upload-folder.yml
+      vars:
+        local_file_or_folder_path: "{{ assets }}"
+    when: cloud_service_provider == "aws"  
+
+  #GCP
+  - name: this block consists of tasks related to azure storage
+    block:
+    - name: set common gcloud variables
+      set_fact:
+        gcp_storage_service_account_name: "{{ cloud_public_storage_accountname }}"
+        gcp_storage_key_file: "{{ cloud_public_storage_secret }}"
+        gcp_bucket_name: "{{ cloud_storage_playercdn_bucketname }}"
+        gcp_path: ""
+        file_delete_pattern: ""
+
+
+    - name: delete files and folders from gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: delete-batch.yml
+
+    - name: upload batch of files to gcloud storage
+      include_role:
+        name: gcp-cloud-storage
+        tasks_from: upload-batch.yml
+      vars:
+        local_file_or_folder_path: "{{ assets }}/*"
+    when: cloud_service_provider == "gcloud"
diff --git a/ansible/bootstrap.yml b/ansible/bootstrap.yml
@@ -8,7 +8,7 @@
     - name: Installing other packages
       apt:
         state: present
-        name: ['python-pkg-resources', 'python2-pip']
+        name: ['python-pkg-resources', 'python-pip']
       when: ansible_distribution_version | float < 18
     - name: Installing other packages
       apt:
@@ -19,10 +19,8 @@
     - bootstrap_any
   tags:
     - bootstrap_any
-    - all
 
-- hosts: "{{hosts}}"
-  gather_facts: no
+- hosts: "{{ hosts }}"
   become: yes
   ignore_unreachable: yes
   vars_files:
@@ -32,6 +30,36 @@
   tags:
     - azure_cli
 
+- hosts: "{{ hosts }}"
+  become: yes
+  ignore_unreachable: yes
+  vars_files:
+    - "{{inventory_dir}}/secrets.yml"
+  roles:
+    - role: gcloud-cli
+  tags:
+    - gcloud_cli
+
+- hosts: "{{ hosts }}"
+  become: yes
+  ignore_unreachable: yes
+  vars_files:
+    - "{{inventory_dir}}/secrets.yml"
+  roles:
+    - role: aws-cli
+  tags:
+    - aws_cli
+
+- hosts: "{{ hosts }}"
+  become: yes
+  ignore_unreachable: yes
+  vars_files:
+    - "{{inventory_dir}}/secrets.yml"
+  roles:
+    - role: oci-cli
+  tags:
+    - oci_cli
+
 - hosts: "{{ hosts| default('all') }}"
   become: yes
   gather_facts: no
@@ -42,4 +70,3 @@
     - vm-agents-nodeexporter
   tags:
     - node_exporter
-    - all
diff --git a/ansible/cassandra-deploy.yml b/ansible/cassandra-deploy.yml
@@ -3,7 +3,8 @@
   environment:
     sunbird_cassandra_host: "{{ groups['cassandra'][0] }}"
     sunbird_cassandra_port: 9042
-    sunbird_cassandra_keyspace: sunbird
+    sunbird_cassandra_keyspace: "{{cassandra_keyspace_to_migrate}}"
+    sunbird_cassandra_migration_location: "filesystem:migration-script/{{cassandra_keyspace_to_migrate}}"
   vars_files:
     - ['{{inventory_dir}}/secrets.yml', 'secrets/{{env}}.yml']
   roles:

diff --git a/ansible/cert-file-upload.yml b/ansible/cert-file-upload.yml