Deploy > DataPipeline > AnalyticsCore

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github Tag: release-4.6.0_RC3
   @gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Checking out git https://github.com/project-sunbird/sunbird-data-pipeline.git into /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@script to read pipelines/deploy/data-products/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-data-pipeline.git

git init /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@script # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
git config core.sparsecheckout # timeout=10
git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@libs/deploy-conf # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-data-pipeline.git
git init /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
git config core.sparsecheckout # timeout=10
git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » DataPipeline » AnalyticsCore" build number 5
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip lpa_core_artifacts.zip:release-4.4.0_RC3
  Archive: lpa_core_artifacts.zip:release-4.4.0_RC3
  inflating: analytics-framework-2.0.jar
  inflating: scruid_2.11-2.4.0.jar
• [ -f batch-models-2.0.jar ]
• [ -f job-manager-2.0.jar ]
• [ -f analytics-framework-2.0.jar ]
• echo analytics-framework-2.0.jar exist
  analytics-framework-2.0.jar exist
• mv analytics-framework-2.0.jar ansible
• [ -f scruid_2.11-2.4.0.jar ]
• echo scruid_2.11-2.4.0.jar exist
  scruid_2.11-2.4.0.jar exist
• mv scruid_2.11-2.4.0.jar ansible
  [Pipeline] echo
  {env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=master, artifact_name=lpa_core_artifacts.zip, artifact_version=release-4.4.0_RC3, currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-4.4.0_RC3, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=master, artifact_name=lpa_core_artifacts.zip, artifact_version=release-4.4.0_RC3, currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-4.4.0_RC3, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/shounakacharya/ansible.git

git init /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private # timeout=10
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 67d7b69b0443b3e7cca95f65bfd0f3b1e26a4938 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 67d7b69b0443b3e7cca95f65bfd0f3b1e26a4938 # timeout=10
Commit message: "Adding API key of google drive for content services"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/common.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/hosts /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/hosts --extra-vars remote=spark /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/AnalyticsCore/ansible/inventory/env/hosts with yaml plugin: Syntax Error
while loading YAML. found unexpected ':' The error appears to be in '/var/li
b/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env
/hosts': line 1, column 5, but may be elsewhere in the file depending on the
exact syntax problem. The offending line appears to be: [all:vars] ^
here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/AnalyticsCore/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/w
orkspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/hosts:57:
Section [lpa:children] includes undefined group: analytics-api

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/A
nalyticsCore/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [spark] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [10.128.0.51]

TASK [data-products-deploy : Ensure azure blob storage container exists] *******
changed: [10.128.0.51]

TASK [data-products-deploy : Copy Framework Library] ***************************
changed: [10.128.0.51]

TASK [data-products-deploy : Copy Framework Library to azure blob] *************
changed: [10.128.0.51]

TASK [data-products-deploy : Copy Scruid Library] ******************************
changed: [10.128.0.51]

TASK [data-products-deploy : Copy Scruid Library to azure blob] ****************
changed: [10.128.0.51]

TASK [data-products-deploy : Copy configuration file] **************************
fatal: [10.128.0.51]: FAILED! => {
"changed": false
}

MSG:

AnsibleUndefinedVariable: 'uci_encryption_key_base64' is undefined

PLAY RECAP *********************************************************************
10.128.0.51 : ok=6 changed=5 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• rev+
  git name-rev --name-only HEAD
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Looks like this variable is missing in the private repo template.
Please add a variable as below in both Core/secrets.yml and DataPipeline/secrets.yml. Please use same value in both the secret files.

uci_encryption_key_base64: SOME_BASE64_STRING

# Please replace SOME_BASE64_STRING with the value generated using below command

You can generate a base64 string using below command on a linux vm.
echo "randomstring" | base64

[shounakacharya]

@keshavprasadms: I updated as you suggested. Now I am getting the following error:

Started by user Admin Deployer
Rebuilds build #1
Checking out git https://github.com/project-sunbird/sunbird-data-pipeline.git into /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@script to read pipelines/deploy/data-products/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@script/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
git config core.sparsecheckout # timeout=10
git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
git rev-list --no-walk 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/analytics-framework-2.0.jar
  Removing ansible/scruid_2.11-2.4.0.jar
  Removing lpa_core_artifacts.zip:release-4.4.0_RC3
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
git config core.sparsecheckout # timeout=10
git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » DataPipeline » AnalyticsCore" build number 5
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip lpa_core_artifacts.zip:release-4.4.0_RC3
  Archive: lpa_core_artifacts.zip:release-4.4.0_RC3
  inflating: analytics-framework-2.0.jar
  inflating: scruid_2.11-2.4.0.jar
• [ -f batch-models-2.0.jar ]
• [ -f job-manager-2.0.jar ]
• [ -f analytics-framework-2.0.jar ]
• echo analytics-framework-2.0.jar exist
  analytics-framework-2.0.jar exist
• mv analytics-framework-2.0.jar ansible
• [ -f scruid_2.11-2.4.0.jar ]
• echo scruid_2.11-2.4.0.jar exist
  scruid_2.11-2.4.0.jar exist
• mv scruid_2.11-2.4.0.jar ansible
  [Pipeline] echo
  {env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=master, artifact_name=lpa_core_artifacts.zip, artifact_version=release-4.4.0_RC3, currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-4.4.0_RC3, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=DataPipeline, jobName=AnalyticsCore, absolute_job_path=ArtifactUpload/dev/DataPipeline/AnalyticsCore, agent=master, artifact_name=lpa_core_artifacts.zip, artifact_version=release-4.4.0_RC3, currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore, artifact=lpa_core_artifacts.zip:release-4.4.0_RC3, ansiblePlaybook=--extra-vars "remote=spark" /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml, ansibleExtraArgs= --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 15f66997e11f55b20fdedc373e8b9dee3e0c53c9 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 15f66997e11f55b20fdedc373e8b9dee3e0c53c9 # timeout=10
Commit message: "after adding uci_encryption_key_base64 to secrets"
git rev-list --no-walk 67d7b69b0443b3e7cca95f65bfd0f3b1e26a4938 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/common.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/hosts /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/private/ansible/inventory/dev/DataPipeline/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/hosts --extra-vars remote=spark /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/lpa_data-products_deploy.yml --tags framework,framework-spark-cluster --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/AnalyticsCore/ansible/inventory/env/hosts with yaml plugin: Syntax Error
while loading YAML. found unexpected ':' The error appears to be in '/var/li
b/jenkins/workspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env
/hosts': line 1, column 5, but may be elsewhere in the file depending on the
exact syntax problem. The offending line appears to be: [all:vars] ^
here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/AnalyticsCore/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/w
orkspace/Deploy/dev/DataPipeline/AnalyticsCore/ansible/inventory/env/hosts:57:
Section [lpa:children] includes undefined group: analytics-api

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/A
nalyticsCore/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [spark] *******************************************************************

TASK [Gathering Facts] *********************************************************
ok: [10.128.0.51]

TASK [data-products-deploy : Ensure azure blob storage container exists] *******
changed: [10.128.0.51]

TASK [data-products-deploy : Copy Framework Library] ***************************
ok: [10.128.0.51]

TASK [data-products-deploy : Copy Framework Library to azure blob] *************
fatal: [10.128.0.51]: FAILED! => {
"ansible_job_id": "455113278671.16611",
"changed": true,
"cmd": [
"az",
"storage",
"blob",
"upload",
"--debug",
"-c",
"telemetry-data-store",
"--name",
"models-2.0/analytics-framework-2.0.jar",
"-f",
"/mount/data/analytics/models-2.0/analytics-framework-2.0.jar"
],
"delta": "0:00:03.618927",
"end": "2022-08-26 09:37:23.322252",
"finished": 1,
"rc": 1,
"start": "2022-08-26 09:37:19.703325"
}

STDERR:

DEBUG: cli.knack.cli: Command arguments: ['storage', 'blob', 'upload', '--debug', '-c', 'telemetry-data-store', '--name', 'models-2.0/analytics-framework-2.0.jar', '-f', '/mount/data/analytics/models-2.0/analytics-framework-2.0.jar']
DEBUG: cli.knack.cli: init debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f325977bbe0>, <function OutputProducer.on_global_arguments at 0x7f32584c04c0>, <function CLIQuery.on_global_arguments at 0x7f32584f16c0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: storage 0.063 56 261
DEBUG: cli.azure.cli.core: Total (1) 0.063 56 261
DEBUG: cli.azure.cli.core: Loaded 56 groups, 261 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : storage blob upload
DEBUG: cli.azure.cli.core: Command table: storage blob upload
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f32557c7400>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/analytics/.azure/commands/2022-08-26.09-37-19.storage_blob_upload.16622.log'.
INFO: az_command_data_logger: command args: storage blob upload --debug -c {} --name {} -f {}
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f3254fe0e50>]
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 623, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2021_09_01.models' has no attribute 'AccountType'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 623, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2021_09_01.models' has no attribute 'ListKeyExpand'

DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f32550852d0>, <function register_cache_arguments..add_cache_arguments at 0x7f32550853f0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f32584c0550>, <function CLIQuery.handle_query_parameter at 0x7f32584f1750>, <function register_ids_argument..parse_ids_arguments at 0x7f3255085360>]
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage account value from environment variables or config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage account_url value from environment variables or config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage connection_string value from environment variables or config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage key value from environment variables or config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage sas_token value from environment variables or config file.
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2021_06_08.blob'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2021_06_08.file'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2021_06_08.blob'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 619, in _get_attr
op = import_module(full_mod_path)
File "/opt/az/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2021_06_08.file'

DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): shounaksunbirdcontent.blob.core.windows.net:443
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (2): shounaksunbirdcontent.blob.core.windows.net:443
DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQSUzRA%3D%3D HTTP/1.1" 201 0

Alive[## ] 4.3742%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNRFF4T1RRek1EUSUzRA%3D%3D HTTP/1.1" 201 0

Alive[##### ] 8.7485%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNRGd6T0RnMk1EZyUzRA%3D%3D HTTP/1.1" 201 0

Alive[######## ] 13.1227%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNVFkzTnpjeU1UWSUzRA%3D%3D HTTP/1.1" 201 0

Alive[########### ] 17.4969%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNVEkxT0RJNU1USSUzRA%3D%3D HTTP/1.1" 201 0

Alive[############# ] 21.8712%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNakE1TnpFMU1qQSUzRA%3D%3D HTTP/1.1" 201 0

Alive[################ ] 26.2454%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNamt6TmpBeE1qZyUzRA%3D%3D HTTP/1.1" 201 0

Alive[################### ] 30.6196%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNalV4TmpVNE1qUSUzRA%3D%3D HTTP/1.1" 201 0

Alive[###################### ] 34.9939%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNek0xTlRRME16SSUzRA%3D%3D HTTP/1.1" 201 0

Alive[######################### ] 39.3681%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOREU1TkRNd05EQSUzRA%3D%3D HTTP/1.1" 201 0

Alive[########################### ] 43.7423%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNemMzTkRnM016WSUzRA%3D%3D HTTP/1.1" 201 0

Alive[############################## ] 48.1166%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdORFl4TXpjek5EUSUzRA%3D%3D HTTP/1.1" 201 0

Alive[################################# ] 52.4908%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOVFExTWpVNU5USSUzRA%3D%3D HTTP/1.1" 201 0

Alive[#################################### ] 56.8650%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOVEF6TXpFMk5EZyUzRA%3D%3D HTTP/1.1" 201 0

Alive[####################################### ] 61.2393%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOVGczTWpBeU5UWSUzRA%3D%3D HTTP/1.1" 201 0

Alive[######################################### ] 65.6135%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOamN4TURnNE5qUSUzRA%3D%3D HTTP/1.1" 201 0

Alive[############################################ ] 69.9877%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOakk1TVRRMU5qQSUzRA%3D%3D HTTP/1.1" 201 0

Alive[############################################### ] 74.3620%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOekV6TURNeE5qZyUzRA%3D%3D HTTP/1.1" 201 0

Alive[################################################## ] 78.7362%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOemsyT1RFM056WSUzRA%3D%3D HTTP/1.1" 201 0

Alive[##################################################### ] 83.1104%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdOelUwT1RjME56SSUzRA%3D%3D HTTP/1.1" 201 0

Alive[####################################################### ] 87.4847%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdPRE00T0RZd09EQSUzRA%3D%3D HTTP/1.1" 201 0

Alive[########################################################## ] 91.8589%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdPRGd3T0RBek9EUSUzRA%3D%3D HTTP/1.1" 201 0

Alive[############################################################# ] 96.2331%DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdPVEl5TnpRMk9EZyUzRA%3D%3D HTTP/1.1" 201 0

Alive[################################################################] 100.0000%
Finished[#############################################################] 100.0000%
DEBUG: urllib3.connectionpool: https://shounaksunbirdcontent.blob.core.windows.net:443 "PUT /telemetry-data-store/models-2.0/analytics-framework-2.0.jar?comp=blocklist HTTP/1.1" 409 220
DEBUG: cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
DEBUG: cli.azure.cli.core.util: Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 632, in upload_blob
response = client.upload_blob(data=stream, length=length, metadata=metadata,
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2021_06_08/_blob_client.py", line 728, in upload_blob
return upload_block_blob(**options)
File "/opt/az/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2021_06_08/_upload_helpers.py", line 177, in upload_block_blob
process_storage_error(error)
File "/opt/az/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2021_06_08/_shared/response_handlers.py", line 181, in process_storage_error
exec("raise error from None") # pylint: disable=exec-used # nosec
File "", line 1, in
File "/opt/az/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2021_06_08/_upload_helpers.py", line 163, in upload_block_blob
return client.commit_block_list(
File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
return func(*args, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2021_06_08/_generated/operations/_block_blob_operations.py", line 1519, in commit_block_list
map_error(status_code=response.status_code, response=response, error_map=error_map)
File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
raise error
azure.core.exceptions.ResourceExistsError: The specified blob already exists.
RequestId:123e36bf-d01e-00a9-032f-b997ef000000
Time:2022-08-26T09:37:23.1346920Z
ErrorCode:BlobAlreadyExists
Content: BlobAlreadyExistsThe specified blob already exists.
RequestId:123e36bf-d01e-00a9-032f-b997ef000000
Time:2022-08-26T09:37:23.1346920Z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 663, in execute
raise ex
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 718, in _run_job
return cmd_copy.exception_handler(ex)
File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/storage/init.py", line 410, in new_handler
first(ex)
File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/storage/init.py", line 410, in new_handler
first(ex)
File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/storage/_exception_handler.py", line 17, in file_related_exception_handler
raise ex
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 333, in call
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 647, in upload_blob
raise AzureResponseError(
azure.cli.core.azclierror.AzureResponseError: The specified blob already exists.
RequestId:123e36bf-d01e-00a9-032f-b997ef000000
Time:2022-08-26T09:37:23.1346920Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.

ERROR: cli.azure.cli.core.azclierror: The specified blob already exists.
RequestId:123e36bf-d01e-00a9-032f-b997ef000000
Time:2022-08-26T09:37:23.1346920Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.
ERROR: az_command_data_logger: The specified blob already exists.
RequestId:123e36bf-d01e-00a9-032f-b997ef000000
Time:2022-08-26T09:37:23.1346920Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f32557c7640>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.main: Command ran in 3.452 seconds (init: 0.140, invoke: 3.312)
INFO: telemetry.save: Save telemetry record of length 3130 in cache
WARNING: telemetry.check: Negative: The /home/analytics/.azure/telemetry.txt was modified at 2022-08-26 09:37:18.337164, which in less than 600.000000 s

MSG:

non-zero return code

PLAY RECAP *********************************************************************
10.128.0.51 : ok=3 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• rev
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
  [Pipeline] sh
• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/DataPipeline/AnalyticsCore
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[shounakacharya]

@keshavprasadms : This was due to the blob being there on the azure telemetry-data-store container. After manually deleting both the files, it ran fine. You might want to look at the az copy command here. Thanks. Marking this as answered

[beepdot]

Its due to this issue - Azure/azure-cli#21779
We are aware of this, but haven't yet fixed. When installing az, we should use a specific version. You can manually downgrade az client on the vm. Below are the versions we are using.

az --version
azure-cli                         2.24.0 *

core                              2.24.0 *
telemetry                          1.0.6

We will fix it in a future release. Thanks for pointing this out.