Deploy status stuck on 'PENDING' #57

ramazulay · 2022-04-07T11:42:51Z

ramazulay
Apr 7, 2022

Hi,
I have a issue when I try to deploy, the status is stuck on 'PENDING'.
The SLD installed on EKS cluster.
The log from worker(stack-deploy-worker-default-7dcbdfd87f-57mvm):
[2022-04-07 13:18:34,045: INFO/MainProcess] Task pipeline git pull[f9c33aed-f988-426f-bdc4-25c99b29d48b] received
[2022-04-07 13:18:34,051: INFO/ForkPoolWorker-2] Download git repo https://github.com/D10S0VSkY-OSS/aws_vpc_poc branch master
[2022-04-07 13:18:34,983: WARNING/ForkPoolWorker-2] [WARNING]: No inventory was parsed, only implicit localhost is available
[2022-04-07 13:18:35,909: WARNING/ForkPoolWorker-2] localhost | SUCCESS => {
"after": "8076b398a8db5f535436298e912f1367b296088a",
"before": "8076b398a8db5f535436298e912f1367b296088a",
"changed": false,
"remote_url_changed": false
}
[2022-04-07 13:18:36,139: INFO/ForkPoolWorker-2] Task pipeline git pull[f9c33aed-f988-426f-bdc4-25c99b29d48b] succeeded in 2.092451297910884s: {'command': 'get_vars_json', 'rc': 0, 'stdout': '{"variable": {"region": {"default": "eu-west-1"}, "vpc_cidr": {"type": "string", "description": "The allowed block size is between a /28 netmask and /16 netmask", "default": "10.0.0.0/16"}, "subnet_cidr": {"type": "map("string")", "description": "Don't change this without first changing vpc_cidr", "default": {"be1": "10.0.0.0/24", "be2": "10.0.1.0/24", "fe1": "10.0.2.0/24", "fe2": "10.0.3.0/24"}}, "vpc_name": {"type": "string", "description": "Set VPC name without special characters", "default": "name"}, "enable_dns_support": {"type": "bool", "description": "Set true or false if you want dns support", "default": false}}}'}
[2022-04-07 13:21:06,506: INFO/MainProcess] Task pipeline git pull[ce2c9f7c-e3bc-4b75-9062-2294eb8db26f] received
[2022-04-07 13:21:06,508: INFO/ForkPoolWorker-2] Download git repo https://github.com/D10S0VSkY-OSS/aws_vpc_poc branch master
[2022-04-07 13:21:07,399: WARNING/ForkPoolWorker-2] [WARNING]: No inventory was parsed, only implicit localhost is available
[2022-04-07 13:21:08,401: WARNING/ForkPoolWorker-2] localhost | SUCCESS => {
"after": "8076b398a8db5f535436298e912f1367b296088a",
"before": "8076b398a8db5f535436298e912f1367b296088a",
"changed": false,
"remote_url_changed": false
}
[2022-04-07 13:21:08,604: INFO/ForkPoolWorker-2] Task pipeline git pull[ce2c9f7c-e3bc-4b75-9062-2294eb8db26f] succeeded in 2.096647311002016s: {'command': 'get_vars_json', 'rc': 0, 'stdout': '{"variable": {"region": {"default": "eu-west-1"}, "vpc_cidr": {"type": "string", "description": "The allowed block size is between a /28 netmask and /16 netmask", "default": "10.0.0.0/16"}, "subnet_cidr": {"type": "map("string")", "description": "Don't change this without first changing vpc_cidr", "default": {"be1": "10.0.0.0/24", "be2": "10.0.1.0/24", "fe1": "10.0.2.0/24", "fe2": "10.0.3.0/24"}}, "vpc_name": {"type": "string", "description": "Set VPC name without special characters", "default": "name"}, "enable_dns_support": {"type": "bool", "description": "Set true or false if you want dns support", "default": false}}}'}
[2022-04-07 13:21:15,040: INFO/MainProcess] Task pipeline git pull[a1fc132c-b86f-4acf-93c5-774e3350ed3c] received
[2022-04-07 13:21:15,043: INFO/ForkPoolWorker-2] Download git repo http://172.16.132.170/test-ram branch master
[2022-04-07 13:21:15,916: WARNING/ForkPoolWorker-2] [WARNING]: No inventory was parsed, only implicit localhost is available
[2022-04-07 13:21:16,408: WARNING/ForkPoolWorker-2] localhost | SUCCESS => {
"after": "a0b548490759552a9813a40cd58de96c0bbe6902",
"before": "a0b548490759552a9813a40cd58de96c0bbe6902",
"changed": false,
"remote_url_changed": false
}
[2022-04-07 13:21:16,604: INFO/ForkPoolWorker-2] Task pipeline git pull[a1fc132c-b86f-4acf-93c5-774e3350ed3c] succeeded in 1.5627696781884879s: {'command': 'get_vars_json', 'rc': 0, 'stdout': '{"variable": {"instance_type": {"type": "string", "description": "instance type", "default": "t2.micro"}}}'}
[2022-04-07 13:22:00,693: INFO/MainProcess] Task schedule remove[b1eb0191-7f1d-4c3e-a717-bd303c2b0e31] received
[2022-04-07 13:22:00,694: INFO/MainProcess] Task schedule add[922932d4-9b4b-40d2-8295-03372ccab376] received
[2022-04-07 13:22:00,701: INFO/ForkPoolWorker-2] Task schedule remove[b1eb0191-7f1d-4c3e-a717-bd303c2b0e31] succeeded in 0.0072300180327147245s: {'result': '{'11': 'removed'}'}
[2022-04-07 13:22:01,044: INFO/ForkPoolWorker-2] Task schedule add[922932d4-9b4b-40d2-8295-03372ccab376] succeeded in 0.34154637693427503s: {'deploy': 'start-test-sld_test-ram_test (trigger: cron[month='', day='', day_of_week='0-4', hour='', minute=''], next run at: 2022-04-07 13:23:00 CEST)', 'destroy': 'destroy-test-sld_test-ram_test (trigger: cron[month='', day='', day_of_week='0-4', hour='18', minute='30'], next run at: 2022-04-07 18:30:00 CEST)'}

The log from dashboard(sld-dashboard-75f9bf9476-l9rcr):

cko) Chrome/99.0.4844.51 Safari/537.36"
[2022-04-07 13:22:45 +0200] [1] [CRITICAL] WORKER TIMEOUT (pid:437)
[2022-04-07 13:22:45 +0200] [437] [INFO] Worker exiting (pid: 437)
[2022-04-07 13:22:45 +0200] [441] [INFO] Booting worker with pid: 441

The log from schedule(schedule-584bc87b48-xqz67):
INFO Get deploy info by id 11 - 200
INFO Update deploy info by id 11 - 409
INFO Job "start-test-sld_test-ram_test (trigger: cron[month='', day='', day_of_week='0-4', hour='', minute=''], next run at: 2022-04-07 13:34:00 CEST)" executed successfully
INFO Get deploy info by id 13 - 200
INFO Update deploy info by id 13 - 409
INFO Job "start-test_vpc_test-ram_test (trigger: cron[month='', day='', day_of_week='0-4', hour='', minute=''], next run at: 2022-04-07 13:34:00 CEST)" executed successfully
INFO Get deploy info by id 12 - 200
INFO Update deploy info by id 12 - 409
INFO Job "start-test-sld2_test-ram_test (trigger: cron[month='', day='', day_of_week='0-4', hour='', minute=''], next run at: 2022-04-07 13:34:00 CEST)" executed successfully
INFO: 172.16.143.162:58988 - "GET / HTTP/1.1" 200 OK

All other microservices show ok in the logs.
I tried to deploy manually with the same cred and I succeeded.

kubectl get all -n sld
NAME READY STATUS RESTARTS AGE
pod/api-backend-f949d77ff-kpkj5 1/1 Running 3 26h
pod/db-544bbbd454-4dc86 1/1 Running 0 26h
pod/rabbit-755f8645d9-9b9zq 1/1 Running 0 26h
pod/redis-b98595559-jhn9x 1/1 Running 0 26h
pod/remote-state-77b6c66b78-mpwpw 1/1 Running 0 26h
pod/schedule-584bc87b48-xqz67 1/1 Running 1 26h
pod/sld-dashboard-75f9bf9476-l9rcr 1/1 Running 0 23h
pod/stack-deploy-worker-default-7dcbdfd87f-57mvm 1/1 Running 0 23h
pod/stack-deploy-worker-squad1-699bdfff74-jlmzv 1/1 Running 0 23h
pod/stack-deploy-worker-squad2-866457b57f-m9b9x 1/1 Running 0 23h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/api-backend NodePort 10.100.227.176 8000:30003/TCP 26h
service/db NodePort 10.100.53.70 3306:30006/TCP 26h
service/rabbit NodePort 10.100.92.172 5672:30001/TCP 26h
service/redis NodePort 10.100.56.110 6379:30002/TCP 26h
service/remote-state NodePort 10.100.10.91 8080:30004/TCP 26h
service/schedule NodePort 10.100.27.100 10000:30007/TCP 26h
service/sld-dashboard LoadBalancer 10.100.52.220 internal-XXXXXXXXX 5000:30005/TCP 26h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/api-backend 1/1 1 1 26h
deployment.apps/db 1/1 1 1 26h
deployment.apps/rabbit 1/1 1 1 26h
deployment.apps/redis 1/1 1 1 26h
deployment.apps/remote-state 1/1 1 1 26h
deployment.apps/schedule 1/1 1 1 26h
deployment.apps/sld-dashboard 1/1 1 1 26h
deployment.apps/stack-deploy-worker-default 1/1 1 1 26h
deployment.apps/stack-deploy-worker-squad1 1/1 1 1 26h
deployment.apps/stack-deploy-worker-squad2 1/1 1 1 26h

NAME DESIRED CURRENT READY AGE
replicaset.apps/api-backend-f949d77ff 1 1 1 26h
replicaset.apps/db-544bbbd454 1 1 1 26h
replicaset.apps/rabbit-755f8645d9 1 1 1 26h
replicaset.apps/redis-b98595559 1 1 1 26h
replicaset.apps/remote-state-77b6c66b78 1 1 1 26h
replicaset.apps/schedule-584bc87b48 1 1 1 26h
replicaset.apps/sld-dashboard-75f9bf9476 1 1 1 26h
replicaset.apps/stack-deploy-worker-default-7dcbdfd87f 1 1 1 26h
replicaset.apps/stack-deploy-worker-squad1-699bdfff74 1 1 1 26h
replicaset.apps/stack-deploy-worker-squad2-866457b57f 1 1 1 26h

Please assist.

D10S0VSkY-OSS · 2022-04-09T00:10:34Z

D10S0VSkY-OSS
Apr 9, 2022
Maintainer

Hello @ramazulay,
I don't see the final result in the log
In SLD, pending status can have two meanings:

That you have launched a deployment and the task is pending because there is no available worker that can handle it (when doing a redeployment you should receive an error in the console "The task already exists in pending status waiting to be executed",
Or that task have passed more than three days (it's the default that values are saved) and you don't know what the status of that task is, this last behavior is done intentionally to enforce good practices and apply every 3 days (you can change the behavior by passing the variable to the worker)
vaiable example

          - name: SLD_RESULT_EXPIRE
            value: "604800"

simple example worker for squad1

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sld-worker-squad1
  labels:
    name: sld-worker-squad1
spec:
  replicas: 2
  selector:
    matchLabels:
      name: sld-worker-squad1
  template:
    metadata:
      labels:
        name: sld-worker-squad1
    spec:
      subdomain: primary
      containers:
        - name: sld-worker-squad1
          image: sld-api
          imagePullPolicy: Always
          env:
          - name: SCHEDULE_SERVER
            value: "http://schedule:10000"
          - name: SLD_RESULT_EXPIRE
            value: "604800"
          - name: BROKER_USER
            valueFrom:
              secretKeyRef:
                name: rabbit-default-user
                key: username
          - name: BROKER_PASSWD
            valueFrom:
              secretKeyRef:
                name: rabbit-default-user
                key: password
          resources:
            limits:
              memory: 900Mi
              cpu: 1
            requests:
              memory: 600Mi
              cpu: 200m
          command: ["celery", "--app", "tasks.celery_worker", "worker","--without-heartbeat", "--without-gossip", "--without-mingle", "--loglevel=info", "-c", "1", "-E", "-O", "fair", "-Q", "squad1-nonprod"]

I understand that you are testing the product, it is recommended to exclude the EKS database to an RDS mysql.
If you need more help, don't hesitate to contact me.

1 reply

D10S0VSkY-OSS Apr 13, 2022
Maintainer

@ramazulay
If you have problems with kind, you can try docker-compose

cd Stack-Lifecycle-Deployment/play-with-sld/docker/
sh play.sh start

you need install dokcer compose docker-compose

D10S0VSkY-OSS · 2022-04-15T15:17:01Z

D10S0VSkY-OSS
Apr 15, 2022
Maintainer

@ramazulay
check if your account name (in the example squad 1) has a dedicated worker listening with the same name.
If you have registered a new account, for example from aws called "myawsaccount01" you must create a worker to listen to that account
in Stack-Lifecycle-Deployment/play-with-sld/docker/docker-compose.yml line 87

entrypoint: ["celery", "--app", "tasks.celery_worker", "worker", "--loglevel=info", "-c", "1", "-E", "-Q", "celery,squad,squad1,myawsaccount01"]

In Stack-Lifecycle-Deployment/play-with-sld/docker/kubernetes/k8s/sld-worker-squad1.yml line 32

command: ["celery", "--app", "tasks.celery_worker", "worker", "--loglevel=info", "-c", "1", "-E", "-Q", "squad1,myawsaccount01"]

Or create new deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stack-deploy-worker-myawsaccount01
  labels:
    name: stack-deploy-worker-myawsaccount01
spec:
  replicas: 1 
  selector:
    matchLabels:
      name: stack-deploy-worker-myawsaccount01
  template:
    metadata:
      labels:
        name: stack-deploy-worker-myawsaccount01
    spec:
      subdomain: primary
      containers:
        - name: stack-deploy-worker-myawsaccount01
          image: d10s0vsky/sld-api:latest
          imagePullPolicy: Always
          env:
          - name: TF_WARN_OUTPUT_ERRORS
            value: "1"
          resources:
            limits:
              memory: 600Mi
              cpu: 1
            requests:
              memory: 300Mi
              cpu: 500m
          command: ["celery", "--app", "tasks.celery_worker", "worker", "--loglevel=info", "-c", "1", "-E", "-Q", "myawsaccount01"]

1 reply

ramazulay Apr 24, 2022
Author

Hi,
Thanks for your reply.
It works! I did not know that the name of the cloud account should be the same as the worker.
Thanks for your help.

ramazulay · 2022-04-24T11:58:15Z

ramazulay
Apr 24, 2022
Author

Hi @D10S0VSkY-OSS ,
Can you send me an example of deployment with ssh key for git repo authentication.
"For ssh you can pass it as a secret in the deployment to the user sld"

Thanks.

1 reply

D10S0VSkY-OSS Apr 26, 2022
Maintainer

Hi @ramazulay
you can add this to your docker-compose.yml:

volumes:
    - ~/.ssh:/home/sld/.ssh

ramazulay · 2022-04-26T12:54:15Z

ramazulay
Apr 26, 2022
Author

Hi @D10S0VSkY-OSS ,
Is it possible to assign a user to a specific env, ie to give permission to each env and not just to the squad?

Thanks.

1 reply

D10S0VSkY-OSS Apr 26, 2022
Maintainer

Hi @ramazulay
no, you can only by squad
but you can create different squad with the same aws account.
squad-env01
squad-env02

ramazulay · 2022-10-11T08:27:11Z

ramazulay
Oct 11, 2022
Author

Hi, There is a tfvars file in my repository and when I create a deploy(by stack) the field is not completed. Am I missing something? Thanks. ‫בתאריך יום ד׳, 27 באפר׳ 2022 ב-2:32 מאת ‪Diego La Falce‬‏ <‪ ***@***.***‬‏>:‬

…

Hi @ramazulay <https://github.com/ramazulay> no, you can only by squad but you can create different squad with the same aws account. squad-env01 squad-env02 — Reply to this email directly, view it on GitHub <#57 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJ44ADD5IRPJAFQOOI4CUIDVHB4HBANCNFSM5TL46SFA> . You are receiving this because you were mentioned.Message ID: <D10S0VSkY-OSS/Stack-Lifecycle-Deployment/repo-discussions/57/comments/2642513 @github.com>

0 replies

D10S0VSkY · 2022-10-11T15:56:14Z

D10S0VSkY
Oct 11, 2022

Hi @ramazulay ,
What do you mean with "the field is not completed." ?
Which version are you using?

Regards,

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy status stuck on 'PENDING' #57

{{title}}

Replies: 6 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Deploy status stuck on 'PENDING' #57

ramazulay Apr 7, 2022

Replies: 6 comments · 4 replies

D10S0VSkY-OSS Apr 9, 2022 Maintainer

D10S0VSkY-OSS Apr 13, 2022 Maintainer

D10S0VSkY-OSS Apr 15, 2022 Maintainer

ramazulay Apr 24, 2022 Author

ramazulay Apr 24, 2022 Author

D10S0VSkY-OSS Apr 26, 2022 Maintainer

ramazulay Apr 26, 2022 Author

D10S0VSkY-OSS Apr 26, 2022 Maintainer

ramazulay Oct 11, 2022 Author

D10S0VSkY Oct 11, 2022

ramazulay
Apr 7, 2022

Replies: 6 comments 4 replies

D10S0VSkY-OSS
Apr 9, 2022
Maintainer

D10S0VSkY-OSS Apr 13, 2022
Maintainer

D10S0VSkY-OSS
Apr 15, 2022
Maintainer

ramazulay Apr 24, 2022
Author

ramazulay
Apr 24, 2022
Author

D10S0VSkY-OSS Apr 26, 2022
Maintainer

ramazulay
Apr 26, 2022
Author

D10S0VSkY-OSS Apr 26, 2022
Maintainer

ramazulay
Oct 11, 2022
Author

D10S0VSkY
Oct 11, 2022