Conductor waits more than an hour to schedule the next task. #163
Assignees
Comments
|
can you share how many running workflows do you have in the system? |
|
Hi @v1r3n, Thanks for the response. There are no other running workflows. Its using redis with the config of the default docker-compose.yaml from the repo |
|
Had the same problem with "JOIN" taking hours to finish. Changed the isAsync on the task back to "true" fixed the long delay #120 (comment) |
|
Hi @anzerr we are reverting this back to async |
|
I see this is fixed on latest main. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Tasks with an execution time of several minutes complete successfully, but the next task is not scheduled until approx. an hour and a half later. The worker is running and idle. The task queue has a size of 0 and shows the working polling every second.
Whenever I execute the workflow below, the tasks in
forkall run fine, The first tasks in each parallel path offork_2complete but theno-op-worker-SET2-1-2andno-op-worker-SET2-2-2tasks don't execute. The execution tab in the UI shows them with status 'Not executed'Details
Conductor version: Tried 3.18.0 and 3.19.0
Running based on Running Conductor Using Docker for evaluation and demo purposes.
On the Orkes playground, these type of workflows are running fine. I checked the documentation, but did not find anything that seemed relevant to configure.
Workflow definition:
Here's a workflow that demonstrates the issue.
{ "updateTime": 1715972548277, "accessPolicy": {}, "name": "why_so_long", "version": 1, "tasks": [ { "name": "fork", "taskReferenceName": "fork", "inputParameters": {}, "type": "FORK_JOIN", "forkTasks": [ [ { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET1-1-1", "inputParameters": { "Time": "00:00:01" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET1-1-2", "inputParameters": { "Time": "00:00:01" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false } ], [ { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET1-2-1", "inputParameters": { "Time": "00:00:01" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET1-2-2", "inputParameters": { "Time": "00:00:01" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false } ] ], "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "join", "taskReferenceName": "join", "inputParameters": {}, "type": "JOIN", "startDelay": 0, "joinOn": [ "no-op-worker-SET1-1-1", "no-op-worker-SET1-1-2", "no-op-worker-SET1-2-1", "no-op-worker-SET1-2-2" ], "optional": false, "asyncComplete": false, "permissive": false }, { "name": "fork_2", "taskReferenceName": "fork_2", "inputParameters": {}, "type": "FORK_JOIN", "forkTasks": [ [ { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET2-1-1", "inputParameters": { "Time": "00:02:00" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET2-1-2", "inputParameters": { "Time": "00:02:00" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false } ], [ { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET2-2-1", "inputParameters": { "Time": "00:02:00" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "no-op-worker", "taskReferenceName": "no-op-worker-SET2-2-2", "inputParameters": { "Time": "00:02:00" }, "type": "SIMPLE", "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false } ] ], "startDelay": 0, "optional": false, "asyncComplete": false, "permissive": false }, { "name": "join_2", "taskReferenceName": "join_2", "inputParameters": {}, "type": "JOIN", "startDelay": 0, "joinOn": [ "no-op-worker-SET2-1-1", "no-op-worker-SET2-1-2", "no-op-worker-SET2-2-1", "no-op-worker-SET2-2-2" ], "optional": false, "asyncComplete": false, "permissive": false } ], "inputParameters": [], "outputParameters": {}, "schemaVersion": 2, "restartable": true, "workflowStatusListenerEnabled": false, "ownerEmail": "[email protected]", "timeoutPolicy": "ALERT_ONLY", "timeoutSeconds": 0, "variables": {}, "inputTemplate": {} }Task definition:
{ "createTime": 1715972548160, "createdBy": "", "accessPolicy": {}, "name": "no-op-worker", "description": "Does nothing", "retryCount": 3, "timeoutSeconds": 0, "inputKeys": [ "Time" ], "outputKeys": [], "timeoutPolicy": "TIME_OUT_WF", "retryLogic": "FIXED", "retryDelaySeconds": 60, "responseTimeoutSeconds": 3600, "inputTemplate": { "Time": "00:00:05.00" }, "rateLimitPerFrequency": 0, "rateLimitFrequencyInSeconds": 1, "ownerEmail": "[email protected]", "backoffScaleFactor": 1 }To Reproduce
The
no-op-workersleeps for for requested Time and return completed state. Its instantiated like this:[WorkerTask(taskType: "no-op-worker", batchSize: 100, domain: null, pollIntervalMs: 200, workerId: "deploy-worker")]Expected behavior
I expect the next task to be scheduled for execution when its predecessor completes.
The text was updated successfully, but these errors were encountered: