Workflow continues to run after cancelled on dashboard

[kahkeong]

Description:
After cancelling a workflow on dashboard, There is a warning 'Thread 6330920960 with run id 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c is still running after cancellation'. The code still continue to run until it finishes.

Logs:

[DEBUG] 🪓 -- 2024-12-23 15:34:48,817 - sending heartbeat
[INFO]  🪓 -- 2024-12-23 15:34:51,904 - rx: start step run: 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c/first-python-workflow:step1
[DEBUG] 🪓 -- 2024-12-23 15:34:51,905 - tx: event: first-python-workflow:step1/1
[INFO]  🪓 -- 2024-12-23 15:34:51,905 - run: start step: first-python-workflow:step1/3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[DEBUG] 🪓 -- 2024-12-23 15:34:51,906 - tx: event: first-python-workflow:step1/1
INFO:root:executed step1
[DEBUG] 🪓 -- 2024-12-23 15:34:51,906 - start time: 0.0013298988342285156
0
[DEBUG] 🪓 -- 2024-12-23 15:34:52,826 - sending heartbeat
1
2
3
4
[DEBUG] 🪓 -- 2024-12-23 15:34:56,871 - sending heartbeat
5
[INFO]  🪓 -- 2024-12-23 15:34:57,469 - rx: cancel step run: 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[INFO]  🪓 -- 2024-12-23 15:34:57,469 - cancel: step run:  /3be4c09f-7ca1-482d-aeee-30fd50f9eb1c
[DEBUG] 🪓 -- 2024-12-23 15:34:57,470 - cancelling step...
6
[WARNING]       🪓 -- 2024-12-23 15:34:58,471 - Thread 6330920960 with run id 3be4c09f-7ca1-482d-aeee-30fd50f9eb1c is still running after cancellation. This could cause the thread pool to get blocked and prevent new tasks from running.
7
8
[DEBUG] 🪓 -- 2024-12-23 15:35:00,933 - sending heartbeat
9
10
11
12
[DEBUG] 🪓 -- 2024-12-23 15:35:04,938 - sending heartbeat
13
14
15
16
[DEBUG] 🪓 -- 2024-12-23 15:35:08,940 - sending heartbeat
17
18
19
[DEBUG] 🪓 -- 2024-12-23 15:35:12,944 - sending heartbeat
[DEBUG] 🪓 -- 2024-12-23 15:35:16,949 - sending heartbeat
[DEBUG] 🪓 -- 2024-12-23 15:35:21,017 - sending heartbeat

Expected behaviour:
It should stop logging new numbers after the workflow is cancelled.

Worker code for reproduction:

from hatchet_sdk import Context, Hatchet, ClientConfig
from dotenv import load_dotenv
import logging

logging.basicConfig(level=logging.INFO)
LOG = logging.getLogger()

load_dotenv()
  
hatchet = Hatchet(
    debug=True,
    config=ClientConfig(
        logger=LOG,
    ),
)

 
@hatchet.workflow(name="first-python-workflow")
class MyWorkflow:
    @hatchet.step(retries=3)
    def step1(self, context: Context):
        LOG.info("executed step1")
        import time
        i = 0
        while i < 20:
            print(i)
            i += 1
            time.sleep(1)
        return {
            "result": "success"
        }
 

if __name__ == "__main__":
    worker = hatchet.worker('first-worker')
    worker.register_workflow(MyWorkflow())
    worker.start()

Configuration Details:

• Self hosted with https://docs.hatchet.run/self-hosting/docker-compose
• Hatched SDK version 0.42.5
• Python version: 3.11.9
• macOS version: 15.1.1

[mrkaye97]

hey @kahkeong - sorry for just circling back to this. The reason for this is that you're doing blocking work (time.sleep) and are in a synchronous function (def step1). We added an option in this PR called HATCHET_CLIENT_ENABLE_FORCE_KILL_SYNC_THREADS which you can set to True if you want to enable force-killing the function that's hanging. Importantly, this is somewhat dangerous, so just be aware that if you're e.g. halfway through a long database write or something and the thread is killed, that write will not finish.

Hopefully this is helpful, sorry for the delay here!