Using ^C on FreeBSD can lead to zombie, unkillable dust executable, using 100% CPU

[axman6]

I’ve run into this issue a few times now, where I’ve been running dust in (which I hope is) a pretty normal way, such as:

$ dust / -X /storage

several times I’ve hit ^C to kill it, and ended up with the executable continuing to run, using 100% CPU, and being unable to kill it with kill <PID> or sudo kill -9 <PID>. The only solution I have is to reboot, which is less than ideal for a server. Let me know if there’s anything I can do to help investigate. For reference, this is happening on FreeBSD 14.0 with all updates installed, dust 0.9.0 built and installed with cargo, rustc 1.76.0, and my file systems are all ZFS.

[bootandy]

Indexing: / 1504966 files, 71G ... -^B

Does the indexing keep updating ?

I have no idea why killing it does not work, but that sounds like a problem in the scheduler rather than something that I have any control over.

If you run du and cancel it - does the same behaviour occur ?

If you run without the '-X' does the same behaviour occur ?

[axman6]

No, the app stops doing anything at all visually, and never terminates (I've had it running for hours before realising it was still going in the past). I haven't seen the problem with du before, and a few tests just now shows du working as expected. I have see the same behaviour without -X too. I can quite reliably lock up dust with ^C (I just tried to run it in lldb to see if I could gget a stack trace, but hitting ^C in it caused it to lock up and I can't access lldb - how fun!)

I will see if I can get some kind of stack trace of what's happening while it's spinning.

[bootandy]

Does running it single threaded make any difference?
export RAYON_NUM_THREADS=1; dust /

[bootandy]

I can quite reliably lock up dust with ^C

I discovered dust doesn't listen for ^C. Fixing now.

[bootandy]

will go out in next major release

[axman6]

@bootandy Thanks so much, I'll give master a go on my machine and see if it's helped.

[axman6]

Initial very basic testing (dust / -X /storage as above and hitting ^C) looks good so far!

[axman6]

... it looks like I spoke too soon, my initial test did work, but now hitting ^C causes it to print Aborting and continue to scan:

~ dust / -X /storage
Indexing: / 2222789 files, 8.3T ... |^C
Aborting
Indexing: / 2232934 files, 8.4T ... |^C
Aborting
Indexing: / 2232934 files, 8.4T ... -^C
Aborting
Indexing: / 2232934 files, 8.4T ... \^C
Aborting
Indexing: / 2232934 files, 8.4T ... \^C
Aborting
Indexing: / 2232934 files, 8.4T ... -^C
Aborting
Indexing: / 2232934 files, 8.4T ... -^C
Aborting
Indexing: / 2232934 files, 8.4T ... -^C
Aborting
Indexing: / 2232934 files, 8.4T ... |^C
Aborting
Indexing: / 2232934 files, 8.4T ... /^C
Aborting
Indexing: / 2232934 files, 8.4T ... -^C
Aborting
Indexing: / 2232934 files, 8.4T ... |^C
Aborting
Indexing: / 2232934 files, 8.4T ... /

And using kill -9 on the executable still doesn't work, and it's still using 100% CPU. Can this issue be reopened?

[bootandy]

It clearly listens to the Ctrl-C because it prints 'Aborting'.

Here's an idea - Can you run in on a folder other than 'storage' and try and ctrl-C it.

I can't cancel the task it if it is in the middle of scanning a really big file. Each thread will need to finish its file before aborting. So if you have a really massive file in storage there's not much I can do.

Assuming the above works, I'd be curious about running it in each of the sub-folders in storage to figure out what the problem directory looks like