tmp space fills up #912

realdevnullius · 2021-08-28T07:19:02Z

realdevnullius
Aug 28, 2021

Apparently not all temp files get removed, slowly filling a disk until no more new plots can be made and 0 bytes space is left despite only 1 active plotman job for that disk is still running and all log files seem OK too. No problems were detected, I've been plotting for 8 days straight on my HP Server (2x NVMe 1.9TB, 2x SSD 1.9TB, 4x HDD Volume).

Sorry for the long title, let me start with some background.

I've been trying to tune my HP for a few weeks now. It's a sloooow process :) With previous testing I ran into the situation that one or more disks would get too full, messing up the plotting process (I'd often get corrupted plots out of it). After a while of testing, I started to notice that when I manually killed a PID, and manually removed it's temp files with their UID, I still got full disks EVEN when they shouldn't be full. If I tried to use plotman to stop a plot on a disk at risk of running out of space, plotman wouldn't actually "see" the temp files and report 0 bytes in size. I ignored that and just did the manual clean up. After a system reboot and a more proper handling on plotman, I started to play with some phase limits and global stagger values. This went great for 7 days, up to when I started to discover a bottleneck (of the two PCIe NVMe's, the 2nd one consistently underperformed for about a good 20% - same file systems and the OS is on the first NVMe, but still: it's substantially slower).

When I made my latest edits (see below) I noticed that for the first time, with a global stagger of 65 minutes, the nvme2 disk would get slower to finish (despite running less tasks), up to a point where a later started plot would finish faster (normally a plot finishes in about 24 hours, I've now seen the 2nd NVMe starting to do close to 28h about a plot all of a sudden). This was last evening and I kept it running overnight.

This morning, I noticed that my FIRST NVMe disk wasn't doing that many plots. In fact, it was only doing ONE plot. This was weird, a faster global plotting value but NOW all of a sudden new plots aren't created fast enough any longer?

I remember from my previous testing last week that I woke up to plotman constantly kicking off new plots, where it SHOULD've been impossible for the scheduler to even find new tasks to create, or so I thought. It looked weird to me, but again I sought blame with myself.

NOW, this morning, my first NVMe disk reported as being completely FULL, 0 bytes free space left!! And, only ONE plotting task running on it :(


       tmp            ready       phases                         dst                plots   GBfree      inbnd phases       pri 
      home/roet/nvm1      OK   2:3                   HDD00/Plots_OK/pooled/plotman   8       14206    1:3 1:6 2:4 3:2 4:0   26
mnt/4x_volume/run-11      --   1:2 1:6 3:1 3:5       HDD01/Plots_OK/pooled/plotman   31      9048     1:5 3:1 3:5           73
mnt/4x_volume/run-22      OK   1:5 2:4 3:4           HDD02/Plots_OK/pooled/plotman   35      8152     1:4 3:3 3:6           69
            mnt/nvm2      OK   1:4 3:2 3:6           HDD03/Plots_OK/pooled/plotman   13      13085    1:4 1:7 3:1           63
           mnt/ssd00      OK   1:4 3:3 4:0           HDD05/Plots_OK/pooled/plotman   30      9272     1:2 2:3 3:4 3:6       64

When I go to its temp folder, I find temp files for 5 different plots and I have 811 files. Yet as you can see, plotman insists there's only 1 plot running atm. It should run 3, with an optional 4th in phase 3.6 or 4.0.

Snippets from the config as it was when I first noticed this bug wasn't something I was messing up:

# Where to plot and log.
directories:
        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /mnt/nvm2
                - /mnt/ssd01
                - /mnt/4x_volume/run-22
                - /mnt/ssd00
                - /home/roet/nvm1
                - /mnt/4x_volume/run-11

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a
        # /home/roet is on nvme01
        # tmp2: /home/roet/plots.tmp-2/plotman
        tmp2: /mnt/4x_volume/tmp.02

        # Optional: A list of one or more directories; the scheduler will
        # use all of them.  These again are presumed to be on independent
        # physical devices so writes (plot jobs) and reads (archivals) can
        # be scheduled to minimize IO contention.
        #
        # If dst is commented out, the tmp directories will be used as the
        # buffer.
        dst:
                - /mnt/farm/HDD00/Plots_OK/pooled/plotman
                - /mnt/farm/HDD01/Plots_OK/pooled/plotman
                - /mnt/farm/HDD02/Plots_OK/pooled/plotman
                - /mnt/farm/HDD03/Plots_OK/pooled/plotman
                - /mnt/farm/HDD05/Plots_OK/pooled/plotman

# Archival configuration.  Optional; if you do not wish to run the
# archiving operation, comment this section out.  Almost everyone
# should be using the archival feature.  It is meant to distribute
# plots among multiple disks filling them all.  This can be done both
# to local and to remote disks.

# Plotting scheduling parameters
scheduling:
        # LIMIT WAS 8 TEMPORARILY TO 9 FOR HDD_VOLUME
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 9

        # Don't run more than this many jobs at a time on a single temp dir.
        # WAS 8 BUT TEMPORARY SET TO 16 FOR HDD VOLUME
        tmpdir_max_jobs: 16

        # Don't run more than this many jobs at a time in total.
        # WAS 16 SET TO 32 FOR HDD VOLUME
        global_max_jobs: 32
        # Don't run any jobs (across all temp dirs) more often than this, in minutes.
        # Next runtest try 165 min global stagger ;-(
        # 70 seemed to work well with nvm1 nvm2 ssd0 ssd1, currently using 40 after adding 3x hdd_volume folders
        # for my system in general, with x the amount of temp folders, this is best for m: x*m=280 or m=280/x
        # 35 seemed ok but let's double it to 70, assuming 21hours for a fully stocked queue
        # 93 mins gave equilibrium with plots made and finished with 4:0 set to 3 I had 2 plots building at all times
        # 81 mins same... no catching up as of yet
        # 78 still not really catching up... back to 70? sigh
        # 75 still not enough pressure to get up to 4 plots at all times per temp, so back to 70
        # let's play a game... 65 is catching up with the new 4 max. up to 67 before it gets too tight - or 66, sigh
        global_stagger_m: 65

        # How often the daemon wakes to consider starting a new plot job, in seconds.
        polling_time_s: 20

        # Optional: Allows the overriding of some scheduling characteristics of the
        # tmp directories specified here.
        # This contains a map of tmp directory names to attributes. If a tmp directory
        # and attribute is not listed here, the default attribute setting from the main
        # configuration will be used
        #
        # Currently support override parameters:
        #     - tmpdir_stagger_phase_major (requires tmpdir_stagger_phase_minor)
        #     - tmpdir_stagger_phase_minor (requires tmpdir_stagger_phase_major)
	#     - tmpdir_stagger_phase_limit
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger and faster than the
                # other tmp dirs and it can hold more plots than the default,
                # allowing more simultaneous plots, so they are being started
                # earlier than the global setting above.
                #"/mnt/tmp/00":
                #        tmpdir_stagger_phase_major: 1
                #        tmpdir_stagger_phase_minor: 5
                #        tmpdir_max_jobs: 5
                # Here, /mnt/tmp/03 is smaller, so a different config might be
                # to space the phase stagger further apart and only allow 2 jobs
                # to run concurrently in it
                # QUESTION HOW TO PLAY WITH THESE PHASES?? :(
                #"/mnt/tmp/03":
                #        tmpdir_stagger_phase_major: 3
                #        tmpdir_stagger_phase_minor: 1
                #        tmpdir_max_jobs: 2
                "/home/roet/nvm1":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/nvm2":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/ssd00":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/ssd01":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/4x_volume/run-11":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/4x_volume/run-22":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
#                "/mnt/4x_volume/run-33":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3
#                "/mnt/4x_volume/run-44":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3

                # The stock plotter: https://github.com/Chia-Network/chia-blockchain
                # https://www.incredigeek.com/home/install-plotman-on-ubuntu-harvester/
                # executable: /home/roet/chia-blockchain/venv/bin
                k: 33                # k-size of plot, leave at 32 most of the time
                e: False             # Use -e plotting option
                n_threads: 4         # Threads per job
                n_buckets: 64       # Number of buckets to split data into default 128 smaller is more ram less wear
                job_buffer: 7400     # 3389 k32 #7400 k33 #14800 k34 #29600 k35     # Per job memory

To upload a zip of all my logs, I need to free up some space from my 1st NVMe disk. I do that by removing the following temp files: plot-k33-2021-08-27-03-49-39e46420990d57d2ee5815573bf4905cf518e2d56ff6eff58fba43342b64b665.plot.sort
Last modification date was yesterday and it's occupying 358GB (115 items).

ram.logs.zip

Answered by altendky

Aug 28, 2021

See a proposed fix in #913.

View full answer

realdevnullius · 2021-08-28T07:26:27Z

realdevnullius
Aug 28, 2021
Author

PS: I ctrl-c'd plotman and I'll leave it plotting for the next 24 hours to finish all plots. Up to then, I'm here for any experiments and questions needed :)

PS: looking at chiaplotgraph (ubuntu), my harvester stopped harvesting a good 3,5 hours ago; I think that's when the first NVMe ran out of space.

0 replies

altendky · 2021-08-28T12:57:06Z

altendky
Aug 28, 2021
Collaborator

I guess there are three scenarios here. The one where you manually kill jobs and you are expected to clean them up. The one where you ask plotman to kill jobs and it is expected to clean up the tmp files. The one where you leave plotman running without killing anything and the plotter itself is expected to clean up the tmp files.

The one specifically relevant to existing plotman features is where you asked plotman to kill a job and it failed to find any tmp files. I guess for that we would want to see the plotman kill output along with the command/prompt/output of lsof -p 1234 (replacing 1234 with the process id (PID) of the relevant plotter process), and the prompt/command/output of ls /path/to/the/tmp/for/this/plotter/process both before and after the plotman kill.

Side note, please don't snippet the configuration file. Just share the entire thing.

2 replies

realdevnullius Aug 28, 2021
Author

I'm on it. For the record: currently, there was no killing on my end - not for 8 days. But let's indeed get started with the basics here :)

I'll start plotman interactive and have it start a new job, kill that one.

altendky Aug 28, 2021
Collaborator

tmp files not being removed when plotting processes are completing successful sounds like an issue with the plotter itself. You may also want to report that to the chia repository. Try to recreate it without using plotman, just calling the plotter directly on a clean disk and show files present after completion.

Now, on to reading the rest below.

realdevnullius · 2021-08-28T15:09:51Z

realdevnullius
Aug 28, 2021
Author

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman kill 491f72b7
Namespace(cmd='kill', force=False, idprefix=['491f72b7'])
Pausing PID 444789, plot id 491f72b7153372178f3ba3b80c6ab802fcc8c49164118a13944f81ea5d0d1fb3
Will kill pid 444789, plot id 491f72b7153372178f3ba3b80c6ab802fcc8c49164118a13944f81ea5d0d1fb3
Will delete 0 temp files

This is for the newly started task "491f72b7".

If I do the same for the nvme1 plot still running, I get the same result:

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman kill 64831721
Namespace(cmd='kill', force=False, idprefix=['64831721'])
Pausing PID 870151, plot id 64831721dba65b3153b87f6e229bd67175ca79d0f4a57435914860927a0130a2
Will kill pid 870151, plot id 64831721dba65b3153b87f6e229bd67175ca79d0f4a57435914860927a0130a2
Will delete 0 temp files
Are you sure? ("y" to confirm):

And while it pains me, I'll kill it next and upload the log files you requested.

Are you sure? ("y" to confirm): y
killing...
cleaning up temp files...

Both processes indeed disappear from plotman status. I'll check on the status of the temp files next.

1 reply

realdevnullius Aug 28, 2021
Author

for lsof output, I had plotman interactive start another plot (on nvme1) and gave the command. This is the output (plotting was underway for a minute or so):

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ lsof -p 491202
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/082f3cb54491
      Output information may be incomplete.
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
chia    491202 roet  cwd    DIR   0,51       50  11076 /home/roet/chia-blockchain
chia    491202 roet  rtd    DIR   0,27       26     34 /
chia    491202 roet  txt    REG   0,27  5490352 656004 /usr/bin/python3.8
chia    491202 roet  mem    REG   0,51  1958984 241388 /home/roet/chia-blockchain/venv/lib/python3.8/site-packages/clvm_rs.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,51   797720 241426 /home/roet/chia-blockchain/venv/lib/python3.8/site-packages/chiapos.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27   104984 640472 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
chia    491202 roet  mem    REG   0,27  1956992 639224 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
chia    491202 roet  mem    REG   0,51   757288 231668 /home/roet/chia-blockchain/venv/lib/python3.8/site-packages/blspy.libs/libgmp-0b97008e.so.10.4.1
chia    491202 roet  mem    REG   0,51  1269192 231602 /home/roet/chia-blockchain/venv/lib/python3.8/site-packages/blspy.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27  2954080 646772 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
chia    491202 roet  mem    REG   0,51  2342916 217988 /home/roet/chia-blockchain/venv/lib/python3.8/site-packages/yaml/_yaml.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    40040 639820 /usr/lib/x86_64-linux-gnu/librt-2.31.so
chia    491202 roet  mem    REG   0,27    31072 657572 /usr/lib/python3.8/lib-dynload/termios.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27   162264 631678 /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.4
chia    491202 roet  mem    REG   0,27    32696 657560 /usr/lib/python3.8/lib-dynload/mmap.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    74848 639752 /usr/lib/x86_64-linux-gnu/libbz2.so.1.0.4
chia    491202 roet  mem    REG   0,27    78088 657526 /usr/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    41544 655834 /usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    19632 657538 /usr/lib/python3.8/lib-dynload/_queue.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    45832 657530 /usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27 14542048  29478 /usr/lib/locale/locale-archive
chia    491202 roet  mem    REG   0,27   108936 638716 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
chia    491202 roet  mem    REG   0,27   182560 655978 /usr/lib/x86_64-linux-gnu/libexpat.so.1.6.11
chia    491202 roet  mem    REG   0,27  1369352 639794 /usr/lib/x86_64-linux-gnu/libm-2.31.so
chia    491202 roet  mem    REG   0,27    14848 639824 /usr/lib/x86_64-linux-gnu/libutil-2.31.so
chia    491202 roet  mem    REG   0,27    18816 639792 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
chia    491202 roet  mem    REG   0,27   157224 639816 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
chia    491202 roet  mem    REG   0,27  2029224 639790 /usr/lib/x86_64-linux-gnu/libc-2.31.so
chia    491202 roet  mem    REG   0,27    14672 655836 /usr/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    28520 657496 /usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so
chia    491202 roet  mem    REG   0,27    27002 640324 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
chia    491202 roet  mem    REG   0,27   191472 639782 /usr/lib/x86_64-linux-gnu/ld-2.31.so
chia    491202 roet    0u   CHR  136,1      0t0      4 /dev/pts/1
chia    491202 roet    1w   REG   0,64     1166    157 /mnt/ram/2021-08-28T17_14_07.444625+02_00.plot.log
chia    491202 roet    2w   REG   0,64     1166    157 /mnt/ram/2021-08-28T17_14_07.444625+02_00.plot.log
chia    491202 roet    3r   CHR    1,9      0t0     10 /dev/urandom
chia    491202 roet    4u   REG   0,51        0 344510 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.sort.tmp
chia    491202 roet    5u   REG   0,51        0 344512 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table1.tmp
chia    491202 roet    6u   REG   0,51        0 344514 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table2.tmp
chia    491202 roet    7u   REG   0,51        0 344516 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table3.tmp
chia    491202 roet    8u   REG   0,51        0 344518 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table4.tmp
chia    491202 roet    9u   REG   0,51        0 344520 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table5.tmp
chia    491202 roet   10u   REG   0,51        0 344522 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table6.tmp
chia    491202 roet   11u   REG   0,51        0 344524 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.table7.tmp
chia    491202 roet   12u   REG  8,145        0   2059 /mnt/4x_volume/tmp.02/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.2.tmp
chia    491202 roet   13u   REG   0,51 59760640 344526 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_000.tmp
chia    491202 roet   14u   REG   0,51 59760640 344528 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_001.tmp
chia    491202 roet   15u   REG   0,51 59760640 344530 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_002.tmp
chia    491202 roet   16u   REG   0,51 59760640 344532 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_003.tmp
chia    491202 roet   17u   REG   0,51 59760640 344534 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_004.tmp
chia    491202 roet   18u   REG   0,51 59760640 344536 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_005.tmp
chia    491202 roet   19u   REG   0,51 59760640 344538 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_006.tmp
chia    491202 roet   20u   REG   0,51 59760640 344540 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_007.tmp
chia    491202 roet   21u   REG   0,51 59760640 344542 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_008.tmp
chia    491202 roet   22u   REG   0,51 59760640 344544 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_009.tmp
chia    491202 roet   23u   REG   0,51 59760640 344546 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_010.tmp
chia    491202 roet   24u   REG   0,51 59760640 344548 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_011.tmp
chia    491202 roet   25u   REG   0,51 59760640 347086 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_012.tmp
chia    491202 roet   26u   REG   0,51 59760640 344550 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_013.tmp
chia    491202 roet   27u   REG   0,51 59760640 344552 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_014.tmp
chia    491202 roet   28u   REG   0,51 59760640 344554 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_015.tmp
chia    491202 roet   29u   REG   0,51 59760640 344556 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_016.tmp
chia    491202 roet   30u   REG   0,51 59760640 344558 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_017.tmp
chia    491202 roet   31u   REG   0,51 59760640 344560 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_018.tmp
chia    491202 roet   32u   REG   0,51 59760640 344562 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_019.tmp
chia    491202 roet   33u   REG   0,51 59760640 344564 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_020.tmp
chia    491202 roet   34u   REG   0,51 59760640 344566 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_021.tmp
chia    491202 roet   35u   REG   0,51 59760640 347088 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_022.tmp
chia    491202 roet   36u   REG   0,51 59760640 347090 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_023.tmp
chia    491202 roet   37u   REG   0,51 59760640 347092 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_024.tmp
chia    491202 roet   38u   REG   0,51 59760640 347094 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_025.tmp
chia    491202 roet   39u   REG   0,51 59760640 347096 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_026.tmp
chia    491202 roet   40u   REG   0,51 59760640 347098 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_027.tmp
chia    491202 roet   41u   REG   0,51 59760640 347100 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_028.tmp
chia    491202 roet   42u   REG   0,51 59760640 347102 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_029.tmp
chia    491202 roet   43u   REG   0,51 59760640 347104 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_030.tmp
chia    491202 roet   44u   REG   0,51 59760640 344568 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_031.tmp
chia    491202 roet   45u   REG   0,51 59760640 344570 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_032.tmp
chia    491202 roet   46u   REG   0,51 59760640 344572 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_033.tmp
chia    491202 roet   47u   REG   0,51 59760640 344574 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_034.tmp
chia    491202 roet   48u   REG   0,51 59760640 357888 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_035.tmp
chia    491202 roet   49u   REG   0,51 59760640 357890 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_036.tmp
chia    491202 roet   50u   REG   0,51 59760640 357892 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_037.tmp
chia    491202 roet   51u   REG   0,51 59760640 357894 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_038.tmp
chia    491202 roet   52u   REG   0,51 59760640 357896 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_039.tmp
chia    491202 roet   53u   REG   0,51 59760640 357898 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_040.tmp
chia    491202 roet   54u   REG   0,51 59760640 357900 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_041.tmp
chia    491202 roet   55u   REG   0,51 59760640 357902 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_042.tmp
chia    491202 roet   56u   REG   0,51 59760640 357904 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_043.tmp
chia    491202 roet   57u   REG   0,51 59760640 357906 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_044.tmp
chia    491202 roet   58u   REG   0,51 59760640 357908 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_045.tmp
chia    491202 roet   59u   REG   0,51 59760640 357910 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_046.tmp
chia    491202 roet   60u   REG   0,51 59760640 357912 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_047.tmp
chia    491202 roet   61u   REG   0,51 59760640 357914 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_048.tmp
chia    491202 roet   62u   REG   0,51 59760640 357916 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_049.tmp
chia    491202 roet   63u   REG   0,51 59760640 357918 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_050.tmp
chia    491202 roet   64u   REG   0,51 59760640 357920 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_051.tmp
chia    491202 roet   65u   REG   0,51 59760640 357922 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_052.tmp
chia    491202 roet   66u   REG   0,51 59760640 357924 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_053.tmp
chia    491202 roet   67u   REG   0,51 59760640 357926 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_054.tmp
chia    491202 roet   68u   REG   0,51 59760640 357928 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_055.tmp
chia    491202 roet   69u   REG   0,51 59760640 357930 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_056.tmp
chia    491202 roet   70u   REG   0,51 59760640 357932 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_057.tmp
chia    491202 roet   71u   REG   0,51 59760640 357934 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_058.tmp
chia    491202 roet   72u   REG   0,51 59760640 357936 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_059.tmp
chia    491202 roet   73u   REG   0,51 59760640 357938 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_060.tmp
chia    491202 roet   74u   REG   0,51 59760640 357940 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_061.tmp
chia    491202 roet   75u   REG   0,51 59760640 357942 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_062.tmp
chia    491202 roet   76u   REG   0,51 59760640 357944 /home/roet/nvm1/plot-k33-2021-08-28-17-14-e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90.plot.p1.t1.sort_bucket_063.tmp

and

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman kill e297043f
Namespace(cmd='kill', force=False, idprefix=['e297043f'])
Pausing PID 491202, plot id e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90
Will kill pid 491202, plot id e297043f20b11c443127c10536c0d12499ec1930b3ee2638e942f75ce554db90
Will delete 0 temp files
Are you sure? ("y" to confirm): y
killing...
cleaning up temp files...
(venv) $

realdevnullius · 2021-08-28T15:32:40Z

realdevnullius
Aug 28, 2021
Author

original output too long, so I put it here: https://pastebin.com/Nevi8QTD

I'll re-do it all, but this time with all temp files first removed (from nvm1 tmp). Then I'll start plotman interactive, wait a minute, kill the newly started plot and give you the ls output of THAT. I'm pretty sure that'll be more readable anyways :)

0 replies

realdevnullius · 2021-08-28T15:37:47Z

realdevnullius
Aug 28, 2021
Author

Short answer: test shows temp files are not removed after plotman kill. Output:

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ ls ~/nvm1/*
ls: cannot access '/home/roet/nvm1/*': No such file or directory
(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman interactive
Traceback (most recent call last):
  File "/home/roet/chia-blockchain/venv/bin/plotman", line 8, in <module>
(SNIP)
  File "/usr/lib/python3.8/threading.py", line 81, in RLock
    def RLock(*args, **kwargs):
KeyboardInterrupt

(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ !ls
ls ~/nvm1/*
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_000.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_001.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_002.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_003.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_004.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_005.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_006.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_007.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_008.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_009.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_010.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_011.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_012.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_013.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_014.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_015.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_016.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_017.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_018.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_019.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_020.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_021.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_022.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_023.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_024.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_025.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_026.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_027.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_028.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_029.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_030.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_031.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_032.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_033.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_034.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_035.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_036.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_037.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_038.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_039.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_040.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_041.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_042.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_043.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_044.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_045.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_046.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_047.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_048.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_049.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_050.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_051.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_052.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_053.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_054.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_055.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_056.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_057.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_058.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_059.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_060.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_061.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_062.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_063.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.sort.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table1.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table2.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table3.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table4.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table5.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table6.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table7.tmp
(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman status
 plot id   plotter    k                     tmp                                       dst    wall   phase    tmp       pid   stat       mem    user    sys     io
6f6b4133      chia   33         /home/roet/nvm1   /mnt/farm/HDD02/Plots_OK/pooled/plotman    0:02     1:1    19G    568726    SLP   417.3Mi    0:02    54s     0s
2e9d84d9      chia   33   /mnt/4x_volume/run-11   /mnt/farm/HDD05/Plots_OK/pooled/plotman    9:32     2:3   405G    946208    RUN     2.5Gi   13:47   0:52   0:08
ba1a86c8      chia   33              /mnt/ssd01   /mnt/farm/HDD00/Plots_OK/pooled/plotman   10:37     2:4   426G   3039185    RUN     2.3Gi   14:40   0:55   0:26
3bdf4eb9      chia   33               /mnt/nvm2   /mnt/farm/HDD03/Plots_OK/pooled/plotman   11:42     3:1   480G   1981943    DSK     8.6Gi   15:50   1:03   0:27
ce82dddd      chia   33              /mnt/ssd00   /mnt/farm/HDD02/Plots_OK/pooled/plotman   12:48     3:2   386G    700804    RUN     7.6Gi   17:31   1:10   0:10
723d5ae2      chia   33   /mnt/4x_volume/run-22   /mnt/farm/HDD01/Plots_OK/pooled/plotman   13:53     3:2   403G   3261258    RUN     7.5Gi   17:33   1:06   0:24
f4e4493a      chia   33   /mnt/4x_volume/run-11   /mnt/farm/HDD00/Plots_OK/pooled/plotman   16:03     3:3   325G    907516    RUN     7.5Gi   19:57   1:20   0:30
d0e616e0      chia   33              /mnt/ssd01   /mnt/farm/HDD03/Plots_OK/pooled/plotman   17:08     3:3   306G    118837    RUN     7.6Gi   20:36   1:33   0:56
b024143c      chia   33   /mnt/4x_volume/run-22   /mnt/farm/HDD00/Plots_OK/pooled/plotman   20:24     3:5   155G   1830876    RUN     7.6Gi   23:52   1:43   1:00
35e6d258      chia   33   /mnt/4x_volume/run-11   /mnt/farm/HDD03/Plots_OK/pooled/plotman   22:34     4:0   115G      5397    RUN     7.5Gi   25:32   1:50   1:12
6023a389      chia   33              /mnt/ssd01   /mnt/farm/HDD01/Plots_OK/pooled/plotman   23:39     4:0      0   3374109    SLP   357.8Mi   26:04   2:13   1:20

Total jobs: 11
Jobs in /home/roet/nvm1: 1
Jobs in /mnt/4x_volume/run-11: 3
Jobs in /mnt/4x_volume/run-22: 2
Jobs in /mnt/nvm2: 1
Jobs in /mnt/ssd00: 1
Jobs in /mnt/ssd01: 3

Updated at: Sat Aug 28 17:36:31 2021
(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ plotman kill 6f6b4133
Namespace(cmd='kill', force=False, idprefix=['6f6b4133'])
Pausing PID 568726, plot id 6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c
Will kill pid 568726, plot id 6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c
Will delete 0 temp files
Are you sure? ("y" to confirm): y
killing...
cleaning up temp files...
(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$ !ls
ls ~/nvm1/*
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_000.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_001.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_002.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_003.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_004.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_005.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_006.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_007.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_008.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_009.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_010.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_011.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_012.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_013.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_014.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_015.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_016.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_017.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_018.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_019.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_020.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_021.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_022.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_023.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_024.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_025.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_026.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_027.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_028.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_029.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_030.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_031.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_032.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_033.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_034.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_035.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_036.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_037.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_038.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_039.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_040.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_041.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_042.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_043.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_044.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_045.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_046.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_047.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_048.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_049.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_050.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_051.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_052.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_053.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_054.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_055.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_056.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_057.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_058.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_059.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_060.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_061.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_062.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.p1.t1.sort_bucket_063.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.sort.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table1.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table2.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table3.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table4.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table5.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table6.tmp
/home/roet/nvm1/plot-k33-2021-08-28-17-34-6f6b41337ea7e20b358872731fdf43f7fa70df70254bdd657a854d6f40aa553c.plot.table7.tmp
(venv) roet@ProLiant-DL380-Gen9:~/chia-blockchain$

0 replies

realdevnullius · 2021-08-28T15:41:04Z

realdevnullius
Aug 28, 2021
Author

current plotman.yaml:

# Default/example plotman.yaml configuration file
# k temp size calculations on https://plot-plan.chia.foxypool.io/

# https://github.com/ericaltendorf/plotman/wiki/Configuration#versions
version: [2]

logging:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        # sudo mount -t tmpfs -o size=20M tmpfs /mnt/ram/
        #        plots: /home/chia/chia/logs
        plots: /mnt/ram/
        transfers: /home/roet/plotman/log.transfer/
        application: /home/roet/plotman/log.app/plotman.log

# Options for display and rendering
user_interface:
        # Call out to the `stty` program to determine terminal size, instead of
        # relying on what is reported by the curses library.   In some cases,
        # the curses library fails to update on SIGWINCH signals.  If the
        # `plotman interactive` curses interface does not properly adjust when
        # you resize the terminal window, you can try setting this to True. 
        use_stty_size: True

# Optional custom settings for the subcommands (status, interactive etc)
commands:
        interactive:
                # Set it to False if you don't want to auto start plotting when 'interactive' is ran.
                # You can override this value from the command line, type "plotman interactive -h" for details
                autostart_plotting: True
                autostart_archiving: True

# Where to plot and log.
directories:
        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /mnt/nvm2
                - /mnt/ssd01
                - /mnt/4x_volume/run-22
                - /mnt/ssd00
                - /home/roet/nvm1
                - /mnt/4x_volume/run-11

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a
        # /home/roet is on nvme01
        # tmp2: /home/roet/plots.tmp-2/plotman
        tmp2: /mnt/4x_volume/tmp.02

        # Optional: A list of one or more directories; the scheduler will
        # use all of them.  These again are presumed to be on independent
        # physical devices so writes (plot jobs) and reads (archivals) can
        # be scheduled to minimize IO contention.
        # 
        # If dst is commented out, the tmp directories will be used as the
        # buffer.
        dst:
                - /mnt/farm/HDD00/Plots_OK/pooled/plotman
                - /mnt/farm/HDD01/Plots_OK/pooled/plotman
                - /mnt/farm/HDD02/Plots_OK/pooled/plotman
                - /mnt/farm/HDD03/Plots_OK/pooled/plotman
                - /mnt/farm/HDD05/Plots_OK/pooled/plotman

# Archival configuration.  Optional; if you do not wish to run the
# archiving operation, comment this section out.  Almost everyone
# should be using the archival feature.  It is meant to distribute
# plots among multiple disks filling them all.  This can be done both
# to local and to remote disks.
#
# As of v0.4, archiving commands are highly configurable.  The basic
# configuration consists of a script for checking available disk space
# and another for actually transferring plots.  Each can be specified
# as either a path to an existing script or inline script contents.
# It is expected that most people will use existing recipes and will
# adjust them by specifying environment variables that will set their
# system specific values.  These can be provided to the scripts via
# the `env` key.  plotman will additionally provide `source` and
# `destination` environment variables to the transfer script so it
# knows the specifically selected items to process.  plotman also needs
# to be able to generally detect if a transfer process is already
# running.  To be able to identify externally launched transfers, the
# process name and an argument prefix to match must be provided.  Note
# that variable substitution of environment variables including those
# specified in the env key can be used in both process name and process
# argument prefix elements but that they use the python substitution
# format.
#
# Complete example: https://github.com/ericaltendorf/plotman/wiki/Archiving
#archiving:
#  target: local_rsync
#  env:
#    command: rsync
#    site_root: /mnt/farm

# Plotting scheduling parameters
scheduling:
        # Run a job on a particular temp dir only if the number of existing jobs
        # before [tmpdir_stagger_phase_major : tmpdir_stagger_phase_minor]
        # is less than tmpdir_stagger_phase_limit.
        # Phase major corresponds to the plot phase, phase minor corresponds to
        # the table or table pair in sequence, phase limit corresponds to
        # the number of plots allowed before [phase major : phase minor].
        # e.g, with default settings, a new plot will start only when your plot
        # reaches phase [2 : 1] on your temp drive. This setting takes precidence
        # over global_stagger_m
        # LIMIT WAS 8 TEMPORARILY TO 9 FOR HDD_VOLUME
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 9

        # Don't run more than this many jobs at a time on a single temp dir.
        # WAS 8 BUT TEMPORARY SET TO 16 FOR HDD VOLUME
        tmpdir_max_jobs: 16

        # Don't run more than this many jobs at a time in total.
        # WAS 16 SET TO 32 FOR HDD VOLUME
        global_max_jobs: 32

        # Don't run any jobs (across all temp dirs) more often than this, in minutes.
        # Next runtest try 165 min global stagger ;-( 
        # 70 seemed to work well with nvm1 nvm2 ssd0 ssd1, currently using 40 after adding 3x hdd_volume folders
        # for my system in general, with x the amount of temp folders, this is best for m: x*m=280 or m=280/x
        # 35 seemed ok but let's double it to 70, assuming 21hours for a fully stocked queue
        # 93 mins gave equilibrium with plots made and finished with 4:0 set to 3 I had 2 plots building at all times
        # 81 mins same... no catching up as of yet
        # 78 still not really catching up... back to 70? sigh
        # 75 still not enough pressure to get up to 4 plots at all times per temp, so back to 70
        # let's play a game... 65 is catching up with the new 4 max. up to 67 before it gets too tight - or 66, sigh
        global_stagger_m: 65

        # How often the daemon wakes to consider starting a new plot job, in seconds.
        polling_time_s: 20

        # Optional: Allows the overriding of some scheduling characteristics of the
        # tmp directories specified here.
        # This contains a map of tmp directory names to attributes. If a tmp directory 
        # and attribute is not listed here, the default attribute setting from the main 
        # configuration will be used
        #
        # Currently support override parameters:
        #     - tmpdir_stagger_phase_major (requires tmpdir_stagger_phase_minor)
        #     - tmpdir_stagger_phase_minor (requires tmpdir_stagger_phase_major)
        #     - tmpdir_stagger_phase_limit
        #     - tmpdir_max_jobs
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger and faster than the
                # other tmp dirs and it can hold more plots than the default,
                # allowing more simultaneous plots, so they are being started
                # earlier than the global setting above.
                #"/mnt/tmp/00":
                #        tmpdir_stagger_phase_major: 1
                #        tmpdir_stagger_phase_minor: 5
                #        tmpdir_max_jobs: 5
                # Here, /mnt/tmp/03 is smaller, so a different config might be
                # to space the phase stagger further apart and only allow 2 jobs
                # to run concurrently in it
                # QUESTION HOW TO PLAY WITH THESE PHASES?? :(
                #"/mnt/tmp/03":
                #        tmpdir_stagger_phase_major: 3
                #        tmpdir_stagger_phase_minor: 1
                #        tmpdir_max_jobs: 2
                "/home/roet/nvm1":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/nvm2":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/ssd00":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/ssd01":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/4x_volume/run-11":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/4x_volume/run-22":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
#                "/mnt/4x_volume/run-33":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3
#                "/mnt/4x_volume/run-44":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3

# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        # Your public keys.  Be sure to use the pool contract address for
        # portable pool plots.  The pool public key is only for original
        # non-portable plots that can not be used with the official pooling
        # protocol.
        farmer_pk: a06153cc93227662742954c316c14a61b2cb071c45accbb1706953f6b50555d523760f2cc885dc456e019aa507b8dc63
        # pool_pk: ...
        pool_contract_address: xch1fx6n53h2zlwchylezxn0d6dwp9655gsxdc3ez0h9u4sqpemwqnhq958pru
        
        # If you enable Chia, plot in *parallel* with higher tmpdir_max_jobs and global_max_jobs
        type: chia
        chia:
                # The stock plotter: https://github.com/Chia-Network/chia-blockchain
                # https://www.incredigeek.com/home/install-plotman-on-ubuntu-harvester/
                # executable: /home/roet/chia-blockchain/venv/bin
                k: 33                # k-size of plot, leave at 32 most of the time
                e: False             # Use -e plotting option
                n_threads: 4         # Threads per job
                n_buckets: 64       # Number of buckets to split data into default 128 smaller is more ram less wear
                job_buffer: 7400     # 3389 k32 #7400 k33 #14800 k34 #29600 k35     # Per job memory

        # If you enable madMAx, plot in *sequence* with very low tmpdir_max_jobs and global_max_jobs
        madmax:
                # madMAx plotter: https://github.com/madMAx43v3r/chia-plotter
                # executable: /path/to/chia_plot
                n_threads: 4          # Default is 4, crank up if you have many cores
                n_buckets: 256        # Default is 256

0 replies

altendky · 2021-08-28T19:40:46Z

altendky
Aug 28, 2021
Collaborator

See a proposed fix in #913.

3 replies

realdevnullius Aug 28, 2021
Author

Thank you, I'm gonna google how to update to a commit ID :)

PS: it surprises me you found some bugs in naming patterns? How comes nobody was mentioning this before? I can hardly have been the only one having problems like this, no? :)

realdevnullius Aug 28, 2021
Author

Update chia:
https://github.com/Chia-Network/chia-blockchain/wiki/INSTALL#ubuntudebian

git pull commit? https://devopscube.com/checkout-clone-specific-git-commit-id-sha/

Will be for tomorrow cause somehow my ubuntu got fracked and has a lot of trouble installing anything. Great timing, all in all, and it gives me the chance to pool my 2x nvme and my 2x ssd :)

Speak you soon - appreciated how you stepped up!

altendky Aug 28, 2021
Collaborator

I think I had had this problem but with everything I'm trying to get done I can't always dig into every last thing the moment I find it.

You should be able to update using the regular plotman installation instructions https://github.com/ericaltendorf/plotman#installation but use altendky/plotman@fixup_tmp_cleanup instead of ericaltendorf/plotman@main.

realdevnullius · 2021-08-28T20:15:51Z

realdevnullius
Aug 28, 2021
Author

For now, 1 question remains though... How comes that after so many days of successful plotting (8), only now suddenly temp files appear to have gotten forgotten? I'll repost in 10 days if the problem occurs once again :)

4 replies

altendky Aug 28, 2021
Collaborator

As mentioned, there are multiple different issues that you shared. The given fix is for plotman kill not cleaning up tmp files. If the plotters are leaving behind tmp files on their own, that's separate.

altendky Aug 28, 2021
Collaborator

*separate and related to the plotter itself.

realdevnullius Aug 29, 2021
Author

so if it goes wrong again, I'm sheit out of luck? :)

altendky Aug 29, 2021
Collaborator

#912 (reply in thread)

realdevnullius · 2021-08-29T07:50:43Z

realdevnullius
Aug 29, 2021
Author

Note to self: SSD0 had temp files left after all plotting was done - not too many, but still.
SSD1 was completely empty. nvm2 had a few more temp files than ssd0 but also not too extreme. Still. Problems appear to have arisen Aug 27th, in the evening.

0 replies

realdevnullius · 2021-09-06T19:41:56Z

realdevnullius
Sep 6, 2021
Author

Just to add to the original question - which appears to be out of the scope of plotman itself - I had the bug re-appear once more.

With a global stagger of 68 minutes I was running smoothly for a few days so I decided to make my global stagger 66 minutes, to see if I could speed things up.

I noticed that all my disks often were 100% busy, indicating that the limits of my hardware were reached. This is also when I suddenly got the bug again when plotman does not know the plot IDs, NOR do the known plots ever finish (despite hard disk activity). After noticing the bug, I closed plotman for half a day. This is the output I got when I started plotman interactive again:

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status
 plot id   plotter    k                      tmp                                       dst    wall   phase    tmp       pid   stat       mem    user    sys     io
--------             33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD01/Plots_OK/pooled/plotman     27s     0:0      0   1556263    SLP   417.6Mi     27s     9s     0s
--------             33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD05/Plots_OK/pooled/plotman   13:45     0:0      0   1188066    RUN     7.5Gi   19:04   1:13   0:02
--------             33     /home/roet/tmp/ssd01   /mnt/farm/HDD03/Plots_OK/pooled/plotman   14:51     0:0      0   1158187    RUN     7.5Gi   20:06   1:16   0:01
--------             33              /mnt/ssd03/   /mnt/farm/HDD06/Plots_OK/pooled/plotman   15:57     0:0      0   1126747    RUN     7.6Gi   20:45   1:21   0:14
--------             33   /mnt/nvme_vol/nvme_0-1   /mnt/farm/HDD02/Plots_OK/pooled/plotman   17:04     0:0      0   1096748    RUN     7.5Gi   22:12   1:32   0:03
--------             33     /home/roet/tmp/ssd00   /mnt/farm/HDD07/Plots_OK/pooled/plotman   18:10     0:0      0   1065744    RUN     7.5Gi   23:30   1:35   0:04
--------             33    /mnt/4x_volume/run-22   /mnt/farm/HDD00/Plots_OK/pooled/plotman   19:16     0:0      0   1034561    RUN     7.6Gi   24:32   1:30   0:06
bfbf6b33      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD01/Plots_OK/pooled/plotman   20:22     1:3   163G   1003098    RUN     7.6Gi   25:40   1:43   0:05
1f7b3843      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD03/Plots_OK/pooled/plotman   21:28     1:4      0    971358    RUN   354.8Mi   26:14   1:50   0:05
b3f7b53f      chia   33              /mnt/ssd03/   /mnt/farm/HDD05/Plots_OK/pooled/plotman   22:35     1:4      0    939971    SLP   354.8Mi   26:17   2:06   0:27

Total jobs: 10
Jobs in /home/roet/tmp/ssd00: 1
Jobs in /home/roet/tmp/ssd01: 2
Jobs in /mnt/4x_volume/run-22: 1
Jobs in /mnt/nvme_vol/nvme_0-0: 3
Jobs in /mnt/nvme_vol/nvme_0-1: 1
Jobs in /mnt/ssd03/: 2

Updated at: Mon Sep  6 21:24:37 2021

Existing and identified plots-in-the-making do have a recent last modified time. But as you can see, no progress is being made.

I suspect it's due to NVMe & SSD maxing out their bus bandwidths. Global stagger of 67 minutes, here I come :)

Edit: I had disk backlog warnings from netstat of 6000ms and more! ;-(

5 replies

realdevnullius Sep 6, 2021
Author

Weird, I did a killall chia, I cleaned up the temps and started anew, but something is still messed up apparently :(

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status
 plot id   plotter    k                      tmp                                       dst   wall   phase   tmp       pid   stat       mem   user   sys   io
--------             33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD01/Plots_OK/pooled/plotman    31s     0:0     0   1567681    SLP   417.7Mi    31s   11s   0s

I'll remove all logfiles and try the cleanup once more :(

Update: log folder was full :) Might have explained all the trouble here.. I'll re-test 66 minutes!

altendky Sep 6, 2021
Collaborator

Yes, not being able to output to the log files could cause the processes to get held up, potentially, and would result in plotman not having log info from which to get the plot IDs etc.

realdevnullius Sep 14, 2021
Author

Existing and identified plots-in-the-making do have a recent last modified time. But as you can see, no progress is being made.

I suspect it's due to NVMe & SSD maxing out their bus bandwidths. Global stagger of 67 minutes, here I come :)

Edit: I had disk backlog warnings from netstat of 6000ms and more! ;-(

The Gremlins have re-apppeared. Per original problem: plots are being forgotten about after running smoothly for about a week? plotman was also unable to clean up the temp files. I did a latest update to 0.5.2 and when its bug gets solved, I'll start plotting again, see and test if it even recognizes the temp files this time around (just in case I didn't properly update last time you fixed this problem). If it does, it's gonna be another week of plotting to see if it starts forgetting plots again :/

For now: no action needed :) I also disabled tmp2, maybe that makes a difference. But I really hate doing that and I'd rather have that dynamic tmp2-option I asked about: make tmp2 the same as the destination: not much diskspeed is needed and it takes some load away from the ssd/nvme disks. It also gives more breathing room concerning free space on these fast disks....

Peace! :)

PS: it is not an out-of-space problem for the log files: that shows itself with ------ or empty as task identifiers... Currently, all tasks are seen correctly and the underlying processes get killed but temp files are not seen and skipped.

realdevnullius Sep 16, 2021
Author

Started plotting again just now. As far as I can tell, all should be working well:

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status
 plot id   plotter    k                      tmp                                       dst   wall   phase   tmp       pid   stat       mem   user   sys   io
5921dd72      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD00/Plots_OK/pooled/plotman    44s     1:1    5G   4160461    SLP   417.6Mi    37s   19s   0s

Total jobs: 1
Jobs in /mnt/nvme_vol/nvme_0-0: 1

Updated at: Thu Sep 16 12:51:30 2021
(venv) roet@Lovecraft:~/chia-blockchain$ plotman kill 5921dd72
Namespace(cmd='kill', force=False, idprefix=['5921dd72'])
Pausing PID 4160461, plot id 5921dd7247e6a73fb8a73796ef6e085f828d6ccd372797a67f24faf73092635a
Will kill pid 4160461, plot id 5921dd7247e6a73fb8a73796ef6e085f828d6ccd372797a67f24faf73092635a
Will delete 73 temp files
Are you sure? ("y" to confirm): y
killing...
cleaning up temp files...

I'll update again in a good week :)

realdevnullius Sep 18, 2021
Author

The 3 tasks that did get recognized on ssd03 by plotman, cleaned up nicely (#912 (comment)). So that seems to work.

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status | grep [s]sd03
d6917c5d      chia   33               /mnt/ssd03   /mnt/farm/HDD06/Plots_OK/pooled/plotman    8:12     1:1    55G   1269301    SLP   417.3Mi    0:07   0:02     3s
81108a03      chia   33               /mnt/ssd03   /mnt/farm/HDD00/Plots_OK/pooled/plotman   22:19     2:4   448G    881365    SLP     2.5Gi   16:46   1:36   0:08
5e798f52      chia   33               /mnt/ssd03   /mnt/farm/HDD02/Plots_OK/pooled/plotman   45:09     5:0   224G    220364    SLP   354.8Mi   26:07   2:07   0:33
Jobs in /mnt/ssd03: 3
(venv) roet@Lovecraft:~/chia-blockchain$ kill d6917c5d
bash: kill: d6917c5d: arguments must be process or job IDs
(venv) roet@Lovecraft:~/chia-blockchain$ plotman kill d6917c5d
Namespace(cmd='kill', force=False, idprefix=['d6917c5d'])
Pausing PID 1269301, plot id d6917c5d88cbf6a4624bc75f9662fa11d130bad81cc6e79ccb3b98c7de139a24
Will kill pid 1269301, plot id d6917c5d88cbf6a4624bc75f9662fa11d130bad81cc6e79ccb3b98c7de139a24
Will delete 73 temp files
Are you sure? ("y" to confirm): y
killing...
cleaning up temp files...
(venv) roet@Lovecraft:~/chia-blockchain$ ls /mnt/ssd03/* | grep -i [d]6917c5d
(venv) roet@Lovecraft:~/chia-blockchain$

realdevnullius · 2021-09-18T16:42:41Z

realdevnullius
Sep 18, 2021
Author

Alright, that went fast this time. SSD03 already is full, 1MB left. This time there might be a good reason for it though (see #928).

But it's one thing that the disk is full because of the for-mentioned problem, it's another that plotman still reports only 3 active plots being made through ssd03...

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status | grep [s]sd03
d6917c5d      chia   33               /mnt/ssd03   /mnt/farm/HDD06/Plots_OK/pooled/plotman    7:56     1:1    55G   1269301    SLP   417.3Mi    0:07   0:02     3s
81108a03      chia   33               /mnt/ssd03   /mnt/farm/HDD00/Plots_OK/pooled/plotman   22:03     2:4   448G    881365    SLP     2.5Gi   16:46   1:36   0:08
5e798f52      chia   33               /mnt/ssd03   /mnt/farm/HDD02/Plots_OK/pooled/plotman   44:52     5:0   224G    220364    SLP   354.8Mi   26:07   2:07   0:33

But when I go to ssd03 and count how many plots still have temp files, I find at least 7 different plots being made. SEVEN! Not 3... Is this still a chia or hardware problem or can we start looking at plotman problems now?

ls -lias output for ssd03: https://pastebin.com/0tXUjvB8
ls outtput for ssd03: https://pastebin.com/Jcp9DRfJ

Config file:

(venv) roet@Lovecraft:~/chia-blockchain$ cat ~/.config/plotman/plotman.yaml
# Default/example plotman.yaml configuration file
# k temp size calculations on https://plot-plan.chia.foxypool.io/

# https://github.com/ericaltendorf/plotman/wiki/Configuration#versions
version: [2]

logging:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        # sudo mount -t tmpfs -o size=20M tmpfs /mnt/ram/
        #        plots: /home/chia/chia/logs
        plots: /mnt/ram/
        transfers: /home/roet/plotman/log.transfer/
        application: /home/roet/plotman/log.app/plotman.log
        disk_spaces: /home/roet/plotman/log.diskspaces/diskspaces.log

# Options for display and rendering
user_interface:
        # Call out to the `stty` program to determine terminal size, instead of
        # relying on what is reported by the curses library.   In some cases,
        # the curses library fails to update on SIGWINCH signals.  If the
        # `plotman interactive` curses interface does not properly adjust when
        # you resize the terminal window, you can try setting this to True. 
        use_stty_size: True

# Optional custom settings for the subcommands (status, interactive etc)
commands:
        interactive:
                # Set it to False if you don't want to auto start plotting when 'interactive' is ran.
                # You can override this value from the command line, type "plotman interactive -h" for details
                autostart_plotting: True
                autostart_archiving: True

# Where to plot and log.
directories:
        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                - /mnt/nvme_vol/nvme_0-0
                - /home/roet/tmp/ssd00
                - /mnt/ssd03/
                - /mnt/nvme_vol/nvme_0-1
                - /home/roet/tmp/ssd01
                - /mnt/4x_volume/run-22

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a
        # /home/roet is on nvme01
        # tmp2: /home/roet/plots.tmp-2/plotman
        #tmp2: /mnt/4x_volume/tmp.02 with global stagger 66 it goes well for a long time
        #then it starts forgetting about plots, slowly filling the disks in a few days time

        # Optional: A list of one or more directories; the scheduler will
        # use all of them.  These again are presumed to be on independent
        # physical devices so writes (plot jobs) and reads (archivals) can
        # be scheduled to minimize IO contention.
        # 
        # If dst is commented out, the tmp directories will be used as the
        # buffer.
        # disk full        - /mnt/farm/HDD04/Plots_OK/pooled/plotman
        dst:
                - /mnt/farm/HDD00/Plots_OK/pooled/plotman
                - /mnt/farm/HDD01/Plots_OK/pooled/plotman
                - /mnt/farm/HDD02/Plots_OK/pooled/plotman
                - /mnt/farm/HDD03/Plots_OK/pooled/plotman
                - /mnt/farm/HDD05/Plots_OK/pooled/plotman
                - /mnt/farm/HDD06/Plots_OK/pooled/plotman
                - /mnt/farm/HDD07/Plots_OK/pooled/plotman

# Archival configuration.  Optional; if you do not wish to run the
# archiving operation, comment this section out.  Almost everyone
# should be using the archival feature.  It is meant to distribute
# plots among multiple disks filling them all.  This can be done both
# to local and to remote disks.
#
# As of v0.4, archiving commands are highly configurable.  The basic
# configuration consists of a script for checking available disk space
# and another for actually transferring plots.  Each can be specified
# as either a path to an existing script or inline script contents.
# It is expected that most people will use existing recipes and will
# adjust them by specifying environment variables that will set their
# system specific values.  These can be provided to the scripts via
# the `env` key.  plotman will additionally provide `source` and
# `destination` environment variables to the transfer script so it
# knows the specifically selected items to process.  plotman also needs
# to be able to generally detect if a transfer process is already
# running.  To be able to identify externally launched transfers, the
# process name and an argument prefix to match must be provided.  Note
# that variable substitution of environment variables including those
# specified in the env key can be used in both process name and process
# argument prefix elements but that they use the python substitution
# format.
#
# Complete example: https://github.com/ericaltendorf/plotman/wiki/Archiving
#archiving:
#  target: local_rsync
#  env:
#    command: rsync
#    site_root: /mnt/farm

# Plotting scheduling parameters
scheduling:
        # Run a job on a particular temp dir only if the number of existing jobs
        # before [tmpdir_stagger_phase_major : tmpdir_stagger_phase_minor]
        # is less than tmpdir_stagger_phase_limit.
        # Phase major corresponds to the plot phase, phase minor corresponds to
        # the table or table pair in sequence, phase limit corresponds to
        # the number of plots allowed before [phase major : phase minor].
        # e.g, with default settings, a new plot will start only when your plot
        # reaches phase [2 : 1] on your temp drive. This setting takes precidence
        # over global_stagger_m
        # LIMIT WAS 8 TEMPORARILY TO 9 FOR HDD_VOLUME
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 9

        # Don't run more than this many jobs at a time on a single temp dir.
        # WAS 8 BUT TEMPORARY SET TO 16 FOR HDD VOLUME
        tmpdir_max_jobs: 16

        # Don't run more than this many jobs at a time in total.
        # WAS 16 SET TO 32 FOR HDD VOLUME
        global_max_jobs: 32

        # Don't run any jobs (across all temp dirs) more often than this, in minutes.
        # Next runtest try 165 min global stagger ;-( 
        # 70 seemed to work well with nvm1 nvm2 ssd0 ssd1, currently using 40 after adding 3x hdd_volume folders
        # for my system in general, with x the amount of temp folders, this is best for m: x*m=280 or m=280/x
        # 35 seemed ok but let's double it to 70, assuming 21hours for a fully stocked queue
        # 93 mins gave equilibrium with plots made and finished with 4:0 set to 3 I had 2 plots building at all times
        # 81 mins same... no catching up as of yet
        # 78 still not really catching up... back to 70? sigh
        # 75 still not enough pressure to get up to 4 plots at all times per temp, so back to 70
        # let's play a game... 65 is catching up with the new 4 max. up to 67 before it gets too tight - or 66, sigh 65 back to 68 after left over temp files bug
        # m68 after a few days remains stable at max 20 parallel plots - there's always a TMP READY and available. Disks may be on max, lots of 100% busy's. Testing 66 next
        # m66 causes bug with ID-less, forgotten plots that get left behind. So m67 next (after a few hours of no new plots) - BULLSHIT; RAM LOG DISK FULL IS WHAT
        # correction concerning m66 it works fine, but a bit more pressure might be ok. Mosst of the disks are at 100% use already but let's see... m64!
        # hmm m64 and I notice a forgotten plot again filling up ssd03 since yesterday 24h to 27h ago. Back to 65
        global_stagger_m: 65

        # How often the daemon wakes to consider starting a new plot job, in seconds.
        polling_time_s: 20

        # Optional: Allows the overriding of some scheduling characteristics of the
        # tmp directories specified here.
        # This contains a map of tmp directory names to attributes. If a tmp directory 
        # and attribute is not listed here, the default attribute setting from the main 
        # configuration will be used
        #
        # Currently support override parameters:
        #     - tmpdir_stagger_phase_major (requires tmpdir_stagger_phase_minor)
        #     - tmpdir_stagger_phase_minor (requires tmpdir_stagger_phase_major)
        #     - tmpdir_stagger_phase_limit
        #     - tmpdir_max_jobs
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger and faster than the
                # other tmp dirs and it can hold more plots than the default,
                # allowing more simultaneous plots, so they are being started
                # earlier than the global setting above.
                #"/mnt/tmp/00":
                #        tmpdir_stagger_phase_major: 1
                #        tmpdir_stagger_phase_minor: 5
                #        tmpdir_max_jobs: 5
                # Here, /mnt/tmp/03 is smaller, so a different config might be
                # to space the phase stagger further apart and only allow 2 jobs
                # to run concurrently in it
                # QUESTION HOW TO PLAY WITH THESE PHASES?? :(
                #"/mnt/tmp/03":
                #        tmpdir_stagger_phase_major: 3
                #        tmpdir_stagger_phase_minor: 1
                #        tmpdir_max_jobs: 2
                # - /mnt/nvme_vol/nvme_0-0
                # - /home/roet/tmp/ssd00
                # - /mnt/4x_volume/run-11
                # - /mnt/nvme_vol/nvme_0-0
                # - /home/roet/tmp/ssd01
                # - /mnt/4x_volume/run-22
                "/mnt/nvme_vol/nvme_0-0":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/home/roet/tmp/ssd00":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/nvme_vol/nvme_0-1":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/home/roet/tmp/ssd01":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/ssd03":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
                "/mnt/4x_volume/run-22":
                        tmpdir_stagger_phase_major: 3
                        tmpdir_stagger_phase_minor: 6
                        tmpdir_stagger_phase_limit: 4
#                "/mnt/4x_volume/run-33":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3
#                "/mnt/4x_volume/run-44":
#                        tmpdir_stagger_phase_major: 3
#                        tmpdir_stagger_phase_minor: 5
#                        tmpdir_stagger_phase_limit: 3

# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        # Your public keys.  Be sure to use the pool contract address for
        # portable pool plots.  The pool public key is only for original
        # non-portable plots that can not be used with the official pooling
        # protocol.
        farmer_pk: blah
        # pool_pk: ...
        pool_contract_address: blah
        
        # If you enable Chia, plot in *parallel* with higher tmpdir_max_jobs and global_max_jobs
        type: chia
        chia:
                # The stock plotter: https://github.com/Chia-Network/chia-blockchain
                # https://www.incredigeek.com/home/install-plotman-on-ubuntu-harvester/
                # executable: /home/roet/chia-blockchain/venv/bin
                k: 33                # k-size of plot, leave at 32 most of the time
                e: False             # Use -e plotting option
                n_threads: 4         # Threads per job
                n_buckets: 64       # Number of buckets to split data into default 128 smaller is more ram less wear
                job_buffer: 7400     # 3389 k32 #7400 k33 #14800 k34 #29600 k35     # Per job memory

        # If you enable madMAx, plot in *sequence* with very low tmpdir_max_jobs and global_max_jobs
        madmax:
                # madMAx plotter: https://github.com/madMAx43v3r/chia-plotter
                # executable: /path/to/chia_plot
                n_threads: 4          # Default is 4, crank up if you have many cores
                n_buckets: 256        # Default is 256
(venv) roet@Lovecraft:~/chia-blockchain$

2 replies

realdevnullius Sep 18, 2021
Author

PS:
- /mnt/nvme_vol/: 3.8TB with 2 folders with each up to 4 active tasks (8 total)
- /home/roet/: 3.8TB with 2 folders with each up to 4 active tasks (8 total)
- /mnt/ssd03/: 1.9TB with 1 folder with up to 4 active tasks

It takes about 22 to 24 hours to complete 1 plot. My temp limits hardly ever get triggered.

(venv) roet@Lovecraft:~/chia-blockchain$ plotman status
 plot id   plotter    k                      tmp                                       dst    wall   phase    tmp       pid   stat       mem    user    sys     io
9373f075      chia   33    /mnt/4x_volume/run-22   /mnt/farm/HDD05/Plots_OK/pooled/plotman    0:42     1:2   133G   1488645    SLP     7.6Gi    1:05   0:08     0s
d6883060      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD03/Plots_OK/pooled/plotman    1:47     1:3   245G   1457601    SLP     7.6Gi    2:59   0:14     4s
bf3f47ce      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD01/Plots_OK/pooled/plotman    2:52     1:4   323G   1426070    SLP     7.6Gi    4:47   0:22     4s
dc73d537      chia   33   /mnt/nvme_vol/nvme_0-1   /mnt/farm/HDD07/Plots_OK/pooled/plotman    3:57     1:4   350G   1394880    SLP     7.6Gi    6:25   0:29    11s
5183ad2f      chia   33     /home/roet/tmp/ssd00   /mnt/farm/HDD02/Plots_OK/pooled/plotman    5:02     1:5   365G   1363511    SLP     7.6Gi    8:23   0:36    10s
39ac574a      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD00/Plots_OK/pooled/plotman    6:07     1:6   369G   1332063    SLP     7.6Gi   10:07   0:42     7s
950762d0      chia   33    /mnt/4x_volume/run-22   /mnt/farm/HDD05/Plots_OK/pooled/plotman    7:12     1:7   370G   1300761    SLP     7.6Gi   12:17   0:44     7s
e7474e2d      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD03/Plots_OK/pooled/plotman    9:23     2:2   420G   1239611    RUN     2.5Gi   14:37   1:01   0:01
5afb6ab6      chia   33   /mnt/nvme_vol/nvme_0-1   /mnt/farm/HDD01/Plots_OK/pooled/plotman   10:28     2:4   441G   1208865    RUN     2.5Gi   15:44   1:03   0:06
1103f5c4      chia   33     /home/roet/tmp/ssd00   /mnt/farm/HDD07/Plots_OK/pooled/plotman   11:33     2:6   460G   1176653    RUN     2.5Gi   16:45   1:04   0:06
706e5896      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD02/Plots_OK/pooled/plotman   12:38     3:1   483G   1149127    RUN     8.6Gi   17:38   1:15   0:08
27db29dd      chia   33    /mnt/4x_volume/run-22   /mnt/farm/HDD00/Plots_OK/pooled/plotman   13:43     3:2   432G   1116264    RUN     7.5Gi   18:49   1:18   0:03
4e99dde9      chia   33   /mnt/nvme_vol/nvme_0-1   /mnt/farm/HDD05/Plots_OK/pooled/plotman   14:48     3:2   420G   1086692    RUN     7.5Gi   19:41   1:24   0:14
3fa96333      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD03/Plots_OK/pooled/plotman   16:58     3:3   387G   1035655    RUN     7.5Gi   21:09   1:53   0:09
85d1ee5f      chia   33   /mnt/nvme_vol/nvme_0-0   /mnt/farm/HDD01/Plots_OK/pooled/plotman   19:09     3:4   341G    971928    RUN     7.6Gi   22:57   2:09   0:23
bfd1a619      chia   33    /mnt/4x_volume/run-22   /mnt/farm/HDD02/Plots_OK/pooled/plotman   21:19     3:6   347G    912960    RUN     7.6Gi   25:35   2:23   0:07
81108a03      chia   33               /mnt/ssd03   /mnt/farm/HDD00/Plots_OK/pooled/plotman   22:24     2:4   448G    881365    SLP     2.5Gi   16:46   1:36   0:08
e833b95f      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD06/Plots_OK/pooled/plotman   23:30     5:0   224G    852370    SLP   355.1Mi   26:26   2:29   0:36
bfeda725      chia   33     /home/roet/tmp/ssd01   /mnt/farm/HDD02/Plots_OK/pooled/plotman   30:01     5:0   224G    664391    SLP   355.1Mi   27:12   2:32   0:48
28b92ea9      chia   33   /mnt/nvme_vol/nvme_0-1   /mnt/farm/HDD02/Plots_OK/pooled/plotman   37:37     5:0   224G    441500    SLP   354.8Mi   26:33   2:31   0:42
5e798f52      chia   33               /mnt/ssd03   /mnt/farm/HDD02/Plots_OK/pooled/plotman   45:14     5:0   224G    220364    SLP   354.8Mi   26:07   2:07   0:33

Total jobs: 21
Jobs in /home/roet/tmp/ssd00: 2
Jobs in /home/roet/tmp/ssd01: 5
Jobs in /mnt/4x_volume/run-22: 4
Jobs in /mnt/nvme_vol/nvme_0-0: 4
Jobs in /mnt/nvme_vol/nvme_0-1: 4
Jobs in /mnt/ssd03: 2

Updated at: Sat Sep 18 18:48:47 2021

realdevnullius Sep 18, 2021
Author

and after checking nvme_0-1, I found 3 more forgotten plots, totaling 699GB.

altendky · 2021-09-19T01:08:13Z

altendky
Sep 19, 2021
Collaborator

Files on a disk are not representative of a plot process running. So, having tmp files on disk for 7 different plots does not indicate that plotman is incorrectly reporting only 3 plotting processes. You have to look at processes.

At some point if you want to debug this you will probably need to start isolating things. In normal operation plotman does not clean up tmp files nor kill processes without you asking. So, if tmp files are being left around when you have not killed processes using plotman then it still doesn't seem likely to be a plotman issue. Maybe look at the logs for those plots.

Maybe just switch to madmax with a phase 1 stagger on your fastest NVMe, or RAID the NVMe if they are the size/make/model etc.

0 replies

tmp space fills up #912

realdevnullius Aug 28, 2021

Replies: 12 comments · 17 replies

realdevnullius Aug 28, 2021 Author

altendky Aug 28, 2021 Collaborator

realdevnullius Aug 28, 2021 Author

altendky Aug 28, 2021 Collaborator

realdevnullius Aug 28, 2021 Author

realdevnullius Aug 28, 2021 Author

realdevnullius Aug 28, 2021 Author

realdevnullius Aug 28, 2021 Author

realdevnullius Aug 28, 2021 Author

altendky Aug 28, 2021 Collaborator

realdevnullius Aug 28, 2021 Author

realdevnullius Aug 28, 2021 Author

altendky Aug 28, 2021 Collaborator

realdevnullius Aug 28, 2021 Author

altendky Aug 28, 2021 Collaborator

altendky Aug 28, 2021 Collaborator

realdevnullius Aug 29, 2021 Author

altendky Aug 29, 2021 Collaborator

realdevnullius Aug 29, 2021 Author

realdevnullius Sep 6, 2021 Author

realdevnullius Sep 6, 2021 Author

altendky Sep 6, 2021 Collaborator

realdevnullius Sep 14, 2021 Author

realdevnullius Sep 16, 2021 Author

realdevnullius Sep 18, 2021 Author

realdevnullius Sep 18, 2021 Author

realdevnullius Sep 18, 2021 Author

realdevnullius Sep 18, 2021 Author

altendky Sep 19, 2021 Collaborator

realdevnullius
Aug 28, 2021

Replies: 12 comments 17 replies

realdevnullius
Aug 28, 2021
Author

altendky
Aug 28, 2021
Collaborator

realdevnullius Aug 28, 2021
Author

altendky Aug 28, 2021
Collaborator

realdevnullius
Aug 28, 2021
Author

realdevnullius Aug 28, 2021
Author

realdevnullius
Aug 28, 2021
Author

realdevnullius
Aug 28, 2021
Author

realdevnullius
Aug 28, 2021
Author

altendky
Aug 28, 2021
Collaborator

realdevnullius Aug 28, 2021
Author

realdevnullius Aug 28, 2021
Author

altendky Aug 28, 2021
Collaborator

realdevnullius
Aug 28, 2021
Author

altendky Aug 28, 2021
Collaborator

altendky Aug 28, 2021
Collaborator

realdevnullius Aug 29, 2021
Author

altendky Aug 29, 2021
Collaborator

realdevnullius
Aug 29, 2021
Author

realdevnullius
Sep 6, 2021
Author

realdevnullius Sep 6, 2021
Author

altendky Sep 6, 2021
Collaborator

realdevnullius Sep 14, 2021
Author

realdevnullius Sep 16, 2021
Author

realdevnullius Sep 18, 2021
Author

realdevnullius
Sep 18, 2021
Author

realdevnullius Sep 18, 2021
Author

realdevnullius Sep 18, 2021
Author

altendky
Sep 19, 2021
Collaborator