Running standalone flux-shell as rank 1

[SteVwonder]

When running flux-shell -s -r 1, the json returned by flux_shell_get_info has rank: 0 in it.

herbein1@a87543a42648:/usr/src/t/trash-directory.t7000-shell-datastaging$ /usr/libexec/flux/flux-shell -s -r 1 -R ../data/shell/staging-R-rank1.json -j ../data/shell/staging-jobspec.json --initrc=test-initrc.lua 0
flux-shell:  INFO: Info data: {"jobid":0,"rank":0,"size":1,"ntasks":1,"service":"","jobspec":{"version":1,"resources":[{"type":"node","count":1,"with":[{"type":"slot","label":"default","count":1,"with":[{"type":"core","count":1},{"type":"storage","count":1,"label":"node-local-scratch"}]}]}],"tasks":[{"command":["hostname"],"slot":"default","count":{"per_slot":1}}],"attributes":{"system":{"duration":0},"user":{"data-staging":[{"test":true,"label":"node-local-scratch","mode":"scratch","granularity":"node","stage-in":{"file":"/usr/src/README"}}]}}},"R":{"version":1,"execution":{"R_lite":[{"rank":"1","node":"corona116","children":{"core":"0"}}],"starttime":1598506718,"expiration":1598564318},"scheduling":{"graph":{"nodes":[{"id":"3","metadata":{"type":"core","basename":"core","name":"core0","id":0,"uniq_id":3,"rank":1,"exclusive":true,"unit":"","size":1,"paths":{"containment":"/cluster0/corona116/core0"}}},{"id":"4","metadata":{"type":"storage","basename":"storage","name":"storage0","id":0,"uniq_id":4,"rank":1,"exclusive":true,"unit":"","size":1562,"properties":{"LinuxDeviceID":"259:0","mountpoint":"/p/ssd1"},"paths":{"containment":"/cluster0/corona116/storage0"}}},{"id":"1","metadata":{"type":"node","basename":"corona116","name":"corona116","id":-1,"uniq_id":1,"rank":1,"exclusive":false,"unit":"","size":1,"paths":{"containment":"/cluster0/corona116"}}},{"id":"0","metadata":{"type":"cluster","basename":"cluster","name":"cluster0","id":0,"uniq_id":0,"rank":-1,"exclusive":false,"unit":"","size":1,"paths":{"containment":"/cluster0"}}}],"edges":[{"source":"1","target":"3","metadata":{"name":{"containment":"contains"}}},{"source":"1","target":"4","metadata":{"name":{"containment":"contains"}}},{"source":"0","target":"1","metadata":{"name":{"containment":"contains"}}}]}}},"options":{"verbose":0,"standalone":true}}


herbein1@a87543a42648:/usr/src/t/trash-directory.t7000-shell-datastaging$ /usr/libexec/flux/flux-shell -s -r 0 -R ../data/shell/staging-R.json -j ../data/shell/staging-jobspec.json --initrc=test-initrc.lua 0
flux-shell:  INFO: Info data: {"jobid":0,"rank":0,"size":1,"ntasks":1,"service":"","jobspec":{"version":1,"resources":[{"type":"node","count":1,"with":[{"type":"slot","label":"default","count":1,"with":[{"type":"core","count":1},{"type":"storage","count":1,"label":"node-local-scratch"}]}]}],"tasks":[{"command":["hostname"],"slot":"default","count":{"per_slot":1}}],"attributes":{"system":{"duration":0},"user":{"data-staging":[{"test":true,"label":"node-local-scratch","mode":"scratch","granularity":"node","stage-in":{"file":"/usr/src/README"}}]}}},"R":{"version":1,"execution":{"R_lite":[{"rank":"0","node":"corona116","children":{"core":"0"}}],"starttime":1598506718,"expiration":1598564318},"scheduling":{"graph":{"nodes":[{"id":"3","metadata":{"type":"core","basename":"core","name":"core0","id":0,"uniq_id":3,"rank":0,"exclusive":true,"unit":"","size":1,"paths":{"containment":"/cluster0/corona116/core0"}}},{"id":"4","metadata":{"type":"storage","basename":"storage","name":"storage0","id":0,"uniq_id":4,"rank":0,"exclusive":true,"unit":"","size":1562,"properties":{"LinuxDeviceID":"259:0","mountpoint":"/p/ssd0"},"paths":{"containment":"/cluster0/corona116/storage0"}}},{"id":"1","metadata":{"type":"node","basename":"corona116","name":"corona116","id":-1,"uniq_id":1,"rank":0,"exclusive":false,"unit":"","size":1,"paths":{"containment":"/cluster0/corona116"}}},{"id":"0","metadata":{"type":"cluster","basename":"cluster","name":"cluster0","id":0,"uniq_id":0,"rank":-1,"exclusive":false,"unit":"","size":1,"paths":{"containment":"/cluster0"}}}],"edges":[{"source":"1","target":"3","metadata":{"name":{"containment":"contains"}}},{"source":"1","target":"4","metadata":{"name":{"containment":"contains"}}},{"source":"0","target":"1","metadata":{"name":{"containment":"contains"}}}]}}},"options":{"verbose":0,"standalone":true}}

[grondo]

The -r standalone shell option specifies the broker rank not the shell rank. To run a standalone shell as shell rank 1, you will need to provide an R that contains 2 broker ranks, and set -r to the 2nd broker rank in R.

Now, the standalone shell option is not designed nor well tested for multiple rank jobs. Things may break in strange ways. You may have to exit early from the shell via a custom initrc to avoid a hang.

For example, here's a simple, multi-rank R:

{
  "version": 1,
  "execution": {
    "R_lite": [
      {
        "rank": "0-1",
        "children": {
          "core": "0"
        }
      }
    ]
  }
}

And an initrc that prints the shell rank:

print("shell rank is: "..shell.info.rank)

If we use broker 1 with the R above, we should get shell rank 1 as well:

$ ./flux-shell  -s -j jobspec -R R  -r 1 --initrc print-shell-rank.lua 1
shell rank is: 1
flux-shell: ERROR: shell_output_write: Function not implemented
flux-shell: ERROR: shell_output_write: Function not implemented
flux-shell: ERROR: shell_output_write: Function not implemented

(note the errors since there is no shell rank 0 to which to send output messages)

We could suppress these messages by exiting from the initrc.

$ cat print-shell-rank.lua
print("shell rank is: "..shell.info.rank)
os.exit(0)
$ ./flux-shell  -s -j jobspec -R R  -r 1 --initrc print-shell-rank.lua 1
shell rank is: 1

Though there may be other side effects from doing that. You could use plugin.register to exit from any of the supported shell.* callbacks as well.

[SteVwonder]

Thanks @grondo! I was indeed getting a hang last night when passing in an R with multiple ranks. I will attempt your suggestion to have certain ranks exit early via the initrc with the understanding that "here be dragons".