Flux not detecting/scheduling GPUs despite hwloc finding them

[SteVwonder]

I'm pretty sure this is a PEBCAK, but I'm not sure what I'm doing wrong. I'm using flux-core 0.19.0. Hwloc is definitely configured with CUDA support, it detects the GPUs as CoProcs, the hwloc XML that Flux generates has the CoProcs too, and the resource.update JSON has 2 GPUs in it. Despite all of that, flux resource list shows no GPUs and any job requesting gpus fails to run due to an unsupported resource.

❯ lstopo-no-graphics| grep CoProc                                                                                                                                                                                              13:16:04 ()
          CoProc L#6 "cuda0"
          CoProc L#12 "cuda1"
❯ flux start flux hwloc topology | grep CoProc                                                                                                                                                                                 13:14:27 ()
            <info name="CoProcType" value="CUDA"/>
            <info name="CoProcType" value="CUDA"/>
❯ flux start flux dmesg                                                                                  13:13:46 ()
2020-11-25T21:13:49.224259Z broker.info[0]: wireup: 0 (complete) 0.0s
2020-11-25T21:13:49.234489Z broker.debug[0]: insmod connector-local
2020-11-25T21:13:49.234555Z broker.info[0]: wireup-complete: none->init
2020-11-25T21:13:49.238022Z connector-local.debug[0]: allow-guest-user=false
2020-11-25T21:13:49.238044Z connector-local.debug[0]: allow-root-owner=false
2020-11-25T21:13:49.297404Z broker.debug[0]: insmod content-sqlite
2020-11-25T21:13:49.300475Z broker.debug[0]: content backing store: enabled content-sqlite
2020-11-25T21:13:49.311838Z broker.debug[0]: insmod barrier
2020-11-25T21:13:49.320124Z broker.debug[0]: insmod aggregator
2020-11-25T21:13:49.354847Z broker.debug[0]: insmod kvs
2020-11-25T21:13:49.412645Z broker.debug[0]: insmod kvs-watch
2020-11-25T21:13:49.450383Z broker.debug[0]: insmod resource
2020-11-25T21:13:49.458588Z resource.debug[0]: reslog_cb: resource-init event posted
2020-11-25T21:13:49.460037Z resource.debug[0]: reslog_cb: hwloc-discover-start event posted
2020-11-25T21:13:49.493714Z broker.debug[0]: insmod cron
2020-11-25T21:13:49.511490Z broker.debug[0]: insmod job-manager
2020-11-25T21:13:49.502791Z cron.info[0]: synchronizing cron tasks to event hb
2020-11-25T21:13:49.520354Z broker.debug[0]: insmod job-info
2020-11-25T21:13:49.521037Z job-manager.info[0]: restart: 0 jobs
2020-11-25T21:13:49.521056Z job-manager.info[0]: restart: 0 running jobs
2020-11-25T21:13:49.521386Z job-manager.info[0]: restart: no checkpoint object
2020-11-25T21:13:49.521405Z job-manager.debug[0]: restart: max_jobid=0
2020-11-25T21:13:49.522163Z job-info.debug[0]: job_state_init_from_kvs: read 0 jobs
2020-11-25T21:13:49.553691Z broker.debug[0]: insmod job-ingest
2020-11-25T21:13:49.555320Z job-ingest.debug[0]: fluid ts=1ms
2020-11-25T21:13:49.590428Z broker.debug[0]: insmod job-exec
2020-11-25T21:13:49.599112Z broker.debug[0]: insmod sched-simple
2020-11-25T21:13:49.599366Z job-exec.debug[0]: using default shell path /collab/usr/global/tools/flux/toss_3_x86_64_ib/flux-c0.19.0-s0.11.0/libexec/flux/flux-shell
2020-11-25T21:13:49.600743Z sched-simple.debug[0]: service_register
2020-11-25T21:13:49.603562Z broker.info[0]: rc1.0: running /collab/usr/global/tools/flux/toss_3_x86_64_ib/flux-c0.19.0-s0.11.0/etc/flux/rc1.d/01-enclosing-instance
2020-11-25T21:13:49.662862Z aggregator.debug[0]: push: resource.hwloc.reload:0-21964: count=1 fwd_count=1 total=1
2020-11-25T21:13:49.662891Z aggregator.debug[0]: sink: resource.hwloc.reload:0-21964: count=1 total=1
2020-11-25T21:13:49.688019Z resource.debug[0]: reslog_cb: hwloc-discover-finish event posted
2020-11-25T21:13:49.688318Z sched-simple.info[0]: resource update: {"resources":{"0":{"NUMANode":2,"Package":2,"Core":16,"PU":32,"GPU":2,"coreids":"0-15","cpuset":"0-31"}},"up":"0"}
2020-11-25T21:13:49.688624Z job-manager.debug[0]: scheduler: hello
2020-11-25T21:13:49.689242Z job-manager.debug[0]: scheduler: ready single
2020-11-25T21:13:49.689540Z sched-simple.debug[0]: ready: 16 of 16 cores: rank0/core[0-15]
2020-11-25T21:13:49.756330Z broker.info[0]: rc1.0: /g/g0/herbein1/opt/packages/toss3/spack-views/daily-driver-2020-07-24/bin/zsh -c /collab/usr/global/tools/flux/toss_3_x86_64_ib/flux-c0.19.0-s0.11.0/etc/flux/rc1 Exited (rc=0) 0.5s
2020-11-25T21:13:49.756396Z broker.info[0]: rc1-success: init->run

❯ flux start flux resource list                                                                          13:13:38 ()
     STATE NNODES   NCORES    NGPUS
      free      1       16        0
 allocated      0        0        0
      down      0        0        0
❯ flux start flux mini run -g1 hostname                                                                                                                                                                                        13:14:10 ()
0.019s: job.exception type=alloc severity=0 Unsupported resource type 'gpu'
2020-11-25T21:14:26.648238Z broker.err[0]: rc2.0: flux mini run -g1 hostname Exited (rc=1) 0.4s

[SteVwonder]

If I install flux-sched v0.11.0, then the fluxion scheduler gets loaded and the problem seems to go away. So maybe this is an issue with sched-simple?

# compile and install flux-sched

❯ flux start flux module list                                                                                                                                                                                                  13:20:30 ()
Module                       Size Digest  Idle  S Service
sched-fluxion-resource   18419856 6993672    0  S
kvs                       1432328 F6B2DDB    0  S
aggregator                1014960 71F78AA    0  S
content-sqlite            1012984 1E54920    0  S content-backing,kvs-checkpoint
job-manager               1231632 1F71036    0  S
cron                      1079928 9E21CAC    0  S
kvs-watch                 1176000 0108280    0  S
sched-fluxion-qmanager    3606648 0CCB290    0  S sched
job-exec                  1176616 DCAE6C4    0  S
barrier                    998016 8939374    0  S
job-ingest                1102872 C4C8A15    0  S
resource                  1082696 1C1646F    0  S
connector-local            991768 C89FC41    0  R
job-info                  1258296 9A67C8C    0  S

herbein1@surface43 ~
❯ flux start flux resource list                                                                                                                                                                                                13:20:36 ()
     STATE NNODES   NCORES    NGPUS
      free      1       16        2
 allocated      0        0        0
      down      0        0        0

[grondo]

sched-simple only supports cores, so this is expected behavior. After v0.19.0, the GPU resources will show up in flux resource list though they won't be schedulable by sched-simple.

[SteVwonder]

Ok. Thanks!