microk8s enable gpu doesnt work

[raulo987]

When using: microk89s enable gpu and deploying + opening CVAT 1.6, new window apperas and there is login box with notifications (with cvat 1 there is no such problem).

The problem is: when using CPU and not enablin microk8s enable gpu everything works but when enabling gpu cvat 1.6 gives error (additional info: also there are some anomalies running other workspaces - it seems like more than 1-2 together doesnt work):

Could not check authorization on the server

Error: Request failed with status code 503. "upstream connect error or disconnect/reset before headers. reset reason: connection failure".

Could not get user agreements from the server
Error: Request failed with status code 503. "upstream connect error or disconnect/reset before headers. reset reason: connection failure".

• $ opctl version
  CLI version: v1.0.2
  Manifest version: v1.0.2
  API version: v1.0.2
  Web UI version: v1.0.2

• opctl init --provider microk8s --enable-metallb --artifact-repository-provider abs

• Cloud provider: AKS

• Using on local machine, microk8s

• sudo snap install microk8s --channel=1.19/stable --classic

• Ubuntu 20.04.5 LTS

• nvidia-smi_
  Mon Dec 19 14:08:52 2022
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 |
  |-------------------------------+----------------------+----------------------+
  | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
  | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
  | | | MIG M. |
  |===============================+======================+======================|
  | 0 NVIDIA GeForce ... On | 00000000:65:00.0 On | N/A |
  | 0% 36C P8 8W / 180W | 1MiB / 8192MiB | 0% Default |
  | | | N/A |
  +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Any ideas whats the problem or how to diagnose/find the problem?