Unable to create workspace based on mcr.microsoft.com/azure-cli (and UI doesn't surface root cause)

[adrienthebo]

Bug description

When creating a workspace based on mcr.microsoft.com/azure-cli:2.39.0, the workspace fails with the following error:

last backup failed: cannot find workspace <uuid> during DisposeWorkspace.

Logs from workspace indicate the following:

cannot ensure gitpod user exists

{
  "insertId": "xl7nrruuf28bczci",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
    "serviceContext": {
      "version": "commit-062938b01dda42dfaad74af2ce8d87a44faf964c",
      "service": "supervisor"
    },
    "message": "cannot ensure Gitpod user exists",
    "error": "[adduser -h /home/gitpod -s /bin/sh -D -G 33333 -u 33333 gitpod]: exit status 1: adduser: unknown group 33333:\n    github.com/gitpod-io/gitpod/supervisor/pkg/supervisor.addUser\n        github.com/gitpod-io/gitpod/supervisor/pkg/supervisor/user.go:172\n  - exit status 1",
    "level": "fatal"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "location": "us-west1",
      "pod_name": "ws-410bb5a4-af36-432d-bab1-7f62fd304041",
      "container_name": "workspace",
      "project_id": "workspace-clusters",
      "cluster_name": "ws-us63",
      "namespace_name": "default"
    }
  },
  "timestamp": "2022-08-24T18:45:41Z",
  "severity": "CRITICAL",
  "labels": {
    "compute.googleapis.com/resource_name": "workspace-ws-us63-pool-rx6r"
  },
  "logName": "projects/workspace-clusters/logs/stderr",
  "receiveTimestamp": "2022-08-24T18:45:48.621058559Z"
}

stat /etc/sudoers: no such file or directory

{
  "insertId": "w544pjzy1n3kclvj",
  "jsonPayload": {
    "level": "error",
    "serviceContext": {
      "version": "commit-062938b01dda42dfaad74af2ce8d87a44faf964c",
      "service": "supervisor"
    },
    "message": "add gitpod sudoers",
    "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
    "error": "stat /etc/sudoers: no such file or directory"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "namespace_name": "default",
      "pod_name": "ws-410bb5a4-af36-432d-bab1-7f62fd304041",
      "cluster_name": "ws-us63",
      "container_name": "workspace",
      "project_id": "workspace-clusters",
      "location": "us-west1"
    }
  },
  "timestamp": "2022-08-24T18:45:41Z",
  "severity": "ERROR",
  "labels": {
    "compute.googleapis.com/resource_name": "workspace-ws-us63-pool-rx6r"
  },
  "logName": "projects/workspace-clusters/logs/stderr",
  "receiveTimestamp": "2022-08-24T18:45:48.621058559Z"
}

From a user's perspective, the only information emitted is the workspace disposal issue:

Workspace disposal failure

An end user attempting to do the same will be stuck with a broken workspace and no obvious means of recourse; surfacing more errors to the user would be very helpful.

Internal slack thread

Steps to reproduce

Create a workspace with a minimal Dockerfile and .gitpod.yml:

.gitpod.Dockerfile

FROM mcr.microsoft.com/azure-cli:2.39.0

.gitpod.yml

image:
  file: .gitpod.Dockerfile

Attempting to launch a workspace from this repo will fail.

Workspace affected

adrienthebo-gitpodazure-me8r3rb8eet

Expected behavior

The given workspace should either launch successfully or indicate why the workspace failed to start and that the root issue is the Dockerfile source image.

Example repository

• adrienthebo/gitpod-azure-toolkit

Anything else?

Last observed on the ws-us63 cluster.

Related issues

• [bridge] Ignored Error Message #12355
• [ide] Investigate the impossibility of connecting via SSH to some workspaces using custom Docker images #12265 (issue triggered due to change the upstream mcr repository)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.