Depending on Helm Chart that installs CRDs does not work

[nathanielkeeler]

What happened?

Hello together,

I am trying to install the Rancher Backup Operator on my Cluster, using Helm. For that, I need to first install the the CRD Helm Chart and make the Operator Chart depend on that. I've tried this using v3 and v4 Charts, but it does not work. Using a Release is also not a good solution since a lot of resources depend on these installs and I don't want to have to wrap my whole program in 'apply' functions. What I noticed is that the Operator Helm Chart seems to want to validate the installation of the CRDs, however they are not there yet. Not sure if this can be solved through pulumi or if it is something inside the Operator Chart that needs to be fixed for it to work with IaC Tools that depend on other resources.

For now the only way to bypass this problem is to add the --skip-preview flag when running pulumi up.

This is the output of the example below using helm.v3.Chart:

pulumi:pulumi:Stack (rancher-config-dev):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1036, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart."
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-07T07:56:11.027870365+00:00", grpc_status:2, grpc_message:"execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart."}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 142, in run_in_stack
        await run_pulumi_func(run)
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 56, in run_pulumi_func
        await wait_for_rpcs()
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 118, in wait_for_rpcs
        await task
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1041, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1038, in do_rpc_call
        handle_grpc_error(exn)
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.

This is the output of the example below using helm.v4.Chart:

pulumi:pulumi:Stack (rancher-config-dev):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1036, in do_rpc_call
        return monitor.RegisterResource(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
        return _end_unary_response_blocking(state, call, False, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
        raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart."
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.", grpc_status:2, created_time:"2024-10-07T08:09:01.234639993+00:00"}"
    >
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 142, in run_in_stack
        await run_pulumi_func(run)
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 56, in run_pulumi_func
        await wait_for_rpcs()
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 118, in wait_for_rpcs
        await task
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1041, in do_register
        resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1038, in do_rpc_call
        handle_grpc_error(exn)
      File "/work/venv/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
        raise grpc_error_to_exception(exn)
    Exception: execution error at (rancher-backup/templates/validate-install-crd.yaml:13:7): Required CRDs are missing. Please install the corresponding CRD chart before installing this chart.

Example

Trying it with Helm v3 Chart like this, using the .ready Status:

# Install the CRD Helm Chart
rancher_backup_restore_crds = helmv3.Chart(
    "rancher-backup-restore-crds",
    helmv3.ChartOpts(
        chart='rancher-backup-crd',
        namespace='cattle-resources-system',
        fetch_opts=helmv3.FetchOpts(
            repo="https://charts.rancher.io"
        )
    ),
    opts=pulumi.ResourceOptions(
        provider=k8s_provider
    )
)

# Apply Secret used for encrypting and decrypting the Backups
encryptionconfig = k8s.core.v1.Secret(
    "encryptionconfig-secret",
    metadata={
        "name": "encryptionconfig",
        "namespace": "cattle-resources-system"
    },
    data={
        "encryption-provider-config.yaml": backup_secrets.apply(
            lambda s: s.data['encryption-key']
        )
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=rancher_backup_restore_crds.ready
    )
)

# Install the Rancher Backup Restore Operator
rancher_backup_restore_operator = helmv4.Chart(
    "rancher-backup-restore-operator",
    chart='rancher-backup',
    namespace='cattle-resources-system',
    repository_opts=helmv4.RepositoryOptsArgs(repo="https://charts.rancher.io"),
    values={xyz},
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=rancher_backup_restore_crds.ready
    )
)

The other way i've tried running it, using Helm v4 Chart, using the variable name of the chart directly in the dependencies:

# Install the CRD Helm Chart
rancher_backup_restore_crds = helmv4.Chart(
    "rancher-backup-restore-crds",
    chart='rancher-backup-crd',
    namespace='cattle-resources-system',
    repository_opts=helmv4.RepositoryOptsArgs(
        repo="https://charts.rancher.io"
    ),
    opts=pulumi.ResourceOptions(
        provider=k8s_provider
    )
)

# Apply Secret used for encrypting and decrypting the Backups
encryptionconfig = k8s.core.v1.Secret(
    "encryptionconfig-secret",
    metadata={
        "name": "encryptionconfig",
        "namespace": "cattle-resources-system"
    },
    data={
        "encryption-provider-config.yaml": backup_secrets.apply(
            lambda s: s.data['encryption-key']
        )
    },
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=rancher_backup_restore_crds
    )
)

# Install the Rancher Backup Restore Operator
rancher_backup_restore_operator = helmv4.Chart(
    "rancher-backup-restore-operator",
    chart='rancher-backup',
    namespace='cattle-resources-system',
    repository_opts=helmv4.RepositoryOptsArgs(repo="https://charts.rancher.io"),
    values={xyz},
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=rancher_backup_restore_crds
    )
)

Output of pulumi about

CLI
Version 3.112.0
Go Version go1.22.1
Go Compiler gc

Plugins
NAME VERSION
command 0.10.0
kubernetes 4.18.1
python unknown
rancher2 6.1.1
tls 4.10.0
vault 5.20.0
vsphere 4.10.0

Host
OS debian
Version 12.6
Arch x86_64

This project is written in python: executable='/usr/local/bin/python3' version='3.12.4'

Backend

Dependencies:
NAME VERSION
openshift 0.13.2
pip 24.2
pulumi_kubernetes 4.18.1
pulumi_rancher2 6.1.1
pulumi_tls 4.10.0
sds-pulumi 0.2.16
setuptools 75.1.0
wheel 0.44.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Hi @nathanielkeeler, sorry to hear you’re experiencing this issue. After investigating, it seems the error stems from this templated file in the rancher-backup Chart (https://github.com/rancher/backup-restore-operator/blob/main/charts/rancher-backup/templates/validate-install-crd.yaml).

For Helm v4 charts, Pulumi performs a server-side dry run (reference:

pulumi-kubernetes/provider/pkg/provider/helm/v4/chart.go

Line 186 in 3c2d98c

cmd.DryRunOption = "server"

). This triggers the validate-install-crd to run against the live cluster. However, because the CRDs aren’t yet available during preview (and preview operations don’t persist on the cluster, causing dependent previews to fail), this error occurs. As a workaround, you can either use the Helm v3 chart object to install the non-CRD chart (i.e., rancher-backup), or skip preview as you mentioned.

Here’s an example:

rancher_backup_restore_crds = helmv4.Chart(
    "rancher-backup-restore-crds",
    chart='rancher-backup-crd',
    namespace='cattle-resources-system',
    repository_opts=helmv4.RepositoryOptsArgs(
        repo="https://charts.rancher.io"
    ),
    opts=pulumi.ResourceOptions(
        provider=k8s_provider
    )
)

# Install the Rancher Backup Restore Operator
rancher_backup_restore_operator = helmv3.Chart(
    "rancher-backup-restore-operator",
    helmv3.ChartOpts(
        chart="rancher-backup",
        namespace="cattle-resources-system",
        fetch_opts=helmv3.FetchOpts(
            repo="https://charts.rancher.io",
        ),
    ),
    opts=pulumi.ResourceOptions(
        provider=k8s_provider,
        depends_on=rancher_backup_restore_crds,
    ),
)

Since Helm v3 charts don’t perform a server-side dry run, this error won’t occur.

Additionally, we’re tracking another issue (#3073) to reassess dry-run behavior for non-Helm resources as well, and will also take this scenario into consideration.

[nathanielkeeler]

This makes sense, thank you for clarifying the problem in such a comprehensible manner! Will be looking forward to see a solution to this case. I will be closing the issue then.