kustomize: Add support for OCI based helm repos

[jkroepke]

Bitnami recently change from classical helm repositories to OCI based repositories, which may have an impact of the support for helm charts from OCI repositories.

this PR add support for fetch charts from OCI repositories.

Example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: external-dns
#  repo: https://charts.bitnami.com/bitnami
  repo: oci://registry-1.docker.io/bitnamicharts
  version: 6.19.2
  releaseName: external-dns
  namespace: external-dns
  valuesFile: values.yaml

This PR does not implement a breaking change.

Fixes #4381
ref #4614

[k8s-ci-robot]

Hi @jkroepke. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

This PR has multiple commits, and the default merge method is: merge.
You can request commits to be squashed using the label: tide/merge-method-squash

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jkroepke]

/label tide/merge-method-squash

[natasha41575]

/assign @koba1t

/triage accepted

[natasha41575]

/ok-to-test

[koba1t]

Hi @jkroepke
Thanks for your contribution!

This PR almost looks good to me.
So, it looks like the lint failed on CI. Could you check it and fix it?

[jkroepke]

I run go fmt, that should be sufficient. I tried to run make lint, but it got killed because of OOM.

[DanielYWoo]

Do we have an ETA for this to be released?

[koba1t]

@jkroepke

I found the below error when running on my laptop. Could you fix it?

api/krusty/helmchartinflationgenerator_test.go:49:7: var-naming: const expectedHelmExternalDns should be expectedHelmExternalDNS (revive)
const expectedHelmExternalDns = `
      ^
make[1]: *** [lint] Error 1
make: *** [lint] Error 2

[jkroepke]

Done

[koba1t]

@jkroepke

Thanks!
Almost looks good to me!
/lgtm

[koba1t]

/cc @natasha41575
Could you check this PR and rerun GitHub Actions?

[MPV]

Beautiful!

FYI: I’m adding it into ArgoCD over at:

• chore(deps): bump kustomize to v5.2.1  argoproj/argo-cd#16054 (comment)

[zimmertr]

Is this not working with GHCR? The repo is public?

helmCharts:
  - name: csi-proxmox
    repo: oci://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin
    version: 0.1.15 # https://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin
    releaseName: csi-proxmox
    namespace: csi-proxmox

Error: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%!A(MISSING)sergelogvinov%!F(MISSING)cha
rts%!F(MISSING)proxmox-csi-plugin%!F(MISSING)csi-proxmox%!A(MISSING)pull&service=ghcr.io: 403 Forbidden

[koba1t]

@zimmertr

I think you made a mistake in your kustomization.yaml file; the below file works on my computer.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: proxmox-csi-plugin
    repo: oci://ghcr.io/sergelogvinov/charts
    version: 0.1.15 # https://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin
    releaseName: csi-proxmox
    namespace: csi-proxmox

https://kubectl.docs.kubernetes.io/references/kustomize/builtins/#_helmchartinflationgenerator_

[zimmertr]

Doi, Thank you :)

[marcusnh]

Hello, I have the same problem with my setup to Azure Container Registery:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Referencing a public repo outside the main repository

helmCharts:
  - name: <ARO-REPO-CHART-NAME>
    repo: oci://ACR-NAME.azurecr.io/
    version: 0.1.6-5
    releaseName:ARO-REPO-NAME
    namespace: poseidon2-dev
    valuesFile: values.yaml

The error message is also similar:

level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `kustomize build <path to cached source>/applicationsets/dev/demo-helm-2 --enable-helm` failed exit status 1: Error: Error: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://<ACR-NAME>.azurecr.io/oauth2/token?scope=repository%!A(MISSING)<ARO-REPO-NAME>%!F(MISSING)<ARO-REPO-NAME>%!A(MISSING)pull&service=<ACR-NAME>.azurecr.io: 401 Unauthorized\n: unable to run: 'helm pull --untar --untardir <path to cached source>/applicationsets/dev/demo-helm-2/charts oci://<ACR-NAME>.azurecr.io/<ARO-REPO-NAME>/<ARO-REPO-NAME> --version 0.1.6-5' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-3509023410/helm HELM_CACHE_HOME=/tmp/kustomize-helm-3509023410/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-3509023410/helm/.data] (is 'helm' installed?): exit status 1" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-01-17T09:44:38Z" grpc.time_ms=2.234 span.kind=server system=grpc

[koba1t]

According to your error message, you failed to exec helm pull command.
Please try to exec helm pull directly to check if the helm chart is valid.

[marcusnh]

This works fine on my local computer. I'm able to run helm pull:

helm pull oci://ACR-NAME.azurecr.io/ARO-REPO-CHART-NAME  --version 0.1.6-5
Pulled: ACR-NAME.azurecr.io/ARO-REPO-CHART-NAME:0.1.6-5
Digest: sha256:9be2d3dd16cee312918f2b321453683e8b-NOT-THE-SAME-57b

I have also tested the chart with a ArgoCD application and that works fine with the kubernetes secret passed to my ArgocD instance:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-<ARO-REPO-NAME>
  namespace: gitops-developers # argocd instance namespace
spec:
  source:
    chart: <ARO-REPO-CHART-NAME>
    repoURL: <ARO-NAME>.azurecr.io
    targetRevision: 0.1.6-5
    helm:
      values: |
        application_name: "<ARO-REPO--CHART-NAME>-test"
        namespace: poseidon2-test

  destination:
    namespace: poseidon2-test
    server: https://kubernetes.default.svc
  project: poseidon2
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true
    # syncOptions:
    #   - Replace=true

[koba1t]

I have also tested the chart with a ArgoCD application and that works fine with the kubernetes secret passed to my ArgocD instance:

I apologize for not knowing the details of ArgoCD. Do you use any credentials to pull helm charts?
kustomize doesn't support private repository and registry authentication for helm pull.
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/

[TeamPoseidonOCP]

To get access to the private ACR, I need some form of authentication, and creating helm credentials with a Kubernetes secret has been sufficient to get access with ARgoCD.

In the link you sent, it says: "We will not add support for:

• private repository or registry authentication
• OCI registries
• other large features that increase the complexity of the feature and/or have significant security implications."

OCI registries are now supported. Are there no planned features for enabling private repositories or registry authentication? This would help us a lot and simplify the setup for our developers.

[koba1t]

Sorry, we don't have a plan to add more functions to helmCharts now.

We discuss in this issue about the features of Helm support. Please check, If you have your opinions.
#4401

[MrFreezeex]

To get access to the private ACR, I need some form of authentication, and creating helm credentials with a Kubernetes secret has been sufficient to get access with ARgoCD.

FYI I put some observations/possible followups on this (private oci support): #5407 (comment) and there is two possibles (ugly) workaround that you could do with existing versions

[felixlut]

I think the above issues are related to this ticket: #5407

My workaround for the time being is to run kustomize build in CI (with a short bash script as a workaround to auth to my private helm registry), generate a manifest.lock.yaml containing the entire k8s manifest, and have Argo cd look at that directly. Essentially move the kustomize build step from ArgoCD to CI instead, since I have more control there.

This is also consistent with the rendered manifests pattern:
https://akuity.io/blog/the-rendered-manifests-pattern

With all that said, I really think that kustomize should respect the local Helm credentials, as mentioned in the issue I linked above

[marcusnh]

Thank you for the feedback. I saw the workarounds, and there is the possibility to add them with a config map to the ArgoCD instance, but I think we will have to use another approach until this is supported. Using the workaround provided in the ticket #5407 will fix the problem, but it is not the cloud-native way that we prefer to do things. I think we will have to stick with ArgoCD applications and helm integration until this is supported.
Is there a feature request that I can follow or contribute to?

[jkroepke]

About the authentication issues, you can use helm login, crane login or docker login before running kustomize. The helm OCI integration will re-use the docker credentials for authentication. If you are running private ACR with Azure RBAC, consider to use az acr login instead docker login.

From my point of view, it's not necessary to integrate the login functionality in kustomize.

[MrFreezeex]

About the authentication issues, you can use helm login, crane login or docker login before running kustomize. The helm OCI integration will re-use the docker credentials for authentication. If you are running private ACR with Azure RBAC, consider to use az acr login instead docker login.

From my point of view, it's not necessary to integrate the login functionality in kustomize.

Did you tested that? My experience about this is that helm launched through kustomize is not able to use the docker config/credential...

[jkroepke]

Tested with ghcr.io. Works fine.

Additionally, I run docker logout ghcr.io after testing and I got auth errors back.

It also works with helm registry login ghcr.io

[MrFreezeex]

Actually it does seems to work with helm 3.13.3 indeed thanks! Pretty sure there is something that affected the usage of helm through kustomize that got fixed at the helm level in the last couple of months as I was not alone affected by this so it would be weird that we all messed up at the same time 🤔. Anyway looks fixed to me!

[marcusnh]

Is there an update from v3.13.2 to v3.13.3 that has fixed the authentication issues?
Our infrastructure is using ArgoCD to provision applications for our developers, so we need a way of doing the helm registry login command through that API. Similar to ArgoCD already does today through ArgoCD applications.

If you are running private ACR with Azure RBAC, consider to use az acr login instead docker login.
Need to be able to use any registry, not just ARC.

[jkroepke]

Is there an update from v3.13.2 to v3.13.3 that has fixed the authentication issues? Our infrastructure is using ArgoCD to provision applications for our developers, so we need a way of doing the helm registry login command through that API. Similar to ArgoCD already does today through ArgoCD applications.

If you are running private ACR with Azure RBAC, consider to use az acr login instead docker login.
Need to be able to use any registry, not just ARC.

Thats a topic for ArgoCD, isn't it?

[marcusnh]

It is kustomize that doesn't support private OCI registries, so the issue needs to be fixed there and not in ArgoCD. This feature request is only to make kustomize more compatible with ArgoCD and Helm. It is possible to add a workaround, but it is not very cloud-native. Take a look at this issue; might clear up why this is a kustomize problem and not an ArgoCD error:

• When I use oci private repository in helm on kustomization It has a issue. argoproj/argo-cd#16623

[jkroepke]

It does support it. See above.

ArgoCD is should provide the authenticated context. Argo does this for helm, too.

You can also run helm registry login before running kustomize.

There is already a linked PR at the ArgoCD issue.