Prometheus CRDS validation failing post upgrade #294
Comments
|
I have an update to share: kustomize build . | kuebconform` However it was constantly failing with Playing around I decided to save the output first to a file and then run kubeconform against that file
|
|
I think I'm seeing the same with clusterpolicies.kyverno.io, which is another very long CRD. I've got a very arcane bash build tool that I'm trying to get a reproducer for at the moment, I'll let you know |
|
@TomHellier could you share the URL of the CRD and a test file to validate? if I can reproduce this easily that would make this much easier to fix 👍 |
|
I think this must be a race condition, I've got a reproducer but am unwilling to share it as it contains all of the kubernetes manifests our cluster runs on. If I change validator.go to output the res.Bytes, The part of the CRD which is reliably failing on is https://github.com/kyverno/kyverno/blob/main/config/crds/kyverno/kyverno.io_clusterpolicies.yaml#L13183 From "e ANDed together.", It doesn't fire every single time, and the error changes each time: Resolutions:
It is strange that the changing maxBufSize doesn't seem to have an impact, but I suspect there is some odd goroutine stuff going on here when attempting to parse very large custom resources, when someone is passing in a very large (40MB worth of kubernetes manifests) into stdout. As changing the number of workers is in my control, I'll do that, but I'd suggest increasing the size of the starting buffer as well as it perhaps isn't growing the way you'd expect. I don't know if @varunthakur2480 was seeing the same problem as me, but I was getting the same error. |
|
Are you a bit familiar with go? Could you run kubeconform using "go run" and -race ? it should output a trace when running in a race condition (if that is the problem), which would help me understand what might be happening. |
|
@yannh took a while, but here you go: note - this was ran with |
|
This is perfect thank you so much! I ll try to fix this when i find some time 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have been trying to upgrade to Prometheus v0.14 https://github.com/prometheus-operator/kube-prometheus/tree/release-0.14 but my pipelines fail during kubeconfm validation .
With following error
stdin - CustomResourceDefinition prometheuses.monitoring.coreos.com failed validation: error unmarshalling resource: error converting YAML to JSON: yaml: line 1737: did not find expected keyI have tried skipping the crds using -skip "monitoring.coreos.com/v1,monitoring.coreos.com/v1alpha1"
which does not seem to work , its worth mentioning that the validation fails even if I remove the schemas from validation path with same error and I also have --ignore-missing-schemas enabled
which leads me to believe that kubeconfm is actually failing before it can even start to match the schema
The text was updated successfully, but these errors were encountered: