Flooding of machine state reconciles during condition message field change #2463
Labels
kind/bug
Categorizes issue or PR as related to a bug.
Comments
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
/kind bug
What steps did you take and what happened:
Hello!
First of all, thanks for all the hard work put into this provider!
I've been playing around with CAPI + CAPO + DevStack quite a bit lately, i've got everything running locally on my laptop (ontop of
libvirt+kind).Today when i restarted my DevStack VM, and when the CAPO controller started up i noticed that it logged a lot of log messages that looked like this (
--v=2):I counted the rate of logs to ~13/s, if i checked the Nova compute API in the DevStack end it's the same rate, it logged this:
At this point the state of the instances and load balancer (Octavia) in DevStack were:
Not the best state 😏, and probably a corner case but still, it was enough to make the OpenStack machine controller behaving a bit off.
When i fetched the
OpenStackMachineobject in a watch loop from the CAPI + CAPO (kind) i observed that these two fields constantly changed, at the same rate as the logs were omitted:resourceVersionmessagefield in theInstanceReadyconditionThat
messagefield was being set to:The problem seems to be that we're writing the pointer memory address to the message field here: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/controllers/openstackmachine_controller.go#L456. When we set that in the condition the
resourceVersionchanges and we retrigger a reconcile immediatly (AFAICT).If i dereference that instance state variable instead it works as expected (SHUTOFF is written instead) and the reconcile flooding stops.
The OpenStack server controller does something similar but instead reads the state off the object which will be correct in this case and not trigger flooding: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/controllers/openstackserver_controller.go#L390
What did you expect to happen:
The OpenStackMachine controller should not write the pointer address to a status condition field causing a flood of reconcile attempts and logs.
Anything else you would like to add:
OpenStackMachine `status`
Here we can see the full
statusof the object in CAPO:Environment:
git rev-parse HEADif manually built): fc2c57akubectl version): v1.32.0/etc/os-release): Arch LinuxThe text was updated successfully, but these errors were encountered: