Error When Deleting Resource in Controller Logs

[tunahanertekin]

What happened?

When deleting submariners.connection-hub.roboscale.io, controller throws this error for once:

1.667211906208583e+09	ERROR	Reconciler error	{"controller": "submarineroperator", "controllerGroup": "connection-hub.roboscale.io", "controllerKind": "SubmarinerOperator", "SubmarinerOperator": {"name":"submariner-operator"}, "namespace": "", "name": "submariner-operator", "reconcileID": "059101c2-9d52-4860-be9a-1f5fae4a3494", "error": "Operation cannot be fulfilled on submarineroperators.connection-hub.roboscale.io \"submariner-operator\": StorageError: invalid object, Code: 4, Key: /registry/connection-hub.roboscale.io/submarineroperators/submariner-operator, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 559c956e-2745-42a1-8472-70dcbc29e1a4, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

What did you expect to happen?

Since it's a graceful deletion, I expect not to see this error that possibly means the controller tries to reconcile the resource after deletion.

How can we reproduce it (as minimally and precisely as possible)?

Create a submariners.connection-hub.roboscale.io instance. Wait for it to reach ReadyToConnect phase. Then delete it:

kubectl delete submariners.connection-hub.roboscale.io submariner

Watch controller logs using:

kubectl logs --follow pod/$(kubectl get pods -n connection-hub-system | tail -n1 | awk '{print $1}') -n connection-hub-system -c manager

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Container network interface (CNI) and version

KubeOVN