vmalert sometimes fails to load updated rules

[umezawatakeshi]

When VMRules are added/deleted/updated, the operator populates them and updates ConfigMaps for vmalerts. The config-reloader sidecar detects it and call the reload API for vmalert. However, vmalert sometimes fail to (re)load some rules. Sometimes, some vmalert replicas successfully load rules but other replicas fail. If the reload API is called manually later, vmalert successfully load all the configured rules.

I suspect that config-reloader calls the reload API "too early" and vmalert begins reload operation before kubelet complete writing all rule files to the volume.

As a workaround, it might be nice for config-reloader to delay calling the reload API for several seconds.

[f41gh7]

Hello,

Yes, it's possible case and default config-reloader image - jimmidyson/configmap-reload:v0.3.0 doesn't provide such feature.

However, it's possible to use alternative config-reloader with the following spec:

kind: VMAlert
spec:
  useVMConfigReloader: true
  configReloaderExtraArgs:
   # change if needed, default value is 3s
    delay-interval: "5s"

https://github.com/VictoriaMetrics/operator/tree/master/cmd/config-reloader

[umezawatakeshi]

I see. I will try it.