K8SPG-619: restart backup jobs on failure

[pooknull]

https://perconadev.atlassian.net/browse/K8SPG-619

DESCRIPTION

Problem:
The backup pod currently fails on the first attempt, resulting in the creation of a new pod on failure. This behavior may not be reliable in all Kubernetes environments, due to potential delays in establishing communication with the Kubernetes API.

Cause:
The backup job’s restartPolicy is set to Never, preventing the existing pod from retrying after a failure.

Solution:
Update the backup job’s restartPolicy to OnFailure and configure backoffLimit to 2. This will allow the existing pod to restart up to two times, with incremental pauses of 10 seconds and 20 seconds between retries.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test name	Status
custom-extensions	passed
custom-tls	passed
demand-backup	failure
finalizers	passed
init-deploy	passed
monitoring	passed
one-pod	passed
operator-self-healing	passed
pitr	passed
scaling	passed
scheduled-backup	passed
self-healing	passed
start-from-backup	passed
tablespaces	passed
telemetry-transfer	passed
upgrade-consistency	passed
upgrade-minor	passed
users	passed
We run 18 out of 18

commit: d109270
image: perconalab/percona-postgresql-operator:PR-969-d1092708f