PB-8410 Incorporate the logic not delete the restore job pods when mo… #402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
License Evaluation Results:
Total License Issues: 17 |
vsundarraj-px
force-pushed
the
PB-8410
branch
from
98b146f to
21aeaf5
Compare
|
License Evaluation Results:
Total License Issues: 17 |
vsundarraj-px
force-pushed
the
PB-8410
branch
from
21aeaf5 to
153f6b8
Compare
|
License Evaluation Results:
Total License Issues: 17 |
vsundarraj-px
requested review from
siva-portworx,
prashanthpx and
vikasit12
vsundarraj-px
force-pushed
the
PB-8410
branch
from
153f6b8 to
a9eb631
Compare
|
License Evaluation Results:
Total License Issues: 17 |
vikasit12
approved these changes
Nov 18, 2024
vikasit12
reviewed
Nov 18, 2024
pkg/drivers/utils/utils.go
Outdated
| timeOut = PxbDefaultJobFailureRetryTimeout | ||
| } | ||
| } | ||
| // skipping error cos we know we will never fail here. |
There was a problem hiding this comment.
nit :: We can remove this comment as this is moot
pkg/drivers/utils/utils.go
Outdated
| @@ -79,6 +79,10 @@ const ( | |||
| kopiaBackupString = "kopiaexecutor backup" | |||
| // if providerType in node spec has this string then it is GCP hosted cluster | |||
| GCPBasedClusterString = "gce://" | |||
| //PxbJobFailureRetryTimeoutKey defines timeout key name to be set after job failure due to mount failure | |||
pkg/drivers/utils/utils.go
Outdated
| // returns error if configMap cannot be read from k8s | ||
| // returns error if key is not found in configMap | ||
| // returns value of the key if key is found in configMap | ||
| func GetDataFromConfigMap(name, namespace, key string) (string, error) { |
There was a problem hiding this comment.
Can we re-use existing API GetConfigValue()?
pkg/drivers/utils/utils.go
Outdated
| // we could fail here if the value set is invalid or has some junk charectors. | ||
| duration, err := time.ParseDuration(timeOut + "s") | ||
| if err != nil || duration <= 0 { | ||
| logrus.Errorf("invalid %v value set. Should be numberic value > 0. Setting to default limit", PxbJobFailureRetryTimeoutKey) |
There was a problem hiding this comment.
Can we add fn in the above debug msg as well?
pkg/drivers/utils/utils.go
Outdated
| logrus.Debugf("%v:failed retry limit not found. Setting to default: %v", fn, err) | ||
| timeOut = PxbDefaultJobFailureRetryTimeout | ||
| } else { | ||
| // we could fail here if the value set is invalid or has some junk charectors. |
|
@vsundarraj-px Should we add it for nfsDelete and kopiaDelete as well? |
|
…unt failure occurs within 5 mins Signed-off-by: vsundarraj-px <[email protected]>
vsundarraj-px
force-pushed
the
PB-8410
branch
from
a9eb631 to
58b8ad2
Compare
|
License Evaluation Results:
Total License Issues: 17 |
|
This PR is closed as the PR #404 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…unt failure occurs within 5 mins
What this PR does / why we need it:
This PR handles Job pod mount failures which are transient. Currently while checking for job status and if we find that mount pvc mount has failed, we immediately kill the jobpod and return error. We noticed that this pvc mount failure are transient errors and mount actually succeeds in next few seconds. Hence, this PR enables a timeout settings through KDMP configMap setting. When set, if mount failure should occur, will not fail immediately but wait till the timout set and if still the mount failure occurs then the job is terminated with error.
Which issue(s) this PR fixes (optional)
Closes # PB-8410
Special notes for your reviewer:
Unit testing done
NFS Restore
NFS Backup
KDMP Backup
KDMP Restore
Since the issue is not easily reproducible, the unit testing is performed with a debug image by forcefully setting mountFailure