PB-8410 Incorporate the logic not delete the restore job pods when mount failure occurs within 5 mins

[vsundarraj-px]

…unt failure occurs within 5 mins

What this PR does / why we need it:
This PR handles Job pod mount failures which are transient. Currently while checking for job status and if we find that mount pvc mount has failed, we immediately kill the jobpod and return error. We noticed that this pvc mount failure are transient errors and mount actually succeeds in next few seconds. Hence, this PR enables a timeout settings through KDMP configMap setting. When set, if mount failure should occur, will not fail immediately but wait till the timout set and if still the mount failure occurs then the job is terminated with error.
Which issue(s) this PR fixes (optional)
Closes #
PB-8410
Special notes for your reviewer:

Unit testing done
NFS Restore
NFS Backup
KDMP Backup
KDMP Restore
Since the issue is not easily reproducible, the unit testing is performed with a debug image by forcefully setting mountFailure

Testing for invalid characters, Tested with negative number, tested with alphanumeric values.

[github-actions]

License Evaluation Results:

Title	Package Name	Package Version	Severity	License Info	Introduced
MPL-2.0 license	github.com/hashicorp/errwrap	1.1.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/portworx@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/snapshot/controllers@#1b6d535ef497', 'github.com/hashicorp/[email protected]', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-multierror	1.1.1	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/portworx@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/snapshot/controllers@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/portworx@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/resourcecollector@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/csi@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/utils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/aws@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/azure@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/gcp@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/portworx@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/snapshot/controllers@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/snapshotter@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/drivers/volume/kdmp@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/utils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']
MPL-2.0 license	github.com/hashicorp/go-version	1.6.0	medium	MPL-2.0	['github.com/portworx/[email protected]', 'github.com/libopenstorage/stork/pkg/applicationmanager/controllers@#1b6d535ef497', 'github.com/libopenstorage/stork/drivers/volume/kdmp@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/utils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/k8sutils@#1b6d535ef497', 'github.com/libopenstorage/stork/pkg/version@#1b6d535ef497', 'github.com/hashicorp/[email protected]']

Total License Issues: 17