Add retry for secondary pool mirroring disablement (canonical#468)

# Description

MicroCeph attempts to disable the remote pool mirroring just after it
disables it for the local pool. Sometimes this operation is performed
before the new state has propagated to the remote cluster. Adding a
retry here to attempt the disablement for remote pool again.

Fixes # (issue)

## Type of change

Please delete options that are not relevant.

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] CleanCode (Code refactor, test updates, does not introduce
functional changes)
- [ ] Documentation update (Doc only change)

## How Has This Been Tested?

> **_NOTE:_** All functional changes should accompany corresponding
tests (unit tests, functional tests etc).

Please describe the addition/modification of tests done to verify this
change. Please also list any relevant details for your test
configuration.

## Contributor's Checklist

Please check that you have:

- [ ] self-reviewed the code in this PR.
- [ ] added code comments, particularly in hard-to-understand areas.
- [ ] updated the user documentation with corresponding changes.
- [ ] added tests to verify effectiveness of this change.

Signed-off-by: utkarsh bhatt <[email protected]>

microceph/ceph/rbd_mirror.go

@@ -6,7 +6,11 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"time"
 
+	"github.com/Rican7/retry"
+	"github.com/Rican7/retry/backoff"
+	"github.com/Rican7/retry/strategy"
 	"github.com/canonical/lxd/shared/logger"
 	"github.com/canonical/microceph/microceph/api/types"
 	"github.com/canonical/microceph/microceph/constants"
@@ -200,14 +204,21 @@ func DisablePoolMirroring(pool string, peer RbdReplicationPeer, localName string
 	// Disable pool mirroring on the local cluster.
 	err = configurePoolMirroring(pool, types.RbdResourceDisabled, "", "")
 	if err != nil {
-		logger.Errorf("REPRBD: %s", err.Error())
+		logger.Errorf("REPRBD: failed to disable the primary pool mirroring %s", err.Error())
 		return err
 	}
 
-	// Disable pool mirroring on the remote cluster.
-	err = configurePoolMirroring(pool, types.RbdResourceDisabled, localName, remoteName)
+	err = retry.Retry(func(i uint) error {
+		// Disable pool mirroring on the remote cluster.
+		err = configurePoolMirroring(pool, types.RbdResourceDisabled, localName, remoteName)
+		if err != nil {
+			logger.Errorf("REPRBD: attempt %d: %s", i, err.Error())
+			return err
+		}
+		return nil
+	}, strategy.Delay(5), strategy.Limit(10), strategy.Backoff(backoff.Linear(5*time.Second)))
 	if err != nil {
-		logger.Errorf("REPRBD: %s", err.Error())
+		logger.Errorf("REPRBD: failed to disable the secondary pool mirroring: %s", err.Error())
 		return err
 	}