PMM-10742 PMM-10470 PMM-10921 PMM-8553 PMM-10824 PMM-10944 Backups (#…

…1320) * PMM-10742 Fix pbmV2 error messages handling * PMM-10470 physical backup restore (#1317) * use describe-restore for restore metadata * allow restore on managed * increase time to cover for physical restores waits * revert interval * fix formatting * PMM-10470 Simplify backups cancel detection, fix restores * PMM-10470 Cleanup Co-authored-by: Artem Gavrilov <[email protected]> * Remove invalid test * Improve validation * Fix * PMM-8553 List PITR timeranges (#1213) * generate proto files for pitr API Signed-off-by: michael.okoko <[email protected]> * implement timeranges parsing and returns Signed-off-by: michael.okoko <[email protected]> * move pitr components to location protobuf * add happy path tests * add tests for services * move pitr listing to artifacts service * fix endpoint url * abstract storage layer operations, add scaffold for tests * improve test cases * regenerate files * fix storage service test * regenerate files * update mock name * add timeline merges * satisfy go linter * fix mock naming * unexport compression types * regen files * rename timelines to timeranges * removed named returns * use artifact name as prefix * rename minio service to client * remove replica set from timeranges result * remove nolint directive * satisfy exhaust struct linter * recreate minio client in methods for consistency * better interface name * match file name to struct * improve comments * fix tests * remove extra call to FileStat previously, we will make an extra API call to storage to get each file's metadata. This is unnecessary as minio already provides same metadata in the original list call, hence we only need one API call to get all that info. * improve log message * fix tests * improved object names Signed-off-by: michael.okoko <[email protected]> * Fix pbm backups * Cleanup * PMM-10824 restore pitr backups (#1300) * Added local storage support for MongoDB. * Added local storage support for MongoDB. * Added local storage support for MongoDB. * Added local storage support for MongoDB. * Added local storage support for MongoDB. * Fix linter suggestions. * Refactoring. * Refactoring. * Refactoring. * Refactoring. * Refactoring, improving tests. * Added license info. * Fix bug. * Fix linter warnings. * Fix linter warnings. * Added, improved tests. * Fix after merge. * Removed PMMServerLocationConfig, some minor changes. * Fix tests, add location prefix. * Fix tests. * Fix tests. * Set new agent version. * PMM-10824 Added restore PITR feature. * Fix after merge. * Fix after merge. * Fix after merge. * Fix after merge. * Upgrade dependencies. * Upgrade dependencies. * Added some precondition checks. * Fix after merge. * Fix after merge. * Fix tests. * Refactored, added tests. * Minor changes. * Refactoring. * Reformatting. * Minor changes. * Minor changes. * Fix after merge. * Added handling for another one pbm error. Co-authored-by: Fábio Silva <[email protected]> * Backup fixes (#1329) Fix bugs, added tests * Prevent PITR restores for local storages * PMM-10944 Restart exporters after restore (#1344) * Restart agents after mongoDB restore * Add missing licence header * Regen * Refactoring * Fix test * Increase pbm timeouts, add logic to prevent overlapping of actions on pbm (#1347) * Prevent running restore if there are scheduled backups, wait for pbm ops complete before running restore. * Disable scheduled backups before starting restore * Make artifact as errored if we failed to start backup job * Update managed/services/backup/compatibility_helpers.go Co-authored-by: Michael Okoko <[email protected]> Co-authored-by: Artem Gavrilov <[email protected]> Co-authored-by: Michael Okoko <[email protected]> * Fix * Remove server local storage from code (#1340) * Removed PMMServerLocationConfig from API. * Rename config PMMClient to Filesystem. * Rename config PMMClient to Filesystem. * Merge. * Minor changes. * Make backups API GA * Fix test * Renamed backupv1beta1 to backuppb in the code. (#1349) * Include backups and alerting APIs to swager.json * Refactoring * Check pmm agent version for pitr restore (#1356) * Check pmm-agent version for PITR restore. * Refactoring Co-authored-by: Artem Gavrilov <[email protected]> * Fix time parsing format. (#1355) * Fix time parsing format. * Fix tests. * Changed PBM response parsing logic. * Changed format to constant. * Add todo Co-authored-by: Artem Gavrilov <[email protected]> * Show correct error when artifact not ready for restore. (#1359) * Show correct error when artifact not ready for restore. * Add license header. Signed-off-by: michael.okoko <[email protected]> Co-authored-by: Michael Okoko <[email protected]> Co-authored-by: Fábio Silva <[email protected]> Co-authored-by: michael.okoko <[email protected]> Co-authored-by: Pavel Khripkov <[email protected]>
percona · Nov 1, 2022 · 3e65511 · 3e65511
1 parent 6ef119b
commit 3e65511
Show file tree

Hide file tree

Showing 90 changed files with 8,210 additions and 2,821 deletions.
diff --git a/Makefile.include b/Makefile.include
@@ -57,13 +57,15 @@ gen: clean            ## Generate files.
 	done
 
 	# generate public API spec, omit agentlocalpb (always private),
-	# and managementpb/dbaas, managementpb/ia, managementpb/alerting, managementpb/backup , managementpb/azure and qanpb (not v1 yet)
+	# and managementpb/dbaas, managementpb/ia, managementpb/azure and qanpb (not v1 yet)
 	bin/swagger mixin --output=api/swagger/swagger.json \
 		api/swagger/header.json \
 		api/serverpb/json/serverpb.json \
 		api/userpb/json/userpb.json \
 		api/inventorypb/json/inventorypb.json \
-		api/managementpb/json/managementpb.json
+		api/managementpb/json/managementpb.json \
+        api/managementpb/backup/json/backup.json \
+        api/managementpb/alerting/json/alerting.json
 	bin/swagger validate api/swagger/swagger.json
 
 	bin/swagger-order --output=api/swagger/swagger.json api/swagger/swagger.json

diff --git a/agent/agents/supervisor/supervisor.go b/agent/agents/supervisor/supervisor.go
@@ -213,6 +213,30 @@ func (s *Supervisor) SetState(state *agentpb.SetStateRequest) {
 	s.setBuiltinAgents(state.BuiltinAgents)
 }
 
+// RestartAgents restarts all existing agents.
+func (s *Supervisor) RestartAgents() {
+	s.rw.Lock()
+	defer s.rw.Unlock()
+
+	for id, agent := range s.agentProcesses {
+		agent.cancel()
+		<-agent.done
+
+		if err := s.startProcess(id, agent.requestedState, agent.listenPort); err != nil {
+			s.l.Errorf("Failed to restart Agent: %s.", err)
+		}
+	}
+
+	for id, agent := range s.builtinAgents {
+		agent.cancel()
+		<-agent.done
+
+		if err := s.startBuiltin(id, agent.requestedState); err != nil {
+			s.l.Errorf("Failed to restart Agent: %s.", err)
+		}
+	}
+}
+
 func (s *Supervisor) storeLastStatus(agentID string, status inventorypb.AgentStatus) {
 	s.arw.Lock()
 	defer s.arw.Unlock()

diff --git a/agent/client/client.go b/agent/client/client.go
@@ -543,10 +543,10 @@ func (c *Client) handleStartJobRequest(p *agentpb.StartJobRequest) error {
 				BucketName:   cfg.S3Config.BucketName,
 				BucketRegion: cfg.S3Config.BucketRegion,
 			}
-		case *agentpb.StartJobRequest_MongoDBBackup_PmmClientConfig:
-			locationConfig.Type = jobs.PMMClientBackupLocationType
-			locationConfig.LocalStorageConfig = &jobs.PMMClientBackupLocationConfig{
-				Path: cfg.PmmClientConfig.Path,
+		case *agentpb.StartJobRequest_MongoDBBackup_FilesystemConfig:
+			locationConfig.Type = jobs.FilesystemBackupLocationType
+			locationConfig.FilesystemStorageConfig = &jobs.FilesystemBackupLocationConfig{
+				Path: cfg.FilesystemConfig.Path,
 			}
 		default:
 			return errors.Errorf("unknown location config: %T", j.MongodbBackup.LocationConfig)
@@ -575,10 +575,10 @@ func (c *Client) handleStartJobRequest(p *agentpb.StartJobRequest) error {
 				BucketName:   cfg.S3Config.BucketName,
 				BucketRegion: cfg.S3Config.BucketRegion,
 			}
-		case *agentpb.StartJobRequest_MongoDBRestoreBackup_PmmClientConfig:
-			locationConfig.Type = jobs.PMMClientBackupLocationType
-			locationConfig.LocalStorageConfig = &jobs.PMMClientBackupLocationConfig{
-				Path: cfg.PmmClientConfig.Path,
+		case *agentpb.StartJobRequest_MongoDBRestoreBackup_FilesystemConfig:
+			locationConfig.Type = jobs.FilesystemBackupLocationType
+			locationConfig.FilesystemStorageConfig = &jobs.FilesystemBackupLocationConfig{
+				Path: cfg.FilesystemConfig.Path,
 			}
 		default:
 			return errors.Errorf("unknown location config: %T", j.MongodbRestoreBackup.LocationConfig)
@@ -591,7 +591,8 @@ func (c *Client) handleStartJobRequest(p *agentpb.StartJobRequest) error {
 			Port:     int(j.MongodbRestoreBackup.Port),
 			Socket:   j.MongodbRestoreBackup.Socket,
 		}
-		job = jobs.NewMongoDBRestoreJob(p.JobId, timeout, j.MongodbRestoreBackup.Name, dbConnCfg, locationConfig)
+
+		job = jobs.NewMongoDBRestoreJob(p.JobId, timeout, j.MongodbRestoreBackup.Name, j.MongodbRestoreBackup.PitrTimestamp.AsTime(), dbConnCfg, locationConfig, c.supervisor)
 	default:
 		return errors.Errorf("unknown job type: %T", j)
 	}

diff --git a/agent/client/deps.go b/agent/client/deps.go
@@ -46,6 +46,7 @@ type supervisor interface {
 	Changes() <-chan *agentpb.StateChangedRequest
 	QANRequests() <-chan *agentpb.QANCollectRequest
 	SetState(*agentpb.SetStateRequest)
+	RestartAgents()
 	AgentLogByID(string) ([]string, uint)
 	// Collector added to use client as Prometheus collector
 	prometheus.Collector

diff --git a/agent/client/mock_supervisor_test.go b/agent/client/mock_supervisor_test.go
diff --git a/agent/runner/jobs/backup_location.go b/agent/runner/jobs/backup_location.go
@@ -19,8 +19,8 @@ type BackupLocationType string
 
 // BackupLocation types. Same as in managed/models/location_model.go.
 const (
-	S3BackupLocationType        BackupLocationType = "s3"
-	PMMClientBackupLocationType BackupLocationType = "pmm-client"
+	S3BackupLocationType         BackupLocationType = "s3"
+	FilesystemBackupLocationType BackupLocationType = "filesystem"
 )
 
 // S3LocationConfig contains required properties for accessing S3 Bucket.
@@ -32,14 +32,14 @@ type S3LocationConfig struct {
 	BucketRegion string
 }
 
-// PMMClientBackupLocationConfig contains config for local storage
-type PMMClientBackupLocationConfig struct {
+// FilesystemBackupLocationConfig contains config for local storage
+type FilesystemBackupLocationConfig struct {
 	Path string
 }
 
 // BackupLocationConfig groups all backup locations configs.
 type BackupLocationConfig struct {
-	Type               BackupLocationType
-	S3Config           *S3LocationConfig
-	LocalStorageConfig *PMMClientBackupLocationConfig
+	Type                    BackupLocationType
+	S3Config                *S3LocationConfig
+	FilesystemStorageConfig *FilesystemBackupLocationConfig
 }
diff --git a/agent/runner/jobs/deps.go b/agent/runner/jobs/deps.go
@@ -0,0 +1,19 @@
+// Copyright 2019 Percona LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//  http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package jobs
+
+type agentsRestarter interface {
+	RestartAgents()
+}
diff --git a/agent/runner/jobs/mongodb_backup_job.go b/agent/runner/jobs/mongodb_backup_job.go
@@ -19,6 +19,7 @@ import (
 	"context"
 	"io"
 	"net/url"
+	"os"
 	"os/exec"
 	"sync/atomic"
 	"time"
@@ -28,7 +29,7 @@ import (
 	"google.golang.org/protobuf/types/known/timestamppb"
 
 	"github.com/percona/pmm/api/agentpb"
-	backupv1beta1 "github.com/percona/pmm/api/managementpb/backup"
+	backuppb "github.com/percona/pmm/api/managementpb/backup"
 )
 
 const (
@@ -48,7 +49,7 @@ type MongoDBBackupJob struct {
 	locationConfig BackupLocationConfig
 	pitr           bool
 	logChunkID     uint32
-	dataModel      backupv1beta1.DataModel
+	dataModel      backuppb.DataModel
 }
 
 // NewMongoDBBackupJob creates new Job for MongoDB backup.
@@ -59,12 +60,12 @@ func NewMongoDBBackupJob(
 	dbConfig DBConnConfig,
 	locationConfig BackupLocationConfig,
 	pitr bool,
-	dataModel backupv1beta1.DataModel,
+	dataModel backuppb.DataModel,
 ) (*MongoDBBackupJob, error) {
-	if dataModel != backupv1beta1.DataModel_PHYSICAL && dataModel != backupv1beta1.DataModel_LOGICAL {
+	if dataModel != backuppb.DataModel_PHYSICAL && dataModel != backuppb.DataModel_LOGICAL {
 		return nil, errors.Errorf("'%s' is not a supported data model for MongoDB backups", dataModel)
 	}
-	if dataModel != backupv1beta1.DataModel_LOGICAL && pitr {
+	if dataModel != backuppb.DataModel_LOGICAL && pitr {
 		return nil, errors.Errorf("PITR is only supported for logical backups")
 	}
 	return &MongoDBBackupJob{
@@ -106,12 +107,19 @@ func (j *MongoDBBackupJob) Run(ctx context.Context, send Send) error {
 	if err != nil {
 		return errors.WithStack(err)
 	}
-	if err := pbmConfigure(ctx, j.l, j.dbURL, conf); err != nil {
+
+	confFile, err := writePBMConfigFile(conf)
+	if err != nil {
+		return errors.WithStack(err)
+	}
+	defer os.Remove(confFile) //nolint:errcheck
+
+	if err := pbmConfigure(ctx, j.l, j.dbURL, confFile); err != nil {
 		return errors.Wrap(err, "failed to configure pbm")
 	}
 
 	rCtx, cancel := context.WithTimeout(ctx, resyncTimeout)
-	if err := waitForPBMState(rCtx, j.l, j.dbURL, pbmNoRunningOperations); err != nil {
+	if err := waitForPBMNoRunningOperations(rCtx, j.l, j.dbURL); err != nil {
 		cancel()
 		return errors.Wrap(err, "failed to wait configuration completion")
 	}
@@ -131,7 +139,7 @@ func (j *MongoDBBackupJob) Run(ctx context.Context, send Send) error {
 		}
 	}()
 
-	if err := waitForPBMState(ctx, j.l, j.dbURL, pbmBackupFinished(pbmBackupOut.Name)); err != nil {
+	if err := waitForPBMBackup(ctx, j.l, j.dbURL, pbmBackupOut.Name); err != nil {
 		j.sendLog(send, err.Error(), false)
 		return errors.Wrap(err, "failed to wait backup completion")
 	}
@@ -156,11 +164,11 @@ func (j *MongoDBBackupJob) startBackup(ctx context.Context) (*pbmBackup, error)
 
 	pbmArgs := []string{"backup"}
 	switch j.dataModel {
-	case backupv1beta1.DataModel_PHYSICAL:
+	case backuppb.DataModel_PHYSICAL:
 		pbmArgs = append(pbmArgs, "--type=physical")
-	case backupv1beta1.DataModel_LOGICAL:
+	case backuppb.DataModel_LOGICAL:
 		pbmArgs = append(pbmArgs, "--type=logical")
-	case backupv1beta1.DataModel_DATA_MODEL_INVALID:
+	case backuppb.DataModel_DATA_MODEL_INVALID:
 	default:
 		return nil, errors.Errorf("'%s' is not a supported data model for backups", j.dataModel)
 	}

diff --git a/agent/runner/jobs/mongodb_backup_job_test.go b/agent/runner/jobs/mongodb_backup_job_test.go
@@ -20,7 +20,7 @@ import (
 
 	"github.com/stretchr/testify/assert"
 
-	backupv1beta1 "github.com/percona/pmm/api/managementpb/backup"
+	backuppb "github.com/percona/pmm/api/managementpb/backup"
 )
 
 func TestCreateDBURL(t *testing.T) {
@@ -83,33 +83,33 @@ func TestNewMongoDBBackupJob(t *testing.T) {
 	tests := []struct {
 		name      string
 		dbConfig  DBConnConfig
-		dataModel backupv1beta1.DataModel
+		dataModel backuppb.DataModel
 		pitr      bool
 		errMsg    string
 	}{
 		{
 			name:      "logical backup model",
 			dbConfig:  DBConnConfig{},
-			dataModel: backupv1beta1.DataModel_LOGICAL,
+			dataModel: backuppb.DataModel_LOGICAL,
 			errMsg:    "",
 		},
 		{
 			name:      "physical backup model",
 			dbConfig:  DBConnConfig{},
-			dataModel: backupv1beta1.DataModel_PHYSICAL,
+			dataModel: backuppb.DataModel_PHYSICAL,
 			errMsg:    "",
 		},
 		{
 			name:      "invalid backup model",
 			dbConfig:  DBConnConfig{},
-			dataModel: backupv1beta1.DataModel_DATA_MODEL_INVALID,
+			dataModel: backuppb.DataModel_DATA_MODEL_INVALID,
 			errMsg:    "'DATA_MODEL_INVALID' is not a supported data model for MongoDB backups",
 		},
 		{
 			name:      "pitr fails for physical backups",
 			dbConfig:  DBConnConfig{},
 			pitr:      true,
-			dataModel: backupv1beta1.DataModel_PHYSICAL,
+			dataModel: backuppb.DataModel_PHYSICAL,
 			errMsg:    "PITR is only supported for logical backups",
 		},
 	}