Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add action and state stores migrations #4305

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions internal/pkg/agent/application/paths/files.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,20 @@ const defaultAgentFleetFile = "fleet.enc"
// defaultAgentEnrollFile is a name of file used to enroll agent on first-start
const defaultAgentEnrollFile = "enroll.yml"

// defaultAgentActionStoreFile is the file that will contain the action that can be replayed after restart.
// defaultAgentActionStoreFile is the file that will contain the action that can
// be replayed after restart.
// It's deprecated and kept for migration purposes.
// Deprecated.
const defaultAgentActionStoreFile = "action_store.yml"

// defaultAgentStateStoreYmlFile is the file that will contain the action that can be replayed after restart.
// defaultAgentStateStoreYmlFile is the file that will contain the action that
// can be replayed after restart.
// It's deprecated and kept for migration purposes.
// Deprecated.
const defaultAgentStateStoreYmlFile = "state.yml"

// defaultAgentStateStoreFile is the file that will contain the action that can be replayed after restart encrypted.
// defaultAgentStateStoreFile is the file that will contain the encrypted state
// store.
const defaultAgentStateStoreFile = "state.enc"

// defaultInputDPath return the location of the inputs.d.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,4 +109,10 @@ func TestEncryptedDiskStorageWindowsLinuxLoad(t *testing.T) {
if diff != "" {
t.Error(diff)
}

// Save something else
err = s.Save(bytes.NewBuffer([]byte("new data")))
if err != nil {
t.Fatal(err)
}
}
9 changes: 7 additions & 2 deletions internal/pkg/agent/storage/encrypted_disk_store.go
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,13 @@ func (d *EncryptedDiskStore) ensureKey(ctx context.Context) error {
return nil
}

// Save will write the encrypted storage to disk.
// Specifically it will write to a .tmp file then rotate the file to the target name to ensure that an error does not corrupt the previously written file.
// Save will read 'in' and write its contents encrypted to disk.
// If EncryptedDiskStore.Load() was called, the io.ReadCloser it returns MUST be
// closed before Save() can be called. It is so because Save() writes to a .tmp
// file then rotate the file to the target name to ensure that an error does not
// corrupt the previously written file.
// Specially on windows systems, if the original files is still open because of
// Load(), Save() would fail.
func (d *EncryptedDiskStore) Save(in io.Reader) error {
// Ensure has agent key
err := d.ensureKey(d.ctx)
Expand Down
4 changes: 3 additions & 1 deletion internal/pkg/agent/storage/storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ const perms os.FileMode = 0600

// Store saves the io.Reader.
type Store interface {
// Save the io.Reader.
// Save the io.Reader. Depending on the underlying implementation, if
// Storage.Load() was called, the io.ReadCloser MUST be closed before Save()
// can be called.
Save(io.Reader) error
}

Expand Down
156 changes: 0 additions & 156 deletions internal/pkg/agent/storage/store/action_store.go

This file was deleted.

84 changes: 0 additions & 84 deletions internal/pkg/agent/storage/store/action_store_test.go

This file was deleted.

95 changes: 95 additions & 0 deletions internal/pkg/agent/storage/store/internal/migrations/migrations.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package migrations

import (
"bytes"
"errors"
"fmt"
"io"
"time"

"gopkg.in/yaml.v2"
)

type StateStore struct {
Action Action `yaml:"action"`
AckToken string `yaml:"ack_token"`
ActionQueue []ActionQueue `yaml:"action_queue"`
}

// Action is a struct to read an Action Policy Change from its old
// YAML format.
type Action struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action has quite a broad meaning in the rest of the agent.
This looks like a very specific action store: it represents a Policy Action.
What about calling this PolicyActionStore or something more specific?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also a private type actionQueue []fleetapi.ScheduledAction in state_store.go that makes following this even more confusion.

Why isn't this the general fleetapi.Action? Or one of the concrete Fleet action API implementations?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package is only for action and state store migrations. It cannot be imported outside the strore package. That's why I thought it would not be confusing, at this level there is only one meaning for Action.

Why isn't this the general fleetapi.Action? Or one of the concrete Fleet action API implementations?

because it's a code for migration, the fleetapi models do not work, they changed. There was a whole PR just to fix the schema: #4240

Copy link
Member Author

@AndersonQ AndersonQ Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it was copied from the old store code. I kept it as it was, even thought it has properties to read different actions, as far as I can tell, there should only be 'Policy Change' actions saved

ActionID string `yaml:"action_id"`
Type string `yaml:"action_type"`
StartTime time.Time `yaml:"start_time,omitempty"`
SourceURI string `yaml:"source_uri,omitempty"`
RetryAttempt int `yaml:"retry_attempt,omitempty"`
Policy map[string]interface{} `yaml:"policy,omitempty"`
IsDetected bool `yaml:"is_detected,omitempty"`
}

// ActionQueue is a struct to read the action queue from its old YAML format.
type ActionQueue struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This queue is quite peculiar: it's the same as Action above with ExpirationTime and Version attributes added.
I assume that the structure follows the old structure we had in the store, but maybe we can add some comments about why it's defined this way 😅

ActionID string `yaml:"action_id"`
Type string `yaml:"type"`
StartTime time.Time `yaml:"start_time,omitempty"`
ExpirationTime time.Time `yaml:"expiration,omitempty"`
Version string `yaml:"version,omitempty"`
SourceURI string `yaml:"source_uri,omitempty"`
RetryAttempt int `yaml:"retry_attempt,omitempty"`
Policy map[string]interface{} `yaml:"policy,omitempty"`
IsDetected bool `yaml:"is_detected,omitempty"`
}

type loader interface {
Load() (io.ReadCloser, error)
}

// LoadActionStore loads an action store from loader.
func LoadActionStore(loader loader) (*Action, error) {
return LoadStore[Action](loader)
}

// LoadYAMLStateStore loads the old YAML state store from loader.
func LoadYAMLStateStore(loader loader) (*StateStore, error) {
return LoadStore[StateStore](loader)
}

// LoadStore loads a YAML file.
func LoadStore[Store any](loader loader) (store *Store, err error) {
// Store is a generic type, this might be needed.
store = new(Store)

reader, err := loader.Load()
if err != nil {
return nil, fmt.Errorf("failed to load action store: %w", err)
}
defer func() {
err2 := reader.Close()
if err != nil {
err = errors.Join(err,
fmt.Errorf("migration storeLoad failed to close reader: %w", err2))
}
}()

data, err := io.ReadAll(reader)
if err != nil {
return nil, err
}
buff := bytes.NewReader(data)

dec := yaml.NewDecoder(buff)
err = dec.Decode(&store)
if errors.Is(err, io.EOF) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("could not YAML unmarshal action from action store: %w", err)
}

return store, nil
}
Loading
Loading