Agent centralized configuration: validate and apply downloaded files

[Nicogp]

Parent issue:

• Agent centralized configuration #32

Description

As part of the development of #32 it is necessary for the module to validate the files received and if they are correct apply the configuration received or restart the agent as needed.

File Reading and Validation:

• The agent must read the configuration files in the specified order.
• The agent must validate each configuration file to ensure it is error-free.

Configuration Application:

• If validation is successful, the agent must apply the new configuration to all relevant modules.
• If required, the agent must be able to restart to apply the new configuration fully.

Other considerations:

• Remove duplicated code.
• Include some useful logs (ie. when the command is not recognized).
• Validate functions pointers (m_setGroupIdFunction, m_getGroupIdFunction, m_downloadGroupFilesFunction).

[Nicogp]

Work update

2024/11/05

• I started to make the changes, added a method in ConfigurationParser to validate files
• I removed the code duplication in CentralizedConfiguration
• Issue on hold due to Attach agent information in communication to the server #256

2024/11/12

• I have taken up the issue
• I added the changes for CentralizedConfiguration to perform the verification of the downloaded files
• I have corrected the creation of the download path for the file
• I have added log messages in case of any failure.

2024/11/19

• Added the changes to move the validated files to the shared folder
• I made a rebasing with master
• I performed manual tests
• I found a bug and corrected it in this PR

2024/11/20

• I have added the changes to moves the valid group file to the 'shared' folder
• I have added the changes to set the SHARED directory in cmake
• I have added the changes to clean up the SHARED directory before downloading new group files

2024/11/21

• I have added the changes to read the shared configuration at agent startup
• I made a PoC to use the observer pattern to notify all subscribing modules that the configuration has changed.

2024/11/22

• I have added the changes to use the observer pattern to notify all subscribing modules that the configuration has changed.
  • I have added the new class
  • I have added the method ReloadConfiguration (empty) to the modules

2024/11/25

• I made multiple rebases
• I adapted the development to the changes introduced with respect to m_datapath
• I unified the use of folder definition constants in ConfigurationParser

2024/11/26

• I added additional validation when loading files
• I added methods to save the agent´s groups information
• I added the mechanism to notify the reload signal, the modules are restarted but so far I am not managing to propagate the configurationparser object correctly.

2024/11/27

• I moved the signal handling to the agent object, which is in charge of stopping the modules through moduleManager and then restarting them with the new configuration.

As for the inventory module, the tests are satisfactory:

initial configuration in wazuh-agent.yml:

agent:
  server_url: http://localhost:27000/
  retry_interval: 3ms
inventory:
  enabled: false
  interval: 60
  scan_on_start: true
  hardware: true
  os: true
  network: true
  packages: true
  ports: true
  ports_all: true
  processes: true
  hotfixes: true
logcollector:
  enabled: false
  localfiles:
    - /var/log/auth.log
  reload_interval: 60
  file_wait: 500

Agent starts without groups:

[2024-11-27 16:56:26.975] [wazuh-agent] [info] [INFO] [process_options_unix.cpp:26] [StartAgent] Starting wazuh-agent
[2024-11-27 16:56:26.978] [wazuh-agent] [info] [INFO] [communicator.hpp:55] [Communicator] Using insecure connection.
[2024-11-27 16:56:26.980] [wazuh-agent] [info] [INFO] [moduleManager.cpp:42] [Setup] Modulemanager
[2024-11-27 16:56:26.983] [wazuh-agent] [info] [INFO] [communicator.cpp:27] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2024-11-27 16:56:27.040] [wazuh-agent] [info] [INFO] [logcollector.cpp:20] [Start] Logcollector is disabled
[2024-11-27 16:56:27.040] [wazuh-agent] [info] [INFO] [inventory.cpp:14] [Start] Module disabled. Exiting...

I assign it to the group validYaml2
{"commands":[{"id":"id12345","status":"sent","info":"string","args":["CentralizedConfiguration","set-group",["validYaml2"]],"agent":{"id":"agentID"}}]}

validYaml2 content:

logcollector:
  enabled: true
  localfiles:
    - /var/log/auth3.log
  reload_interval: 120
inventory:
  enabled: true
  interval: 120
  scan_on_start: false
  hardware: true
  os: false
  network: true
  packages: false
  ports: false
  ports_all: true
  processes: true
  hotfixes: true

processing of set-group:

[2024-11-27 17:00:31.087] [wazuh-agent] [info] [INFO] [command_handler_utils.cpp:74] [DispatchCommand] Dispatching command set-group(CentralizedConfiguration)
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [configuration_parser.cpp:225] [ReloadConfiguration] ReloadConfiguration.
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [configuration_parser.cpp:238] [ReloadConfiguration] ReloadConfiguration done.
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [agent.cpp:68] [ReloadModules] Reloading Modules
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [inventory.cpp:61] [Stop] Module stopped.
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [logcollector.cpp:52] [Stop] Logcollector stopped
[2024-11-27 17:00:31.096] [wazuh-agent] [info] [INFO] [moduleManager.cpp:42] [Setup] Modulemanager
[2024-11-27 17:00:31.097] [wazuh-agent] [info] [INFO] [inventory.cpp:18] [Start] Starting inventory.
[2024-11-27 17:00:31.097] [wazuh-agent] [info] [INFO] [inventory.cpp:101] [ShowConfig] {"inventory":{"enabled":"yes","scan-on-start":"no","interval":120000,"network":"yes","os":"no","hardware":"yes","packages":"no","ports":"no","ports_all":"yes","processes":"yes"}}
[2024-11-27 17:00:31.097] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:868] [SyncLoop] Module started.
[2024-11-27 17:00:31.097] [wazuh-agent] [info] [INFO] [logcollector.cpp:24] [Start] Logcollector started
[2024-11-27 17:00:31.103] [wazuh-agent] [info] [INFO] [command_handler.hpp:68] [operator()] Done processing command: set-group(CentralizedConfiguration)

content of configuration loaded by configparser with local and shared config:

agent:
  server_url: http://localhost:27000/
  retry_interval: 3ms
inventory:
  enabled: true
  interval: 120
  scan_on_start: false
  hardware: true
  os: false
  network: true
  packages: false
  ports: false
  ports_all: true
  processes: true
  hotfixes: true
logcollector:
  enabled: true
  localfiles:
    - /var/log/auth.log
    - /var/log/auth3.log
  reload_interval: 120
  file_wait: 500

inventory performing scans every 2 min as configured

[2024-11-27 17:00:31.103] [wazuh-agent] [info] [INFO] [command_handler.hpp:68] [operator()] Done processing command: set-group(CentralizedConfiguration)
[2024-11-27 17:01:16.606] [wazuh-agent] [info] [INFO] [communicator.cpp:27] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2024-11-27 17:02:14.907] [wazuh-agent] [info] [INFO] [communicator.cpp:27] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2024-11-27 17:02:31.097] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:851] [Scan] Starting evaluation.
[2024-11-27 17:02:31.442] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:863] [Scan] Evaluation finished.
[2024-11-27 17:03:12.916] [wazuh-agent] [info] [INFO] [communicator.cpp:27] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2024-11-27 17:04:11.099] [wazuh-agent] [info] [INFO] [communicator.cpp:27] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2024-11-27 17:04:31.453] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:851] [Scan] Starting evaluation.
[2024-11-27 17:04:31.637] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:863] [Scan] Evaluation finished.

While the logcollector module stops correctly and the configuration is loaded correctly but when launching the internal subroutines they cannot be launched because the reference to the io_context has been lost:

[2024-11-27 20:12:04.437] [wazuh-agent] [debug] [DEBUG] [http_client.cpp:160] [Co_PerformHttpRequest] Response code: 200.
[2024-11-27 20:12:04.437] [wazuh-agent] [debug] [DEBUG] [http_client.cpp:161] [Co_PerformHttpRequest] Response body: {"commands": [{"id": "id12345", "status": "sent", "info": "string", "args": ["CentralizedConfiguration", "set-group", ["validYaml2"]], "agent": {"id": "agentID"}}]}.
[2024-11-27 20:12:04.658] [wazuh-agent] [info] [INFO] [command_handler_utils.cpp:74] [DispatchCommand] Dispatching command set-group(CentralizedConfiguration)
[2024-11-27 20:12:04.666] [wazuh-agent] [debug] [DEBUG] [http_client.cpp:291] [PerformHttpRequestDownload] Response code: 200.
[2024-11-27 20:12:04.667] [wazuh-agent] [info] [INFO] [configuration_parser.cpp:225] [ReloadConfiguration] ReloadConfiguration.
[2024-11-27 20:12:04.667] [wazuh-agent] [debug] [DEBUG] [configuration_parser.cpp:61] [LoadLocalConfig] Loading local config file: /etc/wazuh-agent/wazuh-agent.yml.
[2024-11-27 20:12:04.667] [wazuh-agent] [info] [INFO] [configuration_parser.cpp:238] [ReloadConfiguration] ReloadConfiguration done.
[2024-11-27 20:12:04.667] [wazuh-agent] [info] [INFO] [agent.cpp:68] [ReloadModules] Reloading Modules
[2024-11-27 20:12:04.667] [wazuh-agent] [info] [INFO] [inventory.cpp:61] [Stop] Module stopped.
[2024-11-27 20:12:04.667] [wazuh-agent] [info] [INFO] [logcollector.cpp:68] [Stop] Logcollector stopped
[2024-11-27 20:12:04.668] [wazuh-agent] [info] [INFO] [logcollector.cpp:29] [EnqueueTask] Logcollector EnqueueTask
[2024-11-27 20:12:04.668] [wazuh-agent] [info] [INFO] [logcollector.cpp:30] [EnqueueTask] io_context is running: false

this means that the task of monitoring each file could not be launched successfully.

2024/11/28

• Solved the logcollector error case, as the io_context was stopped, it was not possible to queue Reader tasks.
• Performing rebase with master, the rebase looked satisfactory, but at the time of re-testing the functionalities is giving me a failure, it seems that the reference to the configurationParser of the coroutine is lost.

2024/11/29

• I made a new rebase
• I was able to solve the above failure case
• I performed manual tests and both logcollector and inventory seem to re-start correctly with the new configuration.

2024/12/02

• I made corrections in the MergeYamlNodes method in the way sequence nodes are merged to avoid duplicate entries.
• I started to make the corrections requested in the review.

2024/12/03

• I continue to make the corrections requested in the review.
• I moved MergeYamlNodes method to a separate utils file, I made UTs for MergeYamlNodes

2024/12/04

• I continue to make the corrections requested in the review.
• I created a wrapper class for the filesystem functions in order to mock these functions and correct the CentralizedConfiguration tests.

2024/12/05

• I applied the last changes requested in the review.
• The PR is ready