Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linux agents gets unhealthy on enabling/disabling modules for System/Linux integration. #3654

Closed
amolnater-qasource opened this issue Oct 25, 2023 · 10 comments · Fixed by #3747
Closed

Linux agents gets unhealthy on enabling/disabling modules for System/Linux integration. #3654

amolnater-qasource opened this issue Oct 25, 2023 · 10 comments · Fixed by #3747
Assignees
@leehinman
Labels
bug Something isn't working impact:medium QA:Validated Validated by the QA Team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Comments

@amolnater-qasource
Copy link

Kibana Build details:

VERSION: 8.11.0 BC3
BUILD: 67923
COMMIT: 714189fa2b0f0a4d9f3865a8fce08261211570c8

Host OS: Linux

Preconditions:

  1. 8.11.0 BC3 Kibana cloud environment should be available.
  2. Few Linux agents- .tar and .deb agents should be installed.

Steps to reproduce:

  1. Add System and linux integrations to the agent policy.
  2. Navigate to integration settings and disable the integration modules.
  3. Wait for policy to get updated.
  4. Observe few linux agents gets unhealthy inconsistently.
  5. Restart the disabled modules and wait for policy to get updated.
  6. Observe few linux agents again gets unhealthy inconsistently.

NOTE:

  • We are consistently able to reproduce this issue for linux agents.
  • Issue is reproducible for enabling/disabling system and linux integrations.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2023-10-25.12-49-29.mp4

Expected Result:
Linux agents should remain Healthy on enabling/disabling modules for System/Linux integration.

Agent.json:
ip-172-31-66-47-agent-details.zip
ip-172-31-79-43-agent-details.zip

Logs:
elastic-agent-diagnostics-2023-10-25T07-20-34Z-00.zip
elastic-agent-diagnostics-2023-10-25T07-27-25Z-00.zip
@amolnater-qasource amolnater-qasource added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team impact:medium labels Oct 25, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@amolnater-qasource
Copy link
Author

@manishgupta-qasource Please review.

@manishgupta-qasource
Copy link

Secondary review for this ticket is Done

@cmacknz
Copy link
Member

cmacknz commented Oct 25, 2023

From ip-172-31-66-47-agent-details.zip:

    {
      "id": "beat/metrics-monitoring",
      "type": "beat/metrics",
      "status": "DEGRADED",
      "message": "Degraded: pid '1582' missed 1 check-in",
      "units": [

Same for ip-172-31-79-43-agent-details.zip

    {
      "id": "system/metrics-default",
      "type": "system/metrics",
      "status": "DEGRADED",
      "message": "Degraded: pid '996' missed 1 check-in",
      "units": [
        {
          "id": "system/metrics-default-system/metrics-system-a4cc1da8-f6cd-4ab6-b61b-045da2b42479",
          "type": "input",
          "status": "HEALTHY",
          "message": "Healthy"
        },
        {
          "id": "system/metrics-default",
          "type": "output",
          "status": "HEALTHY",
          "message": "Healthy"
        }
      ]
    },

Seems similar to #3617

@cmacknz
Copy link
Member

cmacknz commented Oct 25, 2023

I can reproduce this on my Mac by adding and removing the system integration.

@cmacknz cmacknz mentioned this issue Oct 25, 2023
Closed
@cmacknz
Copy link
Member

cmacknz commented Oct 25, 2023

Likely also the same problem affecting tests run through elastic-package https://github.com/elastic/ingest-dev/issues/2560

@cmacknz
Copy link
Member

cmacknz commented Oct 26, 2023

#3617 (comment) shows policy reassignment (which stops then restarts every input) causes CPU to spike to 100% so probably this is resource utilization from somewhere.

Diagnostics can capture CPU profiles but for me it didn't show anything useful, it might be easier to directly use the /debug/pprof endpoint of agent (and/or the beats via their unix socket paths) so get more control over it.

Also possible this is a deadlock (or temporary deadlock) on either side of the protocol, but the CPU usage spike makes me think it is resource usage first.

@leehinman
Copy link
Contributor

Just an update so far on what I'm seeing.

On debian arm Linux, I don't see a CPU spike.

I see status go from:

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ beat/metrics-monitoring
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '576'
   │  └─ beat/metrics-monitoring-metrics-monitoring-beats
   │     └─ status: (CONFIGURING) Configuring
   ├─ http/metrics-monitoring
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '589'
   │  └─ http/metrics-monitoring-metrics-monitoring-agent
   │     └─ status: (CONFIGURING) Configuring
   ├─ log-default
   │  └─ status: (DEGRADED) Degraded: pid '525' missed 1 check-in
   └─ system/metrics-default
      └─ status: (DEGRADED) Degraded: pid '536' missed 1 check-in

to

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ log-default
   │  └─ status: (DEGRADED) Degraded: pid '525' missed 1 check-in
   └─ system/metrics-default
      └─ status: (DEGRADED) Degraded: pid '536' missed 1 check-in

That persists for 30s, and then status goes back to "normal"

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running

It isn't always the same components that miss a check-in, I have seem beat/metrics-monitoring, log-default and system/metrics-default. But so far it has always been 2 PIDs that miss a check in.

@leehinman
Copy link
Contributor

update with data so far. Using git bissect we know that #3223 made this behavior visible. Going through parts of that change, I was able to determine that it is the proxying of the changes through another channel that is making this behavior visible. But if I add addition debugging statements I can get the behavior to go away. So the proxying through the channel is just changing the timing so we see the problem now.

Steps to re-produce

  1. Add linux host to a policy with system & linux metrics integrations
  2. change the policy to just system integration
  3. Some components end up in Degraded state for 30sec
  4. After 30 sec state becomes Healthy

Initial policy has the following components

  • log-default
  • system/metrics-default
  • filestream-monitoring
  • beat/metrics-monitoring
  • http/metrics-monitoring
  • linux/metrics-default

Timeline

Time Action Evidence
2023-11-07T18:50:13.301Z Agent Started {"log.level":"info","@timestamp":"2023-11-07T18:50:13.301Z","log.origin":{"file.name":"cmd/run.go","file.line":156},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":44112,"agent.version":"8.12.0","ecs.version":"1.6.0"}
2023-11-07T18:52:48.233Z log-default checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:48.233Z","log.logger":"component.runtime.log-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"dd314b11-8544-49f8-8d7a-85beafafea00" units:{id:"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"log-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:48.324Z system/metrics-default checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:48.324Z","log.logger":"component.runtime.system/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"62b89905-317b-4443-8e6b-64d0537e161f" units:{id:"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:48.465Z linux/metrics-default checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:48.465Z","log.logger":"component.runtime.linux/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"a71572cf-ea34-4005-a82c-f80d48dae2c2" units:{id:"linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"linux/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:48.539Z filestream-monitoring checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:48.539Z","log.logger":"component.runtime.filestream-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"bc831bda-c9fc-45e9-aeab-6f6ff03a2811" units:{id:"filestream-monitoring-filestream-monitoring-agent" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"filestream-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:49.868Z http/metrics-monitoring checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:49.868Z","log.logger":"component.runtime.http/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"2c4b039a-34df-45a8-8745-cf0b3fcf3988" units:{id:"http/metrics-monitoring-metrics-monitoring-agent" config_state_idx:2 state:HEALTHY message:"Healthy"} units:{id:"http/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:49.869Z beat/metrics-monitoring checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:49.869Z","log.logger":"component.runtime.beat/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"5922daf0-78d0-45c7-8b96-f5c1f742f3ff" units:{id:"beat/metrics-monitoring-metrics-monitoring-beats" config_state_idx:2 state:HEALTHY message:"Healthy"} units:{id:"beat/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.868Z new policy received {"log.level":"info","@timestamp":"2023-11-07T18:52:54.868Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1119},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z log-default stopped {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7 (HEALTHY->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z new log-default started {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":587},"message":"Spawned new unit log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52: Starting","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z metricbeat receives termination signal {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","message":"Received signal "terminated", stopping","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"linux/metrics-default","type":"linux/metrics"},"log":{"source":"linux/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"service","log.origin":{"file.line":52,"file.name":"service/service.go"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z metricbeat stops linux/metrics-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","message":"Stopping 2 runners ...","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"linux/metrics-default","type":"linux/metrics"},"log":{"source":"linux/metrics-default"},"log.logger":"centralmgmt","log.origin":{"file.line":188,"file.name":"cfgfile/list.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z linux/metrics-default checkin stopping {"log.level":"error","@timestamp":"2023-11-07T18:52:54.876Z","log.logger":"component.runtime.linux/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"a71572cf-ea34-4005-a82c-f80d48dae2c2" units:{id:"linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"linux/metrics-default" type:OUTPUT config_state_idx:1 state:STOPPING message:"Stopping"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z linux/metrics-default checkin stopping {"log.level":"error","@timestamp":"2023-11-07T18:52:54.876Z","log.logger":"component.runtime.linux/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"a71572cf-ea34-4005-a82c-f80d48dae2c2" units:{id:"linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"linux/metrics-default" type:OUTPUT config_state_idx:1 state:STOPPING message:"Stopping"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z filebeat state change triggered log-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","message":"BeatV2Manager.unitListen UnitChanged.ID(log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7), UnitChanged.Type(modified), UnitChanged.Trigger(16): modified/state_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z filebeat feature change triggered log-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","message":"BeatV2Manager.unitListen UnitChanged.ID(log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
2023-11-07T18:52:54.876Z metricbeat config change triggered beat/metrics-monitoring {"log.level":"info","@timestamp":"2023-11-07T18:52:54.876Z","message":"BeatV2Manager.unitListen UnitChanged.ID(beat/metrics-monitoring-metrics-monitoring-beats), UnitChanged.Type(modified), UnitChanged.Trigger(2): modified/config_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z beat/metrics-monitoring checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:54.877Z","log.logger":"component.runtime.beat/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"5922daf0-78d0-45c7-8b96-f5c1f742f3ff" units:{id:"beat/metrics-monitoring-metrics-monitoring-beats" config_state_idx:3 state:CONFIGURING message:"Configuring"} units:{id:"beat/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z log-default checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:54.877Z","log.logger":"component.runtime.log-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"dd314b11-8544-49f8-8d7a-85beafafea00" units:{id:"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"log-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 message:"Starting"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z metricbeat mmodified/state_change_triggered system/metrics-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.877Z","message":"BeatV2Manager.unitListen UnitChanged.ID(system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7), UnitChanged.Type(modified), UnitChanged.Trigger(16): modified/state_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z metricbeat modified/state_change_triggered system/metrics-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.877Z","message":"BeatV2Manager.unitListen UnitChanged.ID(system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a), UnitChanged.Type(modified), UnitChanged.Trigger(16): modified/state_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z metricbeat added/feature_change_triggered system/metrics-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.877Z","message":"BeatV2Manager.unitListen UnitChanged.ID(system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52), UnitChanged.Type(added), UnitChanged.Trigger(4): added/feature_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z system/metrics-default checkin Healthy, stopping system/metrics-default {"log.level":"error","@timestamp":"2023-11-07T18:52:54.877Z","log.logger":"component.runtime.system/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"62b89905-317b-4443-8e6b-64d0537e161f" units:{id:"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"system/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 message:"Starting"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.877Z metricbeat modified/state_change_triggered http/metrics-monitoring {"log.level":"info","@timestamp":"2023-11-07T18:52:54.877Z","message":"BeatV2Manager.unitListen UnitChanged.ID(http/metrics-monitoring-metrics-monitoring-agent), UnitChanged.Type(modified), UnitChanged.Trigger(2): modified/config_change_triggered","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":486,"file.name":"management/managerV2.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
2023-11-07T18:52:54.878Z http/metrics-monitoriing checkin Healthy, configuring {"log.level":"error","@timestamp":"2023-11-07T18:52:54.878Z","log.logger":"component.runtime.http/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"2c4b039a-34df-45a8-8745-cf0b3fcf3988" units:{id:"http/metrics-monitoring-metrics-monitoring-agent" config_state_idx:3 state:CONFIGURING message:"Configuring"} units:{id:"http/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:54.879Z metricbeat stopping linux/metrics-default {"log.level":"info","@timestamp":"2023-11-07T18:52:54.879Z","message":"Stopping metrics logging.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"linux/metrics-default","type":"linux/metrics"},"log":{"source":"linux/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"monitoring","log.origin":{"file.line":163,"file.name":"log/log.go"},"ecs.version":"1.6.0"}
2023-11-07T18:52:55.891Z log-default checkin Healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:55.891Z","log.logger":"component.runtime.log-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"dd314b11-8544-49f8-8d7a-85beafafea00" units:{id:"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPED message:"Stopped"} units:{id:"log-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:52:55.886Z filebeat input reload for log-default {"log.level":"info","@timestamp":"2023-11-07T18:52:55.886Z","message":"Configured paths: [/var/log/auth.log* /var/log/secure*]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"service.name":"filebeat","input_id":"c672da4e-3406-4233-8527-9fd05d672b6e","ecs.version":"1.6.0","log.logger":"input","log.origin":{"file.line":174,"file.name":"log/input.go"},"ecs.version":"1.6.0"}
2023-11-07T18:52:55.891Z http/metrics-monitoring checkin healthy {"log.level":"error","@timestamp":"2023-11-07T18:52:55.891Z","log.logger":"component.runtime.http/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"2c4b039a-34df-45a8-8745-cf0b3fcf3988" units:{id:"http/metrics-monitoring-metrics-monitoring-agent" config_state_idx:3 state:HEALTHY message:"Healthy"} units:{id:"http/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:09.881Z timeout for linux/metrics-default to stop {"log.level":"error","@timestamp":"2023-11-07T18:53:09.881Z","log.origin":{"file.name":"runtime/manager.go","file.line":772},"message":"timeout exceeded waiting for component "linux/metrics-default" to stop","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
2023-11-07T18:53:11.828Z system/metrics-default state change HEALTHY->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:11.828Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7 (HEALTHY->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:11.829Z system/metrics-default state change HEALTHY->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:11.829Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a (HEALTHY->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a","type":"input","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:11.829Z system/metrics-default new unit Starting {"log.level":"info","@timestamp":"2023-11-07T18:53:11.829Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":587},"message":"Spawned new unit system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52: Starting","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:11.829Z system/metrics-default checkin, Stopping, Healthy {"log.level":"error","@timestamp":"2023-11-07T18:53:11.829Z","log.logger":"component.runtime.system/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"62b89905-317b-4443-8e6b-64d0537e161f" units:{id:"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPING message:"Stopping"} units:{id:"system/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 message:"Starting"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:11.829Z system/metrics-default checkin, Stopped, Healthy {"log.level":"error","@timestamp":"2023-11-07T18:53:11.829Z","log.logger":"component.runtime.system/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"62b89905-317b-4443-8e6b-64d0537e161f" units:{id:"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPED message:"Stopped"} units:{id:"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPED message:"Stopped"} units:{id:"system/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:11.836Z updating component model {"log.level":"info","@timestamp":"2023-11-07T18:53:11.836Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1119},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
2023-11-07T18:53:13.536Z filestream-monitoring checkin, Healthy {"log.level":"error","@timestamp":"2023-11-07T18:53:13.536Z","log.logger":"component.runtime.filestream-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"bc831bda-c9fc-45e9-aeab-6f6ff03a2811" units:{id:"filestream-monitoring-filestream-monitoring-agent" config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"filestream-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:26.847Z timeout for linux/metrics-default to stop {"log.level":"error","@timestamp":"2023-11-07T18:53:26.847Z","log.origin":{"file.name":"runtime/manager.go","file.line":772},"message":"timeout exceeded waiting for component "linux/metrics-default" to stop","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.391Z linux/metrics-default state change HEALTHY->STOPPING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.391Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed linux/metrics-default (HEALTHY->STOPPING): Stopping","log":{"source":"elastic-agent"},"component":{"id":"linux/metrics-default","state":"HEALTHY"},"unit":{"id":"linux/metrics-default","type":"output","state":"STOPPING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.391Z linux/metrics-default state change HEALTHY->STOPPING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.391Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a (HEALTHY->STOPPING): Stopping","log":{"source":"elastic-agent"},"component":{"id":"linux/metrics-default","state":"HEALTHY"},"unit":{"id":"linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a","type":"input","state":"STOPPING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.395Z updating component model {"log.level":"info","@timestamp":"2023-11-07T18:53:27.395Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1119},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.724Z beat/metrics-monitoring state change HEALTHY->CONFIGURING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.724Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (HEALTHY->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.725Z log-default STOPPED->STOPPING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.725Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7 (STOPPED->STOPPING): Stopping","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPING","old_state":"STOPPED"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.726Z http/metrics-monitoring state change HEALTHY->CONFIGURING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.726Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed http/metrics-monitoring-metrics-monitoring-agent (HEALTHY->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"http/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"http/metrics-monitoring-metrics-monitoring-agent","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.726Z system/metrics-default state change STOPPED->STOPPING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.726Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7 (STOPPED->STOPPING): Stopping","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPING","old_state":"STOPPED"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.726Z system/metrics-default state change STOPPED->STOPPING {"log.level":"info","@timestamp":"2023-11-07T18:53:27.726Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a (STOPPED->STOPPING): Stopping","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a","type":"input","state":"STOPPING","old_state":"STOPPED"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.727Z linux/metrics-default state change HEALTHY->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.727Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":577},"message":"Component state changed linux/metrics-default (HEALTHY->STOPPED): Stopped: pid '44292' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"linux/metrics-default","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.728Z linux/metrics-default state change STOPPING->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.728Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed linux/metrics-default (STOPPING->STOPPED): Stopped: pid '44292' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"linux/metrics-default","state":"STOPPED"},"unit":{"id":"linux/metrics-default","type":"output","state":"STOPPED","old_state":"STOPPING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.728Z linux/metrics-default state change STOPPING->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.728Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a (STOPPING->STOPPED): Stopped: pid '44292' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"linux/metrics-default","state":"STOPPED"},"unit":{"id":"linux/metrics-default-linux/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a","type":"input","state":"STOPPED","old_state":"STOPPING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.728Z log-default state change STARTING->HEALTHY {"log.level":"info","@timestamp":"2023-11-07T18:53:27.728Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52 (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.728Z log-default state change STOPPTING->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.728Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7 (STOPPING->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPED","old_state":"STOPPING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.731Z system/metrics-default state change STARTING->HEALTHY {"log.level":"info","@timestamp":"2023-11-07T18:53:27.731Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52 (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.731Z system/metrics-default state change STOPPING->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.731Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7 (STOPPING->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7","type":"input","state":"STOPPED","old_state":"STOPPING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.732Z system/metrics-default state change STOPPING->STOPPED {"log.level":"info","@timestamp":"2023-11-07T18:53:27.732Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a (STOPPING->STOPPED): Stopped","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"HEALTHY"},"unit":{"id":"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a","type":"input","state":"STOPPED","old_state":"STOPPING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.732Z beats/metrics-monitoring state change CONFIGURING->HEALTHY {"log.level":"info","@timestamp":"2023-11-07T18:53:27.732Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.732Z http/metrics-monitoring state change CONFIGURING->HEALTHY {"log.level":"info","@timestamp":"2023-11-07T18:53:27.732Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":595},"message":"Unit state changed http/metrics-monitoring-metrics-monitoring-agent (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"http/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"http/metrics-monitoring-metrics-monitoring-agent","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.733Z post send (pre at 2023-11-07T18:53:11.829Z) to checkinObserverd system/metrics-default Stopped {"log.level":"error","@timestamp":"2023-11-07T18:53:27.733Z","log.logger":"component.runtime.system/metrics-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"62b89905-317b-4443-8e6b-64d0537e161f" units:{id:"system/metrics-default-system/metrics-system-1983394c-8dec-43bd-9739-9aacce986da7" config_state_idx:1 state:STOPPED message:"Stopped"} units:{id:"system/metrics-default-system/metrics-system-b948514c-967d-478c-8450-8cf2a9faaf8a" config_state_idx:1 state:STOPPED message:"Stopped"} units:{id:"system/metrics-default" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} units:{id:"system/metrics-default-system/metrics-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52" config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:27.732Z post send (pre at 2023-11-07T18:53:20.887Z) to checkinObserved beat/metrics-monitoring Healthy {"log.level":"error","@timestamp":"2023-11-07T18:53:27.732Z","log.logger":"component.runtime.beat/metrics-monitoring","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:"5922daf0-78d0-45c7-8b96-f5c1f742f3ff" units:{id:"beat/metrics-monitoring-metrics-monitoring-beats" config_state_idx:3 state:HEALTHY message:"Healthy"} units:{id:"beat/metrics-monitoring" type:OUTPUT config_state_idx:1 state:HEALTHY message:"Healthy"} features_idx:2 component_idx:2","ecs.version":"1.6.0"}
2023-11-07T18:53:27.736Z log-default DEGRADED {"log.level":"warn","@timestamp":"2023-11-07T18:53:27.736Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":577},"message":"Component state changed log-default (HEALTHY->DEGRADED): Degraded: pid '44269' missed 1 check-in","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
2023-11-07T18:53:27.736Z http/metrics-monitoring DEGRADED {"log.level":"warn","@timestamp":"2023-11-07T18:53:27.736Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":577},"message":"Component state changed http/metrics-monitoring (HEALTHY->DEGRADED): Degraded: pid '44329' missed 1 check-in","log":{"source":"elastic-agent"},"component":{"id":"http/metrics-monitoring","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}

log-default processing was stuck trying to write to the checkinObserved channel for 30sec

666:{"log.level":"error","@timestamp":"2023-11-07T18:52:55.891Z","log.logger":"component.runtime.log-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":281},"message":"LEH pre send to checkingObserverd: token:\"dd314b11-8544-49f8-8d7a-85beafafea00\"  units:{id:\"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7\"  config_state_idx:1  state:STOPPED  message:\"Stopped\"}  units:{id:\"log-default\"  type:OUTPUT  config_state_idx:1  state:HEALTHY  message:\"Healthy\"}  units:{id:\"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52\"  config_state_idx:1  state:HEALTHY  message:\"Healthy\"}  features_idx:2  component_idx:2","ecs.version":"1.6.0"}
875:{"log.level":"error","@timestamp":"2023-11-07T18:53:27.739Z","log.logger":"component.runtime.log-default","log.origin":{"file.name":"runtime/runtime_comm.go","file.line":283},"message":"LEH post send to checkingObserverd: token:\"dd314b11-8544-49f8-8d7a-85beafafea00\"  units:{id:\"log-default-logfile-system-1983394c-8dec-43bd-9739-9aacce986da7\"  config_state_idx:1  state:STOPPED  message:\"Stopped\"}  units:{id:\"log-default\"  type:OUTPUT  config_state_idx:1  state:HEALTHY  message:\"Healthy\"}  units:{id:\"log-default-logfile-system-48ef55b9-a50c-4ea8-8f3a-1d3cdfbf4e52\"  config_state_idx:1  state:HEALTHY  message:\"Healthy\"}  features_idx:2  component_idx:2","ecs.version":"1.6.0"}

@cmacknz cmacknz mentioned this issue Nov 9, 2023
Closed
@leehinman leehinman mentioned this issue Nov 10, 2023
Closed
@faec faec mentioned this issue Nov 15, 2023
Merged
7 tasks
@amolnater-qasource amolnater-qasource added the QA:Ready For Testing Code is merged and ready for QA to validate label Nov 19, 2023
@amolnater-qasource
Copy link
Author

Hi Team,

We have revalidated this issue on latest 8.12.0 BC5 kibana cloud environment and found it fixed now.

Observations:

  • Linux agents remains healthy on enabling/disabling modules for System/Linux integration.

Build details:
VERSION: 8.12.0
BUILD: 70053
COMMIT: db9b8921b37139cbb1e11d23f6381f655edeb72b
Artifact Link: https://staging.elastic.co/8.12.0-9f05a310/downloads/beats/elastic-agent/elastic-agent-8.12.0-windows-x86_64.zip

Screenshot:
image

Hence we are closing this issue and marking as QA:Validated.

Thanks!

@amolnater-qasource amolnater-qasource added QA:Validated Validated by the QA Team and removed QA:Ready For Testing Code is merged and ready for QA to validate labels Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
bug Something isn't working impact:medium QA:Validated Validated by the QA Team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Rework runtime manager updates to block the coordinator less faec/elastic-agent
5 participants
@cmacknz @elasticmachine @leehinman @manishgupta-qasource @amolnater-qasource