Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fortigate-exporter exception error report #266

Closed
happinesslijian opened this issue Jan 3, 2024 · 8 comments
Closed

Fortigate-exporter exception error report #266

happinesslijian opened this issue Jan 3, 2024 · 8 comments
Labels
fortigate-issue Something that is an issue in FortiOS that we need to work around wontfix This will not be worked on

Comments

@happinesslijian
Copy link

I have filtered as follows, it was fine at first, but after waiting for a while, the following problem occurred. I'm not sure how to solve this problem. I read related issues. My prometheus fetches every 30 seconds.

global:
  scrape_interval: 30s

exclude:
  - System/SensorInfo

systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 2h 8min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.1M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de0e0}, 0xc000314100)
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc000711620?}, {0x9d8f98?, 0xc0000de0e0?}, 0x6?)
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c000, {0x9d9b08, 0xc0001b61b0})
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 03 12:37:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 03 12:37:57 prometheus-server fortigate-exporter[4159665]: 2024/01/03 12:37:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.583 seconds

This bothers me a lot, I don’t know what the problem is 😣Fortigate exporter service sometimes works well and sometimes bad

systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 3h 0min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.9M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 13:24:52 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:24:52 Probe of "https://10.0.0.1:10443" succeeded, took 2.689 seconds
1月 03 13:25:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:25:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.588 seconds
1月 03 13:25:50 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:25:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.633 seconds
1月 03 13:26:22 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:26:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.693 seconds
1月 03 13:26:50 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:26:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.641 seconds
1月 03 13:27:26 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:27:26 Probe of "https://10.0.0.1:10443" succeeded, took 7.109 seconds
1月 03 13:27:55 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:27:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.636 seconds
1月 03 13:28:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:28:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.628 seconds
1月 03 13:28:57 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:28:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.624 seconds
1月 03 13:29:25 prometheus-server fortigate-exporter[4159665]: 2024/01/03 13:29:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.615 seconds

systemctl status fortigate-exporter.service 
● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 3h 5min ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 9.1M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001a1800)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc00033ecc0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00023e3f0, {0x9d9b08, 0xc0001b61b0})
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 03 13:35:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb

The intuitive feeling is that the fortigate-exporter service is normal for a while and abnormal for a while. I'm not sure how to solve this problem. Need your help thanks.
@happinesslijian happinesslijian mentioned this issue Jan 3, 2024
Closed
@bluecmd
Copy link
Owner

bluecmd commented Jan 3, 2024

Can you attach the full log? Right now it is only possible to see half the error stack trace.

@happinesslijian
Copy link
Author

Can you attach the full log? Right now it is only possible to see half the error stack trace.

To reiterate, my version is as follows:

Fortigate_exporter Version: 1.23.2
Fortigate Version: v7.0.12
prometheus Version:v2.47.0

Now I use systemctl --no-pager status fortigate-exporter.service | lessto view the information of fortigate-exporter.service, as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-01-03 10:29:46 CST; 23h ago
   Main PID: 4159665 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 11.8M
     CGroup: /system.slice/fortigate-exporter.service
             └─4159665 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0003e6630?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:51 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:51 Probe of "https://10.0.0.1:10443" succeeded, took 2.123 seconds
1月 04 10:08:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.603 seconds

Then I used the command journalctl -u fortigate-exporter.service -n 200 | less and got the following information:

-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Thu 2024-01-04 10:15:55 CST. --
1月 04 10:03:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:03:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.332 seconds
1月 04 10:03:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:03:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.626 seconds
1月 04 10:04:26 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:04:26 Probe of "https://10.0.0.1:10443" succeeded, took 7.122 seconds
1月 04 10:04:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:04:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.612 seconds
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:20 http: panic serving 192.168.16.5:59748: runtime error: slice bounds out of range [1:0]
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: goroutine 115345 [running]:
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c2e8?})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0005a4000}, 0xc0000a04b0?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001f0300, {0x9d9bb0, 0xc00059e000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc00059c000)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc00059c000)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0001b50e0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:05:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:05:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.642 seconds
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:50 http: panic serving 192.168.16.5:53386: runtime error: slice bounds out of range [1:0]
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: goroutine 115999 [running]:
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c438?})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00060a000}, 0xc00046c798?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0006040f0, {0x9d9bb0, 0xc000608000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000600000)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000600000)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004c4300?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac480, {0x9d9b08, 0xc0001b61b0})
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:05:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:05:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:05:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.406 seconds
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:20 http: panic serving 192.168.16.5:52678: runtime error: slice bounds out of range [1:0]
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: goroutine 116489 [running]:
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000a0408?})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc000106000}, 0xc0000a1950?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001f0318, {0x9d9bb0, 0xc000652310}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000684000)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000684000)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0005240f0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c2d0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:06:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.615 seconds
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:50 http: panic serving 192.168.16.5:56724: runtime error: slice bounds out of range [1:0]
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: goroutine 116493 [running]:
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc00046c018?})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e4000}, 0xc00046c618?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00027e2d0, {0x9d9bb0, 0xc0003e2000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc000684400)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc000684400)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004ae030?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c360, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:06:50 http: panic serving 192.168.16.5:49372: runtime error: slice bounds out of range [1:0]
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: goroutine 116606 [running]:
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0001e0048?})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e40c0}, 0xc00046c5b8?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000602138, {0x9d9bb0, 0xc000106000}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de0e0}, 0xc000684000)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de0e0?}, 0x6a79ba?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de0e0}, 0xc000684000)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004d44e0?}, {0x9d8f98?, 0xc0000de0e0?}, 0x6?)
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc0001ac7e0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:06:50 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:20 http: panic serving 192.168.16.5:60994: runtime error: slice bounds out of range [1:0]
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: goroutine 116689 [running]:
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0001e0120?})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc0003e4000}, 0xc0000267c8?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00040e6a8, {0x9d9bb0, 0xc0000b0460}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0001fc200)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0003e6630?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c3f0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:07:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:07:51 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:07:51 Probe of "https://10.0.0.1:10443" succeeded, took 2.123 seconds
1月 04 10:08:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.603 seconds
1月 04 10:08:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:08:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.593 seconds
1月 04 10:09:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:09:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.986 seconds
1月 04 10:09:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:09:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.633 seconds
1月 04 10:10:27 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:10:27 Probe of "https://10.0.0.1:10443" succeeded, took 7.369 seconds
1月 04 10:10:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:10:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.573 seconds
1月 04 10:11:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:11:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.650 seconds
1月 04 10:11:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:11:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.466 seconds
1月 04 10:12:25 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:12:25 Probe of "https://10.0.0.1:10443" succeeded, took 5.583 seconds
1月 04 10:12:50 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:12:50 Probe of "https://10.0.0.1:10443" succeeded, took 0.611 seconds
1月 04 10:13:22 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:13:22 Probe of "https://10.0.0.1:10443" succeeded, took 2.736 seconds
1月 04 10:13:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:13:55 Probe of "https://10.0.0.1:10443" succeeded, took 5.597 seconds
1月 04 10:14:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:14:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.655 seconds
1月 04 10:14:57 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:14:57 Probe of "https://10.0.0.1:10443" succeeded, took 7.956 seconds
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:20 http: panic serving 192.168.16.5:44628: runtime error: slice bounds out of range [1:0]
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: goroutine 116688 [running]:
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000261c8?})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00013e000}, 0xc0001e0018?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000010240, {0x9d9bb0, 0xc00064ff10}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc0004475c0?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00045c6c0, {0x9d9b08, 0xc0001b61b0})
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:15:20 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:20 Probe of "https://10.0.0.1:10443" succeeded, took 0.593 seconds
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:55 http: panic serving 192.168.16.5:39620: runtime error: slice bounds out of range [1:0]
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: goroutine 117308 [running]:
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve.func1()
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:1868 +0xb9
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: panic({0x8e5360?, 0xc0000261c8?})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/panic.go:920 +0x270
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9d6540, 0xc00067e000}, 0xc0001e0090?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x7b1
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc000784108, {0x9d9bb0, 0xc000659b90}, 0x6bb51f?, 0x8d38c0?, {0xc0001b42d0, {0x9181c2, 0x5}, 0x1e, 0xa, ...})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:179 +0xae9
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:49 +0x7db
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.HandlerFunc.ServeHTTP(0x445200?, {0x9d8f98?, 0xc0000de000?}, 0x6a79ba?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136 +0x29
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*ServeMux).ServeHTTP(0xd32020?, {0x9d8f98, 0xc0000de000}, 0xc0003d8000)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2514 +0x142
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.serverHandler.ServeHTTP({0xc00058b740?}, {0x9d8f98?, 0xc0000de000?}, 0x6?)
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938 +0x8e
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: net/http.(*conn).serve(0xc00023e120, {0x9d9b08, 0xc0001b61b0})
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009 +0x5f4
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: created by net/http.(*Server).Serve in goroutine 7
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]:         /opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:3086 +0x5cb
1月 04 10:15:55 prometheus-server fortigate-exporter[4159665]: 2024/01/04 10:15:55 Probe of "https://10.0.0.1:10443" succeeded, took 0.591 seconds

By the way, besides the above two ways to view logs, are there any other ways? @bluecmd

@bluecmd
Copy link
Owner

bluecmd commented Jan 4, 2024

Perfect, thanks!

Here is the issue at hand: runtime error: slice bounds out of range [1:0].
This is happening at github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage, line system_resources_usage.go:63.

fortigate_exporter/pkg/probe/system_resources_usage.go

Lines 61 to 66 in 99e11c0

// CPU[0] is the average over all cores, ignore it
m := []prometheus.Metric{}
for i, cpu := range sr.Results.CPU[1:] {
m = append(m, prometheus.MustNewConstMetric(
mResCPU, prometheus.GaugeValue, float64(cpu.Current)/100.0, fmt.Sprintf("%d", i)))
}

Seems like sr.Results.CPU might contain no specific CPU values occasionally in your case.

@happinesslijian Are you comfortable patching and compiling the fortigate exporter yourself? I would suggest adding a print before this code so we can see how the API response looks like when things break for you.
Alternatively, you can try to fetch https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global when the issue happens and see if you can catch the abnormal API response that way.

@happinesslijian
Copy link
Author

happinesslijian commented Jan 5, 2024 •
edited
Loading

很好，谢谢！

这是当前的问题：runtime error: slice bounds out of range [1:0]。 这发生在github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsagesystem_resources_usage.go:63 行。

fortigate_exporter/pkg/probe/system_resources_usage.go

Lines 61 to 66 in 99e11c0

// CPU[0] is the average over all cores, ignore it
m := []prometheus.Metric{}
for i, cpu := range sr.Results.CPU[1:] {
m = append(m, prometheus.MustNewConstMetric(
mResCPU, prometheus.GaugeValue, float64(cpu.Current)/100.0, fmt.Sprintf("%d", i)))
}

在您的情况下，似乎sr.Results.CPU有时可能不包含特定的 CPU 值。

@happinesslijian您是否愿意自己修补和编译 fortigate 导出器？我建议在此代码之前添加打印内容，以便我们可以看到当您出现问题时 API 响应的样子。 或者，您可以尝试https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global在问题发生时进行获取，看看是否可以通过这种方式捕获异常的 API 响应。

@bluecmd I've done it and compiled it as you asked, adding the print operation on line 61
The Fortigate-exporter file has been compiled and has been run in my environment. The latest log is as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-01-05 12:45:37 CST; 2min 46s ago
   Main PID: 300608 (fortigate-expor)
      Tasks: 10 (limit: 19119)
     Memory: 10.0M
     CGroup: /system.slice/fortigate-exporter.service
             └─300608 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e7050, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: [{0} {0} {0} {0} {0} {0} {2} {0} {0}]
1月 05 12:47:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:53 Probe of "https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global" failed, took 2.589 seconds
1月 05 12:48:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 sr.Results.CPU: [{3} {2} {2} {2} {2} {6} {5} {4} {11}]
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 Probe of "https://10.0.0.1:10443" failed, took 0.478 seconds

-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Fri 2024-01-05 12:49:02 CST. --
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00036e780?}, {0x9c8df0, 0xc000182460}, 0xc000171900)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0004760a0, {0x9c9458, 0xc000128b40})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:20 http: panic serving 192.168.16.5:55700: runtime error: slice bounds out of range [1:0]
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: goroutine 5276 [running]:
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000287e0})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc00010e0c0}, 0xc0000284b0?)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0003ecac8, {0x9c9420, 0xc000531800}, {0xc000028438?, 0xc0000833a8?}, 0xc0002a0384?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc000149af0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc0002a038b?, {0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc0000c5080?}, {0x9c8df0, 0xc000182380}, 0xc00040cb00)
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8f00, {0x9c9458, 0xc000128b40})
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:13:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:13:49 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:13:50 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:13:50 Probe of "https://10.0.0.1:10443" failed, took 0.433 seconds
1月 05 12:14:19 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:20 http: panic serving 192.168.16.5:43206: runtime error: slice bounds out of range [1:0]
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: goroutine 5294 [running]:
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0005321e0})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc000744000}, 0xc000028180?)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0004a41e0, {0x9c9420, 0xc000190ba0}, {0xc0000d1848?, 0xc0000833a8?}, 0xc000524004?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc00041baf0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00052400b?, {0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00018f5f0?}, {0x9c8df0, 0xc000182380}, 0xc00078b800)
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc00016a960, {0x9c9458, 0xc000128b40})
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:20 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:21 http: panic serving 192.168.16.5:52160: runtime error: slice bounds out of range [1:0]
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: goroutine 5325 [running]:
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000d0780})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc0007440c0}, 0xc0000d0798?)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0004667b0, {0x9c9420, 0xc000359e60}, {0xc000028270?, 0xc0000833a8?}, 0xc000454044?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc000145af0?, {0x9c8df0?, 0xc000182460?}, 0x0?)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00045404b?, {0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc00064e3f0?}, {0x9c8df0, 0xc000182460}, 0xc000000200)
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8b40, {0x9c9458, 0xc000128b40})
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:21 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:49 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: 2024/01/05 12:14:50 http: panic serving 192.168.16.5:52766: runtime error: slice bounds out of range [1:0]
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: goroutine 5373 [running]:
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve.func1()
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1850 +0xbf
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: panic({0x8d5c80, 0xc0000d0600})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:890 +0x262
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9c57a0, 0xc00010e0c0}, 0xc0000d0858?)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/system_resources_usage.go:63 +0x765
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0000ac6a8, {0x9c9420, 0xc00007d260}, {0xc0000d07c8?, 0xc0000833a8?}, 0xc000524284?, {0xc00016e9f0, {0x905325, 0x5}, 0x1e, ...})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/probe.go:167 +0x809
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /home/runner/work/fortigate_exporter/fortigate_exporter/pkg/probe/main.go:40 +0x4c6
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.HandlerFunc.ServeHTTP(0xc00014baf0?, {0x9c8df0?, 0xc000182380?}, 0x0?)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2109 +0x2f
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*ServeMux).ServeHTTP(0xc00052428b?, {0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2487 +0x149
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.serverHandler.ServeHTTP({0xc000212720?}, {0x9c8df0, 0xc000182380}, 0xc000170700)
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:2947 +0x30c
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: net/http.(*conn).serve(0xc0000b8dc0, {0x9c9458, 0xc000128b40})
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:1991 +0x607
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]: created by net/http.(*Server).Serve
1月 05 12:14:50 prometheus-server fortigate-exporter[288803]:         /opt/hostedtoolcache/go/1.19.0/x64/src/net/http/server.go:3102 +0x4db
1月 05 12:14:50 prometheus-server systemd[1]: Stopping fortigate-exporter...
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Succeeded.
1月 05 12:14:50 prometheus-server systemd[1]: Stopped fortigate-exporter.
1月 05 12:14:50 prometheus-server systemd[1]: Started fortigate-exporter.
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Main process exited, code=exited, status=203/EXEC
1月 05 12:14:50 prometheus-server systemd[1]: fortigate-exporter.service: Failed with result 'exit-code'.
1月 05 12:45:37 prometheus-server systemd[1]: Started fortigate-exporter.
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 FortigateExporter (devel) ( (no hash) )
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 Loaded 1 API keys
1月 05 12:45:37 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:37 Fortigate exporter running, listening on ":9710"
1月 05 12:45:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:45:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:50 sr.Results.CPU: [{0} {0} {1} {1} {0} {0} {1} {1} {0}]
1月 05 12:45:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:45:53 Probe of "https://10.0.0.1:10443" failed, took 3.314 seconds
1月 05 12:46:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:46:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:20 sr.Results.CPU: [{2} {0} {3} {1} {1} {3} {1} {4} {3}]
1月 05 12:46:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:20 Probe of "https://10.0.0.1:10443" failed, took 0.454 seconds
1月 05 12:46:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:46:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:50 sr.Results.CPU: [{1} {1} {0} {1} {0} {1} {1} {1} {0}]
1月 05 12:46:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:46:50 Probe of "https://10.0.0.1:10443" failed, took 0.444 seconds
1月 05 12:47:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 sr.Results.CPU: []
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 http: panic serving 192.168.16.5:59590: runtime error: slice bounds out of range [1:0]
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: goroutine 5 [running]:
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc0005843a8})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0002b8000}, 0xc000028348?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0002b60f0, {0x9ee298, 0xc00031c120}, 0xc00023a160?, 0xc00032b728?, {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc0001b0080?, {0x9edec0?, 0xc000106000?}, 0x40da8a?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc00011c14b?, {0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc0001a2270?}, {0x9edec0, 0xc000106000}, 0xc0002f2400)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0001cc000, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 sr.Results.CPU: []
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:20 http: panic serving 192.168.16.5:51508: runtime error: slice bounds out of range [1:0]
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: goroutine 189 [running]:
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc000584288})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0002b8180}, 0xc0000281f8?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0005a8360, {0x9ee298, 0xc00016ebd0}, 0xc000151e40?, 0xc0001d6728?,
 {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc0001b0480?, {0x9edec0?, 0xc000106460?}, 0x40da8a?)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc0004004cb?, {0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc00016ea20?}, {0x9edec0, 0xc000106460}, 0xc0001aef00)
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e67e0, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:20 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: []
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 http: panic serving 192.168.16.5:53810: runtime error: slice bounds out of range [1:0]
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: goroutine 223 [running]:
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve.func1()
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1854 +0xbf
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: panic({0x9008c0, 0xc000028360})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/runtime/panic.go:890 +0x263
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemResourceUsage({0x9ea7a0, 0xc0003bc000}, 0xc000028390?)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/system_resources_usage.go:64 +0x7a5
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0005a8438, {0x9ee298, 0xc0003fb4a0}, 0xc000304420?, 0xc00078b728?, {0xc000186540, {0x9317d2, 0x5}, 0x1e, 0xa, ...})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/probe.go:179 +0x9cf
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Users/nxf92576/GolandProjects/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.HandlerFunc.ServeHTTP(0xc00026a280?, {0x9edec0?, 0xc0001069a0?}, 0x40da8a?)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2122 +0x2f
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*ServeMux).ServeHTTP(0xc00040060b?, {0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2500 +0x149
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.serverHandler.ServeHTTP({0xc0003fb3b0?}, {0x9edec0, 0xc0001069a0}, 0xc00036ea00)
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:2936 +0x316
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: net/http.(*conn).serve(0xc0003e7050, {0x9ee2d0, 0xc0001a2180})
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:1995 +0x612
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: created by net/http.(*Server).Serve
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]:         C:/Program Files/Go/src/net/http/server.go:3089 +0x5ed
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:47:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:50 sr.Results.CPU: [{0} {0} {0} {0} {0} {0} {2} {0} {0}]
1月 05 12:47:53 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:47:53 Probe of "https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global" failed, took 2.589 seconds
1月 05 12:48:19 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 sr.Results.CPU: [{3} {2} {2} {2} {2} {6} {5} {4} {11}]
1月 05 12:48:20 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:20 Probe of "https://10.0.0.1:10443" failed, took 0.478 seconds
1月 05 12:48:49 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 12:48:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:50 sr.Results.CPU: [{0} {0} {1} {1} {1} {0} {0} {1} {0}]
1月 05 12:48:50 prometheus-server fortigate-exporter[300608]: 2024/01/05 12:48:50 Probe of "https://10.0.0.1:10443" failed, took 0.436 seconds

I try to access https://10.0.0.1:10443/api/v2/monitor/system/resource/usage?scope=global in the browser and get the following results:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}

{
  "http_method":"GET",
  "results":{
    "cpu":[
      {
        "current":0,
        "historical":{
          "1-min":{
            "values":[
              [
                1704430083000,
                0
              ],
              [
                1704430080000,
                0
              ],
              [
                1704430077000,
                1
              ],
···
···
···
              [
                1704350510000,
                0
              ],
              [
                1704346190000,
                0
              ]
            ],
            "max":0,
            "min":0,
            "average":0,
            "start":1704346190000,
            "end":1704430084000
          }
        }
      }
    ]
  },
  "vdom":"root",
  "path":"system",
  "name":"resource",
  "action":"usage",
  "status":"success",
  "serial":"FG200FT922925682",
  "version":"v7.0.13",
  "build":566
}

Please help me check where the problem occurs. Thank you. @bluecmd

bluecmd added a commit that referenced this issue Jan 5, 2024
@bluecmd
fix: retry system resources probe when malformed
4e9cbb4 
Fixes #266

Signed-off-by: Christian Svensson <[email protected]>
@bluecmd bluecmd mentioned this issue Jan 5, 2024
Closed
@bluecmd
Copy link
Owner

bluecmd commented Jan 5, 2024

From your investigations, this seems to be the weird response:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}

This seems like a bug in FortiOS producing empty resources. We can ignore those replies, or we can retry the fetch.

Can you try the patch in PR #269 to see if it makes things work any better?

@happinesslijian
Copy link
Author

From your investigations, this seems to be the weird response:

{
   "http_method":"GET",
   "results":{
   },
   "vdom":"root",
   "path":"system",
   "name":"resource",
   "action":"usage",
   "status":"success",
   "serial":"FG200FT922925682",
   "version":"v7.0.13",
   "build":566
}

This seems like a bug in FortiOS producing empty resources. We can ignore those replies, or we can retry the fetch.

Can you try the patch in PR #269 to see if it makes things work any better?

Hello, @bluecmd I have rebuilt Fortigate-exporter according to your method and it is running. The problem still exists. The detailed log is as follows:

● fortigate-exporter.service - fortigate-exporter
     Loaded: loaded (/etc/systemd/system/fortigate-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-01-05 22:27:55 CST; 21min ago
   Main PID: 369115 (fortigate-expor)
      Tasks: 11 (limit: 19119)
     Memory: 9.8M
     CGroup: /system.slice/fortigate-exporter.service
             └─369115 /usr/local/bin/fortigate-exporter -insecure --auth-file /data/prometheus/fortigate-key.yaml

1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de24b?, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de090?}, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca000, {0x9bf318, 0xc00046a150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:49:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:22 Probe of "https://10.0.0.1:10443" failed, took 2.404 seconds

-- Logs begin at Sat 2023-10-21 12:56:43 CST, end at Fri 2024-01-05 22:50:18 CST. --
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:43:50 Got system resources, will retry fetch..
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:43:50 http: panic serving 192.168.16.5:39056: runtime error: index out of range [0] with length 0
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: goroutine 1289 [running]:
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0001c60a8})
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0002da0c0}, 0xc0000d4378?)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000c510, {0x9bf2e0, 0xc0004729c0}, 0x0?, 0x2?, {0xc>
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0002101cb?, {0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc00039e420?}, {0x9beea8, 0xc0004dc380}, 0xc000238500)
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc00035a0a0, {0x9bf318, 0xc00046a150})
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:43:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:44:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:20 Got system resources, will retry fetch..
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:21 Error: Out of attempts to fetch system resources, giving up
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:21 http: panic serving 192.168.16.5:59406: runtime error: index out of range [0] with length 0
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: goroutine 1310 [running]:
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0001c6108})
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001fc000}, 0xc0005a1690?)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000dc98, {0x9bf2e0, 0xc0004e73e0}, 0x0?, 0xb?, {0xc>
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004600e0?}, 0x40d425?)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0002102cb?, {0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002b6930?}, {0x9beea8, 0xc0004600e0}, 0xc000238800)
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000bee60, {0x9bf318, 0xc00046a150})
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:44:21 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:44:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:44:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:50 Got system resources, will retry fetch..
1月 05 22:44:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:50 Got system resources, will retry fetch..
1月 05 22:44:52 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:44:52 Probe of "https://10.0.0.1:10443" failed, took 3.111 seconds
1月 05 22:45:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:45:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:20 Probe of "https://10.0.0.1:10443" failed, took 0.418 seconds
1月 05 22:45:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:45:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:45:50 Probe of "https://10.0.0.1:10443" failed, took 0.359 seconds
1月 05 22:46:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:22 Probe of "https://10.0.0.1:10443" failed, took 2.352 seconds
1月 05 22:46:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:50 Got system resources, will retry fetch..
1月 05 22:46:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:50 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Out of attempts to fetch system resources, giving up
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 http: panic serving 192.168.16.5:42864: runtime error: index out of range [0] with length 0
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: goroutine 1361 [running]:
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0000d4420})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001e8000}, 0xc000024210?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0001d00f0, {0x9bf2e0, 0xc0000b7ec0}, 0x0?, 0xc0006136>
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004601c0?}, 0x40d425?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de14b?, {0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de120?}, {0x9beea8, 0xc0004601c0}, 0xc000127800)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca5a0, {0x9bf318, 0xc00046a150})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Got system resources, will retry fetch..
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 Error: Out of attempts to fetch system resources, giving up
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:46:51 http: panic serving 192.168.16.5:46512: runtime error: index out of range [0] with length 0
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: goroutine 1530 [running]:
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0000240d8})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001e80c0}, 0xc0005989e0?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000cea0, {0x9bf2e0, 0xc0006c37a0}, 0x0?, 0xb?, {0xc>
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de60b?, {0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0004a8150?}, {0x9beea8, 0xc0004dc380}, 0xc000338a00)
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000bf0e0, {0x9bf318, 0xc00046a150})
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:46:51 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:47:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Got system resources, will retry fetch..
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 Error: Out of attempts to fetch system resources, giving up
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:20 http: panic serving 192.168.16.5:40386: runtime error: index out of range [0] with length 0
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: goroutine 1540 [running]:
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc0004d6240})
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc0001d4000}, 0xc0004d6408?)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00058c930, {0x9bf2e0, 0xc0004e6cc0}, 0x0?, 0xb?, {0xc>
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc460?}, 0x40d425?)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de2cb?, {0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc000167b30?}, {0x9beea8, 0xc0004dc460}, 0xc000126400)
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca0a0, {0x9bf318, 0xc00046a150})
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:47:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:47:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Got system resources, will retry fetch..
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 Error: Out of attempts to fetch system resources, giving up
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:47:50 http: panic serving 192.168.16.5:49440: runtime error: index out of range [0] with length 0
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: goroutine 1601 [running]:
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024168})
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00032a000}, 0xc0004d6300?)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00000c618, {0x9bf2e0, 0xc0006c26c0}, 0x0?, 0xb?, {0xc>
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de70b?, {0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc000385410?}, {0x9beea8, 0xc0004dc380}, 0xc000310800)
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0000befa0, {0x9bf318, 0xc00046a150})
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:47:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:48:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:20 Got system resources, will retry fetch..
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:20 http: panic serving 192.168.16.5:48902: runtime error: index out of range [0] with length 0
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: goroutine 1593 [running]:
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024228})
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00038c000}, 0xc000024228?)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc00058c678, {0x9bf2e0, 0xc000498900}, 0x0?, 0xb?, {0xc>
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc460?}, 0x40d425?)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc00027e44b?, {0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0001b4cf0?}, {0x9beea8, 0xc0004dc460}, 0xc000126b00)
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc000122780, {0x9bf318, 0xc00046a150})
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:20 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:48:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:50 Got system resources, will retry fetch..
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:48:50 http: panic serving 192.168.16.5:45376: runtime error: index out of range [0] with length 0
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: goroutine 1629 [running]:
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve.func1()
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1825 +0xbf
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: panic({0x8ce6e0, 0xc000024150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/runtime/panic.go:844 +0x258
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.probeSystemVDOMResources({0x9bbaa0, 0xc00038c0c0}, 0xc000024138?)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/system_resources_usage.go:136 +0x7f8
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.(*ProbeCollector).Probe(0xc0002121b0, {0x9bf2e0, 0xc0004e6600}, 0x0?, 0xc0004646>
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/probe.go:179 +0x9b9
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: github.com/bluecmd/fortigate_exporter/pkg/probe.ProbeHandler({0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /root/go/fortigate_exporter/pkg/probe/main.go:49 +0x85b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.HandlerFunc.ServeHTTP(0x7f64008f1df8?, {0x9beea8?, 0xc0004dc380?}, 0x40d425?)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2084 +0x2f
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*ServeMux).ServeHTTP(0xc0000de24b?, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2462 +0x149
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.serverHandler.ServeHTTP({0xc0002de090?}, {0x9beea8, 0xc0004dc380}, 0xc000126000)
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:2916 +0x43b
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: net/http.(*conn).serve(0xc0004ca000, {0x9bf318, 0xc00046a150})
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:1966 +0x5d7
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]: created by net/http.(*Server).Serve
1月 05 22:48:50 prometheus-server fortigate-exporter[369115]:         /usr/local/go/src/net/http/server.go:3071 +0x4db
1月 05 22:49:19 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:19 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:22 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:22 Probe of "https://10.0.0.1:10443" failed, took 2.404 seconds
1月 05 22:49:49 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:49 Error: Response code was 424, expected 200 (path: "api/v2/monitor/log/fortianalyzer")
1月 05 22:49:50 prometheus-server fortigate-exporter[369115]: 2024/01/05 22:49:50 Probe of "https://10.0.0.1:10443" failed, took 0.414 seconds

@happinesslijian
Copy link
Author

Hello, @bluecmd is there any other solution? My problem here still exists.😣

@bluecmd
Copy link
Owner

bluecmd commented Jan 8, 2024

@happinesslijian Given the API responses you have shown there is nothing the exporter can do - your fortigate returns invalid data. As long as it does that, we cannot produce metrics.

I would suggest opening an issue with Fortinet and/or upgrading FortiOS version and see if the problem goes away.

@bluecmd bluecmd added wontfix This will not be worked on fortigate-issue Something that is an issue in FortiOS that we need to work around labels Jan 8, 2024
@bluecmd bluecmd closed this as completed Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fortigate-issue Something that is an issue in FortiOS that we need to work around wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: retry system resources probe when malformed bluecmd/fortigate_exporter
2 participants
@bluecmd @happinesslijian