fix metrics server shutdown #761
Conversation
janezpodhostnik
requested review from
peterargue,
m-Peter and
zhangchiqing
as code owners
WalkthroughThe changes introduce a new Changes
Sequence Diagram(s)sequenceDiagram
participant B as Bootstrap
participant MW as metricsWrapper
participant MS as flowMetrics.Server
B->>+MW: newMetricsWrapper(logger, port)
B->>+MW: Start(ctx)
MW->>MS: Initialize and start metrics server
MS-->>MW: Metrics server running
B->>MW: Stop()
MW->>MS: Shutdown metrics server
Suggested labels
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🧰 Additional context used🪛 golangci-lint (1.62.2)bootstrap/bootstrap.go708-708: m.Ready undefined (type *metricsWrapper has no field or method Ready) (typecheck) 742-742: m.Done undefined (type *metricsWrapper has no field or method Done) (typecheck) ⏰ Context from checks skipped due to timeout of 90000ms (1)
🔇 Additional comments (4)
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
bootstrap/bootstrap.go(3 hunks)
🧰 Additional context used
🪛 golangci-lint (1.62.2)
bootstrap/bootstrap.go
708-708: m.Ready undefined (type *metricsWrapper has no field or method Ready)
(typecheck)
737-737: m.Done undefined (type *metricsWrapper has no field or method Done)
(typecheck)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: Test
🔇 Additional comments (3)
bootstrap/bootstrap.go (3)
75-75: LGTM! Good encapsulation of metrics server.The change to use
metricsWrapperimproves the lifecycle management of the metrics server.
374-378: LGTM! Improved metrics server lifecycle management.The changes simplify the lifecycle methods while improving error handling and shutdown logic through proper delegation to the wrapper.
Also applies to: 380-382
729-738: LGTM! Clean shutdown implementation.The Stop method properly handles nil checks and ensures the server is fully stopped by waiting for the Done signal.
🧰 Tools
🪛 golangci-lint (1.62.2)
737-737: m.Done undefined (type *metricsWrapper has no field or method Done)
(typecheck)
| // metricsWrapper is needed since the metric server is a component. | ||
| // We need to start and stop it manually. | ||
| type metricsWrapper struct { | ||
| *flowMetrics.Server | ||
| log zerolog.Logger | ||
| stopFN func() | ||
| } |
There was a problem hiding this comment.
Fix method accessibility in metricsWrapper.
The wrapper embeds *flowMetrics.Server but doesn't properly expose the Ready() and Done() methods, causing compilation errors. Add method forwarding to fix this.
Apply this diff:
type metricsWrapper struct {
*flowMetrics.Server
log zerolog.Logger
stopFN func()
}
+// Ready forwards to the embedded Server's Ready method
+func (m *metricsWrapper) Ready() <-chan struct{} {
+ return m.Server.Ready()
+}
+// Done forwards to the embedded Server's Done method
+func (m *metricsWrapper) Done() <-chan struct{} {
+ return m.Server.Done()
+}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // metricsWrapper is needed since the metric server is a component. | |
| // We need to start and stop it manually. | |
| type metricsWrapper struct { | |
| *flowMetrics.Server | |
| log zerolog.Logger | |
| stopFN func() | |
| } | |
| // metricsWrapper is needed since the metric server is a component. | |
| // We need to start and stop it manually. | |
| type metricsWrapper struct { | |
| *flowMetrics.Server | |
| log zerolog.Logger | |
| stopFN func() | |
| } | |
| // Ready forwards to the embedded Server's Ready method | |
| func (m *metricsWrapper) Ready() <-chan struct{} { | |
| return m.Server.Ready() | |
| } | |
| // Done forwards to the embedded Server's Done method | |
| func (m *metricsWrapper) Done() <-chan struct{} { | |
| return m.Server.Done() | |
| } |
bootstrap/bootstrap.go
Outdated
| func (m *metricsWrapper) Start(ctx context.Context) error { | ||
| m.log.Info().Msg("bootstrap starting metrics server") | ||
|
|
||
| ctx, cancel := context.WithCancel(ctx) | ||
| m.stopFN = cancel | ||
| ictx, errCh := irrecoverable.WithSignaler(ctx) | ||
| m.Server.Start(ictx) | ||
| if err := util.WaitClosed(ctx, m.Ready()); err != nil { | ||
| return fmt.Errorf("failed to start metrics server: %w", err) | ||
| } | ||
| select { | ||
| case err := <-errCh: | ||
| // there might be an error already if the startup failed | ||
| return err | ||
| default: | ||
| } | ||
|
|
||
| go func() { | ||
| err := <-errCh | ||
| if err != nil { | ||
| m.log.Err(err).Msg("error in metrics server") | ||
| panic(err) | ||
| } | ||
| }() | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Improve error handling in Start method.
The error handling could be improved to ensure proper cleanup in case of failures:
- The context cancellation function should be called if startup fails
- The panic in the goroutine could lead to unhandled crashes
Apply this diff:
func (m *metricsWrapper) Start(ctx context.Context) error {
m.log.Info().Msg("bootstrap starting metrics server")
ctx, cancel := context.WithCancel(ctx)
- m.stopFN = cancel
+ defer func() {
+ if err := recover(); err != nil {
+ cancel()
+ m.log.Error().Interface("panic", err).Msg("panic in metrics server")
+ }
+ }()
ictx, errCh := irrecoverable.WithSignaler(ctx)
m.Server.Start(ictx)
if err := util.WaitClosed(ctx, m.Ready()); err != nil {
+ cancel()
return fmt.Errorf("failed to start metrics server: %w", err)
}
select {
case err := <-errCh:
// there might be an error already if the startup failed
+ cancel()
return err
default:
}
+ m.stopFN = cancel
go func() {
err := <-errCh
if err != nil {
m.log.Err(err).Msg("error in metrics server")
- panic(err)
+ cancel()
}
}()
return nil
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func (m *metricsWrapper) Start(ctx context.Context) error { | |
| m.log.Info().Msg("bootstrap starting metrics server") | |
| ctx, cancel := context.WithCancel(ctx) | |
| m.stopFN = cancel | |
| ictx, errCh := irrecoverable.WithSignaler(ctx) | |
| m.Server.Start(ictx) | |
| if err := util.WaitClosed(ctx, m.Ready()); err != nil { | |
| return fmt.Errorf("failed to start metrics server: %w", err) | |
| } | |
| select { | |
| case err := <-errCh: | |
| // there might be an error already if the startup failed | |
| return err | |
| default: | |
| } | |
| go func() { | |
| err := <-errCh | |
| if err != nil { | |
| m.log.Err(err).Msg("error in metrics server") | |
| panic(err) | |
| } | |
| }() | |
| return nil | |
| } | |
| func (m *metricsWrapper) Start(ctx context.Context) error { | |
| m.log.Info().Msg("bootstrap starting metrics server") | |
| ctx, cancel := context.WithCancel(ctx) | |
| defer func() { | |
| if err := recover(); err != nil { | |
| cancel() | |
| m.log.Error().Interface("panic", err).Msg("panic in metrics server") | |
| } | |
| }() | |
| ictx, errCh := irrecoverable.WithSignaler(ctx) | |
| m.Server.Start(ictx) | |
| if err := util.WaitClosed(ctx, m.Ready()); err != nil { | |
| cancel() | |
| return fmt.Errorf("failed to start metrics server: %w", err) | |
| } | |
| select { | |
| case err := <-errCh: | |
| // there might be an error already if the startup failed | |
| cancel() | |
| return err | |
| default: | |
| } | |
| m.stopFN = cancel | |
| go func() { | |
| err := <-errCh | |
| if err != nil { | |
| m.log.Err(err).Msg("error in metrics server") | |
| cancel() | |
| } | |
| }() | |
| return nil | |
| } |
🧰 Tools
🪛 golangci-lint (1.62.2)
708-708: m.Ready undefined (type *metricsWrapper has no field or method Ready)
(typecheck)
Closes: #???
Description
In some cases, like in e2e tests here: #760 the metrics server didn't properly shut down.
This adds a wrapper that encapsulates the metrics server start and stop, and adds an explicit context cancel in case it wasn't already cancelled.
For contributor use:
masterbranchFiles changedin the Github PR explorerSummary by CodeRabbit