Tablet throttler: read and use MySQL host metrics #16904
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| /* | ||
| Copyright 2024 The Vitess Authors. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package base | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
| "sync/atomic" | ||
| "time" | ||
|
|
||
| "vitess.io/vitess/go/timer" | ||
|
|
||
| tabletmanagerdatapb "vitess.io/vitess/go/vt/proto/tabletmanagerdata" | ||
| ) | ||
|
|
||
| var ( | ||
| mysqlHostMetricsRpcTimeout = 5 * time.Second | ||
| mysqlHostMetricsRateLimit = 10 * time.Second | ||
| mysqlHostMetricsRateLimiter atomic.Pointer[timer.RateLimiter] | ||
| lastMySQLHostMetricsResponse atomic.Pointer[tabletmanagerdatapb.MysqlHostMetricsResponse] | ||
| ) | ||
|
|
||
| // getMysqlMetricsRateLimiter returns a rate limiter that is active until the given context is cancelled. | ||
| // This function will be called sequentially, but nonetheless it offers _some_ concurrent safety. Namely, | ||
| // that a created rate limiter is guaranteed to be cleaned up | ||
| func getMysqlMetricsRateLimiter(ctx context.Context, rateLimit time.Duration) *timer.RateLimiter { | ||
| rateLimiter := mysqlHostMetricsRateLimiter.Load() | ||
| if rateLimiter == nil { | ||
| rateLimiter = timer.NewRateLimiter(rateLimit) | ||
| go func() { | ||
| defer mysqlHostMetricsRateLimiter.Store(nil) | ||
| defer rateLimiter.Stop() | ||
| <-ctx.Done() | ||
| }() | ||
| mysqlHostMetricsRateLimiter.Store(rateLimiter) | ||
| } | ||
| return rateLimiter | ||
| } | ||
|
|
||
| // readMysqlHostMetrics reads MySQL host metrics sporadically from the tablet manager (which in turn reads | ||
| // them from mysql deamon). The metrics are then cached, whether successful or not. | ||
| // This idea is that is is very wasteful to read these metrics for every single query. E.g. right now the throttler | ||
| // can issue 4 reads per second, which is wasteful to go through two RPCs to get the disk space usage for example. Even the load | ||
| // average on the MySQL server is not that susceptible to change. | ||
| func readMysqlHostMetrics(ctx context.Context, params *SelfMetricReadParams) error { | ||
| if params.TmClient == nil { | ||
| return fmt.Errorf("tmClient is nil") | ||
| } | ||
| if params.TabletInfo == nil { | ||
| return fmt.Errorf("tabletInfo is nil") | ||
| } | ||
| rateLimiter := getMysqlMetricsRateLimiter(ctx, mysqlHostMetricsRateLimit) | ||
| err := rateLimiter.Do(func() error { | ||
| ctx, cancel := context.WithTimeout(ctx, mysqlHostMetricsRpcTimeout) | ||
| defer cancel() | ||
|
|
||
| resp, err := params.TmClient.MysqlHostMetrics(ctx, params.TabletInfo.Tablet, &tabletmanagerdatapb.MysqlHostMetricsRequest{}) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| lastMySQLHostMetricsResponse.Store(resp) | ||
| return nil | ||
| }) | ||
| return err | ||
| } | ||
|
|
||
| // getMysqlHostMetric gets a metric from the last read MySQL host metrics. The metric will either be directly read from | ||
| // tablet manager (which then reads it from the mysql deamon), or from the cache. | ||
| func getMysqlHostMetric(ctx context.Context, params *SelfMetricReadParams, mysqlHostMetricName string) *ThrottleMetric { | ||
| metric := &ThrottleMetric{ | ||
| Scope: SelfScope, | ||
| } | ||
| if err := readMysqlHostMetrics(ctx, params); err != nil { | ||
| return metric.WithError(err) | ||
| } | ||
| resp := lastMySQLHostMetricsResponse.Load() | ||
| if resp == nil { | ||
| return metric.WithError(ErrNoResultYet) | ||
| } | ||
| mysqlMetric := resp.HostMetrics.Metrics[mysqlHostMetricName] | ||
| if mysqlMetric == nil { | ||
| return metric.WithError(ErrNoSuchMetric) | ||
| } | ||
| metric.Value = mysqlMetric.Value | ||
| if mysqlMetric.Error != nil { | ||
| metric.Err = errors.New(mysqlMetric.Error.Message) | ||
| } | ||
| return metric | ||
| } | ||
|
|
||
| var _ SelfMetric = registerSelfMetric(&MysqldLoadAvgSelfMetric{}) | ||
| var _ SelfMetric = registerSelfMetric(&MysqldDatadirUsedRatioSelfMetric{}) | ||
|
|
||
| // MysqldLoadAvgSelfMetric stands for the load average per cpu, on the MySQL host. | ||
| type MysqldLoadAvgSelfMetric struct { | ||
|
There was a problem hiding this comment. Also curious if this and the datadir type needs to be exported? There was a problem hiding this comment. Good question. Right now we're exporting all |
||
| } | ||
|
|
||
| func (m *MysqldLoadAvgSelfMetric) Name() MetricName { | ||
| return MysqldLoadAvgMetricName | ||
| } | ||
|
|
||
| func (m *MysqldLoadAvgSelfMetric) DefaultScope() Scope { | ||
| return SelfScope | ||
|
There was a problem hiding this comment. Having the type and variable exported makes this interface feel off to me. Since anyone can access the metric and this internal (?) variable as things are now. There was a problem hiding this comment. Yet no one holds an instance of var _ SelfMetric = registerSelfMetric(&MysqldLoadAvgSelfMetric{})So yes, anyone can create their own new instance, but that cannot affect the existing instances. |
||
| } | ||
|
|
||
| func (m *MysqldLoadAvgSelfMetric) DefaultThreshold() float64 { | ||
| return 1.0 | ||
| } | ||
|
|
||
| func (m *MysqldLoadAvgSelfMetric) RequiresConn() bool { | ||
| return false | ||
| } | ||
|
|
||
| func (m *MysqldLoadAvgSelfMetric) Read(ctx context.Context, params *SelfMetricReadParams) *ThrottleMetric { | ||
| return getMysqlHostMetric(ctx, params, "loadavg") | ||
| } | ||
|
|
||
| // MysqldDatadirUsedRatioSelfMetric stands for the disk space usage of the mount where MySQL's datadir is located. | ||
| // Range: 0.0 (empty) - 1.0 (full) | ||
| type MysqldDatadirUsedRatioSelfMetric struct { | ||
| } | ||
|
|
||
| func (m *MysqldDatadirUsedRatioSelfMetric) Name() MetricName { | ||
| return MysqldDatadirUsedRatioMetricName | ||
| } | ||
|
|
||
| func (m *MysqldDatadirUsedRatioSelfMetric) DefaultScope() Scope { | ||
| return SelfScope | ||
| } | ||
|
|
||
| func (m *MysqldDatadirUsedRatioSelfMetric) DefaultThreshold() float64 { | ||
| return 0.98 | ||
| } | ||
|
|
||
| func (m *MysqldDatadirUsedRatioSelfMetric) RequiresConn() bool { | ||
| return false | ||
| } | ||
|
|
||
| func (m *MysqldDatadirUsedRatioSelfMetric) Read(ctx context.Context, params *SelfMetricReadParams) *ThrottleMetric { | ||
| return getMysqlHostMetric(ctx, params, "datadir-used-ratio") | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do these changes relate to the PR? Is it that we're getting rid of the old
loadavgOS metric now that we have the newmysql-loadavgmetric and so we wanted to test another MySQL process metric here?Is there any reason that we're not using any of the new MySQL host metrics in the e2e tests? Or am I missing something here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #16904 (comment): it's about tests failing on MacOS machines.
You're not missing anything here. Let me see what we can do for e2e tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added
endtoendtests that verify the fullMysqlHostMetricspath don to mysqld, and checking response values. We check the existence of both newmysqld-*values, and we specifically check thatmysqld-datadir-used-ratiois non-zero (as can be expected in local hosts and in CI). We do not check the value ofmysqld-loadavgas that could be arbitrarily low and even appear to be zero, as well as because we do not implement it in MacOS.