Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[METRICS] MetricSensor exception handling #1837

Open
chinghongfang opened this issue Aug 7, 2023 · 0 comments
Open

[METRICS] MetricSensor exception handling #1837

chinghongfang opened this issue Aug 7, 2023 · 0 comments

Comments

@chinghongfang
Copy link
Collaborator

MetricStore 在收集指標的時候可能會出現撈取許多指標,但是沒有被保存下來。這是因為 MetricSensor 在過濾的時候丟出 NoSuchElementException。在 MetricStore 中,我們將 receiver 放出來的指標打包成 BeanObjectClient 供 MetricSensor 查詢,但是打包起來的指標不一定包含全部 MetricSensor 需要的指標,只要有一項指標拋出錯誤,其他過濾出來的指標也無法傳回給 MetricStore 儲存了。

舉例來說,NetworkCost 的 MetricSensor 包含了 4 種指標,所以只要 BeanObjectClient 中不包含 "jvmMemory" (指標多時,很常發生) 就會拋出錯誤。

https://github.com/skiptests/astraea/blob/4140b68ca6950206d65c6cc71be0203f1fa1ce74/common/src/main/java/org/astraea/common/cost/NetworkCost.java#L193-L206


#1569 遇到的問題相似,當 MetricSensor 無法從 BeanObjectClient 取得某一個指標時,會拋出錯誤,導致其他已取得的指標也跟著無法"接收"。

當時的解法是在 Partition 中,為每個 MetricSensor 加一個 exception handler ,用來處理特定一個 MetricSensor 撈不到指標的狀況。
而這次是在同一個 MetricSensor 內包含有多種指標索取。

這裡先提出一個先行解法:

  @Override
  public MetricSensor metricSensor() {
    return (client, clusterBean) -> {
      List<HasBeanObject> jvmMemoryBean = List.of();
      try {
        jvmMemoryBean = List.of(HostMetrics.jvmMemory(client));
      } catch (NoSuchElementException ignore) {
        // It returns a list of beans. Do not throw when any one failed.
      }
      return Stream.of(
              jvmMemoryBean,
              ServerMetrics.Topic.BYTES_IN_PER_SEC.fetch(client),
              ServerMetrics.Topic.BYTES_OUT_PER_SEC.fetch(client),
              LogMetrics.Log.SIZE.fetch(client),
              clusterInfoSensor.fetch(client, clusterBean))
          .flatMap(Collection::stream)
          .toList();
    };
  }

先檢查是否報錯,並把錯誤忽略,回傳其他項目。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant