Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding fine-grained stats to observe network traffic #57543

Open
nolouch opened this issue Nov 20, 2024 · 0 comments
Open

Adding fine-grained stats to observe network traffic #57543

nolouch opened this issue Nov 20, 2024 · 0 comments
Labels
type/feature-request Categorizes issue or PR as related to a new feature.

Comments

@nolouch
Copy link
Member

nolouch commented Nov 20, 2024

Feature Request

Is your feature request related to a problem? Please describe:

Background

  • IaaS vendors charge network traffic in a few categories (cross-az/cross-region/Internet IN/OUT) with different pricing. (Amazon EC2 On-Demand Pricing). The pricing varies as per the categories.
    • Internet OUT
    • Cross-region OUT
    • Cross-az IN/OUT
  • For high availability, it's recommended to deploy TiDB clusters across multiple AZ in the same region. This deployment could produce random cross-az traffic which leads to random cost.
  • When cross-az traffic causes significant costs in bills, customers need to know the application or SQL statements that contribute the most. For now, we don't have such meters to satisfy this requirement.

Describe the feature you'd like:

Collecting network traffic produced by SQL statements

- statements_summary

Include extra columns as follows:

Field Data Type  
SUM_UNPACKED_BYTES_SENT_KV_TOTAL bigint unsigned Total volume of unpacked data transmitted from this TiDB instance to all TiKV instances, regardless of zone.
SUM_UNPACKED_BYTES_RECIEVED_KV_TOTAL bigint unsigned Total volume of unpacked data received by this TiDB instance from all TiKV instances, regardless of zone.
SUM_UNPACKED_BYTES_SENT_KV_INTERZONE bigint unsigned Total volume of unpacked data transmitted from this TiDB instance to TiKV instances in different zones.
SUM_UNPACKED_BYTES_RECIEVED_KV_INTERZONE bigint unsigned Total volume of unpacked data received by this TiDB instance from TiKV instances located in different zones.
SUM_UNPACKED_BYTES_SENT_MPP_TOTAL bigint unsigned Total volume of unpacked data sent to all TiFlash instances, with data originating from this TiDB instance or any TiFlash instance as part of inter-instance communication.
SUM_UNPACKED_BYTES_RECIEVED_MPP_TOTAL bigint unsigned Total volume of unpacked data received from all TiFlash instances, directed to this TiDB instance or any TiFlash instance as part of inter-instance communication.
SUM_UNPACKED_BYTES_SENT_MPP_INTERZONE bigint unsigned Total volume of unpacked data sent to TiFlash instances located in different zones, originating from this TiDB instance or any TiFlash instance involved in cross-zone communication.
SUM_UNPACKED_BYTES_RECIEVED_MPP_INTERZONE bigint unsigned Total volume of unpacked data received from TiFlash instances located in different zones, directed to this TiDB instance or any TiFlash instance involved in cross-zone communication.
  • Slow logs
    Include the same set of metrics in slow log. Refer to tidb_statements_states for naming.

  • Metrics in grafana

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

@nolouch nolouch added the type/feature-request Categorizes issue or PR as related to a new feature. label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature-request Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

1 participant