Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JMX metrics for S3 http client usage in native FS #24732

Closed

Conversation

mayurjpatel
Copy link
Contributor

@mayurjpatel mayurjpatel commented Jan 16, 2025

When s3 client was changed to use AWS SDK v2 from AWS SDK v1, the set of s3 pool metrics were missed.

With this PR, these http pool metrics are now exposed via JMX beans. The reference for the http metrics exposed by AWS SDK can be found here:
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/HttpMetric.html

Description

When s3 client was changed to use AWS SDK v2 from AWS SDK v1, the set of s3 pool metrics were missed.

With this PR, these http pool metrics are now exposed via JMX beans. The reference for the http metrics exposed by AWS SDK can be found here:
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/HttpMetric.html

Additional context and related issues

#23944

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

Copy link

cla-bot bot commented Jan 16, 2025

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Mayur Patel.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email [email protected]
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

@mayurjpatel mayurjpatel force-pushed the add_http_client_metrics branch from 8c127bf to 7ec4256 Compare January 16, 2025 22:39
Copy link

cla-bot bot commented Jan 16, 2025

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@zhaner08
Copy link
Contributor

LGTM

public class AwsSdkV2HttpClientStats
{
private final TimeStat connectionAcquireLatency = new TimeStat(MILLISECONDS);
private final DistributionStat availableConcurrency = new DistributionStat();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure these make sense as a DistributionStat since the connection pool stats of availableConcurrency, leasedConcurrency, and pendingConcurrency are point in time snapshot values and not values that are accumulated. In AwsSdkClientCoreStats these are recorded as AtomicLong values instead.

@cla-bot cla-bot bot added the cla-signed label Jan 29, 2025
@github-actions github-actions bot added release-notes docs ui Web UI jdbc Relates to Trino JDBC driver hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector bigquery BigQuery connector mongodb MongoDB connector labels Jan 29, 2025
@mayurjpatel mayurjpatel force-pushed the add_http_client_metrics branch from 6edfa86 to 4971ef1 Compare January 29, 2025 19:53
@mosabua
Copy link
Member

mosabua commented Jan 29, 2025

Did you close this PR accidentally @mayurjpatel ?

@mayurjpatel
Copy link
Contributor Author

Did you close this PR accidentally @mayurjpatel ?

Yeah, sorry, my mistake. I am going to open a new PR with @pettyjamesm 's feedback.

@mosabua
Copy link
Member

mosabua commented Jan 29, 2025

Sounds good .. you probably can just reopen this one (or I can for you) .. as long as you are using the same branch and pushing to it (force pushing..)

@mayurjpatel
Copy link
Contributor Author

Ended up creating #24843 after i sync'ed the fork. Apologies for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery BigQuery connector cla-signed delta-lake Delta Lake connector docs hive Hive connector hudi Hudi connector iceberg Iceberg connector jdbc Relates to Trino JDBC driver mongodb MongoDB connector release-notes ui Web UI
Development

Successfully merging this pull request may close these issues.

4 participants