-
Notifications
You must be signed in to change notification settings - Fork 469
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Azure Storage Account] Add metric_type metadata to the storage_account datastream #7488
Merged
ritalwar
merged 9 commits into
elastic:main
from
ritalwar:azure_tsdb_storageaccount_metrictype_7215
Oct 3, 2023
Merged
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
a82b5b1
[Azure Storage Account] Add metric_type metadata to the datastream
ritalwar a02061f
Update changelog.yml
ritalwar 6b59f3f
Update metric_type mapping to type object
ritalwar 52d1c4f
Merge branch 'main' into azure_tsdb_storageaccount_metrictype_7215
ritalwar 4fa65b5
Merge branch 'main' into azure_tsdb_storageaccount_metrictype_7215
ritalwar 7a68ac6
Update manifest.yml
ritalwar da72daa
Merge branch 'main' into azure_tsdb_storageaccount_metrictype_7215
ritalwar 6b4b22e
Merge branch 'main' into azure_tsdb_storageaccount_metrictype_7215
ritalwar d51fcee
Update docs files
ritalwar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
115 changes: 108 additions & 7 deletions
115
packages/azure_metrics/data_stream/storage_account/fields/fields.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,108 @@ | ||
| - name: azure.storage_account.*.* | ||
| type: object | ||
| object_type: float | ||
| object_type_mapping_type: "*" | ||
| description: > | ||
| storage account | ||
|
|
||
| - name: azure.storage_account | ||
| type: group | ||
| fields: | ||
| - name: availability.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: percent | ||
| description: The percentage of availability for the storage service or the specified API operation. | ||
| - name: egress.total | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of egress data, in bytes. | ||
| - name: ingress.total | ||
| type: float | ||
| unit: byte | ||
| metric_type: gauge | ||
| description: The amount of ingress data, in bytes. | ||
| - name: success_e2elatency.avg | ||
zmoog marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| type: float | ||
| metric_type: gauge | ||
| unit: ms | ||
| description: The end-to-end latency of successful requests made to a storage service or the specified API operation, in milliseconds. | ||
| - name: success_server_latency.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: ms | ||
| description: The latency used by Azure Storage to process a successful request, in milliseconds. | ||
| - name: transactions.total | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of requests made to a storage service or the specified API operation. | ||
| - name: used_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: Account used capacity | ||
| - name: blob_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by the storage account's Blob service in bytes. | ||
| - name: blob_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of Blob in the storage account's Blob service. | ||
| - name: container_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of containers in the storage account's Blob service. | ||
| - name: index_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by ADLS Gen2 (Hierarchical) Index in bytes. | ||
| - name: file_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by the storage account's File service in bytes. | ||
| - name: file_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of file in the storage account's File service. | ||
| - name: file_share_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of file shares in the storage account's File service. | ||
| - name: file_share_capacity_quota.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The upper limit on the amount of storage that can be used by Azure Files Service in bytes. | ||
| - name: file_share_snapshot_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of snapshots present on the share in storage account's Files Service. | ||
| - name: file_share_snapshot_size.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by the snapshots in storage account's File service in bytes. | ||
| - name: queue_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by the storage account's Queue service in bytes. | ||
| - name: queue_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of queue in the storage account's Queue service. | ||
| - name: queue_message_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The approximate number of queue messages in the storage account's Queue service. | ||
| - name: table_capacity.avg | ||
| type: float | ||
| metric_type: gauge | ||
| unit: byte | ||
| description: The amount of storage used by the storage account's Table service in bytes. | ||
| - name: table_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of table in the storage account's Table service. | ||
| - name: table_entity_count.avg | ||
| type: float | ||
| metric_type: gauge | ||
| description: The number of table entities in the storage account's Table service. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are not including:
IndexCapacityfrom https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-classicstorage-storageaccounts-blobservices-metricsFileShareQuotafrom https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-classicstorage-storageaccounts-fileservices-metricsIs this intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, please forget the comment about
IndexCapacity, I could not find it but it's there. My Bad.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FileShareQuota is also there: https://github.com/ritalwar/integrations/blob/a02061f69bdab60414a479a6a6e60b66f1214042/packages/azure_metrics/data_stream/storage_account/fields/fields.yml#L69
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, this is surprising: how can
FileShareQuotafield from the docs becomefile_share_capacity_quotafield in ES?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is being taken from here or sdk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, thanks.
I see we have two documents for Azure Files:
Since the first one mentions the "Microsoft.ClassicStorage/storageAccounts" namespace, this may mean this for the classic version of the Files service, the the second for the current / modern version.
Are we supporting metrics for the classic namespace?
We should also check if we need add some
*IOPS*related field in the current namespace doc, like:FileShareMaxUsedIOPSFileShareProvisionedIOPSFileShareMaxUsedBandwidthMiBpsWDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, my mistake! I referred to the classicStorage link instead of the correct one, which should be Microsoft.Storage/storageAccounts as this is the namespace we are supporting. To address this, I will update the metric_type mapping using the object type instead of the group format like this
This will ensure that all the fields within this namespace are correctly accounted for.
I didn't use this format before due to some issues associated with it. However, it seems those issues have been resolved, as I tested it successfully.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it! Thanks for the check!
I'm not an expert in TSDB, but this makes sense.
I updated the link in the PR description with the non-classic one; please take a look to double-check if it's the correct one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @zmoog!