Skip to content

Commit

Permalink
edits from feedback and more
Browse files Browse the repository at this point in the history
  • Loading branch information
feorlen committed Dec 4, 2024
1 parent 164e8fd commit 35b674e
Show file tree
Hide file tree
Showing 7 changed files with 66 additions and 36 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ The procedure on this page documents the following:
- Configuring a Prometheus service to scrape and display metrics from a MinIO deployment
- Configuring an Alert Rule on a MinIO Metric to trigger an AlertManager action

These instructions use :ref:`version 2 metrics. <minio-metrics-v2>`
For more about metrics API versions, see :ref:`Metrics and alerts. <minio-metrics-and-alerts>`

.. admonition:: Prerequisites
:class: note

Expand All @@ -32,8 +35,6 @@ The procedure on this page documents the following:

- An :mc:`mc` installation on your local host configured to :ref:`access <alias>` the MinIO deployment

These instructions use :ref:`version 2 metrics. <minio-metrics-v2>`
For more about metrics API versions, see :ref:`Metrics and alerts. <minio-metrics-and-alerts>`

Configure Prometheus to Collect and Alert using MinIO Metrics
-------------------------------------------------------------
Expand Down
6 changes: 4 additions & 2 deletions source/operations/monitoring/grafana.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@ Prerequisites
.. admonition:: Grafana dashboards use metrics version 2
:class: note

The MinIO Grafana dashboards are designed for use with :ref:`metrics version 2 <minio-metrics-v2>`.
The MinIO Grafana dashboards use :ref:`metrics version 2 <minio-metrics-v2>`.
For more about metrics API versions, see :ref:`Metrics and alerts. <minio-metrics-and-alerts>`


Version 3 metrics require creating your own dashboard.
For more information about dashboards, see `the Grafana documentation. <https://grafana.com/docs/grafana/latest/dashboards/>`__

MinIO Grafana Dashboard
-----------------------

Expand Down
5 changes: 3 additions & 2 deletions source/operations/monitoring/healthcheck-probe.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,9 @@ A response code of ``200 OK`` indicates the MinIO server is
online and functional. Any other HTTP codes indicate an issue with reaching
the server, such as a transient network issue or potential downtime.

The healthcheck probe alone cannot determine if a MinIO server is offline - only that the current host machine cannot reach the server.
Consider configuring a Prometheus :ref:`alert <minio-metrics-and-alerts>` using the :ref:`metrics v3 <minio-metrics-and-alerts>` ``minio_cluster_health_nodes_offline_count`` or :ref:`v2 <minio-metrics-v2>` ``minio_cluster_nodes_offline_total`` metrics to detect whether one or more MinIO nodes are offline.
The healthcheck probe alone cannot determine if a MinIO server is offline.
Instead, the probe determines whether the current host machine can reach the server.
Consider configuring a Prometheus :ref:`alert <minio-metrics-and-alerts>` using ``minio_cluster_health_nodes_offline_count`` for :ref:`metrics v3 <minio-available-v3-cluster-metrics>` or ``minio_cluster_nodes_offline_total`` for :ref:`metrics v2 <minio-available-cluster-metrics>` to detect whether one or more MinIO nodes are offline.

.. _minio-cluster-write-quorum:

Expand Down
25 changes: 15 additions & 10 deletions source/operations/monitoring/metrics-and-alerts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,24 @@ Metrics and alerts
:local:
:depth: 2

.. admonition:: Metrics version 3

MinIO publishes metrics using the :prometheus-docs:`Prometheus Data Model <concepts/data_model/>`.
You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting.

Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 provides additional endpoints.
MinIO recommends version 3 for new deployments.

.. admonition:: Metrics version 2
:class: note

Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, metrics version 3 provides additional metrics scraping endpoints.
MinIO recommends version 3 for new deployments.
Existing deployments can continue to use version 2 :ref:`metrics <minio-metrics-v2>` and :ref:`Grafana dashboards <minio-grafana>`.

Existing deployments can continue to use :ref:`metrics version 2 <minio-metrics-v2>` and :ref:`the v2 Grafana dashboards <minio-grafana>`.

MinIO publishes cluster and node metrics using the :prometheus-docs:`Prometheus Data Model <concepts/data_model/>`.
You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting.
Version 3 Endpoints
-------------------

For metrics version 3, all metrics are available under the base ``/minio/metrics/v3`` endpoint.
You can scrape the base endpoint to collect all metrics in a single scraping operation, or append an optional path to scrape a single category.
You can scrape the base endpoint to collect all metrics in a single operation, or append an optional path to return a specific category.

For example, the following endpoint returns audit metrics:

Expand Down Expand Up @@ -102,7 +107,7 @@ MinIO provides the following scraping endpoints, relative to the base URL:

``/system/process``

For a complete list of metrics for each endpoint, see :ref:`Available Metrics <minio-metrics-and-alerts-available-metrics>`.
For a complete list of metrics for each endpoint, see :ref:`Available version 3 etrics <minio-metrics-and-alerts-available-metrics>`.

.. cond:: k8s

Expand All @@ -123,8 +128,7 @@ For a complete list of metrics for each endpoint, see :ref:`Available Metrics <m
Available version 3 metrics
---------------------------

MinIO publishes a number of metrics at the cluster, node, or bucket levels.
Each metric includes a label for the MinIO server which generated that metric.
MinIO publishes a number of metrics for clusters, API requests, buckets, and other aspects of the MinIO service:

- :ref:`API Metrics <minio-available-v3-api-metrics>`
- :ref:`Audit Metrics <minio-available-v3-audit-metrics>`
Expand All @@ -137,6 +141,7 @@ Each metric includes a label for the MinIO server which generated that metric.
- :ref:`Scanner Metrics <minio-available-v3-scanner-metrics>`
- :ref:`System Metrics <minio-available-v3-system-metrics>`

Many metrics include labels identifying the resource which generated that metric and other relevant details.

.. _minio-available-v3-api-metrics:

Expand Down
5 changes: 4 additions & 1 deletion source/operations/monitoring/metrics-v2.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Metrics version 2
MinIO publishes cluster and node metrics using the :prometheus-docs:`Prometheus Data Model <concepts/data_model/>`.
You can use any scraping tool to pull metrics data from MinIO for further analysis and alerting.

Version 3 Endpoints
-------------------

Metrics version 2 provides metrics organized into three categories:

- :ref:`Cluster Metrics <minio-available-cluster-metrics>`
Expand All @@ -31,7 +34,7 @@ For example, scraping the following endpoint returns all cluster metrics:
The base endpoint alone, ``/minio/v2/metrics/``, returns cluster metrics.

For more flexible scraping and a wider range of metrics, use :ref:`metrics version 3. <minio-metrics-and-alerts>`
Existing deployments can continue to use :ref:`metrics version 2 <minio-metrics-v2>` and :ref:`the v2 Grafana dashboards <minio-grafana>`.
Existing deployments can continue to use version 2 :ref:`metrics <minio-metrics-v2>` and :ref:`Grafana dashboards <minio-grafana>`.


MinIO Grafana dashboard
Expand Down
27 changes: 19 additions & 8 deletions source/reference/minio-mc-admin/mc-admin-prometheus-generate.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The :mc:`mc admin prometheus generate` command generates a metrics scraping conf
For more complete documentation on using MinIO with Prometheus, see :ref:`How to monitor MinIO server with Prometheus <minio-metrics-collect-using-prometheus>`

Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 (v3) <minio-metrics-and-alerts>` provides additional metrics.
Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 (v3) <minio-metrics-and-alerts>` provides additional endpoints and metrics.
To generate a v3 scrape configuration use the ``--api_version v3`` option.

MinIO recommends new deployments use :ref:`version 3 (v3) <minio-metrics-and-alerts>`.
Expand Down Expand Up @@ -56,7 +56,7 @@ Existing deployments can continue to use :ref:`metrics version 2 <minio-metrics-
ALIAS \
[TYPE] \
[--api_version v3] \
[TYPE --bucket <bucket name> --api_version v3]
[--bucket <bucket name>]
.. include:: /includes/common-minio-mc.rst
:start-after: start-minio-syntax
Expand All @@ -76,14 +76,16 @@ Parameters

To generate a scrape configuration for :ref:`v3 metrics <minio-metrics-and-alerts>`, include an ``--api-version v3`` parameter.
``v3`` is the only accepted value.

Omit ``--api-version`` to generate a :ref:`v2 metrics <minio-metrics-v2>` configuration.

.. mc-cmd:: --bucket
:optional:

Only valid for v3 metrics.

For v3 metric types that return bucket-level metrics, specify a bucket name.
Use with :mc-cmd:`~mc admin prometheus generate --api-version`.
Requires :mc-cmd:`~mc admin prometheus generate --api-version`.

``--bucket`` works for the following v3 metric types:

Expand Down Expand Up @@ -125,7 +127,7 @@ Parameters
- ``resource``

If not specified, a ``v2`` command returns cluster metrics.
Cluster metrics also include rollups of certain node metrics.
Cluster metrics include rollups of certain node metrics.


Global flags
Expand Down Expand Up @@ -163,10 +165,12 @@ The output resembles the following:
static_configs:
- targets: ['localhost:9000']
Generate a v3 cluster metrics config
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc admin prometheus generate --api-version v3 <mc admin prometheus generate --api-version>` to generate a scrape configuration that collects v3 cluster metrics for a MinIO deployment:
Generate a v3 metrics config for another type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To generate a configuration for another metric type, specify the type.
The following generates a scrape configuration for v3 cluster metrics:

.. code-block:: shell
:class: copyable
Expand All @@ -189,6 +193,7 @@ The output resembles the following:
To generate a configuration for a :mc-cmd:`different metric type <mc admin prometheus generate TYPE>`, replace ``cluster`` with the desired type.


Generate a v3 bucket replication metrics config
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -237,6 +242,7 @@ The output resembles the following:
static_configs:
- targets: [`localhost:9000`]
Generate a default metrics v2 config
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand All @@ -261,10 +267,15 @@ The output resembles the following:
static_configs:
- targets: ['localhost:9000']
Generate a v2 config for other metric types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To generate a configuration for another metric type, specify the type.
The following generates a scrape configuration for v2 bucket metrics:

.. code-block:: shell
:class: copyable
mc admin prometheus generate ALIAS bucket
29 changes: 18 additions & 11 deletions source/reference/minio-mc-admin/mc-admin-prometheus-metrics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The output includes additional information about each metric, such as if its val

For more complete documentation on using MinIO with Prometheus, see :ref:`How to monitor MinIO server with Prometheus <minio-metrics-collect-using-prometheus>`

Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 (v3) <minio-metrics-and-alerts>` provides additional metrics.
Starting with MinIO Server :minio-release:`RELEASE.2024-07-15T19-02-30Z` and MinIO Client :mc-release:`RELEASE.2024-07-11T18-01-28Z`, :ref:`metrics version 3 (v3) <minio-metrics-and-alerts>` provides additional endpoints and metrics.
To print v3 metrics use the ``--api_version v3`` option.

MinIO recommends new deployments use :ref:`version 3 (v3) <minio-metrics-and-alerts>`.
Expand Down Expand Up @@ -58,7 +58,7 @@ Existing deployments can continue to use :ref:`metrics version 2 <minio-metrics-
ALIAS \
[TYPE] \
[--api_version v3] \
[TYPE --bucket <bucket name> --api_version v3]
[--bucket <bucket name>]
.. include:: /includes/common-minio-mc.rst
Expand All @@ -85,8 +85,8 @@ Parameters
.. mc-cmd:: --bucket
:optional:

Requires :mc-cmd:`~mc admin prometheus metrics --api-version`.
For v3 metric types that return bucket-level metrics, specify a bucket name.
Use with :mc-cmd:`~mc admin prometheus metrics --api-version`.

``--bucket`` works for the following v3 metric types:

Expand Down Expand Up @@ -128,7 +128,7 @@ Parameters
- ``resource``

If not specified, a ``v2`` command returns cluster metrics.
Cluster metrics also include rollups of certain node metrics.
Cluster metrics include rollups of certain node metrics.


Global flags
Expand Down Expand Up @@ -162,10 +162,10 @@ The following prints all scanner metrics for a deployment:
mc admin prometheus metrics ALIAS scanner --api-version v3
Print v3 bucket replication metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Print v3 API or bucket replication metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Certain metric types accept a :mc-cmd:`~mc admin prometheus metrics --bucket` parameter to specify the bucket for which to print metrics.
Certain v3 metric types accept a :mc-cmd:`~mc admin prometheus metrics --bucket` parameter to specify the bucket for which to print metrics.
The following example prints v3 replication metrics for bucket ``mybucket``:

.. code-block:: shell
Expand All @@ -175,10 +175,11 @@ The following example prints v3 replication metrics for bucket ``mybucket``:
- Replace ``ALIAS`` with the :mc-cmd:`alias <mc alias>` of the MinIO deployment.

To print API metrics for a bucket, replace ``replication`` with ``api``.
To print API metrics for the bucket, replace ``replication`` with ``api``.

Print v2 metrics
~~~~~~~~~~~~~~~~

Print v2 cluster metrics
~~~~~~~~~~~~~~~~~~~~~~~~

By default, :mc-cmd:`mc admin prometheus metrics` prints v2 cluster metrics:

Expand All @@ -189,10 +190,16 @@ By default, :mc-cmd:`mc admin prometheus metrics` prints v2 cluster metrics:
- Replace ``ALIAS`` with the :mc-cmd:`alias <mc alias>` of the MinIO deployment.

To print another type of metrics, specify the desired :mc-cmd:`~mc admin prometheus metrics TYPE`.

Print other types of v2 metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To print another type of v2 metrics, specify the desired :mc-cmd:`~mc admin prometheus metrics TYPE`.
The following example prints v2 bucket metrics:

.. code-block:: shell
:class: copyable
mc admin prometheus metrics ALIAS bucket
Accepted values are ``bucket``, ``cluster``, ``node``, and ``resource``.

0 comments on commit 35b674e

Please sign in to comment.