diff --git a/docs/en/observability/explore-logs.asciidoc b/docs/en/observability/explore-logs.asciidoc index 5769064220..a8522e3e8a 100644 --- a/docs/en/observability/explore-logs.asciidoc +++ b/docs/en/observability/explore-logs.asciidoc @@ -91,4 +91,12 @@ The following actions help you filter and focus on specific fields in the log de * **Filter for value (image:images/plusInCircle.png[filter for value icon]):** Show logs that contain the specific field value. * **Filter out value (image:images/minusInCircle.png[filter out value icon]):** Show logs that do *not* contain the specific field value. * **Filter for field present (image:images/filter.png[filter for present icon]):** Show logs that contain the specific field. -* **Toggle column in table (image:images/listAdd.png[toggle column in table icon]):** Add or remove a column for the field to the main Logs Explorer table. \ No newline at end of file +* **Toggle column in table (image:images/listAdd.png[toggle column in table icon]):** Add or remove a column for the field to the main Logs Explorer table. + +[discrete] +[[view-log-data-set-details]] +== View log data set details + +From the main {kib} menu, go to **Stack Management** → **Data Set Quality* to view more details about your data sets and monitor their overall quality. + +Refer to <> for more information. \ No newline at end of file diff --git a/docs/en/observability/images/green-dot-icon.png b/docs/en/observability/images/green-dot-icon.png new file mode 100644 index 0000000000..05643e0a38 Binary files /dev/null and b/docs/en/observability/images/green-dot-icon.png differ diff --git a/docs/en/observability/images/logs-dataset-overview.png b/docs/en/observability/images/logs-dataset-overview.png new file mode 100644 index 0000000000..0676e3aac0 Binary files /dev/null and b/docs/en/observability/images/logs-dataset-overview.png differ diff --git a/docs/en/observability/images/red-dot-icon.png b/docs/en/observability/images/red-dot-icon.png new file mode 100644 index 0000000000..799035b8cc Binary files /dev/null and b/docs/en/observability/images/red-dot-icon.png differ diff --git a/docs/en/observability/images/yellow-dot-icon.png b/docs/en/observability/images/yellow-dot-icon.png new file mode 100644 index 0000000000..71216ffa3d Binary files /dev/null and b/docs/en/observability/images/yellow-dot-icon.png differ diff --git a/docs/en/observability/index.asciidoc b/docs/en/observability/index.asciidoc index 83247dea79..0a2ce2f146 100644 --- a/docs/en/observability/index.asciidoc +++ b/docs/en/observability/index.asciidoc @@ -57,6 +57,8 @@ include::inspect-log-anomalies.asciidoc[leveloffset=+3] include::configure-logs-sources.asciidoc[leveloffset=+3] +include::logs-monitor-datasets.asciidoc[leveloffset=+2] + include::logs-index-template.asciidoc[leveloffset=+2] include::logs-troubleshooting.asciidoc[leveloffset=+2] diff --git a/docs/en/observability/logs-monitor-datasets.asciidoc b/docs/en/observability/logs-monitor-datasets.asciidoc new file mode 100644 index 0000000000..7f3d14602d --- /dev/null +++ b/docs/en/observability/logs-monitor-datasets.asciidoc @@ -0,0 +1,70 @@ +[[monitor-datasets]] += Monitor log data set quality + +beta:[] + +The **Data Set Quality** page provides an overview of your log data sets. +Use this information to get an idea of your overall log data set quality and find data sets that contain incorrectly parsed documents. +Access the Data Set Quality page from the main {kib} menu at **Stack Management** → **Data Set Quality**. + +[role="screenshot"] +image::../images/logs-dataset-overview.png[Screen capture of the data set overview] + +.Requirements +[NOTE] +==== +Users with the `viewer` role can view the Data Sets Quality summary. To view the Active Data Sets and Estimated Data summaries, users need the `monitor` {ref}/security-privileges.html#privileges-list-indices[index privilege] for the `logs-*-*` index. +==== + +The quality of your data sets is based on the percentage of degraded documents in each data set. +A degraded document in a data set contains the {ref}/mapping-ignored-field.html[_ignored] property because one or more of its fields were ignored during indexing. +Fields are ignored for a variety of reasons. +For example, when the {ref}/mapping-ignored-field.html[ignore_malformed] parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed. + +From the data set table, you'll find information for each data set such as its namespace, size, when the data set was last active, and the percentage of degraded docs. +The percentage of degraded documents determines the data set's quality according to the following scale: + +* Good (image::../images/green-dot-icon.png[Good icon]): 0% of the documents in the data set are degraded. +* Degraded (image::../images/yellow-dot-icon.png[Degraded icon]): Greater than 0% and up to 3% of the documents in the data set are degraded. +* Poor (image::../images/red-dot-icon.png[Poor icon]): Greater than 3% of the documents in the data set are degraded. + +Opening the details of a specific data set shows the degraded documents history, a summary for the data set, and other details that can help you determine if you need to investigate any issues. + +[discrete] +[[investigate-issues]] +== Investigate issues + +The Data Set Quality page has a couple of different ways to help you find ignored fields and investigate issues. +From the data set table, you can expand a data set's details, and view commonly ignored fields and information about those fields. +You can also open a data set in Logs Explorer to find ignored fields in individual logs. + +[discrete] +[[find-ignored-fields-in-data-sets]] +=== Find ignored fields in data sets + +To expand the details of a data set with poor or degraded quality and view ignored fields: + +. From the data set table, click next to a data set with poor or degraded quality. +. From the details, scroll down to **Degraded fields**. + +The **Degraded fields** section shows fields that have been ignored, the number of documents that contain ignored fields, and the timestamp of last occurrence of the field being ignored. + +[discrete] +[[find-ignored-fields-in-individual-logs]] +=== Find ignored fields in individual logs + +To use Logs Explorer to find ignored fields in individual logs: + +. Find data sets with degraded documents using the **Degraded Docs** column of the data sets table. +. Click the percentage in the **Degraded Docs** column to open the data set in Logs Explorer. + +The **Documents** table in Logs Explorer is automatically filtered to show documents that were not parsed correctly. +Under the **actions** column, you'll find the degraded document icon (). + +Now that you know which documents contain ignored fields, examine them more closely to find the origin of the issue: + +. Under the **actions** column, click to open the log details. +. Select the **JSON** tab. +. Scroll towards the end of the JSON to find the `ignored_field_values`. + +Here, you'll find all of the `_ignored` fields in the document and their values, which should provide some clues as to why the fields were ignored. \ No newline at end of file diff --git a/docs/en/serverless/logging/monitor-datasets.mdx b/docs/en/serverless/logging/monitor-datasets.mdx index 87b1ff5d2f..89ccd63bb1 100644 --- a/docs/en/serverless/logging/monitor-datasets.mdx +++ b/docs/en/serverless/logging/monitor-datasets.mdx @@ -6,12 +6,11 @@ description: Monitor log data sets to find degraded documents. tags: [ 'serverless', 'observability', 'how-to' ] --- -

The **Data Set Quality** page provides an overview of your log data sets. Use this information to get an idea of your overall log data set quality and find data sets that contain incorrectly parsed documents. -Access the Data Set Quality page from the main ((kib)) menu and go to **Stack Management** → **Data Set Quality**. +Access the Data Set Quality page from the main menu at **Project settings** → **Management** → **Data Set Quality**. @@ -22,7 +21,7 @@ Access the Data Set Quality page from the main ((kib)) menu and go to **Stack Ma The quality of your data sets is based on the percentage of degraded documents in each data set. A degraded document in a data set contains the [`_ignored`](((ref))/mapping-ignored-field.html) property because one or more of its fields were ignored during indexing. Fields are ignored for a variety of reasons. -For example, when the [`ignore_malformed`](((ref))/mapping-ignored-field.html.html) parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed. +For example, when the [`ignore_malformed`](((ref))/mapping-ignored-field.html) parameter is set to true, if a document field contains the wrong data type, the malformed field is ignored and the rest of the document is indexed. From the data set table, you'll find information for each data set such as its namespace, when the data set was last active, and the percentage of degraded docs. The percentage of degraded documents determines the data set's quality according to the following scale: @@ -39,7 +38,7 @@ From the data set table, you can expand a data set's details, and view commonly You can also open a data set in Logs Explorer to find ignored fields in individual logs. ### Find ignored fields in data sets -To expand the details of a dataset with poor or degraded quality and view ignored fields: +To expand the details of a data set with poor or degraded quality and view ignored fields: 1. From the data set table, click next to a data set with poor or degraded quality. 1. From the details, scroll down to **Degraded fields**. @@ -53,7 +52,7 @@ To use Logs Explorer to find ignored fields in individual logs: 1. Click the percentage in the **Degraded Docs** column to open the data set in Logs Explorer. The **Documents** table in Logs Explorer is automatically filtered to show documents that were not parsed correctly. -Under the *actions* column, you'll find the degraded document icon (). +Under the **actions** column, you'll find the degraded document icon (). Now that you know which documents contain ignored fields, examine them more closely to find the origin of the issue: diff --git a/docs/en/serverless/logging/view-and-monitor-logs.mdx b/docs/en/serverless/logging/view-and-monitor-logs.mdx index 95ef68ddba..cbec4893ed 100644 --- a/docs/en/serverless/logging/view-and-monitor-logs.mdx +++ b/docs/en/serverless/logging/view-and-monitor-logs.mdx @@ -81,6 +81,6 @@ The following actions help you filter and focus on specific fields in the log de ## View log data set details -From the main ((kib)) menu, go to **Stack Management** → **Data Set Quality* to view more details about your data sets and monitor their overall quality. +From the main menu, go to **Project settings** → **Management** → **Data Set Quality** to view more details about your data sets and monitor their overall quality. Refer to Monitor data sets for more information. \ No newline at end of file