Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a logs@custom index template reference #3593

Merged
merged 4 commits into from
Feb 20, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/en/observability/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ include::inspect-log-anomalies.asciidoc[leveloffset=+3]

include::configure-logs-sources.asciidoc[leveloffset=+3]

include::logs-index-template.asciidoc[leveloffset=+2]

include::logs-troubleshooting.asciidoc[leveloffset=+2]

// Infrastructure
Expand Down
12 changes: 10 additions & 2 deletions docs/en/observability/logs-checklist.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -104,11 +104,19 @@ The following resources provide information on viewing and monitoring your logs:
== Application logs

Application logs provide valuable insight into events that have occurred within your services and applications.
See <<application-logs>>.
Refer to <<application-logs>>.

[discrete]
[[logs-alerts-checklist]]
== Create a logs threshold alert

You can create a rule to send an alert when the log aggregation exceeds a threshold.
See <<logs-threshold-alert>>.

Refer to <<logs-threshold-alert>>.

[discrete]
[[logs-template-checklist]]

Configure the default `logs` template using the `logs@custom` component template.

Refer to the <<logs-index-template>>.
57 changes: 57 additions & 0 deletions docs/en/observability/logs-index-template.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
[[logs-index-template]]
= Logs index template reference

Index templates are used to configure the backing indices of data streams as they're created.
These index templates are composed of multiple {ref}/indices-component-template.html[component templates]—reusable building blocks
that configure index mappings, settings, and aliases.

You can view the default `logs` index template in {kib}.
Navigate to **{stack-manage-app}** → **Index Management** → **Index Templates**, and search for `logs`.
Select the `logs` index templates to view relevant component templates.

[discrete]
[[custom-logs-template-edit]]
== Edit the `logs` index template

The default `logs` index template for the `logs-*-*` index pattern is composed of the following component templates:

* `logs@mappings`
* `logs@settings`
* `logs@custom`
* `ecs@mappings`

You can use the `logs@custom` component template to customize your {es} indices. The `logs@custom` component template is not installed by default, but you can create a component template named `logs@custom` to override and extend default mappings or settings. To do this:

. Open {kib} and navigate to **{stack-manage-app}** → **Index Management** → **Component Templates**.
. Click *Create component template*.
. Name the component template logs@custom.
. Add any custom metadata, index settings, or mappings.
mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved

Changes to component templates are not applied retroactively to existing indices. For changes to take effect, create a new write index for impacted data streams by triggering a rollover. Do this using the {es} {ref}/indices-rollover-index.html[Rollover API]. For example, to roll over the `logs-generic-default` data stream, run:

[source,console]
----
POST /logs-generic-default/_rollover/
----

[discrete]
[[custom-logs-template-default-field]]
=== Set the `default_field` using the custom template

The `logs` index template uses `default_field: [*]` meaning queries without specified fields will search across all fields.
You can update the `default_field` to search in the `message` field instead of all fields using the `logs@custom` component template.

Either create the `logs@custom` component template as outlined in the previous section, or add the following code to the *Index settings* of the `logs` index template:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruflin, what do you think about this sentence?

Personally, I wouldn't suggest fiddling with the logs index template but only changing the custom component template.

Thoughts? @gbamparop

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe the initial sentence wasn't clear. My initial intention was to say "Create a logs@custom component template and add the following to the index settings. If you've already created a logs@custom component template, you can edit it and add the following to the index settings." I didn't mean to suggest editing the logs template.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. sorry it wasn't 100% clear.

I would rephrase as "If you don't already have a logs@custom component template you can create it manually. Once you have it, you can add the following to the settings section"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing it out. I updated it to be clear and add steps for updating the template. I guess because of the way the search works, you can't search for anything using the "@" symbol? or it throws the error:
image

Not sure if there's an underlying reason for this, but it does make it pretty difficult to find any custom templates.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdbirnstiehl Yes, I have experienced this. You can't search for "logs@custom". Either for logs or for custom and then iterate through the many pages

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a bug especially as we use it in our names. Has anyone reported this? If not, lets do it.


[source,json]
----
{
"index": {
"query": {
"default_field": [
"message"
]
}
}
}
----