Skip to content

Commit

Permalink
doc fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
georgewallace committed Oct 25, 2024
1 parent 765cb44 commit b5c46bf
Show file tree
Hide file tree
Showing 6 changed files with 36 additions and 34 deletions.
12 changes: 6 additions & 6 deletions docs/reference/reference-architectures/components.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[[reference-architecture-components]]
== Reference Architecture Components
== Architecture components

This page provides an overview of the main node types in Elasticsearch as they relate to the reference architectures. Each node type serves a specific function within the Elasticsearch cluster, contributing to its overall performance and reliability. Understanding these node types is crucial for designing, managing, and optimizing your Elasticsearch deployment.
This page provides an overview of the main components of the reference architectures. Each component serves a specific function within the Elasticsearch cluster, contributing to its overall performance and reliability. Understanding these node types is crucial for designing, managing, and optimizing your Elasticsearch deployment.

[discrete]
[[component-types]]
Expand All @@ -16,14 +16,14 @@ This page provides an overview of the main node types in Elasticsearch as they r
| Responsible for cluster-wide settings and state, including index metadata and node information. Ensures cluster health by managing node joining and leaving.

| Data Node
| image:images/hot.png[Image showing a master node]
image:images/frozen.png[Image showing a master node]
| image:images/hot.png[Image showing a hot data node]
image:images/frozen.png[Image showing a frozen node]
| Stores data and performs CRUD, search, and aggregations. High I/O, CPU, and memory requirements.
| Machine Learning Node
| image:images/machine-learning.png[Image showing a master node]
| image:images/machine-learning.png[Image showing a machine learning node]
| Executes machine learning jobs, including anomaly detection, data frame analysis, and inference.
| Kibana
| image:images/kibana.png[Image showing a master node]
| image:images/kibana.png[Image showing a kibana node]
| Provides the front-end interface for visualizing data stored in Elasticsearch. Essential for creating dashboards and managing visualizations.

| Snapshot Storage
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[elastic-cloud-architecture]]
=== Time Series HA Architecture: Elastic Cloud - Hot-Frozen, Multi-Availability Zone Architecture
=== Elastic Cloud - Multi-Availability Zone

The Hot-Frozen Elasticsearch cluster architecture is cost optimized for large time-series datasets while keeping all of the data **fully searchable**. There is no need to "re-hydrate" archived data. In this architecture, the hot tier is primarily used for indexing and immediate searching (1-3 days) with a majority of the search being handled by the frozen tier. Since the data is moved to searchable snapshots in an object store, the cost of keeping all of the data searchable is dramatically reduced.

Expand All @@ -14,7 +14,6 @@ This architecture is intended for organizations that need to:
* Monitor the performance and health of their applications in real-time, including the creation and tracking of SLOs (Service Level Objectives).
* Provide insights and alerts to ensure optimal performance and quick issue resolution for applications.
* Apply Machine Learning and Artificial Intelligence to assist SREs and Application Teams in dealing with the large amount of data in this type of use case.
* Generative ai? OTEL collector? Highlight the stuff that sells…


[discrete]
Expand Down Expand Up @@ -158,12 +157,15 @@ The following list are important conderations for this architecture:
*** *Distribution:* In a distributed system, any distributed process is only as fast as the slowest node. As a result, it is optimal to maintain indexes with a primary shard count that is a multiple of the node count in a given tier. This creates even distribution of processing and prevents hotspots.
**** Shard distribution should be enforced using the https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#avoid-node-hotspots[‘total shards per node’] index level setting

TIP: For consistent index level settings is it easiest to use index lifecycle management with index templates, please see the section below for more detail.
* **Index lifecyle:** Use index lifecycle management with index templates for consistent index level settings, please see, https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html[Configure a lifecycle policy] for more detail.
** *Hot:* Use this tier for ingestion and the fastest reads on the most current data. This architecture assumes no updates to the data once written.
** **Warm / Cold** - This tier is not considered for this pattern.
** **Frozen:** Data is persisted in a repository; however, it is accessed from the node's cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy. Frozen does not mean slow - it means immutable and saved in durable storage.

* **Architecture Variant - adding a Cold Tier**
** The hot-frozen architecture works well for most time-series use cases. However, when there is a need for more frequent, low-latency searches, introducing a cold tier may be required. Some common examples include detection rule lookback for security use cases or complex custom dashboards. The ILM policy for the example Hot-Frozen architecture above could be modified from 1 day in hot, 89 in frozen to 1 day in hot, 7 days in cold, and 82 days in frozen. Cold nodes fully mount a searchable snapshot for primary shards; replica shards are not needed for reliability. In the event of a failure, cold tier nodes can recover data from the underlying snapshot instead. See https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html[Data tiers] for more details on Elasticsearch data tiers. Note: our Data tiers docs may be slightly at odds with the concept of hot/frozen or hot/cold/frozen. Should they be updated?
* **Limitations of this architecture.**
** This architecture is a high-availability Elasticsearch architecture. It is not intended as a Disaster Recovery architecture since it is deployed across Availability Zones in a single cloud region. This architecture can be enhanced for Disaster Recovery by adding a second deployment in another cloud region. Details on Disaster Recovery for Elasticsearch can be found here.
* **Limitations of this architecture**
** This architecture is a high-availability Elasticsearch architecture. It is not intended as a Disaster Recovery architecture since it is deployed across Availability Zones in a single cloud region. This architecture can be enhanced for Disaster Recovery by adding a second deployment in another cloud region. Details on Disaster Recovery for Elasticsearch can be found https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html#ccr-disaster-recovery[here].

[discrete]
[[cloud-hot-frozen-resources]]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[[reference-architecture-high-availability]]
== High availability
== Time Series Highly Available Architectures

This page outlines reference architectures designed to ensure high availability for time series data in Elasticsearch deployments. These architectures leverage Elasticsearch's features—such as shard allocation, and replication to provide resilient, scalable solutions.

* <<multi-region-two-datacenter-architecture>>
* <<elastic-cloud-architecture>>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[multi-region-two-datacenter-architecture]]
=== Time Series HA Architecture: Multi-Region With Two Datacenters
=== Multi-Region - Two Datacenters

This article defines a scalable and highly available architecture for Elasticsearch using two datacenters in separate geographical regions. The architecture includes all the necessary components of the Elastic Stack and is not intended for sizing workloads, but rather as a basis to ensure the architecture you deploy is foundationally ready to handle any desired workload with resiliency. This architecture does include very high level representations of data flow, but the implementation of which will be included in subsequent documentation.

Expand Down Expand Up @@ -30,12 +30,15 @@ The following list are important conderations for this architecture:
*** **Distribution:** In a distributed system, any distributed process is only as fast as the slowest node. As a result, it is optimal to maintain indexes with a primary shard count that is a multiple of the node count in a given tier. This creates even distribution of processing and prevents hotspots.
**** Shard distribution should be enforced using the https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#avoid-node-hotspots['total shards per node'] index level setting

TIP: For consistent index level settings is it easiest to use index lifecycle management with index templates, please see the section below for more detail.
* **Index lifecyle:** Use index lifecycle management with index templates for consistent index level settings, please see, https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html[Configure a lifecycle policy] for more detail.
** *Hot:* Use this tier for ingestion and the fastest reads on the most current data. This architecture assumes no updates to the data once written.
** **Warm / Cold** - This tier is not considered for this pattern.
** **Frozen:** Data is persisted in a repository; however, it is accessed from the node's cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy. Frozen does not mean slow - it means immutable and saved in durable storage.

*** **Shard allocation awareness:** To prevent both a primary and a replica from being copied to the same zone, or in this case the same pod, you can use https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness[shard allocation awareness] and define a simple attribute in the elaticsearch.yaml file on a per-node basis to make Elasticsearch aware of the physical topology and route shards appropriately. In deployment models with multiple availability zones, AZ's would be used in place of pod location.

*** **Disadvantages of this architecture**
**** No region resilience
* **Limitations of this architecture**
** No region resilience

[discrete]
[[multi-region-resources]]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[self-managed-single-datacenter]]
=== Elastic Highly Available Architecture: Self Managed - Single Datacenter
=== Self Managed - Single Datacenter

This architecture ensures high availability during normal operations and node maintenance. It includes all necessary Elastic Stack components but is not intended for workload sizing. Use this pattern as a foundation and extend it to meet your specific needs. While it represents data flow for context, implementations may vary. Key design elements include the number and location of master nodes, data nodes, zone awareness, and shard allocation strategy. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design-large-clusters.html#high-availability-cluster-design-two-zones[Resilience in larger clusters - Two-zone clusters]. This design does not cover cross-region (geographically diverse) disaster recovery.

Expand Down Expand Up @@ -64,8 +64,10 @@ The following list are important conderations for this architecture:

**** Shard distribution should be enforced using the https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#avoid-node-hotspots[‘total shards per node’] index level setting

TIP: For consistent index level settings is it easiest to use index lifecycle management with index templates, please see the section below for more detail.

* **Index lifecyle:** Use index lifecycle management with index templates for consistent index level settings, please see, https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html[Configure a lifecycle policy] for more detail.
** *Hot:* Use this tier for ingestion and the fastest reads on the most current data. This architecture assumes no updates to the data once written.
** **Warm / Cold** - This tier is not considered for this pattern.
** **Frozen:** Data is persisted in a repository; however, it is accessed from the node's cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy. Frozen does not mean slow - it means immutable and saved in durable storage.
*** *Shard allocation awareness:* To prevent both a primary and a replica from being copied to the same zone, or in this case the same pod, you can use https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness[shard allocation awareness] and define a simple attribute in the elaticsearch.yaml file on a per-node basis to make Elasticsearch aware of the physical topology and route shards appropriately. In deployment models with multiple availability zones, AZ's would be used in place of pod location.

* **Forced Awareness:** This should be set in In order to prevent Elastic from trying to create replica shards when a given POD is down for maintenance.
Expand All @@ -79,11 +81,11 @@ TIP: For consistent index level settings is it easiest to use index lifecycle ma
***** Data is persisted in a repository; however, it is accessed from the node’s cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy.
***** Frozen does not mean slow - it means immutable and saved in durable storage.
* https://www.elastic.co/guide/en/elasticsearch/reference/8.16/snapshots-take-snapshot.html#automate-snapshots-slm[SLM (Snapshot Lifecycle Management): Considerations]
* *Limitations of this pattern*
* **Limitations of this architecture**
** No region resilience
** Only a single copy of (some of … i.e. the most recently written data that is not yet part of a snapshot) data exists during maintenance windows - (Note: This could be addressed by adding data nodes to POD 3 and setting the sharding strategy to 1 Primary and 2 Replicas)
** Assumes write once (no updating of documents)
* **Benefits of this pattern**
* **Benefits of this architecture**
** Reduces cost by leveraging the Frozen tier as soon as that makes sense from an ingest and most frequently read documents perspective
** Significantly reduces the likelihood of hot-spotting due to the sharding strategy
** Eliminates network and disk overhead caused by rebalancing attempts that would occur during maintenance due to setting forced awareness.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[three-availability-zones]]
=== Time Series HA Architecture: Three availability zones
=== Three availability zones

This article outlines a scalable and highly available architecture for Elasticsearch using three availability zones. The architecture encompasses all essential components of the Elastic Stack and serves as a foundational blueprint to ensure your deployment is resilient and ready to handle any desired workload. While this overview includes high-level representations of data flow, detailed implementation will be covered in subsequent documentation.

Expand Down Expand Up @@ -42,23 +42,17 @@ The following list are important conderations for this architecture:
*** **Distribution:** In a distributed system, any distributed process is only as fast as the slowest node. As a result, it is optimal to maintain indexes with a primary shard count that is a multiple of the node count in a given tier. This creates even distribution of processing and prevents hotspots.
**** Shard distribution should be enforced using the https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#avoid-node-hotspots['total shards per node'] index level setting

TIP: For consistent index level settings is it easiest to use index lifecycle management with index templates, please see the section below for more detail.

* https://www.elastic.co/guide/en/elasticsearch/reference/8.16/data-tiers.html[ILM (Information Lifecycle Management): Considerations]
** Hot:
*** Use this tier for ingestion. (Note: we are assuming for this pattern no updates to the data once written).
*** Use this tier for fastest reads on the most current data.
** Warm / Cold - not considered for this pattern.
** Frozen:
*** Data is persisted in a repository; however, it is accessed from the node’s cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy.
*** Frozen does not mean slow - it means immutable and saved in durable storage.
* **Index lifecyle:** Use index lifecycle management with index templates for consistent index level settings, please see, https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html[Configure a lifecycle policy] for more detail.
** *Hot:* Use this tier for ingestion and the fastest reads on the most current data. This architecture assumes no updates to the data once written.
** **Warm / Cold** - This tier is not considered for this pattern.
** **Frozen:** Data is persisted in a repository; however, it is accessed from the node's cache. It may not be as fast as the Hot tier; however, it can still be fast depending on the caching strategy. Frozen does not mean slow - it means immutable and saved in durable storage.

* https://www.elastic.co/guide/en/elasticsearch/reference/8.16/snapshots-take-snapshot.html#automate-snapshots-slm[SLM (Snapshot Lifecycle Management): Considerations]
* *Limitations of this pattern*
* **Limitations of this architecture**
** No region resilience
** Only a single copy of (some of … i.e. the most recently written data that is not yet part of a snapshot) data exists during maintenance windows - (Note: This could be addressed by adding data nodes to POD 3 and setting the sharding strategy to 1 Primary and 2 Replicas)
** Assumes write once (no updating of documents)
* **Benefits of this pattern**
* **Benefits of this architecture**
** Reduces cost by leveraging the Frozen tier as soon as that makes sense from an ingest and most frequently read documents perspective
** Significantly reduces the likelihood of hot-spotting due to the sharding strategy
** Eliminates network and disk overhead caused by rebalancing attempts that would occur during maintenance due to setting forced awareness.
Expand Down

0 comments on commit b5c46bf

Please sign in to comment.