Skip to content

Commit

Permalink
updates
Browse files Browse the repository at this point in the history
  • Loading branch information
georgewallace committed Oct 29, 2024
1 parent e6bb795 commit 1d642a2
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

The Hot-Frozen Elasticsearch cluster architecture is cost optimized for large time-series datasets while keeping all of the data **fully searchable**. There is no need to "re-hydrate" archived data. In this architecture, the hot tier is primarily used for indexing and immediate searching (1-3 days) with a majority of the search being handled by the frozen tier. Since the data is moved to searchable snapshots in an object store, the cost of keeping all of the data searchable is dramatically reduced.

This architecture is ideal for observability use cases. The architecture includes all the necessary components of the Elastic Stack and is not intended for sizing workloads, but rather as a basis to ensure the architecture you deploy is foundationally ready to handle any desired workload with resiliency. This architecture shows a very high level representation of data flow. For more details on that, see this <LINK>.
TIP: This architecture includes all the necessary components of the Elastic Stack and is not intended for sizing workloads, but rather as a basis to ensure the architecture you deploy is foundationally ready to handle any desired workload with resiliency.

[discrete]
[[cloud-hot-use-case]]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,22 @@
[[self-managed-single-datacenter]]
=== Self managed - single datacenter

This architecture ensures high availability during normal operations and node maintenance.
This architecture is designed to ensure high availability during normal operations and node maintenance. It reduces cost by leveraging the frozen tier as soon as it makes sense from an ingest and most frequently read documents perspective. It significantly reduces the likelihood of hot-spotting due to the sharding strategy. Additionally it eliminates network and disk overhead caused by rebalancing attempts that would occur during maintenance due to setting forced awareness. For more information on elements of this architecture, see https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design-large-clusters.html#high-availability-cluster-design-two-zones[Resilience in larger clusters - Two-zone clusters].

TIP: This architecture includes all the necessary components of the Elastic Stack and is not intended for sizing workloads, but rather as a basis to ensure the architecture you deploy is foundationally ready to handle any desired workload with resiliency.

Key design elements include the number and location of master nodes, data nodes, zone awareness, and shard allocation strategy. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design-large-clusters.html#high-availability-cluster-design-two-zones[Resilience in larger clusters - Two-zone clusters].

While this architecture does include a representation of a data flow, this is being provided for contextual understanding and may differ from implementation to implementation. The critical portion of the design is the number and location of master nodes, the location of data nodes, zone awareness and the shard allocation strategy. For additional information see this https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-design-large-clusters.html#high-availability-cluster-design-two-zones[reference].

[discrete]
[[single-datacenter-use-case]]
==== Use case

This architecture is intended for organizations that need to do the following:

* Store data that is written once and not updated (e.g. logs, metrics or even an accounting ledger where balance updates are done via additional offsetting entries)
* Be resilient to hardware failures
* Ensure availability during operational maintenance of any given (zone i.e. POD in the diagram)
* Maintain a single copy of the data during maintenance
* Leverage a Frozen Data tier as part of the https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html[Information Lifecycle]
* Leverage a https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html[Snapshot Repository] for additional recovery options
* Store data that is written once and not updated including logs. metrics or even accounting ledger where balance updates are done via additional offsetting entries.
* Be resilient to hardware failures.
* Ensure availability during operational maintenance of any given pod.
* Maintain a single copy of the data during maintenance.
* Leverage a Frozen Data tier as part of the https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html[Information Lifecycle].
* Leverage a https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html[Snapshot Repository] for additional recovery options.

[discrete]
[[single-datacenter-architecture]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

This article outlines a scalable and highly available architecture for Elasticsearch using three availability zones. The architecture encompasses all essential components of the Elastic Stack and serves as a foundational blueprint to ensure your deployment is resilient and ready to handle any desired workload. While this overview includes high-level representations of data flow, detailed implementation will be covered in subsequent documentation.

TIP: This architecture includes all the necessary components of the Elastic Stack and is not intended for sizing workloads, but rather as a basis to ensure the architecture you deploy is foundationally ready to handle any desired workload with resiliency.

[discrete]
[[three-availability-zones-use-case]]
==== Use case
Expand Down

0 comments on commit 1d642a2

Please sign in to comment.