7.1.1

Elasticsearch, Kibana and Logstash 7.1.1

This release adds support for deploying Elasticsearch Kibana and Logstash 7.1.1 to Azure!

Features

#281 Add Elastic Stack 7.1.1 🥇

This release adds Elastic Stack 7.1.1 to the template, with 7.1.1 now the default version.

The most notable feature of Elastic Stack 7.1.1 is that some Security features are now free with the basic license tier, starting in 6.8.0 and 7.1.0. These features include

• File and Native realms for creating and managing users, the latter of which the template uses to configure the built-in users such as elastic, kibana
• Transport Layer Security (TLS)

This means that if you're deploying 6.8.0 or 7.1.1 with this version of the template, Basic Authentication will be configured for the built-in users using the supplied passwords whether you deploy with the basic license or a trial license.

If deploying with a basic license by specifying xPackPlugins with a value of No, you must also configure TLS for at least the Elasticsearch Transport layer by supplying a PKCS#12 archive for the esTransportCaCertBlob parameter. It is highly recommended that you also configure TLS for the Elasticsearch HTTP layer and Kibana too.

If deploying with a trial license by specifying xPackPlugins with a value of Yes, configuring TLS is optional, however, it is still highly recommended that you configure TLS, since TLS is required for at least the Elasticsearch Transport layer with basic, gold and platinum license levels, so configuring at deployment time may be easier than configuring TLS on each Elasticsearch node at a later point.

#282 Certificates can now be configured in the Azure Marketplace

A step has been added to the UI definition to allow Transport Layer Security (TLS) to be configured as part of template deployment from the Azure Marketplace

Since a deployment from the Marketplace always deploys with a trial license, configuring TLS is optional, to keep the getting started experience simple for those wishing to try the offering out. It is strongly recommended that you configure TLS however.

Breaking Changes

#285 Prefix loadbalancers, application gateway and ip addresses with vmHostNamePrefix

Loadbalancer, Application Gateway and public IP address resources are now all prefixed with the value of vmHostNamePrefix, if provided. This allows a new cluster to be deployed to an existing resource group and connected to an existing vnet where another cluster has already been deployed. Note that the vnet resource, if deployed, is not prefixed with vmHostNamePrefix, since its name can be supplied as part of template deployment.

Miscellaneous

• #284 Add important admonition to the docs about existing vnet subnet sizes

7.0.0

Elasticsearch, Kibana and Logstash 7.0.0

This release adds support for deploying Elasticsearch Kibana and Logstash 7.0.0 to Azure!

Features

#275 Add Elastic Stack 7.0.0 🥇

This release adds Elastic Stack 7.0.0 to the template, with 7.0.0 now the default version. 7.0.0 comes with a heap of new features, including:

New Kibana design and navigation

The most prominent change is a switch to a new global navigation, which introduces a constant header to switch Kibana spaces, display breadcrumbs, and initiate user actions like change password or logout.

Zen2 Cluster Coordination

Zen2 is a completely new cluster coordination layer for Elasticsearch, which is faster, safer, and easier to use. If you're interested in more details, the Zen2 blog post is a great read! The ARM template handles configuring zen2 for Elastic Stack 7.0.0 in addition to still configuring the original zen coordination layer for older versions.

Nanosecond precision

A migration from Joda-Time to the official Java time API in JDK 8 within the core of Elasticsearch, now provides better precision to accurately store and sequence data, useful in cases where data is collected at high frequencies.

plus many more. Check out the release blog post for more details..

#275 Set up passwords for apm_system and remote_monitoring_user built-in user accounts

As the Stack grows, so has the built-in user accounts used by Stack components. In addition to configuring passwords for elastic, kibana, logstash_system and beats_system built-in user accounts, the ARM template will allows you to set passwords for apm_system and remote_monitoring_user built-in user accounts. apm_system is used by the APM server to send APM data to Elasticsearch, and remote_monitoring_user is used by MetricBeat to send data.

Breaking Changes

#275 Remove es_read user account

Since the initial template deployment, an es_read user account has been configured as part of the deployment. The value of configuring this account on deployment has diminished over time, given how easy it is to configure users as a post-deployment step, using either the API or Users view in Kibana. For this reason, and to make way for apm_system and remote_monitoring_user built-in user account configuration, this account is no longer configured. If you'd like to add it as a post-deployment step, the following curl requests against the cluster will do so

# create the user role
curl -XPOST "<base uri>/_security/role/user" -u "elastic:<elastic superuser password>" -H 'Content-Type: application/json' -d'
{
    "cluster": ["monitor"],
    "indices": [{
        "names": ["*"],
        "privileges": ["read", "monitor", "view_index_metadata"]
    }]
}'

# create the es_read user
curl -XPOST "<base uri>/_security/user/es_read" -u "elastic:<elastic superuser password>" -H 'Content-Type: application/json' -d'
{
    "password":"<password for es_read>",
    "roles":["user"]
}'

6.7.0

Elasticsearch, Kibana and Logstash 6.7.0

This release adds support for deploying Elasticsearch Kibana and Logstash 6.7.0 to Azure!

Features

#270 Add Elastic Stack 6.7.0 🥇

This release adds Elastic Stack 6.7.0 to the template, with 6.7.0 now the default version. 6.7.0 comes with a heap of new features, including:

Elastic Maps

a new dedicated solution for mapping, querying, and visualizing geospatial data in Kibana. As an example, here's a layer of Australian suburb polygons and multipolygons from the Australian Bureau of Statistics

Logs and Infrastructure UIs go GA

Real-time log tailing and birds eye views of container and pods infrastructure, oh my!

plus many more. Check out the release blog post for more details..

#267 Add support for 8TiB, 16TiB and 32TiB managed disks

Need more disk space per VM? The template now supports 8TiB, 16TiB and 32TiB managed disks, providing capacity for a variety of use cases.

In addition to these larger disks, support has now been extended to 32GiB, 64GiB and 256GiB managed disk sizes.

#265 Add support for Lsv2-series VMs

With the general availability of Lsv2 series VMs, the template has been extended to support this new SKU. Lsv2-series features high throughput, low latency, and directly mapped local NVMe storage. This change adds support for the Lsv2-series VM, with an open issue #272 to add support for using the local NVMe storage instead of the tempoary disk when 0 managed disks are selected.

Breaking Changes

#267 vmDataDiskSize no longer accepts "T-Shirt" size disks

The template has supported providing "T-Shirt" size e.g. Small, Medium, Large, etc. inputs for the data disks size vmDataDiskSize parameter. With the inclusion of 8TiB, 16TiB and 32TiB managed disks, this sizing nomenclature was starting to hurt more than help. For example, the 32TiB disk size would be XXXXXLarge! This has now been replaced with sizes that represent the underlying disk size e.g. 32GiB, 64GiB up to 32TiB.

6.6.1

Elasticsearch, Kibana and Logstash 6.6.1

This release adds support for deploying Elasticsearch Kibana and Logstash 6.6.1 to Azure!

Features

#264 Add Elastic Stack 6.6.1 🥇

This release adds Elastic Stack 6.6.1 to the template, with 6.6.1 now the default version.

#264 Add support for 5.6.15

Bug Fixes

#263 Remove double // in linked resource URIs

A recent change in the Azure Marketplace means that URIs containing // as opposed to a single /, no longer correctly resolve to the resource.

6.6.0

Elasticsearch, Kibana and Logstash 6.6.0

This release adds support for deploying Elasticsearch Kibana and Logstash 6.6.0 to Azure!

Features

#257 Add Elastic Stack 6.6.0 🥇

This release adds Elastic Stack 6.6.0 to the template, with 6.6.0 now the default version.

6.6.0 includes new features such as

• Index Lifecycle Management
• Logs and Infrastructure UIs
• Frozen Indices

Plus many more! Check out the blog post on 6.6 to see all the new features available

#257 Add support for 6.5.4 and 5.6.14

Deprecations

#257 Remove support for 5.6.12, 5.5.x, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.4.1, 6.4.2

These versions have all been superseded by newer build versions in the minor lineage, or are now EOL.

6.5.3

Elasticsearch, Kibana and Logstash 6.5.3

This release adds support for deploying Elasticsearch Kibana and Logstash 6.5.3 to Azure!

Features

#254 Add Elastic Stack 6.5.3 🥇

This release adds Elastic Stack 6.5.3 to the template, with 6.5.3 now the default version.

#253 Add Elastic Stack 6.5.1 and 6.5.2 🏅

This release adds Elastic Stack 6.5.1 and 6.5.2 to the template.

Misc

#252 Replace Oracle JDK 8 with OpenJDK 8

With the licensing change with Oracle Java SE 8 coming into effect in January 2019, the template now deploys with OpenJDK 8 instead of Oracle JDK 8. This is currently the OpenJDK 8 build 191 apt package, which you should see as

$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

when executed on an Elasticsearch node or Logstash VM.

6.5.0

Elasticsearch, Kibana and Logstash 6.5.0

This release adds support for deploying Elasticsearch Kibana and Logstash 6.5.0 to Azure!

Features

#248 Add Elastic Stack 6.5.0 and 6.4.3 🏆

This release adds Elastic Stack 6.5.0 to the template, with 6.5.0 now the default version.

Bug fixes

• #237 Fix sha checksum for Elastic Stack versions less than 5.6.2

Deprecations

• #240 Remove 5.4.x version from the template

  This version has now reached support EOL

Misc

• #234 Remove locations with 2 Fault domains from configuration

  Template defaults to 2 Fault domains, so remove those from config that explicitly define 2.

6.4.2

Elasticsearch, Kibana and Logstash 6.4.2

This release adds support for deploying Elasticsearch Kibana and Logstash 6.4.2 to Azure!

Features

#223 Add Logstash to Marketplace UI and ARM template 🌳

Logstash has been added to the Marketplace UI and ARM template. This is an initial implementation, with possibility to expand in future based on feedback and usage. It deploys a single instance of Logstash, with configurable VM SKU, when logstash is set to Yes, with the default being No.

#225 and bc04447 Add support for Elasticsearch, Kibana and Logstash 6.4.1 and 6.4.2 🥇

Support is added for the latest Elastic Stack release, which brings a heap of new goodness including:

• Logstash integration with Azure Monitoring module
• Field aliases
• New Elasticsearch SQL enhancements
• Custom rules for fine tuning machine learning results

amongst others

#230 Enable accelerated networking for supported VM SKUs

Accelerated networking enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. This high-performance path bypasses the host from the datapath, reducing latency, jitter, and CPU utilization.

This feature enables accelerated networking by default for those VM SKUs that support it, with the ability to override this using specific parameters for

• vmMasterNodeAcceleratedNetworking: for master nodes
• vmDataNodeAcceleratedNetworking: for data nodes
• vmClientNodeAcceleratedNetworking: for coordinating nodes
• vmKibanaAcceleratedNetworking: for Kibana
• vmLogstashAcceleratedNetworking: for Logstash

#232 Use systemd to configure all of the Elastic Stack

systemd is now used for configuration Elasticsearch, Kibana and Logstash, and removes monit from the template; monit was introduced into the template in its infancy to work around Azure DNS availability on startup. Since that time, DNS is massively improved and systemd is capable of doing all monit is being used for, and is better known and understood.

#226 Add support for FSv2 series VMs 🖥

Compute optimized VM sizes with a high CPU-to-memory ratio

#228 Configure beats_system built-in user when configuring Security

Allows a cluster to be future Beats ready!

Bug fixes

• #231 Fix issue where Bash special characters in inputs such as passwords would cause deployment failures.

Breaking changes

• #229 Use resource group to get details for the Azure Repository account, and remove the Azure Storage account key and suffix parameters.

  We think this will provide an easier experience; fewer and less sensitive parameters.

6.3.1

Elasticsearch, Kibana and Logstash 6.3.1 🎈

This release adds support for deploying Elasticsearch and Kibana 6.3.1 to Azure!

Features

#209 Add support for Elasticsearch and Kibana 6.3.1

Add support for Elasticsearch and Kibana 6.3.1

#208 Configure shard allocation awareness over Fault and Update domains

When data, master and coordinating nodes are provisioned, they are added to an Availability Set and assigned a Fault and Update domain. These values are now added as node attributes and used to configure shard allocation awareness on the node.

#207 Larger disk options 💾

Since the beginning, a user has been able to configure the size of data disks to attach to each data node, to store data. Two new disk sizes have now been added, giving choices of

• 128GB (Small)
• 512GB (Medium)
• 1024GB (Large)
• 2048GB (XLarge)
• 4095GB (XXLarge)

With the default remaining as Large. Disks smaller than 2199023255040 bytes (~2199 GB) are partitioned with fdisk, while those larger are partitioned with parted, since they cannot use DOS partition table format and
instead must use Guid Partition table format (GPT).

#212 Use shared storage account for Azure repository plugin

The shared storage account deployed as part of the template can now be configured for use with the Azure repository plugin, reducing the need to provision a separate storage account. To use, pass a value of Yes for
azureCloudPlugin parameter.

#216 Documentation 📝

More comprehensive documentation for the template has now been added to the elastic.co website. Be sure to check out the version that corresponds to the template version being deployed.

#206 Configure custom probe for Application Gateway

When Application Gateway is selected as the loadBalancerType and xPackPlugins is Yes, a custom health probe is configured that now also considers a 401 response as healthy. This removes the need to configure anonymous access to the cluster base path.

Breaking changes

• #210 Update to Azure CLI 2.0 and remove support for Azure CLI 1.0

The template has been updated to now use Azure CLI 2.0 for integration testing. Support for Azure CLI 1.0 has now been removed since the last released version does not support newer API versions of resources.

• #211 Remove user information from template

The template no longer requires the separate collection of user information, simplifying the number of parameters that need to be provided. This data can now be surfaced from Azure infrastructure.

6.3.0

Elasticsearch and Kibana 6.3.0

This release adds support for deploying Elasticsearch and Kibana 6.3.0 to Azure! the default Elasticsearch and Kibana version in this template version is 6.3.0 with X-Pack bundled.

Check out the Doubling down on Open blog post to learn more and also take a look at the Opening X-Pack FAQ.

Features

#196 Elasticsearch and Kibana 6.3.0 🎉

Add support for Elasticsearch and Kibana 6.3.0 to the template. The template deploys the default distribution of 6.3.0 that includes X-Pack features bundled by default, with applied license level determining which features are activated. For 6.3.0, the xPackPlugins parameter determines whether a trial license is applied to the cluster (a value of Yes), providing access to all platinum features for 30 days. A value of No will apply a perpetual basic license that does not need to be registered.

#178 Managed disks 💾

The template has been updated to move from unmanaged to managed disks. Aside from being easier to manage and deploy, they provide encryption at rest by default, through Storage Service Encryption (SSE).

#187 SSL/TLS configuration for Elasticsearch HTTP and Transport layers 🔐

The template has supported SSL/TLS from the browser to Kibana and SSL/TLS to Elasticsearch through Application Gateway for a long time now. This new feature now brings the ability to configure SSL/TLS for both the HTTP and Transport layers of Elasticsearch:

• For HTTP layer, provide either

  • a PKCS#12 archive (esHttpCertBlob) and optional passphase (esHttpCertPassword) containing the certificate and key to use to encrypt the HTTP layer on all nodes

  or

  • a PKCS#12 archive (esHttpCaCertBlob) and optional passphase (esHttpCaCertPassword) containing the CA certificate and key to use to generate a unique certificate for each node to encrypt the HTTP layer

• For Transport layer provide a PKCS#12 archive (esTransportCaCertBlob) and optional passphase (esTransportCaCertPassword) containing the certificate and key to use to generate a unique certificate for each node to encrypt the Transport layer

#188 SAML based Single Sign-On with Azure Active Directory 🌐

It's now super simple to configure SAML based Single Sign-On (SSO) with Azure Active Directory; you just need to configure an Enterprise application for SAML SSO and provide the App Federation Metadata Url to the template in the samlMetadataUri. You can also optionally pass the domain name that you'll use to access Kibana with the samlServiceProviderUri. A blog post will be coming out soon with full details, so keep an eye on the elastic blog.

#187 Kibana additional YAML configuration 🍰

It's been possible to pass additional YAML configuration for Elasticsearch for a while, now it's possible to do the same for Kibana! This is an expert level feature, so be sure to run your yaml through a linter before running a deployment.

Enhancements

• #196 Support Elasticsearch and Kibana 5.6.10
• #193 Support Elasticsearch and Kibana 5.6.9
• #198 Support Dsv3-series VMs
• #205 Support additional VM SKUs, including L-series VMs

Bug fixes

• #184 Create keystore if it doesn't already exist

Deprecations

• #187 Remove support for Elasticsearch and Kibana less than 5.3.0