From ceebfd0fc50844a6a1332b825d9248e438349796 Mon Sep 17 00:00:00 2001 From: Michael Russell Date: Wed, 20 Jun 2018 07:46:13 +0200 Subject: [PATCH] Update readme and add changelog into git for the 6.3 release --- CHANGELOG.md | 72 ++++++++++++++++++ Makefile | 2 +- README.md | 211 +++++++++++++++++++++++++-------------------------- 3 files changed, 177 insertions(+), 108 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..35bb0a5a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,72 @@ +## 6.3.0 - 2018/06/18 + +### Breaking changes + +Elasticsearch 6.3 includes several big changes that are reflected in this role. +When upgrading from module versions prior to 6.3, there are a number of upgrade considerations to take into account: + +* This role defaults to the upstream package repositories, which now include X-Pack bundled by default. To preserve previous behavior which does _not_ include X-Pack be sure to explicitly set `es_enable_xpack: true` which will install the `elasticsearch-oss` package. +* Great care has been taken in making sure that all upgrade paths work, however as always please take extra caution when upgrading and test in a non-production environment. New automated tests have been added to make sure that the following upgrade paths work: + * oss to oss + * oss to xpack + * xpack to xpack +* X-Pack configuration files which used to be in `${ES_PATH_CONF}/x-pack` are now in `${ES_PATH_CONF}/`. If you have any configuration files in this directory not managed by ansible you will need to move them manually. + +#### Features + +* Integration testing has been refactored in [#457](https://github.com/elastic/ansible-elasticsearch/pull/457). This removed a lot of duplicate tests and added new tests to make sure all upgrade paths work. +* It is now possible to test elasticsearch snapshot builds by setting `es_use_snapshot_release` to `true` + +#### Fixes + +* Installing `es_plugins` from custom urls is now idempotent. Previously the plugin name was being compared to the url which meant it would be reinstalled every time ansible was run because they didn't match + +#### Pull requests + +* [#452](https://github.com/elastic/ansible-elasticsearch/pull/452) - @Crazybus - Add initial 6.3 support +* [#454](https://github.com/elastic/ansible-elasticsearch/pull/454) - @Crazybus - Move jenkins matrix file into the repo so test suites are controlled via the pull request workflow +* [#455](https://github.com/elastic/ansible-elasticsearch/pull/455) - @Crazybus - Add automated test for upgrading from oss to oss +* [#457](https://github.com/elastic/ansible-elasticsearch/pull/457) - @Crazybus - Refactor integration tests to remove duplication and add extra suites to make sure all upgrade paths are covered + +## 6.2.4.1 - 2018/06/14 + +Patch release requested by @average-joe in #453 + +#### Pull requests + +* [#445](https://github.com/elastic/ansible-elasticsearch/pull/445) - @gekkeharry13 - Added configuration options for configuring x-pack notifications via email with some other nice fixes. +* [#450](https://github.com/elastic/ansible-elasticsearch/pull/450) - @Crazybus - improving some flakey tests which were randomly failing. +* [#447](https://github.com/elastic/ansible-elasticsearch/pull/447) - @chaintng - Fix to make sure sudo is used when running `update-alternatives` for java. +* [#423](https://github.com/elastic/ansible-elasticsearch/pull/423) - @eRadical - Fixing the until condition being used when installing rpms from a custom repository. + + +## 6.2.4 - 2018/04/24 + +* `6.2.4` and `5.6.9` as the default versions. + +## 6.2.3 - 2018/04/21 + +* Thanks to @cl0udf0x for adding proper names to all tasks which were unnamed [#417](https://github.com/elastic/ansible-elasticsearch/pull/417) +* Thanks @cyrilleverrier for having a keen eye and spotting this typo. [#432](https://github.com/elastic/ansible-elasticsearch/pull/432) + +## 6.2.2 - 2018/02/22 + +* `6.2.2` and `5.6.8` as the default versions +* Thanks to @pemontto for fixing up all of the ansible conditional logic https://github.com/elastic/ansible-elasticsearch/pull/429 +* Thanks @cyrilleverrier for https://github.com/elastic/ansible-elasticsearch/pull/427 which makes sure x-pack settings are not in the config file when x-pack isn't enabled + +## 6.1.3 - 2018/02/01 + +* `6.x` is now the default `es_major_version` with `6.1.3` as the default `es_version` +* Special thanks to @shribigb, @toddlers and @remil1000 for their efforts in getting `6.x` support working! +* `.kitchen.yml` has been updated to allow testing both `6.x` and `5.x` versions +* A new [Jenkins job](https://devops-ci.elastic.co/job/elastic+ansible-elasticsearch+pull-request/) has been added for pull requests to automatically test all combinations of `6.x` and `5.x` on ubuntu-1404, ubuntu-1604, debian-8 and centos-7 with the various test suites. + +## 5.5.1 - 2017/08/20 + +* Fixes with respect to issues on restart. +* 5.5.1 update with supporting package scripts. +* Documentation clarification. +* Fixes for loading of templates +* Support for ML +* Ability to install x-pack from remote. \ No newline at end of file diff --git a/Makefile b/Makefile index 7fe4508c..d5e4d146 100644 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ default: build SHELL:=/bin/bash -eux VERSION := 6.x -PATTERN := xpack-standard-ubuntu-1604 +PATTERN := xpack-ubuntu-1604 .PHONY: converge cerify test login destroy list diff --git a/README.md b/README.md index 191658ba..9cf99606 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,10 @@ Ansible role for 6.x/5.x Elasticsearch. Currently this works on Debian and RedHat based linux systems. Tested platforms are: -* Ubuntu 14.04/16.04 +* Ubuntu 14.04 +* Ubuntu 16.04 * Debian 8 -* Centos 7 +* CentOS 7 The latest Elasticsearch versions of 6.x and 5.x are actively tested. **Only Ansible versions > 2.4.3.0 are supported, as this is currently the only version tested.** @@ -19,12 +20,8 @@ This role uses the json_query filter which [requires jmespath](https://github.co Create your Ansible playbook with your own tasks, and include the role elasticsearch. You will have to have this repository accessible within the context of playbook. -``` -cd /my/repos/ -git clone https://github.com/elastic/ansible-elasticsearch.git -cd /my/ansible/playbook -mkdir -p roles -ln -s /my/repos/ansible-elasticsearch ./roles/elasticsearch +```sh +ansible-galaxy install elastic.elasticsearch ``` Then create your playbook yaml adding the role elasticsearch. By default, the user is only required to specify a unique es_instance_name per role application. This should be unique per node. @@ -32,12 +29,12 @@ The application of the elasticsearch role results in the installation of a node The simplest configuration therefore consists of: -``` +```yaml - name: Simple Example hosts: localhost roles: - - { role: elasticsearch, es_instance_name: "node1" } - vars: + - role: elasticsearch + es_instance_name: "node1" ``` The above installs a single node 'node1' on the hosts 'localhost'. @@ -58,42 +55,42 @@ This playbook uses [Kitchen](https://kitchen.ci/) for CI and local testing. ### Running the tests If you want to test X-Pack features with a license you will first need to export the `ES_XPACK_LICENSE_FILE` variable. -``` +```sh export ES_XPACK_LICENSE_FILE="$(pwd)/license.json" ``` To converge an Ubuntu 16.04 host running X-Pack -``` +```sh $ make converge ``` To run the tests -``` +```sh $ make verify ``` To list all of the different test suits -``` +```sh $ make list ``` The default test suite is Ubuntu 16.04 with X-Pack. If you want to test another suite you can override this with the `PATTERN` variable -``` -$ make converge PATTERN=standard-centos-7 +```sh +$ make converge PATTERN=oss-centos-7 ``` The `PATTERN` is a kitchen pattern which can match multiple suites. To run all tests for CentOS -``` +```sh $ make converge PATTERN=centos-7 ``` The default version is 6.x If you want to test 5.x you can override it with the `VERSION` variable to test 5.x -``` -$ make converge VERSION=5.x PATTERN=standard-centos-7 +```sh +$ make converge VERSION=5.x PATTERN=oss-centos-7 ``` When you are finished testing you can clean up everything with -``` +```sh $ make destroy-all ``` @@ -104,26 +101,27 @@ The use of a map ensures the Ansible playbook does not need to be updated to ref In addition to the es_config map, several other parameters are supported for additional functions e.g. script installation. These can be found in the role's defaults/main.yml file. -The following illustrates applying configuration parameters to an Elasticsearch instance. By default, Elasticsearch 5.1.2is installed. +The following illustrates applying configuration parameters to an Elasticsearch instance. -``` +```yaml - name: Elasticsearch with custom configuration hosts: localhost roles: - #expand to all available parameters - - { role: elasticsearch, es_instance_name: "node1", es_data_dirs: "/opt/elasticsearch/data", es_log_dir: "/opt/elasticsearch/logs", - es_config: { - node.name: "node1", - cluster.name: "custom-cluster", - discovery.zen.ping.unicast.hosts: "localhost:9301", - http.port: 9201, - transport.tcp.port: 9301, - node.data: false, - node.master: true, - bootstrap.memory_lock: true, - } - } + - role: elasticsearch vars: + es_instance_name: "node1" + es_data_dirs: + - "/opt/elasticsearch/data" + es_log_dir: "/opt/elasticsearch/logs" + es_config: + node.name: "node1" + cluster.name: "custom-cluster" + discovery.zen.ping.unicast.hosts: "localhost:9301" + http.port: 9201 + transport.tcp.port: 9301 + node.data: false + node.master: true + bootstrap.memory_lock: true es_scripts: false es_templates: false es_version_lock: false @@ -145,24 +143,25 @@ The role makes no attempt to enforce the setting of these are requires users to A more complex example: -``` +```yaml - name: Elasticsearch with custom configuration hosts: localhost roles: - #expand to all available parameters - - { role: elasticsearch, es_instance_name: "node1", es_data_dirs: "/opt/elasticsearch/data", es_log_dir: "/opt/elasticsearch/logs", - es_config: { - node.name: "node1", - cluster.name: "custom-cluster", - discovery.zen.ping.unicast.hosts: "localhost:9301", - http.port: 9201, - transport.tcp.port: 9301, - node.data: false, - node.master: true, - bootstrap.memory_lock: true, - } - } + - role: elasticsearch vars: + es_instance_name: "node1" + es_data_dirs: + - "/opt/elasticsearch/data" + es_log_dir: "/opt/elasticsearch/logs" + es_config: + node.name: "node1" + cluster.name: "custom-cluster" + discovery.zen.ping.unicast.hosts: "localhost:9301" + http.port: 9201 + transport.tcp.port: 9301 + node.data: false + node.master: true + bootstrap.memory_lock: true es_scripts: false es_templates: false es_version_lock: false @@ -171,9 +170,9 @@ A more complex example: es_plugins_reinstall: false es_api_port: 9201 es_plugins: - - plugin: ingest-geoip - proxy_host: proxy.example.com - proxy_port: 8080 + - plugin: ingest-geoip + proxy_host: proxy.example.com + proxy_port: 8080 ``` #### Important Note @@ -189,21 +188,21 @@ An example of a two server deployment is shown below. The first server holds th **Note the structure of the below playbook for the data nodes. Whilst a more succinct structures are possible which allow the same role to be applied to a host multiple times, we have found the below structure to be the most reliable with respect to var behaviour. This is the tested approach.** -``` +```yaml - hosts: master_nodes roles: - - { role: elasticsearch, es_instance_name: "node1", es_heap_size: "1g", - es_config: { - cluster.name: "test-cluster", - discovery.zen.ping.unicast.hosts: "elastic02:9300", - http.port: 9200, - transport.tcp.port: 9300, - node.data: false, - node.master: true, - bootstrap.memory_lock: false, - } - } + - role: elasticsearch vars: + es_instance_name: "node1" + es_heap_size: "1g" + es_config: + cluster.name: "test-cluster" + discovery.zen.ping.unicast.hosts: "elastic02:9300" + http.port: 9200 + transport.tcp.port: 9300 + node.data: false + node.master: true + bootstrap.memory_lock: false es_scripts: false es_templates: false es_version_lock: false @@ -211,53 +210,50 @@ An example of a two server deployment is shown below. The first server holds th es_plugins: - plugin: ingest-geoip - - - hosts: data_nodes roles: - - { role: elasticsearch, es_instance_name: "node1", es_data_dirs: "/opt/elasticsearch", - es_config: { - discovery.zen.ping.unicast.hosts: "elastic02:9300", - http.port: 9200, - transport.tcp.port: 9300, - node.data: true, - node.master: false, - bootstrap.memory_lock: false, - cluster.name: "test-cluster" - } - } + - role: elasticsearch vars: + es_instance_name: "node1" + es_data_dirs: + - "/opt/elasticsearch" + es_config: + discovery.zen.ping.unicast.hosts: "elastic02:9300" + http.port: 9200 + transport.tcp.port: 9300 + node.data: true + node.master: false + bootstrap.memory_lock: false + cluster.name: "test-cluster" es_scripts: false es_templates: false es_version_lock: false ansible_user: ansible es_api_port: 9200 es_plugins: - - plugin: ingest-geoip - + - plugin: ingest-geoip - hosts: data_nodes roles: - - { role: elasticsearch, es_instance_name: "node2", es_api_port:9201, - es_config: { - discovery.zen.ping.unicast.hosts: "elastic02:9300", - http.port: 9201, - transport.tcp.port: 9301, - node.data: true, - node.master: false, - bootstrap.memory_lock: false, - cluster.name: "test-cluster", - } - } + - role: elasticsearch vars: + es_instance_name: "node2" + es_api_port: 9201 + es_config: + discovery.zen.ping.unicast.hosts: "elastic02:9300" + http.port: 9201 + transport.tcp.port: 9301 + node.data: true + node.master: false + bootstrap.memory_lock: false + cluster.name: "test-cluster" es_scripts: false es_templates: false es_version_lock: false es_api_port: 9201 ansible_user: ansible es_plugins: - - plugin: ingest-geoip - + - plugin: ingest-geoip ``` Parameters can additionally be assigned to hosts using the inventory file if desired. @@ -266,7 +262,7 @@ Make sure your hosts are defined in your ```inventory``` file with the appropria Then run it: -``` +```sh ansible-playbook -i hosts ./your-playbook.yml ``` @@ -283,7 +279,7 @@ The following additional parameters allow X-Pack to be configured: * ```es_role_mapping``` Role mappings file declared as yml as described [here](https://www.elastic.co/guide/en/x-pack/current/mapping-roles.html) -``` +```yaml es_role_mapping: power_user: - "cn=admins,dc=example,dc=com" @@ -294,7 +290,7 @@ es_role_mapping: * ```es_users``` - Users can be declared here as yml. Two sub keys 'native' and 'file' determine the realm under which realm the user is created. Beneath each of these keys users should be declared as yml entries. e.g. -``` +```yaml es_users: native: kibana4_server: @@ -316,7 +312,7 @@ es_users: * ```es_roles``` - Elasticsearch roles can be declared here as yml. Two sub keys 'native' and 'file' determine how the role is created i.e. either through a file or http(native) call. Beneath each key list the roles with appropriate permissions, using the file based format described [here] (https://www.elastic.co/guide/en/x-pack/current/file-realm.html) e.g. -``` +```yaml es_roles: file: admin: @@ -359,13 +355,13 @@ es_roles: * ```es_xpack_license``` - X-Pack license. The license is a json blob. Set the variable directly (possibly protected by Ansible vault) or from a file in the Ansible project on the control machine via a lookup: -``` +```yaml es_xpack_license: "{{ lookup('file', playbook_dir + '/files/' + es_cluster_name + '/license.json') }}" ``` X-Pack configuration parameters can be added to the elasticsearch.yml file using the normal `es_config` parameter. -For a full example see [here](https://github.com/elastic/ansible-elasticsearch/blob/master/test/integration/xpack.yml) +For a full example see [here](https://github.com/elastic/ansible-elasticsearch/blob/master/test/integration/xpack-upgrade.yml) #### Important Note for Native Realm Configuration @@ -381,8 +377,9 @@ These can either be set to a user declared in the file based realm, with admin p In addition to es_config, the following parameters allow the customization of the Java and Elasticsearch versions as well as the role behaviour. Options include: +* ```es_enable_xpack``` Default `true`. Setting this to `false` will install the oss release of elasticsearch * ```es_major_version``` Should be consistent with es_version. For versions >= 5.0 and < 6.0 this must be "5.x". For versions >= 6.0 this must be "6.x". -* ```es_version``` (e.g. "6.1.2"). +* ```es_version``` (e.g. "6.3.0"). * ```es_api_host``` The host name used for actions requiring HTTP e.g. installing templates. Defaults to "localhost". * ```es_api_port``` The port used for actions requiring HTTP e.g. installing templates. Defaults to 9200. **CHANGE IF THE HTTP PORT IS NOT 9200** * ```es_api_basic_auth_username``` The Elasticsearch username for making admin changing actions. Used if Security is enabled. Ensure this user is admin. @@ -390,10 +387,10 @@ In addition to es_config, the following parameters allow the customization of th * ```es_start_service``` (true (default) or false) * ```es_plugins_reinstall``` (true or false (default) ) * ```es_plugins``` an array of plugin definitions e.g.: -```yml - es_plugins: - - plugin: ingest-geoip -``` + ```yaml + es_plugins: + - plugin: ingest-geoip + ``` * ```es_path_repo``` Sets the whitelist for allowing local back-up repositories * ```es_action_auto_create_index ``` Sets the value for auto index creation, use the syntax below for specifying indexes (else true/false): es_action_auto_create_index: '[".watches", ".triggered_watches", ".watcher-history-*"]' @@ -410,7 +407,7 @@ Earlier examples illustrate the installation of plugins using `es_plugins`. For If installing Monitoring or Alerting, ensure the license plugin is also specified. Security configuration currently has limited support, but more support is planned for later versions. To configure X-pack to send mail, the following configuration can be added to the role. When require_auth is true, you will also need to provide the user and password. If not these can be removed: -``` +```yaml es_mail_config: account: profile: standard @@ -451,7 +448,7 @@ To define proxy globally, set the following variables: To define proxy only for a particular plugin during its installation: -``` +```yaml es_plugins: - plugin: ingest-geoip proxy_host: proxy.example.com @@ -478,4 +475,4 @@ To define proxy only for a particular plugin during its installation: ## Questions on Usage -We welcome questions on how to use the role. However, in order to keep the GitHub issues list focused on "issues" we ask the community to raise questions at https://discuss.elastic.co/c/elasticsearch. This is monitored by the maintainers. +We welcome questions on how to use the role. However, in order to keep the GitHub issues list focused on "issues" we ask the community to raise questions at https://discuss.elastic.co/c/elasticsearch. This is monitored by the maintainers. \ No newline at end of file