diff --git a/README.md b/README.md index ef33203a..9efda46b 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ You can use PEADM to deploy and manage PE installations for standard, large, and #### Table of contents - [Puppet Enterprise Administration Module (PEADM)](#puppet-enterprise-pe-administration-adm-module) - - [Table of contents](#table-of-contents) + - [Table of contents](#table-of-contents) - [Support](#support) - [Overview](#overview) - [What PEADM affects](#what-peadm-affects) @@ -24,7 +24,6 @@ You can use PEADM to deploy and manage PE installations for standard, large, and PEADM is a supported PE module. If you are a PE customer with the standard or premium support service, you can contact [Support](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet) or your Technical Account Manager for assistance. - ## Overview This is the standard workflow for installing PEADM. @@ -32,6 +31,7 @@ This is the standard workflow for installing PEADM. 1. [Install Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) on a jump host. 2. To create a Bolt project that includes the PEADM module, run:
`mkdir && cd && bolt project init --modules puppetlabs-peadm` 3. In the Bolt project directory, update the `inventory.yaml` file with connection information for the servers you plan to use for hosting PE infrastructure. For example: + ```yaml --- groups: @@ -55,56 +55,55 @@ groups: ### What PEADM affects -* The `peadm::install` plan adds a number of custom original identifier (OID) trusted facts to the certificates of deployed PE infrastructure nodes. These trusted facts are used by PEADM plans to identify nodes that host PE infrastructure components. -* Depending on the scale of your architecture, up to four node groups may be created to configure `puppet_enterprise` class parameters for the following PE infrastructure components: - * The primary server - * The primary server replica - * PostgreSQL nodes (database servers) - * Compilers (compiler hosts are designated as belonging to availability group A or B) +- The `peadm::install` plan adds a number of custom original identifier (OID) trusted facts to the certificates of deployed PE infrastructure nodes. These trusted facts are used by PEADM plans to identify nodes that host PE infrastructure components. +- Depending on the scale of your architecture, up to four node groups may be created to configure `puppet_enterprise` class parameters for the following PE infrastructure components: + - The primary server + - The primary server replica + - PostgreSQL nodes (database servers) + - Compilers (compiler hosts are designated as belonging to availability group A or B) ### What PEADM does not affect -* PEADM does not impact regular PE operations. After using it to deploy a new PE installation or upgrade an existing one, PEADM is not required until you want to use it to upgrade PE or expand your installation. -* Using PEADM to install PE or upgrade PE does not prevent you from using documented PE procedures such as setting up disaster recovery or performing a manual upgrade. +- PEADM does not impact regular PE operations. After using it to deploy a new PE installation or upgrade an existing one, PEADM is not required until you want to use it to upgrade PE or expand your installation. +- Using PEADM to install PE or upgrade PE does not prevent you from using documented PE procedures such as setting up disaster recovery or performing a manual upgrade. ### Requirements -* PEADM is compatible with Puppet Enterprise 2019.8.1 or newer versions. -* PEADM requires [Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) version 3.17.0 or newer and works on operating systems supported by Bolt. -**Note:** You can use PEADM to install Puppet Enterprise (PE) onto any operating system platform that is supported by the specific PE version you want to install. To review operating system platforms supported by specific PE versions, refer to the PE documentation. -* To successfully convert your current PE installation to a PEADM-managed installation, ensure that the PE setting for editing classifier configuration data is enabled. This setting is enabled by default on new PE installations, but it could be disabled if the relevant configuration was removed from your global hiera.yaml file. See the [PE docs](https://www.puppet.com/docs/pe/latest/config_console.html#enable_console_configuration_data) for more information. +- PEADM is compatible with Puppet Enterprise 2019.8.1 or newer versions. +- PEADM requires [Bolt](https://www.puppet.com/docs/bolt/latest/bolt_installing) version 3.17.0 or newer and works on operating systems supported by Bolt. + **Note:** You can use PEADM to install Puppet Enterprise (PE) onto any operating system platform that is supported by the specific PE version you want to install. To review operating system platforms supported by specific PE versions, refer to the PE documentation. +- To successfully convert your current PE installation to a PEADM-managed installation, ensure that the PE setting for editing classifier configuration data is enabled. This setting is enabled by default on new PE installations, but it could be disabled if the relevant configuration was removed from your global hiera.yaml file. See the [PE docs](https://www.puppet.com/docs/pe/latest/config_console.html#enable_console_configuration_data) for more information. ## Usage For instructions on using PEADM plans, see the following PEADM docs: -* [Install](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/install.md) -* [Upgrade](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/upgrade.md) -* [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md) -* [Status](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/status.md) +- [Install](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/install.md) +- [Upgrade](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/upgrade.md) +- [Convert](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/convert.md) +- [Status](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/status.md) ## Reference To understand which architecture is right for you, see the following information on the Puppet documentation site: -* [PE architectures](https://puppet.com/docs/pe/latest/choosing_an_architecture.html) -* [PE multi-region reference architectures](https://puppet.com/docs/patterns-and-tactics/latest/reference-architectures/pe-multi-region-reference-architectures.html) - +- [PE architectures](https://puppet.com/docs/pe/latest/choosing_an_architecture.html) +- [PE multi-region reference architectures](https://puppet.com/docs/patterns-and-tactics/latest/reference-architectures/pe-multi-region-reference-architectures.html) To learn more about the PEADM module and its uses, see the following PEADM docs: -* [Recovery procedures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/recovery.md) -* [Architectures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/architectures.md) -* [Expanding deployment](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/expanding.md) -* [Classification](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/classification.md) -* [Testing](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/pre_post_checks.md) -* [Docker based examples](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/docker_examples.md) -* [Release process](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/release_process.md) +- [Recovery procedures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/recovery.md) +- [Architectures](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/architectures.md) +- [Expanding deployment](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/expanding.md) +- [Classification](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/classification.md) +- [Testing](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/pre_post_checks.md) +- [Docker based examples](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/docker_examples.md) +- [Release process](https://github.com/puppetlabs/puppetlabs-peadm/blob/main/documentation/release_process.md) ## Getting help -* If you find a bug, you can [create a GitHub issue](https://github.com/puppetlabs/puppetlabs-peadm/issues). -* For PE customers using PEADM and experiencing outages or other issues, [contact the Support team](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet). +- If you find a bug, you can [create a GitHub issue](https://github.com/puppetlabs/puppetlabs-peadm/issues). +- For PE customers using PEADM and experiencing outages or other issues, [contact the Support team](https://portal.perforce.com/s/topic/0TO4X000000DbNgWAK/puppet). ## License diff --git a/REFERENCE.md b/REFERENCE.md index 065b6d75..bf4d04f9 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -10,6 +10,7 @@ * `peadm::setup::convert_node_manager`: Used during the peadm::convert plan * `peadm::setup::convert_pre20197`: Defines configuration needed for converting PE 2018 +* `peadm::setup::legacy_compiler_group` * `peadm::setup::node_manager`: Configures PEAdm's required node groups * `peadm::setup::node_manager_yaml`: Set up the node_manager.yaml file in the temporary Bolt confdir @@ -55,6 +56,7 @@ * [`backup_classification`](#backup_classification): A task to call the classification api and write to file * [`cert_data`](#cert_data): Return certificate data related to the Puppet agent * [`cert_valid_status`](#cert_valid_status): Check primary for valid state of a certificate +* [`classify_compilers`](#classify_compilers): Classify compilers as legacy or non-legacy * [`code_manager`](#code_manager): Perform various code manager actions * [`code_sync_status`](#code_sync_status): A task to confirm code is in sync accross the cluster for clusters with code manager configured * [`divert_code_manager`](#divert_code_manager): Divert the code manager live-dir setting @@ -107,6 +109,7 @@ Supported use cases: #### Private Plans +* `peadm::convert_compiler_to_legacy` * `peadm::misc::divert_code_manager`: This plan exists to account for a scenario where a PE XL * `peadm::modify_cert_extensions` * `peadm::subplans::component_install`: Install a new PEADM component @@ -116,6 +119,7 @@ Supported use cases: * `peadm::subplans::modify_certificate` * `peadm::subplans::prepare_agent` * `peadm::uninstall`: Single-entry-point plan for uninstalling Puppet Enterprise +* `peadm::update_compiler_extensions` * `peadm::util::code_sync_status` * `peadm::util::copy_file` * `peadm::util::db_disable_pglogical` @@ -134,7 +138,7 @@ Type: Puppet Language Assert that the architecture given is a supported one -#### `peadm::assert_supported_architecture(TargetSpec $primary_host, Variant[TargetSpec, Undef] $replica_host = undef, Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef)` +#### `peadm::assert_supported_architecture(TargetSpec $primary_host, Variant[TargetSpec, Undef] $replica_host = undef, Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef, Variant[TargetSpec, Undef] $legacy_compilers = undef)` The peadm::assert_supported_architecture function. @@ -170,6 +174,12 @@ Data type: `Variant[TargetSpec, Undef]` +##### `legacy_compilers` + +Data type: `Variant[TargetSpec, Undef]` + + + ### `peadm::assert_supported_bolt_version` Type: Puppet Language @@ -1054,6 +1064,20 @@ Data type: `String` The certifcate name to check validation of +### `classify_compilers` + +Classify compilers as legacy or non-legacy + +**Supports noop?** false + +#### Parameters + +##### `compiler_hosts` + +Data type: `Array[String]` + +List of FQDNs of compilers + ### `code_manager` Perform various code manager actions @@ -1791,6 +1815,7 @@ The following parameters are available in the `peadm::convert` plan: * [`primary_host`](#-peadm--convert--primary_host) * [`replica_host`](#-peadm--convert--replica_host) * [`compiler_hosts`](#-peadm--convert--compiler_hosts) +* [`legacy_compilers`](#-peadm--convert--legacy_compilers) * [`primary_postgresql_host`](#-peadm--convert--primary_postgresql_host) * [`replica_postgresql_host`](#-peadm--convert--replica_postgresql_host) * [`compiler_pool_address`](#-peadm--convert--compiler_pool_address) @@ -1819,6 +1844,14 @@ Data type: `Optional[TargetSpec]` +Default value: `undef` + +##### `legacy_compilers` + +Data type: `Optional[TargetSpec]` + + + Default value: `undef` ##### `primary_postgresql_host` @@ -1904,6 +1937,7 @@ The following parameters are available in the `peadm::install` plan: * [`primary_host`](#-peadm--install--primary_host) * [`replica_host`](#-peadm--install--replica_host) * [`compiler_hosts`](#-peadm--install--compiler_hosts) +* [`legacy_compilers`](#-peadm--install--legacy_compilers) * [`primary_postgresql_host`](#-peadm--install--primary_postgresql_host) * [`replica_postgresql_host`](#-peadm--install--replica_postgresql_host) * [`console_password`](#-peadm--install--console_password) @@ -2020,6 +2054,14 @@ Data type: `Optional[TargetSpec]` +Default value: `undef` + +##### `legacy_compilers` + +Data type: `Optional[TargetSpec]` + + + Default value: `undef` ##### `primary_postgresql_host` diff --git a/documentation/convert.md b/documentation/convert.md index 6d2c8095..c0ec001f 100644 --- a/documentation/convert.md +++ b/documentation/convert.md @@ -1,6 +1,6 @@ -# Convert infrastructure for use with the peadm module +# Convert infrastructure for use with the PEADM module -The peadm::convert plan can be used to adopt manually deployed infrastructure for use with peadm, or to adopt infrastructure deployed with an older version of peadm. +The peadm::convert plan can be used to adopt manually deployed infrastructure for use with PEADM or to adopt infrastructure deployed with an older version of peadm. ## Convert an Existing Deployment @@ -14,19 +14,36 @@ Prepare to run the plan against all servers in the PE infrastructure, using a pa "pe-xl-compiler-0.lab1.puppet.vm", "pe-xl-compiler-1.lab1.puppet.vm" ], - - "compiler_pool_address": "puppet.lab1.puppet.vm", + "legacy_compilers": [ + "pe-xl-legacy-compiler-0.lab1.puppet.vm", + "pe-xl-legacy-compiler-1.lab1.puppet.vm" + ], + "compiler_pool_address": "puppet.lab1.puppet.vm" } ``` -See the [install](install.md#reference-architectures) documentation for a list of supported architectures. Note that for convert, *all infrastructure being converted must already be functional*; you cannot use convert to add new systems to the infrastructure, nor can you use it to change your architecture. +See the [install](install.md#reference-architectures) documentation for a list of supported architectures. Note that for convert, _all infrastructure being converted must already be functional_; you cannot use convert to add new systems to the infrastructure, nor can you use it to change your architecture. ``` -bolt plan run peadm::convert --params @params.json +bolt plan run peadm::convert --params @params.json ``` ## Retry or resume plan This plan is broken down into steps. Normally, the plan runs through all the steps from start to finish. The name of each step is displayed during the plan run, as the step begins. -The `begin_at_step` parameter can be used to facilitate re-running this plan after a failed attempt, skipping past any steps that already completed successfully on the first try and picking up again at the step specified. The step name to resume at can be read from the previous run logs. A full list of available values for this parameter can be viewed by running `bolt plan show peadm::convert`. +The `begin_at_step` parameter can be used to facilitate re-running this plan after a failed attempt, skipping past any steps that were already completed successfully on the first try and picking up again at the step specified. The step name to resume can be read from the previous run logs. A full list of available values for this parameter can be viewed by running `bolt plan show peadm::convert`. + +## Convert compilers to legacy + +### Puppet Enterprise installed with puppetlabs-peadm version 3.21 or later + +To convert compilers to legacy compilers use the `peadm::convert_compiler_to_legacy` plan. This plan will create the needed Node group and Classifier rules to make compilers legacy. Also will add certificate extensions to those nodes. + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` + +### Puppet Enterprise installed with puppetlabs-peadm version prior to 3.21 + +Follow Steps 1 to 3 in the [Upgrade Puppet Enterprise with legacy compilers](upgrade_with_legacy_compilers.md) documentation. diff --git a/documentation/convert_compiler_to_legacy.md b/documentation/convert_compiler_to_legacy.md new file mode 100644 index 00000000..badb4832 --- /dev/null +++ b/documentation/convert_compiler_to_legacy.md @@ -0,0 +1,9 @@ +# Convert compilers to legacy + +### Puppet Enterprise installed with puppetlabs-peadm version 3.25 or later + +To convert compilers to legacy compilers use the `peadm::convert_compiler_to_legacy` plan. This plan will create the needed Node group and Classifier rules to make compilers legacy. Also will add certificate extensions to those nodes. + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` diff --git a/documentation/install.md b/documentation/install.md index 22e9d70c..72db9afc 100644 --- a/documentation/install.md +++ b/documentation/install.md @@ -1,69 +1,68 @@ -# Install Puppet Enterprise using the peadm module +# Install Puppet Enterprise using the PEADM module -The peadm module can be used to install Puppet Enterprise on new infrastructure. Supported architectures include Standard, Large, and Extra Large. +The PEADM module can be used to install Puppet Enterprise on new infrastructure. Supported architectures include Standard, Large, and Extra Large. -The peadm install plan creates a base install. Once a base cluster is installed, you may need to continue and perform additional configuration and adjustments to reach your target state, depending on your use case. +The PEADM install plan creates a base install. Once a base cluster is installed, you may need to continue and perform additional configuration and adjustments to reach your target state, depending on your use case. ## Reference Architectures -When installing a new PE cluster using peadm, there are several different host parameters which can be specified. At a minimum, you must always specify the primary parameter. Depending on which architecture you are deploying, other host parameters may be needed as well. The following is a list of the architectures peadm can install and the required parameters. - -* Standard - - primary -* Standard with DR - - primary - - primary-replica -* Large - - primary - - compilers -* Large with DR - - primary - - primary-replica - - compilers -* Extra Large - - primary - - pdb-database - - compilers (optional) -* Extra Large with DR - - primary - - primary-replica - - pdb-database - - pdb-database-replica - - compilers (optional) - -Supplying a combination of host parameters which does not match one of the supported architectures above will result in an unsupported architecture error. +When installing a new PE cluster using PEADM, several different host parameters can be specified. At a minimum, you must always specify the primary parameter. Depending on which architecture you are deploying, other host parameters may be needed as well. The following is a list of the architectures PEADM can install and the required parameters. + +- Standard + - primary +- Standard with DR + - primary + - primary-replica +- Large + - primary + - compilers +- Large with DR + - primary + - primary-replica + - compilers +- Extra Large + - primary + - pdb-database + - compilers (optional) +- Extra Large with DR + - primary + - primary-replica + - pdb-database + - pdb-database-replica + - compilers (optional) + +Supplying a combination of host parameters that do not match one of the supported architectures above will result in an unsupported architecture error. ## Requirements -* Puppet must not be installed on any of the target PE cluster hosts prior to beginning install. +Puppet must not be installed on any of the target PE cluster hosts before beginning installation. ## Usage ### Bolt 3 usage + We will name the bolt project `large_ha_peadm` in this example but the project name can be anything. 1. Install Bolt on a jumphost. This can be the primary, or any other system. (via package) 2. Run `mkdir large_ha_peadm && cd large_ha_peadm && bolt project init large_ha_peadm --modules puppetlabs-peadm` -4. Create an inventory file with connection information. Example included below. -5. Create a parameters file. Example included below. -6. Run `bolt plan run peadm::install --params @params.json ` after the inventory and params files are created. - +3. Create an inventory file with connection information. Example included below. +4. Create a parameters file. Example included below. +5. Run `bolt plan run peadm::install --params @params.json ` after the inventory and params files are created. ### Bolt 2 usage -1. Install Bolt on a jumphost. This can be the primary, or any other system. -2. Download or git clone the peadm module and put it somewhere on the jumphost. e.g. ~/modules/peadm. -3. Download or git clone the module dependencies, and put them somewhere on the jumphost. e.g. ~/modules/stdlib, ~/modules/node\_manager, etc. -4. Create an inventory file with connection information. Example included below. -5. Create a parameters file. Example included below. -6. Run the peadm::install plan with the inputs created. Example: +1. Install Bolt on a jumphost. This can be the primary or any other system. +2. Download or git clone the PEADM module and put it somewhere on the jumphost. e.g. ~/modules/peadm. +3. Download or git clone the module dependencies, and put them somewhere on the jumphost. e.g. ~/modules/stdlib, ~/modules/node_manager, etc. +4. Create an inventory file with connection information. Example included below. +5. Create a parameters file. Example included below. +6. Run the peadm::install plan with the inputs created. Example: bolt plan run peadm::install \ --inventory inventory.yaml \ --modulepath ~/modules \ --params @params.json - Example inventory.yaml Bolt inventory file: ```yaml @@ -85,9 +84,13 @@ groups: uri: 10.234.6.45 - name: pe-xl-compiler-1.lab1.puppet.vm uri: 10.234.14.131 + - name: pe-xl-legacy-compiler-0.lab1.puppet.vm + uri: 10.234.6.46 + - name: pe-xl-legacy-compiler-1.lab1.puppet.vm + uri: 10.234.6.47 ``` -Example params.json Bolt parameters file (shown: Extra Large with DR): +example params.json bolt parameters file (shown: extra large with dr): ```json { @@ -99,9 +102,12 @@ Example params.json Bolt parameters file (shown: Extra Large with DR): "pe-xl-compiler-0.lab1.puppet.vm", "pe-xl-compiler-1.lab1.puppet.vm" ], - + "legacy_compilers": [ + "pe-xl-legacy-compiler-0.lab1.puppet.vm", + "pe-xl-legacy-compiler-1.lab1.puppet.vm" + ], "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "compiler_pool_address": "puppet.lab1.puppet.vm", "version": "2021.7.0" } @@ -113,12 +119,12 @@ Example params.json Bolt parameters file (shown: Standard): { "primary_host": "pe-xl-core-0.lab1.puppet.vm", "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "version": "2021.7.9" } ``` -Review the [peadm::install plan](../plans/install.pp) to learn about more advanced installation options. For example, it is possible to: supply an ssh private key and git clone URL for a control-repo as part of installation; supply the LDAP configuration data for PE; specify where the installer tarball is uploaded; and similar complete automation tie-ins. +Review the [peadm::install plan](../plans/install.pp) to learn about more advanced installation options. For example, it is possible to: supply an SSH private key and git clone URL for a control-repo as part of installation; supply the LDAP configuration data for PE; specify where the installer tarball is uploaded; and similar complete automation tie-ins. ## Offline usage @@ -126,28 +132,27 @@ The peadm::install plan downloads installation content from an online repository The default staging directory is `/tmp`. If a different staging dir is being used, it can be specified using the `stagingdir` parameter to the peadm::install plan. -The content needed is the PE installation tarball for the target version. The installation content should be in the staging dir, and should have its original name. E.g. `/tmp/puppet-enterprise-2021.4.0-el-7-x86_64.tar.gz`. +The content needed is the PE installation tarball for the target version. The installation content should be in the staging dir and should have its original name. E.g. `/tmp/puppet-enterprise-2021.4.0-el-7-x86_64.tar.gz`. Installation content can be downloaded from [https://puppet.com/try-puppet/puppet-enterprise/download/](https://puppet.com/try-puppet/puppet-enterprise/download/). -If you wish to prevent the bolt host from transferring the installer tarball to the targets you can place the installer tar files on the target hosts in the _upload directory_. This can save time over slow networks, but is best to just perform the bolt task on the target node (puppetserver). The default _upload directory_ is `/tmp`. If a different upload dir is being used, it can be specified using the `uploaddir` parameter to the peadm::install plan. With default parameters the tar file will need to exist in the directories for offline configuration: +If you wish to prevent the bolt host from transferring the installer tarball to the targets you can place the installer tar files on the target hosts in the _upload directory_. This can save time over slow networks, but is best to just perform the bolt task on the target node (puppetserver). The default _upload directory_ is `/tmp`. If a different upload dir is being used, it can be specified using the `uploaddir` parameter to the peadm::install plan. With default parameters, the tar file will need to exist in the directories for offline configuration: -* /tmp on the Bolt host -* /tmp on the primary -* /tmp on the primary PuppetDB PostgreSQL (if used) -* /tmp on the replica PuppetDB PostgreSQL (if used) +- /tmp on the Bolt host +- /tmp on the primary +- /tmp on the primary PuppetDB PostgreSQL (if used) +- /tmp on the replica PuppetDB PostgreSQL (if used) ## Online usage -The peadm::install plan can be configured to download installation content directly to hosts. To configure online installation, set the `download_mode` parameter of the `peadm::install` plan to `direct`. The direct mode is often more efficient when PE hosts have a route to the internet. +The peadm::install plan can be configured to download installation content directly to hosts. To configure the online installation, set the `download_mode` parameter of the `peadm::install` plan to `direct`. The direct mode is often more efficient when PE hosts have a route to the internet. ## Hostnames and Certificate Names The various host parameters given to the peadm::install or peadm::action::install plans will be set as Puppet certificate names. You must use the names here that you want the servers to be identified as by Puppet. While it is not required that target names match hostnames, it _is_ required that target names be resolvable. - -In the event that Bolt will reach servers by IP address or external DNS name rather than internal DNS name or desired certname, a Bolt inventory file should be used to specify URIs for each name. For example: +If Bolt will reach servers by IP address or external DNS name rather than internal DNS name or desired certname, a Bolt inventory file should be used to specify URIs for each name. For example: ```yaml --- @@ -158,7 +163,7 @@ targets: uri: 10.234.14.131 ``` -A parameters JSON file can then reference the target names, which will become the Puppet certificate names, and Bolt will still be able to reach the systems by using the IP addresses or other DNS name specified as the URIs in the inventory.yaml file. +A parameters JSON file can then reference the target names, which will become the Puppet certificate names, and Bolt will still be able to reach the systems by using the IP addresses or other DNS names specified as the URIs in the inventory.yaml file. ```json { @@ -166,7 +171,7 @@ A parameters JSON file can then reference the target names, which will become th "replica_host": "pe-xl-core-1.lab1.puppet.vm", "console_password": "puppetlabs", - "dns_alt_names": [ "puppet", "puppet.lab1.puppet.vm" ], + "dns_alt_names": ["puppet", "puppet.lab1.puppet.vm"], "compiler_pool_address": "puppet.lab1.puppet.vm", "version": "2021.7.0" } @@ -174,12 +179,11 @@ A parameters JSON file can then reference the target names, which will become th ## Implementation Reference -Provisioning can be broken down into two actions: [install](../plans/action/install.pp), and [configure](../plans/subplans/configure.pp). Installation currently requires ssh access to the un-installed nodes, but configure can be performed using the Orchestrator transport if installation has already been completed. - -Besides getting Puppet Enterprise installed, the key configuration supporting Large and Extra Large architectures is laid out in four classification groups. Links are provided below to a Markdown document that describes the groups, and also to the Puppet manifest that actually configures them: +Provisioning can be broken down into two actions: [install](../plans/action/install.pp), and [configure](../plans/subplans/configure.pp). The installation currently requires SSH access to the uninstalled nodes, but configuration can be performed using the Orchestrator transport if the installation has already been completed. -* [classification.md](classification.md) -* [peadm::setup::node\_manager class](../manifests/setup/node_manager.pp) +Besides getting Puppet Enterprise installed, the key configuration supporting Large and Extra-large architectures is laid out in four classification groups. Links are provided below to a Markdown document that describes the groups, and also to the Puppet manifest that configures them: -The reference implementation uses trusted facts to put nodes in the right groups. Because the important puppet\_enterprise::\* class parameters and data are specified in the console, it should also be safe to have a pe.conf present on both the primary, and the primary replica nodes. +- [classification.md](classification.md) +- [peadm::setup::node_manager class](../manifests/setup/node_manager.pp) +The reference implementation uses trusted facts to put nodes in the right groups. Because the important puppet_enterprise::\* class parameters and data are specified in the console, it should also be safe to have a pe.conf present on both the primary and the primary replica nodes. diff --git a/documentation/upgrade_with_legacy_compilers.md b/documentation/upgrade_with_legacy_compilers.md new file mode 100644 index 00000000..398a7bec --- /dev/null +++ b/documentation/upgrade_with_legacy_compilers.md @@ -0,0 +1,27 @@ +# Upgrade Puppet Enterprise with legacy compilers + +## What is a legacy compiler and a current compiler + +As a legacy compiler we refer to a compiler that doesn't have PuppetDB. And a current Compiler is a compiler that has PuppetDB. By default, latest versions of Puppet enterprise comes with compilers that have PuppetDB.If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers. + +## Who is this documentation for + +For those users that have installed Puppet Enterprise with puppetlabs-peadm prior version 3.25 and manually converted their existing complilers (all of the or at least 1) to legacy compilers. + +## Who is this documentation not for + +For those users that have installed Puppet Enterprise with PEADM with 3.25 version or later, there is no need to follow this documentation. The install process will automatically have created the necessary configurations for you and you can use the `peadm::convert_compiler_to_legacy` plan if you need a legacy compiler. example: + +```shell +bolt plan run peadm::convert_compiler_to_legacy legacy_hosts=compiler1.example.com,compiler2.example.com primary_host=primary.example.com +``` + +## How to upgrade Puppet Enterprise with legacy compilers + +### 1. Revert changes to the legacy compilers nodes + +Usually users pin the nodes in the Pe Master Node Group and then manually removing PuppetDB from compilers nodes. To revert this changes go to your Puppet Enterprise console and unpin the compilers nodes from the Group. + +### 2. Upgrade Puppet Enterprise + +You can proceed with the upgrade of Puppet Enterprise as usual using the puppetlabs-peadm module 3.25 or later and pass legacy compilers to the upgrade plan. diff --git a/functions/assert_supported_architecture.pp b/functions/assert_supported_architecture.pp index 0c705ddd..25709e86 100644 --- a/functions/assert_supported_architecture.pp +++ b/functions/assert_supported_architecture.pp @@ -5,7 +5,31 @@ function peadm::assert_supported_architecture ( Variant[TargetSpec, Undef] $primary_postgresql_host = undef, Variant[TargetSpec, Undef] $replica_postgresql_host = undef, Variant[TargetSpec, Undef] $compiler_hosts = undef, + Variant[TargetSpec, Undef] $legacy_compilers = undef, ) >> Hash { + # Normalize $legacy_compilers to an array + $legacy_compilers_array = $legacy_compilers ? { + undef => [], + String => [$legacy_compilers], + Array => $legacy_compilers, + default => fail("Unexpected type for \$legacy_compilers: ${legacy_compilers}"), + } + + # Normalize $compiler_hosts to an array + $compiler_hosts_array = $compiler_hosts ? { + undef => [], + String => [$compiler_hosts], + Array => $compiler_hosts, + default => fail("Unexpected type for \$compiler_hosts: ${compiler_hosts}"), + } + $all_compilers = $legacy_compilers_array + $compiler_hosts_array + + # Set $has_compilers to undef if $all_compilers is empty, otherwise set it to true + $has_compilers = empty($all_compilers) ? { + true => undef, + default => true, + } + $result = case [ !!($primary_host), !!($replica_host), @@ -13,13 +37,13 @@ function peadm::assert_supported_architecture ( !!($replica_postgresql_host), ] { [true, false, false, false]: { # Standard or Large, no DR - ({ 'disaster-recovery' => false, 'architecture' => $compiler_hosts ? { + ({ 'disaster-recovery' => false, 'architecture' => $has_compilers ? { undef => 'standard', default => 'large', } }) } [true, true, false, false]: { # Standard or Large, DR - ({ 'disaster-recovery' => true, 'architecture' => $compiler_hosts ? { + ({ 'disaster-recovery' => true, 'architecture' => $has_compilers ? { undef => 'standard', default => 'large', } }) @@ -44,7 +68,7 @@ function peadm::assert_supported_architecture ( <% if $replica_postgresql_host { -%> - pdb-database-replica <% } -%> - <% if $compiler_hosts { -%> + <% if $has_compilers { -%> - compilers <% } -%> diff --git a/functions/oid.pp b/functions/oid.pp index 2fc735d0..0f03a43c 100644 --- a/functions/oid.pp +++ b/functions/oid.pp @@ -4,6 +4,7 @@ function peadm::oid ( case $short_name { 'peadm_role': { '1.3.6.1.4.1.34380.1.1.9812' } 'peadm_availability_group': { '1.3.6.1.4.1.34380.1.1.9813' } + 'peadm_legacy_compiler': { '1.3.6.1.4.1.34380.1.1.9814' } 'pp_application': { '1.3.6.1.4.1.34380.1.1.8' } 'pp_cluster': { '1.3.6.1.4.1.34380.1.1.16' } 'pp_role': { '1.3.6.1.4.1.34380.1.1.13' } diff --git a/manifests/setup/legacy_compiler_group.pp b/manifests/setup/legacy_compiler_group.pp new file mode 100644 index 00000000..e3601200 --- /dev/null +++ b/manifests/setup/legacy_compiler_group.pp @@ -0,0 +1,72 @@ +# @api private +class peadm::setup::legacy_compiler_group ( + String[1] $primary_host, + Optional[String] $internal_compiler_a_pool_address = undef, + Optional[String] $internal_compiler_b_pool_address = undef, +) { + Node_group { + purge_behavior => none, + } + + node_group { 'PE Legacy Compiler': + parent => 'PE Master', + rule => ['and', + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + } + + node_group { 'PE Legacy Compiler Group A': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_b_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + node_group { 'PE Legacy Compiler Group B': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + node_group { 'PE Compiler': + rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], + } +} diff --git a/manifests/setup/node_manager.pp b/manifests/setup/node_manager.pp index 65c69044..f74cb217 100644 --- a/manifests/setup/node_manager.pp +++ b/manifests/setup/node_manager.pp @@ -79,6 +79,12 @@ variables => { 'pe_master' => true }, } + # PE Compiler group comes from default PE and already has the pe compiler role + node_group { 'PE Compiler': + parent => 'PE Master', + rule => ['and', ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false']], + } + # This group should pin the primary, and also map to any pe-postgresql nodes # which are part of the architecture. node_group { 'PE Database': @@ -115,6 +121,7 @@ rule => ['and', ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false'], ], classes => { 'puppet_enterprise::profile::puppetdb' => { @@ -173,6 +180,7 @@ rule => ['and', ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'false'], ], classes => { 'puppet_enterprise::profile::puppetdb' => { @@ -192,4 +200,66 @@ }, }, } + + node_group { 'PE Legacy Compiler': + parent => 'PE Master', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_b_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + } + + # Configure the A pool for legacy compilers. There are up to two pools for DR, each + # having an affinity for one "availability zone" or the other. + node_group { 'PE Legacy Compiler Group A': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'A'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_b_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } + + # Configure the B pool for legacy compilers. There are up to two pools for DR, each + # having an affinity for one "availability zone" or the other. + node_group { 'PE Legacy Compiler Group B': + ensure => 'present', + parent => 'PE Legacy Compiler', + rule => ['and', + ['=', ['trusted', 'extensions', 'pp_auth_role'], 'pe_compiler'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_availability_group')], 'B'], + ['=', ['trusted', 'extensions', peadm::oid('peadm_legacy_compiler')], 'true'], + ], + classes => { + 'puppet_enterprise::profile::master' => { + 'puppetdb_host' => [$internal_compiler_a_pool_address, $internal_compiler_a_pool_address].filter |$_| { $_ }, + 'puppetdb_port' => [8081], + }, + }, + data => { + # Workaround for GH-118 + 'puppet_enterprise::profile::master::puppetdb' => { + 'ha_enabled_replicas' => [], + }, + }, + } } diff --git a/plans/convert.pp b/plans/convert.pp index 1995a0b0..e697e87d 100644 --- a/plans/convert.pp +++ b/plans/convert.pp @@ -10,6 +10,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -36,6 +37,7 @@ $replica_target = peadm::get_targets($replica_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) $all_targets = peadm::flatten_compact([ @@ -43,6 +45,7 @@ $replica_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, $primary_postgresql_target, ]) @@ -53,6 +56,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) out::message('# Gathering information') @@ -115,10 +119,36 @@ $index % 2 != 0 } } + $legacy_compiler_a_targets = $legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'A' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'A' + } + else { + $index % 2 == 0 + } + } + $legacy_compiler_b_targets = $legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'B' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'B' + } + else { + $index % 2 != 0 + } + } } else { $compiler_a_targets = $compiler_targets $compiler_b_targets = [] + $legacy_compiler_a_targets = $legacy_compiler_targets + $legacy_compiler_b_targets = [] } # Modify csr_attributes.yaml and insert the peadm-specific OIDs to identify @@ -185,6 +215,7 @@ add_extensions => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'false', }, ) }, @@ -194,6 +225,27 @@ add_extensions => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'false', + }, + ) + }, + background('modify-compilers-a-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_a_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + background('modify-compilers-b-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_b_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', }, ) }, @@ -252,6 +304,20 @@ # completion run_command('systemctl restart pe-puppetserver.service pe-puppetdb.service', $all_targets - $primary_target - $primary_postgresql_target - $replica_postgresql_target) + + # Run puppet on all targets again to ensure everything is fully up-to-date + run_task('peadm::puppet_runonce', $all_targets) + } + + if $legacy_compilers { +# lint:ignore:strict_indent + $warning_msg = run_task('peadm::check_legacy_compilers', $primary_host, legacy_compilers => $legacy_compilers.join(',') ).first.message + if $warning_msg.length > 0 { + out::message(@("WARN"/L)) + WARNING: ${warning_msg} + | WARN + } +# lint:endignore } return("Conversion to peadm Puppet Enterprise ${arch['architecture']} completed.") diff --git a/plans/convert_compiler_to_legacy.pp b/plans/convert_compiler_to_legacy.pp new file mode 100644 index 00000000..c75924bd --- /dev/null +++ b/plans/convert_compiler_to_legacy.pp @@ -0,0 +1,154 @@ +# @api private +plan peadm::convert_compiler_to_legacy ( + Peadm::SingleTargetSpec $primary_host, + TargetSpec $legacy_hosts, + Optional[Boolean] $remove_pdb = true, +) { + $primary_target = peadm::get_targets($primary_host, 1) + $convert_legacy_compiler_targets = peadm::get_targets($legacy_hosts) + + $cluster = run_task('peadm::get_peadm_config', $primary_host).first.value + $error = getvar('cluster.error') + if $error { + fail_plan($error) + } + + apply($primary_target) { + class { 'peadm::setup::node_manager_yaml': + primary_host => $primary_target.peadm::certname() ? { + undef => $primary_target, + default => $primary_target.peadm::certname(), + }, + } + + class { 'peadm::setup::legacy_compiler_group': + primary_host => $primary_target.peadm::certname() ? { + undef => $primary_target, + default => $primary_target.peadm::certname(), + }, + internal_compiler_a_pool_address => getvar('cluster.params.internal_compiler_a_pool_address'), + internal_compiler_b_pool_address => getvar('cluster.params.internal_compiler_b_pool_address'), + require => Class['peadm::setup::node_manager_yaml'], + } + } + + $replica_host = getvar('cluster.params.replica_host') + $primary_postgresql_host = getvar('cluster.params.primary_postgresql_host') + $replica_postgresql_host = getvar('cluster.params.replica_postgresql_host') + $compiler_hosts = getvar('cluster.params.compiler_hosts') + $legacy_compilers = getvar('cluster.params.legacy_hosts') + + $replica_target = peadm::get_targets($replica_host, 1) + $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) + $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) + $compiler_targets = peadm::get_targets($compiler_hosts) - $convert_legacy_compiler_targets + $legacy_targets = peadm::get_targets($legacy_compilers) + $convert_legacy_compiler_targets + + $all_targets = peadm::flatten_compact([ + $primary_target, + $replica_target, + $primary_postgresql_target, + $replica_postgresql_target, + $compiler_targets, + $legacy_targets, + ]) + + # Ensure input valid for a supported architecture + $arch = peadm::assert_supported_architecture( + $primary_host, + $replica_host, + $primary_postgresql_host, + $replica_postgresql_host, + $compiler_hosts, + $legacy_compilers, + ) + + if $arch['disaster-recovery'] { + # Gather certificate extension information from all systems + $cert_extensions = run_task('peadm::cert_data', $legacy_targets).reduce({}) |$memo,$result| { + $memo + { $result.target.peadm::certname => $result['extensions'] } + } + $legacy_compiler_a_targets = $convert_legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'A' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'A' + } + else { + $index % 2 == 0 + } + } + $legacy_compiler_b_targets = $convert_legacy_compiler_targets.filter |$index,$target| { + $exts = $cert_extensions[$target.peadm::certname()] + if ($exts[peadm::oid('peadm_availability_group')] in ['A', 'B']) { + $exts[peadm::oid('peadm_availability_group')] == 'B' + } + elsif ($exts[peadm::oid('pp_cluster')] in ['A', 'B']) { + $exts[peadm::oid('pp_cluster')] == 'B' + } + else { + $index % 2 != 0 + } + } + } else { + $legacy_compiler_a_targets = $convert_legacy_compiler_targets + $legacy_compiler_b_targets = [] + } + + wait([ + background('modify-compilers-certs') || { + run_plan('peadm::modify_certificate', $compiler_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('peadm_legacy_compiler') => 'false', + }, + ) + }, + background('modify-compilers-a-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_a_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + background('modify-compilers-b-certs') || { + run_plan('peadm::modify_certificate', $legacy_compiler_b_targets, + primary_host => $primary_target, + add_extensions => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', + }, + ) + }, + ]) + + if $remove_pdb { + run_command('puppet resource service puppet ensure=stopped', $convert_legacy_compiler_targets) + run_command('puppet resource service pe-puppetdb ensure=stopped enable=false', $convert_legacy_compiler_targets) + } + + run_task('peadm::puppet_runonce', $convert_legacy_compiler_targets) + run_task('peadm::puppet_runonce', $compiler_targets) + run_task('peadm::puppet_runonce', $primary_target) + run_task('peadm::puppet_runonce', $all_targets) + + if $remove_pdb { + run_command('puppet resource package pe-puppetdb ensure=purged', $convert_legacy_compiler_targets) + run_command('puppet resource user pe-puppetdb ensure=absent', $convert_legacy_compiler_targets) + + run_command('rm -rf /etc/puppetlabs/puppetdb', $convert_legacy_compiler_targets) + run_command('rm -rf /var/log/puppetlabs/puppetdb', $convert_legacy_compiler_targets) + run_command('rm -rf /opt/puppetlabs/server/data/puppetdb', $convert_legacy_compiler_targets) + } + + run_command('systemctl start pe-puppetserver.service', $convert_legacy_compiler_targets) + run_command('puppet resource service puppet ensure=running', $convert_legacy_compiler_targets) + + return("Converted host ${convert_legacy_compiler_targets} to legacy compiler.") +} diff --git a/plans/install.pp b/plans/install.pp index fb1052c5..58b512ca 100644 --- a/plans/install.pp +++ b/plans/install.pp @@ -38,6 +38,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -85,6 +86,7 @@ # Large compiler_hosts => $compiler_hosts, + legacy_compilers => $legacy_compilers, # Extra Large primary_postgresql_host => $primary_postgresql_host, @@ -123,6 +125,7 @@ # Large compiler_hosts => $compiler_hosts, + legacy_compilers => $legacy_compilers, # Extra Large primary_postgresql_host => $primary_postgresql_host, diff --git a/plans/subplans/component_install.pp b/plans/subplans/component_install.pp index df74079e..daf11688 100644 --- a/plans/subplans/component_install.pp +++ b/plans/subplans/component_install.pp @@ -22,6 +22,10 @@ peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => $avail_group_letter, } + } elsif $role == 'pe_compiler_legacy' { + $certificate_extensions = { + peadm::oid('peadm_role') => $role, + } } else { $certificate_extensions = { peadm::oid('peadm_role') => $role, diff --git a/plans/subplans/configure.pp b/plans/subplans/configure.pp index bccc0503..2afa0084 100644 --- a/plans/subplans/configure.pp +++ b/plans/subplans/configure.pp @@ -27,6 +27,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -51,6 +52,7 @@ $replica_target = peadm::get_targets($replica_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) # Ensure input valid for a supported architecture @@ -60,6 +62,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) # Source list of files on Primary and synchronize to new Replica @@ -74,6 +77,7 @@ run_plan('peadm::util::copy_file', peadm::flatten_compact([ $replica_target, $compiler_targets, + $legacy_compiler_targets, ]), source_host => $primary_target, path => $common_content_source @@ -139,6 +143,7 @@ $primary_target, $primary_postgresql_target, $compiler_targets, + $legacy_compiler_targets, $replica_target, $replica_postgresql_target, ])) @@ -161,6 +166,7 @@ $primary_postgresql_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, ])) return("Configuration of Puppet Enterprise ${arch['architecture']} succeeded.") diff --git a/plans/subplans/install.pp b/plans/subplans/install.pp index 7d48cb8a..d349e23e 100644 --- a/plans/subplans/install.pp +++ b/plans/subplans/install.pp @@ -41,6 +41,7 @@ # Large Optional[TargetSpec] $compiler_hosts = undef, + Optional[TargetSpec] $legacy_compilers = undef, # Extra Large Optional[Peadm::SingleTargetSpec] $primary_postgresql_host = undef, @@ -79,6 +80,7 @@ $primary_postgresql_target = peadm::get_targets($primary_postgresql_host, 1) $replica_postgresql_target = peadm::get_targets($replica_postgresql_host, 1) $compiler_targets = peadm::get_targets($compiler_hosts) + $legacy_compiler_targets = peadm::get_targets($legacy_compilers) # Ensure input valid for a supported architecture $arch = peadm::assert_supported_architecture( @@ -87,6 +89,7 @@ $primary_postgresql_host, $replica_postgresql_host, $compiler_hosts, + $legacy_compilers, ) $all_targets = peadm::flatten_compact([ @@ -95,6 +98,7 @@ $replica_target, $replica_postgresql_target, $compiler_targets, + $legacy_compiler_targets, ]) $primary_targets = peadm::flatten_compact([ @@ -115,6 +119,7 @@ $agent_installer_targets = peadm::flatten_compact([ $compiler_targets, + $legacy_compiler_targets, $replica_target, ]) @@ -122,10 +127,14 @@ if $arch['disaster-recovery'] { $compiler_a_targets = $compiler_targets.filter |$index,$target| { $index % 2 == 0 } $compiler_b_targets = $compiler_targets.filter |$index,$target| { $index % 2 != 0 } + $legacy_a_targets = $legacy_compiler_targets.filter |$index,$target| { $index % 2 == 0 } + $legacy_b_targets = $legacy_compiler_targets.filter |$index,$target| { $index % 2 != 0 } } else { $compiler_a_targets = $compiler_targets $compiler_b_targets = [] + $legacy_a_targets = $legacy_compiler_targets + $legacy_b_targets = [] } $dns_alt_names_csv = $dns_alt_names.reduce |$csv,$x| { "${csv},${x}" } @@ -143,7 +152,7 @@ true } elsif $replica_host { true - } elsif $compiler_hosts { + } elsif $compiler_hosts or $legacy_compilers { true } else { $code_manager_auto_configure @@ -179,7 +188,7 @@ # puppet and are present in PuppetDB, it is not necessary anymore. $puppetdb_database_temp_config = { 'puppet_enterprise::profile::database::puppetdb_hosts' => ( - $compiler_targets + $primary_target + $replica_target + $compiler_targets + $legacy_compiler_targets + $primary_target + $replica_target ).map |$t| { $t.peadm::certname() }, } @@ -278,6 +287,7 @@ extension_requests => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'false', } ) }, @@ -286,6 +296,25 @@ extension_requests => { peadm::oid('pp_auth_role') => 'pe_compiler', peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'false', + } + ) + }, + background('compiler-a-csr.yaml') || { + run_plan('peadm::util::insert_csr_extension_requests', $legacy_a_targets, + extension_requests => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'A', + peadm::oid('peadm_legacy_compiler') => 'true', + } + ) + }, + background('compiler-b-csr.yaml') || { + run_plan('peadm::util::insert_csr_extension_requests', $legacy_b_targets, + extension_requests => { + peadm::oid('pp_auth_role') => 'pe_compiler', + peadm::oid('peadm_availability_group') => 'B', + peadm::oid('peadm_legacy_compiler') => 'true', } ) }, diff --git a/plans/update_compiler_extensions.pp b/plans/update_compiler_extensions.pp new file mode 100644 index 00000000..784f919e --- /dev/null +++ b/plans/update_compiler_extensions.pp @@ -0,0 +1,25 @@ +# @api private +plan peadm::update_compiler_extensions ( + TargetSpec $compiler_hosts, + Peadm::SingleTargetSpec $primary_host, + Boolean $legacy = false, +) { + $primary_target = peadm::get_targets($primary_host, 1) + $host_targets = peadm::get_targets($compiler_hosts) + + run_plan('peadm::modify_certificate', $host_targets, + primary_host => $primary_target, + add_extensions => { peadm::oid('peadm_legacy_compiler') => String($legacy) }, + ) + + run_task('peadm::puppet_runonce', $primary_target) + run_task('peadm::puppet_runonce', $host_targets) + + if $legacy { + run_command('systemctl restart pe-puppetserver.service', $host_targets) + } else { + run_command('systemctl restart pe-puppetserver.service pe-puppetdb.service', $host_targets) + } + + return("Added legacy cert with value ${legacy} to compiler hosts ${compiler_hosts}") +} diff --git a/plans/upgrade.pp b/plans/upgrade.pp index b9adcd38..63375829 100644 --- a/plans/upgrade.pp +++ b/plans/upgrade.pp @@ -167,11 +167,25 @@ == $cert_extensions.dig($primary_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) } + $compiler_m1_nonlegacy_targets = $compiler_targets.filter |$target| { + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) + == $cert_extensions.dig($primary_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) and + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_legacy_compiler')) + == 'false') + } + $compiler_m2_targets = $compiler_targets.filter |$target| { ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) == $cert_extensions.dig($replica_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) } + $compiler_m2_nonlegacy_targets = $compiler_targets.filter |$target| { + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_availability_group')) + == $cert_extensions.dig($replica_target[0].peadm::certname, peadm::oid('peadm_availability_group'))) and + ($cert_extensions.dig($target.peadm::certname, peadm::oid('peadm_legacy_compiler')) + == 'false') + } + peadm::plan_step('preparation') || { if $download_mode == 'bolthost' { # Download the PE tarball on the nodes that need it @@ -239,7 +253,7 @@ peadm::plan_step('upgrade-primary') || { # Shut down PuppetDB on CMs that use the PM's PDB PG. Use run_command instead # of run_task(service, ...) so that upgrading from 2018.1 works over PCP. - run_command('systemctl stop pe-puppetdb', $compiler_m1_targets) + run_command('systemctl stop pe-puppetdb', $compiler_m1_nonlegacy_targets) run_task('peadm::pe_install', $primary_postgresql_target, tarball => $upload_tarball_path, @@ -344,7 +358,7 @@ # Shut down PuppetDB on CMs that use the replica's PDB PG. Use run_command # instead of run_task(service, ...) so that upgrading from 2018.1 works # over PCP. - run_command('systemctl stop pe-puppetdb', $compiler_m2_targets) + run_command('systemctl stop pe-puppetdb', $compiler_m2_nonlegacy_targets) run_task('peadm::pe_install', $replica_postgresql_target, tarball => $upload_tarball_path, diff --git a/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp b/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp index aecd8197..4529dc9b 100644 --- a/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp +++ b/spec/acceptance/peadm_spec/plans/provision_test_cluster.pp @@ -20,6 +20,9 @@ 'large': { ['primary', 'compiler'] } + 'large-with-two-compilers': { + ['primary', 'compiler', 'compiler'] + } 'large-with-dr': { ['primary', 'compiler', 'replica', 'compiler'] } diff --git a/spec/functions/assert_supported_architecture_spec.rb b/spec/functions/assert_supported_architecture_spec.rb index d48342f3..5f67a33b 100644 --- a/spec/functions/assert_supported_architecture_spec.rb +++ b/spec/functions/assert_supported_architecture_spec.rb @@ -24,6 +24,15 @@ let(:compiler_hosts) do 'pup-c1.puppet.vm' end + let(:legacy_compilers) do + 'pup-c2.puppet.vm' + end + let(:compiler_hosts_array) do + ['pup-c1.puppet.vm'] + end + let(:legacy_compilers_array) do + ['pup-c2.puppet.vm'] + end it { is_expected.to run.with_params(primary_host) @@ -59,6 +68,28 @@ 'disaster-recovery' => false, 'architecture' => 'large') end + it do + is_expected.to run.with_params(primary_host, + nil, + nil, + nil, + compiler_hosts, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'large') + end + it do + is_expected.to run.with_params(primary_host, + nil, + nil, + nil, + nil, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'large') + end it do is_expected.to run.with_params(primary_host, @@ -81,4 +112,26 @@ 'disaster-recovery' => false, 'architecture' => 'extra-large') end + it do + is_expected.to run.with_params(primary_host, + nil, + primary_postgresql_host, + nil, + compiler_hosts, + legacy_compilers) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'extra-large') + end + it do + is_expected.to run.with_params(primary_host, + nil, + primary_postgresql_host, + nil, + compiler_hosts_array, + legacy_compilers_array) + .and_return('supported' => true, + 'disaster-recovery' => false, + 'architecture' => 'extra-large') + end end diff --git a/spec/plans/subplans/install_spec.rb b/spec/plans/subplans/install_spec.rb index f4ed169e..c83a47a7 100644 --- a/spec/plans/subplans/install_spec.rb +++ b/spec/plans/subplans/install_spec.rb @@ -12,7 +12,15 @@ allow_task('peadm::precheck').return_for_targets( 'primary' => { 'hostname' => 'primary', - 'platform' => 'el-7.11-x86_64' + 'platform' => 'el-7.11-x86_64', + }, + 'compiler1' => { + 'hostname' => 'compiler1', + 'platform' => 'el-7.11-x86_64', + }, + 'compiler2' => { + 'hostname' => 'compiler2', + 'platform' => 'el-7.11-x86_64', }, ) @@ -39,9 +47,9 @@ it 'minimum variables to run' do params = { - 'primary_host' => 'primary', + 'primary_host' => 'primary', 'console_password' => 'puppetLabs123!', - 'version' => '2019.8.12', + 'version' => '2019.8.12', } expect(run_plan('peadm::subplans::install', params)).to be_ok @@ -49,10 +57,10 @@ it 'installs 2023.4 without r10k_known_hosts' do params = { - 'primary_host' => 'primary', - 'console_password' => 'puppetLabs123!', - 'version' => '2023.4.0', - 'r10k_remote' => 'git@github.com:puppetlabs/nothing', + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2023.4.0', + 'r10k_remote' => 'git@github.com:puppetlabs/nothing', 'r10k_private_key_content' => '-----BEGINfoo', } @@ -61,16 +69,16 @@ it 'installs 2023.4+ with r10k_private_key and r10k_known_hosts' do params = { - 'primary_host' => 'primary', - 'console_password' => 'puppetLabs123!', - 'version' => '2023.4.0', - 'r10k_remote' => 'git@github.com:puppetlabs/nothing', + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2023.4.0', + 'r10k_remote' => 'git@github.com:puppetlabs/nothing', 'r10k_private_key_content' => '-----BEGINfoo', - 'r10k_known_hosts' => [ + 'r10k_known_hosts' => [ { 'name' => 'test', 'type' => 'key-type', - 'key' => 'abcdef', + 'key' => 'abcdef', }, ], 'permit_unsafe_versions' => true, @@ -78,4 +86,14 @@ expect(run_plan('peadm::subplans::install', params)).to be_ok end + + it 'installs 2021.7.9 with legacy compilers' do + params = { + 'primary_host' => 'primary', + 'console_password' => 'puppetLabs123!', + 'version' => '2021.7.9', + 'legacy_compilers' => ['compiler1', 'compiler2'], + } + expect(run_plan('peadm::subplans::install', params)).to be_ok + end end diff --git a/tasks/check_legacy_compilers.rb b/tasks/check_legacy_compilers.rb new file mode 100755 index 00000000..7ca7220b --- /dev/null +++ b/tasks/check_legacy_compilers.rb @@ -0,0 +1,67 @@ +#!/opt/puppetlabs/puppet/bin/ruby +# frozen_string_literal: true + +require 'json' +require 'uri' +require 'net/http' +require 'puppet' + +# CheckLegacyCompilers task class +class CheckLegacyCompilers + def initialize(params) + @nodes = params['legacy_compilers'].split(',') if params['legacy_compilers'].is_a?(String) + end + + def execute! + pinned_nodes = [] + @nodes.each do |node| + node_classification = get_node_classification(node) + pinned = false + node_classification['groups'].each do |group| + if group['name'] == 'PE Master' + pinned_nodes << node + pinned = true + end + end + next if pinned + next unless node_classification.key?('parameters') + next unless node_classification['parameters'].key?('pe_master') + if node_classification['parameters']['pe_master'] + pinned_nodes << node + end + end + + return unless !pinned_nodes.empty? + puts 'The following legacy compilers are classified as Puppet primary nodes:' + puts pinned_nodes.join(', ') + puts 'To continue with the upgrade, ensure that these compilers are no longer recognized as Puppet primary nodes.' + end + + def https(port) + https = Net::HTTP.new('localhost', port) + https.use_ssl = true + https.cert = @cert ||= OpenSSL::X509::Certificate.new(File.read(Puppet.settings[:hostcert])) + https.key = @key ||= OpenSSL::PKey::RSA.new(File.read(Puppet.settings[:hostprivkey])) + https.verify_mode = OpenSSL::SSL::VERIFY_NONE + https + end + + def get_node_classification(certname) + pdb = https(4433) + pdb_request = Net::HTTP::Post.new('/classifier-api/v2/classified/nodes/' + certname) + pdb_request['Content-Type'] = 'application/json' + + response = JSON.parse(pdb.request(pdb_request).body) + + response + end +end + +# Run the task unless an environment flag has been set, signaling not to. The +# environment flag is used to disable auto-execution and enable Ruby unit +# testing of this task. +unless ENV['RSPEC_UNIT_TEST_MODE'] + Puppet.initialize_settings + task = CheckLegacyCompilers.new(JSON.parse(STDIN.read)) + task.execute! +end diff --git a/tasks/classify_compilers.json b/tasks/classify_compilers.json new file mode 100644 index 00000000..cb85a1dd --- /dev/null +++ b/tasks/classify_compilers.json @@ -0,0 +1,15 @@ +{ + "description": "Classify compilers as legacy or non-legacy", + "parameters": { + "compiler_hosts": { + "type": "Array[String]", + "description": "List of FQDNs of compilers" + } + }, + "implementations": [ + { + "name": "classify_compilers.rb", + "requirements": ["shell"] + } + ] +} \ No newline at end of file diff --git a/tasks/classify_compilers.rb b/tasks/classify_compilers.rb new file mode 100755 index 00000000..754f6ad9 --- /dev/null +++ b/tasks/classify_compilers.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby + +require 'json' +require 'open3' + +def classify_compiler(services) + if services.any? { |service| service['type'] == 'puppetdb' } + :non_legacy + else + :legacy + end +end + +params = JSON.parse(STDIN.read) +compiler_hosts = params['compiler_hosts'] + +legacy_compilers = [] +non_legacy_compilers = [] + +compiler_hosts.each do |compiler| + cmd = "puppet infra status --host #{compiler} --format=json" + stdout, stderr, status = Open3.capture3(cmd) + + if status.success? + services = JSON.parse(stdout) + classification = classify_compiler(services) + + if classification == :legacy + legacy_compilers << compiler + else + non_legacy_compilers << compiler + end + else + STDERR.puts "Error running command for #{compiler}: #{stderr}" + end +end + +result = { + 'legacy_compilers' => legacy_compilers, + 'compilers' => non_legacy_compilers +} + +puts result.to_json diff --git a/tasks/get_peadm_config.rb b/tasks/get_peadm_config.rb index 30d8ad21..071229ac 100755 --- a/tasks/get_peadm_config.rb +++ b/tasks/get_peadm_config.rb @@ -47,6 +47,7 @@ def config 'primary_postgresql_host' => postgresql[primary_letter], 'replica_postgresql_host' => postgresql[replica_letter], 'compilers' => compilers.map { |c| c['certname'] }, + 'legacy_compilers' => legacy_compilers.map { |c| c['certname'] }, 'compiler_pool_address' => groups.dig('PE Master', 'config_data', 'pe_repo', 'compile_master_pool_address'), 'internal_compiler_a_pool_address' => groups.dig('PE Compiler Group B', 'classes', 'puppet_enterprise::profile::master', 'puppetdb_host', 1), 'internal_compiler_b_pool_address' => groups.dig('PE Compiler Group A', 'classes', 'puppet_enterprise::profile::master', 'puppetdb_host', 1), @@ -63,7 +64,11 @@ def config 'compilers' => { 'A' => compilers.select { |c| c['letter'] == 'A' }.map { |c| c['certname'] }, 'B' => compilers.select { |c| c['letter'] == 'B' }.map { |c| c['certname'] }, - } + }, + 'legacy_compilers' => { + 'A' => legacy_compilers.select { |c| c['letter'] == 'A' }.map { |c| c['certname'] }, + 'B' => legacy_compilers.select { |c| c['letter'] == 'B' }.map { |c| c['certname'] }, + }, }, } end @@ -81,7 +86,24 @@ def groups # Returns a list of compiler certnames and letters, based on a PuppetDB query def compilers @compilers ||= - pdb_query('inventory[certname,trusted.extensions] { trusted.extensions.pp_auth_role = "pe_compiler" }').map do |c| + pdb_query('inventory[certname,trusted.extensions] { + trusted.extensions.pp_auth_role = "pe_compiler" and + trusted.extensions."1.3.6.1.4.1.34380.1.1.9814" = "false" + }').map do |c| + { + 'certname' => c['certname'], + 'letter' => c.dig('trusted.extensions', '1.3.6.1.4.1.34380.1.1.9813'), + } + end + end + + # Returns a list of legacy compiler certnames and letters, based on a PuppetDB query + def legacy_compilers + @legacy_compilers ||= + pdb_query('inventory[certname,trusted.extensions] { + trusted.extensions.pp_auth_role = "pe_compiler" and + trusted.extensions."1.3.6.1.4.1.34380.1.1.9814" = "true" + }').map do |c| { 'certname' => c['certname'], 'letter' => c.dig('trusted.extensions', '1.3.6.1.4.1.34380.1.1.9813'),