Skip to content

Commit

Permalink
Merge pull request #124 from StatensPensjonskasse/multihost-and-roles…
Browse files Browse the repository at this point in the history 
…-in-setfile

Add option to generate setfile with multiple hosts and roles
  • Loading branch information
@bastelfreak
bastelfreak authored Apr 25, 2024
2 parents ae37836 + 6ea0b1c commit ea38f86
Show file tree
Hide file tree
Showing 6 changed files with 330 additions and 129 deletions.
146 changes: 109 additions & 37 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,23 @@ To get outputs [usable in Github Actions](https://docs.github.com/en/free-pro-te

```console
$ metadata2gha
puppet_major_versions=[{"name":"Puppet 7","value":7,"collection":"puppet7"},{"name":"Puppet 6","value":6,"collection":"puppet6"}]
puppet_unit_test_matrix=[{"puppet":7,"ruby":"2.7"},{"puppet":6,"ruby":"2.5"}]
github_action_test_matrix=[{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 7","value":7,"collection":"puppet7"}},{"setfile":{"name":"Debian 11","value":"debian11-64"},"puppet":{"name":"Puppet 6","value":6,"collection":"puppet6"}}]
puppet_major_versions=[{"name":"Puppet 8","value":8,"collection":"puppet8"},{"name":"Puppet 7","value":7,"collection":"puppet7"}]
puppet_unit_test_matrix=[{"puppet":8,"ruby":"3.2"},{"puppet":7,"ruby":"2.7"}]
puppet_beaker_test_matrix=[{"name":"Puppet 8 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet8","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet8}"}},{"name":"Puppet 7 - Debian 12","env":{"BEAKER_PUPPET_COLLECTION":"puppet7","BEAKER_SETFILE":"debian12-64{hostname=debian12-64-puppet7}"}}]
```

Puppet major versions formatted for readability:
```json
[
{
"name": "Puppet 8",
"value": 8,
"collection": "puppet8"
},
{
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
},
{
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
}
]
```
Expand All @@ -41,45 +41,37 @@ Puppet unit test matrix formatted for readability:
```json
[
{
"puppet": 7,
"ruby": "2.7"
"puppet": 8,
"ruby": "3.2"
},
{
"puppet": 6,
"ruby": "2.5"
"puppet": 7,
"ruby": "2.7"
}
]
```

GitHub Action test matrix formatted for readability
Beaker test matrix formatted for readability
```json
[
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 7",
"value": 7,
"collection": "puppet7"
"name": "Puppet 8 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet8",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet8}"
}
},
{
"setfile": {
"name": "Debian 11",
"value": "debian11-64"
},
"puppet": {
"name": "Puppet 6",
"value": 6,
"collection": "puppet6"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}"
}
}
]
```

It is also possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).
It is possible to specify the path to metadata.json and customize the setfiles. For example, to ensure the setfiles use FQDNs and apply the [systemd PIDFile workaround under docker](https://github.com/docker/for-linux/issues/835). This either means either using an older image (CentOS 7, Ubuntu 16.04) or skipping (CentOS 8).

```console
$ metadata2gha --use-fqdn --pidfile-workaround true /path/to/metadata.json
Expand All @@ -89,21 +81,101 @@ This results in the following JSON data
```json
[
{
"name": "CentOS 7",
"value": "centos7-64{hostname=centos7-64.example.com,image=centos:7.6.1810}"
"name": "Puppet 7 - CentOS 7",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "centos7-64{hostname=centos7-64-puppet7.example.com,image=centos:7.6.1810}"
}
},
{
"name": "Debian 10",
"value": "debian10-64{hostname=debian10-64.example.com}"
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7.example.com}"
}
},
{
"name": "Ubuntu 18.04",
"value": "ubuntu1804-64{hostname=ubuntu1804-64.example.com}"
"name": "Puppet 7 - Ubuntu 22.04",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "ubuntu2204-64{hostname=ubuntu2204-64-puppet7.example.com}"
}
}
]
```

It is also possible to specify a comma separated list of operating systems as used in `metadata.json` (`CentOS,Ubuntu`).
If you need custom hostname or multiple hosts in your integration tests this could be achived by using the --beaker-hosts option

Option argument is 'HOSTNAME:ROLES;HOSTNAME:..;..' where
- hosts are separated by ';'
- host number and roles are separated by ':'
- Roles follow beaker-hostgenerator syntax
If you don't need any extra roles use '1;2;..'

```console
$ metadata2gha --beaker-hosts 'foo:primary.ma;bar:secondary.a'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64primary.ma{hostname=foo-puppet7}-debian12-64secondary.a{hostname=bar-puppet7}"
}
}
]
```

If you need to Expand the matrix ie by product versions it could be achived by using the --beaker-facter option

Option argument is 'FACT:LABEL:VALUE,VALUE,..' where
- Fact, label and values are separated by ':'
- Values are separated by ','

```console
$ metadata2gha --beaker-facter 'mongodb_repo_version:MongoDB:4.4,5.0,6.0,7.0'
```

This results in the following JSON data
```json
[
{
"name": "Puppet 7 - Debian 12 - MongoDB 4.4",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "4.4"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 5.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "5.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 6.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "6.0"
}
},
{
"name": "Puppet 7 - Debian 12 - MongoDB 7.0",
"env": {
"BEAKER_PUPPET_COLLECTION": "puppet7",
"BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}",
"BEAKER_FACTER_mongodb_repo_version": "7.0"
}
}
]
```

## Work with the API

Expand Down
10 changes: 10 additions & 0 deletions bin/metadata2gha
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ options = {
domain: nil,
minimum_major_puppet_version: nil,
beaker_fact: nil,
beaker_hosts: nil,
}

OptionParser.new do |opts|
Expand Down Expand Up @@ -40,6 +41,15 @@ OptionParser.new do |opts|
options[:beaker_facter] = [fact, label, values.split(',')]
end
end
opts.on('--beaker-hosts HOSTNAME:ROLES;HOSTNAME:ROLES;...', 'Expand the setfile string to create multiple hosts with custom roles. Roles string; see beaker-hostgenerator') do |opt|
options[:beaker_hosts] = {}
if opt != 'false'
opt.split(';').each do |host|
hostname, roles = host.split(':', 2)
options[:beaker_hosts][hostname] = roles
end
end
end
end.parse!

filename = ARGV[0]
Expand Down
51 changes: 43 additions & 8 deletions lib/puppet_metadata/beaker.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,34 +55,69 @@ def adjusted_os(os)
# Enforce a domain to be appended to the hostname, making it an FQDN
# @param [Optional[String]] puppet_version
# The desired puppet version. Will be appended to the hostname
# @param [Optional[Hash]] hosts
# Key: hostname, Value: roles (roles string as defined by beaker-hostgenerator )
# Override the automatically generated hostname and optionally add roles
# If more than one entry this will generate multiple hosts in the setfile
# The domain may still be set via the `domain` param.
#
# @return [nil] If no setfile is available
# @return [Array<(String, String)>] The beaker setfile description with a readable name
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil)
def os_release_to_setfile(os, release, use_fqdn: false, pidfile_workaround: false, domain: nil, puppet_version: nil, hosts: nil)
return unless os_supported?(os)

aos = adjusted_os(os)

name = "#{aos}#{release.tr('.', '')}-64"
hostname = (puppet_version.nil? && puppet_version != 'none') ? name : "#{name}-#{puppet_version}"
human_name = "#{os} #{release}"
domain ||= 'example.com' if use_fqdn

options = {}
options[:hostname] = "#{hostname}.#{domain}" if domain
hosts_settings = []
if hosts
hosts.each do |hostname, roles|
hosts_settings << {
'name' => if roles
name + roles
elsif hosts.size > 1
hosts_settings.empty? ? "#{name}.ma" : "#{name}.a"
else
name
end,
'hostname' => ((puppet_version.nil? || puppet_version == 'none') ? hostname : "#{hostname}-#{puppet_version}") + (domain ? ".#{domain}" : ''),
}
end
else
hosts_settings << {
'name' => name,
'hostname' => if puppet_version && puppet_version != 'none'
"#{name}-#{puppet_version}" + (domain ? ".#{domain}" : '')
elsif domain
name + (domain ? ".#{domain}" : '')
else
''
end,
}
end

options = {}
# Docker messes up cgroups and some systemd versions can't deal with
# that when PIDFile is used.
image_to_use = nil
if pidfile_workaround?(pidfile_workaround, os)
return if PIDFILE_INCOMPATIBLE[os]&.include?(release)

if (image = PIDFILE_COMPATIBLE_IMAGES.dig(os, release))
options[:image] = image
image_to_use = image
end
end

human_name = "#{os} #{release}"
setfile_parts = []
hosts_settings.each do |host_settings|
options[:hostname] = host_settings['hostname'] unless host_settings['hostname'].empty?
options[:image] = image_to_use if image_to_use
setfile_parts << build_setfile(host_settings['name'], options)
end

[build_setfile(name, options), human_name]
[setfile_parts.join('-'), human_name]
end

# Return whether a Beaker setfile can be generated for the given OS
Expand Down
1 change: 1 addition & 0 deletions lib/puppet_metadata/github_actions.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ def os_release_to_beaker_setfile(os, release, puppet_collection)
pidfile_workaround: options[:beaker_pidfile_workaround],
domain: options[:domain],
puppet_version: puppet_collection,
hosts: options[:beaker_hosts],
)
end
end
Expand Down
55 changes: 54 additions & 1 deletion spec/beaker_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

it { expect(described_class.os_release_to_setfile('SLES', '11')).to be_nil }

describe 'pidfile_workaround' do
context 'with pidfile_workaround' do
describe 'true' do
[
['CentOS', '6', ['centos6-64', 'CentOS 6']],
Expand Down Expand Up @@ -57,5 +57,58 @@
it { expect(described_class.os_release_to_setfile('CentOS', '8', pidfile_workaround: ['CentOS'])).to be_nil }
end
end

context 'with domain' do
it {
expect(described_class.os_release_to_setfile('CentOS', '7', domain: 'mydomain.org')).to eq(['centos7-64{hostname=centos7-64.mydomain.org}', 'CentOS 7'])
}
end

context 'with puppet_version' do
[
['CentOS', '7', 'none', ['centos7-64', 'CentOS 7']],
['CentOS', '7', 'puppet7', ['centos7-64{hostname=centos7-64-puppet7}', 'CentOS 7']],
].each do |os, release, puppet_version, expected|
it { expect(described_class.os_release_to_setfile(os, release, puppet_version: puppet_version)).to eq(expected) }
end
end

context 'with hosts and no roles' do
[
['Debian', '12', { 'foo' => nil }, ['debian12-64{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => nil, 'bar' => nil }, ['debian12-64.ma{hostname=foo}-debian12-64.a{hostname=bar}', 'Debian 12']],
].each do |os, release, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts and roles' do
[
['Debian', '12', { 'foo' => 'myrole.ma' }, ['debian12-64myrole.ma{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => 'myrole,primary.ma' }, ['debian12-64myrole,primary.ma{hostname=foo}', 'Debian 12']],
['Debian', '12', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo}-debian12-64myrole,secondary.a{hostname=bar}', 'Debian 12'],],
].each do |os, release, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts, roles and domain' do
[
['Debian', '12', 'mydomain.org', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo.mydomain.org}-debian12-64myrole,secondary.a{hostname=bar.mydomain.org}', 'Debian 12'],],
].each do |os, release, domain, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, domain: domain, hosts: hosts)).to eq(expected) }
end
end

context 'with hosts, roles, domain and puppet_version' do
[
['Debian', '12', 'mydomain.org', 'puppet7', { 'foo' => 'myrole,primary.ma', 'bar' => 'myrole,secondary.a' },
['debian12-64myrole,primary.ma{hostname=foo-puppet7.mydomain.org}-debian12-64myrole,secondary.a{hostname=bar-puppet7.mydomain.org}', 'Debian 12'],],
].each do |os, release, domain, puppet_version, hosts, expected|
it { expect(described_class.os_release_to_setfile(os, release, domain: domain, puppet_version: puppet_version, hosts: hosts)).to eq(expected) }
end
end
end
end
Loading

0 comments on commit ea38f86

Please sign in to comment.