-
Notifications
You must be signed in to change notification settings - Fork 14
FAQ
For each period of time defined by the polling_interval parameter, the load balancer server does SNMP get requests on the alias member nodes to retrieve a load metric. The section How to define DNS Load Balanced Alias Members explains how to configure the load metric as well as health monitoring conditions for the alias members. You may configure them to use combinations of Collectd metrics.
The best_hosts parameter defines the number of nodes exposed by the load balanced alias at any time.
The load balancer orders the alias member nodes using the load metric. It only considers nodes with load metric values between 1 and 99999 (i.e. higher than 0 and less than 100000).This means that nodes with values that are negative, 0 or higher than 99999 are in principle excluded from the alias. They may only be included when there are no nodes available and a previous alias state is kept. Within the nodes that are considered, the load balancer takes the best_hosts nodes that are the least loaded and updates the DNS A and AAAA records if needed for their IP addresses to be exposed by the alias.
The best_hosts special value of -1 means that you will expose the IP addresses of all alias members that are configured. This is for instance used for message brokers where you need to consume from all alias members.
Please note that if you have best_hosts greater than 1 or equal to -1, then the order of the list of IPs that you see from the alias may be deterministically biased. This happens for instance in SLC5 due to the bug of getaddrinfo() in glibc described in the following twiki https://twiki.cern.ch/twiki/bin/view/LinuxSupport/GlibcDnsLoadBalancing.
Please note, that in the case of the SLC6 and CC7 resolvers, the list of IPs that you see from the alias is randomly shuffled on each new DNS resolution so if you display multiple IPs for an alias, you cannot rely on them being ordered by load.
You have to bear in mind that a DNS LB alias does not insure session persistence (stickiness).
If your application needs session persistence, you may insure it by installing Memcache in the alias member nodes. You may also consider using load balancers that can keep track of sessions such as Apache or HAProxy.
The lbclient Puppet module allows UDP traffic to port 161 from the LBD servers. This is needed in all LB alias member nodes so that the LBD servers can get the load metric values calculated by the lbclient using the SNMP protocol.
If communication is not established the LBD server will consider the node as not available for the alias.
If the external parameter is set to 'yes', then the LB alias will be visible in the CERN external DNS server. However, this will not open external access to the LB alias member nodes in the CERN firewall. To do so, you would have to create a LANDB set or add the member nodes to an existing one that grants external access.
The CERN Configuration Management System User Guide section CERN Firewall describes how to configure your machines in a LANDB set using Puppet.
By High Availability (HA) use case, we mean the case when we want a specific node to be exposed all the time as long as it is available. We call this node the master. The other failover node is only exposed when the master is not available.
You can solve the HA use case of the lbclient by using a constant load metric, tuned to different values on each of the nodes. You can define constant load metrics with lbclient::loadconstant. For instance you may configure the lbclient in the master node to give a constant 1 as follows:
lbclient::alias::load{ 'Add number one':
load => {'type' => 'constant', data => '1' },
}In the same was you may configure the lbclient in the failover node to give a constant 2 so that it is left behind unless the master is down as follows:
lbclient::alias::load { 'Add number two':
load => {'type' => 'constant', data => '2' },
}Like in the previous cases, this could also be defined in hiera. So on a hostgroup, you could consider something like the following:
include ::lbclient
$master = '<master_node_name>.cern.ch'
if $::fqdn == $master {
lbclient::alias::load { 'Add to load number 1':
load => {'type' => 'constant', data => '1',},
}
} else {
lbclient::alias::load { 'Add to load number 2':
load => {'type' => 'constant', data => '2',},
}
}data/hostgroup/my_hostroup.yaml:
---
lbclient::aliases::definitions:
# All the nodes in the hostgroup will have the definition of the alias
xxx.cern.ch:
checks:
- nologin
- roger
loads:
# The load by default will be 1 on all the ndoes
- type: constant
data: 1
data/fqdns/<failover1>.yaml:
---
# On this particular node, we add a second load
# That will make the load 1+5=6, and this node will be the last one to be selected
lbclient::aliases::definitions:
xxx.cern.ch:
loads:
- type: constant
data: 5
You could also consider to configure the lbclient in the failover node to give a constant worst_value as another way to make sure that it is left behind unless the master is down as follows:
$worst_value = 99999
lbclient::alias::load { 'Add worst value':
load => {type => 'constant', data => $worst_value},} ,
}Multiple LB aliases are supported on a single node. By default they share the lbclient configuration. In this case you need a single lbclient and lbclient::config definitions per node but you may have as many lbd::client definitions as aliases you have defined in the box.
There are two possibilities here: to have the same list of checks for the different aliases defined in the node, or to have different lists per alias. For the case when the aliases share the same list of checks, a possible way to proceed is to create the aliases and the checks in different objects:
class zzzz::loadbalmulti {
#Include the module
include lbclient
#Define the aliases. Note that the list of checks is empty
lbclient::alias{ 'xxxx.cern.ch':}
lbclient::alias{ 'yyyy.cern.ch':}
#Define each of the checks. Note that they will apply to all the aliases on the node
['roger', 'nologin', 'tmpfull'].each | $mycheck| {
lbclient::alias::check {"Check for ${mycheck}":
check => $mycheck,
}
}
...
}
A different option is to have a complete different set of tests for each of the aliases. This can be achieved by defining a /usr/local/etc/lbclient.conf.<aliasname>.cern.ch config file were <aliasname> stands for the name of the specific alias. You may do this with Puppet by having multiple lbclient::config using the target parameter to point to the alias specific config file /usr/local/etc/lbclient.conf.<aliasname>.cern.ch. For instance:
class zzzz::loadbalmulti {
include lbclient
lbclient::alias{ 'xxxx.cern.ch':
checks => ['nologin', 'roger', 'sshdaemon', 'tmpfull'],
target => '/usr/local/etc/lbclient.conf.xxxx.cern.ch',
}
lbclient::alias { 'yyyy.cern.ch':
checks => ['roger', 'nologin', 'webdaemon',],
target => '/usr/local/etc/lbclient.conf.yyyy.cern.ch',
}
}Each alias may be then independently disabled touching the file /etc/iss.nologin.<aliasname>.cern.ch. For instance, in the example above you would touch /etc/iss.nologin.yyyy.cern.ch to disable the node for alias yyyy.cern.ch.
You can achieve the same complete different set of tests for each of the aliases by using the lbclient::aliases class that will define automatically the specific /usr/local/etc/lbclient.conf.<aliasname>.cern.ch targets. For instance:
class zzzz::loadbalmultialiases {
include lbclient
class{'::lbclient::aliases':
definitions => {
'xxxx.cern.ch' => {
checks => ['nologin', 'roger', 'sshdaemon', 'tmpfull'],
},
'yyyy.cern.ch' => {
checks => ['roger', 'nologin', 'webdaemon',],
},
},
}
}Using the using the lbclient::aliases class, you may also add an additional set of checks applied to all different aliases. For instance:
class zzzz::loadbalmultialiases2 {
include lbclient
class{'::lbclient::aliases':
definitions => {
'xxxx.cern.ch' => {
checks => ['nologin', 'roger', 'sshdaemon', 'tmpfull', 'eos'],
},
'yyyy.cern.ch' => {
checks => ['roger', 'nologin', 'webdaemon',],
},
},
checks => ['afs'],
}
}As mentioned before, this could also be achived using hiera.
class zzzz::loadbalmulti {
include lbclient
...
}lbclient::aliases::definitions:
<xxx.cern.ch>:
# These tests will be for only this alias
checks:
- eos
- tmpfull
- roger
- sshdaemon
- nologin
<yyy.cern.ch>:
checks:
- roger
- nologin
- webdaemonPlease, note that unfortunately the roger state seems to be exclusively per node so we cannot consider using it to disable a single alias in the node.
In order to support multiple LB aliases per node with differentiated state, we did the following:
- By default, the lbclient gives a single state for all the aliases in the node with a single integer metric. This is a value of an
integer OIDtype. - When finding independent config files for each alias defined like
/usr/local/etc/lbclient.conf.aliasname.cern.ch, the lbclient then produces a metric ofstring OIDtype with a comma separated key-value pair list such asaliasname1.cern.ch=49,aliasname2.cern.ch=45. - The support of the polymorphism of OID in the LBD server, ie. if the value is not a number try to decode it as a list of comma separated key value pairs and take the value for the current alias. This is running in the production LBDs since August 2014.
- A compatible implementation of the lbclient in Python that we have extended to support multiple aliases with independent states. This implementation uses the type string OID with a comma separated list of key value pairs. This is described in https://its.cern.ch/jira/browse/AI-3885. This version of the lbclient has been tested by the messaging team since August 2014. It is the default in production for SLC6 and CC7 since May 2015.
By default, snmpd is quite verbose on its output. When the lbd contacts lbclient (by default, every five minutes), there will be some lines like the following that appear in the log files
Jan 14 15:45:36 <host> snmpd[2406]: Connection from UDP/IPv6: [xxxx:xxxx:xxx:xx:xx]:41496
Jan 14 15:45:37 <host> snmpd[2406]: Connection from UDP/IPv6: [xxxx:xxxx:xxx:xx:xx]:41496
Jan 14 15:45:37 <host> snmpd[2406]: Connection from UDP: [xxx.xxx.xxx.xxx]->[xxx.xxx.xxx.xxx]:161
Jan 14 15:45:37 <host> snmpd[2406]: Connection from UDP: [xxx.xxx.xxx.xxx]:51451->[xxx.xxx.xxx.xxx]:161
To avoid having these lines in the log, add the following parameter to the yaml configuration of your hostgroup:
snmp::do_not_log_tcpwrappers: 'yes'`
# Node Management in Ermis
## 1. Overview
Node management is a new feature of Ermis API service. As the rest of the service, Node management targets internal users with advanced Networking knowledge. This feature enables the modification of the nodes behind a certain alias. The user interface is accessible through [AIErmis](https://aiermis.cern.ch), under the “Modify Alias” section.
The purpose of Node management feature is to define the nodes that will sit behind a certain alias, during a certain time. The user can allow or forbid nodes to be checked for a particular alias.
## 2.How To Add
1\. Visit : [AIErmis](https://aiermis.cern.ch/lbweb/modifyform).
2\. Choose the alias for which you want to allow or forbid nodes.
3\. Under the node management table, declare the alias names in the format : <name>.cern.ch (sub-domains are also allowed).
4\. Select the access control , *Allow* or *Forbidden*.
5\. Add in the table.
6\. Repeat this process for all the nodes.
7\. Submit.
## 3. How to delete/edit
1\. Visit: [AIErmis](https://aiermis.cern.ch/lbweb/modifyform), and select the alias.
2\. If you want to edit the name of privileges of a certain node, select *Edit*, and when you are finished with the editing, make sure to select *Save*, before submitting.
3\. In case you want to delete an entry, select *Delete* and *Submit*.
## 4. Possible scenarios
By default, the list of nodes that can be presented by an alias is retrieved from PuppetDB. The list will contain all the nodes configured following the instruction on [the alias members](aliasmembers.md). On top of that, there are two possible scenarios :
1\. Any node that has been defined with the privilege *Allow* for that alias --→ Node will be checked.
2\. Any node that has been defined with privilege *Forbidden*--→ Node will not even be checked. It will not be behind that alias.
Note that the fact that a node is been checked does not imply that the node will be behind the alias. The healthy nodes with lowest load will be selected. If there are no healthy nodes, the default behavior is to keep the current nodes (if any).
Therefore, if a node that has been granted the *Allow* privileges does not appear behind the alias, it could be either because *lbclient* returns a negative number on the node, or because there are other nodes behind the alias with a lower load.
## 5. Technical details
Ermis REST service is implemented in Python 2.7 with a MySQL database.
All the alias data, including the ones related to the Node management, are saved in the database. The list of nodes and their relations with each alias is retrieved during the compilation of the configuration file *load-balancing.conf*, which is composed at the load-balancer daemon machine.
It is important to specify that the node does not have to be managed by puppet.
As a result, the *load-balancing.conf* file is composed. The forbidden nodes are omitted from the definition of the cluster for that particular alias.
# Metric Policy
## Definition
The Metric Policy decides what to do in the case when none of the member nodes appears as available to the LBD when evaluating the state of the LB alias.
## Currently Available Options
The currently avaiable options are the following:
- "minimum": (Obsolete and strongly not recommended). With this policy, when none of the member nodes appears as available to the LBD, a random pick is done among the alias members in the configuration (although they are not available at this time) to select which hosts to present (the IP addresses) for the LB alias.
- "minino": (Obsolete and strongly not recommended). With this policy, when none of the member nodes appears as available to the LBD, the LB alias is let to appear as depleted, ie. to present no IP addresses.
- "cmsfrontier": (Current default. Strongly recommended). with this policy, when none of the member nodes appears as available to the LBD, the LB alias presents the IP addresses in the last valid state before the depletion.
Please note that while the Metric Policy is not exposed in the Ermis GUI, it can still be changed with the kermis CLI:
-bash-4.2$ kermis --help ... -m METRIC, --metric=METRIC Define the metric policy that should be used. Default 'cmsfrontier' for create. May be updated
However, there is a large consensus that the "cmsfrontier" is the best one for all the existing use cases and there is no interest to change it.
## Historical Note
The original LBD implementation proposed as default the "minimum" metric policy. This policy did a random pick among the alias members in the configuration (although they are not available at this time) to select which hosts to present (the IP addresses) for the LB alias.
This policy had the positive property that the alias did not appear as depleted, the user was just getting a node that was not responsive.
In some cases, this policy presented the risk on falling on a node not in production with a "wrong" database (It happened, for instance with CASTOR).
This is why when migrating the LDB from Quattor to Puppet environment in 2012 we decided to set as default the "minino" policy that was allowing the LB alias to appear as depleted when no alias member appeared as available rather than doing a random pick among them.
While we certainly avoided the risk of the random pick of a development node with a "wrong" database backing, with some time, we realized that, in many cases, a depleted alias, ie. a DNS name presenting no IP addresses was more confusing to users than one pointing to a non-responsive node. This lead to some critical services staying with the "minimum" policy.
So end of 2016 after a request in https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF0679888 we set off to implement a metric policy that would avoid alias depletion by presenting the IPs in the last valid state before the depletion. We called this policy "cmsfrontier". While the implementation of this policy adds the complexity that the LBD to keep the previous state of the LB aliases, on the other hand, it provides a flexible behaviour that prevents alias depletion without a random pick (best of all worlds).
In 2017, we started moving aliases to this policy on demand.
In 2018 we set it as the default for new aliases when we deployed the LBD implementation in Go.
In Q1 2021 we did a campaign to contact all alias owners still on "minimum" policy to persuade them to migrate to "cmsfrontier".
So since end of March 2021 all aliases in the LBD are using the "cmsfrontier" policy.