Ignoring labels adapter/port on metric group partition-attached-network-interface

[andy-maier]

The following errors happen continuously in zhmc-prometheus-exporter version TBD:

2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'adapter' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/c16de202-c35b-11ee-97ac-525412345600'
2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'port' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/c16de202-c35b-11ee-97ac-525412345600'
2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'adapter' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/c448de8c-c35b-11ee-a0ab-525412345600'
2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'port' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/c448de8c-c35b-11ee-a0ab-525412345600'
2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'adapter' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/bb399b4c-c35b-11ee-a0ab-525412345600'
2024-02-04 13:14:11+0000 WARNING zhmcexporter: Ignoring label 'port' on metric group 'partition-attached-network-interface' due to error in rendering label value Jinja2 expression: KeyError: '/api/partitions/b7aaf25a-c35b-11ee-97ac-525412345600/nics/bb399b4c-c35b-11ee-a0ab-525412345600'

[Charles1000Chen]

The zhmc-prometheus-exporter version is 1.5.0.

[andy-maier]

Initial analysis:

The KeyError is raised when the NIC URI is looked up in adapter_name_func() and adapter_port_func() which try to get the NIC resource object from a uri2resource dictionary that is set up during startup of the exporter. If a NIC is not found there, this means that the NIC came into existence after the start of the exporter.

Solving this ideally should be done with issue #227.

A stop-gap solution might be to expand the uri2resource dictionary whenever a resource is not found. This requires some support in the zhmcclient library for determining the backing port of a NIC. I created issue zhmcclient/python-zhmcclient#1451 for that.

[andy-maier]

PR #457 expands the uri2resource dictionary when a resource is not found. It turned out that it is not necessary to use the new Nic.backing_port()method added in the PR for zhmcclient/python-zhmcclient#1451.

I tested the change on the HMC of A224. However, I could not reproduce the new log message when adding NICs.

Also, in tests on dal12z1.CPCB where the problem originally was observed on partition dal2-qz2-sr2-rk070-m09 it could not be reproduced. We ran the old exporter and an exporter with the code from this PR in parallel for 2 days, but creating NICs did not trigger the situation.

So we don't understand yet what triggers the situation. However, the change in PR #457 handles the situation definitely better than before when it occurs.