CA-389176: Fix missing xcp-rrdd-plugin logs (preserve directory collection order)

[bernhardkaindl]

This PR fixes a regression caused by #51 (Run commands before traversing directories):
The files listed in these messages are currently not collected, fix this regression:

Omitting /var/log/xcp-rrdd-plugins.log.3.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.6.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.19.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.16.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.22.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.13.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.18.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.17.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.15.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.12.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.11.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.24.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.29.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.31.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.28.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.25.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.30.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.7.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.26.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.10.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.5.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.14.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.4.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.23.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.1, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.2.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.21.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.20.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.8.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.9.gz, size constraint of xcp-rrdd-plugins exceeded
Omitting /var/log/xcp-rrdd-plugins.log.27.gz, size constraint of xcp-rrdd-plugins exceeded

For the external bugtool plugin xcp-rrdd-plugins, the order of the directory traversal of the trees it collects is important:

File /etc/xensource/bugtool/xcp-rrdd-plugins.xml:

<capability pii="maybe" max_size="1638400" max_time="60" mime="text/plain" checked="true"/>

File /etc/xensource/bugtool/xcp-rrdd-plugins/stuff.xml:

<collect>
  <directory pattern=".*xcp-rrdd-plugins\.log.*">/var/log</directory>
  <directory label="rrdd_shm">/dev/shm/metrics</directory>
</collect>

/dev/shm/metrics/xcp-rrdd-iostat is a sparse file that may hold just 8k of actual data, but its apparent size is ~8 MB. Using the current algorithm, the large, sparse file gets added nonetheless. But after it, the max_size limit of 1.6 MB set by the plugin's .xml is exceeded, and file_output then rejects further files in the same cap.

When xen-bugtool --all is used, Python2 no longer preserves the order of the directories in the directory_specifications variable.

Then, /dev/shm/metrics is collected before /var/log/xcp-rrdd-plugins, which means that -rrdd-plugins log files are NOT collected.

Of course, as described, also other factors are to blame:

• Not getting the actual, but the apparent sparse file size
• Collecting too large files that exceed the limits instead of rejecting them (which would allow small files after them).

But the immediate trigger for this regression is not preserving the directory order in Python 2.

Fix it by using OrderedDict() (like Python 3 always does anyway) for the directory_specifications dictionary (only two dozen entries).

[coveralls]

Pull Request Test Coverage Report for Build 8015564219

Details

• 0 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
• 74 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-3.0%) to 87.776%

---

Files with Coverage Reduction	New Missed Lines	%
xen-bugtool	74	84.28%

Totals
Change from base Build 8006372356:	-3.0%
Covered Lines:	1788
Relevant Lines:	2037

---

💛 - Coveralls

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.29%. Comparing base (33b6d6e) to head (cc0b8ab).
Report is 14 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master      #88   +/-   ##
=======================================
  Coverage   88.29%   88.29%           
=======================================
  Files          18       18           
  Lines        2195     2196    +1     
=======================================
+ Hits         1938     1939    +1     
  Misses        257      257           

Flag	Coverage Δ
python2.7	86.95% <100.00%> (+<0.01%)	⬆️
python3.10.13	87.77% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ashwin9390]

Python3.7+ maintains insertion order using builtin dictionary {} where ,older versions of python less than 3.7 would require ordered dictionary(OrderedDict()). Since this is would be python 2.x the change holds good here.

[psafont]

This class is available in both 3.1+ and 2.7, so it should bring consistent behaviour.

[bernhardkaindl]

Python3.7+ maintains insertion order using builtin dictionary {} where ,older versions of python less than 3.7 would require ordered dictionary(OrderedDict()). Since this is would be python 2.x the change holds good here.

Thank you for your review and your analysis!

And, just to complete the justification for this change:
There is one major reason to not yet change to Python3 in XS8:

• We need to backport the change from CP-41238: To save RRDs before capturing files, run commands before collecting directories #51 (which caused this regression) for "Yangtze" for the hotfix UPD-990.

• Delivering this regression from CP-41238: To save RRDs before capturing files, run commands before collecting directories #51 to customers would impact the completeness of the bug-report info for 8.2 CU1 (Python 2-only) for customer support teams as well as our own development analysis of possible customer XSI tickets.