Update baseline script to ignore emerg logs that irq handler is missing

[musamaanjum]

An emergy log is printed if no irq handler is found in the vector during boot. Because of this log, we get lots of false positives. Investigate first if it is safe to ignore, then update the baseline script to ignore these. In this way, the results would be cleaner. Example job

kern  :emerg : call_irq_handler: 1.55 No irq handler for vector

kern  :emerg : call_irq_handler: 2.55 No irq handler for vector
kern  :emerg : call_irq_handler: 3.55 No irq handler for vector
[   12.324058] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=emerg RESULT=fail UNITS=lines MEASUREMENT=3>

The updates are needed in ./config/rootfs/debos/overlays/baseline/opt/kernelci/dmesg.sh.

[musamaanjum]

dmesg.sh file is copied into the rootfs at creation time. We'll have to build the rootfs again.

I'm not sure why dmesg.sh file is copied from kernelci.org instead of master. It seems kernelci.org branch was being kept to store updated data files. @nuclearcat Are we still using kernelci.org branch or should I update the dmesg.sh file in the master?

[laura-nao]

@musamaanjum thanks for looking into this! There was a discussion about this when we first enabled baseline/baseline-nfs tests on Chromebooks (see kernelci/kernelci-core#2483). The errors are known and harmless on zork and grunt devices (and their derivatives). There's a way to exclude specific Tast tests that are known/expected to fail (see: kernelci/kernelci-core#2489) but I don't think we have a counterpart for non-ChromeOS tests at the moment.

I think it would be useful to auto-match these errors on the affected boards, so the tests will keep failing but the users will be able to see it's a known issue (and will be able to filter the results out as well).

@helen-fornazier I'm not up to speed on the status of auto-matching issues, is this something we can already add or is it still in the works?

[laura-nao]

Duplicate of kernelci/kernelci-core#2483

[musamaanjum]

@laura-nao Thanks for letting me know about this. I didn't know. Let's wait for @helen-fornazier to comment. Let's try to expedite the solution as it is going to help me a lot in opening fewer failed reports and speed up my overall work.

[musamaanjum]

Let's close this ticket and continue discussion over kernelci/kernelci-core#2483.

[musamaanjum]

As discussed in the KernelCI weekly, as we don't have the ability to ignore known issues at this time, let's update the dmesg.sh script to ignore these warnings.

[laura-nao]

As discussed in the KernelCI weekly, as we don't have the ability to ignore known issues at this time, let's update the dmesg.sh script to ignore these warnings.

How about auto-generated issues based on log matching, as mentioned in #496 (comment)? Has this option already been excluded?

I don't think completely hiding the warnings is a good idea, we know they're harmless on grunt and zork devices but I'm not sure they won't be fatal on other platforms. And filtering out the messages in the dmesg.sh script itself only for specific boards looks a bit hacky.

[musamaanjum]

In the meeting, participants agreed that Auto log matching would be the best solution. But as we don't have any plan to add this, we can ignore these specific warnings temporarily. Additionally, I'll try to report them to upstream to see if we can reduce their log level if they aren't important errors.
cc: @nuclearcat @spbnick

[laura-nao]

Another option that comes to mind is related to Grafana’s log_line field (in the All tests view), which uses regex to match specific error messages in the logs. Sometimes, the kern :emerg : call_irq_handler: 3.55 No irq handler for vector message is matched correctly, but in other cases, different errors are prioritized. If I remember correctly, these regex patterns were manually set up to catch recurring issues. Maybe it’s possible to adjust them so that the IRQ handler message gets matched first? This way we can easily identify these cases and filter them out.

It’s not an ideal solution, but since we’re considering temporary workarounds, this could be an alternative to modifying the dmesg.sh script, which was supposed to be generic.