debug: Fix tests so they pass/not_applicable on HiFive1

[timsifive]

#485 introduces a list of tests that are known to cause an error on HiFive1. This issue is to go through that list, and fix the tests/infrastructure/whatever so that all these tests either pass, or return not_applicable.

[TommyMurphyTM1234]

Hi @timsifive (and/or others) - I'm a bit confused about the how exactly specific tests are categorised as "not_applicable".
If I use the following (i.e. without appending --exclude-tests hifive1_excludes.yaml to specify the HiFive1 specific exclude list):

./gdbserver.py targets/SiFive/HiFive1.py

then I get this:

:::::::::::::::::::::::::::[ ran 72 tests in 355s ]:::::::::::::::::::::::::::
18 tests returned not_applicable
43 tests returned pass
5 tests returned fail
   EtriggerTest > logs/20231006-114621-HiFive1-EtriggerTest.log
   IcountTest > logs/20231006-114642-HiFive1-IcountTest.log
   InstantHaltTest > logs/20231006-114655-HiFive1-InstantHaltTest.log
   Semihosting > logs/20231006-114937-HiFive1-Semihosting.log
   SemihostingFileio > logs/20231006-114944-HiFive1-SemihostingFileio.log
6 tests returned exception
   Hwbp2 > logs/20231006-114632-HiFive1-Hwbp2.log
   ItriggerTest > logs/20231006-114713-HiFive1-ItriggerTest.log
   MemTestReadInvalid > logs/20231006-114826-HiFive1-MemTestReadInvalid.log
   MemorySampleMixed > logs/20231006-114833-HiFive1-MemorySampleMixed.log
   MemorySampleSingle > logs/20231006-114845-HiFive1-MemorySampleSingle.log
   RepeatReadTest > logs/20231006-114931-HiFive1-RepeatReadTest.log

What I don't understand is what dictates this:

18 tests returned not_applicable

I've skimmed the Python scripts but cannot answer this question so far.
Can anybody else?
Thanks.

[TommyMurphyTM1234]

Ah - I think I get it now - class Target sets certain attributes to default values that disable certain tests because they depend on specific target capabilities/features. Those tests are only enabled if a specific target derived from class Target (e.g. class HiFive1) overrides the relevant attributes in order to declare support for the tests/required capabities?

Specifically - the following tests are assumed to be "not_applicable" by default and the list links to the relevant class Target attributes and checks in gdbserver.py:

• CeaseStepiTest
  • gdbserver.py check:

    riscv-tests/debug/gdbserver.py

    Line 1864 in 9905a43

    return self.hart.support_cease

  • class Target attribute:

    riscv-tests/debug/targets.py

    Line 54 in 9905a43

    support_cease = False

• CustomRegisterTest

  • riscv-tests/debug/gdbserver.py

    Line 193 in 9905a43

    return self.target.implements_custom_test

  • riscv-tests/debug/targets.py

    Line 101 in 9905a43

    implements_custom_test = False

• FreeRtosTest

  • riscv-tests/debug/gdbserver.py

    Line 1949 in 9905a43

    return self.target.freertos_binary

  • riscv-tests/debug/targets.py

    Line 130 in 9905a43

    freertos_binary = None

• HwbpManual

  • riscv-tests/debug/targets.py

    Line 122 in 9905a43

    support_manual_hwbp = True

  • riscv-tests/debug/gdbserver.py

    Line 612 in 9905a43

    return self.hart.instruction_hardware_breakpoint_count > 0

• MemTestBlockReadInvalid

  • riscv-tests/debug/gdbserver.py

    Line 298 in 9905a43

    return self.target.invalid_memory_returns_zero

  • riscv-tests/debug/targets.py

    Line 104 in 9905a43

    invalid_memory_returns_zero = False

• MulticoreRegTest
• MulticoreRtosSwitchActiveHartTest
• MulticoreRunAllHaltOne

  • riscv-tests/debug/gdbserver.py

    Line 1170 in 9905a43

    return len(self.target.harts) > 1

  • riscv-tests/debug/targets/SiFive/HiFive1.py

    Line 11 in 9905a43

    harts = [HiFive1Hart()]

• PrivChange

  • riscv-tests/debug/gdbserver.py

    Line 1607 in 9905a43

    if 0 not in self.supported:

  • riscv-tests/debug/gdbserver.py

    Line 1574 in 9905a43

    self.supported.add(0)

• SmpSimultaneousRunHalt

  • riscv-tests/debug/gdbserver.py

    Line 1227 in 9905a43

    return len(self.target.harts) > 1 and self.target.support_hasel

  • riscv-tests/debug/targets/SiFive/HiFive1.py

    Line 11 in 9905a43

    harts = [HiFive1Hart()]

• StepThread2Test

  • riscv-tests/debug/gdbserver.py

    Line 2020 in 9905a43

    return len(self.target.harts) > 1

  • riscv-tests/debug/targets/SiFive/HiFive1.py

    Line 11 in 9905a43

    harts = [HiFive1Hart()]

• Sv32Test
• Sv39Test
• Sv48Test

  • riscv-tests/debug/gdbserver.py

    Line 1639 in 9905a43

    return self.hart.ram_size >= 32 * 1024

  • riscv-tests/debug/targets/SiFive/HiFive1.py

    Line 6 in 9905a43

    ram_size = 16 * 1024

• UnavailableCycleTest

  • riscv-tests/debug/gdbserver.py

    Line 1924 in 9905a43

    return self.target.support_unavailable_control

  • riscv-tests/debug/targets.py

    Line 133 in 9905a43

    support_unavailable_control = False

• UnavailableMultiTest

  • riscv-tests/debug/gdbserver.py

    Line 1820 in 9905a43

    self.target.support_unavailable_control) \

  • riscv-tests/debug/targets.py

    Line 133 in 9905a43

    support_unavailable_control = False

• UnavailableRunTest

  • riscv-tests/debug/gdbserver.py

    Line 1887 in 9905a43

    return self.hart.support_cease or \

  • riscv-tests/debug/targets.py

    Line 54 in 9905a43

    support_cease = False

• VectorTest

  • riscv-tests/debug/gdbserver.py

    Line 1713 in 9905a43

    if not self.hart.extensionSupported('V'):

  • riscv-tests/debug/targets/SiFive/HiFive1.py

    Line 8 in 9905a43

    misa = 0x40001105

[timsifive]

Right. The not_applicable() method is older than the HiFive1 exclusion list. In theory it's more flexible if we add support for more hardware in the future. In practice I haven't really kept it up-to-date. Ideally the tests/target definitions would be updated so that HiFive1 wouldn't need the separate exclusion list.