Intermittent failure with test case String::compareTo(const char *)

[jboynes]

Running the unit tests on a VM on a laptop, I get intermittent failures of this testcase. On one run it will pass, on another, with the same binary, it will fail. I realize this isn't a typical target CPU but I wonder if there's an underlying issue.

ubuntu@foo:~/ArduinoCore-API/test/build$ bin/test-ArduinoCore-API
===============================================================================
All tests passed (968 assertions in 304 test cases)

ubuntu@foo:~/ArduinoCore-API/test/build$ bin/test-ArduinoCore-API

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test-ArduinoCore-API is a Catch v2.13.9 host application.
Run with -? for options

-------------------------------------------------------------------------------
Testing String::compareTo(const char *)
       When: Passed string is empty
-------------------------------------------------------------------------------
/home/ubuntu/ArduinoCore-API/test/src/String/test_compareTo.cpp:48
...............................................................................

/home/ubuntu/ArduinoCore-API/test/src/String/test_compareTo.cpp:51: FAILED:
  REQUIRE( str1.compareTo("") == strcmp(str1.c_str(), str2.c_str()) )
with expansion:
  144 == 143

===============================================================================
test cases: 304 | 303 passed | 1 failed
assertions: 968 | 967 passed | 1 failed

ubuntu@foo:~/ArduinoCore-API/test/build$ 

Environment is an Ubuntu VM running under Multipass on an Apple M1 laptop.

$ uname -a
Linux foo 6.2.0-32-generic #32-Ubuntu SMP PREEMPT_DYNAMIC Mon Aug 14 12:37:51 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

[aentinger]

Hi @jboynes ☕ 👋

Thanks for reporting this 🙇 . I've not observed this myself, but I'll keep an eye out for it.

Please let us know, if you should find the issue behind this intermittent failure.

[jboynes]

I'm not sure why it's an issue in my environment but the cause is that the return value from strcmp varies. It's allowed to do that as only the sign of the result is defined, the actual value returned is undefined behaviour. I think comparing to that in the test assertion is the problem.

I created PR #205 to address this.

[aentinger]

Reopen due to insights and fix by @jboynes 🚀 .