QUnit.todo() with a failure are reported as failling

[hashar]

I had the use case of a test being properly implemented but the feature code being incomplete and failing. When running the tests with the QUnit web interface, the test is not erroring out and is properly labeled todo.

However when running karma, the test is wrongfully reported as failing:

SUMMARY:
✖ 1 test failed

FAILED TESTS:
  ext.graph.visualEditor
    ✖ ve.ui.TableWidget
      Chrome 66.0.3359 (Linux 0.0.0)
    Died on test #4     at 
....
    TypeError: Cannot read property '0' of undefined
        at VeUiTableWidget.ve.ui.TableWidget.onInsertColumn   // expected failure
...
        at runTest (node_modules/qunit/qunit/qunit.js:1530:30)
        at Test.run (node_modules/qunit/qunit/qunit.js:1516:6)
        at node_modules/qunit/qunit/qunit.js:1728:12
        at advanceTaskQueue (node_modules/qunit/qunit/qunit.js:1129:6)
        at advance (node_modules/qunit/qunit/qunit.js:1110:4)

In lib/adapter.js I adjusted runner.testDone to print the test and testResult to the console which yields:

LOG: Object{
  name: 've.ui.TableWidget',
  module: 'ext.graph.visualEditor',
  skipped: false,
  todo: true,
  failed: 1,
  passed: 3,
  total: 4,
LOG: Object{success: false, errors: ['Died on test #4 ...

The testResult is set by runner.log which mark it with success = false. However in runner.testDone, the todo flag should be used to invert the result. Potentially:

--- orig.js	2018-06-26 22:31:26.880087441 +0200
+++ new.js	2018-06-26 22:32:36.218186993 +0200
@@ -8,6 +8,11 @@
 		time: new Date().getTime() - timer
 	}
 
+	// For QUnit.todo(): flip success report
+	if ( test.todo === true ) {
+		result.success = !testResult.success
+	}
+
 	if (result.description.indexOf('global failure') !== -1) {
 		return
 	}

Environment:

• karma 2.0.2
• karma-qunit 2.0.1
• qunit 2.6.0

[hashar]

From a discussion I had with @Krinkle , it is left to determine whether QUnit itself should handle the todo flag and flip the success or whether it should be handled by the reporter.