Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Error objects in TestEnd.error assertions list #123

Closed
Krinkle opened this issue Sep 30, 2020 · 1 comment
Closed

Allow Error objects in TestEnd.error assertions list #123

Krinkle opened this issue Sep 30, 2020 · 1 comment

Comments

@Krinkle
Copy link
Member

Krinkle commented Sep 30, 2020

This follows up from #105 where @keithamus brought up the following:

the desire is the next major version of Chai (v5) will implement 3.5 Assertion events as part of the CRI. Any failed assertion will emit an assertion event (and if there are no listeners to that event it'll throw). In a CRI world, I would expect assertion library agnostic runners such as Mocha to simply reflect events coming from assertion libraries like Chai - in other words they wouldn't create Assertion event objects, merely listen to the assertion libraries events and re-dispatch them on their own event bus.

Although I'm not an author of a test runner library, I would imagine for uncaught non-assert runtime errors, marking the TestEnd event as failed would be sufficient. Perhaps it should be allowed that errors is an Array<Assertion | Error> meaning that non-assertion errors can be added to the errors property? This is drifting from the topic though.

I support this idea.

In concrete terms for CRI, how much of the Error object shape should we specify as something reporters may rely on?

In terms of the ECMAScript spec, instances of Error effectively boil down to:

  • string message: Error message.
  • string name: Error class name, or custom prefix.
  • (non-standard) string stack: Optional property with a stack trace.

The CRI spec only specifies object shape and properties and thus any class inheritence is safe to ignore, serialize as JSON, transport over the network, and re-emit elsewhere to a reporter. But, this also means we can't rely on instanceof checks and thus need to standardise a reliable way for reporters to distinguish between Error and Assertion object shapes.

Some ideas:

  • Have separate lists, e.g. errors and assertions and require that reporters interested in printing failures to also iterate assertions but to ignore entries having assertion.passed === true.
  • Double down on separate lists, e.g. errors, failedAssertions, allAssertions.
  • Go the other way, by merging the two object shapes into one. E.g. make Error a valid Assertion object by making actual and expected optional. This seems messy, but might be better? It would simplify things for reporters and would require some amount of defensive coding and/or duck-typing when printing failures. Although for the most part, using name and (optional) stack which both implement would suffice.
    • The asssertion.passed property would become optional, with undefined basically meaning false so that an Error is seen as a failed assertion (instead of a passed assertion).
    • The actual and expected properties would also become optional. This might make sense regardless, given issue Consider diff property on assertion object #106. Because we want to accomodate assertions where there the expected value isn't a single/explicit JS value.

Do any of these seem great? Perhaps there's another way we can do this better?

@Krinkle
Copy link
Member Author

Krinkle commented Feb 21, 2021

I'm declining this for now, effectively freezing the CRI event model mostly as-is. Instead I recommend new adopters that are blocked for one reason or another to instead invest in TAP streams. See #133.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant