Skip to content

@expectedFailure decorator should result in # TODO directives #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
smcv opened this issue May 10, 2018 · 4 comments · Fixed by #75
Closed

@expectedFailure decorator should result in # TODO directives #74

smcv opened this issue May 10, 2018 · 4 comments · Fixed by #75

Comments

@smcv
Copy link
Contributor

smcv commented May 10, 2018

Other TAP producers, such as Perl, report expected failures and unexpected successes like this:

not ok 1 - this should be equal but we have not implemented it yet # TODO not implemented yet
ok 2 - likewise # TODO not implemented yet
1..2

Here test 1 is an expected failure, and test 2 is an unexpected success (it was marked as an expected failure but succeeded anyway).

@expectedFailure currently prints _('(expected failure)') or _('(unexpected success)'). If it printed # TODO first, resulting in something like

not ok 1 - this should be equal but we have not implemented it yet # TODO (expected failure)
ok 2 - likewise # TODO (unexpected success)
1..2

then that would interoperate with other consumers.

@mblayman
Copy link
Member

@smcv thanks for the report. I made the decision when I started the project that I would strictly stick with the TAP specification to be as consistent as I think is reasonable.

Since unexpected success and failure is not part of the TAP spec for version 12 or 13 (I checked again just now to confirm), then I think my call is to continue to exclude this functionality from tappy.

If this kind of feature is included in future versions of the spec, I'm happy to include it then. You might want to comment on the proposed TAP 14 spec to see if it makes sense to include it. TestAnything/testanything.github.io#36

@smcv
Copy link
Contributor Author

smcv commented May 11, 2018

unexpected success and failure is not part of the TAP spec for version 12 or 13

The spec calls it a "TODO test", but I'm fairly sure it's the same thing as Python @expectedFailure. The phrasing in the v12 spec says:

These tests represent a feature to be implemented or a bug to be fixed and act as something of an executable “things to do” list. They are not expected to succeed.

which sounds a lot like @expectedFailure to me?

@mblayman
Copy link
Member

mblayman commented May 11, 2018

Sorry, maybe I got confused about what you are looking for specifically. My apologies for a premature closure.

I was being very myopic (which may be because I'm reading in spare minutes at PyCon) and thought that you were looking for an expected failure to report as TODO not implemented yet.

I hacked in an expectedFailure on a test and was surprised by the result. I expected to see:

ok 15 - test_category (tap.tests.test_line.TestResult) # TODO (unexpected success)

but instead got

ok 15 - test_category (tap.tests.test_line.TestResult) # (unexpected success)

which is probably what you were trying to point out from the beginning but I was being too slow. I agree with your interpretation of the spec. There should be a TODO in there.

Sorry for the misunderstanding! 😇 P.S. I thought it already did this. Thanks for pointing it out.

@mblayman mblayman reopened this May 11, 2018
mblayman added a commit that referenced this issue May 11, 2018
This aligns better with the specification.

> These tests represent a feature to be implemented or a bug to be fixed
and act as something of an executable “things to do” list. They are not
expected to succeed.

Fixes #74
@smcv
Copy link
Contributor Author

smcv commented May 11, 2018

Sorry, no, the text given in my example was from the Perl equivalent of a hypothetical

@expectedFailureWithDiagnostic("not implemented yet")

in Python (but unfortunately expectedFailure doesn't currently support giving a reason anyway).

You are correct to say that what I wanted was the addition of a # TODO directive.

mblayman added a commit that referenced this issue May 12, 2018
* expectedFailure should be treated as a TODO.

This aligns better with the specification.

> These tests represent a feature to be implemented or a bug to be fixed
and act as something of an executable “things to do” list. They are not
expected to succeed.

Fixes #74

* Update the release docs.

* Try to fix brew.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants