When using --tap option the exit code of alsatian is alway's 0

[winksaville]

Here is the output if --tap is not used and we see the exit code is 1:

$ DEBUG=client.spec alsatian 'build/client/**/*.spec.js'
  client.spec setupFixture:+ +0ms
Client tests

wd: nop which is defined in a TS file
|====================|
  client.spec teardownFixture:+ +44ms
  client.spec teardownFixture:- +119ms

Pass: 0/1
Fail: 1/1
Ignore: 0/1

FAIL: wd: nop which is defined in a TS file
The test threw an unhandled error.
Expected: no unhandled errors to be thrown
  Actual: an unhandled error
wink@wink-envy:~/prgs/web-client-server-app-ts (master)
$ echo $?
1

But if I run it with --tap the exit code is 0

[jamesadarich]

Hey @winksaville,

I believe this is expected behaviour at the moment as there is some discussion around TAP providers and TAP consumers as to who should be determining the process exit code. The discussion at the time of implementing said that TAP providers (e.g. Alsatian in this case) should always output a process status of zero as it is up to the consumer (e.g. a TAP reporter) as to whether or not the process failed.

I've looked at the current version of the TAP spec at the moment and they have a to be confirmed for exit codes so perhaps this is worth revisiting. I believe as it stood at the time that giving an exit code of 1 in the case of failures made some TAP reporters fail. But I think this is worth revisiting.

What is your use case here? Output for Travis CI?

[winksaville]

I'm using --tap so I can see the my debug/log output and on the console and there is no reporter perse. How about --tap1oe which would mean 1 on error and that way it would work either way?

[jamesadarich]

Ok so essentially what you're looking for is the tap output but with an exit code of 1 of any failures occur? Maybe worth knocking up a quick reporter to cover this. Can have a look into this if that's what you're looking for?

[winksaville]

That's it exactly. But does it take creating reporter, or can you just have l Alsatian record if any tests fail and decide what it's exit code should be?

[jamesadarich]

I'd say probably reporter, but i think I'm gonna test popular tap reporters to see how they react and then report back findings here. If many of them are expecting 0 then we'll stick the otherwise switch to 1 on fail by default. I should be able to knock up a quick reporter for you in the meantime though if that would help?

[winksaville]

I'm not in any hurry, you do what you feel is right. Just registering my desire, although I think my request would generally be useful.

…

On Sun, Mar 12, 2017, 3:57 PM James Richford ***@***.***> wrote: I'd say probably reporter, but i think I'm gonna test popular tap reporters to see how they react and then report back findings here. If many of them are expecting 0 then we'll stick the otherwise switch to 1 on fail by default. I should be able to knock up a quick reporter for you in the meantime though if that would help? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#334 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-hHIf1BLV5C3fVdDbUHJ-Dwc3yz8sVks5rlHhQgaJpZM4MaF8l> .

[Jameskmonger]

I asked a similar question on the "official" TAP specification discussion about a year ago:

TestAnything/Specification#22

[winksaville]

Please add a flag so the exit code of Alsatian can be controlled. Another reason is that my debug logs will likely screw up the parser. Although, maybe there is a convention so the TAP output canned be easily separated from other output.

Anyway, I still feel allowing Alsatian exit code to be controlled by me the test author is the best solution. But it is your baby :)

[Jameskmonger]

I think we should maybe investigate the behaviour if both the producer and the parser give an exit code of 1 on error.

[winksaville]

SGTM

[winksaville]

Is a solution for coming?

[jamesadarich]

Hey @winksaville, this is not forgotten just requires a bit more thought as to the affect on TAP reporters. I'll try and get a quick reporter done with the functionality described above :)

[winksaville]

TXS

…

On Fri, Mar 31, 2017, 10:49 AM James Richford ***@***.***> wrote: Hey @winksaville <https://github.com/winksaville>, this is not forgotten just requires a bit more thought as to the affect on TAP reporters. I'll try and get a quick reporter done with the functionality described above :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#334 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-hHPEGxGcd2lgH1BH01NaXmq9moJ8Cks5rrTx8gaJpZM4MaF8l> .

[jamesadarich]

@winksaville I've just created a simple reporter to achieve the above I believe.

npm install tap-fail-exit-one

to use it just pipe as below

alsatian "./tests/**/*.spec.js" --tap | tap-fail-exit-one

Let me know if it fits what you need for now :)

[winksaville]

@jamesrichford for some reason the line endings are crlf instead of just lf and it won't execute unless I use dos2unix:

$ npm install tap-fail-exit-one --save-dev
[email protected] /home/wink/prgs/web-client-server-app-ts
└─┬ [email protected] 
  └── [email protected] 

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
wink@wink-envy:~/prgs/web-client-server-app-ts (master)
$ ./node_modules/.bin/tap-fail-exit-one 
/usr/bin/env: ‘node\r’: No such file or directory

After converting dos2unix ./node_modules/tap-fail-exit-one index.js it worked. The weird thing was I forked your repo and expected to find crlf's in the file but didn't see any. Therefore I'm guessing you're publishing via a windows machine but git is handling line endings "properly" but npm publish doesn't. I see this issue but not sure if there is an "automatic" solution or you need to do something. Acutally, I suspect you already know how to fix this in that Alsatian doesn't have the problem :)

[winksaville]

Actually I'd prefer a switch in Alsatian so I don't have to have additional dependencies and to make it simpler for me, but I'm selfish.

[winksaville]

FYI: found mind-the-end-of-your-line a long read but informative. It seems a .gitattributes file is a good idea but I've never seen one before.

So I just looked and found that DefinitelyTyped does have one:

# Auto detect text files and perform LF normalization
* text=auto

# Custom for Visual Studio
*.cs     diff=csharp
*.sln    merge=union
*.csproj merge=union
*.vbproj merge=union
*.fsproj merge=union
*.dbproj merge=union

# Standard to msysgit
*.doc	 diff=astextplain
*.DOC	 diff=astextplain
*.docx diff=astextplain
*.DOCX diff=astextplain
*.dot  diff=astextplain
*.DOT  diff=astextplain
*.pdf  diff=astextplain
*.PDF	 diff=astextplain
*.rtf	 diff=astextplain
*.RTF	 diff=astextplain

Your thoughts?

[jamesadarich]

@winksaville - the above is not the final solution just a temporary whilst we are discussing with the guys at TAP what the spec should be for this but also whether there is any impact on TAP reporters.

Good catch on the line endings - I can make an update for that it's actually to do with TypeScript compiling where we need to set the line endings.

I think .gitattributes is a good idea maybe not to this degree though as I've never had issues with git detecting file types correctly perhaps this has been solved in later versions of git as the article is from 2012?Definitely good for setting things like line ending normalisation though 👍

[winksaville]

OK, I'll wait impatiently for the update to fix the EOL problem :)

…

On Tue, Apr 4, 2017, 5:52 AM James Richford ***@***.***> wrote: @winksaville <https://github.com/winksaville> - the above is not the final solution just a temporary whilst we are discussing with the guys at TAP what the spec should be for this but also whether there is any impact on TAP reporters. Good catch on the line endings - I can make an update for that it's actually to do with TypeScript compiling where we need to set the line endings. I think .gitattributes is a good idea maybe not to this degree though as I've never had issues with git detecting file types correctly perhaps this has been solved in later versions of git as the article is from 2012?Definitely good for setting things like line ending normalisation though 👍 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#334 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-hHCjSfgRi5UkDT5Yc5Lp-wORW_LEGks5rsjz1gaJpZM4MaF8l> .

[Jameskmonger]

I remember when we first created tap-bark we had an issue with the line endings due to you working on Windows and me on Ubuntu @jamesrichford

[jamesadarich]

@winksaville just published what is hopefully a fix :)

[Jameskmonger]

@jamesrichford Nothing pushed to GitHub?

[winksaville]

@Jameskmonger: It was a change to the tap-fail-exit-one.

@jamesrichford: works perfectly now, thanks.

[winksaville]

status?

[jamesadarich]

@winksaville Progress is blocked currently on the guys at TAP defining the strategy at the moment as we have a workaround for the moment we'll see what they come back with first. I've chased them on the issue mentioned above

[winksaville]

@jamesrichford, ok its just somewhat inconvenient and I seem to have a mental block remembering tap-fail-exit-one.