cucumber-json-formatter (cucumber-gherkin-windows-amd64) not creating proper JSON outcome

[binodtcs]

👓 What did you see?

Cucumber-report.json created by the cucumber-json-formatter for windows is not a valid JSON and also the report looks different from the Json report created in Mac system

✅ What did you expect to see?

Content of the json report for windows and Mac should be same and also json report should be valid

📦 Which tool/library version are you using?

@badeball/cypress-cucumber-preprocessor": "^9.2.1",
@cypress/browserify-preprocessor": "^3.0.2",
"cypress": "latest",
"typescript": "^4.6.4"

package.json file:
"cypress-cucumber-preprocessor": {
"json": {
"enabled": true,
"formatter": "formatter/cucumber-json-formatter.exe",
"output": "reports/cucumber-report.json"
}
}

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install cypress latest
2. Install cypress-cucumber package and add index.ts with below content:
   import * as browserify from "@cypress/browserify-preprocessor";
   import { preprocessor } from "@badeball/cypress-cucumber-preprocessor/browserify";
   import { addCucumberPreprocessorPlugin } from "@badeball/cypress-cucumber-preprocessor";

export default async(
on: Cypress.PluginEvents,
config: Cypress.PluginConfigOptions
): Promise<Cypress.PluginConfigOptions> => {
await addCucumberPreprocessorPlugin(on, config);
on(
"file:preprocessor",
preprocessor(config, {
...browserify.defaultOptions,
typescript: require.resolve("typescript"),
})
);

return config;
};

4. Create a feature & spec file

5. Create a formatter folder at root and place the latest version of cucumber-json-formatter file from https://github.com/cucumber/common/releases for windows 64 and keep without .exe extension as mentioned in the link https://github.com/badeball/cypress-cucumber-preprocessor/blob/master/docs/json-report.md

6. Added below section in package.json file:
   "cypress-cucumber-preprocessor": {
   "json": {
   "enabled": true,
   "formatter": "formatter/cucumber-json-formatter.exe",
   "output": "reports/cucumber-report.json"
   }
   }

7. Run the npx cypress run to execute the test and create the report (cucumber-report.json)

8. Content of the cucumber-report.json generated in Mac and Windows should be same.

Please let me know for any additional information.

[aslakhellesoy]

Please provide an NDJSON input file that reproduces the incorrect JSON.

[binodtcs]

Please provide an NDJSON input file that reproduces the incorrect JSON.
Please find attached the ndjson file
cucumber-messages.zip

[oguzhantx]

any update on this, having same issue.

[binodtcs]

Please refer https://www.npmjs.com/package/cucumber-json-report-formatter

[youvegotnigel]

I'm also facing this issue. any update?

[aslakhellesoy]

@mattwynne can you remind me whether we decided to stop maintaining this tool? Personally I don’t see the need for it anymore since we decided to keep a built-in JSON formatter in all our Cucumber implementations.

I don’t understand why this is needed with Cypress. If Cypress runs Cucumber, can’t it just tell Cucumber to output JSON directly?

[badeball]

I don’t understand why this is needed with Cypress. If Cypress runs Cucumber, can’t it just tell Cucumber to output JSON directly?

Hi again, Aslak

I'm the author of https://github.com/badeball/cypress-cucumber-preprocessor, which is responsible for the (sudden) surge of users for this tool. Let me try to answer your question. cypress-cucumber-preprocessor is a plugin for Cypress, a testing framework, that enables behavior which mimics Cucumber.

Here's the thing, Cypress itself is a test runner, with a complete runtime and not something you can put inside of Cucumber like you can with EG. Selenium. The plugin will translate Gherkin into calls to describe() and it(), which you might know from EG. Mocha. These methods are provided by Cypress' runtime and is the only structure it recognize.

So the plugin doesn't actually "run Cucumber", but rather re-implements it (largely, but makes use of several of your modules), despite many referring to it as "Cypress + Cucumber". It's essentially just giving familiar behavior to a tool that's not actually even remotely compatible with Cucumber.

As to messages, JSON reports and this.. I developed outputting messages instead of JSON as it was significantly easier, with messages requiring little to no state in the implementation and being definitely typed. Thus appeared the sudden surge of users to https://github.com/cucumber/json-formatter.

[badeball]

So the plugin doesn't actually "run Cucumber", but rather re-implements it (largely, but makes use of several of your modules), despite many referring to it as "Cypress + Cucumber". It's essentially just giving familiar behavior to a tool that's not actually even remotely compatible with Cucumber.

Expanding to this...

The plugin is popular with the Cypress community. The typical user however, I suspect is fully unaware of the absence of relationship between it and Cucumber and you. The fact that it's re-implementing behavior (with feature identically being the target) remains unknown to most, I'm afraid. I know that the terminology is used carelessly in blogs, articles, guides, videos, etc.

I'm sorry that you got sucked in the Cypress vortex here. It's was never my intention. Feel totally free to close indiscriminately the issues where Cypress is even mentioned.

[aslakhellesoy]

Ohhhhh I see! Thanks for explaining @badeball. I think it's really cool what you've done here.

I'm guessing that people want Cucumber JSON output because this format is more widely supported in 3rd party reporting tools.

I'm not sure how to best accommodate that need.
One strategy could be to implement the JSON formatter natively in your library. That might be easier if the JSON formatter in Cucumber.js could be reused, but I think it might be too tightly coupled to the internals of Cucumber.js. It might have been easier if it just consumed cucumber messages and produced JSON (like the go library does).

Another alternative is to keep maintaining this library. Since it's only used by users of your cypress plugin, I think it might be better to hand it over to that community.

[badeball]

I'm guessing that people want Cucumber JSON output because this format is more widely supported in 3rd party reporting tools.

Yeah, exactly.

I'm fairly certain that I'm going to have to just suck it up a little bit and implement it natively myself. I benefited a lot by the existence of this though, despite no ones intention of maintaining it, it let me focus on other things for a while and I'm grateful for that.

[aslakhellesoy]

@davidjgoss has done a lot of refactorings in cucumber-js lately to make it more usable as a library. I’m sure additional work would be needed to make it work in this extension, but maybe it’s worth pursuing still?

Fancy scheduling a call to discuss this?

I’d love to see 1st class support for Cucumber in Cypress in a way that is supported by the official Cucumber-js implementation.

[davidjgoss]

One of the initiatives we have going in cucumber-js is the reimagining of formatters as a simpler kind of plug-in focused on messages. I think we could reimplement the existing JSON formatter in this vein such that it could be used as a function by other projects I.e. just give it a message stream.

Happy to have a call!

[badeball]

I would also be happy to have a call! What TZ are you two in and what time would suit you the best?

[binodtcs]

I have used the below package to publish the cucumber report https://www.npmjs.com/package/zephyr-scale-results-publisher

[elchupanebrej]

This formatter could be a good integration point:
There I no well-defined json-schema for json reports - but this is a place, where it could be actually defined. Old tools could stick to it without huge refactorings(as it is needed for porting to messages).
This gives space and time to migrate tools to messages.

[davidjgoss]

Sorry for the late reply @badeball - to get this scheduled, would you be able to drop into the Cucumber Slack and DM us?

[badeball]

Hey @davidjgoss, sorry myself for late replying. I'm super-busy nowadays with exams, but I'll message you there when I feel it's time to re-visit this issue on my end :)

[aslakhellesoy]

There I no well-defined json-schema for json reports - but this is a place, where it could be actually defined.

Actually, there is. Sort of. See https://github.com/cucumber/cucumber-json-converter

We’re using this library in a commercial product to process the wildly inconsistent Cucumber JSON documents.

[mpkorstanje]

Actually, there is. Sort of. See https://github.com/cucumber/cucumber-json-converter

I do not see how a tool that can consume many disparate formats is any help for producing a correct document.

Perhaps you are saying that every one of these formats is correct but the ramifications of that aren't trivial. Could you clarify what you mean?

[aslakhellesoy]

What I mean is that one of the schemas in this repo could be elected the official one. Then all the implementations could be updated to conform to that schema.

I don’t think it is possible to come up with a single schema that is conformant to what current Cucumbers currently output. They are too inconsistent.

[mpkorstanje]

I suspect we'll run into the same problems we had when we tried to deprecate the format. There is a lot of inertia in the ecosystem.

Though those problems should be less if we stay in the same ecosystem. So we could atleast share the implementation between cucumber-js and cypress.