Client unit tests not working on Meteor3 with beta

[ToyboxZach]

On the beta
#150

Describe the bug
The client tests never run

I get this log and then it just stops:

I20240115-09:07:57.916(-8)? --------------------------------
I20240115-09:07:57.916(-8)? ----- RUNNING CLIENT TESTS -----
I20240115-09:07:57.916(-8)? --------------------------------
=> Started your app.

=> App running at: http://localhost:3050/
I20240115-09:07:59.999(-8)? HeadlessChrome/106.0.5249.0
I20240115-09:08:00.993(-8)? HMR: connected

To Reproduce

in client.test.ts

describe("BASE", () => {
  it("can rotate x axis", () => {
    console.log("RUN TEST");
  });
});

Run the command:
TEST_SERVER=0 CLIENT_TEST_REPORTER='min' TEST_BROWSER_DRIVER=puppeteer meteor test --driver-package meteortesting:mocha --port 3050 --once

Expected behavior
It runs my test

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• MacOs
• Puppeteer (I tried other browsers and they didn't work, but puppeteer is what I had 2.0 working on)
• Beta

Additional context
My tests were working just fine in 2.0 but the move to 3.0 fails. My app roughly works when not testing, but unit tests fail like this, and functional tests have other issues, that I am not confident are related or not

This is my package list if it helps (on Meteor 3.beta)

[email protected]             # Packages every Meteor app needs to have
[email protected]                   # The database Meteor supports right now
[email protected]                 # Meteor's client-side reactive programming library
cultofcoders:redis-oplog
fortawesome:fontawesome
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
tmeasday:acceptance-test-driver
react-meteor-data
aldeed:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
meteortesting:[email protected]
quave:migrations

[email protected]
http
[email protected]
[email protected]
activitree:scss3

[github-actions]

Thank you for submitting this issue!

We, the Members of Meteor Community Packages take every issue seriously.
Our goal is to provide long-term lifecycles for packages and keep up
with the newest changes in Meteor and the overall NodeJs/JavaScript ecosystem.

However, we contribute to these packages mostly in our free time.
Therefore, we can't guarantee your issues to be solved within certain time.

If you think this issue is trivial to solve, don't hesitate to submit
a pull request, too! We will accompany you in the process with reviews and hints
on how to get development set up.

Please also consider sponsoring the maintainers of the package.
If you don't know who is currently maintaining this package, just leave a comment
and we'll let you know

[ToyboxZach]

Oh I think it was some weird setup I have.

In order to use other tools I have a .mocharc.js

let args = process.env.HEADLESS ? ["--headless", "--disable-gpu"] : [];
let grep = process.env.HEADLESS ? "^((?!@circleciignore).)*$" : "";

module.exports = {
  ui: "bdd",
  require: ["@babel/register"],
  grep: grep,
  spec: ["tests/specs/*.spec.js"],
  timeout: 30000,
  fullStackTrace: true,

  fullTrace: true,
  recursive: true,
};

I believe in previous versions this was completely ignored by this use case, but it existing causes problems in this beta

[jankapunkt]

Which puppeteer version do you have installed?

[ToyboxZach]

"puppeteer": "^17.1.3",

[jankapunkt]

Can you please run one test for me without puppeteer but actually in the browser (the default without any browser driver) and paste the console output here, please?

The problem is that the puppeteer process does not pipe the errors back so we can't see the root cause of this. In browser it will show up

[ToyboxZach]

This is the only error I get. I don't know what this error is, but I have been getting it forever without issue before. SO I guess it could be related to error handling of this

meteortesting_mocha-core.js?hash=c76426b69ff65052a24efd005756c3697c5b13a4:20659 Uncaught TypeError: self[opt] is not a function
    at meteortesting_mocha-core.js?hash=c76426b69ff65052a24efd005756c3697c5b13a4:20659:20
    at Array.forEach (<anonymous>)
    at mocha.setup (meteortesting_mocha-core.js?hash=c76426b69ff65052a24efd005756c3697c5b13a4:20657:8)
    at module (client.js:19:7)
    at fileEvaluate (modules-runtime-hot.js?hash=82c21ce5103770e3ed6096f6e6eee4aeb50ee752:386:7)
    at Module.require (modules-runtime-hot.js?hash=82c21ce5103770e3ed6096f6e6eee4aeb50ee752:268:27)
    at mod.require (modules.js?hash=a43c70045df2a904a400cb5e2631082dca285d6a:305:33)
    at Object.require (modules-runtime-hot.js?hash=82c21ce5103770e3ed6096f6e6eee4aeb50ee752:308:21)
    at evaluateNextModule (core-runtime.js?hash=f962403d0ec4acf462876ccc7e2e785c6ce6c700:193:26)
    at runEagerModules (core-runtime.js?hash=f962403d0ec4acf462876ccc7e2e785c6ce6c700:232:3)

[ToyboxZach]

Some further investigation definitely shows that its the .mocharc.js file that causes that erorr and that causes the tests not to load.

This leads to two things.

1. Why does this error even throw?
   I can get it not to throw with a simplified .mocharc.js

module.exports = {
  timeout: 30000,
};

But can't get all the options I had back in even trying to mimic this format https://github.com/mochajs/mocha/blob/master/example/config/.mocharc.js

2. Why does this error actually prevent the tests from running? Unlike in v2 that just let it happen.

[jankapunkt]

Thanks for getting closer to the root cause. I will check on the RC file but I assume there is some config issue since we updated mocha from V8 to V10 and maybe this included breaking changes.

[ToyboxZach]

Yeah wouldn't be surprised if it was related to a mocha update.

But I will reiterate, that I always got that error on Meteor2, so its more about how it reacts to that error. I'm guessing its an unhandled promise error vs an error that was just caught and ignored at some level.

[jankapunkt]

Can you narrow it down, to which property of the rc file this actually roots? By looking at the Mocha changelog the closest candidates are require options (they went esm first and dropped amd/require) and ui: 'bdd' which also had some breaking changes on how things are imported.

My first assumption here is, something is not imported right internally, causing it to be undefined then causing the error above.

[jankapunkt]

It seems that programmatic spawn of mocha has a limited set of options: https://mochajs.org/api/mocha
Can you confirm that stripping all options from your code, that are not in this list will make it work?

Furthermore: do your tests still work in general, once these options are corrected?

[ToyboxZach]

That resource helped. "require" also broke it even though that list says it whould work but I am able to get it to run with these options:

let args = process.env.HEADLESS ? ["--headless", "--disable-gpu"] : [];
let grep = process.env.HEADLESS ? "^((?!@circleciignore).)*$" : "";

module.exports = {
  ui: "bdd",
  grep: grep,
  timeout: 30000,
  fullTrace: true,
};

Being limited to these options though does make it more annoying to run tests in vscode which is what we had originally created this file to accomplish, it would be nice if this got back to the state it was before, but understand if this is minor enough to not warrant it.

[jankapunkt]

@ToyboxZach can you give me some resources on how you integrate it with vscode? I use IntelliJ so I have not so much knowledge on it. Maybe it's easy to integrate on package level before passing it to mocha

[ToyboxZach]

It has actually been a while since I have tried it because we figured out some other command line flows that just worked better for me so I don't have the full integration going but if I remember correctly we used this extension

https://github.com/compulim/vscode-mocha

And then the key was actually including these values in the mocharc
require: ["@babel/register"],
spec: ["tests/specs/*.spec.js"],

So that it knew correctly to babel compile it, and where to find the specs.

Essentially it just ran a mocha instance in vscode and we just had to figure out how to make sure it compiled and loaded stuff correctly, so it was pretty much copying what this package was doing, but with a third part tool.

Thats why this change makes it a bit more annoying, as we can't share the mocharc between the two use cases, and if I remember correctly it was either not possible, or cumbersome to actually pass these options via the command line that vscode worked off of.

But we don't use it anymore, so at the moment its not an issue for me.

[jankapunkt]

Let me see if we can use Meteor to make the require option somehow. From my end this may be important as well as it's a part of integration of Meteor into the broader JS ecosystem.

For many devs VScode integration is a must so we should look for being compatible, too

[ToyboxZach]

Just an update about this.

I seem to be running into a similar problem if I update puppeteer to the latest version.

I'm going to dive a bit deeper, but there is definitely some lack of error being propagated that is making it hard to figure out.

[jankapunkt]

Yes puppeteer had breaking changes which is why we stick to 18 something in our test projects. That will work for sure but is no long term solution.