[investigate] Replace Puppeteer with Playwright (Vitest Browser Mode?)

[birkskyum]

Ongoing work on improving the test suite.

Initially we had Selenium in Query / Browser suite, and custom mocks of webgl1 in Render suite.

February 2022 - Browser, Query: Selenium to Playwright

• Move browser test to playwright #974
• Migrate query tests from puppeteer to playwright #966

May 8th 2023 - Browser, Query: Playwright to Puppeteer

Google chrome made a splash. Until this point headless and headful chrome were two different browsers, and now they cleaned it up so the headless became a "real chrome", which is a lot better. Only puppeteer supported this initially, so we switched to be able to leverage it in (query + browser):

• Replace playwright with puppeteer #2494

May 9th 2023 - Render: Custom Webgl1 Mocks to Puppeteer:

• Render tests - Replace mocks with Puppeteer/Chrome #2497

Now - Browser, Query, Render: Puppeteer to Playwright

This is now old history, and since, playwright has due to a nicer dx, and multi-browser support (firefox / webkit / chromium), managed to become the No. 1 browser runner, and it's possible to it thoughout.

As an example of the better DX, this is i.e. a diff from browser.test.ts:

Puppeteer:

await page.emulateMediaFeatures([
    {name: 'prefers-reduced-motion', value: 'reduce'},
]);

Playwright:

await page.emulateMedia({reducedMotion: 'reduce'});

Rendet test HTML reporting works - here a full run on a mac.

Launch Checklist

• Confirm your changes do not include backports from Mapbox projects (unless with compliant license) - if you are not sure about this, please ask!

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.81%. Comparing base (d3f2bca) to head (38cc518).
Report is 4 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (d3f2bca) and HEAD (38cc518). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (d3f2bca)	HEAD (38cc518)
	9	8

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #4949       +/-   ##
===========================================
- Coverage   90.48%   71.81%   -18.68%     
===========================================
  Files         265      265               
  Lines       38113    38113               
  Branches     3166     2607      -559     
===========================================
- Hits        34488    27372     -7116     
- Misses       2650     9490     +6840     
- Partials      975     1251      +276     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[HarelM]

Looks like a good direction, let's make sure the coverage is working as expected as I had to fight with it back then...

[birkskyum]

@HarelM , how to I check the coverage? is it specific to the render test suite?

[HarelM]

If this is only a refactoring, the coverage should stay the same between main branch and this one.
You can open an empty PR and compare the report in codecov, probably selectively.

[birkskyum]

It seems like the render tests take 2.5x longer to run with the current state of this PR (26 min. vs ~ 11 min), so that has to be reduced significantly before this PR can be merged.

[HarelM]

Yes, that's a concern...

[birkskyum]

The performance overhead appear to come from improved isolation / cross-browser architecture with some abstractions layers - it just is a bit steep price. I'll revisit this when the vitest browser mode is stable, as it might bring forward some options for performance improvements:

• https://vitest.dev/guide/browser/

[birkskyum]

The test suite for Integration / Unit is now Vitest.

That means we can explore Vitest Browser Mode for render tests - if it's helping on the perf issues with playwright:

• https://vitest.dev/guide/browser/