Core + Addon Test: Refactor test API and fix total test count

[ghengeveld]

Closes #29541

What I did

• TESTING_MODULE_RUN_ALL_REQUEST is removed in favor of TESTING_MODULE_RUN_REQUEST. So we have a single event to run tests now.
• The payload to TESTING_MODULE_RUN_REQUEST is greatly simplified:

Before:

export type TestingModuleRunRequestPayload = {
  providerId: TestProviderId;
  payload: {
    stories: { id: string; name: string }[];
    importPath: string;
    componentPath: string;
  }[];
};

After:

export type TestingModuleRunRequestPayload = {
  providerId: TestProviderId;
  indexUrl: string; // e.g. http://localhost:6006/index.json
  storyIds?: string[]; // ['button--primary', 'button--secondary']
};

• The VitestManager now fetches the indexUrl to be able to match Vitest test specs (which only have a file path) to the storyIds provided. This way we make sure that only tests are executed for stories that exist in the index (which may otherwise not be true considering Vitest uses a different glob and globbing library than Storybook does).
• testNamePattern is now only set for single story runs, rather than whenever stories are specified in the request, because this testNamePattern trick will only reliably work when targeting a single test file, and we don't / won't have a UI pattern to run a subset of stories on a single component even though the API would support it. The story name is also retrieved from the index rather than passed along in the request payload.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	78.4 MB	78.4 MB	0 B	0.19	0%
initSize	144 MB	144 MB	193 kB	1.5	0.1%
diffSize	65.1 MB	65.3 MB	193 kB	4.09	0.3%
buildSize	6.83 MB	7.03 MB	198 kB	4.11	2.8%
buildSbAddonsSize	1.51 MB	1.51 MB	-211 B	-Infinity	0%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.86 MB	2.06 MB	198 kB	4.11	🔺9.6%
buildSbPreviewSize	271 kB	271 kB	-147 B	-Infinity	-0.1%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.83 MB	4.03 MB	198 kB	4.1	4.9%
buildPreviewSize	3 MB	3 MB	-39 B	0.95	0%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	16.4s	23.2s	6.7s	0.93	29.2%
generateTime	21.8s	19.4s	-2s -454ms	-0.59	-12.6%
initTime	15.3s	13.1s	-2s -232ms	-1.11	-17%
buildTime	7.6s	8.6s	1s	-0.21	11.6%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	7s	5.2s	-1s -726ms	-0.67	-32.6%
devManagerResponsive	4.1s	3.4s	-705ms	-0.26	-20.3%
devManagerHeaderVisible	631ms	550ms	-81ms	-0.13	-14.7%
devManagerIndexVisible	668ms	627ms	-41ms	-0.04	-6.5%
devStoryVisibleUncached	1.3s	895ms	-468ms	-0.57	-52.3%
devStoryVisible	666ms	583ms	-83ms	-0.41	-14.2%
devAutodocsVisible	592ms	486ms	-106ms	-0.39	-21.8%
devMDXVisible	600ms	552ms	-48ms	0.85	-8.7%
buildManagerHeaderVisible	491ms	525ms	34ms	-0.47	6.5%
buildManagerIndexVisible	502ms	544ms	42ms	-0.43	7.7%
buildStoryVisible	485ms	523ms	38ms	-0.47	7.3%
buildAutodocsVisible	403ms	463ms	60ms	-0.22	13%
buildMDXVisible	464ms	454ms	-10ms	0.21	-2.2%

Greptile Summary

This PR refactors Storybook's test API to simplify test execution and improve test progress reporting accuracy.

• Consolidates test execution into single TESTING_MODULE_RUN_REQUEST event, removing separate "run all" event
• Simplifies test request payload to use indexUrl and optional storyIds instead of detailed story information
• Improves test progress accuracy by using static story count from index rather than Vitest's dynamic test count
• Adds server-side story index fetching to ensure tests only run for existing stories
• Implements new context menu system for test providers with improved status handling and accessibility

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit d645bb5. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 1 target

• nx run-many -t build --parallel=3

---

Sent with 💌 from NxCloud.

[storybook-pr-benchmarking]

Package Benchmarks

^{Commit: d645bb5, ran on 22 November 2024 at 08:58:14 UTC}

No significant changes detected, all good. 👏

[valentinpalkovic]

The Vite Backend part LGTM.

[greptile-apps]

_{41 file(s) reviewed, 25 comment(s)
Edit PR Review Bot Settings | Greptile}

[JReinhold]

Based on the diff I see, I think you need to pull in next

[JReinhold]

Apart from the failing skip test, I think it looks good.

[JReinhold]

based on the failing test, I don't think skip should be part of isValidTest

[ndelangen]

@ghengeveld can we get this in a green state please, it's my base-branch for this work:
#29662