[Chromatic] Disable snapshots globally, only keep regression stories #1911
Conversation
Right now, Chromatic snapshots are only supposed to run on files that end with `.regression.stories.tsx`, but they seem to be running on other stories as well in the github checks. Instead of using the .yml to specify this, I'm disabling chromatic using the `disableSnapshot` chormatic parameter globablly, and manually enabling it in the one regression stories file we have. Issue: https://khanacademy.atlassian.net/browse/LEMS-2579 Test plan: Check the chromatic info in the github checks below.
npm Snapshot: PublishedGood news!! We've packaged up the latest commit from this PR (08ee321) and published it to npm. You Example: yarn add @khanacademy/perseus@PR1911If you are working in Khan Academy's webapp, you can run: ./dev/tools/bump_perseus_version.sh -t PR1911 |
|
Size Change: 0 B Total Size: 1.29 MB ℹ️ View Unchanged
|
GeraldRequired Reviewers
Don't want to be involved in this pull request? Comment |
|
The new "test" number on the dashboard matches the expected snapshot number in the checks! I think this means it's working. 
|
| */} | ||
| <div className="framework-perseus"> | ||
| <Story /> | ||
| </div> |
There was a problem hiding this comment.
NOTE: This is causing some trivial snapshot updates within the UI tests for the regression stories.
| @@ -43,4 +43,3 @@ jobs: | |||
| token: ${{ secrets.GITHUB_TOKEN }} | |||
| projectToken: ${{ secrets.CHROMATIC_APP_CODE }} | |||
| autoAcceptChanges: "main" | |||
| onlyStoryFiles: "**/*regression.stories.tsx" | |||
There was a problem hiding this comment.
So I think the issue is that onlyStoryFiles is an option inside the chromatic.config.json file, the option is here: https://www.chromatic.com/docs/configure/#options
There was a problem hiding this comment.
Ah, that's good to know!
I think we're gonna enable a few key stories other than the regression stories soon, so I think it would still be worth removing this from the .yml file.
There was a problem hiding this comment.
It seems like this is working as designed. This setting is documented as being supported in the GitHub Action:
On an example PR I checked, it seems like it is obeying the setting:
19:50:09.030 Starting partial build
19:50:09.030 → Snapshots will be limited to story files matching '**/*regression.stories.tsx'
...
19:50:58.649 Build 5134 passed!
→ Tested 412 stories across 130 components; captured 27 snapshots in 39 seconds
It captured only 27 snapshots.
There was a problem hiding this comment.
@jeremywiebe It does say "captured 27 snapshots" on all the PRs. I checked a bunch of older unrelated Perseus PRs to check this, and it's true that that messaging is consistent. However, the PRs would also show other snapshots that aren't part of these 27 in the "UI test" check. It also says there are 130 tests on the chromatic dashboard for other PRs when there should only be 27:
I'd like to move away from using the onlyStoryFiles: "**/*regression.stories.tsx" because I think there's a disconnect somewhere that isn't super obvious.
There was a problem hiding this comment.
I'm going to retract my last comment. Just for a confidence check, I put up another PR that would have had a ton of UI test updates if my assertion was correct. It only reported changes in the regression stories, so I think my statements before this were all false, and I just don't know what "tests" means on the chromatic dashboard.
There was a problem hiding this comment.
@jeremywiebe we can still continue with this PR if we want to enable other stories to have snapshots later, rather than just limiting to regression stories.
|
It turns out that the |
Summary:
Right now, Chromatic snapshots are only supposed to run on files
that end with
.regression.stories.tsx, but they seem to be runningon other stories as well in the github checks.
Instead of using the .yml to specify this, I'm disabling chromatic
using the
disableSnapshotchromatic parameter globally, andmanually enabling it in the one regression stories file we have.
Issue: https://khanacademy.atlassian.net/browse/LEMS-2579
Test plan:
Check the chromatic info in the github checks below.