test(playwright-ct): fix issues and flake #8016

skogsmaskin · 2024-12-11T19:32:20Z

Description

I have identified several issues that caused the playwright-ct tests to be flaky.

Some were real issues, some were with the library itself, and some were issues with the tests (see individual commits for details).

I have upgraded the Playwright packages used in the mono-repo.

There might still be flake, but this should help at least.

This PR is also tied into #8021 and #8022 which was moved into own separate PRs

What to review

That the tests pass and have become less flaky.

Testing

Should be tested automatically.

Notes for release

N/A

vercel · 2024-12-11T19:32:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
page-building-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2024 11:39am
performance-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2024 11:39am
test-next-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2024 11:39am
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2024 11:39am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2024 11:39am

github-actions · 2024-12-11T19:36:45Z

No changes to documentation

socket-security · 2024-12-11T19:39:12Z

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@playwright/[email protected]	Transitive: environment, filesystem	`+3`	11.2 MB	yurys
npm/@playwright/[email protected]	None	`+2`	11.1 MB	dgozman-ms, mxschmitt, pavelfeldman, ...1 more

🚮 Removed packages: npm/@playwright/[email protected], npm/@playwright/[email protected]

View full report↗︎

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx

github-actions · 2024-12-11T19:44:40Z

⚡️ Editor Performance Report

Updated Thu, 12 Dec 2024 11:37:46 GMT

Benchmark	reference ^{latency of sanity@latest}	experiment ^{latency of this branch}	Δ (%) ^{latency difference}
article (title)	25.6 efps (39ms)	25.0 efps (40ms)	+1ms (+2.6%)	✅
article (body)	67.6 efps (15ms)	71.9 efps (14ms)	-1ms (-/-%)	✅
article (string inside object)	27.8 efps (36ms)	26.3 efps (38ms)	+2ms (+5.6%)	✅
article (string inside array)	23.8 efps (42ms)	24.1 efps (42ms)	-1ms (-1.2%)	✅
recipe (name)	50.0 efps (20ms)	50.0 efps (20ms)	+0ms (-/-%)	✅
recipe (description)	58.8 efps (17ms)	58.8 efps (17ms)	+0ms (-/-%)	✅
recipe (instructions)	99.9+ efps (5ms)	99.9+ efps (5ms)	+0ms (-/-%)	✅
synthetic (title)	19.2 efps (52ms)	19.6 efps (51ms)	-1ms (-1.9%)	✅
synthetic (string inside object)	19.0 efps (53ms)	18.9 efps (53ms)	+1ms (+1.0%)	✅

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	39ms	43ms	49ms	174ms	154ms	10.0s
article (body)	15ms	18ms	40ms	325ms	370ms	6.0s
article (string inside object)	36ms	39ms	48ms	210ms	138ms	6.7s
article (string inside array)	42ms	45ms	52ms	167ms	142ms	7.0s
recipe (name)	20ms	23ms	28ms	39ms	0ms	6.9s
recipe (description)	17ms	18ms	20ms	27ms	0ms	4.4s
recipe (instructions)	5ms	7ms	8ms	18ms	0ms	3.1s
synthetic (title)	52ms	55ms	64ms	270ms	598ms	13.0s
synthetic (string inside object)	53ms	55ms	65ms	234ms	476ms	8.0s

🧪 Experiment result

The performance result of this branch

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	40ms	42ms	46ms	181ms	426ms	10.4s
article (body)	14ms	16ms	18ms	74ms	67ms	4.8s
article (string inside object)	38ms	40ms	44ms	209ms	242ms	6.7s
article (string inside array)	42ms	44ms	49ms	253ms	301ms	7.2s
recipe (name)	20ms	22ms	24ms	53ms	1ms	7.1s
recipe (description)	17ms	19ms	21ms	29ms	0ms	4.5s
recipe (instructions)	5ms	7ms	9ms	23ms	0ms	3.0s
synthetic (title)	51ms	53ms	55ms	79ms	109ms	13.0s
synthetic (string inside object)	53ms	58ms	90ms	379ms	903ms	8.7s

📚 Glossary

column definitions

benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".

latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.

p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.

p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.

p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.

blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.

test duration — how long the test run took to complete.

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx

bjoerge · 2024-12-12T10:31:00Z

packages/sanity/package.json

@@ -266,8 +266,8 @@
    "yargs": "^17.3.0"
  },
  "devDependencies": {
-    "@playwright/experimental-ct-react": "1.44.1",
-    "@playwright/test": "1.44.1",
+    "@playwright/experimental-ct-react": "1.49.1",


wonder when it will no longer be experimental 🤔

packages/sanity/playwright-ct.config.ts

Upgrade @playwright/experimental-ct-react from 1.44.1 to 1.49.1 Upgrade @playwright/test from 1.49.1 to 1.49.1

…y being used for chromium/darwin

github-actions · 2024-12-12T11:00:19Z

Component Testing Report Updated Dec 12, 2024 11:37 AM (UTC)

❌ Failed Tests (5) -- expand for details

File	Status	Duration	Passed	Skipped	Failed
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	1m 9s	15	0	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	14s	3	0	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	40s	6	0	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	56s	11	7	0
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx	✅ Passed (Inspect)	0s	0	12	0
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	28s	6	0	0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx	✅ Passed (Inspect)	15s	3	0	0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx	❌ Failed (Inspect)	3m 47s	1	0	5
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	1m 11s	15	0	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	2m 52s	21	0	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 49s	18	0	0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx	✅ Passed (Inspect)	14s	3	9	0
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx	✅ Passed (Inspect)	42s	9	0	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	28s	6	0	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	1m 52s	21	0	0
formBuilder/tree-editing/TreeEditing.spec.tsx	✅ Passed (Inspect)	0s	0	3	0
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx	✅ Passed (Inspect)	0s	0	3	0

This test is now likely to work as expected. Remove exception to skip Chromium

skogsmaskin · 2024-12-12T11:33:46Z

packages/sanity/playwright-ct/tests/formBuilder/inputs/PortableText/Decorators.spec.tsx

@@ -44,12 +44,6 @@ test.describe('Portable Text Input', () => {
      page,
      browserName,
    }) => {
-      // avoid flakiness to make sure the test has the best chance despite being slow
-      test.slow()


Increased timeouts will not combat the flake here.

bjoerge

This is great, thank you so much!

* next: (49 commits) fix: delays rendering the Start in Create banner until document is ready (#8020) fix(deps): update dependency @sanity/ui to ^2.10.9 (#8009) chore(deps): dedupe pnpm-lock.yaml (#8023) test(playwright-ct): fix issues and flake (#8016) feat(cli): misc copy changes (#8003) feat(cli): remove is-builtin-module (#6579) v3.67.1 chore(deps): bump minimum requirement of @sanity/import and @sanity/export (#8012) chore(deps): update dependency @sanity/pkg-utils to v6.12.0 (#8010) fix(core): tasks UpdatedTimeAgo should be a hook (#8011) v3.67.0 chore(prettier): fix unformatted files 🤖 ✨ (#8006) chore: reduce renovate double PR noise fix(deps): update dependency @sanity/ui to ^2.10.7 (#8005) fix(deps): update dependency @sanity/ui to ^2.10.7 (#7998) feat(cli): use `@vercel/frameworks` in `bootstrapRemoteTemplate` (#8001) feat: switch create integration to opt-out flow (#8002) feat: use eslint 9 for new studios (#7978) fix(sanity): optimise getLeafWeights to not stack overflow (#7999) feat(test-studio): enable `groq2024` search strategy ...

skogsmaskin force-pushed the fix/playwright-ct-test branch from 8346a5f to 0c4024a Compare December 11, 2024 19:33

skogsmaskin commented Dec 11, 2024

View reviewed changes

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx Outdated Show resolved Hide resolved

vercel bot deployed to Preview – page-building-studio December 11, 2024 19:43 View deployment

vercel bot deployed to Preview – performance-studio December 11, 2024 19:43 View deployment

skogsmaskin force-pushed the fix/playwright-ct-test branch from 0c4024a to b365756 Compare December 11, 2024 19:44

vercel bot deployed to Preview – page-building-studio December 11, 2024 19:50 View deployment

vercel bot deployed to Preview – performance-studio December 11, 2024 19:52 View deployment

vercel bot deployed to Preview – test-studio December 11, 2024 19:52 View deployment

skogsmaskin commented Dec 11, 2024

View reviewed changes

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx Outdated Show resolved Hide resolved

vercel bot deployed to Preview – test-next-studio December 11, 2024 20:02 View deployment

skogsmaskin commented Dec 11, 2024

View reviewed changes

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx Outdated Show resolved Hide resolved

skogsmaskin commented Dec 11, 2024

View reviewed changes

packages/sanity/src/core/form/inputs/PortableText/object/modals/PopoverModal.tsx Outdated Show resolved Hide resolved

vercel bot deployed to Preview – test-studio December 11, 2024 21:45 View deployment

vercel bot deployed to Preview – page-building-studio December 11, 2024 21:45 View deployment

vercel bot deployed to Preview – performance-studio December 11, 2024 21:45 View deployment

skogsmaskin force-pushed the fix/playwright-ct-test branch from 042ba35 to 8cac812 Compare December 11, 2024 21:52

vercel bot deployed to Preview – performance-studio December 11, 2024 21:58 View deployment

vercel bot deployed to Preview – page-building-studio December 11, 2024 21:59 View deployment

vercel bot deployed to Preview – test-studio December 11, 2024 21:59 View deployment

vercel bot deployed to Preview – test-next-studio December 11, 2024 22:10 View deployment

vercel bot deployed to Preview – page-building-studio December 11, 2024 22:21 View deployment

vercel bot deployed to Preview – performance-studio December 11, 2024 22:21 View deployment

vercel bot deployed to Preview – test-studio December 11, 2024 22:21 View deployment

skogsmaskin changed the title ~~test(playwright-ct): fix issues~~ test(playwright-ct): fix issues and flake Dec 11, 2024

vercel bot deployed to Preview – test-next-studio December 11, 2024 22:30 View deployment

sanity-io deleted a comment from github-actions bot Dec 11, 2024

skogsmaskin requested review from bjoerge and removed request for a team December 11, 2024 22:37

skogsmaskin mentioned this pull request Dec 12, 2024

fix(core/form/inputs): fix issue with tabbing to popover toolbar buttons in PT-input #5057

Open

bjoerge reviewed Dec 12, 2024

View reviewed changes

skogsmaskin added 2 commits December 12, 2024 11:35

chore: upgrade playwright packages to latest

a15568c

Upgrade @playwright/experimental-ct-react from 1.44.1 to 1.49.1 Upgrade @playwright/test from 1.49.1 to 1.49.1

test(playwright-ct): workaround for issue regarding incorrect Meta-ke…

b67482f

…y being used for chromium/darwin

skogsmaskin force-pushed the fix/playwright-ct-test branch from 663f8bf to e941fce Compare December 12, 2024 10:48

vercel bot deployed to Preview – page-building-studio December 12, 2024 10:53 View deployment

vercel bot deployed to Preview – performance-studio December 12, 2024 10:53 View deployment

vercel bot deployed to Preview – test-studio December 12, 2024 10:56 View deployment

vercel bot deployed to Preview – test-next-studio December 12, 2024 11:01 View deployment

skogsmaskin added 4 commits December 12, 2024 12:25

test(playwright-ct): enable test for chromium after Meta-key-bug-fix

be333d0

This test is now likely to work as expected. Remove exception to skip Chromium

test(playwright-ct): enable test again after Meta-bug-fix

8a73d4f

test(playwright-ct): tweak timouts and meta key

3e572a8

test(playwright-ct): remove timeout

f9f6e84

skogsmaskin force-pushed the fix/playwright-ct-test branch from e941fce to f9f6e84 Compare December 12, 2024 11:25

skogsmaskin requested a review from bjoerge December 12, 2024 11:32

vercel bot deployed to Preview – page-building-studio December 12, 2024 11:32 View deployment

vercel bot deployed to Preview – performance-studio December 12, 2024 11:32 View deployment

vercel bot deployed to Preview – test-studio December 12, 2024 11:32 View deployment

skogsmaskin commented Dec 12, 2024

View reviewed changes

bjoerge enabled auto-merge December 12, 2024 11:39

vercel bot deployed to Preview – test-next-studio December 12, 2024 11:39 View deployment

bjoerge approved these changes Dec 12, 2024

View reviewed changes

bjoerge added this pull request to the merge queue Dec 12, 2024

Merged via the queue into next with commit 6bf4d2a Dec 12, 2024
59 checks passed

bjoerge deleted the fix/playwright-ct-test branch December 12, 2024 11:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(playwright-ct): fix issues and flake #8016

test(playwright-ct): fix issues and flake #8016

skogsmaskin commented Dec 11, 2024 •

edited

Loading

vercel bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024

socket-security bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

bjoerge Dec 12, 2024

skogsmaskin Dec 12, 2024

github-actions bot commented Dec 12, 2024 •

edited

Loading

skogsmaskin Dec 12, 2024

bjoerge left a comment

test(playwright-ct): fix issues and flake #8016

test(playwright-ct): fix issues and flake #8016

Conversation

skogsmaskin commented Dec 11, 2024 • edited Loading

Description

What to review

Testing

Notes for release

vercel bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 11, 2024

socket-security bot commented Dec 11, 2024 • edited Loading

github-actions bot commented Dec 11, 2024 • edited Loading

⚡️ Editor Performance Report

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

bjoerge Dec 12, 2024

Choose a reason for hiding this comment

skogsmaskin Dec 12, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 12, 2024 • edited Loading

skogsmaskin Dec 12, 2024

Choose a reason for hiding this comment

bjoerge left a comment

Choose a reason for hiding this comment

skogsmaskin commented Dec 11, 2024 •

edited

Loading

vercel bot commented Dec 11, 2024 •

edited

Loading

socket-security bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 11, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading