test(cli): remove v2 fixtures and bump timeouts #7974

bjoerge · 2024-12-06T15:21:22Z

Description

Given that v2 has been EOL for a while, we can now safely remove v2 test and v2 specific code paths in our CLI test suite.

I started out debugging CI test failures, initially suspecting it had something to do with our old v2 tests still being run, but found out that it was caused simply by the tests taking longer to complete than usual, and now seem to have exceeded the default test timeout of 30s the cli tests are using. This PR also includes bumping the default timeouts to 60s.

Also includes a small change to make it easier to run CLI tests locally.

lmk if I should split these changes into separate PRs.

What to review

Does it look sensible? Are CI passing now?

Testing

n/a - this is improving tests

Notes for release

n/a - internal

vercel · 2024-12-06T15:21:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
page-building-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 9, 2024 9:08am
performance-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 9, 2024 9:08am
test-next-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 9, 2024 9:08am
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 9, 2024 9:08am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Dec 9, 2024 9:08am

github-actions · 2024-12-06T15:22:23Z

No changes to documentation

socket-security · 2024-12-06T15:22:33Z

Removed dependencies detected. Learn more about Socket for GitHub ↗︎

🚮 Removed packages: npm/[email protected], npm/[email protected]

View full report↗︎

github-actions · 2024-12-06T15:28:01Z

Component Testing Report Updated Dec 9, 2024 9:04 AM (UTC)

✅ All Tests Passed -- expand for details

File	Status	Duration	Passed	Skipped
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	1m 10s	15	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	12s	3	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	37s	6	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	52s	11	7
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx	✅ Passed (Inspect)	0s	0	12
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	26s	6	0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx	✅ Passed (Inspect)	15s	3	0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx	✅ Passed (Inspect)	2m 33s	1	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	1m 6s	15	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	2m 51s	21	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 49s	18	0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx	✅ Passed (Inspect)	14s	3	9
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx	✅ Passed (Inspect)	39s	9	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	27s	6	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	54s	12	0
formBuilder/tree-editing/TreeEditing.spec.tsx	✅ Passed (Inspect)	0s	0	3
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx	✅ Passed (Inspect)	0s	0	3

github-actions · 2024-12-06T15:33:40Z

⚡️ Editor Performance Report

Updated Mon, 09 Dec 2024 09:05:47 GMT

Benchmark	reference ^{latency of sanity@latest}	experiment ^{latency of this branch}	Δ (%) ^{latency difference}
article (title)	23.3 efps (43ms)	23.3 efps (43ms)	+0ms (-/-%)	✅
article (body)	56.2 efps (18ms)	59.9 efps (17ms)	-1ms (-6.2%)	✅
article (string inside object)	25.0 efps (40ms)	24.4 efps (41ms)	+1ms (+2.5%)	✅
article (string inside array)	22.2 efps (45ms)	21.3 efps (47ms)	+2ms (+4.4%)	✅
recipe (name)	50.0 efps (20ms)	47.6 efps (21ms)	+1ms (+5.0%)	✅
recipe (description)	55.6 efps (18ms)	55.6 efps (18ms)	+0ms (-/-%)	✅
recipe (instructions)	99.9+ efps (6ms)	99.9+ efps (6ms)	+0ms (-/-%)	✅
synthetic (title)	18.5 efps (54ms)	18.5 efps (54ms)	+0ms (-/-%)	✅
synthetic (string inside object)	18.2 efps (55ms)	17.9 efps (56ms)	+1ms (+1.8%)	✅

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	43ms	45ms	61ms	243ms	220ms	12.0s
article (body)	18ms	21ms	29ms	103ms	252ms	6.0s
article (string inside object)	40ms	42ms	48ms	163ms	160ms	6.8s
article (string inside array)	45ms	46ms	55ms	186ms	147ms	7.4s
recipe (name)	20ms	22ms	25ms	47ms	0ms	7.7s
recipe (description)	18ms	19ms	20ms	36ms	0ms	4.6s
recipe (instructions)	6ms	8ms	9ms	9ms	0ms	3.1s
synthetic (title)	54ms	57ms	62ms	190ms	383ms	14.3s
synthetic (string inside object)	55ms	56ms	58ms	113ms	350ms	7.8s

🧪 Experiment result

The performance result of this branch

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	43ms	47ms	77ms	193ms	284ms	12.1s
article (body)	17ms	20ms	25ms	187ms	181ms	5.5s
article (string inside object)	41ms	44ms	47ms	225ms	235ms	7.2s
article (string inside array)	47ms	51ms	59ms	169ms	165ms	7.3s
recipe (name)	21ms	22ms	24ms	58ms	3ms	7.0s
recipe (description)	18ms	20ms	21ms	31ms	0ms	4.6s
recipe (instructions)	6ms	8ms	9ms	10ms	0ms	3.2s
synthetic (title)	54ms	59ms	65ms	124ms	351ms	14.4s
synthetic (string inside object)	56ms	60ms	72ms	281ms	748ms	8.5s

📚 Glossary

column definitions

benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".

latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.

p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.

p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.

p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.

blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.

test duration — how long the test run took to complete.

…cally

juice49 · 2024-12-09T09:19:23Z

packages/@sanity/cli/test/build.test.ts

@@ -27,23 +27,7 @@ describeCliTest('CLI: `sanity build` / `sanity deploy`', () => {
        expect(result.code).toBe(0)

        // These _could_ theoretically change, but is unlikely to with v2 being in support mode


Tiny nit: this comment looks to be redundant now?

juice49

Looks good to me!

vercel bot deployed to Preview – performance-studio December 6, 2024 15:25 View deployment

vercel bot deployed to Preview – page-building-studio December 6, 2024 15:26 View deployment

vercel bot deployed to Preview – test-studio December 6, 2024 15:26 View deployment

vercel bot deployed to Preview – test-next-studio December 6, 2024 15:34 View deployment

vercel bot deployed to Preview – page-building-studio December 6, 2024 15:40 View deployment

vercel bot deployed to Preview – performance-studio December 6, 2024 15:40 View deployment

vercel bot deployed to Preview – test-studio December 6, 2024 15:40 View deployment

vercel bot deployed to Preview – test-next-studio December 6, 2024 15:43 View deployment

vercel bot deployed to Preview – page-building-studio December 6, 2024 15:49 View deployment

vercel bot deployed to Preview – test-studio December 6, 2024 15:50 View deployment

vercel bot deployed to Preview – performance-studio December 6, 2024 15:51 View deployment

vercel bot deployed to Preview – test-next-studio December 6, 2024 15:56 View deployment

vercel bot deployed to Preview – performance-studio December 6, 2024 16:09 View deployment

vercel bot deployed to Preview – page-building-studio December 6, 2024 16:09 View deployment

bjoerge force-pushed the fix/remove-v2-fixtures branch from 5e43a2c to d6cdc47 Compare December 6, 2024 16:10

vercel bot deployed to Preview – performance-studio December 6, 2024 16:16 View deployment

vercel bot deployed to Preview – page-building-studio December 6, 2024 16:17 View deployment

vercel bot deployed to Preview – test-studio December 6, 2024 16:19 View deployment

bjoerge changed the title ~~Fix/remove v2 fixtures~~ test(cli) remove v2 fixtures and bump timeouts Dec 6, 2024

bjoerge force-pushed the fix/remove-v2-fixtures branch from d6cdc47 to 038b610 Compare December 6, 2024 16:30

bjoerge changed the title ~~test(cli) remove v2 fixtures and bump timeouts~~ test(cli): remove v2 fixtures and bump timeouts Dec 6, 2024

vercel bot deployed to Preview – page-building-studio December 6, 2024 16:35 View deployment

vercel bot deployed to Preview – performance-studio December 6, 2024 16:36 View deployment

vercel bot deployed to Preview – test-studio December 6, 2024 16:36 View deployment

vercel bot deployed to Preview – test-next-studio December 6, 2024 16:45 View deployment

bjoerge marked this pull request as ready for review December 6, 2024 16:50

bjoerge requested a review from a team as a code owner December 6, 2024 16:50

bjoerge requested review from juice49 and rexxars and removed request for a team December 6, 2024 16:50

bjoerge enabled auto-merge December 6, 2024 16:52

bjoerge added 4 commits December 9, 2024 09:53

chore(cli): remove v2 test fixtures

adfafaf

chore(cli): remove v2 test branches

397194c

test(cli): make it easier to configure token for running cli tests lo…

0799a4a

…cally

test(cli): increase test timeouts

b8cc9b3

bjoerge force-pushed the fix/remove-v2-fixtures branch from 038b610 to b8cc9b3 Compare December 9, 2024 08:53

vercel bot deployed to Preview – performance-studio December 9, 2024 08:59 View deployment

vercel bot deployed to Preview – page-building-studio December 9, 2024 09:00 View deployment

vercel bot deployed to Preview – test-studio December 9, 2024 09:00 View deployment

vercel bot deployed to Preview – test-next-studio December 9, 2024 09:08 View deployment

juice49 reviewed Dec 9, 2024

View reviewed changes

juice49 approved these changes Dec 9, 2024

View reviewed changes

bjoerge added this pull request to the merge queue Dec 9, 2024

Merged via the queue into next with commit 583bb6f Dec 9, 2024
57 checks passed

bjoerge deleted the fix/remove-v2-fixtures branch December 9, 2024 09:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(cli): remove v2 fixtures and bump timeouts #7974

test(cli): remove v2 fixtures and bump timeouts #7974

bjoerge commented Dec 6, 2024 •

edited

Loading

vercel bot commented Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024

socket-security bot commented Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024 •

edited

Loading

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

juice49 Dec 9, 2024

juice49 left a comment

		@@ -27,23 +27,7 @@ describeCliTest('CLI: `sanity build` / `sanity deploy`', () => {
		expect(result.code).toBe(0)

		// These _could_ theoretically change, but is unlikely to with v2 being in support mode

test(cli): remove v2 fixtures and bump timeouts #7974

test(cli): remove v2 fixtures and bump timeouts #7974

Conversation

bjoerge commented Dec 6, 2024 • edited Loading

Description

What to review

Testing

Notes for release

vercel bot commented Dec 6, 2024 • edited Loading

github-actions bot commented Dec 6, 2024

socket-security bot commented Dec 6, 2024 • edited Loading

github-actions bot commented Dec 6, 2024 • edited Loading

github-actions bot commented Dec 6, 2024 • edited Loading

⚡️ Editor Performance Report

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

juice49 Dec 9, 2024

Choose a reason for hiding this comment

juice49 left a comment

Choose a reason for hiding this comment

bjoerge commented Dec 6, 2024 •

edited

Loading

vercel bot commented Dec 6, 2024 •

edited

Loading

socket-security bot commented Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024 •

edited

Loading