[Turbopack] add rocksdb #72513

sokra · 2024-11-08T14:47:57Z

What?

switches the database for persistent caching from lmdb to rocksdb.

Why?

No preallocated sparse file which cause problems
Compression support
Multiple files instead of a big single file
Better performance

Closes PACK-3424

sokra · 2024-11-08T14:48:19Z

[Turbopack] add rocksdb #72513 👈 (View in Graphite)
[Turbopack] Invalidation and Persisting fixes #72950
[Turbopack] concurrent write batches and little endian encoding #72929
canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

sokra · 2024-11-08T14:51:31Z

https://github.com/vercel/next.js/actions/runs/11744294464

ijjk · 2024-11-08T14:56:53Z

Failing test suites

Commit: 46717a0

pnpm test test/integration/export-default-map/test/index.test.js (turbopack)

Export with default map > production mode > should export with folder that has dot in name
Export with default map > production mode > should export an amp only page to clean path
Export with default map > production mode > should export hybrid amp page correctly
Export with default map > production mode > should export nested hybrid amp page correctly
Export with default map > production mode > should export nested hybrid amp page correctly with folder
Export with default map > production mode > should export hybrid index amp page correctly

Expand output

● Export with default map › production mode › should export with folder that has dot in name

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

● Export with default map › production mode › should export an amp only page to clean path

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

● Export with default map › production mode › should export hybrid amp page correctly

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

● Export with default map › production mode › should export nested hybrid amp page correctly

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

● Export with default map › production mode › should export nested hybrid amp page correctly with folder

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

● Export with default map › production mode › should export hybrid index amp page correctly

command failed with code 1 signal null
 ⚠ Linting is disabled.
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   ▲ Next.js 15.0.4-canary.45 (Turbopack)

   Checking validity of types ...
   Creating an optimized production build ...
   Loading config from /root/actions-runner/_work/next.js/next.js/test/integration/export-default-map/next.config.js
   Building (0/8) ...
   Building (2/8) 
   Building (4/8) 
   Building (6/8) 
 ✓ Building (8/8)
 ✓ Compiled successfully in 4.2s
   Collecting page data ...
   Generating static pages (0/9) ...
   Generating static pages (2/9) 
Error occurred prerendering page "/some". Read more: https://nextjs.org/docs/messages/prerender-error
AssertionError: Assertion failed: WebAssembly is uninitialized

  at new module$contents$goog$asserts_AssertionError (../evalmachine.<anonymous>:102:1441)
  at module$contents$goog$asserts_doAssertFailure (../evalmachine.<anonymous>:103:354)
  at goog.asserts.assertExists (../evalmachine.<anonymous>:104:374)
  at Object.module$contents$amp$validator_validateString [as validateString] (../evalmachine.<anonymous>:2238:108)
  at Validator.validateString (../packages/next/dist/compiled/amphtml-validator/index.js:1:20650)
  at validateAmp (../packages/next/dist/export/routes/pages.js:100:34)
  at async exportPagesPage (../packages/next/dist/export/routes/pages.js:134:17)
  at async Span.traceAsyncFn (../packages/next/dist/trace/trace.js:153:20)
  at async exportPage (../packages/next/dist/export/worker.js:334:18)
  Export encountered an error on /some, exiting the build.
   ⨯ Static worker exited with code: 1 and signal: null
  at ChildProcess.<anonymous> (lib/next-test-utils.ts:310:11)

Read more about building and testing Next.js in contributing.md.

pnpm test-dev test/development/acceptance-app/hydration-error.test.ts

Error overlay for hydration errors in App router > should collapse and uncollapse properly when there are many frames

Expand output

● Error overlay for hydration errors in App router › should collapse and uncollapse properly when there are many frames

expect(received).toBe(expected) // Object.is equality

Expected: 4
Received: NaN

  788 |
  789 |     retry(async () => {
> 790 |       expect(await getRedboxTotalErrorCount(browser)).toBe(4)
      |                                                       ^
  791 |     })
  792 |
  793 |     const description = await session.getRedboxDescription()

  at toBe (development/acceptance-app/hydration-error.test.ts:790:55)
  at retry (lib/next-test-utils.ts:806:14)

Read more about building and testing Next.js in contributing.md.

ijjk · 2024-11-08T15:01:26Z

Stats from current PR

Default Build (Increase detected ⚠️)

General

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
buildDuration	20.5s	17.5s	N/A
buildDurationCached	16.6s	14.7s	N/A
nodeModulesSize	409 MB	409 MB	✓
nextStartRea..uration (ms)	493ms	480ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
1187-HASH.js gzip	49.2 kB	49.2 kB	N/A
8276.HASH.js gzip	169 B	168 B	N/A
8377-HASH.js gzip	5.3 kB	5.3 kB	N/A
bccd1874-HASH.js gzip	53 kB	53 kB	N/A
framework-HASH.js gzip	57.5 kB	57.5 kB	N/A
main-app-HASH.js gzip	232 B	235 B	N/A
main-HASH.js gzip	33.7 kB	33.7 kB	N/A
webpack-HASH.js gzip	1.71 kB	1.71 kB	N/A
Overall change	0 B	0 B	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Overall change	39.4 kB	39.4 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
_app-HASH.js gzip	193 B	193 B	✓
_error-HASH.js gzip	193 B	193 B	✓
amp-HASH.js gzip	513 B	511 B	N/A
css-HASH.js gzip	343 B	342 B	N/A
dynamic-HASH.js gzip	1.84 kB	1.84 kB	N/A
edge-ssr-HASH.js gzip	265 B	265 B	✓
head-HASH.js gzip	363 B	362 B	N/A
hooks-HASH.js gzip	393 B	392 B	N/A
image-HASH.js gzip	4.44 kB	4.43 kB	N/A
index-HASH.js gzip	268 B	268 B	✓
link-HASH.js gzip	2.35 kB	2.34 kB	N/A
routerDirect..HASH.js gzip	328 B	328 B	✓
script-HASH.js gzip	397 B	397 B	✓
withRouter-HASH.js gzip	323 B	326 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	1.75 kB	1.75 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
_buildManifest.js gzip	746 B	747 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
index.html gzip	523 B	524 B	N/A
link.html gzip	537 B	537 B	✓
withRouter.html gzip	519 B	521 B	N/A
Overall change	537 B	537 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
edge-ssr.js gzip	128 kB	128 kB	N/A
page.js gzip	200 kB	200 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
middleware-b..fest.js gzip	665 B	664 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	31.1 kB	31.1 kB	N/A
edge-runtime..pack.js gzip	844 B	844 B	✓
Overall change	844 B	844 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
797-experime...dev.js gzip	322 B	322 B	✓
797.runtime.dev.js gzip	314 B	314 B	✓
app-page-exp...dev.js gzip	321 kB	321 kB	✓
app-page-exp..prod.js gzip	126 kB	126 kB	✓
app-page-tur..prod.js gzip	139 kB	139 kB	✓
app-page-tur..prod.js gzip	134 kB	134 kB	✓
app-page.run...dev.js gzip	311 kB	311 kB	✓
app-page.run..prod.js gzip	122 kB	122 kB	✓
app-route-ex...dev.js gzip	36.8 kB	36.8 kB	✓
app-route-ex..prod.js gzip	25 kB	25 kB	✓
app-route-tu..prod.js gzip	25 kB	25 kB	✓
app-route-tu..prod.js gzip	24.8 kB	24.8 kB	✓
app-route.ru...dev.js gzip	38.5 kB	38.5 kB	✓
app-route.ru..prod.js gzip	24.8 kB	24.8 kB	✓
pages-api-tu..prod.js gzip	9.56 kB	9.56 kB	✓
pages-api.ru...dev.js gzip	11.4 kB	11.4 kB	✓
pages-api.ru..prod.js gzip	9.56 kB	9.56 kB	✓
pages-turbo...prod.js gzip	21.3 kB	21.3 kB	✓
pages.runtim...dev.js gzip	27 kB	27 kB	✓
pages.runtim..prod.js gzip	21.3 kB	21.3 kB	✓
server.runti..prod.js gzip	916 kB	916 kB	✓
Overall change	2.34 MB	2.34 MB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js sokra/rocksdb-2	Change
0.pack gzip	2.04 MB	2.04 MB	⚠️ +345 B
index.pack gzip	145 kB	145 kB	N/A
Overall change	2.04 MB	2.04 MB	⚠️ +345 B

Diff details

Diff for main-HASH.js

Diff too large to display

Commit: 46717a0

sokra · 2024-11-18T17:41:30Z

https://github.com/vercel/next.js/actions/runs/11897981769

This is extracted from #72513. This override appears to break the build for rocksdb. It doesn't seem like there's any good reason for us to do this. It looks like we used to do a cross-build for macos arm64 (#28138), which would explain the need to override the sysroot, but we don't do that anymore. We don't do this override for x86-64 builds (even though that one _is_ a cross-build? The napi cli probably does the right thing for us...). Manually triggered `build-and-deploy` CI job: https://github.com/vercel/next.js/actions/runs/12018322261

make rocksdb the default optimize add feature flags for different databases remove logging enable blob storage Revert "remove logging" This reverts commit 3385639. sharding and parallel flush fix atomic flush ignore deprecated warnings disable invalid option drop backend correctly to avoid locked DB clippy concurrent write batches

ijjk added the created-by: Turbopack team PRs by the Turbopack team. label Nov 8, 2024

sokra changed the title ~~add rocksdb~~ [Turbopack] add rocksdb Nov 8, 2024

sokra added the CI Bypass Graphite Optimization Ignore Graphite CI optimizations, run the full CI suite. https://graphite.dev/docs/stacking-and-ci label Nov 18, 2024 — with Graphite App

sokra force-pushed the sokra/rocksdb-2 branch from 328bfd0 to af5b549 Compare November 18, 2024 17:39

sokra changed the base branch from canary to sokra/write-batch-concurrent November 18, 2024 17:39

sokra mentioned this pull request Nov 18, 2024

[Turbopack] concurrent write batches and little endian encoding #72929

Merged

sokra force-pushed the sokra/rocksdb-2 branch from af5b549 to 53577c8 Compare November 18, 2024 17:58

sokra force-pushed the sokra/write-batch-concurrent branch from 8b73506 to e2e7913 Compare November 19, 2024 09:06

sokra force-pushed the sokra/rocksdb-2 branch from 53577c8 to a1113ab Compare November 19, 2024 09:06

ijjk added the Turbopack Related to Turbopack with Next.js. label Nov 19, 2024

sokra changed the base branch from sokra/write-batch-concurrent to sokra/invalidation-improvements November 19, 2024 09:06

This was referenced Nov 19, 2024

[Turbopack] Invalidation and Persisting fixes #72950

Merged

[Turbopack] Custom persistence layer #73029

Merged

sokra force-pushed the sokra/invalidation-improvements branch from ffeced0 to ebe8016 Compare November 21, 2024 10:45

sokra force-pushed the sokra/rocksdb-2 branch from a1113ab to 20fc734 Compare November 21, 2024 10:47

sokra force-pushed the sokra/invalidation-improvements branch 2 times, most recently from 38d8d87 to a35dca4 Compare November 21, 2024 11:10

sokra force-pushed the sokra/rocksdb-2 branch from 20fc734 to 44f6c56 Compare November 21, 2024 11:10

sokra changed the base branch from sokra/invalidation-improvements to graphite-base/72513 November 21, 2024 11:36

sokra force-pushed the sokra/rocksdb-2 branch from 44f6c56 to e848ca2 Compare November 21, 2024 11:36

sokra force-pushed the graphite-base/72513 branch from a35dca4 to 7885f88 Compare November 21, 2024 11:36

sokra changed the base branch from graphite-base/72513 to canary November 21, 2024 11:37

sokra force-pushed the sokra/rocksdb-2 branch 2 times, most recently from bdc7c2b to b4a7721 Compare November 23, 2024 13:06

bgw mentioned this pull request Nov 25, 2024

chore(CI): Remove sysroot override for macos arm64 #73188

Merged

sokra added 9 commits December 6, 2024 10:27

skip native build on wasm build

ac688d0

fix test case for old backend

31dd0a3

skip native build on devlow action

f4cc209

use thread to build write batch

1c9a5d4

make write thread an const option

8a058fb

clippy

542fc43

remove logging

a0ae5f2

remove warning

46717a0

sokra force-pushed the sokra/rocksdb-2 branch from b4a7721 to 46717a0 Compare December 6, 2024 09:27

padmaia closed this Feb 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Turbopack] add rocksdb #72513

[Turbopack] add rocksdb #72513

sokra commented Nov 8, 2024 •

edited

Loading

sokra commented Nov 8, 2024 •

edited

Loading

sokra commented Nov 8, 2024

ijjk commented Nov 8, 2024 •

edited

Loading

ijjk commented Nov 8, 2024 •

edited

Loading

sokra commented Nov 18, 2024

[Turbopack] add rocksdb #72513

[Turbopack] add rocksdb #72513

Conversation

sokra commented Nov 8, 2024 • edited Loading

What?

Why?

sokra commented Nov 8, 2024 • edited Loading

sokra commented Nov 8, 2024

ijjk commented Nov 8, 2024 • edited Loading

Failing test suites

ijjk commented Nov 8, 2024 • edited Loading

Stats from current PR

sokra commented Nov 18, 2024

sokra commented Nov 8, 2024 •

edited

Loading

sokra commented Nov 8, 2024 •

edited

Loading

ijjk commented Nov 8, 2024 •

edited

Loading

ijjk commented Nov 8, 2024 •

edited

Loading