bench(qe): findUnique batching performance #4856

Druue · 2024-05-07T14:27:14Z

github-actions · 2024-05-07T14:31:32Z

WASM Query Engine file Size

Engine	This PR	Base branch	Diff
Postgres	2.153MiB	2.153MiB	0.000B
Postgres (gzip)	846.057KiB	846.060KiB	-3.000B
Mysql	2.119MiB	2.119MiB	0.000B
Mysql (gzip)	831.746KiB	831.748KiB	-2.000B
Sqlite	2.015MiB	2.015MiB	0.000B
Sqlite (gzip)	793.077KiB	793.079KiB	-2.000B

codspeed-hq · 2024-05-07T14:37:25Z

CodSpeed Performance Report

Merging #4856 will not alter performance

_{Comparing bench/6573 (555eec2) with main (793f7fb)}

Summary

✅ 11 untouched benchmarks

🆕 1 new benchmarks

Benchmarks breakdown

	Benchmark	`main`	`bench/6573`	Change
🆕	`batching`	N/A	216 ms	N/A

github-actions · 2024-05-07T14:41:24Z

✅ WASM query-engine performance won't change substantially (1.008x)

Full benchmark report

DATABASE_URL="postgresql://postgres:postgres@localhost:5432/bench?schema=imdb_bench&sslmode=disable" \
node --experimental-wasm-modules query-engine/driver-adapters/executor/dist/bench.mjs
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
"
cpu: AMD EPYC 7763 64-Core Processor
runtime: node v18.20.2 (x64-linux)

benchmark                   time (avg)             (min … max)       p75       p99      p999
-------------------------------------------------------------- -----------------------------
• movies.findMany() (all - ~50K)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline     364 ms/iter       (363 ms … 366 ms)    366 ms    366 ms    366 ms
Web Assembly: Latest       446 ms/iter       (443 ms … 450 ms)    448 ms    450 ms    450 ms
Web Assembly: Current      452 ms/iter       (447 ms … 458 ms)    456 ms    458 ms    458 ms
Node API: Current          194 ms/iter       (192 ms … 196 ms)    195 ms    196 ms    196 ms

summary for movies.findMany() (all - ~50K)
  Web Assembly: Current
   2.33x slower than Node API: Current
   1.24x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movies.findMany({ take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  14'955 µs/iter (14'706 µs … 16'997 µs) 14'882 µs 16'997 µs 16'997 µs
Web Assembly: Latest    18'275 µs/iter (18'071 µs … 20'081 µs) 18'309 µs 20'081 µs 20'081 µs
Web Assembly: Current   18'360 µs/iter (18'220 µs … 19'129 µs) 18'359 µs 19'129 µs 19'129 µs
Node API: Current        8'139 µs/iter  (7'858 µs … 10'931 µs)  8'035 µs 10'931 µs 10'931 µs

summary for movies.findMany({ take: 2000 })
  Web Assembly: Current
   2.26x slower than Node API: Current
   1.23x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movies.findMany({ where: {...}, take: 2000 })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   2'330 µs/iter   (2'217 µs … 3'562 µs)  2'308 µs  3'391 µs  3'562 µs
Web Assembly: Latest     2'823 µs/iter   (2'739 µs … 3'530 µs)  2'823 µs  3'306 µs  3'530 µs
Web Assembly: Current    2'877 µs/iter   (2'774 µs … 3'842 µs)  2'857 µs  3'703 µs  3'842 µs
Node API: Current        1'378 µs/iter   (1'290 µs … 1'715 µs)  1'395 µs  1'615 µs  1'715 µs

summary for movies.findMany({ where: {...}, take: 2000 })
  Web Assembly: Current
   2.09x slower than Node API: Current
   1.23x slower than Web Assembly: Baseline
   1.02x slower than Web Assembly: Latest

• movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline     568 ms/iter       (558 ms … 582 ms)    576 ms    582 ms    582 ms
Web Assembly: Latest       767 ms/iter       (761 ms … 783 ms)    773 ms    783 ms    783 ms
Web Assembly: Current      778 ms/iter       (764 ms … 803 ms)    802 ms    803 ms    803 ms
Node API: Current          470 ms/iter       (455 ms … 489 ms)    487 ms    489 ms    489 ms

summary for movies.findMany({ include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.65x slower than Node API: Current
   1.37x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline  78'625 µs/iter (78'423 µs … 79'028 µs) 78'838 µs 79'028 µs 79'028 µs
Web Assembly: Latest       108 ms/iter       (107 ms … 109 ms)    108 ms    109 ms    109 ms
Web Assembly: Current      108 ms/iter       (108 ms … 109 ms)    109 ms    109 ms    109 ms
Node API: Current       61'654 µs/iter (59'751 µs … 66'007 µs) 62'164 µs 66'007 µs 66'007 µs

summary for movies.findMany({ where: {...}, include: { cast: true } take: 2000 }) (m2m)
  Web Assembly: Current
   1.76x slower than Node API: Current
   1.38x slower than Web Assembly: Baseline
   1.01x slower than Web Assembly: Latest

• movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1'006 ms/iter     (997 ms … 1'025 ms)  1'020 ms  1'025 ms  1'025 ms
Web Assembly: Latest     1'286 ms/iter   (1'272 ms … 1'321 ms)  1'310 ms  1'321 ms  1'321 ms
Web Assembly: Current    1'286 ms/iter   (1'275 ms … 1'307 ms)  1'303 ms  1'307 ms  1'307 ms
Node API: Current          868 ms/iter       (848 ms … 884 ms)    882 ms    884 ms    884 ms

summary for movies.findMany({ take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.48x slower than Node API: Current
   1.28x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline     141 ms/iter       (141 ms … 142 ms)    141 ms    142 ms    142 ms
Web Assembly: Latest       180 ms/iter       (178 ms … 191 ms)    179 ms    191 ms    191 ms
Web Assembly: Current      180 ms/iter       (179 ms … 183 ms)    181 ms    183 ms    183 ms
Node API: Current          105 ms/iter       (103 ms … 107 ms)    106 ms    107 ms    107 ms

summary for movie.findMany({ where: { ... }, take: 2000, include: { cast: { include: { person: true } } } })
  Web Assembly: Current
   1.71x slower than Node API: Current
   1.27x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

• movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1'059 µs/iter     (984 µs … 1'786 µs)  1'047 µs  1'704 µs  1'786 µs
Web Assembly: Latest     1'353 µs/iter   (1'301 µs … 1'867 µs)  1'355 µs  1'730 µs  1'867 µs
Web Assembly: Current    1'392 µs/iter   (1'323 µs … 2'167 µs)  1'393 µs  1'846 µs  2'167 µs
Node API: Current          751 µs/iter     (693 µs … 1'129 µs)    772 µs    817 µs  1'129 µs

summary for movie.findMany({ where: { reviews: { author: { ... } }, take: 100 }) (to-many -> to-one)
  Web Assembly: Current
   1.85x slower than Node API: Current
   1.31x slower than Web Assembly: Baseline
   1.03x slower than Web Assembly: Latest

• movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
-------------------------------------------------------------- -----------------------------
Web Assembly: Baseline   1'040 µs/iter     (986 µs … 1'735 µs)  1'035 µs  1'658 µs  1'735 µs
Web Assembly: Latest     1'376 µs/iter   (1'302 µs … 2'349 µs)  1'367 µs  1'994 µs  2'349 µs
Web Assembly: Current    1'380 µs/iter   (1'324 µs … 1'989 µs)  1'384 µs  1'773 µs  1'989 µs
Node API: Current          753 µs/iter     (704 µs … 1'015 µs)    778 µs    845 µs  1'015 µs

summary for movie.findMany({ where: { cast: { person: { ... } }, take: 100 }) (m2m -> to-one)
  Web Assembly: Current
   1.83x slower than Node API: Current
   1.33x slower than Web Assembly: Baseline
   1x faster than Web Assembly: Latest

After changes in 555eec2

SevInf · 2024-05-27T09:34:22Z

It's not actually fixes the issue, it just adds a benchmark, right? Can we remove closes from the body?

Druue · 2024-05-27T13:48:40Z

Specifically benching convert_single doesn't seem to indicate the issue:

Running benches/query_batching.rs (/Users/soph/src/prisma-engines/target/release/deps/query_batching-27a756169b13a5a3)
convert_single          time:   [384.73 ps 393.84 ps 407.00 ps]
                        change: [-0.9007% +1.3046% +3.3588%] (p = 0.24 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  8 (8.00%) high mild
  1 (1.00%) high severe

And neither the benches on this PR nor the benches locally (same code as here) indicate the issues highlighted in the reproduction in our repros:

Running benches/query_batching.rs (/Users/soph/src/prisma-engines/target/release/deps/query_batching-27a756169b13a5a3)
Benchmarking batching: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.6s, or reduce sample count to 80.
batching                time:   [55.888 ms 57.016 ms 58.388 ms]
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) high mild
6 (6.00%) high severe

Comparison:

multiple `prisma.findFirst(...)` calls: 417.636ms
multiple `prisma.findUnique(...)` calls: 1.481s
using DataLoader: 41.626ms
`prisma.findMany(...)`: 35.207ms

So we somehow went from a total round trip of +4s with the majority of that being in engines to only seeing ~.5 - 2s

janpio · 2024-06-03T08:28:41Z

Note that part of those measurements in the reproduction were probably wrong: prisma/prisma#6573 (comment)

Druue requested a review from a team as a code owner May 7, 2024 14:27

Druue requested review from Weakky and removed request for a team May 7, 2024 14:27

Druue added this to the 5.14.0 milestone May 7, 2024

Druue force-pushed the bench/6573 branch from e2bb768 to c394d7c Compare May 23, 2024 14:24

Druue modified the milestones: 5.14.0, 5.15.0 May 23, 2024

Druue self-assigned this May 23, 2024

Druue requested a review from SevInf May 23, 2024 14:31

Druue added 2 commits May 27, 2024 12:33

bench prisma/prisma#6573

d7ebb9e

clippy

555eec2

Druue force-pushed the bench/6573 branch from 7d53088 to 555eec2 Compare May 27, 2024 12:15

Druue marked this pull request as draft May 27, 2024 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bench(qe): findUnique batching performance #4856

bench(qe): findUnique batching performance #4856

Druue commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

codspeed-hq bot commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

SevInf commented May 27, 2024

Druue commented May 27, 2024 •

edited

Loading

janpio commented Jun 3, 2024

bench(qe): findUnique batching performance #4856

Are you sure you want to change the base?

bench(qe): findUnique batching performance #4856

Conversation

Druue commented May 7, 2024 • edited Loading

github-actions bot commented May 7, 2024 • edited Loading

WASM Query Engine file Size

codspeed-hq bot commented May 7, 2024 • edited Loading

CodSpeed Performance Report

Merging #4856 will not alter performance

Summary

Benchmarks breakdown

github-actions bot commented May 7, 2024 • edited Loading

✅ WASM query-engine performance won't change substantially (1.008x)

SevInf commented May 27, 2024

Druue commented May 27, 2024 • edited Loading

janpio commented Jun 3, 2024

Druue commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

codspeed-hq bot commented May 7, 2024 •

edited

Loading

github-actions bot commented May 7, 2024 •

edited

Loading

Druue commented May 27, 2024 •

edited

Loading