WIP: Semantic search

[vangberg]

#2

Todo:

• Computing embeddings in NotesDB.save slows test down.
• Combine FTS and distance search with reciprocal rank fusion

[vangberg]

Computing embeddings in NotesDB.save slows test down

Caching the pipeline makes this negligible.

The tests runs, but running the app fails. Probably due to Webpack:

[electron] ERROR in ./node_modules/onnxruntime-node/bin/napi-v3/win32/x64/onnxruntime_binding.node 1:2
[electron] Module parse failed: Unexpected character '�' (1:2)
[electron] You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
[electron] (Source code omitted for this binary file)
[electron]  @ ./node_modules/onnxruntime-node/bin/napi-v3/ sync ^\.\/.*\/.*\/onnxruntime_binding\.node$ ./win32/x64/onnxruntime_binding.node
[electron]  @ ./node_modules/onnxruntime-node/dist/binding.js 9:0-86
[electron]  @ ./node_modules/onnxruntime-node/dist/backend.js 18:18-38
[electron]  @ ./node_modules/onnxruntime-node/dist/index.js 21:16-36 25:18-38
[electron]  @ ./node_modules/@huggingface/transformers/dist/transformers.mjs 2:0-90 24:17-70
[electron]  @ ./src/skrift/huggingface.ts 1:0-53 10:44-52
[electron]  @ ./src/skrift/notes_db/index.ts 7:0-67 181:28-58
[electron]  @ ./src/skrift-electron/main/ipc.ts 4:0-57 12:17-29 13:12-30 24:31-46 28:8-22 38:17-37 44:17-28 45:4-18 48:21-41 56:10-22 58:17-37 64:23-34 65:10-22 67:22-42 79:21-41 85:22-36 86:24-44
[electron]  @ ./src/skrift-electron/main/index.ts 3:0-36 42:0-8
[electron] 
[electron] webpack 5.97.1 compiled with 6 errors and 5 warnings in 8507 ms
[electron] npm run start:electron exited with code 1

[vangberg]

So, sqlite-vec works nicely. Including transformers.js seems difficult and error-prone. Maybe it's better to simply use an external API (Claude etc.) to compute embeddings, while storing them locally in sqlite-vec.

[vangberg]

It seems to actually … work

[vangberg]

I'm not sure RRF works that well in this case. E.g.:

Clearly interleaved -- 1st, 3rd and 5th result from keyword search, 2nd, 4th and 6th from semantic search (it's crazy how often "Kommen", Danish for caraway, comes up as a search result, actually)

[vangberg]

Making it explicit that some results are keyword matches and other are semantic matches might be better, e.g.: