computeDigest always return [0, 0]?

[Spxg]

sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs

Line 13510 in 7c1b309

computeDigest(byteArray) {

sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs

Line 13321 in 7c1b309

#apBody = new Uint8Array(HEADER_CORPUS_SIZE);

computeDigest seems to only return [0, 0], because the length of apBody Uint8Array is 516, which will cause h1 h2 to be infinite.

[tomayac]

Closing here, as this affects the library upstream (if it's a bug), but nonetheless @sgbeal FYI. The present project wraps the upstream library for npm.

[sgbeal]

i will look at this in the upstream tree in the next few hours. @Spxg please feel free to re-post this in the sqlite forum. If it's misbehaving this way then it's a fluke that it ever worked.

[sgbeal]

i can definitely reproduce this, and the fault is absolutely mine, but we have a slight problem with fixing it. From the documentation:

    /**
       Computes a digest for the given byte array and returns it as a
       two-element Uint32Array. This digest gets stored in the
       metadata for each file as a validation check. Changing this
       algorithm invalidates all existing databases for this VFS, so
       don't do that.
    */

i'll need to come up with a fix which is only applied when re-populating the VFS from scratch. (Edit: that introduces another problem, though: the VFS has no persistent place to store the fact that it needs to use the updated/working digest. So i'm not sure how to fix this without invaliding every db out there. Sheesh!)

Thank you for the report!

[Spxg]

that introduces another problem, though: the VFS has no persistent place to store the fact that it needs to use the updated/working digest.

Yes, fixing this will be a breaking change.

In addition, another issue also mentioned you, you can take a look if you have time: #92

Thank you for the fast turnaround.

[sgbeal]

Yes, fixing this will be a breaking change.

It currently appears so but maybe not: we have 32 bits for flags in the per-file header and what i'd like to do is, when creating a new file, set that bit to indicate that we can use a working computeDigest(). When opening an a file created before this (as yet hypothetical) fix, that flag bit will be unset and we'll use the no-op/broken digest.

That's the plan, anyway, and this will be my priority today. It's unfortunately too late to get into the 3.49 release, which is currently being prepared, but it would be in any subsequent 3.49.x patch release.

Thank you again for the report!

[sgbeal]

@Spxg i have a potential fix for this checked in, but (A) it won't be part of 3.49.0 and (B) it requires a good deal more testing before i'm satisfied that it behaves sanely with older-format files. In short, it only applies the digest when creating new VFS entries, and then records the fact that the digest should be checked in one of the file's flags. Older-format files have a 0 in that flag bit and will revert to the old behavior of (effectively) no checksum.

In any case, this will, assuming it works out, be part of 3.50 and possibly 3.49.1.

BTW: i'd be interested in hearing how you found this bug.

[Spxg]

BTW: i'd be interested in hearing how you found this bug.

I wrapped sqlite-wasm, and expect to provide a usable C-like API for Rust (https://github.com/Spxg/sqlite-wasm-rs).

Because there are two different wasm instances (wasm32-unknown-unknown vs wasm32-unknown-emscripten), many operations require memory copying and memory management. The advantage is that can directly use the persistent storage implemented by sqlite-wasm. Currently, wasm32-unknown-unknown support has been added to diesel: diesel-rs/diesel#4411.

But I want to provide an additional, more "native" version, compiling sqlite directly to wasm32-unknown-uknown.
Given that sqlite mainly supports emscripten, linking emscripten to wasm32-unknown-unknown is the best approach (otherwise you need to implement some methods of libc yourself). rustwasm/wasm-bindgen#3454 makes this possible.

After a simple polyfill (enable -DSQLITE_OS_OTHER), all diesel tests can now run (db is :memory:). Now just need to implement a persistent storage. Since opfs-sahpool looks more like an internal implementation, I tried to port opfs-sahpool to Rust and found this bug during the porting process.

Thanks to you and the community for making it possible to run SQLite on the web.

[sgbeal]

Thanks to you and the community for making it possible to run SQLite on the web.

FWIW, i strongly recommend following the works of Roy Hashimoto over at:

https://github.com/rhashimoto

in particularly his wa-sqlite project. To the best of my knowledge he was the first person to get OPFS working with SQLite and he has undoubtedly done far, far more research and experimentation into how to improve upon it than anyone else. His project provides a large handful of VFSes which explore various benefits and trade-offs. Plus, he writes modern JS using modern JS-centric toolchains, so his code is certainly more approachable than ours for most JS developers. Contrast that with us in the SQLite project, who deliberately/willfully/stubbornly stick to "vanilla JS" without using any of all-too-fluid modern JS frameworks.

[Spxg]

FWIW, i strongly recommend following the works of Roy Hashimoto over at:

Very helpful! Because I also need to implement a memvfs as the default vfs, the examples provided by wa-sqlite can help me very well.