Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webapp: fix GPT storage #803

Merged
merged 7 commits into from
Oct 24, 2024
Merged

webapp: fix GPT storage #803

merged 7 commits into from
Oct 24, 2024

Conversation

tharvik
Copy link
Collaborator

@tharvik tharvik commented Oct 8, 2024

as outlined in #800, the storage space of the browser does break when huge models (such as GPT) are stored.

  • change the persist backend to support larger stores as the old one doesn't support theses
  • add a storage backend based on OPFS
  • a smaller storage backend based on LocalStorage, to still work if OPFS isn't supported (such as in the tests)
    • stores the smaller models until capacity, where it discards the large ones
  • also add a layer to check that deserialized values are valid
  • also reduce the serialized size of weights & models by using UInt8Array instead of array of numbers
  • change the msgpack library to a one not eating up all memory (and recent, and standard)
  • update server tests to wait for closing the server, and reduce the number of loaded task per test

@tharvik tharvik marked this pull request as draft October 8, 2024 16:41
@tharvik tharvik force-pushed the 800-fix-gpt-storage-tharvik branch 3 times, most recently from 27fdea3 to be730f3 Compare October 9, 2024 15:49
@tharvik tharvik force-pushed the 800-fix-gpt-storage-tharvik branch from 2269be1 to c953f52 Compare October 18, 2024 14:37
@tharvik tharvik force-pushed the 800-fix-gpt-storage-tharvik branch from c953f52 to b90e4ae Compare October 18, 2024 14:40
@tharvik tharvik marked this pull request as ready for review October 18, 2024 14:50
@tharvik
Copy link
Collaborator Author

tharvik commented Oct 18, 2024

finally managed to find a way to avoid clogging up memory: don't trust msgpack-lite to do the job correctly and instead use the official JS library

@tharvik tharvik requested a review from JulienVig October 18, 2024 15:52
Copy link
Collaborator

@JulienVig JulienVig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great work!

Using Chrome or Firefox (131) smaller models are saved with LocalStorage but GPT never seems to be persisted (refreshing clears it), is there anything to do to enable OPFS?
Edit: that was my mistake

Copy link
Collaborator

@JulienVig JulienVig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!! I left mostly questions and pleas for more comments in the code, nothing blocking :)

@tharvik tharvik force-pushed the 800-fix-gpt-storage-tharvik branch from b90e4ae to f66420f Compare October 23, 2024 20:16
@tharvik tharvik merged commit 10b4be7 into develop Oct 24, 2024
23 checks passed
@tharvik tharvik deleted the 800-fix-gpt-storage-tharvik branch October 24, 2024 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants