Skip to content

Latest commit

 

History

History
30 lines (16 loc) · 1.01 KB

README.md

File metadata and controls

30 lines (16 loc) · 1.01 KB

sblex-rs

Memory usage

Morphology is the beast in the room that uses a lot of memory.

Python reference: Diagram showing memory usage of Python version

NOTE All rust examples also serializes the trie to json.

Baseline:

Diagram showing memory usage of load_morphology

Use ArcStr instead of String

We can replace String with arcstr::ArcStr as value in the Trie to reduce memory usage.

Diagram showing memory usage of load_morphology

And if also use ArcStr as keys in the inner map we can reduce it some more:

Diagram showing memory usage of load_morphology

Decomposing builder

The builder already holds the number of states and all states, lets grab them (and reuse the memory).

Diagram showing memory usage of load_morphology