perf(index): reduce memory usage

[cmdoret]

The index is consuming too much memory in high load situations. It is currently a simple HashMap<String, String> mapping instance-> type. This implies high redundancy as #instances >> #types.

Objective: change index structure to optimize memory usage while keeping O(1) lookup time.

Proposal:

File format:

types: [Person, Account, Organization]
map:
  hash(urn:alice): 0
  hash(urn:acme):  2
  hash(urn:test-account): 1
  hash(urn:bob): 0

Struct:

//pseudocode

struct Index {
  types: Vec<str>,
  map: HashMap<[u8], usize>,
}

impl Index {
  fn get(&self, instance: &str) -> String {
    type_idx = self.map.get(instance.hash())
    
    return self.types[type_idx]
  }
}

[cmdoret]

Further optimizations are possible:

• Allow usize types in Rules to support int comparison instead of String
  • e.g. by using enum Type {Str(String), Int(usize)}
• Binary index format
  • e.g. via serde_binary or HDT

[cmdoret]

Note: need to use map hash->enum {usize, Vec}
Also we can filter out unneeded types from the index at pseudonymization step