[core] Use ref hash to avoid key copy

[dentiny]

For the current implementation, each key is stored twice in LRU cache, one in list, another in list.
But considering the invariant that, as long as the entry exists in LRU cache, the address of the key never changes (since we're using std::list), we could use our self-defined hash wrapper to store key reference in hash map instead of value.

[rynewang]

General comment: I am worried about 1 extra indirection this PR introduces. Say the flat map wants to rehash - it needs to do at least O(n) Hashes and Eqs. Now all these operations need a dereference to read the Key in list. Is this saved memory space worth the extra compute time cost?

[rynewang]

instead of a custom Hash and Equal functor, what about creating a MyWrapper class that contains a std::reference_wrapper and implements Hash and Eq?

[dentiny]

It complicated the implementation. For example,

struct KeyReferenceWrapper {
  std::reference_wrapper<Key> key_ref;
  // Implement hash function: absl::container_internal::hash_default_hash<Key>
  // Implement eq function: absl::container_internal::hash_default_eq<Key>
}

don't see benefit.

[dentiny]

General comment: I am worried about 1 extra indirection this PR introduces. Say the flat map wants to rehash - it needs to do at least O(n) Hashes and Eqs. Now all these operations need a dereference to read the Key in list.

I'm not sure what you're worrying about? The only overhead is the reference wrapper deference.

Is this saved memory space worth the extra compute time cost?

I'm not sure the indirection (aka. reference wrapper deference) could be saved easily from somewhere else.
For example, the shared pointer cleanup I'm working on these days. I'm not worried about the performance cost.

[dentiny]

General comment: I am worried about 1 extra indirection this PR introduces. Say the flat map wants to rehash - it needs to do at least O(n) Hashes and Eqs. Now all these operations need a dereference to read the Key in list. Is this saved memory space worth the extra compute time cost?

Some numbers:

• a cache miss and memory lookup is 10s of nanoseconds, which is far far far less than ray scheduling overhead
• in constrast, considering the only existing use key is serialized runtime env, we don't have a hard cap on how large the serialized string could be, I think saving a key storage is more important

Some reference:

• SO link: https://stackoverflow.com/questions/431469/how-expensive-is-it-to-dereference-a-pointer
• latency measurement: https://gist.github.com/jboner/2841832