mito: Null value should also contribute to memtable usage

[evenyag]

What type of enhancement is this?

Other

What does the enhancement do?

Now the ValueRef always considers null as zero size

greptimedb/src/datatypes/src/value.rs

Lines 1172 to 1174 in 19413eb

	pub fn data_size(&self) -> usize {
	match *self {
	ValueRef::Null => 0,

The memtable uses ValueRef::data_size() to compute the allocated size, which is incorrect if there are too many nulls.

greptimedb/src/mito2/src/memtable/time_series.rs

Line 194 in 19413eb

allocated += fields.iter().map(|v| v.data_size()).sum::<usize>();

The arrow array allocates space for nulls.

Implementation challenges

We might assume nulls occupy spaces or find a new way to estimate the memory used by array builders.

[killme2008]

I wonder if we can do a quick fix.

[tisonkun]

@evenyag what will be the size of a single Null in memtable? Is it a constant or it can be dynamic?

[etolbakov]

was just passing by and noticed this conversation
@tisonkun @evenyag
how do you folks feel about the following?

ValueRef::Null => std::mem::size_of::<*const u8>(),

[evenyag]

It depends on the data type and the arrow array's implementation. Primitive arrays always push a default value into the data buffer so it occupies the same space as the native type. BinaryArray and StringArray also push an offset into the offset buffer so the size is the size of the offset type. I ignore the size of the null bitmap and buffer padding for simplicity.

It's hard to maintain the correct memory usage of the memtable via accumulating ValueRef::data_size(). Maybe we can use a constant for a quick fix? Maybe 8, the size of int64. At least, it is better than 0.