Encoding the byte array in JSON with base64

[WenyXu]

What type of enhancement is this?

Performance

What does the enhancement do?

Encoding byte arrays in JSON is inefficient. One acceptable solution is encoding the byte array with base64.

greptimedb/src/common/meta/src/rpc/ddl.rs

Lines 276 to 292 in 17b385a

	impl Serialize for CreateTableTask {
	fn serialize<S>(&self, serializer: S) -> result::Result<S::Ok, S::Error>
	where
	S: serde::Serializer,
	{
	let table_info = serde_json::to_vec(&self.table_info)
	.map_err(|err| serde::ser::Error::custom(err.to_string()))?;
	let pb = PbCreateTableTask {
	create_table: Some(self.create_table.clone()),
	partitions: self.partitions.clone(),
	table_info,
	};
	let buf = pb.encode_to_vec();
	serializer.serialize_bytes(&buf)
	}
	}

greptimedb/src/common/meta/src/rpc/ddl.rs

Lines 347 to 358 in 17b385a

	impl Serialize for AlterTableTask {
	fn serialize<S>(&self, serializer: S) -> result::Result<S::Ok, S::Error>
	where
	S: serde::Serializer,
	{
	let pb = PbAlterTableTask {
	alter_table: Some(self.alter_table.clone()),
	};
	let buf = pb.encode_to_vec();
	serializer.serialize_bytes(&buf)
	}
	}

Implementation challenges

No response

[killme2008]

I think the performance of ddl procedure is not a major concern right now, and this serialization can't be the major bottleneck of it. I prefer the JSON format personally, it is more readable and friendly for debugging and management.

[WenyXu]

I think the performance of ddl procedure is not a major concern right now, and this serialization can't be the major bottleneck of it. I prefer the JSON format personally, it is more readable and friendly for debugging and management.

In this case, we only encode protobuf-encoded byte array with base64. By default, JSON format will treat the u8 array as the u32 array, which results in an expansion ratio of up to 4:1.

let pb = PbAlterTableTask { 
             alter_table: Some(self.alter_table.clone()), 
         }; 
         let buf = pb.encode_to_vec(); 
         serializer.serialize_bytes(&buf) 

[fengjiachun]

By the way, it seems that we haven't gained any benefits from using the JSON format in our current code processing. Formatting JSON from protobuf bytes doesn't enhance readability or user-friendliness.

[killme2008]

Got the point. If you want to make this change, please do it ASAP before v0.4.0 is released.