feat!: move serialization into zenoh-ext

[wyfo]

Related to #1472, but with extension traits being replaced by free functions/dedicated types

[github-actions]

PR missing one of the required labels: {'documentation', 'dependencies', 'breaking-change', 'new feature', 'bug', 'internal', 'enhancement'}

[github-actions]

PR missing one of the required labels: {'enhancement', 'dependencies', 'breaking-change', 'documentation', 'new feature', 'bug', 'internal'}

[milyin]

Looks good for me. The only problem is that old serialization examples were removed / commented out, but no new serialization examples/tests were added

[milyin]

This example is less complete than the removed one. No examples of usage ZSerializer / ZDeserializer wrriter wrapping types.
I suppose that writing real code with new API may cause changes in the API

[wyfo]

Just didn't took the time to write them.

[milyin]

Serialization for zenoh types Encoding, Timestamp, may be other zenoh types should be restored. They existed in previous implementation, "zenoh-backend-filesystem" uses it

    pub(crate) async fn put_data_info<P: AsRef<Path>>(
        &self,
        file: P,
        encoding: Encoding,
        timestamp: &Timestamp,
    ) -> ZResult<()> {
        let key = file.as_ref().to_string_lossy();
        trace!("Put data-info for {}", key);

        let z_bytes = {
            let mut zb = ZBytes::new();
            let mut writer = zb.writer();
            writer.serialize(timestamp);
            writer.serialize(encoding);
            zb
        };

[wyfo]

These types were serialized using zenoh codec directly, and I'm not sure I want to depend on the codec directly for serialization (I've removed the dependency on the codec for VLE, which is done using leb128 crate now).
What should I do?

[milyin]

These types were serialized using zenoh codec directly, and I'm not sure I want to depend on the codec directly for serialization (I've removed the dependency on the codec for VLE, which is done using leb128 crate now). What should I do?

It seems that the current serialization of these types through zenoh-codec is basically wrong. This implementation means that we are using the same format both for sending data through the wire and for storing/retrieving this data from some possibly persistent storage.

Imagine that at some moment we are releasing Zenoh 2.0 with new wire format. This change also inadvertently affects user stored data, which is a big trouble.

So I think we need to implement serialization of these types in zenoh-ext using their public api, without addressing zenoh-codec

[wyfo]

That's a valid point, but if we change the codec, we can still keep the previous version in zenoh-ext to not mess the storage up. Nevertheless, I agree with you.

However, the Encoding public API is its string representation, which is way heavier than just writing its id. Even so, I don't think we care too much, because the file transferred should be bigger than the encoding. Do you agree with using the plain string representation? Or should we expose the form id+suffix?

[milyin]

However, the Encoding public API is its string representation, which is way heavier than just writing its id. Even so, I don't think we care too much, because the file transferred should be bigger than the encoding. Do you agree with using the plain string representation? Or should we expose the form id+suffix?

The zenoh-ext is able to access 'id' field of encoding: it imports zenoh with internal feature enabled:

impl Encoding {
    #[zenoh_macros::internal]
    pub fn id(&self) -> EncodingId {
        self.0.id
    }
    #[zenoh_macros::internal]
    pub fn schema(&self) -> Option<&ZSlice> {
        self.0.schema.as_ref()
    }
    #[zenoh_macros::internal]
    pub fn new(id: EncodingId, schema: Option<ZSlice>) -> Self {
        Encoding(zenoh_protocol::core::Encoding { id, schema })
    }
}

But for future compatibility I'd vote for storing string only. This is not a big overhead and in most cases encoding is transferred as sample's field, not as part of payload.

Anyway we still have time to change it, it's not a big decision. Let's store it simply as a sting

[milyin]

minor formating issues to be fixed