Fix absolute address of the end of hint input

[matthiasgoergens]

https://github.com/scroll-tech/ceno/blob/matthias/ceno_host/ceno_host/src/lib.rs test_rkyv_padding has some example code to show that we can left-pad rkyv serialised data without any problems, but right padding doesn't work, at least not naively.

Thus we should fix the last byte of the input to always be in the same VM address, and make the start move further and further left, if we add more data.

See also #607 and #622

[naure]

How does one use rkyv with the usual rightwards growth, like Vec or "write" APIs? And what's the difference between left and right? I understand the difference between headers and footers in a streameable format, but I don't see how that applies here.

[matthiasgoergens]

Have a look at the linked code for two example that illustrate the difference. Please also follow the link to the rkyv docs that's in the code.

[matthiasgoergens]

How does one use rkyv with the usual rightwards growth, like Vec or "write" APIs?

That would be interesting to look up! But luckily it's irrelevant here.

And what's the difference between left and right?

Have a look at the linked code. Basically in rkyv: let buffer = chain(random garbage, your data) is deserialisable the same as your data, but let buffer = chain(your data, random garbage) ain't. You can replace 'random garbage' with 'memory addresses that throw an error when you try to access them' for the same result.

I understand the difference between headers and footers in a streameable format, but I don't see how that applies here.

Fortunately, we don't need to understand anything here. (But I encourage you to look into it, if you are curious.)

Why do you bring up streamable formats? Everything we do is statically known up front.

[matthiasgoergens]

To make your life easier, let me copy and paste the relevant link from the docs in the code that I mentioned: https://rkyv.org/format.html

This deterministic layout means that you don't need to store the position of the root object in most cases. As long as your buffer ends right at the end of your root object, you can use access with your buffer.

[naure]

As long as your buffer ends right at the end of your root object

Ok sure that just means something else does the framing (e.g. Rust slice length). Just store the length or entry point in the first word then. Maybe the library actually has this feature already. That is when the data has to have a dynamic layout in the first place.

[matthiasgoergens]

As long as your buffer ends right at the end of your root object

Ok sure that just means something else does the framing (e.g. Rust slice length). Just store the length or entry point in the first word then. Maybe the library actually has this feature already. That is when the data has to have a dynamic layout in the first place.

The library does not store the length or 'entry point' at the start of the slice. What makes you think so?
The two experiments I linked you to wouldn't work, if it did.

It's not that hard. No need to speculate. If you are curious, you can either have a look at the test code I linked or you can read the rkyv sources. (Or both.)

I don't know what you mean by framing and why you think anyone would need that? (I have a guess about what you mean by that, but it seems to contradict what you say. So I'm confused.)

[matthiasgoergens]

To spell it out:

Rust slices have all the information necessary to determine eg their end, start, midpoint, length etc.

Rkyv's derserialisation expects to find a specific 'root' at a fixed offset from the end of the slice you pass it.

It then re-assembles the whole DAG of your data structure from that root, by following pointers.

If you have some rkyv serialised data in a slice of absolute memory addresses of start..end, you can extend it to earlier_start..end and still get exactly the same data out, as long as earlier_start <= start.

Specifically in our case, we can fill the available hint input to always end at end when we prepare the VM's memory. Then we can ask rkyv to deserialise from hint_io_start..end, even if hint_io_start is much smaller than the actual start of the data in memory we prepared.

Does this make sense?

(Everything here is a bit simplified.)

[naure]

This sounds like you have misinterpreted my last message which was admittedly compact, and assumed I didn’t know what I was talking about. I will try to expand some other time.

[matthiasgoergens]

Yes, that's sounds likely. Please expand, thanks.

[matthiasgoergens]

@naure Please expand when you have time.

[naure]

Somebody has explained that solution: #631 (comment)