Try to preserve metatable information in core.serialize

[y5nw]

Add compact, short information about your PR for easier understanding:

• Goal of the PR
  This PR attempts to preserve metatable information in the data serialized by minetest.serialize.
• How does the PR work?
  This PR works by keeping a record of metatables that are assigned to a name. This name is then kept along with the serialized data which can later be used to construct the wanted type during the deserialization process.
  Note that there are certain limitations that apply when an object contains a recognized metatable. These limitations will be documented as part of this PR.
• Does it resolve any reported issue?
  Fixes Serialization does not preserve vector metatable #14357.
• Does this relate to a goal in the roadmap?
  No, but it may be considered a bugfix depending on whether the issue linked above is considered as a bug.
• If not a bug fix, why is this PR needed? What usecases does it solve?
  See linked issue.

To do

This PR is Ready for Review.

• Compatibility: serialized data that do not use this feature remain compatible with older versions of core.deserialize; compatibility is otherwise not guaranteed.
• Documentation
• Support custom (de)serializers
• Support for MT's builtin types:
  • vector
  • ItemStack (*)
• Support for recursive objects and reference counting.
• Unittests

How to test

• Make sure that there is a test case for each entry in the To-Do.
• Make sure that the To-Do covers every case.
• Test the new core.serialize on various data structures and make sure they work.

(*) The testcase for ItemStacks is put into the devtest as it appears to be implemented as userdata.

[fluxionary]

would this also fix #13644?

[sfan5]

would this also fix #13644?

no, different code.

[y5nw]

With #14369 being reviewed, should I change this to depend on that PR? It turned out that the only "special case" to handle here would be ItemStacks, and regular tables can quite well use core.known_metatables from that PR.

Edit: Another option would be to have the serializer also check core.known_metatables if the metatable is not recorded by the serializer. This would still make it possible for mods to serialize data in "special" ways if this is desired.

[appgurueu]

I think there's a more general abstraction possible that doesn't require special-casing ItemStacks (or other userdata which we might want to preserve in the future).

We simply need two maps:

• One which maps metatables to named serializers, for example: [itemstack_mt] = {name = "ItemStack", serializer = ItemStack().to_table}. The serializer returns a single value (or perhaps a vararg) to be fed to the named function. So serializing an item stack would result in something like ItemStack({name = "default:dirt", count = 42}).
• Another which maps these function names to actual functions, in this case, simply {ItemStack = ItemStack}.

So all we need to know for each "class" is a quadruple: Name; metatable / methodtable; serializer (to_*); deserializer (constructor).

For the case of pure Lua data structures, the serializer can be simply the identity function, and the deserializer only has to set the metatable.

So I think it would make sense to add an extended version, say, core.register_serializable(name, metatable, serializer, deserializer). Then you could just do core.register_serializable("ItemStack", itemstack_mt, ItemStack().to_table, ItemStack).

core.register_portable_metatable(name, metatable) would just be convenience for core.register_serializable(name, metatable, function(x) return x end, function(x) return setmetatable(x, metatable) end).

(Something like this is what modlib does, but it only does it for userdata (hence the todo 😅). The usage is relatively clean though. Peek at this with a grain of salt, I'd write it a bit differently nowadays.)

---

I think we should also ask ourselves: Don't we want to expose these metatables in some global variables for convenience? It's a bit nasty to have to obtain an instance to obtain the metatable. See #11639.

[y5nw]

* One which maps metatables to named serializers, for example: `[itemstack_mt] = {name = "ItemStack", serializer = ItemStack().to_table}`. The serializer returns a single value (or perhaps a vararg) to be fed to the named function. So serializing an item stack would result in something like `ItemStack({name = "default:dirt", count = 42})`.

* Another which maps these function names to actual functions, in this case, simply `{ItemStack = ItemStack}`.

This should be doable, although I would prefer to not guarantee the correctness of preserving recursive data structures with custom serializers.

I think we should also ask ourselves: Don't we want to expose these metatables in some global variables for convenience? It's a bit nasty to have to obtain an instance to obtain the metatable. See #11639.

That would make sense IMO, but let's leave that to a different PR.