[FRAME] Paginated Storage Primitives

[ggwpez]

The paged-list pallet provides a quite narrow solution to the problem of paginated data structures.
This MR implements StorageList and StoragePagedNMap. Only the latter contains code, the former is an adapter type of it. Alike it should be possible to use it as StoragePagedDoubleMap and StoragePagedMap. All variants are counted, so the Counted prefix is omitted.

Metadata

It currently uses fudged metadata by setting the metadata of a StoragePagedList<Value> to the metadata of a StorageMap<u32, Vec<Value>>. This is correct on the storage level, since it maps the page-index (u32) to a page (Vec<Value>), but when reading the page, it is unclear to the reader which offset to use within the page.
FRAME metadata V16 would be needed to fix this properly so that the reader is aware of the metadata field in storage.

Changes

• The paged-list pallet now uses a proper heap and supports len().
• TODO: explain

[paritytech-cicd-pr]

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3359288

[jasl]

All variants are counted

I have yet to learn the source code, but I am curious about the counter of DoubleMap or NMap.
There's a common scenario, for example, DoubleMap<CollectionId, ItemId>, I would like to list items in a collection, and it's useful to have the counter of how many items are in a collection.

Does this is the scope of the PR aim? Does the counter support multi-level?

Does the pagination ensure the results are ordered for map structures?

[ggwpez]

@jasl after reading your comment irealize that the name of the storage primitives is badly chosen. Its not really a "Map" in the common sense; its rather a List structure that can be accessed through a map key.
So for example a StoargePagedNMap has the functions append_one(key, value), iter(key) and len(key) (and some other).
All operations always need the full key. This means that these prefix operations like on normal NMaps are not possible... It would therefore not support sub-level counting operations.
The full key is AFAIK always needed, since the metadata needs to be updated.

[jasl]

Thank you for clarifying!

So this feature is for optimizing iterating a full list of storage? like when doing migration?