From a0a62ff10ec2468af0f9d34fc710c0fb48a98c0e Mon Sep 17 00:00:00 2001 From: garikbesson Date: Thu, 1 Aug 2024 16:53:25 +0300 Subject: [PATCH] Update Collections page (#2146) * added Iterable Set/Map + changed code urls (temporarily) * additional changes * nested errors explanation changes * update js collections examples * updated Pagination section * additional comment about using collections in JS * fix: snippets and text --------- Co-authored-by: Guille --- .../2.smart-contracts/anatomy/collections.md | 417 +++++++++--------- 1 file changed, 211 insertions(+), 206 deletions(-) diff --git a/docs/2.build/2.smart-contracts/anatomy/collections.md b/docs/2.build/2.smart-contracts/anatomy/collections.md index 566afb6d2c9..61859ca936a 100644 --- a/docs/2.build/2.smart-contracts/anatomy/collections.md +++ b/docs/2.build/2.smart-contracts/anatomy/collections.md @@ -6,17 +6,14 @@ import {CodeTabs, Language, Github} from "@site/src/components/codetabs" import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -When deciding on data structures to use for the data of the application, it is important to understand the tradeoffs of data structures in your smart contract. - -Choosing the wrong structure can create a bottleneck as the application scales, and migrating the state to the new data structures will come at a cost. +When deciding on data structures it is important to understand their tradeoffs. Choosing the wrong structure can create a bottleneck as the application scales, and migrating the state to the new data structures will come at a cost. You can choose between two types of collections: 1. Native collections (e.g. `Array`, `Map`, `Set`), provided by the the language -2. SDK collections (e.g. `UnorderedMap`, `Vector`), provided by the NEAR SDK - -Since the SDK reads all the contract's attributes when a function is executed - and writes them back when it finishes - understanding how the SDK stores and loads both types of collections is crucial to decide which one to use. +2. SDK collections (e.g. `IterableMap`, `Vector`), provided by the NEAR SDK +Understanding how the contract stores and loads both types of collections is crucial to decide which one to use. :::tip @@ -24,9 +21,9 @@ Use native collections for small amounts of data that need to be accessed all to ::: -:::info +:::info How the State is Handled -Contracts store all their data in a `key-value` database. The SDK handles this database, and stores values [serialized in JSON or Borsh](./serialization.md) +Each time the contract is executed, the first thing it will do is to read the values and [deserialize](./serialization.md) them into memory, and after the function finishes, it will [serialize](./serialization.md) and write the values back to the database. ::: @@ -35,10 +32,10 @@ Contracts store all their data in a `key-value` database. The SDK handles this d ## Native Collections Native collections are those provided by the language: -- JS: `Array`, `Set`, `Map` -- Rust: `Vector`, `HashMap`, `Set` +- JS: `Array`, `Set`, `Map`, `Object` ... +- Rust: `Vector`, `HashMap`, `Set` ... -All entries in a native collection are serialized into a single value and stored together into the state. This means that every time a function execute, the SDK will read and deserialize all entries in the native collection. +All entries in a native collection are **serialized into a single value** and **stored together** into the state. This means that every time a function execute, the SDK will read and **deserialize all entries** in the native collection.
@@ -54,9 +51,9 @@ Native collections are useful if you are planning to store smalls amounts of dat ::: -:::warning Keep Native Collections Small +:::danger Keep Native Collections Small -As the collection grows, reading and writing it will cost more and more gas. If the collections grows too large, your contract might end up expending all its available gas in reading/writing the state, thus becoming unusable +As the native collection grows, deserializing it from memory will cost more and more gas. If the collections grows too large, your contract might expend all the gas trying to read its state, making it fail on each function call ::: @@ -64,9 +61,9 @@ As the collection grows, reading and writing it will cost more and more gas. If ## SDK Collections -The NEAR SDKs expose collections that are optimized to store large amounts of data in the contract's state. These collections are built to have an interface similar to native collections. +The NEAR SDKs expose collections that are optimized for random access of large amounts of data. SDK collections are instantiated using a "prefix", which is used as an index to split the data into chunks. This way, SDK collections can defer reading and writing to the store until needed. -SDK collections are instantiated using a "prefix", which is used as an index to split the data into chunks. This way, SDK collections can defer reading and writing to the store until needed. +These collections are built to have an interface similar to native collections.
@@ -93,32 +90,44 @@ SDK collections are useful when you are planning to store large amounts of data | SDK Collection | Native Equivalent | Description | |----------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `Vector` | `Array` | A growable array type. The values are sharded in memory and can be used for iterable and indexable values that are dynamically sized. | -| `LookupMap` | `Map` | This structure behaves as a thin wrapper around the key-value storage available to contracts. This structure does not contain any metadata about the elements in the map, so it is not iterable. | -| `UnorderedMap` | `Map` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | | `LookupSet` | `Set` | A set, which is similar to `LookupMap` but without storing values, can be used for checking the unique existence of values. This structure is not iterable and can only be used for lookups. | | `UnorderedSet` | `Set` | An iterable equivalent of `LookupSet` which stores additional metadata for the elements contained in the set. | +| `LookupMap` | `Map` | This structure behaves as a thin wrapper around the key-value storage available to contracts. This structure does not contain any metadata about the elements in the map, so it is not iterable. | +| `UnorderedMap` | `Map` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | -:::info Note +| SDK collection | `std` equivalent | Description | +|-----------------------------------------------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `store::Vector` | `Vec` | A growable array type. The values are sharded in memory and can be used for iterable and indexable values that are dynamically sized. | +| store::LookupMap`` | HashMap`` | This structure behaves as a thin wrapper around the key-value storage available to contracts. This structure does not contairn any metadata about the elements in the map, so it is not iterable. | +| store::IterableMap`` | HashMap`` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | +| store::UnorderedMap`` | HashMap`` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | +| `store::LookupSet` | `HashSet` | A set, which is similar to `LookupMap` but without storing values, can be used for checking the unique existence of values. This structure is not iterable and can only be used for lookups. | +| `store::IterableSet` | `HashSet` | An iterable equivalent of `LookupSet` which stores additional metadata for the elements contained in the set. | +| `store::UnorderedSet` | `HashSet` | An iterable equivalent of `LookupSet` which stores additional metadata for the elements contained in the set. | -The `near_sdk::collections` will be moving to `near_sdk::store` and have updated APIs. If you would like to access these updated structures as they are being implemented, enable the `unstable` feature on `near-sdk`. + -::: + + +:::info Note +The `near_sdk::collections` is now deprecated in favor of `near_sdk::store`. To use `near_sdk::collections` you will have to use the [`legacy` feature](https://github.com/near-examples/storage-examples/blob/2a138a6e8915e08ce76718add3e36c04c2ea2fbb/collections-rs/legacy/Cargo.toml#L11). -| SDK collection | `std` equivalent | Description | -|----------------------------------------|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `LazyOption` | `Option` | Optional value in storage. This value will only be read from storage when interacted with. This value will be `Some` when the value is saved in storage, and `None` if the value at the prefix does not exist. | -| `Vector` | `Vec` | A growable array type. The values are sharded in memory and can be used for iterable and indexable values that are dynamically sized. | -| LookupMap`` | HashMap`` | This structure behaves as a thin wrapper around the key-value storage available to contracts. This structure does not contain any metadata about the elements in the map, so it is not iterable. | -| UnorderedMap`` | HashMap`` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | -| TreeMap`` | BTreeMap`` | An ordered equivalent of `UnorderedMap`. The underlying implementation is based on an [AVL tree](https://en.wikipedia.org/wiki/AVL_tree). This structure should be used when a consistent order is needed or accessing the min/max keys is needed. | -| `LookupSet` | `HashSet` | A set, which is similar to `LookupMap` but without storing values, can be used for checking the unique existence of values. This structure is not iterable and can only be used for lookups. | -| `UnorderedSet` | `HashSet` | An iterable equivalent of `LookupSet` which stores additional metadata for the elements contained in the set. | +::: +| SDK collection | `std` equivalent | Description | +|----------------------------------------------------|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `collections::Vector` | `Vec` | A growable array type. The values are sharded in memory and can be used for iterable and indexable values that are dynamically sized. | +| collections::LookupMap`` | HashMap`` | This structure behaves as a thin wrapper around the key-value storage available to contracts. This structure does not contairn any metadata about the elements in the map, so it is not iterable. | +| collections::UnorderedMap`` | HashMap`` | Similar to `LookupMap`, except that it stores additional data to be able to iterate through elements in the data structure. | +| collections::TreeMap`` | BTreeMap`` | An ordered equivalent of `UnorderedMap`. The underlying implementation is based on an [AVL tree](https://en.wikipedia.org/wiki/AVL_tree). This structure should be used when a consistent order is needed or accessing the min/max keys is needed. | +| `collections::LookupSet` | `HashSet` | A set, which is similar to `LookupMap` but without storing values, can be used for checking the unique existence of values. This structure is not iterable and can only be used for lookups. | +| `collections::UnorderedSet` | `HashSet` | An iterable equivalent of `LookupSet` which stores additional metadata for the elements contained in the set. | +| `collections::LazyOption` | `Option` | Optional value in storage. This value will only be read from storage when interacted with. This value will be `Some` when the value is saved in storage, and `None` if the value at the prefix does not exist. | @@ -131,8 +140,11 @@ The `near_sdk::collections` will be moving to `near_sdk::store` and have updated | `Vector` | ✅ | ✅ | ✅ | ✅ | | `LookupSet` | | | | | | `UnorderedSet` | ✅ | ✅ | | ✅ | +| `IterableSet` | ✅ | ✅ | | ✅ | | `LookupMap` | | | | | | `UnorderedMap` | ✅ | ✅ | | ✅ | +| `IterableMap` | ✅ | ✅ | | ✅ | +| `TreeMap` | ✅ | ✅ | ✅ | ✅ |
@@ -143,8 +155,9 @@ The `near_sdk::collections` will be moving to `near_sdk::store` and have updated | `Vector` | O(1) | O(1)\* | O(1)\*\* | O(n) | O(n) | O(n) | | `LookupSet` | O(1) | O(1) | O(1) | O(1) | N/A | N/A | | `UnorderedSet` | O(1) | O(1) | O(1) | O(1) | O(n) | O(n) | +| `IterableSet` | O(1) | O(1) | O(1) | O(1) | O(n) | O(n) | | `LookupMap` | O(1) | O(1) | O(1) | O(1) | N/A | N/A | -| `UnorderedMap` | O(1) | O(1) | O(1) | O(1) | O(n) | O(n) | +| `IterableMap` | O(1) | O(1) | O(1) | O(1) | O(n) | O(n) | | `TreeMap` | O(1) | O(log n) | O(log n) | O(log n) | O(n) | O(n) | _\* - to insert at the end of the vector using `push_back` (or `push_front` for deque)_ @@ -152,7 +165,11 @@ _\*\* - to delete from the end of the vector using `pop` (or `pop_front` for deq --- -## Collections Cookbook +## SDK Collections Cookbook + +Let's see how to use the SDK collections in practice + +
### Instantiation @@ -160,32 +177,45 @@ All structures need to be initialized using a **unique `prefix`**, which will be - + + + +:::tip + +Do not forget to use the `schema` to define how your contract's state is structured + +::: + + url="https://github.com/near-examples/storage-examples/blob/main/collections-rs/store/src/lib.rs" + start="24" end="47"/> :::tip - Notice how we use `enums` to ensure all collections have a different prefix. Moreover, `enums` are very efficient since they get serialized into a single `byte` prefix. + Notice how we use `enums` to ensure all collections have a different prefix. Another advantage of using `enums` is that they are serialized into a single `byte` prefix. ::: - + + -:::warning + :::tip -Because the values are not kept in memory and are lazily loaded from storage, it's important to make sure if a collection is replaced or removed, that the storage is cleared. In addition, it is important that if the collection is modified, the collection itself is updated in state because most collections will store some metadata. + Notice how we use `enums` to ensure all collections have a different prefix. Another advantage of using `enums` is that they are serialized into a single `byte` prefix. -::: + ::: + + + :::danger @@ -199,60 +229,94 @@ Be careful of not using the same prefix in two collections, otherwise, their sto Implements a [vector/array](https://en.wikipedia.org/wiki/Array_data_structure) which persists in the contract's storage. Please refer to the Rust and JS SDK's for a full reference on their interfaces. - - - - - - - - - + + + + + + + + + + +
-### Map +### LookupMap Implements a [map/dictionary](https://en.wikipedia.org/wiki/Associative_array) which persists in the contract's storage. Please refer to the Rust and JS SDK's for a full reference on their interfaces. - - - - - - - - - + + + + + + + +
-### Set +### UnorderedMap / IterableMap + +Implements a [map/dictionary](https://en.wikipedia.org/wiki/Associative_array) which persists in the contract's storage. Please refer to the Rust and JS SDK's for a full reference on their interfaces. + + + + + + + + + + +
+ +### LookupSet Implements a [set](https://en.wikipedia.org/wiki/Set_(abstract_data_type)) which persists in the contract's storage. Please refer to the Rust and JS SDK's for a full reference on their interfaces. - - - - - - - - - - + + + + + + + + + + +
+ +### UnorderedSet / IterableSet + +Implements a [map/dictionary](https://en.wikipedia.org/wiki/Associative_array) which persists in the contract's storage. Please refer to the Rust and JS SDK's for a full reference on their interfaces. + + + + + + + + +
@@ -260,23 +324,18 @@ Implements a [set](https://en.wikipedia.org/wiki/Set_(abstract_data_type)) which An ordered equivalent of Map. The underlying implementation is based on an [AVL](https://en.wikipedia.org/wiki/AVL_tree). You should use this structure when you need to: have a consistent order, or access the min/max keys. - - - - - - + + + + +
-### `LazyOption` +### LazyOption (Legacy) -It's a type of persistent collection that only stores a single value. -The goal is to prevent a contract from deserializing the given value until it's needed. -An example can be a large blob of metadata that is only needed when it's requested in a view call, -but not needed for the majority of contract operations. +LazyOptions are great to store large values (i.e. a wasm file), since its value will not be read from storage until it is interacted with. It acts like an `Option` that can either hold a value or not and also requires a unique prefix (a key in this case) like other persistent collections. @@ -287,90 +346,26 @@ Compared to other collections, `LazyOption` only allows you to initialize the va ## Nesting Collections -It is possible to nest collections. When nesting SDK collections, remember to **assign different prefixes to all collections** (including the nested ones). +When nesting SDK collections, be careful to **use different prefixes** for all collections, including the nested ones. - While you can create nested maps, you first need to construct or deconstruct the structure from state. This is a temporary solution that will soon be automatically handled by the SDK. - - ```ts - import { NearBindgen, call, view, near, UnorderedMap } from "near-sdk-js"; - - @NearBindgen({}) - class StatusMessage { - records: UnorderedMap; - constructor() { - this.records = new UnorderedMap("a"); - } - - @call({}) - set_status({ message, prefix }: { message: string; prefix: string }) { - let account_id = near.signerAccountId(); - - const inner: any = this.records.get("b" + prefix); - const inner_map: UnorderedMap = inner - ? UnorderedMap.deserialize(inner) - : new UnorderedMap("b" + prefix); - - inner_map.set(account_id, message); - - this.records.set("b" + prefix, inner_map); - } - - @view({}) - get_status({ account_id, prefix }: { account_id: string; prefix: string }) { - const inner: any = this.records.get("b" + prefix); - const inner_map: UnorderedMap = inner - ? UnorderedMap.deserialize(inner) - : new UnorderedMap("b" + prefix); - return inner_map.get(account_id); - } - } - ``` + - In Rust the simplest way to avoid collisions between nested collections is by using `enums` + - ```rust - use near_sdk::borsh::{self, BorshDeserialize, BorshSerialize}; - use near_sdk::collections::{UnorderedMap, UnorderedSet}; - use near_sdk::{env, near, AccountId, BorshStorageKey, CryptoHash}; - - #[near(contract_state)] - pub struct Contract { - pub accounts: UnorderedMap>, - } - - impl Default for Contract { - fn default() -> Self { - Self { - accounts: UnorderedMap::new(StorageKeys::Accounts), - } - } - } - - #[near(serializers = [borsh])] - pub enum StorageKeys { - Accounts, - SubAccount { account_hash: CryptoHash }, - } - - #[near] - impl Contract { - pub fn get_tokens(&self, account_id: &AccountId) -> Vec { - let tokens = self.accounts.get(account_id).unwrap_or_else(|| { - UnorderedSet::new(StorageKeys::SubAccount { - account_hash: env::sha256_array(account_id.as_bytes()), - }) - }); - tokens.to_vec() - } - } - ``` + :::tip + + Notice how we use `enums` that take a `String` argument to ensure all collections have a different prefix + + ::: @@ -384,13 +379,6 @@ Because the values are not kept in memory and are lazily loaded from storage, it Some error-prone patterns to avoid that cannot be restricted at the type level are: - - - - - - - @@ -446,9 +434,9 @@ assert!( // as reads, and the writes are performed on [`Drop`](https://doc.rust-lang.org/std/ops/trait.Drop.html) // so if the collection is kept in static memory or something like `std::mem::forget` is used, // the changes will not be persisted. -use near_sdk::store::LookupSet; +use near_sdk::store::IterableSet; -let mut m: LookupSet = LookupSet::new(b"l"); +let mut m: IterableSet = IterableSet::new(b"l"); m.insert(1); assert!(m.contains(&1)); @@ -456,7 +444,7 @@ assert!(m.contains(&1)); // m.flush(); std::mem::forget(m); -m = LookupSet::new(b"l"); +m = IterableSet::new(b"l"); assert!(!m.contains(&1)); ``` @@ -470,7 +458,7 @@ Some issues for more context: - https://github.com/near/near-sdk-rs/issues/560 - https://github.com/near/near-sdk-rs/issues/703 -The following cases are the most commonly encountered bugs that cannot be restricted at the type level: +The following cases are the most commonly encountered bugs that cannot be restricted at the type level (only relevant for `near_sdk::collections`, not `near_sdk::store`): ```rust use near_sdk::borsh::{self, BorshSerialize}; @@ -502,8 +490,7 @@ let n = root.get(&1).unwrap(); assert!(n.is_empty()); assert!(n.contains(&"test".to_string())); -// Bug 2 (only relevant for `near_sdk::collections`, not `near_sdk::store`): Nested -// collection is modified without updating the collection itself in the outer collection. +// Bug 2: Nested collection is modified without updating the collection itself in the outer collection. // // This is fixed at the type level in `near_sdk::store` because the values are modified // in-place and guarded by regular Rust borrow-checker rules. @@ -527,35 +514,53 @@ assert!(n.contains(&"some value".to_string())); ## Pagination -Persistent collections such as `UnorderedMap`, `UnorderedSet` and `Vector` may +Persistent collections such as `IterableMap/UnorderedMap`, `IterableSet/UnorderedSet` and `Vector` may contain more elements than the amount of gas available to read them all. In order to expose them all through view calls, we can use pagination. -This can be done using iterators with [`Skip`](https://doc.rust-lang.org/std/iter/struct.Skip.html) and [`Take`](https://doc.rust-lang.org/std/iter/struct.Take.html). This will only load elements from storage within the range. -Example of pagination for `UnorderedMap`: -```rust -#[near(contract_state)] -#[derive(PanicOnDefault)] -pub struct Contract { - pub status_updates: UnorderedMap, -} + + + With JavaScript this can be done using iterators with [`toArray`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Iterator/toArray) and [`slice`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice). -#[near] -impl Contract { - /// Retrieves multiple elements from the `UnorderedMap`. - /// - `from_index` is the index to start from. - /// - `limit` is the maximum number of elements to return. - pub fn get_updates(&self, from_index: usize, limit: usize) -> Vec<(AccountId, String)> { - self.status_updates - .iter() - .skip(from_index) - .take(limit) - .collect() - } -} -``` + ```ts + /// Returns multiple elements from the `UnorderedMap`. + /// - `from_index` is the index to start from. + /// - `limit` is the maximum number of elements to return. + @view({}) + get_updates({ from_index, limit }: { from_index: number, limit:number }) { + return this.status_updates.toArray().slice(from_index, limit); + } + ``` + + + + With Rust this can be done using iterators with [`Skip`](https://doc.rust-lang.org/std/iter/struct.Skip.html) and [`Take`](https://doc.rust-lang.org/std/iter/struct.Take.html). This will only load elements from storage within the range. + + ```rust + #[near(contract_state)] + #[derive(PanicOnDefault)] + pub struct Contract { + pub status_updates: IterableMap, + } + + #[near] + impl Contract { + /// Retrieves multiple elements from the `IterableMap`. + /// - `from_index` is the index to start from. + /// - `limit` is the maximum number of elements to return. + pub fn get_updates(&self, from_index: usize, limit: usize) -> Vec<(AccountId, String)> { + self.status_updates + .iter() + .skip(from_index) + .take(limit) + .collect() + } + } + ``` + + ---