Setup for ETH-implicit accounts #10020

staffik · 2023-10-27T14:37:56Z

Part of #10018.

This PR introduces some changes from #9969 laying groundwork for real protocol changes to be done in a separate PR.
Some changes might look redundant (especially match receiver_id.get_account_type() { ...) but they will make it easier to read changes done in the next PR.

Changes done in near-account-id 1.0.0-alpha.2:

Add AccountType enum: NamedAccount, NearImplicitAccount, or EthImplicitAccount.
Parse 40 characters long hexadecimal addresses prefixed with '0x' as EthImplicitAccount.
AccountType::is_implicit() returns true for both NearImplicitAccount and EthImplicitAccount.

Summary of this PR:

Bump version of near-account-id to 1.0.0-alpha.2.
Do not use is_implicit() for now, as we do not want to treat EthImplicitAccount as implicit as for now.
Add derive_account_id_from_public_key (aka derive_near_implicit_account_id) function that currently only supports ED25519 key type and returns hex-encoded copy of the key.
Refactor/rename tests a bit.

core/account-id/src/lib.rs

birchmd

Looks like a good start to me. Just a couple of nit picks

runtime/runtime/src/actions.rs

core/primitives/src/utils.rs

wacban

Looks great! I have a few nits and questions but you're really close.

core/account-id/src/lib.rs

integration-tests/src/tests/standard_cases/mod.rs

runtime/near-vm-runner/src/logic/logic.rs

runtime/runtime/src/actions.rs

tools/mirror/src/key_mapping.rs

runtime/near-vm-runner/src/logic/logic.rs

wacban

Thanks a lot!
One final request and let's have someone from runtime have a quick look and then we should be ready to go.

core/primitives-core/src/runtime/fees.rs

wacban

LGTM!

core/primitives-core/src/runtime/fees.rs

staffik · 2023-11-02T08:46:54Z

This commit removed core/account-id and it is now under https://github.com/near/near-account-id-rs. I created a PR: near/near-account-id-rs#14 in the second repo and it probably should be merged first.

…unt_id

staffik · 2023-11-06T07:42:29Z

Cargo.lock

@@ -3350,9 +3350,9 @@ dependencies = [

 [[package]]
 name = "near-account-id"
-version = "1.0.0-alpha.1"
+version = "1.0.0-alpha.2"


1.0.0-alpha.2 now has some changes that previously have been in this PR, see near/near-account-id-rs#14

staffik · 2023-11-06T07:51:07Z

runtime/runtime/src/actions.rs

-                if config.wasm_config.implicit_account_creation && account_id.is_implicit() {
-                    // If the account doesn't exist and it's 64-length hex account ID, then you
+                if config.wasm_config.implicit_account_creation
+                    // TODO(eth-implicit) Change back to is_implicit() when ETH-implicit accounts are supported.


In this and several other places, I changed account_id.get_account_type().is_implicit() to account_id.get_account_type() == AccountType::NearImplicitAccount. This is because in the new version of near-account-id we now have that is_implicit() returns true for EthImplicitAccount, but we want to treat EthImplicitAccount as NamedAccount in this PR. See PR discussion.

staffik · 2023-11-06T07:55:40Z

@akhi3030 @wacban Can you take a look?

akhi3030 · 2023-11-06T08:22:37Z

@akhi3030 @wacban Can you take a look?

@staffik: I don't think I have sufficient context to review this. I'll delegate to @wacban. He is a codeowner so his approval should be sufficient.

Let me know if you have specific questions you want me input on though.

wacban

LGTM

wacban · 2023-11-06T10:22:38Z

integration-tests/src/tests/client/features/delegate_action.rs

+        AccountType::EthImplicitAccount => {
+            PublicKey::from_seed(KeyType::SECP256K1, sender.as_ref())
+        }


What does this do? Please add a TODO if you need to change it later.

@wacban It is to return a public key based on account address. Now I think it is confusing here, as well as eth_implicit_test_account in test_utils. I am removing it, it is currently unused in integration tests and it will make more sense in the next PR.

## Context Tracking issue: #10018. Design: near/NEPs#518 (comment). ### Goal We want the NEAR Protocol's ecosystem to be closer with Ethereum by integrating Web3 wallet support. To accomplish that, some changes on the protocol level are needed with an emphasis on user experience continuity. Following the design, the main change on the protocol level would be to have ETH-style addresses managed by a new `Wallet Contract` integrated into the protocol. For seamless onboarding of EVM users, the contract will be free of charge. ### Previous work This PR is built on top of a [preparatory PR](#10020). It reintroduces some changes from a [closed PR](#10056) that follows an old design. ## Summary On a transfer to an ETH-implicit address: 1. An account is created at this address. 2. The account does not have any access key added. Thus, it is not possible to make a transaction from this account directly. 3. The Wallet Contract (as a part of the protocol) is deployed to the account. For this PR, an empty smart contract is used as a placeholder. ### Changes - On a transfer to an ETH-implicit address, if an account does not exist yet at this address: - Create account at this address. - Deploy a `Wallet Contract` placeholder (empty contract now) to this account. - Update fee for a transfer to an ETH-implicit address to include account creation cost. `Wallet Contract` deployment is free of charge (user only pays for an increased storage). - Tests: - Test whether it is possible to create an ETH-implicit account by making a transfer to that address, and whether further funds can be sent to this account. - Test that no access key is added to an ETH-implicit account (the account is locked) and in consequence no transactions are possible from this account. - Guard the changes with `EthImplicitAccounts` protocol feature and `eth_implicit_accounts` runtime config flag. ## Next work - Discuss how the `Wallet Contract` should be integrated into the protocol for efficiency. - Use the `Wallet Contract` placeholder implementation that allows to transfer funds only.

## Context NEP: near/NEPs#518 Tracking issue: #10018. ### Goal We want the NEAR Protocol's ecosystem to be closer with Ethereum by integrating Web3 wallet support. To accomplish that, some changes on the protocol level are needed with an emphasis on user experience continuity. Following the design, the main change on the protocol level would be to have ETH-style addresses managed by a new `Wallet Contract` integrated into the protocol. For seamless onboarding of EVM users, the contract will be free of charge. ### Previous work This PR is built on top of two PRs: * #10020: no-op PR, laying groundwork for further changes. * #10224: empty contract as a placeholder, literally deployed to new ETH accounts. ## Summary This PR adds `near-wallet-contract` crate (based on `runtime/near-test-contracts`) that exposes `Wallet Contract` WASM code to the runtime. It stops deploying a copy of the `Wallet Contract` to each newly created ETH-implicit account. Instead, it uses an in-memory cached contract code on `execute_function_call` action from such account. ### Changes - Add `wallet-contract` crate (separated from the workspace) with placeholder `Wallet Contract` implementation. - Add `near-wallet-contract` crate (part of the workspace) that generates (through `build.rs`) and exposes the `Wallet Contract` WASM code. - Do not literally deploy `Wallet Contract` when creating ETH-implicit account. Just store reference to the `Wallet Contract`. - Treat ETH-implicit accounts specially when retrieving contract code from an account (in such case returns in-memory cached `Wallet Contract` code). - Add tests calling `Wallet Contract` where a transfer from an ETH-implicit account is possible depending on the public key passed with `rlp_transaction` argument.

* build fuzzers on Big machines (#10252) See eg. the likely-OOM at https://github.com/near/nearcore/actions/runs/6890897651/job/18744849948 * test: add tests for SnapshotHostInfo broadcast behavior (#10204) Add a few tests which test that `SnapshotHostInfo` is properly broadcast between peers. `SnapshotHostInfo` lets other peers know that a certain peer has a state snapshot at a specific point in time, which is later used for state sync. Nodes publish this information periodically and it should be broadcast to all nodes in the network. Fixes: #10172 * allow warnings for fuzzer building (#10253) Fuzzer building runs on rustc nightly, which can have a different set of warnings than rustc stable. Rather than introduce a new compilation step for all the PRs, just compile fuzzers without caring about warnings. Fixes a recent fuzzer building issue, probably caused by a nightly upgrade: https://github.com/near/nearcore/actions/runs/6929988512/job/18848791016 * fix: use header_head to determine validators for forwarding transactions (#10256) Functions `find_chunk_producer_for_forwarding()`, `find_validator_for_forwarding()` are not used much, one calls another, and they both call `chain.head()`. That seems unnecessary. Refactored to call `get_chunk_producer()` directly, and not worry about 3 layers of functions using different types of heads. See https://github.com/near/nearcore/pull/10251 for context * drop buildkite pipeline (#10227) We have now fully migrated off buildkite, so let’s remove the old pipeline from the repository :) * configurable tx routing horizon (#10251) Community reports that sometimes it takes minutes for a transaction to be recorded on chain. My guess is that it takes that much time to record a transaction on chain, because it gets forwarded to nodes which either: * fail to produce their chunks * don't have time to process incoming tx RPCs before producing their chunks * the node producing a tx is out of sync and the set of validators is already too late All of these can be fixed by multicasting transactions further into the future. This PR is the simplest attempt to send transactions further into the future. Tested: Had a testnet node and submitted a tx that was supposed to be forwarded to chunk producers of the next 100 blocks. In practice it gets forwarded to 5 validators. Sending a message to a validator takes about 100µs for a trivial tx. * refactor: more work removing `num_shards()` (#10255) Building on top of https://github.com/near/nearcore/pull/10249, further reduces use of `num_shards()` in chain.rs. Also removes some "premature" optimisations that I don't think would necessarily help in production and just add complexity to the code. * fix(config): disable epoch sync, enable state sync (#10260) * Simplifies `state_sync_enabled` and `enable_multiline_logging`. * Adjusts {min,max}_block_production_delay config options when generating configs for testnet and mainnet. * Simplifies config initialization logic. * Deletes betanet and stakewars * bump openssl (#10262) Fixes https://rustsec.org/advisories/RUSTSEC-2023-0072, reported by the cargo-audit CI job * add a Justfile for reproducing CI locally, use it for CI (#10235) * feat: support partial mandates in chunk validator assignment (#10241) Adds support for partial mandates in chunk validator assignment. Currently only full mandates are considered. Assume `stake_per_mandate = 5`, then: - Valdiator `V1` with stake 12 holds 2 full mandates corresponding to a stake of 10. The remaining stake of 2 does not participate in validation. - Validator `V2` with stake 4 holds 0 full mandates and does not participate in validation at all. With support for partial mandates `V1` gets a partial mandate with weight 2 and `V2` gets a partial mandate of 4. So partial mandates allow to increase the number of validators and allows them to commit their entire stake to validation. ### Probability of shard corruption In [these simulations](https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/chunk.20validator.20assignment.20simulation.20with.20partial.20seats), the probability of shard corruption decreases as partial seats are included. In general, the effect of including partial seats depends on the distribution of (malicious) stake across validators and the stake required per mandate. ### `ValidatorMandatesAssignment` This type is introduced to more clearly document the result of sampling chunk validators and to avoid passing around a raw `Vec<HashMap<ValidatorId, _>>`. ### Shuffling of shard ids [This thread](https://github.com/mooori/sim-validator-assignment/pull/12#discussion_r1389895314) points out a bias towards small shard ids and brings up shard id shuffling as a way to fix it. For now shard ids are not shuffled in this PR, since I think it would make the diff harder to review. I would suggest to introduce shard id shuffling in a separate PR. This should not pose a risk as chunk validator assignment is not enabled in production. * [Snyk] Upgrade @types/react-dom from 18.2.14 to 18.2.15 (#10270) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/react-dom from 18.2.14 to 18.2.15.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **1 version** ahead of your current version. - The recommended version was released **22 days ago**, on 2023-11-07. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/react-dom</summary> <ul> <li> 18.2.15 - 2023-11-07 </li> <li> 18.2.14 - 2023-10-18 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react-dom GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI5MDM2MzdmYS0yNGU2LTRiZjktYjY1NC1hYzYzNzFiMDY5MDEiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjkwMzYzN2ZhLTI0ZTYtNGJmOS1iNjU0LWFjNjM3MWIwNjkwMSJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react-dom&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> * feat: use the < operator instead of cmp method to improve readability (#10268) The following code is currently used for comparing `boundary_account` and `account_id`: ```rust if boundary_account.cmp(account_id) == Greater { ... } ``` IMO it's a bit confusing - it isn't immediately obvious whether this means `boundary_account` > `account_id` or `account_id` > `boundary_account`. I misread this line and because of that I made a mistake in #10240. Let's change it to something that is easier to read: ```rust if account_id < boundary_account { ... } ``` * feat: add database analyse-gas-usage command (#10240) Add a command which allows to analyse gas usage on some part of the blockchain. The command goes over the specified blocks and gathers information about how much gas was used by each block, shard and account. Then it displays an analysis of all the data. The analysis contains: * Total gas used * Gas used on each shard * Top 10 accounts by gas usage * The optimal account by which a shard could be split into two halves with similar gas usage Help page: ```bash $ ./neard database analyse-gas-usage --help Analyse gas usage in a chosen sequnce of blocks Usage: neard database analyse-gas-usage [OPTIONS] Options: --last-blocks <LAST_BLOCKS> Analyse the last N blocks in the blockchain --from-block-height <FROM_BLOCK_HEIGHT> Analyse blocks from the given block height, inclusive --to-block-height <TO_BLOCK_HEIGHT> Analyse blocks up to the given block height, inclusive -h, --help Print help ``` Example usage: ```bash # Analyse gas usage in the last 1000 blocks cargo run --bin neard -- database analyse-gas-usage --last-blocks 1000 # Analyse gas usage in blocks with heights between 200 and 300 (inclusive) ./neard database analyse-gas-usage --from-block-height 200 --to-block-height 300 ``` * refactor: replace all remaining uses of `num_shards()` with `shard_ids()` (#10271) Closes https://github.com/near/nearcore/issues/10237 * Update resharding.md (#10264) A pretty heavy handed rewrite of the resharding documentation. I'm open to some heavy handed suggestions ;) I'm not sure what's the best trade off between duplicating some content from the NEPs here and keeping in minimal with just links. There is some new content here, mainly the rollout section, I would love to hear some thoughts about this in particular from @gmilescu and @posvyatokum. * introduce hack for fuzzers building (#10276) This introduces a hack in our fork of cargo-bolero, needed for proper fuzzer building. The reason is still the same as the patch previously applied: the current bolero way of auto-detecting fuzzers is suboptimal. All these patches should be able to go away with some work on cargo-bolero, but we have other priorities, and this hack seems to make the tarball builds work locally at least. With this, we run with two patches: - https://github.com/Ekleog-NEAR/bolero/commit/877f19959e708f288e2e3c814768743f86f69c80, which is a bit hacky but might deserve being included upstream were it not for backward compatibility reasons - https://github.com/Ekleog-NEAR/bolero/commit/56da8e6d1d018519a30b36d85d3a53fe35a42eaf, which is a real, bad hack, but seems to be enough to stop having to care about it for now, until we can go back to it and fix the underlying issue that fuzzer listing is currently not well-implemented. Its consequence is, any fuzzer in the `neard` crate will not actually run on our fuzzing infrastructure. We’ll have to fix it at some point. * refactor: move runtime tests into a separate file (#10274) This PR moves NightshadeRuntime tests module into a separate file without changing anything else. See https://github.com/near/nearcore/issues/10275 for more context. In case having tests as a separate file is a good idea for the following reasons: - these are mostly integration-style tests with heavy setup (see `TestEnv`) and quite a lot of code, so to me it seems like it deserves to be in a separate file - the code is already in `mod.rs` file, so we don't need to create a separate directory structure. * [ChunkValidation] Remove the feature flag protocol_feature_chunk_validation. (#10257) We discussed as a team earlier that we don't need these feature flags. Removing it to simplify further development of chunk validation. * Chore: Removed unused imports/exports (#10288) * crypto: simplify split usage [trivial] (#10285) * [Snyk] Upgrade @types/react from 18.2.25 to 18.2.37 (#10284) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/react from 18.2.25 to 18.2.37.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **12 versions** ahead of your current version. - The recommended version was released **25 days ago**, on 2023-11-07. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/react</summary> <ul> <li> 18.2.37 - 2023-11-07 </li> <li> 18.2.36 - 2023-11-06 </li> <li> 18.2.35 - 2023-11-05 </li> <li> 18.2.34 - 2023-11-01 </li> <li> 18.2.33 - 2023-10-26 </li> <li> 18.2.32 - 2023-10-25 </li> <li> 18.2.31 - 2023-10-20 </li> <li> 18.2.30 - 2023-10-19 </li> <li> 18.2.29 - 2023-10-18 </li> <li> 18.2.28 - 2023-10-10 </li> <li> 18.2.27 - 2023-10-09 </li> <li> 18.2.26 - 2023-10-09 </li> <li> 18.2.25 - 2023-10-04 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiJkYjA5OTM4YS0zZGY5LTRkMTYtYTM5MS00MDkyMzVkNTllMDUiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6ImRiMDk5MzhhLTNkZjktNGQxNi1hMzkxLTQwOTIzNWQ1OWUwNSJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> * [Snyk] Security upgrade react-query from 3.39.3 to 4.0.0 (#10282) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to fix one or more vulnerable packages in the `npm` dependencies of this project.</h3> #### Changes included in this PR - Changes to the following files to upgrade the vulnerable dependencies to a fixed version: - tools/debug-ui/package.json - tools/debug-ui/package-lock.json #### Vulnerabilities that will be fixed ##### With an upgrade: Severity | Priority Score (*) | Issue | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:------------------------- ![high severity](https://res.cloudinary.com/snyk/image/upload/w_20,h_20/v1561977819/icon/h.png "high severity") | **661/1000** **Why?** Recently disclosed, Has a fix available, CVSS 7.5 | Missing Release of Resource after Effective Lifetime [SNYK-JS-INFLIGHT-6095116](https://snyk.io/vuln/SNYK-JS-INFLIGHT-6095116) | Yes | No Known Exploit (*) Note that the real score may have changed since the PR was raised. <details> <summary>Commit messages</summary> <details> <summary>Package name: react-query</summary> The new version differs by 1 commits. <ul> <li><a href="https://snyk.io/redirect/github/TanStack/query/commit/357ec041a6fcc4a550f3df02c12ecc7bcdefbc05">357ec04</a> v4 release (#3842)</li> </ul> <a href="https://snyk.io/redirect/github/TanStack/query/compare/ef684205cb4890db904db5b387513fb9042e0bb6...357ec041a6fcc4a550f3df02c12ecc7bcdefbc05">See the full diff</a> </details> </details> Check the changes in this PR to ensure they won't cause issues with your project. ------------ **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiIyMzRmMTE2Yi0wYWFjLTRkOTMtYWRlZi0zMjNhMDgyMTQ2MzAiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjIzNGYxMTZiLTBhYWMtNGQ5My1hZGVmLTMyM2EwODIxNDYzMCJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr) 🛠 [Adjust project settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr/settings) 📚 [Read more about Snyk's upgrade and patch logic](https://support.snyk.io/hc/en-us/articles/360003891078-Snyk-patches-to-fix-vulnerabilities) [//]: # (snyk:metadata:{"prId":"234f116b-0aac-4d93-adef-323a08214630","prPublicId":"234f116b-0aac-4d93-adef-323a08214630","dependencies":[{"name":"react-query","from":"3.39.3","to":"4.0.0"}],"packageManager":"npm","projectPublicId":"98480bdc-d80b-4fd1-89d7-c4c56a706763","projectUrl":"https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr","type":"auto","patch":[],"vulns":["SNYK-JS-INFLIGHT-6095116"],"upgrade":["SNYK-JS-INFLIGHT-6095116"],"isBreakingChange":true,"env":"prod","prType":"fix","templateVariants":["updated-fix-title","priorityScore"],"priorityScoreList":[661],"remediationStrategy":"vuln"}) --- **Learn how to fix vulnerabilities with free interactive lessons:** 🦉 [Learn about vulnerability in an interactive lesson of Snyk Learn.](https://learn.snyk.io/?loc=fix-pr) Co-authored-by: snyk-bot <[email protected]> * build(deps-dev): bump @adobe/css-tools from 4.3.1 to 4.3.2 in /tools/debug-ui (#10272) Bumps [@adobe/css-tools](https://github.com/adobe/css-tools) from 4.3.1 to 4.3.2. <details> <summary>Changelog</summary> Sourced from <a href="https://github.com/adobe/css-tools/blob/main/History.md"><code>@adobe/css-tools</code>'s changelog</a>. <blockquote> <h1>4.3.2 / 2023-11-28</h1> <ul> <li>Fix redos vulnerability with specific crafted css string - CVE-2023-48631</li> <li>Fix Problem parsing with :is() and nested :nth-child() <a href="https://redirect.github.com/adobe/css-tools/issues/211">#211</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/adobe/css-tools/commits">compare view</a></li> </ul> </details> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=@adobe/css-tools&package-manager=npm_and_yarn&previous-version=4.3.1&new-version=4.3.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/near/nearcore/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update resharding.md (#10281) Added some metrics from tests in mainnet and testnet. ref data mainnet https://nearinc.grafana.net/goto/aFBwfLHSg?orgId=1 ref data testnet https://nearinc.grafana.net/goto/XgBUBYNSg?orgId=1 * fix: Document HeaderSync (#10280) Made no functional changes except: * Replaced a tuple with a struct * Introduced a function `made_enough_progress()` to make the code much simpler to understand. * Renamed a ban reason * fix: format tcp connection error using `Display` (#10286) Reported by @wacban. Debug printing results in multiline strings, which are hard to interpret. It also breaks tooling one might want to use to analyze logs. Example: ``` 2023-12-01T10:22:08.186809Z INFO network: failed to connect to ed25519:[email protected]:54063 result=Err(tcp::Stream::connect() Caused by: deadline has elapsed) ``` I believe this is coming from the default `Debug` impl for `anyhow::Error`. Using `Display` impl should fix it as it concatenates the context via `:` instead. --------- Co-authored-by: nikurt <[email protected]> * refactor: remove some more usage of `num_shards()` (#10291) See https://github.com/near/nearcore/issues/10237. * feat: limit the number of shards ids in SnapshotHostInfo (#10294) `SnapshotHostInfo` contains a list of shard ids for which a peer has a snapshot. Under normal circumstances this list is pretty small - the maximum size would be the total number of shards. But a malicious peer could craft a `SnapshotHostInfo` message with very large number of shards - millions of shard ids. This could cause problems on the receiver node, so there has to be limit on the number of shard ids in a message. Let's limit the number of shard ids in a single message to `MAX_SHARDS_PER_SNAPSHOT_HOST_INFO = 512`. It's a constant that describes how many shards a single peer can have snapshots for. A peer doesn't keep more state snapshots than the number of tracked shards, so this limit is reasonable. 512 shards ought to be enough for anyone. In an ideal world we could check the current number of shards and reject messages that contain more shard ids than the current number, but sadly this can't be implemented. The problem is that the receiving node might be behind the rest of the blockchain, and the latest information just isn't available, so it can't check what the current number of shards is. We could reject messages in such situations, but this would lead to loss of information. Limiting the number of shard ids to a constant number is an okay alternative. * feat(resharding): dynamic config (#10290) make the resharding config dynamic * Remove rust-version.workspace = true from published crates (#10292) This comes at the request of dev-hub, where dependencies downstream have been having issues with our aggressive rust-version policy. Supporting only the latest rust-version is not enough for them. As a consequence, we’re removing rust-version here. This sounds like a more semantically reasonable use for rust-version anyway: a crate without rust-version set should be assumed to work with the latest stable version anyway, and maybe more without guarantees… which is exactly our guarantees. Also update the themis tool to make sure we don’t make this mistake again. * [resharding] Add pytest for checking RPC calls after resharding (#10296) This PR adds a basic pytest to check if the RPC calls to tx endpoint work after a resharding event. Note that this test currently works under the assumption that the node is tracking all shards and future improvements to the tests where we may need to redirect the request to a node tracking the specific shard we are interested in, would come later. Resolving https://github.com/near/nearcore/issues/5047 * build fuzzers on ubuntu-20.04 (#10299) Currently, our clusterfuzz runners do not have the glibc version required to run builds made on ubuntu-22.04. Building fuzzers on ubuntu-20.04 should fix the issue (that is only apparent in fuzzer logs); hopefully once we upgrade our clusterfuzz runners we’ll be able to bump to a newer ubuntu LTS. * refactor: Rename BodySync to BlockSync for consistency (#10293) And a bit of documentation for header sync. And a bit of documentation of block sync. And a bit of refactoring of block sync. And a tiny refactoring related to state sync. And a lot of TODOs. * fix(resharding): fix dynamic config (#10297) Fixing the dynamic config since #10290 was accidentaly merged prematurely before it was fully tested and fixed. Tested on localnet for now, will then test on mocknet and only then merge. * refactor: don't use `Fn` when `FnOnce` will suffice (#10301) This function calls the closure just once so no need to take a more restrictive closure. * enable resharding v2 (#10303) # Feature to stabilize This PR stabilizes the Resharding V2 protocol feature. It introduces a new implementation for resharding and a new shard layout for the production networks - mainnet and testnet. The resharding process from the current shard layout to the new shard layout will be triggered automatically an epoch before the switch to the new protocol version. # Context - https://github.com/near/NEPs/pull/508 - https://github.com/near/nearcore/issues/8992 # Testing and QA This feature was extensively tested in unit tests, integration tests, nayduck tests, mocknet, testnet and mainnet. # Checklist - [ ] nayduck - https://nayduck.near.org/#/run/3308 - [x] Update CHANGELOG.md to include this protocol feature in the `Unreleased` section. * Reintroduce configurable cache size for the FlatState column (#10304) One way to reduce flat state read latency during the NEAT incident was to increase the cache size 3x. As the config parameter was helpful during the incident, re-introduce this config parameter. Additionally make caches for State and FlatState configurable separately. * refactor: same treatment for `get_shard_uids()` as for `shard_ids()` (#10305) - renames `get_shard_uids()` to `shard_uids()` - `shard_uids()` return an iterator instead of a vector. * refactor(state-sync): Request sync block in client actor before starting state sync (#10175) We only use this logic for state sync when the nodes starts with old blocks. If the nodes has been running during the epoch change, it is supposed to have the block so it does not need it at state sync during catchup. * refactor: group runtime apply_transactions block/chunk-related arguments (#10279) This PR groups block- and chunk-related `RuntimeAdapter::apply_transactions` arguments into a separate structs. It also replaces usage `update_shard::BlockContext` with `ApplyTransactionsBlockContext` as well as a bit of refactoring around `ShardUpdateReason` to only require data which is actually used. Implements the first idea from https://github.com/near/nearcore/issues/10261. * fix: State Sync Header Request no longer crashes a node (#10313) * feat: add a time limit for processing transactions in produce_chunk (#10310) To produce a chunk the node fetches transactions from the transaction pool and adds them to the chunk. Adding a transaction to the chunk can be computationally expensive - the transaction is converted to receipt, which burns gas. Because of that the number of transactions has to be limited. The node will fetch transactions until some kind of limit is reached, and then decide that the chunk is ready and publish it. Currently we have a limit of gas that can be used for preparing transactions, but there is a big problem with it - it doesn't take into account discarding invalid transactions from the pool. Processing an invalid transaction doesn't burn gas, so the node could spend a lot of time discarding invalid transactions without ever reaching the limit. This would make chunk production very slow and the others might decide to just skip the chunk instead of waiting for it. Let's add another limit to `prepare_transactions`: a time limit. When producing a chunk the node will be allowed to add new transactions to the chunk for up to X milliseconds. This will ensure that a chunk is produced quickly, even when there are many invalid transactions. Choosing a good value for the limit is difficult, so a new config option is added to allow changing it on the fly: `produce_chunk_add_transactions_time_limit`. The default value for the time limit is 200ms. The reasoning for choosing this value is as follows: * Assuming that we want to produce ~1 block per second, 200ms is the largest value that would support that. If my understanding of the flow is correct, a 200ms chunk production would look like this: * Chunk producer produces a chunk (200ms) * Chunk producer publishes the chunk (50ms (?)) * Validator receives the chunk * Validator validates the transactions, redoing chunk producer's work (200ms) * Validator runs receipts (500ms) This gives us a 1s time to produce a block. * Looking at the metrics from mainnet, the P99 time to produce a chunk is ~60ms (and maximum is similar). 200ms is more than three times that, so it shouldn't break anything on the chain: https://nearinc.grafana.net/d/cd668436-f19a-4f3d-b4a5-a35ebd3701b0/chunk-production-stats?from=now-12h&to=now&refresh=1h&orgId=1 * During the minting of NEAT, the P99 time to produce a chunk jumped to ~860ms. This is unacceptable, as this will cause the chain to produce less than one block per second. This limit would prevent that. https://nearinc.grafana.net/d/cd668436-f19a-4f3d-b4a5-a35ebd3701b0/chunk-production-stats?from=1701298800000&to=1701342000000&orgId=1 For a more detailed discussion, see the comments in https://github.com/near/nearcore/issues/10278 Fixes: #10278 * fix(resharding): fix tests (#10314) This should fix a bunch of the nayduck tests. https://nayduck.near.org/#/run/3318 * feat(chunk-validator-assignment): shuffle shard ids (#10298) Introduces a shuffling of shard ids when assigning chunk validators to shards. Without that shuffling it is predictable which shards receive fewer (partial) mandates when they cannot be distributed evenly across shards. See this [comment](https://github.com/mooori/sim-validator-assignment/pull/12#discussion_r1389895314) which also brings up the idea of shard id shuffling. ### Separate shuffles for full and partial mandate assignment If there was only one shuffling for both full and partial mandates, then the shard(s) that get a full mandate less may also get a partial mandate less. Separate shufflings counter that and ideally (depending on randomness) stake is distributed more evenly as shards that receive a full mandate less may still get the maximum amount of partial mandates. ### Related issue #10014 * docs(resharding) - described how to dynamically update the config (#10315) * feat: stateless validation jobs in test mode (#10248) This is a next step for #9982. Here I introduce jobs which will perform stateless validation of newly received chunk by executing txs and receipts. Later they should be executed against state witness, but for now I just set a foundation by running these jobs against state data in DB. All passing tests verify that old and new jobs generate the same result. The final switch will happen when stateful jobs will be replaced with stateless ones. ### Details This doesn't introduce any load on stable version. On nightly version there will be `num_shards` extra jobs which will check that stateless validation results are consistent with stateful execution. But as we use nightly only for testing, it shouldn't mean much overhead. I add more fields to `ShardContext` structure to simplify code. Some of them are needed to break early if there is resharding, and the logic is the same for both kinds of jobs. `StorageDataSource::DbTrieOnly` is introduced to read data only from trie in stateless jobs. This is annoying but still needed if there are a lot of missing chunks and flat storage head moved above the block at which previous chunk was created. When state witness will be implemented, `Recorded` will be used instead. ## Testing * Failure to update current_chunk_extra on the way leads to >20 tests failing in process_blocks, with errors like `assertion `left == right` failed: For stateless validation, chunk extras for block CMV88CBcnKoxa7eTnkG64psLoJzpW9JeAhFrZBVv6zDc and shard s3.v2 do not match...` * If I update current_chunk_extra only once, `tests::client::resharding::test_latest_protocol_missing_chunks_high_missing_prob` fails which was specifically introduced for that. Actually this helped to realize that `validate_chunk_with_chunk_extra` is still needed but I will introduce it later. * Nayduck: ~https://nayduck.near.org/#/run/3293 - +10 nightly tests failing, will take a look~ https://nayduck.near.org/#/run/3300 --------- Co-authored-by: Longarithm <[email protected]> * [Snyk] Upgrade @types/node from 16.18.3 to 16.18.62 (#10312) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/node from 16.18.3 to 16.18.62.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **59 versions** ahead of your current version. - The recommended version was released **22 days ago**, on 2023-11-18. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/node</summary> <ul> <li> 16.18.62 - 2023-11-18 </li> <li> 16.18.61 - 2023-11-07 </li> <li> 16.18.60 - 2023-10-31 </li> <li> 16.18.59 - 2023-10-18 </li> <li> 16.18.58 - 2023-10-06 </li> <li> 16.18.57 - 2023-10-02 </li> <li> 16.18.56 - 2023-10-02 </li> <li> 16.18.55 - 2023-09-29 </li> <li> 16.18.54 - 2023-09-23 </li> <li> 16.18.53 - 2023-09-20 </li> <li> 16.18.52 - 2023-09-16 </li> <li> 16.18.51 - 2023-09-15 </li> <li> 16.18.50 - 2023-09-08 </li> <li> 16.18.49 - 2023-09-08 </li> <li> 16.18.48 - 2023-09-02 </li> <li> 16.18.47 - 2023-09-01 </li> <li> 16.18.46 - 2023-08-24 </li> <li> 16.18.45 - 2023-08-24 </li> <li> 16.18.44 - 2023-08-23 </li> <li> 16.18.43 - 2023-08-22 </li> <li> 16.18.42 - 2023-08-22 </li> <li> 16.18.41 - 2023-08-17 </li> <li> 16.18.40 - 2023-08-08 </li> <li> 16.18.39 - 2023-07-21 </li> <li> 16.18.38 - 2023-06-30 </li> <li> 16.18.37 - 2023-06-26 </li> <li> 16.18.36 - 2023-06-13 </li> <li> 16.18.35 - 2023-06-10 </li> <li> 16.18.34 - 2023-05-26 </li> <li> 16.18.33 - 2023-05-25 </li> <li> 16.18.32 - 2023-05-21 </li> <li> 16.18.31 - 2023-05-16 </li> <li> 16.18.30 - 2023-05-13 </li> <li> 16.18.29 - 2023-05-11 </li> <li> 16.18.28 - 2023-05-10 </li> <li> 16.18.27 - 2023-05-08 </li> <li> 16.18.26 - 2023-05-05 </li> <li> 16.18.25 - 2023-04-25 </li> <li> 16.18.24 - 2023-04-19 </li> <li> 16.18.23 - 2023-03-30 </li> <li> 16.18.22 - 2023-03-28 </li> <li> 16.18.21 - 2023-03-25 </li> <li> 16.18.20 - 2023-03-24 </li> <li> 16.18.19 - 2023-03-23 </li> <li> 16.18.18 - 2023-03-20 </li> <li> 16.18.17 - 2023-03-20 </li> <li> 16.18.16 - 2023-03-14 </li> <li> 16.18.15 - 2023-03-13 </li> <li> 16.18.14 - 2023-03-02 </li> <li> 16.18.13 - 2023-02-26 </li> <li> 16.18.12 - 2023-02-04 </li> <li> 16.18.11 - 2022-12-26 </li> <li> 16.18.10 - 2022-12-16 </li> <li> 16.18.9 - 2022-12-13 </li> <li> 16.18.8 - 2022-12-10 </li> <li> 16.18.7 - 2022-12-08 </li> <li> 16.18.6 - 2022-12-05 </li> <li> 16.18.5 - 2022-12-05 </li> <li> 16.18.4 - 2022-11-30 </li> <li> 16.18.3 - 2022-10-30 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/node GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI0NTA4MzY5Mi1lOWNjLTQyMjYtOWRkMi01NDFhODBhNTYzYTgiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjQ1MDgzNjkyLWU5Y2MtNDIyNi05ZGQyLTU0MWE4MGE1NjNhOCJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/node&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> * test(resharding) - test for getting the shard layout info by rpc (#10320) We need a way for near ecosystem developers to query the current shard layout. The EXPERIMENTAL_protocol_config allows that although with some extra steps. It should be good enough until we have time to implement a dedicated endpoint just for the shard layout. In this PR I extended the resharding nayduck test to check that the mentioned rpc endpoint works as expected and that changes in the shard layout are reflected in the rpc. * fix: remove flakiness in the test try_map_result_stops_on_error (#10318) `try_map_result` is a function which takes a list of tasks and runs them on multiple threads producing task outputs until the first error occurs. The test `try_map_result_stops_on_error` asks to execute an infinite list of tasks, but the 100th task throws an error, so the function should stop after executing ~100 tasks. The test checked that the function produces outputs for the first 10 tasks, but this isn't always true. The first 10 tasks could end up on a thread that is preempted and the tasks aren't executed before the error is hit on the 100th task. Because of this the test is flaky. I assumed that the first 10 tasks would be executed first, but this was wrong, we can't assume anything here. Let's remove all checks that could be broken by race conditions, no matter how unlikely. This makes the test weaker, but at least it won't be flaky. * make crates publish-able for other branches than master (#10322) * Fix sync_ban.py (#10323) The log message tracked by this test was changed. * [Snyk] Upgrade @types/react-dom from 18.2.15 to 18.2.16 (#10324) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/react-dom from 18.2.15 to 18.2.16.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **1 version** ahead of your current version. - The recommended version was released **22 days ago**, on 2023-11-21. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/react-dom</summary> <ul> <li> 18.2.16 - 2023-11-21 </li> <li> 18.2.15 - 2023-11-07 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react-dom GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiIyNjJjMzRiNi1iYjljLTQ2YjYtYmQzMy04NDdkZDU2ZjRiNDciLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjI2MmMzNGI2LWJiOWMtNDZiNi1iZDMzLTg0N2RkNTZmNGI0NyJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react-dom&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> * [Snyk] Upgrade @types/node from 16.18.62 to 16.18.63 (#10326) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/node from 16.18.62 to 16.18.63.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **1 version** ahead of your current version. - The recommended version was released **22 days ago**, on 2023-11-21. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/node</summary> <ul> <li> 16.18.63 - 2023-11-21 </li> <li> 16.18.62 - 2023-11-18 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/node GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI0NmJkMTg5Yy02MDg5LTRlNTQtOTZmMC0yOTFlNzZlY2ZkMzkiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjQ2YmQxODljLTYwODktNGU1NC05NmYwLTI5MWU3NmVjZmQzOSJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/node&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> Co-authored-by: nikurt <[email protected]> * refactor: various cleanups and fixes in `chain.rs` (#10327) get_state_response_part(): - remove the contiguous shard_ids assumption. Instead of checking the shard id against the number of chunks, look up the list of shard ids instead. - since the epoch id is remains the same for the current and the prev block, the set of shard ids should also remain the same so we only need to do the check once. - remove some unnecessary clones - finally rename some variables to make them shorter. Mainly remove the `sync_` prefix. I am not sure how much value this prefix is adding and the shorter variables are still conveying the same amount of information. compute_state_response_header(): - Similar to above, remove the contiguous shard_ids assumption. - Remove some unnecessary clones and dereferences save_receipt_id_to_shard_id_for_block(): - there is a potential bug in the function where if encounter an error in middle of iteration, we do not roll back the changes. Hence, pre compute the list of hashmaps before saving receipts. - Take a list of shard ids instead of num_shards to remove the contiguous shard ids assumption * Remove a scary warning from GCS State Sync (#10329) * [Snyk] Upgrade @types/react-dom from 18.2.16 to 18.2.17 (#10332) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade @types/react-dom from 18.2.16 to 18.2.17.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **1 version** ahead of your current version. - The recommended version was released **22 days ago**, on 2023-11-22. <details> <summary>Release notes</summary> <details> <summary>Package name: @types/react-dom</summary> <ul> <li> 18.2.17 - 2023-11-22 </li> <li> 18.2.16 - 2023-11-21 </li> </ul> from <a href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react-dom GitHub release notes</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI3MTZkNzFlYS01ZDExLTRjYTAtYjA1NS05YzMzZjdlMDVlODUiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjcxNmQ3MWVhLTVkMTEtNGNhMC1iMDU1LTljMzNmN2UwNWU4NSJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react-dom&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> * [Snyk] Upgrade react-router-dom from 6.19.0 to 6.20.0 (#10331) This PR was automatically created by Snyk using the credentials of a real user. <h3>Snyk has created this PR to upgrade react-router-dom from 6.19.0 to 6.20.0.</h3> :information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project. <hr/> - The recommended version is **2 versions** ahead of your current version. - The recommended version was released **21 days ago**, on 2023-11-22. <details> <summary>Release notes</summary> <details> <summary>Package name: react-router-dom</summary> <ul> <li> 6.20.0 - <a href="https://snyk.io/redirect/github/remix-run/react-router/releases/tag/react-router-native%406.20.0">2023-11-22</a>[email protected] </li> <li> 6.20.0-pre.0 - 2023-11-21 </li> <li> 6.19.0 - 2023-11-16 </li> </ul> from <a href="https://snyk.io/redirect/github/remix-run/react-router/releases">react-router-dom GitHub release notes</a> </details> </details> <details> <summary>Commit messages</summary> <details> <summary>Package name: react-router-dom</summary> <ul> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/3cc38eac4753702a9a8a1fe239e2138d63ac6cc5">3cc38ea</a> chore: Update version for release (#11050)</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/77862f95f71397d92b8b583a16674c56efccc55f">77862f9</a> Exit prerelease mode</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/1621319ffa353bf33f33064d7611859df16286ee">1621319</a> Update Release Notes TOC</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/f1f8ed0acffb3d6c2860c362fc2b376dbf87df24">f1f8ed0</a> Update release notes</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/1e026b6f1ac34a774b4f77e5e3696251e8f79940">1e026b6</a> chore: Update version for release (pre) (#11047)</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/4a08d64c368c07816e753632345a13b8da050111">4a08d64</a> Enter prerelease mode</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/c0b4e12d12c5abdf5f7723e71959c4eb5e9effd9">c0b4e12</a> Merge branch 'main' into release-next</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/58d421fc4c592661a68dea59edc507fc4668ba5d">58d421f</a> Fix other code paths for resolveTo from a splat route (#11045)</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/5f530a775cd266940f725894277b6ea7bc55b5d0">5f530a7</a> Do not revalidate unmounted fetchers when v7_persistFetcher is enabled (#11044)</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/f320378b5145f59bb266a35a7655b563f712daef">f320378</a> Add additional test case for #10983</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/a48c43c8118bbf75b23b3ee748648bb3ee4d688e">a48c43c</a> feat: export `PathParam` type (#10719)</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/1d56e55d3f95730f99617dff23cf153f82394921">1d56e55</a> Remove tag links from headings in CHANGELOG.md</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/3b1a7364c730209b4baed9454c7f6c17c55e3ba8">3b1a736</a> Fix flaky test</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/406a1ddecd399ede2a517d7cae5f3ee63d02ed91">406a1dd</a> Merge branch 'release-next' into dev</li> <li><a href="https://snyk.io/redirect/github/remix-run/react-router/commit/6a5939b07c06c9dacd82705645dbbbe46de90e5e">6a5939b</a> Merge branch 'release-next'</li> </ul> <a href="https://snyk.io/redirect/github/remix-run/react-router/compare/dcf0c2a85aac3a78059a287ea478ff12adcb6a2d...3cc38eac4753702a9a8a1fe239e2138d63ac6cc5">Compare</a> </details> </details> <hr/> **Note:** *You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.* For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiIyZmI3YjkxYi0zN2NiLTQzYzgtOWNkYS02ODAzNTFmYzMwNmQiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjJmYjdiOTFiLTM3Y2ItNDNjOC05Y2RhLTY4MDM1MWZjMzA2ZCJ9fQ==" width="0" height="0"/> 🧐 [View latest project report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr) 🛠 [Adjust upgrade PR settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr) 🔕 [Ignore this dependency or unsubscribe from future upgrade PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=react-router-dom&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)  Co-authored-by: snyk-bot <[email protected]> Co-authored-by: nikurt <[email protected]> * feat(resharding): making resharding interruptible (#10328) Currently ctrl-c doesn't fully stop neard while it's in the middle of resharding. Adding StateSplitHandle that allows for stopping resharding in the `build_state_for_split_shards` method. to be tested in mocknet soon * bump shared crates version to 0.18, as released on the 1.36.1 branch (#10336) * better wording for workflow option (#10337) * fix: remove potential flakiness from a snapshot_hosts test (#10317) The test `snapshot_hosts::invalid_signature_not_broadcast` sends out a message containing three instances of `SnapshotHostInfo`, one of which has an invalid signature, and then verifies that this message is rejected. There is a subtelty here that I didn't take into account when I wrote this test. When the `PeerManager` receives a message with multiple `SnapshotHostInfos`, it can accept the valid instances, even in the case where there are other invalid instances in the message. Because of this the test could fail - `PeerManager` could accept `ok_info_a` or `ok_info_b` and then the final assert would fail. Whether it accepts them or not depends on the timing, as the instances are processed on multiple threads and the processing stops at the first invalid instance. Because of that the test could be flaky. Let's fix it by filtering out `ok_info_a` and `ok_info_b` in the final check. This way even if the `PeerManager` accepts them, it won't break the test. The code is the same as in `snapshot_hosts::too_many_shards_not_broadcast()`. * Implement simple state witness distribution, chunk validation, and chunk approval distribution skeletons. (#10287) This is a pretty rough PR, but I'm making an attempt to concretely implement the flow from distributing the state witness to sending the chunk endorsement. The following is left out: (1) production of the state witness; (2) actual validation logic; (3) receiving the chunk endorsement. However, the skeleton of the flow is there, including multithreading of the validation logic. Very open to comments. * nit: remove unnecessary `?` and `return`s in chain.rs (#10340) * init project * Wallet Contract placeholder (#10269) ## Context NEP: https://github.com/near/NEPs/issues/518 Tracking issue: https://github.com/near/nearcore/issues/10018. ### Goal We want the NEAR Protocol's ecosystem to be closer with Ethereum by integrating Web3 wallet support. To accomplish that, some changes on the protocol level are needed with an emphasis on user experience continuity. Following the design, the main change on the protocol level would be to have ETH-style addresses managed by a new `Wallet Contract` integrated into the protocol. For seamless onboarding of EVM users, the contract will be free of charge. ### Previous work This PR is built on top of two PRs: * https://github.com/near/nearcore/pull/10020: no-op PR, laying groundwork for further changes. * https://github.com/near/nearcore/pull/10224: empty contract as a placeholder, literally deployed to new ETH accounts. ## Summary This PR adds `near-wallet-contract` crate (based on `runtime/near-test-contracts`) that exposes `Wallet Contract` WASM code to the runtime. It stops deploying a copy of the `Wallet Contract` to each newly created ETH-implicit account. Instead, it uses an in-memory cached contract code on `execute_function_call` action from such account. ### Changes - Add `wallet-contract` crate (separated from the workspace) with placeholder `Wallet Contract` implementation. - Add `near-wallet-contract` crate (part of the workspace) that generates (through `build.rs`) and exposes the `Wallet Contract` WASM code. - Do not literally deploy `Wallet Contract` when creating ETH-implicit account. Just store reference to the `Wallet Contract`. - Treat ETH-implicit accounts specially when retrieving contract code from an account (in such case returns in-memory cached `Wallet Contract` code). - Add tests calling `Wallet Contract` where a transfer from an ETH-implicit account is possible depending on the public key passed with `rlp_transa…

staffik requested a review from a team as a code owner October 27, 2023 14:37

staffik requested a review from wacban October 27, 2023 14:37

staffik mentioned this pull request Oct 27, 2023

[ProjectTracking]: Ethereum address implicit accounts #10018

Closed

8 tasks

wacban reviewed Oct 27, 2023

View reviewed changes

core/account-id/src/lib.rs Outdated Show resolved Hide resolved

staffik changed the title ~~refactor: eth-implicit accounts~~ ETH-implicit accounts non-functional changes Oct 27, 2023

staffik changed the title ~~ETH-implicit accounts non-functional changes~~ Setup for ETH-implicit accounts Oct 27, 2023

staffik force-pushed the eth-implicit-account branch from 52a7754 to 068b0f6 Compare October 30, 2023 10:53

wacban requested a review from birchmd October 30, 2023 11:22

birchmd approved these changes Oct 30, 2023

View reviewed changes

runtime/runtime/src/actions.rs Outdated Show resolved Hide resolved

core/primitives/src/utils.rs Outdated Show resolved Hide resolved

staffik force-pushed the eth-implicit-account branch from 92cdc36 to 4319933 Compare October 30, 2023 15:05

wacban reviewed Oct 30, 2023

View reviewed changes

tools/mirror/src/key_mapping.rs Show resolved Hide resolved

runtime/near-vm-runner/src/logic/logic.rs Outdated Show resolved Hide resolved

staffik force-pushed the eth-implicit-account branch from 4319933 to cb1a914 Compare October 31, 2023 11:28

staffik requested a review from wacban October 31, 2023 12:25

wacban reviewed Oct 31, 2023

View reviewed changes

core/primitives-core/src/runtime/fees.rs Outdated Show resolved Hide resolved

staffik requested a review from wacban October 31, 2023 15:06

wacban approved these changes Oct 31, 2023

View reviewed changes

staffik force-pushed the eth-implicit-account branch from 1c6cf23 to a96103c Compare October 31, 2023 16:24

walnut-the-cat reviewed Oct 31, 2023

View reviewed changes

core/primitives-core/src/runtime/fees.rs Outdated Show resolved Hide resolved

core/primitives-core/src/runtime/fees.rs Show resolved Hide resolved

staffik force-pushed the eth-implicit-account branch 3 times, most recently from 1aa7719 to e6b1d59 Compare November 2, 2023 08:24

staffik force-pushed the eth-implicit-account branch from 5779553 to f7ba202 Compare November 2, 2023 09:46

staffik added 6 commits November 5, 2023 21:14

refactor: eth-implicit accounts

4f33fac

Cargo fmt check style fixes

f4b490a

Rename derive_account_id_from_public_key -> derive_near_implicit_acco…

e3b385d

…unt_id

nit PR fixes

1123a41

transfer_exec_fee / transfer_send_fee refactor

6a3895c

Style fix for cargo fmt check

dc0d40f

staffik added 2 commits November 5, 2023 21:14

Remove core/account-id/src/lib.rs

0e82e48

Update near-account-id version

861bdca

staffik force-pushed the eth-implicit-account branch from f7ba202 to 861bdca Compare November 5, 2023 20:14

Do not use is_implicit for now

d9b8e53

staffik commented Nov 6, 2023

View reviewed changes

wacban approved these changes Nov 6, 2023

View reviewed changes

Remove eth_implicit_test_account as it is currently unused

79c1ddd

staffik added this pull request to the merge queue Nov 6, 2023

Merged via the queue into master with commit 00ca2f3 Nov 6, 2023
8 of 9 checks passed

staffik deleted the eth-implicit-account branch November 6, 2023 13:10

staffik mentioned this pull request Nov 7, 2023

ETH-implicit account support (old design) #10056

Closed

staffik mentioned this pull request Nov 21, 2023

ETH-implicit account support #10224

Merged

staffik mentioned this pull request Nov 29, 2023

Wallet Contract placeholder #10269

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup for ETH-implicit accounts #10020

Setup for ETH-implicit accounts #10020

staffik commented Oct 27, 2023 •

edited

Loading

birchmd left a comment

wacban left a comment

wacban left a comment

wacban left a comment

staffik commented Nov 2, 2023 •

edited

Loading

staffik Nov 6, 2023

staffik Nov 6, 2023

staffik commented Nov 6, 2023

akhi3030 commented Nov 6, 2023

wacban left a comment

wacban Nov 6, 2023

staffik Nov 6, 2023

Setup for ETH-implicit accounts #10020

Setup for ETH-implicit accounts #10020

Conversation

staffik commented Oct 27, 2023 • edited Loading

birchmd left a comment

Choose a reason for hiding this comment

wacban left a comment

Choose a reason for hiding this comment

wacban left a comment

Choose a reason for hiding this comment

wacban left a comment

Choose a reason for hiding this comment

staffik commented Nov 2, 2023 • edited Loading

staffik Nov 6, 2023

Choose a reason for hiding this comment

staffik Nov 6, 2023

Choose a reason for hiding this comment

staffik commented Nov 6, 2023

akhi3030 commented Nov 6, 2023

wacban left a comment

Choose a reason for hiding this comment

wacban Nov 6, 2023

Choose a reason for hiding this comment

staffik Nov 6, 2023

Choose a reason for hiding this comment

staffik commented Oct 27, 2023 •

edited

Loading

staffik commented Nov 2, 2023 •

edited

Loading