… into code-coverage

… code-coverage

… code-coverage

… code-coverage

)

remove the space to escape the newline instead.

* external access to kaspad core rpc services API

* rename wallet event members for serde serialization
misc wallet functionality update

* wip - event renaming

* wip - event serialization tuneup

* remove default wrpc listening ports on kaspad Args + fix port mapping bug for automatic JSON RPC port mapping.

* Box wallet events

* introduce RpcCtl + isolate wRPC from DynRpcApi

* debug trait for secret

* de-couple wRPC from the runtiem wallet making RPC bindings generic

* convert wallet_list to return Vec<WalletDescriptor>

* user-friendly error message on decryption error

* UtxoProcessor - ignore task shutdown if not running

* wallet storage data versioning

* wallet - propagate wallet title

* sort traits on WalletDescriptor

* wallet - use precise decimal conversion when parsing strings to sompi

* display kaspa on get-balance-by-address-rpc

* wallet - rename settings fns to have try_ prefixes

* rename library names "kaspad" and "kaspa_cli" to "kaspad_lib" and "kaspa_cli_lib" to prevent build collisions

* no-unsafe-eval feature

* allow for chrome-extension schema in the RPC url

* wallet - expose bip32 wordlist for external access

* wallet - delegate rpc ownership to utxo_processor; reorder rpc disconnect to occur before utxo_processor shutdown;

* wallet - migrate wallet settings loading to cli

* rename RpcCtlOp (Open, Close) to RpcState (Opened, Closed)

* remove automatic RPC disconnect from wallet stop() to allow retention of the RPC channel after closing the wallet

* allow UtxoProcessor to bind to an already existing/connected RPC channel, allow retention of a connected RPC channel past UtxoProcessor shutdown. (API centric defensive functionality)

* code formatting

* fix clippy warnings

* migrate chrome-extension to use ws:// schema by default

* wallet list formatting

* staging for WRS 0.7.0

* add bip32 word list iterator

* update WRS to 0.7.0

* fix rustdoc errors

* wip - initial WalletApi trait implementation

* wallet - macros and account descriptors

* wallet - ping_call

* cleanup

* wip - wallet api cleanup & testing

* kaspa-metrics as a separate crate

* wip - metrics cleanup

* cargo.lock update

* async/sync api  changes for storage

* merge

* master merge

* wip

* wip

* wip

* wallet - (wip) changes to AccountDescriptor and Wallet API

* metrics - make sink future return as optional

* wip - refactor wallet open; improve wallet event ordering; cleanup

* wip (debugging UtxoChanged notifications) - fix issuance during failed tx

* adding workflow-rs as github dependency (allows CI)

* revert github deps for WRS

* metrics

* refactor metrics to include bdi counts and peer counts

* wallet api updates
refactoring transaction access interface
misc updates to wallet APIs
refactor wallet open sequence
refactor legacy accounts

* wallet framework - fix a but resulting in the early issuane of UTXO maturity events.
Refactor handling of outgoing transaction UTXOs (sending outgoing tx ACKs on UtxoChange removal), minor fixes in the Transaction file-system storage.  Add "outgoing" property to account balances (to track in-flight transactions being submitted until acceptance). Introduce new Events::Change that gets fired specifically when receiving change of the currently outgoing transaction. Initial steps toward Daa property wrapper.

* npm 0.13.0

* metrics - cleanup after RPC refactoring

* rename rust-side of wasm bindings as naming conventions interfere with rust use (make fns more explicitly named)

* make metrics data formatting fns as pub

* expose log levels in kaspa-core::log

* rename metrics & improve metrics fetch task

* add API calls to pre-configure wallet framework stroage folders

* cumulative updates for Cargo.lock

* Improve address error messages for human readability

* PrvKeyData enumeration, initial account transfer support (wip), wallet descriptor handling

* wallet - change error to retain PrvKeyDataId instead of a String

* propagate bip39 flag to runtime account and it's descriptor

* wip - keydata refactoring

* metrics refactoring (bandwidth)

* wip (cleanup)

* misc improvements to metrics

* code formatting

* wRPC bandwidth monitoring + scaffolding for global bandwidth metrics

* bump WRS to 0.9.0

* metrics - add mempool_size and convert some items to u32/f32

* metrics cleanup

* wallet and account rename, remove "title" from accounts

* add missing mempool size metric

* UTXO "stasis" mode for new coinbase transactions

* remove useless Option from the rpc url parser

* fix daa score in maturity state check

* add name_or_id to PrvKeyDataInfo

* wallet RPC Connect/Disconnect and GetStatus API calls

* cleanup

* support for 12-word bip39 mnemonics

* add a hidden --daemon flag to kaspad

* refactor get_daa_score_timestamp helper to return Vec<u64> instead of Response struct

* Add Scan event

* implement support for unixtime in transactions

* kaspa value formatting functions

* fix a deadlock due to re-entrant mutex lock when loading multiple accounts from storage

* introduce utxo stasis mode + make balance carry utxo counts as well as the totals

* refactor wallet events, replace reset() with close() where appropriate

* manual debug impl for tx components (display script as hex, address as address etc)

* fix incorrect gas assignment in WASM bindings

* updates for wallet event handling

* refactor sign_with_multiple to return Err(tx) when partially signed

* refactor Storage and Wallet descriptors

* fix account derivation index handling in DerivationCapable

* track acceptance DAA stamp in outgoing transaction record

* temporary harness for embedding script engine directly into tx generator (used for debugging)

* convert PendingTransaction Utxo entries into HashSet

* rename pending tx fns

* outgoing tx wrapper around pending tx

* refactor notification events during wallet ops

* as_ref() for PrvKeyData

* wip - initial RpcCoreMocks for wallet framework tests

* wallet api messages update for StorageDescriptor usage

* update AddressDerivationMeta to track addresses from index 0

* add AHash and async_channel to wallet_core crate

* allow Args to be used externally for daemon argument pre-validation

* impl Hash for UtxoEntryReference and rename "fake()" to "simulated()"

* update wallet events

* expose tx validator check_scripts() as pub

* update wallet wasm fns for wallet and storage descriptors

* rename kaspa-metrics to kaspa-metrics-core

* code formatting

* code formatting

* migrate transaction console formatting to cli/extensions

* refactor wallet events

* use ahash for faster processing in wallet hash maps (utxo context)

* wallet - add SendError handling for error::Error

* wallet event refactoring, scan/discovery process refactoring, wallet change secret api handling

* refactoring: transfer and deep derivation scan

* transaction record refactoring

* wallet - support for cross-account transfers in generator

* wallet - outgoing transaction impl

* wallet - refactor maturity handling

* wallet/utxo - initial tests scaffolding

* minor optimization of the discovery/scan process

* wallet - utxo processor outgoing tx and discovery timestamp unixtime handling

* refactor utxo context to use output transactions, refactor transfers, compound and change notifications

* code formatting + Cargo.lock update

* Signed enum to represent Fully or Partially signed tx

* bind p2p and grpc bandwidth metrics

* misc clippy fixes

* cleanup

* remove --daemon as Args::parse() offers external mutations

* remove previously implemented mnemonic variant in favor of client-side mnemonic creation

* remove AccessContextT, convert to wallet_secret, wip WalletApi traits, refactor account args

* code formatting

* separate runtime wallet into multiple modules

* wallet - revert create_accounts to process a single account at a time

* cleanup

* wallet - JSON import and export + preliminary docs

* address - AddressList - fix TryFrom deadlock caused by recursive trait invocation

* wallet - api rustdoc + cleanup, transaction calls

* wallet docs

* change address debug formatting

* adding cli rpc addpeer

* wip - wallet restructuring

* wip - restructure wallet modules

* wip - wallet restructuring - cleanup

* wip - AccountKind refactoring

* wallet - refactoring storage serialization & encryption

* fold in PR kaspanet#349

* wallet framework - misc helper fns and cleanup

* storage unit tests + prv key data module restructuring

* wallet - account factory registrations

* Remove panics caused by sudden loss of connection while submitting transactions.

* cleanup

* cleanup

* cleanup + docs

* docs

* adjust metrics formatting

* cleanup

* bump workflow-rs to 0.10.1

* fix wallet storage unit tests

* fix generator unit tests

* convert missing listener_id panic to error

* docs

* code formatting

* lints

* misc updates following github reviews

* refactor AccountKind to use fixedstr and implement Copy

* refactor AccountKind to use fixedstr (str64) and re-implement Copy, removing clone() where necessary

* change js-bound fn names from *as* to *to* to follow Rust conventions

* replace AssocPrvKeyDataIds iter().any() to contains()

* remove vec![] in MetricsGroup iterator fn; add TODO comment on Metric::group()

* scope magic numbers within their associated structs

* add try_fully_signed() and try_partially_signed() fns to Signed that allow recovery of the tx

* refactor local storage cache to use RwLock instead of a Mutex

* refactor serialization as per github feedback (partial trait impl)

* Optimize AssocPrvKeyDataIds to use Either in the IntoIter impl, removing allocations.

* Refactor to bypass aux buffer creation when storing wallet on native platforms.

* impl custom Debug for storage structs to display data in hex

* adding misc SAFETY comments on unsafe ops

* TODO comments for later SignerT+multisig refactoring

* Change panic!() to unreachable!() to make it clearer that a state should not be reached.

* rename TransactionRecord::unixtime to unixtime_msec

* fix wrong references in comments

* Fix min_cosigner_index handling in multisig fns

* add comments to UtxoContext::insert() error handling

* bind AccountStorable:Storable for AccountStorage, migrate Storable into a separate module, add comments to MAGIC values in some locations

* make framework init functions unsafe

* update workflow-rs to 0.10.2

* fix comments

* fix data cloning when getting collection counts

* adding TODO notes to wasm macros and KAS value decimal parsing

---------

Co-authored-by: Surinder Singh Matoo <[email protected]>

* mimalloc

* mimalloc disable purge_decommits

* Add to simpa and integration tests

* remove redundant unix and replace with win/linux/mac

* Add comment

* Add comment

* Sort out features

* Get rid of ctor

---------

Co-authored-by: Michael Sutton <[email protected]>

* fix account derivation index relay in bip32

* fix serializations and sanity checks

* zeroize bip39 mnemonic after bip44 scan

* code formatting

… data sizes (kaspanet#351)

* cache inner struct: prep for tracking size

* initial cache policy and mem size estimator

* use new cache policies

* fix storage calcs

* update TN11 genesis

* temp: limit max block parents in virtual processor

* use more conservative constants (wip)

* ensure sufficient cache allocs for high level stores

* increase caches related to daa window

* Squashed commit of the following:

commit a4ba784
Author: Ori Newman <[email protected]>
Date:   Tue Dec 26 16:22:36 2023 +0200

    Add headers first validation to simpa (kaspanet#364)

commit fb1aa5f
Author: Ori Newman <[email protected]>
Date:   Tue Dec 26 16:07:19 2023 +0200

    Move to mimalloc (kaspanet#361)

    * mimalloc

    * mimalloc disable purge_decommits

    * Add to simpa and integration tests

    * remove redundant unix and replace with win/linux/mac

    * Add comment

    * Add comment

    * Sort out features

    * Get rid of ctor

    ---------

    Co-authored-by: Michael Sutton <[email protected]>

commit b6f7e09
Author: aspect <[email protected]>
Date:   Tue Dec 26 13:38:24 2023 +0200

    Cumulative PR for Wallet & Metrics Refactoring (kaspanet#360)

    * external access to kaspad core rpc services API

    * rename wallet event members for serde serialization
    misc wallet functionality update

    * wip - event renaming

    * wip - event serialization tuneup

    * remove default wrpc listening ports on kaspad Args + fix port mapping bug for automatic JSON RPC port mapping.

    * Box wallet events

    * introduce RpcCtl + isolate wRPC from DynRpcApi

    * debug trait for secret

    * de-couple wRPC from the runtiem wallet making RPC bindings generic

    * convert wallet_list to return Vec<WalletDescriptor>

    * user-friendly error message on decryption error

    * UtxoProcessor - ignore task shutdown if not running

    * wallet storage data versioning

    * wallet - propagate wallet title

    * sort traits on WalletDescriptor

    * wallet - use precise decimal conversion when parsing strings to sompi

    * display kaspa on get-balance-by-address-rpc

    * wallet - rename settings fns to have try_ prefixes

    * rename library names "kaspad" and "kaspa_cli" to "kaspad_lib" and "kaspa_cli_lib" to prevent build collisions

    * no-unsafe-eval feature

    * allow for chrome-extension schema in the RPC url

    * wallet - expose bip32 wordlist for external access

    * wallet - delegate rpc ownership to utxo_processor; reorder rpc disconnect to occur before utxo_processor shutdown;

    * wallet - migrate wallet settings loading to cli

    * rename RpcCtlOp (Open, Close) to RpcState (Opened, Closed)

    * remove automatic RPC disconnect from wallet stop() to allow retention of the RPC channel after closing the wallet

    * allow UtxoProcessor to bind to an already existing/connected RPC channel, allow retention of a connected RPC channel past UtxoProcessor shutdown. (API centric defensive functionality)

    * code formatting

    * fix clippy warnings

    * migrate chrome-extension to use ws:// schema by default

    * wallet list formatting

    * staging for WRS 0.7.0

    * add bip32 word list iterator

    * update WRS to 0.7.0

    * fix rustdoc errors

    * wip - initial WalletApi trait implementation

    * wallet - macros and account descriptors

    * wallet - ping_call

    * cleanup

    * wip - wallet api cleanup & testing

    * kaspa-metrics as a separate crate

    * wip - metrics cleanup

    * cargo.lock update

    * async/sync api  changes for storage

    * merge

    * master merge

    * wip

    * wip

    * wip

    * wallet - (wip) changes to AccountDescriptor and Wallet API

    * metrics - make sink future return as optional

    * wip - refactor wallet open; improve wallet event ordering; cleanup

    * wip (debugging UtxoChanged notifications) - fix issuance during failed tx

    * adding workflow-rs as github dependency (allows CI)

    * revert github deps for WRS

    * metrics

    * refactor metrics to include bdi counts and peer counts

    * wallet api updates
    refactoring transaction access interface
    misc updates to wallet APIs
    refactor wallet open sequence
    refactor legacy accounts

    * wallet framework - fix a but resulting in the early issuane of UTXO maturity events.
    Refactor handling of outgoing transaction UTXOs (sending outgoing tx ACKs on UtxoChange removal), minor fixes in the Transaction file-system storage.  Add "outgoing" property to account balances (to track in-flight transactions being submitted until acceptance). Introduce new Events::Change that gets fired specifically when receiving change of the currently outgoing transaction. Initial steps toward Daa property wrapper.

    * npm 0.13.0

    * metrics - cleanup after RPC refactoring

    * rename rust-side of wasm bindings as naming conventions interfere with rust use (make fns more explicitly named)

    * make metrics data formatting fns as pub

    * expose log levels in kaspa-core::log

    * rename metrics & improve metrics fetch task

    * add API calls to pre-configure wallet framework stroage folders

    * cumulative updates for Cargo.lock

    * Improve address error messages for human readability

    * PrvKeyData enumeration, initial account transfer support (wip), wallet descriptor handling

    * wallet - change error to retain PrvKeyDataId instead of a String

    * propagate bip39 flag to runtime account and it's descriptor

    * wip - keydata refactoring

    * metrics refactoring (bandwidth)

    * wip (cleanup)

    * misc improvements to metrics

    * code formatting

    * wRPC bandwidth monitoring + scaffolding for global bandwidth metrics

    * bump WRS to 0.9.0

    * metrics - add mempool_size and convert some items to u32/f32

    * metrics cleanup

    * wallet and account rename, remove "title" from accounts

    * add missing mempool size metric

    * UTXO "stasis" mode for new coinbase transactions

    * remove useless Option from the rpc url parser

    * fix daa score in maturity state check

    * add name_or_id to PrvKeyDataInfo

    * wallet RPC Connect/Disconnect and GetStatus API calls

    * cleanup

    * support for 12-word bip39 mnemonics

    * add a hidden --daemon flag to kaspad

    * refactor get_daa_score_timestamp helper to return Vec<u64> instead of Response struct

    * Add Scan event

    * implement support for unixtime in transactions

    * kaspa value formatting functions

    * fix a deadlock due to re-entrant mutex lock when loading multiple accounts from storage

    * introduce utxo stasis mode + make balance carry utxo counts as well as the totals

    * refactor wallet events, replace reset() with close() where appropriate

    * manual debug impl for tx components (display script as hex, address as address etc)

    * fix incorrect gas assignment in WASM bindings

    * updates for wallet event handling

    * refactor sign_with_multiple to return Err(tx) when partially signed

    * refactor Storage and Wallet descriptors

    * fix account derivation index handling in DerivationCapable

    * track acceptance DAA stamp in outgoing transaction record

    * temporary harness for embedding script engine directly into tx generator (used for debugging)

    * convert PendingTransaction Utxo entries into HashSet

    * rename pending tx fns

    * outgoing tx wrapper around pending tx

    * refactor notification events during wallet ops

    * as_ref() for PrvKeyData

    * wip - initial RpcCoreMocks for wallet framework tests

    * wallet api messages update for StorageDescriptor usage

    * update AddressDerivationMeta to track addresses from index 0

    * add AHash and async_channel to wallet_core crate

    * allow Args to be used externally for daemon argument pre-validation

    * impl Hash for UtxoEntryReference and rename "fake()" to "simulated()"

    * update wallet events

    * expose tx validator check_scripts() as pub

    * update wallet wasm fns for wallet and storage descriptors

    * rename kaspa-metrics to kaspa-metrics-core

    * code formatting

    * code formatting

    * migrate transaction console formatting to cli/extensions

    * refactor wallet events

    * use ahash for faster processing in wallet hash maps (utxo context)

    * wallet - add SendError handling for error::Error

    * wallet event refactoring, scan/discovery process refactoring, wallet change secret api handling

    * refactoring: transfer and deep derivation scan

    * transaction record refactoring

    * wallet - support for cross-account transfers in generator

    * wallet - outgoing transaction impl

    * wallet - refactor maturity handling

    * wallet/utxo - initial tests scaffolding

    * minor optimization of the discovery/scan process

    * wallet - utxo processor outgoing tx and discovery timestamp unixtime handling

    * refactor utxo context to use output transactions, refactor transfers, compound and change notifications

    * code formatting + Cargo.lock update

    * Signed enum to represent Fully or Partially signed tx

    * bind p2p and grpc bandwidth metrics

    * misc clippy fixes

    * cleanup

    * remove --daemon as Args::parse() offers external mutations

    * remove previously implemented mnemonic variant in favor of client-side mnemonic creation

    * remove AccessContextT, convert to wallet_secret, wip WalletApi traits, refactor account args

    * code formatting

    * separate runtime wallet into multiple modules

    * wallet - revert create_accounts to process a single account at a time

    * cleanup

    * wallet - JSON import and export + preliminary docs

    * address - AddressList - fix TryFrom deadlock caused by recursive trait invocation

    * wallet - api rustdoc + cleanup, transaction calls

    * wallet docs

    * change address debug formatting

    * adding cli rpc addpeer

    * wip - wallet restructuring

    * wip - restructure wallet modules

    * wip - wallet restructuring - cleanup

    * wip - AccountKind refactoring

    * wallet - refactoring storage serialization & encryption

    * fold in PR kaspanet#349

    * wallet framework - misc helper fns and cleanup

    * storage unit tests + prv key data module restructuring

    * wallet - account factory registrations

    * Remove panics caused by sudden loss of connection while submitting transactions.

    * cleanup

    * cleanup

    * cleanup + docs

    * docs

    * adjust metrics formatting

    * cleanup

    * bump workflow-rs to 0.10.1

    * fix wallet storage unit tests

    * fix generator unit tests

    * convert missing listener_id panic to error

    * docs

    * code formatting

    * lints

    * misc updates following github reviews

    * refactor AccountKind to use fixedstr and implement Copy

    * refactor AccountKind to use fixedstr (str64) and re-implement Copy, removing clone() where necessary

    * change js-bound fn names from *as* to *to* to follow Rust conventions

    * replace AssocPrvKeyDataIds iter().any() to contains()

    * remove vec![] in MetricsGroup iterator fn; add TODO comment on Metric::group()

    * scope magic numbers within their associated structs

    * add try_fully_signed() and try_partially_signed() fns to Signed that allow recovery of the tx

    * refactor local storage cache to use RwLock instead of a Mutex

    * refactor serialization as per github feedback (partial trait impl)

    * Optimize AssocPrvKeyDataIds to use Either in the IntoIter impl, removing allocations.

    * Refactor to bypass aux buffer creation when storing wallet on native platforms.

    * impl custom Debug for storage structs to display data in hex

    * adding misc SAFETY comments on unsafe ops

    * TODO comments for later SignerT+multisig refactoring

    * Change panic!() to unreachable!() to make it clearer that a state should not be reached.

    * rename TransactionRecord::unixtime to unixtime_msec

    * fix wrong references in comments

    * Fix min_cosigner_index handling in multisig fns

    * add comments to UtxoContext::insert() error handling

    * bind AccountStorable:Storable for AccountStorage, migrate Storable into a separate module, add comments to MAGIC values in some locations

    * make framework init functions unsafe

    * update workflow-rs to 0.10.2

    * fix comments

    * fix data cloning when getting collection counts

    * adding TODO notes to wasm macros and KAS value decimal parsing

    ---------

    Co-authored-by: Surinder Singh Matoo <[email protected]>

commit 10d0ebf
Author: D-Stacks <[email protected]>
Date:   Tue Dec 26 10:43:59 2023 +0100

    make step 3. of linux installation not fail copy and paste (kaspanet#363)

    remove the space to escape the newline instead.

* reduce children cache policy

* reduce caches 25%

* increase tx cache

* fix gd alloc and slightly increase other numbers

* fix a cache alloc bug by redesigning the cache and adding a min units option + fix bps constants

* fix simpa short cmd op name colision

* move impls to inner struct

* comments, docs and technical refactoring

* refactor MemSizeEstimator

* refactor bool to MemMode

* memsize docs

* remove unneeded min limits

* comments and name enhancements

* rename `CachePolicy::Units` to `CachePolicy::Count` to avoid confusion with `MemMode`

* use estimated num of unique headers in validate pruning point headers cache

* cache policy builder

* add todos

* median window has different size + comments

* read ghostdag data from compact access if possible

* remove wrong upper bound for reachability sets

* p2p connmanager - log the connect err

* IBD with headers proof logs

* log if search for missing bodies is taking too long

* Exit early on pruning processor when exit signal is received

* Exit early on pruning point validation when process exit is initiated

* Rename is_process_exiting to is_consensus_exiting

* Change atomic ordering for exit signal to relaxed

* Split logging of performance metrics into better readable lines (#4)

* Split logging of performance metrics

Affects:
- daemon.rs
- simpa

* Make daemon.rs log perf metrics if enabled

* Log perf metrics via impl Display

* Remove loglevel default changes

With impl Display, we don't need to change loglevels
to display perf metrics

* use 2 sub-structs for metrics display

* print precision + relocate fd

* use RAM (over RES/RSS) + optimize and fix edge case in `to_human_readable`

---------

Co-authored-by: coderofstuff <[email protected]>

* fix new unused imports warns by running `cargo clippy --fix --all --tests --benches --features devnet-prealloc`
note that all these `pub use` were not really exposed publicly on the crate level

* fix `variant name ends with the enum's name` warnings

* Add rothschild tx benchmark

So we can compare the performance difference between serial generation and parallel generation

* Optimize high TPS generation

1. Parallelize TX generation - hashing is a bottleneck so generating TXs seriallize will slow
the entire loop down significantly
2. Optimize UTXO selection - previously it was running at O(UTXO) due to filtering logic

* Move criterion to dev-dependency

* Some more rothschild improvements

* Fix from_fmillis typo

* Add flags

* Parallelize txid hashing in simpa

* Moving ClientPool to grpc/client

* Fix rothschild sending less than 100 txs

Due to integer divide, when tps is < 100, we end up creating no tx

* Remove obsolete temp init comment

---------

Co-authored-by: Ori Newman <[email protected]>

* track avg mergesets

* unorphan ancestors if they exist w/o roots

* refactor processing batch object + test fix

* use warn

* improve log

* comment

* orphan eviction policy

* oops: use the correct len

* review comments

…anet#374)

* track block txs by approx bytes

* 0.5 scale factor on byte budgets

* fix obsolete comment (unrelated to this pr)

* reduce orphan block flood

* use tracked bytes mode for acceptance data cache

* impl `estimate_mem_units` only when actually used

* propagate ram scale arg from the command line

* add to simpa as well

* make it clear that mass is an average

* fix cpu usage stat

* check_subscription_before_vccn

* simplfy to root notifer only

* simplify one last time. no need to be in notify trait at all.

* Add a pruning exit point on advance_pruning_utxoset

* Fix log

* Add timestamps to IBD progress

* Add timezone

* Address review comments

* Change error message

* Implement basic tx throttling on high P2P load

When P2P load is high:
1. Limit the number of peers to broadcast to
2. Limit the number of transactions requested when requesting missing
   transactions
3. Reduce the size of the invs channel and allow dropping invs when
   it's full

* Renames and small fixes

* Saturating sub for unknown tx limit

* Some fixes

* Remove redundant collect

* Fix lint

* restore collect

---------

Co-authored-by: Ori Newman <[email protected]>
Co-authored-by: Michael Sutton <[email protected]>
Co-authored-by: Michael Sutton <[email protected]>

* move frequent logs to debug

* wip storage mass calculation

* use checked add and return Option<u64>

* add mass field

* add mass to all necessary communication structs

* rename `calculated_mass` to `calculated_compute_mass` to make it explicit that storage mass is not included in this field

* apply bew mass as mempool rule

* apply storage mass rule to body processor

* apply storage mass rule to virtual in context validation

* some important prints

* simplify inner hashing condition + add todo

* add low-level storage mass tests

* add future todo

* fix tx hashing for BBT

* fix mass counting in body validation

* set mass in the correct place (in consensus)

* remove spam blocking patch

* fix fee rate (safe bcs fee is unwrapped which happens only for validated txs)

* fix tests

* renames and minor review comments

* fix mass sum

* add storage mass to simpa + activation switch

* prevent rothschild from sending dust

* new genesis

* new dns seeder

* activate storage mass on start

* new db version

* bump version

* make is synced smaller on TNets

* fix min items allocation for GD data

---------

Co-authored-by: Ori Newman <[email protected]>

Co-authored-by: aspect <[email protected]>
Co-authored-by: Ori Newman <[email protected]>

* Update testnet11.md

* Another update

* Some fixes

* Some fixes

* Some fixes

Removed "first experiment" section

* fix timestamp deviation tolerance for 10 bps (note that the change does not affect 1bps -> target time =1000)

* bump version

* Add nogrpc flag

When set, the gRPC service is not initialized

* Set default grpc listen to 127.0.0.1

* Preserve rpc registration order

…kaspanet#394)

* migrate settings to network params containing consts

* fix unit tests

…transaction acceptance. (kaspanet#393)

* Fix balance calculation overflow when waiting for compound transaction acceptance.

* account for fees in compound transactions

* cleanup

* do not inject port based on network type in wRPC URL when using SSL or have path

* update URL port injection logic

* Remove mimalloc "override" feature for macos only

* Move mimalloc dependency fully in utils/alloc

* Refactor sync event emission order and UtxoProcessor cleanup. Remove legacy RPC support.

* Refactor sync event emission order and UtxoProcessor cleanup. Remove legacy RPC support.

* update comments

* externalize wallet.reload() to the wallet API as wallet_reload_call()

…panet#396)

* Update workflow-rs to 0.10.3

* add encryption example

* fix decrypt_xchacha20poly1305 JS bindings

---------

Co-authored-by: Michael Sutton <[email protected]>

* When tx throttled, preferrentially relay to outbound peers

* Convert check tx throttling to be easily testable

* Fix TPS logic; Refactor rpc tx counter snapshot

* Optimize hub peer selection logic

Co-authored-by: Michael Sutton <[email protected]>

* Fix rpc -> p2p tx throttling typo

* Further optimize hub peer selection to avoid alloc

Co-authored-by: Michael Sutton <[email protected]>

---------

Co-authored-by: Michael Sutton <[email protected]>

* Add indexed db dependencies

* Implementation progress for load_single

* progress idb transaction store

* use `Arc<Inner>` pattern

* implementing try from jsvalue for types to be stored in idb

* try from js value for utxo record

* remove manual parsing and use borsh data to deserialize

* indexdb progress

* indexdb store and load multiple

* indexdb remove, store tx note, store tx metadata

* Add encryptable to transaction record for jsvalue

* Remove logs

* unnecessary import removed

* Move from/to js value to idb implementation itself

* fix some of the clippy warnings

* fix all clippy warnings

---------

Co-authored-by: aspect <[email protected]>

Co-authored-by: aspect <[email protected]>

* make a better estimation of header count (during IBD this will now count for the headers phase)

* rpc service responsiveness: use `transaction_count_sample` for inferring mempool size

* use counters snapshot

* introduce last known good virtual state cache

* move processing counters to consensus core and remove rpc service crate dep on consensus

* extract consensus stats to a single call

* turn some consensus session calls into non-async + fix get_block_dag_info_call

* use saturating_sub instead of checked_sub + unwrap_or_default

* use join over all async calls

---------

Co-authored-by: coderofstuff <[email protected]>

---------

Co-authored-by: coderofstuff <[email protected]>

* bump to 0.13.4

* bump to 0.13.4

* fix incorrect version due to a toolchain issue

* replace NetworkType with NetworkId in the tx generator and related APIs

* fix url path rendering in format_url

* Improve mass related errors

* refactor fee handling

* renaming fee variants

* mass combination strategy

* wip - storage mass

* additional unit tests and limit checks

* Introduce per-network configurable additional compound transaction compute mass (currently set to 100 on tn10 and tn11)

* expand sanity checks to handle 0 value outputs

* documentation updates

* code formatting

* Set new input sequence to 0

* change aggregate input value check to use saturating_sub() to prevent potential overflows (re github feedback)

* change storage mass fee calc to return u64::MAX if the resulting mass is None (re GitHub feedback)

* increase semaphore yield duration to 30 micros + tighten the test (so that w/o this fix it would fail certianly)

* allow the test 2 read capture attempts + add macos experiment results