Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wallet 5 revision 2 - features and optimisations #13

Merged
merged 99 commits into from
Feb 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
d399976
Optimized unneccessary cell loads and operations
Skydev0h Oct 2, 2023
c35b90b
Recorded failed attempt at defragmenting storage
Skydev0h Oct 2, 2023
fcc45d6
Removed unneccessary always true check
Skydev0h Oct 2, 2023
7ff7d07
Unrolled the common internal handler code
Skydev0h Oct 2, 2023
01cf03d
Implicitly return from the external handler
Skydev0h Oct 2, 2023
557c0d7
Reaped benefits of separated internal loaders
Skydev0h Oct 2, 2023
a8ee4c0
Discarded unneccessary slice remains in dispatcher
Skydev0h Oct 2, 2023
5e65d1c
Loaded auth_kind optionally using LDUQ instruction
Skydev0h Oct 3, 2023
44345e4
Is ifnot a joke for you? (emits less instructions)
Skydev0h Oct 3, 2023
bacb468
Localize extensions in loop and short-circ simple
Skydev0h Oct 3, 2023
46033e9
Reordering int msg handlers somehow saves 10 gas
Skydev0h Oct 3, 2023
57d1a9f
Moving signature check higher saves some gas
Skydev0h Oct 3, 2023
ff9a1b2
Reordering checks somehow sames some more gas
Skydev0h Oct 3, 2023
3601ebe
Removing end_parse is -gas and +reliability
Skydev0h Oct 3, 2023
eaf48f7
Keep your functions close and vars even closer
Skydev0h Oct 3, 2023
15f3dc6
OK, can remove end_parse, will write rationale later
Skydev0h Oct 3, 2023
ea38004
Add global gas counters to keep an eye on non-contest paths
Skydev0h Oct 9, 2023
f245407
Recalculated GGC for every commit to find a tradeoff
Skydev0h Oct 9, 2023
99cd428
Refactored internal message flows, good GGC value
Skydev0h Oct 9, 2023
48f8868
Github-friendly RST improvements page
Skydev0h Oct 9, 2023
9ecfb79
Reorganized inlining point for extension message flow
Skydev0h Oct 9, 2023
fe7f68e
Fix improvements table width for commit name
Skydev0h Oct 9, 2023
50ccdc0
[Internal] Fix ext-storer bug (negligible GGC gas impact)
Skydev0h Oct 9, 2023
a4890b2
Do not carry around params not needed (ext opt)
Skydev0h Oct 9, 2023
75c5bac
Add failed rewrite attempt (signed processing)
Skydev0h Oct 9, 2023
c45523e
Optimize argument order to match stack
Skydev0h Oct 9, 2023
fc3bc96
More reliable tail processing at cost of very small GGC increase
Skydev0h Oct 9, 2023
79ab86d
Swapping extn and sign order back saves some net gas
Skydev0h Oct 9, 2023
4f0cea7
Substantial code rewrite to make it more readable
Skydev0h Oct 10, 2023
4c6ee62
Record failed attempt to cache one ext w/o dict (increased gas)
Skydev0h Oct 10, 2023
1a08eae
Short-circuit optimization of LDUQ with IFNOTRET
Skydev0h Oct 10, 2023
3c26248
Short-circuited some returns with asm
Skydev0h Oct 10, 2023
7adaf5f
ASM-optimized simple action cases
Skydev0h Oct 10, 2023
bd454a7
Created scalpel tool for TVM delidding and analyzing under microscope
Skydev0h Oct 10, 2023
d8cd02d
Optimized out more unneeded instructions if may RET
Skydev0h Oct 10, 2023
204dddd
Removed another unneccessary DROP with preload uint
Skydev0h Oct 10, 2023
d19f689
Reordered argument order to optimize stack operations
Skydev0h Oct 10, 2023
ba29b84
Rewritten RETALT to IFNOTJMP - less gas, more reliable
Skydev0h Oct 10, 2023
8e5696b
Another argument stack optimization (psr -> dr call)
Skydev0h Oct 10, 2023
f77dded
Black magic route optimization (drop some result later)
Skydev0h Oct 10, 2023
4192634
Supercharged scalpel
Skydev0h Oct 10, 2023
542b241
This quite did not worked out. git good.
Skydev0h Oct 11, 2023
f461946
Another black magic optimization (drop auth_kind later)
Skydev0h Oct 11, 2023
e62691e
WIP: Documentation on optimizations
Skydev0h Oct 11, 2023
272892e
Update README.md
Skydev0h Oct 11, 2023
be920b8
Center contest logo
Skydev0h Oct 11, 2023
870e9ff
Fix centering
Skydev0h Oct 11, 2023
2ddb845
Update README
Skydev0h Oct 11, 2023
18a3e62
I have not noticed this achievement earlier!
Skydev0h Oct 11, 2023
15b4d6c
Revised code, started researching gas under microscope
Skydev0h Oct 12, 2023
653fa9d
Add tools to inspect generated code cell tree
Skydev0h Oct 12, 2023
8bdcf47
fift stuff
Skydev0h Oct 12, 2023
5e9bfa0
Prepare ground for radical optimizations
Skydev0h Oct 12, 2023
c11545f
Allow to turn on and off detailed gas tracing of main cases in tests
Skydev0h Oct 12, 2023
0794b7d
More preparations for radical optimization
Skydev0h Oct 12, 2023
8041686
Backport some tooling from entrypoint branch
Skydev0h Oct 13, 2023
886b9fe
Backported FunC optimizations from entrypoint branch
Skydev0h Oct 13, 2023
fcdb880
Use SDBEGINS to enforce "sign" prefix in external message
Skydev0h Oct 13, 2023
07b73f2
Use SDBEGINSQ to check internal message prefixes
Skydev0h Oct 13, 2023
4f2cf58
Backported some opts, organized code and docs
Skydev0h Oct 13, 2023
22a2f68
Finally written documentation about improvements
Skydev0h Oct 15, 2023
e104fda
Optimized instructions order for extension and fix args
Skydev0h Oct 15, 2023
17337eb
Clean up failed optimization attempts
Skydev0h Nov 2, 2023
33c4e87
Cleaned up unused code
Skydev0h Nov 2, 2023
5709f2a
Fixed calculations in table (type in origin amount)
Skydev0h Nov 2, 2023
b475fda
Fixed more calculations in improvements file
Skydev0h Nov 2, 2023
ea879ad
Merge pull request #3 from Skydev0h/main
siandreev Nov 20, 2023
6145f6b
Merge branch 'main' into feature/optimisation
Skydev0h Jan 23, 2024
6a245a2
Added root cell repacking algorithm with enforced Asm version
Skydev0h Jan 24, 2024
e70ff08
Added size constants and changed stored seqno type from ui32 to i33
Skydev0h Jan 24, 2024
194f803
Changed sign op for internal messages to sint to avoid replay attacks
Skydev0h Jan 24, 2024
04a363d
Removed set_data and temporarily disabled the relevant tests
Skydev0h Jan 24, 2024
3222372
Added actions checking (send_msg w/ mode 2 enforced) and set pk enabled
Skydev0h Jan 24, 2024
54a28d3
Add guidelines and code to be able to make unsafe version
Skydev0h Jan 24, 2024
7abb84d
Optimized verify_actions by rearranging loop and conditions
Skydev0h Jan 24, 2024
12b6e05
Added more info about sequence and fixed some cases in getters
Skydev0h Jan 24, 2024
c5c65d1
Adjusted comments to be more correct and precise
Skydev0h Jan 24, 2024
0c9fa87
Removed unneccessary noop middleware and renamed method to classic name
Skydev0h Jan 24, 2024
74cc7f6
Removed throwif from the comment about seqno
Skydev0h Jan 24, 2024
4217b8a
Made internal messages bounce-safe (replaced throws with soft checks)
Skydev0h Jan 24, 2024
dc6b4e9
Added quick return for empty internal messages
Skydev0h Jan 24, 2024
3ad3a7a
[Docs] Moved contest note to separate file and tidied up README a bit
Skydev0h Jan 24, 2024
0d9ffef
[Docs] Slight readme tidy-up
Skydev0h Jan 24, 2024
507b821
[Docs] Updated specification in accordance with new contract features
Skydev0h Jan 24, 2024
b5bdf24
[Docs] Update types.tlb
Skydev0h Jan 24, 2024
f2601ae
[Docs] Add use cases for disabling public key
Skydev0h Jan 27, 2024
9545120
Updated actions, allow naming, empty ref check, docs, actions wrapper
Skydev0h Jan 27, 2024
c7bd87a
Added gas tests for incoming messages
Skydev0h Jan 28, 2024
916ed76
Removed fift optimizations that might create problems for tooling
Skydev0h Jan 28, 2024
43eba0d
Flattened and split some asm functions for understanding and toolings
Skydev0h Jan 28, 2024
167347a
Add external message bit to signed message part
Skydev0h Jan 30, 2024
e580224
Changed signed message prefix from 1 bit to 32-bit message tag
Skydev0h Feb 3, 2024
41f3063
Moved signature to the end of the message (includes opcode)
Skydev0h Feb 9, 2024
033b63e
Adjusted TLB definitions to match signature at the end
Skydev0h Feb 9, 2024
34a8556
Adjusted all tests to match new signature place (in end)
Skydev0h Feb 9, 2024
c724855
Fixed accounting for seqno twice on signature disabling
Skydev0h Feb 12, 2024
4bf0feb
Added throw for unknown ext-act, setcode/data neg tests, test-only acts
Skydev0h Feb 15, 2024
3f76dbb
Added tests for signature disallow cases, incorrect allow act now throws
Skydev0h Feb 15, 2024
7f786ea
Added a test for separate auth disable by extension
Skydev0h Feb 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions Contest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## Contest note!

<div align="center">
<img alt="Contest logo" src="contest.png" height="280" width="280">
</div>

**Because of the extreme amount of optimizations, developer's discretion is advised!** *Evil laugh*

The build system is the same as in the original Wallet V5, **no security features have been sacrificed**
for performance improvements, that is - there are **practically no tradeoffs or compromises**.

Message and storage layouts were **not changed**, although some rearragement might squeeze a little more gas,
but that may break existing optimizations due to stack reordering.

Also, **tests were improved** - a **Global Gas Counter** mechanism was added that accounts for gas in all transactions
of all test suites and cases (except for negative and getter ones). This allows to keep an eye on other non-contest
cases to track how bad is tradeoff when performing optimizations here and there.

Another utility that was developed for contest is ***scalpel script***, that allows for a detailed, *really* detailed optimizations
of the code by comparing lines of code function by function, printing out diffs, and providing detailed TVM files with
stack comments and rewrites. This utility allowed to make some latter optimizations, since with each optimization
next one becomes exponentionally harder to make. While result is not entirely precise and is needed to be verified
by tests, this allows to instantly estimate whether there is some progress or not, since scalpel is executed immediately,
while tests take approximately 10 seconds to execute.

### Details of optimizations, their rationale and explanations, comparison of consumed gas both in test cases and not in test cases (global gas counter) are provided on a dedicated page: [Gas improvements](Improvements.rst).
447 changes: 447 additions & 0 deletions Improvements.rst

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,13 @@ Wallet V5 has 25% lower fees, can delegate payments for gas to third parties and
- `contracts` - source code of all the smart contracts of the project and their dependencies.
- `wrappers` - wrapper classes (implementing `Contract` from ton-core) for the contracts, including any [de]serialization primitives and compilation functions.
- `tests` - tests for the contracts.
- `scripts` - scripts used by the project, mainly the deployment scripts.
- `scripts` - scripts used by the project, mainly the deployment scripts, additionally contains utilities for gas optimisation.
- `fift` - contains standard Fift v0.4.4 library including the assembler and disassembler for gas optimisation utilities.

### Additional documentation

- [Gas improvements](Improvements.rst) - a log of improvements, detailed by primary code paths, global gas counters per commit.
- [Contest](Contest.md) - a note showing some information about interesting improvements during the optimisation contest.

## How to use

Expand Down
86 changes: 30 additions & 56 deletions Specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,18 @@ Thanks to [Andrew Gutarev](https://github.com/pyAndr3w) for the idea to set c5 r

Thanks to [@subden](https://t.me/subden), [@botpult](https://t.me/botpult) and [@tvorogme](https://t.me/tvorogme) for ideas and discussion.

Thanks to [Skydev](https://github.com/Skydev0h) for optimization and preparing the second revision of the contract.


## Features

* 25% smaller computation fees.
* Arbitrary amount of outgoing messages is supported via action list.
* Wallet code can be upgraded transparently without breaking user's address in the future.
* Wallet code can be extended by anyone in a decentralized and conflict-free way: multiple feature extensions can co-exist.
* Extensions can perform the same operations as the signer: emit arbitrary messages on behalf of the owner, add and remove extensions.
* Signed requests can be delivered via internal message to allow 3rd party pay for gas.
* For consistency and ease of indexing, external messages also receive a 32-bit opcode.
* To lay foundation for support of scenarios like 2FA or access recovery it is possible to disable signature authentication.

## Overview

Expand All @@ -39,8 +41,8 @@ Authentication:
* by extension

Operation types:
* standard output actions
* “set data” operation
* standard output send message action
* enable or disable public key (signature authentication)
* install extension
* remove extension

Expand All @@ -63,40 +65,12 @@ User may delegate this job to other apps via extensions.

### Extending the wallet

**A. Use extensions**

The best way to extend functionality of the wallet is to use the extensions mechanism that permit delegating access to the wallet to other contracts.

From the perspective of the wallet, every extension can perform the same actions as the owner of a private key. Therefore limits and capabilities can be embedded in such an extension with a custom storage scheme.

Extensions can co-exist simultaneously, so experimental capabilities can be deployed and tested independently from each other.

**B. Code optimization**

Backwards compatible code optimization **can be performed** with a single `set_code` action (`action_set_code#ad4de08e`) signed by the user. That is, hypothetical upgrade from `v5R1` to `v5R2` can be done in-place without forcing users to change their wallet address.

If the optimized code requires changes to the data layout (e.g. reordering fields) the user can sign a request with two actions: `set_code` (in the standard action) and `set_data` (an extended action per this specification). Note that `set_data` action must make sure `seqno` is properly incremented after the upgrade as to prevent replays. Also, `set_data` must be performed right before the standard actions to not get overwritten by extension actions. The updated wallet **must** have the new subwallet ID to prevent accidental repeated migrations.

User agents **should not** make `set_code` and `set_data` actions available via general-purpose API to prevent misuse and mistakes. Instead, they should be used as a part of migration logic for a specific wallet code.

To restore the wallet by a seed phrase, user agent should use the original code and should expect the upgraded code to work in exactly the same way as previously.

**C. Emergency upgrades**

This is a variant of (B), so the same consideration apply. The difference is that functionality may be modified as to prevent user from suffering loss of funds. E.g. some previously possible actions or signed messages would lead to a failure.

Just like with (B), user agents **should not** make `set_code` and `set_data` actions available via general-purpose API to prevent misuse and mistakes. Instead, they should be used as a part of migration logic for a specific wallet code.

New users’ wallets **should not** be deployed with upgraded code. Instead, the improved wallet code should also be released as a new wallet version (e.g. v6, with a separate subwallet ID) and new wallets should be deployed with that code. This way `set_code` would be used as an emergency patch for existing wallets, while new wallets would be deployed directly with the major next version.


**D. Substantial upgrades**

We **do not recommend** performing substantial wallet upgrades in-place using `set_code`/`set_data` actions. Instead, user agents should have support for multiple accounts and easy switching between them.

In-place migration requires maintaining backwards compatibility for all wallet features, which in turn could lead to increase in code size and higher gas and rent costs.


### Can the wallet outsource payment for gas fees?

Yes! You can deliver signed messages via an internal message from a 3rd party wallet. Also, the message is handled exactly like an external one: after the basic checks the wallet takes care of the fees itself, so that 3rd party does not need to overpay for users who actually do have TONs.
Expand All @@ -123,6 +97,14 @@ You need to put two requests in your message body:

Yes. We have considered constant-size schemes where the wallet only stores trusted extension code. However, extension authentication becomes combursome and expensive: plugin needs to transmit additional data and each request needs to recompute plugin’s address. We estimate that for the reasonably sized wallets (less than 100 plugins) authentication via the dictionary lookup would not exceed costs of indirect address authentication.

### Why it can be useful to disallow authentication with signature?

Ability to disallow authentication with signature enables two related use-cases:

1. Two-factor authentication schemes: where control over wallet is fully delegated to an extension that checks two signatures: the user’s one and the signature from the auth service. Naturally, if the signature authentication in the wallet remains allowed, the second factor check is bypassed.

2. Account recovery: delegating full control to another wallet in case of key compromise or loss. Wallet may contain larger amount of assets and its address could be tied to long-term contracts, therefore delegation to another controlling account is preferred to simply transferring the assets.

### What is library on masterchain?

Library is a special code storage mechanism that allows to reduce storage cost for a new Wallet V5 contract instance. Wallet V5 contract code is stored into a masterchain library.
Expand All @@ -143,7 +125,7 @@ wallet_id$_ global_id:int32 wc:int8 version:(## 8) subwallet_number:(## 32) = Wa
- `global_id` is a TON chain identifier. TON Mainnet `global_id = -239` and TON Testnet `global_id = -3`.
- `wc` is a Workchain. -1 for Masterchain and 0 for Basechain.
- `version`: current version of wallet v5 is `0`.
- `subwallet_number` can be used to get multiplie wallet contracts binded to the single keypair.
- `subwallet_number` can be used to get multiple wallet contracts bound to the single keypair.

## Packed address

Expand All @@ -166,48 +148,40 @@ Action types:
```tl-b
// Standard actions from block.tlb:
out_list_empty$_ = OutList 0;
out_list$_ {n:#} prev:^(OutList n) action:OutAction
= OutList (n + 1);
action_send_msg#0ec3c86d mode:(## 8)
out_msg:^(MessageRelaxed Any) = OutAction;
action_set_code#ad4de08e new_code:^Cell = OutAction;
action_reserve_currency#36e6b809 mode:(## 8)
currency:CurrencyCollection = OutAction;
libref_hash$0 lib_hash:bits256 = LibRef;
libref_ref$1 library:^Cell = LibRef;
action_change_library#26fa1dd4 mode:(## 7) { mode <= 2 }
libref:LibRef = OutAction;
out_list$_ {n:#} prev:^(OutList n) action:OutAction = OutList (n + 1);
action_send_msg#0ec3c86d mode:(## 8) out_msg:^(MessageRelaxed Any) = OutAction;

// Extended actions in W5:
action_list_basic$0 {n:#} actions:^(OutList n) = ActionList n 0;
action_list_extended$1 {m:#} {n:#} action:ExtendedAction prev:^(ActionList n m) = ActionList n (m+1);

action_set_data#1ff8ea0b data:^Cell = ExtendedAction;
action_add_ext#1c40db9f addr:MsgAddressInt = ExtendedAction;
action_delete_ext#5eaef4a4 addr:MsgAddressInt = ExtendedAction;
action_set_signature_auth_allowed#20cbb95a allowed:(## 1) = ExtendedAction;
```

Authentication modes:

```tl-b
signed_request$_
signature: bits512 // 512
subwallet_id: uint32 // 512+32
valid_until: uint32 // 512+32+32
msg_seqno: uint32 // 512+32+32+32 = 608
inner: InnerRequest = SignedRequest;

internal_signed#7369676E signed:SignedRequest = InternalMsgBody;
internal_extension#6578746E inner:InnerRequest = InternalMsgBody;
external_signed#7369676E signed:SignedRequest = ExternalMsgBody;
signed_request$_ // 32 (opcode from outer)
wallet_id: WalletID // 80
valid_until: # // 32
msg_seqno: # // 32
inner: InnerRequest // 1 .. (1 + 32 + 256) + ^Cell
signature: bits512 // 512
= SignedRequest; // Total: 688 .. 976 + ^Cell

internal_signed#73696e74 signed:SignedRequest = InternalMsgBody;
internal_extension#6578746e inner:InnerRequest = InternalMsgBody;
external_signed#7369676e signed:SignedRequest = ExternalMsgBody;

actions$_ {m:#} {n:#} actions:(ActionList n m) = InnerRequest;
```

Contract state:
```tl-b
wallet_id$_ global_id:int32 wc:int8 version:(## 8) subwallet_number:(## 32) = WalletID;
contract_state$_ seqno:# wallet_id:WalletID public_key:(## 256) extensions_dict:(HashmapE 256 int8) = ContractState;
wallet_id$_ global_id:# wc:int8 version:(## 8) subwallet_number:# = WalletID;
contract_state$_ seqno:int33 wallet_id:WalletID public_key:(## 256) extensions_dict:(HashmapE 256 int8) = ContractState;
```

## Source code
Expand Down
Binary file added contest.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading