perf: paginate account credit debits

[avichalp]

• Paginate account debiting using HAMT's for_each_ranged method. Store the next address process in the current debit cycle in the state.
• Paginate blob deletion by keeping the next index of AMT in the state

closes #478

[sanderpick]

this is looking good to me. are there any red flags popping up for you?

[avichalp]

@sanderpick: no red flags but was just worrying if i am missing an edge case.

where should we put the pagination batch sizes? it is currently hardcoded in this PR

[sanderpick]

@sanderpick: no red flags but was just worrying if i am missing an edge case.

where should we put the pagination batch sizes? it is currently hardcoded in this PR

you could put in the recall_config actor. note: that will require an SDK update. there are commands to set the config.

re: the tests... can you manually try and figure out what the threshholds are for these batch sizes? ie, at what point do they run out of gas. maybe there's a way to use the recall loader-cli (cc @brunocalza) to add many 1000s of accounts and an bucket and object from each? or maybe there's a theoretical way to understand, ie, how much gas each iteration takes and if it grows linearly (it should since we're using hamt/amt). that would be nice, since load testing the expiries sounds like it would take some extra custom work.

[avichalp]

Here is a break down of manually testing the debit_accounts method and different batch sized.

When walking the outer expiry AMT N times, we could go upto ~ 330 before running into block gas limit (10B).

1 account and N blobs

1: 29_528_398
100: 2_756_052_956
250: 7_392_148_602
300: 8_881_775_388
325: 9_651_803_132
330: 9_868_023_508
335: 9_995_108_942
336: out of gas

However, debit_accounts also iterate over Accounts in addition to expiry AMT. With Accounts traversal the breakdown looks something like this:

N accounts with 1 blob each (excludes expiry traversal gas)

1: 11_519_090
335: 820_064_125
1000: 2_096_304_681
2000: 3_617_460_618
5000: 7_618_952_490
6000: 8_976_209_296
6500: 9_659_512_057
6700: 9_912_982_462
6800: out of gas

---

Combining these two I have been trying account debiting with the following. It consumes ~ 9.2-9.3B gas units which almost fills the whole block but remains within the BLOCK_GAS_LIMIT

• expiry batch size: 150
• accounts batch size: 2000

Note: we should also reduce the debit interval if we go with expiry batch size of 150 to ensure there is not much backlog

[sanderpick]

thanks for the great rundown here! maybe,

expiry batch size: 100
accounts batch size: 1000

would be better? more conservative? i wonder also, is there a way we can not propagate the error is this system level tx fails? kind of like we did for the blob reader callback method?

[avichalp]

@sanderpick: 100 and 1000 sounds good to me!

I added the error handling for debit accounts call here: 084a3d1. I had to create the return object with default values in the error case to satisfy the return type. Is this okay?

Also, here are the two PRs for

• config command updates in rust-recall: Add batch size config rust-recall#214
• event field updates in recall-contracts: fix: update set config event contracts#84

I believe the set_config tests are failing at the moment due to changed event interface.

Also, where do we call the set_config? I think, that is the place where we actually set the values 1000 and 100.

[sanderpick]

@sanderpick: 100 and 1000 sounds good to me!

I added the error handling for debit accounts call here: 084a3d1. I had to create the return object with default values in the error case to satisfy the return type. Is this okay?

I think so... does it work when you hit the out of gas error? ie, does the chain keep progressing? I wonder also, is there some way to know if the debiting falls behind with some logging?

Also, here are the two PRs for

• config command updates in rust-recall: Add batch size config rust-recall#214
• event field updates in recall-contracts: fix: update set config event contracts#84

I believe the set_config tests are failing at the moment due to changed event interface.

👀

Also, where do we call the set_config? I think, that is the place where we actually set the values 1000 and 100.

it's called from the SDK recall subnet config set command. so far we haven't used it -- the initial values are used.

[avichalp]

@sanderpick:

I think so... does it work when you hit the out of gas error? ie, does the chain keep progressing?

interestingly, we don't need to explicitly handle the error here. having state.execute_implicit(msg)? doesn't stop the chain. i recall, i saw this behaviour early on with blob reader too. but i thought i might have missed something at the time, so later changed that to explicit handling

[sanderpick]

ah, so the tx just fails like a normal user txn that fails? and then the chain would just try again during the next "cron" tick?

[avichalp]

yes, it seems so. a WARN is logged

[avichalp]

adds two new config parameters

[avichalp]

Uses "next index" to continue the cursor based pagination

[avichalp]

next key for pagination

[avichalp]

logs next index and current epoch so we can get a sense of backlog

[sanderpick]

sorry for delay here... will look asap

[sanderpick]

looks great! this is a big improvement 👍