Adding an array of state lists #4599

kozlovb · 2023-08-04T13:17:37Z

Description

Replace free and used list by an array of lists.

Replace appendElement removeElement by introducing a change state method.

Benchmark

benchmark performed with -
https://github.com/onflow/flow-go/blob/master/module/mempool/stdmap/backDataHeapBenchmark_test.go#L45-L65

results for current PR

{"level":"info","interaction_time":450172.126897,"time":"2023-08-28T14:04:00Z","message":"adding elements done"}
{"level":"info","gc-elapsed-time":12.706404735,"time":"2023-08-28T14:04:13Z","message":"garbage collection done"}
goos: darwin
goarch: amd64
pkg: github.com/onflow/flow-go/module/mempool/stdmap
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkArrayBackDataLRU-12 1 612802990076 ns/op 15210332256 B/op 600000676 allocs/op
PASS
ok github.com/onflow/flow-go/module/mempool/stdmap 616.531s

results for flow-go/master

{"level":"info","interaction_time":496179.697692,"time":"2023-08-28T14:20:11Z","message":"adding elements done"}
{"level":"info","gc-elapsed-time":8.206771913,"time":"2023-08-28T14:20:19Z","message":"garbage collection done"}
goos: darwin
goarch: amd64
pkg: github.com/onflow/flow-go/module/mempool/stdmap
cpu: Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
BenchmarkArrayBackDataLRU-12 1 652748634400 ns/op 15210392456 B/op 600000773 allocs/op
PASS
ok github.com/onflow/flow-go/module/mempool/stdmap 656.203s

kozlovb · 2023-08-04T14:23:45Z

Regarding the 5. of the definition of done -
"Implementation should be supported by appropriate test coverage."

It's an a bit tricky situation cause this change doesn't modify public methods. And we have discussed with @yhassanzadeh13 that news tests should call only public methods. Last time in a similar situation I implemented an additional test - TestAddAndRemoveEntities and added checks like checkEachEntityIsInFreeOrUsedState and discoverEntitiesBelongingToStateList to existing tests. Given that this PR is sized, I think 2's less than the previous one, may be we could agree on some improvement of tests or could skip this.
I am not sure about it and ready to discuss.

kc1116 · 2023-08-11T16:11:53Z

module/mempool/herocache/backdata/heropool/pool_test.go

@@ -726,13 +726,13 @@ func testAddingEntities(t *testing.T, pool *Pool, entitiesToBeAdded []*unittest.
 			usedTraverseStep = uint32(len(pool.poolEntities))
 		}
 		tailAccessibleFromHead(t,
-			pool.used.head,
-			pool.used.tail,
+			pool.states[stateUsed].head,


nit: Should add funcs to states like used() and free() since we use pool.states[SOME_INDEX] often.

Just to clarify that I understood you correctly. You suggest that pool.states[stateUsed].head is converted to pool.states.used().head ?
or pool.used().head - if a used() method is implemented in the pool since the pool has control over a number of states and state for now has fields allowing it to represent any linked list with in the pool.

yhassanzadeh13

Please also post your benchmarking comparison against master in PR description, similar to #4470.

module/mempool/herocache/backdata/heropool/pool.go

kozlovb · 2023-08-25T11:36:28Z

Hi @yhassanzadeh13 I applied most of the comments except for one where I need clarifications. Please take a look.

I wanted to ask you, do you know Bastian Müller ? I have this PR - onflow/cadence-tools#176. Initially, was my home task for BuiildSquad, so with Nozim we are doing for free. I need some clarifications from Bastian or if for whatever reason this is not relevant then I'd like to know if I can close it. Could you please ask him if he would be ok that I contact him or if he can reach out to me on discord - BorisK#0954. It's not super important but if we can finalize it somehow that would be great.

yhassanzadeh13 · 2023-08-25T21:12:04Z

Hi @yhassanzadeh13 I applied most of the comments except for one where I need clarifications. Please take a look.

I wanted to ask you, do you know Bastian Müller ? I have this PR - onflow/cadence-tools#176. Initially, was my home task for BuiildSquad, so with Nozim we are doing for free. I need some clarifications from Bastian or if for whatever reason this is not relevant then I'd like to know if I can close it. Could you please ask him if he would be ok that I contact him or if he can reach out to me on discord - BorisK#0954. It's not super important but if we can finalize it somehow that would be great.

cc @turbolent for the Cadence related issue.

Also @kozlovb please let me know which comment you need clarification on.

comments except for one where I need clarifications

And also;

Please also post your benchmarking comparison against master in PR description, similar to #4470.

kozlovb · 2023-08-28T13:20:38Z

@yhassanzadeh13 I got a bit confused -
"Should add funcs to states like used() and free() since we use pool.states[SOME_INDEX] often."
I feel like @kc1116 meant pool.states.free() ( not sure that I understood correctly)
However, in the current design it seems that state is more like a linked list that doesn't know about how many different states the pool might need.
Currently, we have this in pool.go -

// StateIndex is a type of a state of a placeholder in a pool.
type StateIndex uint

const numberOfStates = 2
const ( // iota is reset to 0
   stateFree StateIndex = iota
   stateUsed
)

So, I was thinking may be I should apply pool.free() instead of pool.states.free() or may be pool.free() was meant on the first place. What do you think ? If pool.states.free()is required then I'll move a definition of number of states in linked list.go too.

Will do, for a benchmark.

thx, Bastian replied.

yhassanzadeh13

Thank you for incorporating the comments. Everything looks good to me! Additionally, I ran the benchmark locally and compared the results; both the garbage collection time and object allocation remain nearly unchanged, which is a positive outcome.

@kozlovb please merge the master into your branch to merge your PR.

Build-Squad:boriskozlov/array-of-lists-rebased

{"level":"info","total_slots_written":2000,"total_interactions_since_last_log":10000,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":0,"available_slots":5,"total_buckets":0,"available_slots":6,"total_buckets":0,"available_slots":7,"total_buckets":0,"available_slots":8,"total_buckets":0,"available_slots":9,"total_buckets":0,"available_slots":10,"total_buckets":0,"available_slots":11,"total_buckets":0,"available_slots":12,"total_buckets":0,"available_slots":13,"total_buckets":0,"available_slots":14,"total_buckets":2,"available_slots":15,"total_buckets":78,"available_slots":16,"total_buckets":1920,"time":"2023-09-05T21:19:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":4489337,"total_interactions_since_last_log":22436686,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":3,"available_slots":5,"total_buckets":23,"available_slots":6,"total_buckets":141,"available_slots":7,"total_buckets":849,"available_slots":8,"total_buckets":3853,"available_slots":9,"total_buckets":15326,"available_slots":10,"total_buckets":53374,"available_slots":11,"total_buckets":160454,"available_slots":12,"total_buckets":401617,"available_slots":13,"total_buckets":803693,"available_slots":14,"total_buckets":1211363,"available_slots":15,"total_buckets":1216770,"available_slots":16,"total_buckets":621871,"time":"2023-09-05T21:19:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":9129587,"total_interactions_since_last_log":23201249,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":6,"available_slots":5,"total_buckets":48,"available_slots":6,"total_buckets":313,"available_slots":7,"total_buckets":1729,"available_slots":8,"total_buckets":7844,"available_slots":9,"total_buckets":31393,"available_slots":10,"total_buckets":108955,"available_slots":11,"total_buckets":327830,"available_slots":12,"total_buckets":820478,"available_slots":13,"total_buckets":1640051,"available_slots":14,"total_buckets":2467548,"available_slots":15,"total_buckets":2473556,"available_slots":16,"total_buckets":1249836,"time":"2023-09-05T21:19:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":13658878,"total_interactions_since_last_log":22646456,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":7,"available_slots":5,"total_buckets":75,"available_slots":6,"total_buckets":495,"available_slots":7,"total_buckets":2604,"available_slots":8,"total_buckets":11729,"available_slots":9,"total_buckets":46923,"available_slots":10,"total_buckets":163559,"available_slots":11,"total_buckets":490697,"available_slots":12,"total_buckets":1228677,"available_slots":13,"total_buckets":2458123,"available_slots":14,"total_buckets":3694005,"available_slots":15,"total_buckets":3699330,"available_slots":16,"total_buckets":1862654,"time":"2023-09-05T21:19:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":18069763,"total_interactions_since_last_log":22054425,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":18,"available_slots":5,"total_buckets":118,"available_slots":6,"total_buckets":678,"available_slots":7,"total_buckets":3440,"available_slots":8,"total_buckets":15415,"available_slots":9,"total_buckets":62035,"available_slots":10,"total_buckets":216863,"available_slots":11,"total_buckets":650133,"available_slots":12,"total_buckets":1626844,"available_slots":13,"total_buckets":3253127,"available_slots":14,"total_buckets":4887336,"available_slots":15,"total_buckets":4894177,"available_slots":16,"total_buckets":2459579,"time":"2023-09-05T21:19:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":22598752,"total_interactions_since_last_log":22644945,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":1,"available_slots":4,"total_buckets":25,"available_slots":5,"total_buckets":142,"available_slots":6,"total_buckets":822,"available_slots":7,"total_buckets":4253,"available_slots":8,"total_buckets":19283,"available_slots":9,"total_buckets":77618,"available_slots":10,"total_buckets":271663,"available_slots":11,"total_buckets":814296,"available_slots":12,"total_buckets":2035387,"available_slots":13,"total_buckets":4070019,"available_slots":14,"total_buckets":6112933,"available_slots":15,"total_buckets":6120275,"available_slots":16,"total_buckets":3072034,"time":"2023-09-05T21:19:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":27058307,"total_interactions_since_last_log":22297775,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":3,"available_slots":4,"total_buckets":34,"available_slots":5,"total_buckets":167,"available_slots":6,"total_buckets":986,"available_slots":7,"total_buckets":5118,"available_slots":8,"total_buckets":23151,"available_slots":9,"total_buckets":93072,"available_slots":10,"total_buckets":325190,"available_slots":11,"total_buckets":974746,"available_slots":12,"total_buckets":2436664,"available_slots":13,"total_buckets":4875639,"available_slots":14,"total_buckets":7319307,"available_slots":15,"total_buckets":7328927,"available_slots":16,"total_buckets":3675301,"time":"2023-09-05T21:20:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":31571249,"total_interactions_since_last_log":22564710,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":4,"available_slots":4,"total_buckets":41,"available_slots":5,"total_buckets":191,"available_slots":6,"total_buckets":1119,"available_slots":7,"total_buckets":5930,"available_slots":8,"total_buckets":26960,"available_slots":9,"total_buckets":108678,"available_slots":10,"total_buckets":379620,"available_slots":11,"total_buckets":1137179,"available_slots":12,"total_buckets":2844217,"available_slots":13,"total_buckets":5688811,"available_slots":14,"total_buckets":8541049,"available_slots":15,"total_buckets":8551644,"available_slots":16,"total_buckets":4285804,"time":"2023-09-05T21:20:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":36117374,"total_interactions_since_last_log":22730625,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":4,"available_slots":4,"total_buckets":44,"available_slots":5,"total_buckets":213,"available_slots":6,"total_buckets":1277,"available_slots":7,"total_buckets":6803,"available_slots":8,"total_buckets":30831,"available_slots":9,"total_buckets":124058,"available_slots":10,"total_buckets":433977,"available_slots":11,"total_buckets":1301113,"available_slots":12,"total_buckets":3255187,"available_slots":13,"total_buckets":6509599,"available_slots":14,"total_buckets":9772079,"available_slots":15,"total_buckets":9781613,"available_slots":16,"total_buckets":4900574,"time":"2023-09-05T21:20:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":40630660,"total_interactions_since_last_log":22566426,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":5,"available_slots":4,"total_buckets":48,"available_slots":5,"total_buckets":237,"available_slots":6,"total_buckets":1418,"available_slots":7,"total_buckets":7616,"available_slots":8,"total_buckets":34661,"available_slots":9,"total_buckets":139247,"available_slots":10,"total_buckets":488092,"available_slots":11,"total_buckets":1463622,"available_slots":12,"total_buckets":3662595,"available_slots":13,"total_buckets":7325215,"available_slots":14,"total_buckets":10994376,"available_slots":15,"total_buckets":11002534,"available_slots":16,"total_buckets":5510992,"time":"2023-09-05T21:20:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":45088171,"total_interactions_since_last_log":22287559,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":5,"available_slots":4,"total_buckets":52,"available_slots":5,"total_buckets":269,"available_slots":6,"total_buckets":1601,"available_slots":7,"total_buckets":8490,"available_slots":8,"total_buckets":38607,"available_slots":9,"total_buckets":154588,"available_slots":10,"total_buckets":541793,"available_slots":11,"total_buckets":1624354,"available_slots":12,"total_buckets":4064721,"available_slots":13,"total_buckets":8130448,"available_slots":14,"total_buckets":12201077,"available_slots":15,"total_buckets":12208052,"available_slots":16,"total_buckets":6114112,"time":"2023-09-05T21:20:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":49503921,"total_interactions_since_last_log":22078750,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":2,"available_slots":3,"total_buckets":8,"available_slots":4,"total_buckets":57,"available_slots":5,"total_buckets":313,"available_slots":6,"total_buckets":1776,"available_slots":7,"total_buckets":9369,"available_slots":8,"total_buckets":42455,"available_slots":9,"total_buckets":169978,"available_slots":10,"total_buckets":595187,"available_slots":11,"total_buckets":1783482,"available_slots":12,"total_buckets":4462607,"available_slots":13,"total_buckets":8926868,"available_slots":14,"total_buckets":13395946,"available_slots":15,"total_buckets":13404398,"available_slots":16,"total_buckets":6711475,"time":"2023-09-05T21:20:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":53955145,"total_interactions_since_last_log":22256116,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":10,"available_slots":4,"total_buckets":63,"available_slots":5,"total_buckets":359,"available_slots":6,"total_buckets":1956,"available_slots":7,"total_buckets":10234,"available_slots":8,"total_buckets":46228,"available_slots":9,"total_buckets":185157,"available_slots":10,"total_buckets":648754,"available_slots":11,"total_buckets":1944660,"available_slots":12,"total_buckets":4863564,"available_slots":13,"total_buckets":9729973,"available_slots":14,"total_buckets":14600395,"available_slots":15,"total_buckets":14609851,"available_slots":16,"total_buckets":7313938,"time":"2023-09-05T21:21:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":58433759,"total_interactions_since_last_log":22393074,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":10,"available_slots":4,"total_buckets":65,"available_slots":5,"total_buckets":387,"available_slots":6,"total_buckets":2142,"available_slots":7,"total_buckets":11098,"available_slots":8,"total_buckets":50074,"available_slots":9,"total_buckets":200594,"available_slots":10,"total_buckets":702472,"available_slots":11,"total_buckets":2105778,"available_slots":12,"total_buckets":5267629,"available_slots":13,"total_buckets":10538260,"available_slots":14,"total_buckets":15813264,"available_slots":15,"total_buckets":15822958,"available_slots":16,"total_buckets":7919025,"time":"2023-09-05T21:21:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":62853798,"total_interactions_since_last_log":22100191,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":11,"available_slots":4,"total_buckets":71,"available_slots":5,"total_buckets":415,"available_slots":6,"total_buckets":2306,"available_slots":7,"total_buckets":11865,"available_slots":8,"total_buckets":53692,"available_slots":9,"total_buckets":215546,"available_slots":10,"total_buckets":755130,"available_slots":11,"total_buckets":2265036,"available_slots":12,"total_buckets":5666474,"available_slots":13,"total_buckets":11335862,"available_slots":14,"total_buckets":17010203,"available_slots":15,"total_buckets":17020040,"available_slots":16,"total_buckets":8517144,"time":"2023-09-05T21:21:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":67282349,"total_interactions_since_last_log":22142759,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":11,"available_slots":4,"total_buckets":77,"available_slots":5,"total_buckets":453,"available_slots":6,"total_buckets":2479,"available_slots":7,"total_buckets":12678,"available_slots":8,"total_buckets":57515,"available_slots":9,"total_buckets":230762,"available_slots":10,"total_buckets":808137,"available_slots":11,"total_buckets":2424944,"available_slots":12,"total_buckets":6066313,"available_slots":13,"total_buckets":12135173,"available_slots":14,"total_buckets":18209096,"available_slots":15,"total_buckets":18218723,"available_slots":16,"total_buckets":9115985,"time":"2023-09-05T21:21:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":71731376,"total_interactions_since_last_log":22245131,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":81,"available_slots":5,"total_buckets":485,"available_slots":6,"total_buckets":2646,"available_slots":7,"total_buckets":13542,"available_slots":8,"total_buckets":61328,"available_slots":9,"total_buckets":246128,"available_slots":10,"total_buckets":861674,"available_slots":11,"total_buckets":2585378,"available_slots":12,"total_buckets":6467227,"available_slots":13,"total_buckets":12937639,"available_slots":14,"total_buckets":19413095,"available_slots":15,"total_buckets":19423780,"available_slots":16,"total_buckets":9718358,"time":"2023-09-05T21:21:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":76158658,"total_interactions_since_last_log":22136413,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":87,"available_slots":5,"total_buckets":516,"available_slots":6,"total_buckets":2802,"available_slots":7,"total_buckets":14393,"available_slots":8,"total_buckets":65110,"available_slots":9,"total_buckets":261410,"available_slots":10,"total_buckets":914662,"available_slots":11,"total_buckets":2745208,"available_slots":12,"total_buckets":6866944,"available_slots":13,"total_buckets":13737437,"available_slots":14,"total_buckets":20612189,"available_slots":15,"total_buckets":20619616,"available_slots":16,"total_buckets":10318269,"time":"2023-09-05T21:21:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":80623691,"total_interactions_since_last_log":22325166,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":95,"available_slots":5,"total_buckets":551,"available_slots":6,"total_buckets":2954,"available_slots":7,"total_buckets":15265,"available_slots":8,"total_buckets":68892,"available_slots":9,"total_buckets":276821,"available_slots":10,"total_buckets":968343,"available_slots":11,"total_buckets":2907448,"available_slots":12,"total_buckets":7270441,"available_slots":13,"total_buckets":14543235,"available_slots":14,"total_buckets":21818888,"available_slots":15,"total_buckets":21827780,"available_slots":16,"total_buckets":10922963,"time":"2023-09-05T21:22:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":85040925,"total_interactions_since_last_log":22086170,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":101,"available_slots":5,"total_buckets":587,"available_slots":6,"total_buckets":3125,"available_slots":7,"total_buckets":16105,"available_slots":8,"total_buckets":72699,"available_slots":9,"total_buckets":291921,"available_slots":10,"total_buckets":1021359,"available_slots":11,"total_buckets":3066733,"available_slots":12,"total_buckets":7668866,"available_slots":13,"total_buckets":15340260,"available_slots":14,"total_buckets":23014678,"available_slots":15,"total_buckets":23023543,"available_slots":16,"total_buckets":11520933,"time":"2023-09-05T21:22:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":89500474,"total_interactions_since_last_log":22297744,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":105,"available_slots":5,"total_buckets":607,"available_slots":6,"total_buckets":3275,"available_slots":7,"total_buckets":16981,"available_slots":8,"total_buckets":76583,"available_slots":9,"total_buckets":307115,"available_slots":10,"total_buckets":1074534,"available_slots":11,"total_buckets":3227509,"available_slots":12,"total_buckets":8070281,"available_slots":13,"total_buckets":16145392,"available_slots":14,"total_buckets":24222329,"available_slots":15,"total_buckets":24231689,"available_slots":16,"total_buckets":12124058,"time":"2023-09-05T21:22:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":93910177,"total_interactions_since_last_log":22048515,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":15,"available_slots":4,"total_buckets":112,"available_slots":5,"total_buckets":635,"available_slots":6,"total_buckets":3460,"available_slots":7,"total_buckets":17812,"available_slots":8,"total_buckets":80412,"available_slots":9,"total_buckets":322173,"available_slots":10,"total_buckets":1127711,"available_slots":11,"total_buckets":3387075,"available_slots":12,"total_buckets":8468807,"available_slots":13,"total_buckets":16941698,"available_slots":14,"total_buckets":25414243,"available_slots":15,"total_buckets":25425809,"available_slots":16,"total_buckets":12720212,"time":"2023-09-05T21:22:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":98355959,"total_interactions_since_last_log":22228907,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":3,"available_slots":3,"total_buckets":15,"available_slots":4,"total_buckets":114,"available_slots":5,"total_buckets":663,"available_slots":6,"total_buckets":3627,"available_slots":7,"total_buckets":18649,"available_slots":8,"total_buckets":84210,"available_slots":9,"total_buckets":337628,"available_slots":10,"total_buckets":1181683,"available_slots":11,"total_buckets":3547872,"available_slots":12,"total_buckets":8869739,"available_slots":13,"total_buckets":17743136,"available_slots":14,"total_buckets":26617654,"available_slots":15,"total_buckets":26629208,"available_slots":16,"total_buckets":13321758,"time":"2023-09-05T21:22:40Z","message":"logging telemetry"}
{"level":"info","interaction_time":223656.095792,"time":"2023-09-05T21:22:43Z","message":"adding elements done"}
{"level":"info",".Alloc":15212373432,".AllocObjects":500007183,".TotalAlloc":15218045344,".NumGC":5,"time":"2023-09-05T21:22:43Z","message":"heap allocation digest"}
{"level":"info","gc-elapsed-time":0.4294355,"time":"2023-09-05T21:22:44Z","message":"garbage collection done"}
{"level":"info",".Alloc":2403920,".AllocObjects":6847,".TotalAlloc":15218047784,".NumGC":6,"time":"2023-09-05T21:22:44Z","message":"heap allocation digest"}
BenchmarkArrayBackDataLRU-10    	       1	274607587375 ns/op
PASS

flow-go/master

pkg: github.com/onflow/flow-go/module/mempool/stdmap
BenchmarkArrayBackDataLRU
{"level":"info","total_slots_written":2000,"total_interactions_since_last_log":10000,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":0,"available_slots":5,"total_buckets":0,"available_slots":6,"total_buckets":0,"available_slots":7,"total_buckets":0,"available_slots":8,"total_buckets":0,"available_slots":9,"total_buckets":0,"available_slots":10,"total_buckets":0,"available_slots":11,"total_buckets":0,"available_slots":12,"total_buckets":0,"available_slots":13,"total_buckets":0,"available_slots":14,"total_buckets":1,"available_slots":15,"total_buckets":89,"available_slots":16,"total_buckets":1910,"time":"2023-09-05T21:04:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":4418600,"total_interactions_since_last_log":22083000,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":0,"available_slots":4,"total_buckets":7,"available_slots":5,"total_buckets":30,"available_slots":6,"total_buckets":160,"available_slots":7,"total_buckets":798,"available_slots":8,"total_buckets":3809,"available_slots":9,"total_buckets":15298,"available_slots":10,"total_buckets":52637,"available_slots":11,"total_buckets":158069,"available_slots":12,"total_buckets":396045,"available_slots":13,"total_buckets":791905,"available_slots":14,"total_buckets":1189784,"available_slots":15,"total_buckets":1197415,"available_slots":16,"total_buckets":612643,"time":"2023-09-05T21:04:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":8826515,"total_interactions_since_last_log":22039576,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":2,"available_slots":4,"total_buckets":12,"available_slots":5,"total_buckets":61,"available_slots":6,"total_buckets":345,"available_slots":7,"total_buckets":1685,"available_slots":8,"total_buckets":7768,"available_slots":9,"total_buckets":30580,"available_slots":10,"total_buckets":105674,"available_slots":11,"total_buckets":317608,"available_slots":12,"total_buckets":794950,"available_slots":13,"total_buckets":1586741,"available_slots":14,"total_buckets":2382176,"available_slots":15,"total_buckets":2389873,"available_slots":16,"total_buckets":1209040,"time":"2023-09-05T21:04:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":13433633,"total_interactions_since_last_log":23035590,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":3,"available_slots":4,"total_buckets":23,"available_slots":5,"total_buckets":102,"available_slots":6,"total_buckets":509,"available_slots":7,"total_buckets":2549,"available_slots":8,"total_buckets":11789,"available_slots":9,"total_buckets":46180,"available_slots":10,"total_buckets":160779,"available_slots":11,"total_buckets":483210,"available_slots":12,"total_buckets":1211491,"available_slots":13,"total_buckets":2419348,"available_slots":14,"total_buckets":3628231,"available_slots":15,"total_buckets":3635887,"available_slots":16,"total_buckets":1833532,"time":"2023-09-05T21:04:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":18027245,"total_interactions_since_last_log":22968056,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":5,"available_slots":4,"total_buckets":31,"available_slots":5,"total_buckets":154,"available_slots":6,"total_buckets":705,"available_slots":7,"total_buckets":3484,"available_slots":8,"total_buckets":15679,"available_slots":9,"total_buckets":61744,"available_slots":10,"total_buckets":215709,"available_slots":11,"total_buckets":647916,"available_slots":12,"total_buckets":1625834,"available_slots":13,"total_buckets":3248504,"available_slots":14,"total_buckets":4872858,"available_slots":15,"total_buckets":4879557,"available_slots":16,"total_buckets":2455065,"time":"2023-09-05T21:05:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":22382750,"total_interactions_since_last_log":21777529,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":5,"available_slots":4,"total_buckets":35,"available_slots":5,"total_buckets":195,"available_slots":6,"total_buckets":865,"available_slots":7,"total_buckets":4303,"available_slots":8,"total_buckets":19378,"available_slots":9,"total_buckets":76656,"available_slots":10,"total_buckets":268014,"available_slots":11,"total_buckets":804387,"available_slots":12,"total_buckets":2018520,"available_slots":13,"total_buckets":4033805,"available_slots":14,"total_buckets":6053685,"available_slots":15,"total_buckets":6059353,"available_slots":16,"total_buckets":3043549,"time":"2023-09-05T21:05:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":26604842,"total_interactions_since_last_log":21110459,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":5,"available_slots":4,"total_buckets":40,"available_slots":5,"total_buckets":235,"available_slots":6,"total_buckets":1045,"available_slots":7,"total_buckets":5094,"available_slots":8,"total_buckets":22937,"available_slots":9,"total_buckets":90960,"available_slots":10,"total_buckets":318759,"available_slots":11,"total_buckets":956711,"available_slots":12,"total_buckets":2400476,"available_slots":13,"total_buckets":4796688,"available_slots":14,"total_buckets":7196532,"available_slots":15,"total_buckets":7200223,"available_slots":16,"total_buckets":3615137,"time":"2023-09-05T21:05:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":30728103,"total_interactions_since_last_log":20616306,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":7,"available_slots":4,"total_buckets":45,"available_slots":5,"total_buckets":263,"available_slots":6,"total_buckets":1191,"available_slots":7,"total_buckets":5859,"available_slots":8,"total_buckets":26404,"available_slots":9,"total_buckets":105152,"available_slots":10,"total_buckets":368270,"available_slots":11,"total_buckets":1106014,"available_slots":12,"total_buckets":2771995,"available_slots":13,"total_buckets":5541447,"available_slots":14,"total_buckets":8312659,"available_slots":15,"total_buckets":8315428,"available_slots":16,"total_buckets":4173369,"time":"2023-09-05T21:05:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":35099287,"total_interactions_since_last_log":21855916,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":7,"available_slots":4,"total_buckets":45,"available_slots":5,"total_buckets":287,"available_slots":6,"total_buckets":1354,"available_slots":7,"total_buckets":6663,"available_slots":8,"total_buckets":30111,"available_slots":9,"total_buckets":120315,"available_slots":10,"total_buckets":420558,"available_slots":11,"total_buckets":1263955,"available_slots":12,"total_buckets":3166863,"available_slots":13,"total_buckets":6329722,"available_slots":14,"total_buckets":9495734,"available_slots":15,"total_buckets":9499235,"available_slots":16,"total_buckets":4764438,"time":"2023-09-05T21:05:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":39425847,"total_interactions_since_last_log":21632802,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":7,"available_slots":4,"total_buckets":51,"available_slots":5,"total_buckets":316,"available_slots":6,"total_buckets":1512,"available_slots":7,"total_buckets":7528,"available_slots":8,"total_buckets":33888,"available_slots":9,"total_buckets":135546,"available_slots":10,"total_buckets":472909,"available_slots":11,"total_buckets":1420311,"available_slots":12,"total_buckets":3556964,"available_slots":13,"total_buckets":7110404,"available_slots":14,"total_buckets":10666576,"available_slots":15,"total_buckets":10669797,"available_slots":16,"total_buckets":5350038,"time":"2023-09-05T21:05:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":43660308,"total_interactions_since_last_log":21172307,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":8,"available_slots":4,"total_buckets":56,"available_slots":5,"total_buckets":343,"available_slots":6,"total_buckets":1680,"available_slots":7,"total_buckets":8345,"available_slots":8,"total_buckets":37590,"available_slots":9,"total_buckets":150263,"available_slots":10,"total_buckets":524024,"available_slots":11,"total_buckets":1572968,"available_slots":12,"total_buckets":3938384,"available_slots":13,"total_buckets":7875236,"available_slots":14,"total_buckets":11811702,"available_slots":15,"total_buckets":11816991,"available_slots":16,"total_buckets":5922718,"time":"2023-09-05T21:06:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":47932758,"total_interactions_since_last_log":21362250,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":0,"available_slots":3,"total_buckets":8,"available_slots":4,"total_buckets":59,"available_slots":5,"total_buckets":375,"available_slots":6,"total_buckets":1865,"available_slots":7,"total_buckets":9138,"available_slots":8,"total_buckets":41219,"available_slots":9,"total_buckets":164829,"available_slots":10,"total_buckets":575328,"available_slots":11,"total_buckets":1727327,"available_slots":12,"total_buckets":4323901,"available_slots":13,"total_buckets":8647119,"available_slots":14,"total_buckets":12967955,"available_slots":15,"total_buckets":12972708,"available_slots":16,"total_buckets":6500927,"time":"2023-09-05T21:06:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":52303873,"total_interactions_since_last_log":21855573,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":10,"available_slots":4,"total_buckets":61,"available_slots":5,"total_buckets":410,"available_slots":6,"total_buckets":2044,"available_slots":7,"total_buckets":10005,"available_slots":8,"total_buckets":45157,"available_slots":9,"total_buckets":180114,"available_slots":10,"total_buckets":628057,"available_slots":11,"total_buckets":1885335,"available_slots":12,"total_buckets":4719109,"available_slots":13,"total_buckets":9435088,"available_slots":14,"total_buckets":14150420,"available_slots":15,"total_buckets":14155328,"available_slots":16,"total_buckets":7092734,"time":"2023-09-05T21:06:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":56902651,"total_interactions_since_last_log":22993892,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":70,"available_slots":5,"total_buckets":432,"available_slots":6,"total_buckets":2212,"available_slots":7,"total_buckets":10843,"available_slots":8,"total_buckets":48976,"available_slots":9,"total_buckets":196127,"available_slots":10,"total_buckets":683713,"available_slots":11,"total_buckets":2051614,"available_slots":12,"total_buckets":5133748,"available_slots":13,"total_buckets":10264122,"available_slots":14,"total_buckets":15395398,"available_slots":15,"total_buckets":15400216,"available_slots":16,"total_buckets":7715167,"time":"2023-09-05T21:06:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":61334841,"total_interactions_since_last_log":22160949,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":76,"available_slots":5,"total_buckets":453,"available_slots":6,"total_buckets":2361,"available_slots":7,"total_buckets":11668,"available_slots":8,"total_buckets":52711,"available_slots":9,"total_buckets":211210,"available_slots":10,"total_buckets":737380,"available_slots":11,"total_buckets":2211720,"available_slots":12,"total_buckets":5533763,"available_slots":13,"total_buckets":11065133,"available_slots":14,"total_buckets":16595046,"available_slots":15,"total_buckets":16598274,"available_slots":16,"total_buckets":8315033,"time":"2023-09-05T21:06:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":65782998,"total_interactions_since_last_log":22240786,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":12,"available_slots":4,"total_buckets":80,"available_slots":5,"total_buckets":485,"available_slots":6,"total_buckets":2538,"available_slots":7,"total_buckets":12529,"available_slots":8,"total_buckets":56558,"available_slots":9,"total_buckets":226375,"available_slots":10,"total_buckets":790623,"available_slots":11,"total_buckets":2371889,"available_slots":12,"total_buckets":5935095,"available_slots":13,"total_buckets":11868800,"available_slots":14,"total_buckets":17798817,"available_slots":15,"total_buckets":17802280,"available_slots":16,"total_buckets":8916916,"time":"2023-09-05T21:06:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":70261284,"total_interactions_since_last_log":22391429,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":85,"available_slots":5,"total_buckets":520,"available_slots":6,"total_buckets":2709,"available_slots":7,"total_buckets":13421,"available_slots":8,"total_buckets":60395,"available_slots":9,"total_buckets":241804,"available_slots":10,"total_buckets":844535,"available_slots":11,"total_buckets":2533427,"available_slots":12,"total_buckets":6338724,"available_slots":13,"total_buckets":12675880,"available_slots":14,"total_buckets":19012160,"available_slots":15,"total_buckets":19015277,"available_slots":16,"total_buckets":9522333,"time":"2023-09-05T21:07:00Z","message":"logging telemetry"}
{"level":"info","total_slots_written":74426833,"total_interactions_since_last_log":20827745,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":87,"available_slots":5,"total_buckets":552,"available_slots":6,"total_buckets":2880,"available_slots":7,"total_buckets":14213,"available_slots":8,"total_buckets":63916,"available_slots":9,"total_buckets":256069,"available_slots":10,"total_buckets":895414,"available_slots":11,"total_buckets":2683566,"available_slots":12,"total_buckets":6714479,"available_slots":13,"total_buckets":13426304,"available_slots":14,"total_buckets":20140543,"available_slots":15,"total_buckets":20143096,"available_slots":16,"total_buckets":10085700,"time":"2023-09-05T21:07:10Z","message":"logging telemetry"}
{"level":"info","total_slots_written":78629521,"total_interactions_since_last_log":21013441,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":93,"available_slots":5,"total_buckets":577,"available_slots":6,"total_buckets":3048,"available_slots":7,"total_buckets":14993,"available_slots":8,"total_buckets":67580,"available_slots":9,"total_buckets":270644,"available_slots":10,"total_buckets":946081,"available_slots":11,"total_buckets":2835328,"available_slots":12,"total_buckets":7094769,"available_slots":13,"total_buckets":14184826,"available_slots":14,"total_buckets":21277659,"available_slots":15,"total_buckets":21279571,"available_slots":16,"total_buckets":10654338,"time":"2023-09-05T21:07:20Z","message":"logging telemetry"}
{"level":"info","total_slots_written":83027955,"total_interactions_since_last_log":21992166,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":99,"available_slots":5,"total_buckets":605,"available_slots":6,"total_buckets":3203,"available_slots":7,"total_buckets":15851,"available_slots":8,"total_buckets":71362,"available_slots":9,"total_buckets":285690,"available_slots":10,"total_buckets":998550,"available_slots":11,"total_buckets":2993264,"available_slots":12,"total_buckets":7491756,"available_slots":13,"total_buckets":14978978,"available_slots":14,"total_buckets":22469536,"available_slots":15,"total_buckets":22469696,"available_slots":16,"total_buckets":11249351,"time":"2023-09-05T21:07:30Z","message":"logging telemetry"}
{"level":"info","total_slots_written":87357040,"total_interactions_since_last_log":21645425,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":13,"available_slots":4,"total_buckets":104,"available_slots":5,"total_buckets":634,"available_slots":6,"total_buckets":3401,"available_slots":7,"total_buckets":16682,"available_slots":8,"total_buckets":75000,"available_slots":9,"total_buckets":300490,"available_slots":10,"total_buckets":1050807,"available_slots":11,"total_buckets":3149262,"available_slots":12,"total_buckets":7883215,"available_slots":13,"total_buckets":15760835,"available_slots":14,"total_buckets":23639913,"available_slots":15,"total_buckets":23640909,"available_slots":16,"total_buckets":11835774,"time":"2023-09-05T21:07:40Z","message":"logging telemetry"}
{"level":"info","total_slots_written":91893052,"total_interactions_since_last_log":22680064,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":14,"available_slots":4,"total_buckets":110,"available_slots":5,"total_buckets":663,"available_slots":6,"total_buckets":3578,"available_slots":7,"total_buckets":17567,"available_slots":8,"total_buckets":78998,"available_slots":9,"total_buckets":316269,"available_slots":10,"total_buckets":1105266,"available_slots":11,"total_buckets":3313149,"available_slots":12,"total_buckets":8293433,"available_slots":13,"total_buckets":16579172,"available_slots":14,"total_buckets":24866897,"available_slots":15,"total_buckets":24867901,"available_slots":16,"total_buckets":12450034,"time":"2023-09-05T21:07:50Z","message":"logging telemetry"}
{"level":"info","total_slots_written":96476561,"total_interactions_since_last_log":22917545,"available_slots":0,"total_buckets":0,"available_slots":1,"total_buckets":0,"available_slots":2,"total_buckets":1,"available_slots":3,"total_buckets":15,"available_slots":4,"total_buckets":115,"available_slots":5,"total_buckets":691,"available_slots":6,"total_buckets":3737,"available_slots":7,"total_buckets":18423,"available_slots":8,"total_buckets":83038,"available_slots":9,"total_buckets":332250,"available_slots":10,"total_buckets":1160719,"available_slots":11,"total_buckets":3478387,"available_slots":12,"total_buckets":8708450,"available_slots":13,"total_buckets":17404647,"available_slots":14,"total_buckets":26106850,"available_slots":15,"total_buckets":26108400,"available_slots":16,"total_buckets":13070838,"time":"2023-09-05T21:08:00Z","message":"logging telemetry"}
{"level":"info","interaction_time":228196.294958,"time":"2023-09-05T21:08:08Z","message":"adding elements done"}
{"level":"info",".Alloc":15212376600,".AllocObjects":500007219,".TotalAlloc":15218051360,".NumGC":4,"time":"2023-09-05T21:08:08Z","message":"heap allocation digest"}
{"level":"info","gc-elapsed-time":0.479802792,"time":"2023-09-05T21:08:09Z","message":"garbage collection done"}
{"level":"info",".Alloc":2407080,".AllocObjects":6884,".TotalAlloc":15218053800,".NumGC":5,"time":"2023-09-05T21:08:09Z","message":"heap allocation digest"}
BenchmarkArrayBackDataLRU-10    	       1	279933742542 ns/op
PASS

yhassanzadeh13 · 2023-09-05T21:27:12Z

@yhassanzadeh13 I got a bit confused - "Should add funcs to states like used() and free() since we use pool.states[SOME_INDEX] often." I feel like @kc1116 meant pool.states.free() ( not sure that I understood correctly) However, in the current design it seems that state is more like a linked list that doesn't know about how many different states the pool might need. Currently, we have this in pool.go -
// StateIndex is a type of a state of a placeholder in a pool.
type StateIndex uint

const numberOfStates = 2
const ( // iota is reset to 0
   stateFree StateIndex = iota
   stateUsed
)
So, I was thinking may be I should apply pool.free() instead of pool.states.free() or may be pool.free() was meant on the first place. What do you think ? If pool.states.free()is required then I'll move a definition of number of states in linked list.go too.

Will do, for a benchmark.

thx, Bastian replied.

Thanks for the suggestion, but the current implementation looks good, and there is no need for further abstractions.

codecov-commenter · 2023-09-05T21:42:13Z

Codecov Report

Patch coverage: 65.38% and project coverage change: -1.61% ⚠️

Comparison is base (2528190) 56.25% compared to head (c4913f0) 54.65%.
Report is 1146 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4599      +/-   ##
==========================================
- Coverage   56.25%   54.65%   -1.61%     
==========================================
  Files         653      918     +265     
  Lines       64699    85611   +20912     
==========================================
+ Hits        36396    46787   +10391     
- Misses      25362    35225    +9863     
- Partials     2941     3599     +658

Flag	Coverage Δ
unittests	`54.65% <65.38%> (-1.61%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
cmd/bootstrap/transit/cmd/utils.go	`31.54% <ø> (ø)`
cmd/execution_builder.go	`0.00% <0.00%> (ø)`
engine/access/rest/middleware/metrics.go	`0.00% <0.00%> (ø)`
engine/access/rest/routes/execution_result.go	`56.75% <0.00%> (ø)`
engine/access/rpc/backend/backend_network.go	`34.11% <0.00%> (ø)`
engine/access/state_stream/engine.go	`0.00% <0.00%> (ø)`
engine/common/rpc/convert/blocks.go	`78.76% <ø> (ø)`
fvm/environment/env.go	`100.00% <ø> (ø)`
model/flow/identifierList.go	`22.36% <0.00%> (+0.84%)`	⬆️
model/verification/chunkDataPackRequest.go	`43.75% <0.00%> (-6.25%)`	⬇️
... and 102 more

... and 250 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kozlovb · 2023-09-05T22:57:21Z

@yhassanzadeh13 I got a bit confused - "Should add funcs to states like used() and free() since we use pool.states[SOME_INDEX] often." I feel like @kc1116 meant pool.states.free() ( not sure that I understood correctly) However, in the current design it seems that state is more like a linked list that doesn't know about how many different states the pool might need. Currently, we have this in pool.go -
// StateIndex is a type of a state of a placeholder in a pool.
type StateIndex uint

const numberOfStates = 2
const ( // iota is reset to 0
   stateFree StateIndex = iota
   stateUsed
)
So, I was thinking may be I should apply pool.free() instead of pool.states.free() or may be pool.free() was meant on the first place. What do you think ? If pool.states.free()is required then I'll move a definition of number of states in linked list.go too.
Will do, for a benchmark.
thx, Bastian replied.
Thanks for the suggestion, but the current implementation looks good, and there is no need for further abstractions.

thank you for the feedback.

kozlovb · 2023-09-05T22:58:09Z

I'll try to merge soon.

kozlovb · 2023-09-06T16:53:57Z

@yhassanzadeh13 , @franklywatson I've just rebased :
bkozlov@C02X84NSJGH6 flow-go % git rebase remotes/upstream-flow-go/master
Successfully rebased and updated refs/heads/boriskozlov/array-of-lists-rebased.
Pls retry to submit.

kozlovb mentioned this pull request Aug 4, 2023

Boriskozlov/array of lists #4535

Closed

franklywatson requested review from kc1116 and yhassanzadeh13 August 10, 2023 16:50

kc1116 reviewed Aug 11, 2023

View reviewed changes

kc1116 approved these changes Aug 11, 2023

View reviewed changes

yhassanzadeh13 reviewed Aug 18, 2023

View reviewed changes

module/mempool/herocache/backdata/heropool/pool.go Outdated Show resolved Hide resolved

module/mempool/herocache/backdata/heropool/pool.go Outdated Show resolved Hide resolved

module/mempool/herocache/backdata/heropool/pool.go Outdated Show resolved Hide resolved

yhassanzadeh13 approved these changes Sep 5, 2023

View reviewed changes

franklywatson added this pull request to the merge queue Sep 5, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 5, 2023

franklywatson added this pull request to the merge queue Sep 5, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 5, 2023

franklywatson added this pull request to the merge queue Sep 5, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 5, 2023

Boris Kozlov added 3 commits September 6, 2023 18:51

Adding an array of state lists

7c13338

Improved comments

3b5cc0f

Applied minor comments

21129fe

kozlovb force-pushed the boriskozlov/array-of-lists-rebased branch from c4913f0 to 21129fe Compare September 6, 2023 16:52

franklywatson added this pull request to the merge queue Sep 6, 2023

Merged via the queue into onflow:master with commit 104ca4f Sep 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding an array of state lists #4599

Adding an array of state lists #4599

kozlovb commented Aug 4, 2023 •

edited

Loading

kozlovb commented Aug 4, 2023 •

edited

Loading

kc1116 Aug 11, 2023

kozlovb Aug 11, 2023

yhassanzadeh13 left a comment

kozlovb commented Aug 25, 2023

yhassanzadeh13 commented Aug 25, 2023 •

edited

Loading

kozlovb commented Aug 28, 2023 •

edited

Loading

yhassanzadeh13 left a comment •

edited

Loading

yhassanzadeh13 commented Sep 5, 2023

codecov-commenter commented Sep 5, 2023

kozlovb commented Sep 5, 2023

kozlovb commented Sep 5, 2023

kozlovb commented Sep 6, 2023

Adding an array of state lists #4599

Adding an array of state lists #4599

Conversation

kozlovb commented Aug 4, 2023 • edited Loading

Description

Benchmark

kozlovb commented Aug 4, 2023 • edited Loading

kc1116 Aug 11, 2023

Choose a reason for hiding this comment

kozlovb Aug 11, 2023

Choose a reason for hiding this comment

yhassanzadeh13 left a comment

Choose a reason for hiding this comment

kozlovb commented Aug 25, 2023

yhassanzadeh13 commented Aug 25, 2023 • edited Loading

kozlovb commented Aug 28, 2023 • edited Loading

yhassanzadeh13 left a comment • edited Loading

Choose a reason for hiding this comment

yhassanzadeh13 commented Sep 5, 2023

codecov-commenter commented Sep 5, 2023

Codecov Report

kozlovb commented Sep 5, 2023

kozlovb commented Sep 5, 2023

kozlovb commented Sep 6, 2023

kozlovb commented Aug 4, 2023 •

edited

Loading

kozlovb commented Aug 4, 2023 •

edited

Loading

yhassanzadeh13 commented Aug 25, 2023 •

edited

Loading

kozlovb commented Aug 28, 2023 •

edited

Loading

yhassanzadeh13 left a comment •

edited

Loading