feat: pinning index in memory when building, second try

[usamoi]

issue: #117
first try: #150

[VoVAllen]

Let's try it on 100M dataset and see how flamegraph goes

[usamoi]

2c4t, on laion-100m:

• estimated build time on main: ~60h
• estimated build time on pinning-v2: ~50h

next steps:

• merge two shared memory segments to one
• make it optional

cc @VoVAllen

[VoVAllen]

Why do we need to merge two segments together?

The optimization result is pretty close to the flamegraph result.

[VoVAllen]

And what is 2c4t?

[usamoi]

And what is 2c4t?

4 vcpu

Why do we need to merge two segments together?

not necessary; helpful to make it optional

[usamoi]

i4i.xlarge, laion-100m:

• estimated build time on main: ~61h
• estimated build time on pinning-v2: ~44h