Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: override same chunk #4749

Merged
merged 14 commits into from
Aug 26, 2024
Merged

fix: override same chunk #4749

merged 14 commits into from
Aug 26, 2024

Conversation

acha-bill
Copy link
Contributor

@acha-bill acha-bill commented Aug 6, 2024

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Override chunk with the same address.

When we attempt to delete the chunk, the delete operation is appended to the current batch. However, since this Get reads from disk, it returns the old refCnt. So it ignores the new chunk.

To fix this, we introduce a new replace method that does not relly on refCnt.

Related Issue (Optional)

#4739

@acha-bill acha-bill requested a review from ldeffenb August 6, 2024 13:19
Copy link
Collaborator

@ldeffenb ldeffenb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the look of this better, simply replace it within the transaction rather than the possibility of dangling/orphan chunks.

@@ -242,6 +242,13 @@ func (c *chunkStoreTrx) Iterate(ctx context.Context, fn storage.IterateChunkFn)
return chunkstore.Iterate(ctx, c.indexStore, c.sharkyTrx, fn)
}

func (c *chunkStoreTrx) Replace(ctx context.Context, ch swarm.Chunk) (err error) {
defer handleMetric("chunkstore_replace", c.metrics)(&err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See if I can remember to add this metric to one of my Grafana dashboards!

Copy link
Member

@nugaon nugaon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it resolves #4739 , I hoped we can handle two problems at the same time.

This fix does not cover the case when one uses postage stamp for the replaced single owner chunk; that stamp is used up for a non-living chunk on the uploader side when storage providers do not benefit from this pull syncing.
Former is not that significant problem especially if the batch is mutable while the last could be a huge improvement IMO since this is a supported feature in storage incentives with which operators can grow they reserve.

Nevertheless, there is no chunk retrieval mechanism to download different payloads of SOC at the moment, (also may require sharky changes) so we should merge this in order to have a more solid shelling point.

pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
@acha-bill acha-bill requested a review from istae August 12, 2024 08:32
Copy link
Member

@istae istae left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much better, let's also add some comments though

pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Outdated Show resolved Hide resolved
Copy link
Member

@nugaon nugaon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I was not aware of these kind of different stores used for storing chunk, but I have some questions/comments regarding that below.

pkg/storer/internal/reserve/reserve.go Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Show resolved Hide resolved
pkg/storer/internal/reserve/reserve.go Show resolved Hide resolved
@acha-bill acha-bill requested a review from istae August 19, 2024 12:37
Copy link
Member

@istae istae left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!
one final change request https://github.com/ethersphere/bee/pull/4749/files#diff-e37eed2ab5dfe1533be0257473f5b76738361f02d2be71da1b7bf4d5216a00c7R294: the size change here now also has to consider the same chunk address check.

should be:

if !loadedStamp && sameAddressOldChunkStamp == nil {
   r.size.Add(1)
}

@acha-bill acha-bill merged commit cd86a9f into master Aug 26, 2024
14 checks passed
@acha-bill acha-bill deleted the fix-override-chunk branch August 26, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants