Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(pinning): check before writing root address #4558

Merged
merged 5 commits into from
Jan 30, 2024
Merged

Conversation

notanatol
Copy link
Contributor

@notanatol notanatol commented Jan 25, 2024

Checklist

  • I have read the coding guide.
  • I have filled out the description and linked the related issues.

Description

Closes: #4527

@notanatol notanatol changed the title fix: check before writing root address fix(pinning): check before writing root address Jan 25, 2024
@notanatol notanatol marked this pull request as draft January 25, 2024 00:30
Copy link
Collaborator

@ldeffenb ldeffenb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I corrected my original review by moving that comment to Close() where it belongs, but added a new concern with the Has() check that was added to Put().

err := writer.Put(c.collection)
if err != nil {
return fmt.Errorf("pin store: failed updating collection: %w", err)
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't solve the problem of residual collectionChunks with their corresponding forever-remaining increments of the chunkstore RefCnt. Consider:

  1. Two or more simultaneous operations start pinning the exact same content.
  2. They all do the individual Puts, and the root collection doesn't yet exist, so they all actually store their collectionChunks and invoke chunkstore.Put incrementing RefCnt on those chunks.
  3. One of the pinners actually executes the Close causing the pinCollectionItem to exist. That one "wins" and actually completes the pin.
  4. When the other pinners finally invoke Close, they'll correctly not replace the original pinCollectionItem, but all of those incorrect RefCnt increments are still there causing those chunks to never be able to be removed from the chunkstore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically returning an error on line :276 will make cleanupOnErrWriter wrapper object to perform a cleanup (by calling the Cleanup method) whenever the error is detected

bee/pkg/api/api.go

Lines 870 to 874 in bf56d82

func (r *cleanupOnErrWriter) WriteHeader(statusCode int) {
// if there is an error status returned, cleanup.
if statusCode >= http.StatusBadRequest {
err := r.onErr()
if err != nil {

notice the onErr pointing to the .Cleanup method

bee/pkg/api/bzz.go

Lines 106 to 110 in bf56d82

ow := &cleanupOnErrWriter{
ResponseWriter: w,
onErr: putter.Cleanup,
logger: logger,
}

which in turn will decrement the ref count(s) to the correct value. See the test below.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for that description. I should have followed the error path back out from Close() myself. I'm setting up a testnet environment to verify this on a full OSM tileset load whenever it is merged.

@notanatol notanatol self-assigned this Jan 28, 2024
@notanatol notanatol added the ready for review The PR is ready to be reviewed label Jan 28, 2024
@notanatol notanatol marked this pull request as ready for review January 28, 2024 16:16
Copy link
Collaborator

@ldeffenb ldeffenb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Been tracking this one as it developed and the current version looks like it should solve the pinning refCnt issue that it intends to solve. Looking forward to testing it once it merges.

@notanatol notanatol merged commit cffd1cc into master Jan 30, 2024
12 checks passed
@notanatol notanatol deleted the fix-pin-refcnt branch January 30, 2024 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pull-request ready for review The PR is ready to be reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upload w/Pinning duplicates RefCnt
4 participants