-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(pinning): check before writing root address #4558
Conversation
bf608d2
to
6ca0ffe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I corrected my original review by moving that comment to Close() where it belongs, but added a new concern with the Has() check that was added to Put().
err := writer.Put(c.collection) | ||
if err != nil { | ||
return fmt.Errorf("pin store: failed updating collection: %w", err) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't solve the problem of residual collectionChunks with their corresponding forever-remaining increments of the chunkstore RefCnt. Consider:
- Two or more simultaneous operations start pinning the exact same content.
- They all do the individual Puts, and the root collection doesn't yet exist, so they all actually store their collectionChunks and invoke chunkstore.Put incrementing RefCnt on those chunks.
- One of the pinners actually executes the Close causing the pinCollectionItem to exist. That one "wins" and actually completes the pin.
- When the other pinners finally invoke Close, they'll correctly not replace the original pinCollectionItem, but all of those incorrect RefCnt increments are still there causing those chunks to never be able to be removed from the chunkstore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically returning an error on line :276 will make cleanupOnErrWriter
wrapper object to perform a cleanup (by calling the Cleanup
method) whenever the error is detected
Lines 870 to 874 in bf56d82
func (r *cleanupOnErrWriter) WriteHeader(statusCode int) { | |
// if there is an error status returned, cleanup. | |
if statusCode >= http.StatusBadRequest { | |
err := r.onErr() | |
if err != nil { |
notice the
onErr
pointing to the .Cleanup
methodLines 106 to 110 in bf56d82
ow := &cleanupOnErrWriter{ | |
ResponseWriter: w, | |
onErr: putter.Cleanup, | |
logger: logger, | |
} |
which in turn will decrement the ref count(s) to the correct value. See the test below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for that description. I should have followed the error path back out from Close() myself. I'm setting up a testnet environment to verify this on a full OSM tileset load whenever it is merged.
db8dce2
to
c0f555f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Been tracking this one as it developed and the current version looks like it should solve the pinning refCnt issue that it intends to solve. Looking forward to testing it once it merges.
Checklist
Description
Closes: #4527