Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle null preconditions correctly and simplify gcs lib again #8400

Merged
merged 3 commits into from
Feb 13, 2025

Conversation

tylerwilliams
Copy link
Member

No description provided.

bw := g.bucketHandle.Object(blobName).NewWriter(ctx)
// See https://pkg.go.dev/cloud.google.com/go/storage#Writer
// Always disable buffering in the client for these writes.
bw.ChunkSize = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from the gcs docs:

	// If ChunkSize is set to zero, chunking will be disabled and the object will
	// be uploaded in a single request without the use of a buffer. This will
	// further reduce memory used during uploads, but will also prevent the writer
	// from retrying in case of a transient error from the server or resuming an
	// upload that fails midway through, since the buffer is required in order to
	// retry the failed request.

is there some higher-level retry logic i'm missing to compensate for this? (it didn't look like it to me, but i'm also not sure we should care?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't add anything, but given that we're already disabling this for chunks under 16M (most chunks) it seems fairly safe. LMK if you disagree tho, not attached to this part if you feel strongly.

@tylerwilliams tylerwilliams merged commit 7022bb0 into master Feb 13, 2025
15 checks passed
@tylerwilliams tylerwilliams deleted the gcs_cleanup branch February 13, 2025 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants