Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid client-side file upload to bucket; use server-side #101

Closed
n8kim1 opened this issue Nov 10, 2020 · 2 comments
Closed

Avoid client-side file upload to bucket; use server-side #101

n8kim1 opened this issue Nov 10, 2020 · 2 comments

Comments

@n8kim1
Copy link
Contributor

n8kim1 commented Nov 10, 2020

NOTE we went w a different approach. See later comments for more info

Related to submissions randomly disappearing (#10, among others), here's a tweak that would be better practice and get safer results:
From Jerry:
File upload to bucket is used for processing submissions and resumes. Currently, in both cases the API returns a signed URL which the client can then use to upload the file directly to the bucket.

However if the client goes wrong (which internet browsers are wont to do), then the backend database records an upload, but the upload does not occur successfully. Specific causes may include the end-user refreshing the page before the upload completes, or unexpected errors during upload (e.g. CORS). This may be a cause of battlecode/battlecode20#194.

Using server-side file uploads will better ensure that anything recorded in the database actually exists. Additionally we can do the same thing to implement battlecode/battlecode20#24.
battlecode/battlecode20#379

@n8kim1
Copy link
Contributor Author

n8kim1 commented Dec 25, 2020

Note: the order should be: save to db, get link, upload to bucket, save in db again (to note that the submission's made it into the bucket), send message to comp pub/sub, save in db again (to note that the submission is queued for compiling)
This may require an intermediate submission status, eg "uploading"? (to have a more clear indicator of empty buckets) (note -- indeed, ultimately, two more intermediate submission statuses were made)
Out of this order could cause funny errors -- eg, comp pub/sub requires presence in bucket, so we respect this dependency through good timing. Also getting link requires a submission id (to properly order/assign buckets).

@n8kim1
Copy link
Contributor Author

n8kim1 commented Jan 2, 2021

Ultimately we decided to go with client-side upload:

  • server-side upload could cause a huge strain on our servers (receiving each submission zip, and then uploading it too), especially during pre-deadline rush
  • client-side bugs are much less frequent now, after revisions to submission process

We still had to follow some careful orders, as dictated by the second comment.

relevant changes have been made, and merged to master!

@n8kim1 n8kim1 closed this as completed Jan 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant