Feature Request: Option to copy submissions when cloning a project

[jbaileyhandle]

Currently, when a project is copied, the submissions to the original project are not copied over to the cloned project. This would useful in some cases.
A recurring issue for EECS 370 (and I imagine other classes) is that we often want to upgrade our autograder scripts, but validation can make this challenging.

That is, it is very important that our scripts be well tested before releasing our projects to our 800 students. We do not want to change our scripts after releasing the projects to avoid student confusion, so we can't really use our students as guinea pigs for our new scripts.

At the same time, it is hard to truely validate our scripts just on example submissions we come up with. Our test submissions will never give as much coverage as 100s of students do.

So, we have a chicken and egg problem. Ideally, we'd like to be able to test new scripts on old student submissions. This could be enabled by copying student submissions to a project when we make a copy of the project. Then, we can update the scripts for the copy and rerun submissions. Note that we do not wish to rerun all submissions on the original project to avoid potentially losing student scores.

[james-perretta]

I have gotten a request for this once, maybe twice before, but for different reasons. In principle I'm not opposed to this. There are a few things that will need to be investigated though, off the top of my head, in no particular order:

1. Implications for storage space on the server - For most courses the space used by all submissions is pretty small, but there are a few where student submissions come close to the maximum allowed size (around 50MB), and that can add up.
2. Total runtime for the copy - I'd have to take some measurements of how long it takes to copy all the database objects (probably not that long) and all the files (potentially a while).
3. Where to allow this - Copying a whole course already comes close to the HTTP request time limit, so this would likely only be an option for copying a single project (which is perhaps all you're asking for).

Could you tell me a bit more about your desired workflow for checking your script changes? That will help me understand the advantages of this approach over other potential workarounds.

Thanks!

[nelsontm]

Currently we have internal testing projects hidden to students where we upload the new instructor files, and test against all student submissions to see if the scores changed drastically or not as intended. If all the numbers look right, we upload the new instructor files to the main project public to students.

[james-perretta]

To check the scores, are you doing something like downloading scores before and after and running a diff?

[jbaileyhandle]

More or less. I think the idea is to download the scores and compare the distributions + mean, median etc. before and after using our own scripts.

[james-perretta]

Got it, I'll keep thinking about the best way to address this. I'm open to ideas and suggestions if you happen to have any : )

[james-perretta]

Here's a possible solution that I'd like to propose. It's not the most efficient strategy for uploading a bunch of submissions at once, but it would also let me address a couple other occasionally-requested features. The gist of it is 1) Add an API endpoint that lets admins upload a submission for a particular group, and 2) Add an API endpoint that lets admins pseudo delete a submission.

In more detail:

1. Add an API endpoint that imports groups, but not their submissions from one project into another. /projects/{project_id}/import_groups/import_from={other_project_id}
2. Add an API endpoint that lets admins import a single submission for a group. The endpoint URL would be something like /groups/{group_id}/import_submission/ and it would take the files to submit in the request body. It would immediately mark the submission as finished grading so that it doesn't go through the regular grading process. A rerun can then be used to grade the submission.
3. Add an API endpoint that lets admins pseudo-delete a submission. The endpoint URL would be /submissions/{submission_id}/pseudo_delete/. It would work by: 1) Creating a pseudo-deleted copy of the group the submission belongs to, and 2) Move that submission into that pseudo-deleted group

With those in place, you'd be able to import groups into your private project, download the submission files you want to test, use the API to "submit" those files to the appropriate groups, and rerun those submissions.

Thoughts?