Add GENERIC storage quota check to the file upload framework.

[landreev]

Several issues have been opened for adding mechanisms for storage quotas. There is some overlap between them, but different kinds of quotas have been requested - per-user, per-dataset, per-collection, per-user-per-day. Some appear to be more urgent than the others. (case in point, the most recent one, #8549, opened by me, is a specific request from our curation team that we wanted to address soon).

• Add mechanism for collection-wise storage size quotas #8549
• Limit the amount of data depositors may upload every day #7829
• Storage allocation quota per user #4339
• File and dataset limits: Add a programmatic way to limit file size and dataset size #3939

(there may be a couple more that are complete duplicates of something requested by one or more of the issues above, those are omitted).

Each one of these issues does require the actual quota check to be present somewhere in the workflow of uploading and creating a file. So this new issue is just for adding that mechanism. Perhaps it can be tested against a single setting that defines the limit for ALL uploads, but once it's there it should be extendable to add more types of quota checks to satisfy the specific cases and scenarios requested in the issues above. It should be programmatically configurable, and it should be enforced everywhere where the application allows a creation of a new file - via the GUI, the API and in the direct upload workflow.

It can be as simple as slightly modifying the individual file size check that's already there; but it should probably be made more programmatic, etc.

[landreev]

We may want to consider adding a dedicated command for file creation. Otherwise, file uploads is one area/subsystem that's notoriously resource-intensive, but is completely outside the Command Engine framework and not traceable in the ActionLogRecord.
This would take some careful consideration; it's not clear to me if this is actually viable or practical necessary, especially as the place where we could enforce the quotas.

[mreekie]

grooming
just a note for self - linked to #9409 which is still on the board.

[cmbz]

2023/09/11

• Added NIH 2.1.2 label and moved to the NIH GREI column for prioritization.

[cmbz]

2023/09/25: Added to 6.1 milestone as per conversation during prioritization meeting.