Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/minio file upload #125

Merged
merged 29 commits into from
Jan 15, 2024
Merged

Conversation

sandstromviktor
Copy link
Contributor

@sandstromviktor sandstromviktor commented Jan 12, 2024

Description

Ok, this is a big one. Sorry about that.

The main goal was to enable users to upload files in a simple manner. We decided that minio does the job.
After this PR, you can upload files with Minio using the S3-API and the UI.

How to test? Right now, this is up and running on serve-dev, so please go there and test it in different ways.

What have i done?

  1. Fixed the minio helm chart to start up and create a bucket and a non admin user.
  2. Added a task that deletes minio after 24 hours
  3. Created an additional minio app called "minio admin" (name can change) that can run in a ML project without being automatically terminated.
  4. In the UI, i have made a new part called "manage files" with its own logic to reveal passwords, copy on click etc.

In addition: To help with the locust tests, i've made 5 django admin commands that populate the database with temp users, projects and apps. This should probably be a separate PR, but i included it so i could test it on dev simultaneously with the file manage part.

Let me know what you think!

Checklist

If you're unsure about any of the items below, don't hesitate to ask. We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

  • This pull request is against develop branch (not applicable for hotfixes)
  • I have included a link to the issue on GitHub or JIRA (if any)
  • I have included migration files (if there are changes to the model classes)
  • I have included, reviewed and executed tests (unit and end2end) to complement my changes
  • I have updated the related documentation (if necessary)
  • I have updated the release notes (releasenotes.md)
  • I have added a reviewer for this pull request
  • I have added myself as an author for this pull request
  • In the case I have modified settings.py, then I have also updated the studio-settings-configmap.yaml file in serve-charts

@sandstromviktor sandstromviktor added the new feature A new feature label Jan 12, 2024
@sandstromviktor sandstromviktor requested a review from a team January 12, 2024 07:50
@sandstromviktor sandstromviktor self-assigned this Jan 12, 2024
@sandstromviktor sandstromviktor marked this pull request as ready for review January 12, 2024 09:14
Copy link
Collaborator

@akochari akochari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Tested thoroughly on dev.

@sandstromviktor sandstromviktor merged commit 0a0f24a into develop Jan 15, 2024
2 checks passed
@sandstromviktor sandstromviktor deleted the feature/minio-file-upload branch January 15, 2024 07:55
Copy link
Contributor

@churnikov churnikov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General question, why are we calling it "manage files" while it could be called "temporary file storage"?
And are we describing somewhere why one might need it?

Comment on lines +428 to +432
def generate_password(len: int) -> str:
return "".join(
secrets.choice(string.octdigits + string.ascii_uppercase + string.ascii_lowercase + string.digits)
for i in range(len)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why aren't you using django's password generator? I believe, that this way is most probably insecure and exploitable

Comment on lines +23 to +26
except IntegrityError:
self.stdout.write(self.style.WARNING("User with the given username or email already exists."))

self.stdout.write(self.style.SUCCESS(f"Successfully created {i} users"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If IntegrityError is raised, then there should be less successful creations of users



@register.filter
def dict_key(dictionary, key):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of words about what is that would have helped understand its purpose. I feel like I get it, but still

@churnikov
Copy link
Contributor

image On minio bucket creation date looks weird

Is there a way to see how much storage is given to me?


I also feel like I don't understand the time limit? What would happen to data afterwards?


Tried to use s3 via api. Didn't work for me

aws s3 ls --profile serve_tmp_storage --endpoint-url https://api-r2964ba7f.serve-dev.scilifelab.se

SSL validation failed for https://api-r2964ba7f.serve-dev.scilifelab.se/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

@sandstromviktor sandstromviktor mentioned this pull request Jan 19, 2024
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants