Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK: Add MinIO compatibility #37

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

etlam
Copy link

@etlam etlam commented Oct 15, 2020

This pull requests adds compatibilty for MinIO.

Problems with MinIO:

  • resources beneath resources are not supported by MinIO
  • use_path_style_endpoint=true must be used

@@ -394,6 +394,9 @@ protected function getRelativePublicationPathAndFilename(ResourceMetaDataInterfa
{
if ($object->getRelativePublicationPath() !== '') {
$pathAndFilename = $object->getRelativePublicationPath() . $object->getFilename();
} elseif (isset($this->s3DefaultProfile['minio']) && $this->s3DefaultProfile['minio']) {
// If MinIO is used we have to change to url style. Creating an object beneath an object is not supported
$pathAndFilename = $object->getSha1() . '_folder/' . $object->getFilename();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@etlam Can you point to the Minio document discussing this limititation ? I use Minio for a few project with Flow ... don't know if i'm lucky but it work with just the path style configuration.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@kdambekalns kdambekalns Mar 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a valid discussion, but is it actually failing for you?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we use Sitegeist.Kaleidoscope in our project. I do not know if it would fail without Kaleidoscope.

@dfeyer
Copy link
Contributor

dfeyer commented Mar 20, 2021

@robertlemke @kdambekalns I can confirm that Minio did not have the limitation that this PR try to solve. Running a project and currently URL like http://storage-medialib.cms.ttree.localhost/public/sites/core/c31d44f75b3ba322d5219cd1e17c2ba933a6db5a/9WO0CELoWq6tSdjqCs34Y0qek7l-420x236.jpg are perfectly valid and works with latest docker image of Minio

But use_path_style_endpoint is required and the README update can be nice even if the Minio document is clear about this https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html

@etlam
Copy link
Author

etlam commented Mar 22, 2021

If you use for example Sitegeist.Kaleidoscope the original image gets saved as jhgsjsjhkshgezwwef and the smaller thumbnails try to get saved as jhgsjsjhkshgezwwef/<hash_of_variant>[...].
So the original image should be used as file and folder at the same time. This is not possible with MinIO because of the used underlying linux file system.

A folder and a file can not have the same name and be located in the same folder:
minio/minio#7335 (comment)
minio/minio#10160

Maybe there is a more elegant way to prevent that there is a file and a folder with the same name.

@kdambekalns
Copy link
Member

See #42

@etlam
Copy link
Author

etlam commented Jul 20, 2021

Yes, you have to use use_path_style_endpoint=true for MinIO.
But the problem with same file and folder names remains, as linked in my comment above:
minio/minio#7335 (comment)
minio/minio#10160

AWS S3 can use a file as a folder and store more files inside of it, MinIO can not do this.

We can close the pull request if using Sitegeist.Kaleidoscope with MinIO is a special case.

kdambekalns added a commit that referenced this pull request Jul 20, 2021
See #42 and #37 for the rationale.
@kdambekalns
Copy link
Member

Still not sure this is an issue with Sitegeist.Kaleidoscope, nor that it is a general issue:

  • Flow stores persistent in resources in a storage using just the hash
  • Resources and variants are published in a target using a pattern of hash/name

Since the hash is dependent on the file contents, two different files will have a different hash, and as long storage and target have separate buckets (as required for the S3 adapter), there should never be a clash…

Are you in fact using seperate buckets for storage and target, @etlam?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants