Merge branch 'develop'

.github/workflows/publish-to-pypi.yml

@@ -2,6 +2,9 @@ name: PyPI Publish
 
 on:
   push:
+    branches-ignore:
+      - 'develop'
+      - 'feature/*'
     tags:
       - v**
 

DjangoExampleApplication/models.py

@@ -43,7 +43,7 @@ def delete(self, *args, **kwargs):
         Delete must be overridden because the inherited delete method does not call `self.image.delete()`.
         """
         # noinspection PyUnresolvedReferences
-        self.image.delete()
+        self.file.delete()
         super(GenericAttachment, self).delete(*args, **kwargs)
 
 

Dockerfile

@@ -0,0 +1,14 @@
+# syntax=docker/dockerfile:1
+FROM python:3
+ENV PYTHONUNBUFFERED=1
+WORKDIR /code
+
+# Copy Demo Project
+COPY ./manage.py /code/manage.py
+COPY ./django_minio_backend /code/django_minio_backend
+COPY ./DjangoExampleProject /code/DjangoExampleProject
+COPY ./DjangoExampleApplication /code/DjangoExampleApplication
+
+# Copy and install requirements.txt
+COPY requirements.txt /code/
+RUN pip install -r /code/requirements.txt

README.Docker.md

@@ -0,0 +1,46 @@
+# Docker Compose Description for django-minio-backend
+Execute the following step to start a demo environment using Docker Compose:
+
+**Start the Docker Compose services:**
+ ```shell
+ docker-compose up -d
+ docker-compose exec web python manage.py createsuperuser --noinput
+ docker-compose exec web python manage.py collectstatic --noinput
+ ```
+
+## About docker-compose.yml
+Note the following lines in `docker-compose.yml`:
+```yaml
+ environment:
+   GH_MINIO_ENDPOINT: "nginx:9000"
+   GH_MINIO_USE_HTTPS: "false"
+   GH_MINIO_EXTERNAL_ENDPOINT: "localhost:9000"
+   GH_MINIO_EXTERNAL_ENDPOINT_USE_HTTPS: "false"
+```
+
+MinIO is load balanced by nginx, so all connections made from Django towards MinIO happens through the internal `nginx` FQDN. <br>
+Therefore, the value of `GH_MINIO_ENDPOINT` is `nginx:9000`.
+
+# Web Access
+Both Django(:8000) and MinIO(:9001) expose a Web GUI and their ports are mapped to the host machine.
+
+## Django Admin
+Open your browser at http://localhost:8000/admin to access the Django admin portal:
+  * username: `admin`
+  * password: `123123`
+
+## MinIO Console
+Open your browser at http://localhost:9001 to access the MiniIO Console:
+  * username: `minio`
+  * password: `minio123`
+
+# Developer Environment
+An alternative docker-compose file is available for **django-minio-backend** which does not copy the source files into the container, but maps them as a volume.
+**Input file**: `docker-compose.develop.yml`
+
+If you would like to develop in a Docker Compose environment, execute the following commands:
+```shell
+docker-compose --project-name "django-minio-backend-DEV" -f docker-compose.develop.yml up -d
+docker-compose --project-name "django-minio-backend-DEV" -f docker-compose.develop.yml exec web python manage.py createsuperuser --noinput
+docker-compose --project-name "django-minio-backend-DEV" -f docker-compose.develop.yml exec web python manage.py collectstatic --noinput
+```

README.md

@@ -159,7 +159,7 @@ For a reference implementation, see [Examples](examples).
 
 ## Behaviour
 The following list summarises the key characteristics of **django-minio-backend**:
-  * Bucket existence is **not** checked on save by default.
+  * Bucket existence is **not** checked on a save by default.
     To enable this guard, set `MINIO_BUCKET_CHECK_ON_SAVE = True` in your `settings.py`.
   * Bucket existences are **not** checked on Django start by default.
     To enable this guard, set `MINIO_CONSISTENCY_CHECK_ON_START = True` in your `settings.py`.
@@ -170,8 +170,21 @@ The following list summarises the key characteristics of **django-minio-backend*
   * Depending on your configuration, **django-minio-backend** may communicate over two kind of interfaces: internal and external.
     If your `settings.py` defines a different value for `MINIO_ENDPOINT` and `MINIO_EXTERNAL_ENDPOINT`, then the former will be used for internal communication
     between Django and MinIO, and the latter for generating URLs for users. This behaviour optimises the network communication.
+    See **Networking** below for a thorough explanation
   * The uploaded object's content-type is guessed during save. If `mimetypes.guess_type` fails to determine the correct content-type, then it falls back to `application/octet-stream`.
 
+## Networking and Docker
+If your Django application is running on a shared host with your MinIO instance, you should consider using the `MINIO_EXTERNAL_ENDPOINT` and `MINIO_EXTERNAL_ENDPOINT_USE_HTTPS` parameters.
+This way most traffic will happen internally between Django and MinIO. The external endpoint parameters are required for external pre-signed URL generation.
+
+If your Django application and MinIO instance are running on different hosts, you can omit the `MINIO_EXTERNAL_ENDPOINT` and `MINIO_EXTERNAL_ENDPOINT_USE_HTTPS` parameters, 
+and **django-minio-backend** will default to the value of `MINIO_ENDPOINT`.
+
+Setting up and configuring custom networks in Docker is not in the scope of this document. <br>
+To learn more about Docker networking, see [Networking overview](https://docs.docker.com/network/) and [Networking in Compose](https://docs.docker.com/compose/networking/).
+
+See [README.Docker.md](README.Docker.md) for a real-life Docker Compose demonstration.
+
 ## Compatibility
   * Django 2.2 or later
   * Python 3.6.0 or later

django_minio_backend/models.py

@@ -83,8 +83,8 @@ def __init__(self,
 
         self._REPLACE_EXISTING = kwargs.get('replace_existing', False)
 
-        self.__CLIENT: Union[minio.Minio, None] = None
-        self.__CLIENT_EXTERNAL: Union[minio.Minio, None] = None
+        self.__CLIENT: Union[minio.Minio, None] = None  # This client is used for internal communication only. Communication this way should not leave the host network's perimeter
+        self.__CLIENT_FAKE: Union[minio.Minio, None] = None  # This fake client is used for pre-signed URL generation only; it does not execute HTTP requests
         self.__MINIO_ENDPOINT: str = get_setting("MINIO_ENDPOINT")
         self.__MINIO_EXTERNAL_ENDPOINT: str = get_setting("MINIO_EXTERNAL_ENDPOINT", self.__MINIO_ENDPOINT)
         self.__MINIO_ACCESS_KEY: str = get_setting("MINIO_ACCESS_KEY")
@@ -165,7 +165,7 @@ def get_available_name(self, name, max_length=None):
             return name
         return super(MinioBackend, self).get_available_name(name, max_length)
 
-    def _open(self, object_name, mode='rb', **kwargs):
+    def _open(self, object_name, mode='rb', **kwargs) -> File:
         """
         Implements the Storage._open(name,mode='rb') method
         :param name (str): object_name [path to file excluding bucket name which is implied]
@@ -235,20 +235,18 @@ def url(self, name: str):
         """
         if self.is_bucket_public:
             return f'{self.base_url_external}/{self.bucket}/{name}'
+        if self.same_endpoints:
+            # in this scenario the fake client is not needed
+            client = self.client
+        else:
+            client = self.client_fake
 
         try:
-            if self.same_endpoints:
-                u: str = self.client.presigned_get_object(
-                    bucket_name=self.bucket,
-                    object_name=name.encode('utf-8'),
-                    expires=get_setting("MINIO_URL_EXPIRY_HOURS", timedelta(days=7))  # Default is 7 days
-                )
-            else:
-                u: str = self.client_external.presigned_get_object(
-                    bucket_name=self.bucket,
-                    object_name=name.encode('utf-8'),
-                    expires=get_setting("MINIO_URL_EXPIRY_HOURS", timedelta(days=7))  # Default is 7 days
-                )
+            u: str = client.presigned_get_object(
+                bucket_name=self.bucket,
+                object_name=name.encode('utf-8'),
+                expires=get_setting("MINIO_URL_EXPIRY_HOURS", timedelta(days=7))  # Default is 7 days
+            )
             return u
         except urllib3.exceptions.MaxRetryError:
             raise ConnectionError("Couldn't connect to Minio. Check django_minio_backend parameters in Django-Settings")
@@ -332,19 +330,17 @@ def is_minio_available(self) -> MinioServerStatus:
 
     @property
     def client(self) -> minio.Minio:
-        """Get handle to an (already) instantiated minio.Minio instance (for internal URL access)"""
+        """Get handle to an (already) instantiated minio.Minio instance"""
         if not self.__CLIENT:
-            self.new_client()
-            return self.__CLIENT
+            return self._create_new_client()
         return self.__CLIENT
 
     @property
-    def client_external(self) -> minio.Minio:
-        """Get handle to an (already) instantiated minio.Minio instance (for external URL access)"""
-        if not self.__CLIENT_EXTERNAL:
-            self.new_client(internal=False)
-            return self.__CLIENT_EXTERNAL
-        return self.__CLIENT_EXTERNAL
+    def client_fake(self) -> minio.Minio:
+        """Get handle to an (already) instantiated FAKE minio.Minio instance for generating signed URLs for external access"""
+        if not self.__CLIENT_FAKE:
+            return self._create_new_client(fake=True)
+        return self.__CLIENT_FAKE
 
     @property
     def base_url(self) -> str:
@@ -356,7 +352,7 @@ def base_url_external(self) -> str:
         """Get external base URL to MinIO"""
         return self.__BASE_URL_EXTERNAL
 
-    def new_client(self, internal: bool = True):
+    def _create_new_client(self, fake: bool = False) -> minio.Minio:
         """
         Instantiates a new Minio client and assigns it to their respective class variable
         """
@@ -369,16 +365,18 @@ def new_client(self, internal: bool = True):
             )
 
         mc = minio.Minio(
-            endpoint=self.__MINIO_ENDPOINT if internal else self.__MINIO_EXTERNAL_ENDPOINT,
+            endpoint=self.__MINIO_EXTERNAL_ENDPOINT if fake else self.__MINIO_ENDPOINT,
             access_key=self.__MINIO_ACCESS_KEY,
             secret_key=self.__MINIO_SECRET_KEY,
-            secure=self.__MINIO_USE_HTTPS if internal else self.__MINIO_EXTERNAL_ENDPOINT_USE_HTTPS,
+            secure=self.__MINIO_EXTERNAL_ENDPOINT_USE_HTTPS if fake else self.__MINIO_USE_HTTPS,
             http_client=self.HTTP_CLIENT,
+            region='us-east-1' if fake else None,
         )
-        if internal:
-            self.__CLIENT = mc
+        if fake:
+            self.__CLIENT_FAKE = mc
         else:
-            self.__CLIENT_EXTERNAL = mc
+            self.__CLIENT = mc
+        return mc
 
     # MAINTENANCE
     def check_bucket_existence(self):

docker-compose.develop.yml

@@ -0,0 +1,108 @@
+# ORIGINAL SOURCE
+# https://docs.min.io/docs/deploy-minio-on-docker-compose.html
+# https://github.com/minio/minio/blob/master/docs/orchestration/docker-compose/docker-compose.yaml?raw=true
+
+# IN THIS CONFIGURATION, THE PROJECT FILES ARE VOLUME MAPPED INTO THE CONTAINER FROM THE HOST
+
+version: "3.9"
+
+# Settings and configurations that are common for all containers
+x-minio-common: &minio-common
+  image: minio/minio:RELEASE.2021-07-30T00-02-00Z
+  command: server --console-address ":9001" http://minio{1...4}/data{1...2}
+  expose:
+    - "9000"
+    - "9001"
+  environment:
+    MINIO_ROOT_USER: minio
+    MINIO_ROOT_PASSWORD: minio123
+  healthcheck:
+    test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
+    interval: 30s
+    timeout: 20s
+    retries: 3
+
+services:
+  # starts Django from DjangoExampleProject + DjangoExampleApplication
+  web:
+    image: python:3
+    command: bash -c "
+      pip install -r /code/requirements.txt
+      && python manage.py migrate
+      && python manage.py runserver 0.0.0.0:8000
+      "
+    volumes:
+      - .:/code
+    working_dir: /code
+    environment:
+      PYTHONUNBUFFERED: "1"
+      GH_MINIO_ENDPOINT: "nginx:9000"
+      GH_MINIO_USE_HTTPS: "false"
+      GH_MINIO_EXTERNAL_ENDPOINT: "localhost:9000"
+      GH_MINIO_EXTERNAL_ENDPOINT_USE_HTTPS: "false"
+      GH_MINIO_ACCESS_KEY: "minio"
+      GH_MINIO_SECRET_KEY: "minio123"
+      # CREATE AN ADMIN ACCOUNT FOR INTERNAL DEMO PURPOSES ONLY!
+      DJANGO_SUPERUSER_USERNAME: "admin"
+      DJANGO_SUPERUSER_PASSWORD: "123123"
+      DJANGO_SUPERUSER_EMAIL: "[email protected]"
+    ports:
+      - "8000:8000"
+    depends_on:
+      - nginx
+  # starts 4 docker containers running minio server instances.
+  # using nginx reverse proxy, load balancing, you can access
+  # it through port 9000.
+  minio1:
+    <<: *minio-common
+    hostname: minio1
+    volumes:
+      - data1-1:/data1
+      - data1-2:/data2
+
+  minio2:
+    <<: *minio-common
+    hostname: minio2
+    volumes:
+      - data2-1:/data1
+      - data2-2:/data2
+
+  minio3:
+    <<: *minio-common
+    hostname: minio3
+    volumes:
+      - data3-1:/data1
+      - data3-2:/data2
+
+  minio4:
+    <<: *minio-common
+    hostname: minio4
+    volumes:
+      - data4-1:/data1
+      - data4-2:/data2
+
+  nginx:
+    image: nginx:1.19.2-alpine
+    hostname: nginx
+    volumes:
+      - ./nginx.conf:/etc/nginx/nginx.conf:ro
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    depends_on:
+      - minio1
+      - minio2
+      - minio3
+      - minio4
+
+## By default this config uses default local driver,
+## For custom volumes replace with volume driver configuration.
+volumes:
+  data1-1:
+  data1-2:
+  data2-1:
+  data2-2:
+  data3-1:
+  data3-2:
+  data4-1:
+  data4-2: