Skip to content

Commit

Permalink
Change to postgres.
Browse files Browse the repository at this point in the history
  • Loading branch information
zzhlogin committed Jan 29, 2024
2 parents ea5e681 + 23c62c8 commit 291bced
Show file tree
Hide file tree
Showing 22 changed files with 64 additions and 412 deletions.
35 changes: 16 additions & 19 deletions sample-applications/vehicle-dealership-sample-app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ This directory contains code for a microservices based sample app that is used t

### EKS
To get started with AWS EKS, you can run the one touch script as below.
`bash script.sh <account_id> <cluster_name> <region> <mysql_password> <s3_bucket_name>`
`bash script.sh <account_id> <cluster_name> <region> <postgres_password> <s3_bucket_name>`

This will create the docker images, upload them to ECR and then create pods on EKS with those images.

Expand Down Expand Up @@ -52,7 +52,7 @@ To deploy to EC2, you will have to go through the following steps.
- AmazonS3FullAccess
- AmazonSQSFullAccess
- Name one vehicle-service and the other image-service
5. Go to RDS and create a MySQL DB with the following configurations:
5. Go to RDS and create a Postgres DB with the following configurations:
- Use the Dev/Test template
- Update the Master username to `root` and create a password of your choosing. Write it down since you will need it later.
- In the Connectivity settings, choose the VPC and security group created in step 3.
Expand All @@ -61,18 +61,15 @@ To deploy to EC2, you will have to go through the following steps.
```
sudo dnf install python3.11
sudo dnf install python3.11-pip
sudo dnf install mariadb105
sudo dnf install -y mariadb105-devel gcc python3.11-devel
sudo dnf install postgresql15
mysql -h <RDS_DB_Endpoint> -P 3306 -u root -p<password_from_step_5>
createdb vehicle_inventory -h <rds_url> -U root
createuser djangouser -h <rds_url> -U root
CREATE DATABASE vehicle_inventory;
psql -h <rds_url> -d vehicle_inventory -U root
CREATE USER 'djangouser'@'%' IDENTIFIED BY '<password_of_your_choosing>';
GRANT ALL PRIVILEGES ON vehicle_inventory.* TO 'djangouser'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
alter user djangouser with encrypted password '<password_of_your_choosing>';
grant all privileges on database vehicle_inventory to djangouser;
aws s3 sync s3://<s3_bucket_that_has_python_code> .
Expand All @@ -81,10 +78,10 @@ cd to the vehicle microservice directory and run:
python3.11 -m pip install -r requirements.txt
Create a .env file with the following:
MYSQL_ROOT_PASSWORD=<password_from_RDS_setup>
MYSQL_DATABASE=vehicle_inventory
MYSQL_USER=djangouser
MYSQL_PASSWORD=<password_from_this_step>
POSTGRES_ROOT_PASSWORD=<password_from_RDS_setup>
POSTGRES_DATABASE=vehicle_inventory
POSTGRES_USER=djangouser
POSTGRES_PASSWORD=<password_from_this_step>
DB_SERVICE_HOST=<RDS_DB_endpoint>
DB_SERVICE_PORT=3306
IMAGE_BACKEND_SERVICE_HOST=<image-service_ec2_public_IP>
Expand Down Expand Up @@ -113,18 +110,18 @@ Now you should be able to access the APIs below through the EC2 addr:port of eac

### Locally with Docker
To get started, make sure you either have you AWS creds in `$HOME/.aws` or the following: `AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN` are exported.
1. Run `bash local_script.sh <mysql_pass> <s3_bucket_name>`.
1. Run `bash local_script.sh <postgres_pass> <s3_bucket_name>`.
This will create docker containers, move the requirement env variables there and start them up.

They should be accessible through `0.0.0.0:8000` for the image service and `0.0.0.0:8001` for the vehicle service.

## APIs

The following are the APIs and what they do:
1. GET /vehicle-inventory/: returns all the vehicles entries for mysql db
2. PUT /vehicle-inventory/: puts vehicle into db. For example: `curl -X POST http://0.0.0.0:8001/vehicle-inventory/ -d '{"make": "BMW","model": "M340","year": 2022,"image_name": "newCar.jpg"}'`
1. GET /vehicle-inventory/: returns all the vehicles entries for postgres db
2. POST /vehicle-inventory/: puts vehicle into db. For example: `curl -X POST http://0.0.0.0:8001/vehicle-inventory/ -d '{"make": "BMW","model": "M340","year": 2022,"image_name": "newCar.jpg"}'`
3. GET /vehicle-inventory/<int>: returns vehicle entry with id = <int>
4. GET /vehicle-inventory/<int>/image: returns image file information from S3 for the specific vehicle by calling the image microservice
5. GET /images/name/<image_name>: returns image information for <image_name> from S3 if present.
6. PUT /images/name/<image_name>: creates an empty file in S3. This is an async endpoint since it will put image name in an SQS queue and not wait for the file to be created in S3. Instead, a long running thread will poll SQS and then create the image file later.
6. POST /images/name/<image_name>: creates an empty file in S3. This is an async endpoint since it will put image name in an SQS queue and not wait for the file to be created in S3. Instead, a long running thread will poll SQS and then create the image file later.
7. GET /image/remote-image: makes a remote http call to google.com.
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@
# SPDX-License-Identifier: Apache-2.0
import json
import os
import time

import requests
from django.http import HttpResponse, HttpResponseBadRequest, HttpResponseNotAllowed, HttpResponseNotFound
from django.http import HttpResponse, HttpResponseBadRequest, HttpResponseNotAllowed
from django.views.decorators.csrf import csrf_exempt
from dotenv import load_dotenv
from MainService.models import Vehicle
Expand Down Expand Up @@ -41,23 +40,14 @@ def vehicle(request):

def get_vehicle_by_id(request, vehicle_id):
if request.method == "GET":
throttle_time = request.GET.get("throttle")
if throttle_time:
print("going to throttle for " + throttle_time + " seconds")
time.sleep(int(throttle_time))

vehicle_objects = Vehicle.objects.filter(id=vehicle_id).values()
if not vehicle_objects:
return HttpResponseNotFound("Vehicle with id=" + str(vehicle_id) + " is not found")
return HttpResponse(vehicle_objects)
return HttpResponseNotAllowed("Only GET requests are allowed!")


def get_vehicle_image(request, vehicle_id):
if request.method == "GET":
vehicle_object = Vehicle.objects.filter(id=vehicle_id).first()
if not vehicle_object:
return HttpResponseNotFound("Vehicle with id=" + str(vehicle_id) + " is not found")
image_name = getattr(vehicle_object, "image_name")
return HttpResponse(requests.get(get_image_endpoint() + "/images/name/" + image_name, timeout=10))
return HttpResponseNotAllowed("Only GET requests are allowed!")
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,10 @@

DATABASES = {
"default": {
"ENGINE": "django.db.backends.mysql",
"USER": os.environ.get("MYSQL_USER"),
"NAME": os.environ.get("MYSQL_DATABASE"),
"PASSWORD": os.environ.get("MYSQL_PASSWORD"),
"ENGINE": "django.db.backends.postgresql",
"USER": os.environ.get("POSTGRES_USER"),
"NAME": os.environ.get("POSTGRES_DATABASE"),
"PASSWORD": os.environ.get("POSTGRES_PASSWORD"),
"HOST": os.environ.get("DB_SERVICE_HOST"),
"PORT": os.environ.get("DB_SERVICE_PORT"),
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Django~=5.0
requests~=2.31.0
mysqlclient~=2.2.1
psycopg2~=2.9.9
python-dotenv~=1.0.0
opentelemetry-distro==0.43b0
# We don't use the distro[otlp] option which automatically includes exporters since gRPC is not appropriate for
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,24 @@
version: '3'
services:
db:
image: mysql:8.0
image: postgres:14.0
container_name: vehicle_inventory_db
restart: always
volumes:
- data:/var/lib/mysql
- data:/var/lib/postgres
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
POSTGRES_DB: ${POSTGRES_DATABASE}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
PGUSER: ${POSTGRES_USER}
ports:
- "3306:3306"
- "5432:5432"
healthcheck:
test: ["CMD", "mysql", "-h", "localhost", "-u", "root", "-p${MYSQL_PASSWORD}", "-e", "SELECT 1"]
test: ["CMD", "pg_isready", "-d", "vehicle_inventory", "-U", "djangouser"]
timeout: 20s
retries: 10
expose:
- 3306
- 5432

vehicle-inventory-backend:
image: pythonsampleapp/vehicle-inventory-service
Expand Down Expand Up @@ -54,7 +54,7 @@ services:
context: .
dockerfile: ImageServiceApp/Dockerfile
container_name: image-service-backend
command: sh -c "python3 manage.py migrate --noinput && python3 manage.py collectstatic --noinput && opentelemetry-instrument python3 manage.py runserver 0.0.0.0:8000 --noreload"
command: sh -c "opentelemetry-instrument python3 manage.py runserver 0.0.0.0:8000 --noreload"
restart: always
volumes:
- ./ImageServiceApp/.:/image-service-app
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,11 @@ spec:
image: ${REPOSITORY_PREFIX}/pythonsampleapp/vehicle-inventory-service:latest
name: vehicle-inventory-backend
env:
- name: MYSQL_DATABASE
- name: POSTGRES_DATABASE
value: vehicle_inventory
- name: MYSQL_PASSWORD
value: ${MYSQL_PASSWORD}
- name: MYSQL_ROOT_PASSWORD
value: ${MYSQL_PASSWORD}
- name: MYSQL_USER
- name: POSTGRES_PASSWORD
value: ${POSTGRES_PASSWORD}
- name: POSTGRES_USER
value: djangouser
- name: OTEL_SERVICE_NAME
value: "VehicleInventoryApp"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,36 +29,31 @@ spec:
spec:
containers:
- env:
- name: MYSQL_DATABASE
- name: POSTGRES_DB
value: vehicle_inventory
- name: MYSQL_PASSWORD
value: ${MYSQL_PASSWORD}
- name: MYSQL_ROOT_PASSWORD
value: ${MYSQL_PASSWORD}
- name: MYSQL_USER
- name: POSTGRES_PASSWORD
value: ${POSTGRES_PASSWORD}
- name: POSTGRES_USER
value: djangouser
image: mysql:8.0
image: postgres:14.0
livenessProbe:
exec:
command:
- mysql
- -h
- localhost
- -u
- root
- -p${MYSQL_PASSWORD}
- -e
- SELECT 1
- pg_isready
- -d
- vehicle_inventory
- -U
- djangouser
failureThreshold: 10
timeoutSeconds: 20
name: vehicle-inventory-db
ports:
- containerPort: 3306
hostPort: 3306
- containerPort: 5432
hostPort: 5432
protocol: TCP
resources: {}
volumeMounts:
- mountPath: /var/lib/mysql
- mountPath: /var/lib/postgres
name: data
restartPolicy: Always
volumes:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ metadata:
name: db
spec:
ports:
- port: 3306
- port: 5432
selector:
io.kompose.service: db
status:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ spec:
- args:
- sh
- -c
- python3 manage.py migrate --noinput && python3 manage.py collectstatic --noinput && opentelemetry-instrument python3 manage.py runserver 0.0.0.0:8000 --noreload
- opentelemetry-instrument python3 manage.py runserver 0.0.0.0:8000 --noreload
image: ${REPOSITORY_PREFIX}/pythonsampleapp/image-service:latest
name: image-service-backend
imagePullPolicy: Always
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,11 @@ rm VehicleInventoryApp/.env
rm ImageServiceApp/.env
rm .env

echo "MYSQL_ROOT_PASSWORD=${password}" >> VehicleInventoryApp/.env
echo "MYSQL_DATABASE=vehicle_inventory" >> VehicleInventoryApp/.env
echo "MYSQL_USER=djangouser" >> VehicleInventoryApp/.env
echo "MYSQL_PASSWORD=${password}" >> VehicleInventoryApp/.env
echo "POSTGRES_DATABASE=vehicle_inventory" >> VehicleInventoryApp/.env
echo "POSTGRES_USER=djangouser" >> VehicleInventoryApp/.env
echo "POSTGRES_PASSWORD=${password}" >> VehicleInventoryApp/.env
echo "DB_SERVICE_HOST=db" >> VehicleInventoryApp/.env
echo "DB_SERVICE_PORT=3306" >> VehicleInventoryApp/.env
echo "DB_SERVICE_PORT=5432" >> VehicleInventoryApp/.env
echo "IMAGE_BACKEND_SERVICE_HOST=image-service-backend" >> VehicleInventoryApp/.env
echo "IMAGE_BACKEND_SERVICE_PORT=8000" >> VehicleInventoryApp/.env

Expand All @@ -37,9 +36,8 @@ echo "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" >> ImageServiceApp/.env
echo "AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}" >> ImageServiceApp/.env
echo "S3_BUCKET=${s3_bucket}" >> ImageServiceApp/.env

echo "MYSQL_ROOT_PASSWORD=${password}" >> .env
echo "MYSQL_DATABASE=vehicle_inventory" >> .env
echo "MYSQL_USER=djangouser" >> .env
echo "MYSQL_PASSWORD=${password}" >> .env
echo "POSTGRES_DATABASE=vehicle_inventory" >> .env
echo "POSTGRES_USER=djangouser" >> .env
echo "POSTGRES_PASSWORD=${password}" >> .env

docker-compose up --build

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Loading

0 comments on commit 291bced

Please sign in to comment.