Dockerized Data Ingestion and Visualization with Python, PostgreSQL, pgAdmin, and Tableau
Source: https://data.mendeley.com/datasets/8gx2fvg2k6/5
Downloading the data
wget https://github.com/trquanganhng/DataCoSupplyChain/releases/download/Download/DataCoSupplyChainDataset.csv
I have split the Dataset into multiple tables based on the file "DescriptionDataCoSupplyChain.csv"(https://github.com/trquanganhng/DataCoSupplyChain/releases/download/Download/DescriptionDataCoSupplyChain.csv).
The dataset will be separated into tables as follows (https://github.com/trquanganhng/DataCoSupplyChain/files/15068615/DataCoSupplyChain.pdf)
After successfully inputting data into the Database. I created a few Dashboards from these data using Tableau.
Summary: https://github.com/trquanganhng/DataCoSupplyChain/files/15068619/Visualization.pdf
git clone https://github.com/trquanganhng/DataCoSupplyChain.git
docker network create pgnetwork
docker run -it \
-e POSTGRES_USER="admin" \
-e POSTGRES_PASSWORD="admin" \
-e POSTGRES_DB="DataCoSupplyChain" \
-v $(pwd)/DataCoSupplyChain_postgres_data:/var/lib/postgresql/data \
-p 5432:5432 \
--network=pgnetwork \
--name pgdatabase \
postgres:14
pgcli -h localhost -p 5432 -u admin -d DataCoSupplyChain
docker run -it \
-e PGADMIN_DEFAULT_EMAIL="[email protected]" \
-e PGADMIN_DEFAULT_PASSWORD="admin" \
-p 8080:80 \
--network=pgnetwork \
--name pgadmin \
dpage/pgadmin4
URL="https://github.com/trquanganhng/DataCoSupplyChain/releases/download/Download/DataCoSupplyChainDataset.csv"
python ingest_data.py \
--user=admin \
--password=admin \
--host=localhost \
--port=5432 \
--db=DataCoSupplyChain \
--url=${URL} \
docker build -t ingest_data:v001 .
URL="https://github.com/trquanganhng/DataCoSupplyChain/releases/download/Download/DataCoSupplyChainDataset.csv"
docker run -it \
--network=pgnetwork \
ingest_data:v001 \
--user=admin \
--password=admin \
--host=pgdatabase \
--port=5432 \
--db=DataCoSupplyChain \
--url=${URL} \