The transform layer of the data pipeline is constructed using dbt and duckdb.
In order to create a working environment, the docker is used. To start it, please, follow the next steps.
- Launch the docker daemon.
- Get to the repository root folder:
cd transform/ - Build the docker image with a proper tag:
docker build --build-arg S3_ACCESS_KEY_ID=foo --build-arg S3_SECRET_ACCESS_KEY=foo --tag ffc-be-transform:latest -f DockerfileDev . - Run docker container:
docker run -it -p 8080:8080 -v $(pwd):/usr/src/app ffc-be-transform:latest /bin/bash - Run dbt transformation inside the docker container. For example:
dbt run --select +fact_standings --target prod - To view the documentation, generate it using
dbt docs generate && dbt docs serveThen, navigate to http://localhost:8080 to access the documentation.
- dbt docs to learn more about the tool
- dbt discourse for commonly asked questions and answers
- dbt slack for live discussions and support
- dbt blog for the latest news on dbt's development and best practices
- duckdb docs to learn more about the tool