A dbt adapter for ksqlDB
Like in many database projects, managing models becomes increasingly challenging. dbt-ksql helps to modularize the code, manage the model dependency, and facilitate the model deployment.
This project is in a proof-of-concept phase.
We are using part of dbt features to manage ksqlDB by
- Separating production/staging/development environments through dbt's profile and target.
- Modularize the SQL with dbt's variable, reference, macro, etc.
Not all dbt features are supported. ksqlDB transforms streaming data. Some of dbt features are assuming batch operation (e.g. incremental and snapshot).
Supported:
- stream
- table
Not-suported:
- ephemeral
- incremental
See example folder for a dbt project. It has examples for creating stream, stream as, table. It also has a example of using macros.
- This project is an experimental phase.
- This has not been published to pypi.
To install, fetch this repository and run a local install:
git clone [email protected]:anelendata/dbt-ksql.git
cd dbt-ksql
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install -e .
After the successful instsall, the following command will show
dbt --version
Something like this:
Core:
- installed: 1.1.2
- latest: 1.2.1 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- ksqldb: 1.1.0 - Could not determine latest version
After set up profiles.yml try running the following commands:
- compile:
dbt compile --profiles-dir . --profile prod - run:
dbt run --profiles-dir . --profile prod
- drop stream:
dbt run-operation drop_stream --args "{relation: some_stream, delete_topic: true}" --profiles-dir . --profile dev
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.
This project is developed by ANELEN and friends. Please check out the ANELEN's open innovation philosophy and other projects
Copyright © 2022~ Anelen Co., LLC