Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Data structures and migrations library

Workflow Engine for Kubernetes

Tasks and projects solved while passing Machine Learning Engineer course by karpov.courses and AI Talent Hub (ITMO University)

Python (scrapy, asyncIO), Apache Airflow, GCP (Storage Bucket, Functions, BigQuery, VM), dbt, Terraform, Docker

Docker compose yml of diffrent tools setup locally

AWS Summit 2022 ASEAN --- COM203 Using IaC with Terraform to provision Big Data Platform on Amazon EMR

Production Grade Terraform for Provisioning Infrastructure

Desenvolvimento de uma pipeline robusta de dados para ingestão e transformação de grandes volumes de dados de internações hospitalares (AIH - Autorização de Internação Hospitalar) do DATASUS.

Airflow DAGs for the Stellar ETL project

Open drug data pipelines curated by pharmacists.

Airflow configuration for Telemetry

Data Engeenering Project - Data Pipeline

Example DAGs

An MLOps pipeline for stock price forecasting with Airflow, DVC, and Google Cloud, featuring advanced financial models and continuous data integration.

This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.

✨ Automate fashion trend analysis with Apache Airflow! Extract data from X & Pinterest, transform into insights, and load into PostgreSQL. Predict seasonal styles & visualize trends. 💃📊

Best in class Open Source tools, in production, with the press of a button.