Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Engineer Exam Solution #3

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open

Data Engineer Exam Solution #3

Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
928a95e
Initialize meltano
mkdlt Jan 13, 2022
ccfedeb
Complete data generation task
mkdlt Jan 13, 2022
be6204a
Move original README text
mkdlt Jan 13, 2022
2e701fa
Add tools, ELT notes
mkdlt Jan 13, 2022
670a979
Add links, formatting
mkdlt Jan 13, 2022
1a17874
Add extractor for flat files
mkdlt Jan 13, 2022
2783ce0
Add extractor config
mkdlt Jan 13, 2022
c57bf56
Add loader for postgres
mkdlt Jan 13, 2022
fb9bf55
Fix authentication error
mkdlt Jan 13, 2022
6d34213
Add extraction and loading notes
mkdlt Jan 13, 2022
e936076
Add dbt as transformer
mkdlt Jan 13, 2022
ddfb768
Merge branch 'master' of github.com:mkdlt/dataengineer_test
mkdlt Jan 13, 2022
efc0b41
Implement star schema
mkdlt Jan 13, 2022
51c7686
Add star schema ERD
mkdlt Jan 13, 2022
d95ccad
Add notes on star schema and dbt
mkdlt Jan 13, 2022
5fce20c
Add commands for transform process
mkdlt Jan 13, 2022
4a031e8
Add dbt_date package for date dimension table
mkdlt Jan 13, 2022
9ccf2e2
Fix dbt target schema and config paths
mkdlt Jan 13, 2022
9b7a20a
Add reports
mkdlt Jan 13, 2022
3e0f4ef
Add notes on reporting, deployment, architecture
mkdlt Jan 13, 2022
123b189
Add architecture notes
mkdlt Jan 13, 2022
3c21119
Trim answers
mkdlt Jan 13, 2022
1376ab0
Fix star schema reference
mkdlt Jan 13, 2022
eb5cd3a
Add Airflow as orchestrator
mkdlt Jan 13, 2022
44304cf
Fix Airflow config
mkdlt Jan 13, 2022
9f7caad
Fix typo
mkdlt Jan 13, 2022
95f2230
Fix tools list
mkdlt Jan 13, 2022
724f1b5
Add step by step instructions
mkdlt Jan 13, 2022
cb32bdd
Fix report references
mkdlt Jan 13, 2022
a72f700
Fix instructions
mkdlt Jan 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add extractor for flat files
  • Loading branch information
@mkdlt
mkdlt committed Jan 13, 2022
commit 1a178746278ced92e1c9f11fbf0fe0ca52620480
5 changes: 5 additions & 0 deletions meltano/meltano.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
version: 1
send_anonymous_usage_stats: true
project_id: 1b6e33ad-3fcb-4280-9ac7-7c12ae357a80
plugins:
extractors:
- name: tap-spreadsheets-anywhere
variant: ets
pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git