Musings on the Data Engineering Zoomcamp: Week 2
Post motivated by #learninginpublic and Alexey!
Continuing with the Week-2 of the Data Engineering Zoomcamp organized by DataTalks.club. This week’s topic was workflow orchestration using Prefect (prefect was chosen this year instead of Airflow; Airflow gave a lot of trouble to DE Zoomcamp-2022 participants; my experiences on running airflow discussed here and here).
Topics covered in the Week-2:
Data Lake vs. Data Warehouse
Using Prefect for data flow orchestration
Advanced prefect functionalities relevant to data engineering
GitHub-prefect integration
Prefect with docker
Prefect cloud and automated notifications (of data flow deployments)
Relevant links:
My experiences on GitHub-Prefect integration for data pipeline: link.
Week-1 of DE Zoomcamp 2023: link.
#dezoomcamp #dataengineering #learninginpublic