Skip to content

Let's apply the concepts of data modelling to design a pipeline that loads mecessary data into target storag so that it can be used for Analytics and insights

Notifications You must be signed in to change notification settings

chronnix/Simple-Data-Engineering-Project-

Repository files navigation

Data Engineering Project Using Python, PostgreSQL and Airflow

As we all noticed, the pandemic took everyone by surprise and left many families in poverty. An organization has been tasked with analyzing the fiscal position of families pre-covid, and use the results to prredict families who were impacted the most, as well as age groups. We will then use that data to determine the level of help people would recieve from Government relief progams.

Our job for this project is to design a schema for our database, extract data from various sources that we would need, trasnform the data and load it into our database. We are going to utilize the pandas library for data maipulation, psycopg2 library to connect to our database and excute sql quesries, and lastly airflow so that we can schedule jobs and they can run as dependencies.

Now let's try to follow along and do it, feel free to leave comments or questions

About

Let's apply the concepts of data modelling to design a pipeline that loads mecessary data into target storag so that it can be used for Analytics and insights

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages