This project demonstrates a real-time ETL pipeline built using Apache Airflow (Astro CLI) and PostgreSQL.
We fetch live weather data (based on latitude & longitude) from the Open Meteo API, process it through an Airflow DAG, and load the results into a PostgreSQL database.
Later, this data can be used for:
- 📊 Dashboarding & Visualization (e.g., Power BI, Tableau, Grafana)
- 🌤 Weather Forecasting models with ML/AI
- 📡 Building scalable data pipelines with Airflow
Step 1: Extract → Weather API (latitude & longitude based)
Step 2: Transform → Processed inside Airflow task
Step 3: Load → Stored into PostgreSQL table
The DAG is fully orchestrated in Apache Airflow, ensuring scheduled execution and monitoring.
The processed weather data is finally stored into PostgreSQL, making it ready for analytics.
git clone https://github.com/<your-username>/etl-weather-astro.git
cd etl-weather-astro