Calendly Marketing Insights

Cross-platform Data Engineering & Analytics Pipeline (End-to-End)

Overview

Calendly Marketing Insights is an end-to-end data engineering project designed to ingest, transform, model, and analyze marketing-related data using modern data engineering practices. The build followed a structured implementation workflow, progressing from initial design, enabling needed infrastructure (setting up storage and compute credential to connect with AWS) to a fully operational analytics pipeline.

This repository documents the architecture, development workflow, and outputs across ingestion, transformation, orchestration, and analytics.

---

Project Goals

• Build a scalable data engineering pipeline for marketing analytics.
• Centralize Calendly marketing data (ads, web analytics, performance metrics).
• Apply robust data modeling (Bronze, Silver, Gold).
• Automate workflows using Databricks and supporting tools.
• Deliver dashboards and insights for marketing decision-making.

---

Architecture

1. Data Ingestion

• Ingest raw data from multiple marketing sources (e.g., Calendly event streams captured via webhook, campaign metrics, web analytics).
• Implement ingestion through Databricks notebooks and scheduled jobs.
• Store raw unvalidated data in Bronze tables.

2. Data Transformation

• Clean, standardize, and enrich data following the medallion architecture.
• Apply business rules, data quality checks, and incremental updates in Silver tables.
• Build analytical, denormalized Gold tables for reporting.

3. Data Modeling

• Dimensional model with Fact & Dimension tables.

• Star schema applied to marketing events, sessions, conversions, and campaign metadata.
• Optimized for BI tools such as Looker Studio/Tableau or custom visualization app deployed through Streamlit.

4. Analytics Layer

• Create dashboards and aggregated views for:

  • Campaign performance
  • Conversion paths
  • Traffic sources
  • User engagement patterns
  • Time-series trends

5. Orchestration

• Use Databricks Jobs and Workflows for scheduled runs.
• Implement dependency ordering (Bronze → Silver → Gold).

• Logging, monitoring, and failure alerts.

---

Development Summary

This project was executed systematically across different phases. Below is a consolidated summary of the implementation flow:

Phase 1: Planning & Foundations

• Define project scope, data sources, and KPIs.
• Design the architecture and medallion layers.

• Set up Databricks workspace, repos, clusters, and notebooks that can access the Amazon s3 buckets for executing jobs and also validating data written to different layers.
• Establish naming conventions and folder structure.

Phase 2: Ingestion Layer (Bronze)

• Create ingestion lambda for each data source: Calendly Events, Spend.

Calendly Event Lambda

A 1-day snapshot of hourly lambda invocations to write the events data into the bronze layer:

Spend Lambda Monitoring

Failure of the lambda invocation observed via error rate triggers retries and alerts. The errors were cause due to the upstream data unavailability - 2 retries were setup in total.

• Implement schema inference and raw storage.
• Register Bronze tables in the Hive Metastore.
• Introduce initial automated workflows for ingestion using the webhooks:API Gateway setup for the events data and designated S3:AWS lambda setup for the spend data collection.

Phase 3: Transformation Layer (Silver)

• Write ETL logic for cleaning and normalizing raw data.
• Apply data quality checks (null handling, type casting, deduping).
• Build intermediate conformance tables.
• Validate transformations through unit checks.

Phase 4: Modeling & Gold Layer

• Implement Fact tables (MarketingEvents, Conversions, Sessions).
• Implement Dimension tables (Campaign, Channel, UTM, Date).
• Optimize using Z-Order clustering, CDF, and Delta Lake capabilities.
• Build aggregated Gold views for BI use cases.

Phase 5: Analytics & Dashboards

• Connect Gold tables to BI tools.
• Create marketing performance dashboards.
• Build funnels, cohorts, and time-series views.
• Validate metrics with sample marketing datasets.
• Deployment of (viz/) the streamlit app required a SQL warehouse cluster Vs a Compute cluster used to run the jobs. SQL warehouse cluster is quick to start relative to the Compute cluster withour Photon acceleration and that comes at a higher DBU/h.

Phase 6: Automation & Finalization

• Add scheduled workflows for full pipeline automation.
• Enforce dependency patterns across jobs.
• Document runbooks, table dictionaries, and data flow diagrams.
• Prepare README, architecture diagrams, and demo notebooks.

---

Repository Structure

├── .github/
│   ├── workflows/
│
├── delta_lake/
│   ├── bronze/
│   ├── silver/
│   └── gold/
│
├── notebooks/      # Test, Change Data Capture, Modeling and DDL scripts
│   ├── test_transform
│   ├── analyze_cdc
│
├── viz/
│  
├── docs/
│   ├── architecture-diagram.png
│   ├── data-dictionary.md
│   └── dimensional-model-schema.png
│
├── test/
│   
└── README.md

---

Key Outputs

• Automated ingestion workflows
• Cleaned, modeled marketing datasets
• Fact and dimension schema
• Delta Lake medallion-layer implementation
• Production-ready Gold tables
• Marketing dashboards for insights
• Fully documented ETL job

---

Tech Stack

• Databricks (Delta Lake, Workflows, Repos, Notebooks)
• Python (PySpark)
• SQL
• Delta Tables
• GitHub for version control
• Streamlit for BI

---

How to Run

1. Clone the repository into Databricks Repos.
2. Attach notebooks, jobs to a cluster (recommended: Photon runtime if not on Free version of the Databricks).
3. Run Bronze Python Script to ingest raw data.
4. Run Silver Python Script to apply transformations.
5. Run Gold Python Script to generate analytical models.
6. Connect BI tools to the Gold layer for visualization.
7. Configure Databricks Workflows for automation.

---

Future Enhancements

• Add streaming ingestion for real-time analytics.
• Implement data quality framework (e.g., Deequ, Great Expectations).
• Add CI/CD workflows for Databricks deployments.
• Expand dashboard coverage to more marketing channels.

---