UIDAI Enrolment Risk Engine

Team ID: UIDAI_7554
Author: S Aadarsh (@adarshvision1)
Kaggle Notebook: UIDAI-7554

---

🎯 Executive Summary

The UIDAI Enrolment Risk Engine is a compact, reproducible machine learning pipeline designed to identify district-months where Aadhaar enrolments deviate materially from learned norms. The system combines statistical baselines, temporal concentration metrics, demographic segmentation, and behavioral anomaly detection to create actionable risk assessments for policy intervention.

Key Achievements

• Processed: 1,006,007 raw enrolment records (Mar–Dec 2025)
• Generated: 4,848 monthly feature rows across districts
• Scored: 3,847 district-month combinations
• Flagged: 77 high-risk cases requiring attention
• Forecast: ~36,049 daily enrolments (national average)

Policy Outputs

• ✅ Decision Map — Interactive visualization prioritizing audit-worthy districts
• ✅ Top-10 Flagged Cases — C3 priority list with detailed metadata
• ✅ District Segmentation — Demographic clusters for customized verification
• ✅ 30-Day Forecast — Proactive resource planning for workload management

---

📋 Table of Contents

• Problem Statement
• Approach
• Technical Architecture
• Dataset
• Methods
• Results
• Operational Playbook
• Installation & Usage
• Artifacts
• Limitations
• Future Work
• License

---

🎯 Problem Statement

Aadhaar enrolment volumes are influenced by social, administrative, and operational events. UIDAI requires a reliable, interpretable system that distinguishes:

• ✅ Routine demographic demand
• ⚠️ Concentrated surges requiring verification
• 🚨 Anomalous patterns meriting policy attention

Challenge: Identify high-risk enrolment patterns without overwhelming audit resources or creating false positives.

---

🔬 Approach

The system employs a multi-layered anomaly detection strategy:

1. Dynamic Baselining (Context)

   • LightGBM quantile regressors (q50, q90) predict district-specific demand
   • q90 represents upper bounds of historically plausible volumes

2. Temporal Burstiness (Behavior)

   • Gini coefficient measures intra-month concentration
   • Identifies whether volume is distributed or concentrated into few days

3. Operational Segmentation (Demographics)

   • K-Means clustering on child/adult ratios
   • Creates operational archetypes: Newborn Hubs, Camp/Sporadic, Mixed

4. Composite Risk Scoring

   • IsolationForest behavioral scores
   • Weighted anomaly score combining excess, burst, and peer deviation
   • Policy mapping: A1 (Monitor) → B2 (Analyst Review) → C3 (Field Verification)

---

🏗️ Technical Architecture

Pipeline Stages

graph LR
    A[Raw Data] --> B[ETL & Canonicalization]
    B --> C[District-Month Aggregation]
    C --> D[Feature Engineering]
    D --> E[Quantile Baseline Models]
    E --> F[Temporal & Demographic Features]
    F --> G[Risk Scoring Engine]
    G --> H[Policy Classification]
    H --> I[Interactive Dashboards]

Loading

Technology Stack

• Machine Learning: LightGBM (quantile regression), IsolationForest, Exponential Smoothing
• Data Processing: Pandas, NumPy
• Visualization: Plotly (interactive HTML dashboards)
• Storage: Parquet (efficient columnar format)

Key Design Rationale

Choice	Rationale
District-month units	Aligns with administrative action, reduces daily noise
Quantile baselines	Avoids mean-driven false positives in heavy-tailed data
Gini coefficient	Single interpretable signal for temporal concentration
Composite scoring	Balances statistical deviation and behavioral irregularity

---

📊 Dataset

Input Specifications

• Source: Anonymized Aadhaar enrolment CSV shards
• Required Columns:
  • date — Enrolment date
  • state, district, pincode — Geographic identifiers
  • age_0_5, age_5_17, age_18_greater — Demographic buckets

Scale

• Raw Records: 1,006,007
• Time Period: March–December 2025
• Districts Covered: Multi-state coverage with 4,848 district-month combinations

---

🔧 Methods

1. ETL Pipeline

• Text canonicalization (lowercase, strip whitespace)
• Date parsing and validation
• Compute total_enrolments from age buckets or fallback fields
• Aggregate to (state, district, month) level

2. Feature Engineering

Lag Features: 1, 2, 3, 6-month historical volumes
Rolling Statistics: 3-month mean and standard deviation
Event Intensity: Month-based proxies (admissions, fiscal windows)
Peer Deviation: Comparison to state/regional averages

3. Modeling

Baseline Models (LightGBM Quantile Regression)

• q50 (Median): Expected typical demand
• q90 (Upper Baseline): Extreme but historically plausible demand
• Features: Lags, rolling stats, demographic ratios, event intensity, peer deviation

Behavioral Anomaly (IsolationForest)

• Detects unusual patterns in feature space
• Identifies outliers not captured by quantile models

Forecast (Exponential Smoothing)

• National-level 30-day daily enrolment forecast
• Resource planning and workload anticipation

4. Risk Scoring

risk_index = MinMaxScale(
    weighted_sum(
        excess_score,
        burstiness_score,
        peer_deviation_score,
        isolation_forest_score
    )
)

Policy Mapping:

• C3 (Field Verification): risk_index >= threshold_high AND no exoneration
• B2 (Analyst Review): threshold_med <= risk_index < threshold_high
• A1 (Monitor): risk_index < threshold_med

Exoneration Rule:
If event_intensity >= 0.8 AND demographic ratios match expected patterns → Classified as authorized surge

---

📈 Results

Top 10 Districts by Monthly Volume

State	District	Month	Total Enrolments	Age 0-5	Age 5-17	Age 18+
West Bengal	Murshidabad	2025-09	13,877	12,270	1,592	15
Karnataka	Bengaluru Urban	2025-07	12,219	6,740	3,752	1,727
West Bengal	North 24 Parganas	2025-09	10,642	8,255	2,149	238
Uttar Pradesh	Bahraich	2025-07	10,547	3,919	6,096	532
Karnataka	Bengaluru Urban	2025-06	10,123	5,062	3,223	1,838

Top 10 Flagged District-Months (C3 Priority)

State	District	Month	Total	Expected q90	Excess	Burst Gini	Risk Index	Policy
Gujarat	Banas Kantha	2025-07	9,228	6,907	2,321	0.96	1.00	C3
West Bengal	Murshidabad	2025-09	13,877	7,409	6,468	0.34	1.00	C3
Bihar	Pashchim Champaran	2025-07	9,642	7,342	2,300	0.95	0.98	C3
Gujarat	Dohad	2025-07	8,001	6,310	1,691	0.90	0.95	C3
Meghalaya	East Khasi Hills	2025-04	7,798	5,057	2,741	0.82	0.93	C3

Note: High Burst Gini (>0.7) indicates concentration into few days

System Performance Summary

{
  "rows_processed_raw": 1006007,
  "monthly_feature_rows": 4848,
  "scored_rows": 3847,
  "high_risk_count_ge_t_med": 77,
  "forecast_mean_per_day": 36049,
  "anomalies_detected": 0,
  "run_timestamp_utc": "2026-01-23T16:47:38.968599"
}

Visual Analytics

The system generates three interactive HTML dashboards:

1. 🗺️ Decision Map (decision_map.html)

   • X-axis: Burstiness (Gini coefficient)
   • Y-axis: Excess over q90 baseline
   • Upper-right quadrant = High priority audit targets

2. 👶 District Segmentation (district_segmentation.html)

   • Child vs Adult enrolment mix
   • Clusters: Newborn Hubs, Camp/Sporadic, Mixed demographics

3. 📅 Workload Forecast (workload_forecast.html)

   • Historical daily series + 30-day forecast
   • Spike detection and trend analysis

---

📋 Operational Playbook

Automated Triage

For Every C3 Case, Auto-Generate:

• Daily time series visualization
• Top-3 PIN codes (geographic concentration)
• Top operator IDs (account review)
• Event intensity score
• Excess and burstiness metrics

Auto-Call Checklist:

if burst_gini > 0.7 and top_3_day_share > 0.6:
    flag_for_immediate_operator_audit()

Exoneration Check:

if event_intensity >= 0.8 and demographic_mix_matches_expected:
    classify_as("Administrative surge - Request documentation")
else:
    proceed_to_field_audit()

Field & Analyst Actions

Policy Class	Action Required	Responsibility
C3	Field verification, document check, operator review, temporary suspension if breach found	District Officers
B2	Request event permits and rosters, monitor recurrence	State Analysts
A1	Monitor only, include in resource planning	Regional Coordinators

6-8 Week Pilot Plan

1. Week 1-2: Run programmatic triage checks (PIN, operator, temporal concentration)
2. Week 3-4: Contact district offices, request permits/rosters
3. Week 5-6: Schedule and conduct field audits for cases without valid documentation
4. Week 7-8: Log audit outcomes, compute Precision@10, retrain models

---

💻 Installation & Usage

Prerequisites

# Python 3.8+
pip install pandas numpy lightgbm scikit-learn plotly statsmodels joblib

Quick Start

# Clone the repository
git clone https://github.com/adarshvision1/UIDAI_7554.git
cd UIDAI_7554

# Open Jupyter Notebook
jupyter notebook uidai-7554.ipynb

Running the Pipeline

The notebook executes the following stages:

1. Data Loading — Read anonymized CSV shards
2. ETL — Clean and aggregate to district-month level
3. Feature Engineering — Generate lags, rolling stats, event proxies
4. Model Training — Fit quantile regressors and anomaly detectors
5. Scoring — Compute risk indices and policy classifications
6. Visualization — Generate interactive HTML dashboards
7. Export — Save artifacts to results/uidai_artifacts/

---

📦 Artifacts

All outputs are saved to results/uidai_artifacts/:

Data Artifacts

• feature_store_enrolment.parquet — Engineered features (4,848 rows)
• engineered_with_predictions.parquet — Features + baseline predictions
• scored_cases.parquet — Full risk scoring results
• top_cases.json — Top-N flagged district-months
• daily_forecast_30d.csv — National workload forecast

Model Artifacts

• model_q50.txt / .joblib — Median quantile regressor
• model_q90.txt / .joblib — Upper baseline regressor
• model_training_meta.json — Training configuration and metrics

Visual Artifacts

• decision_map.html — Interactive burstiness vs excess plot
• district_segmentation.html — Demographic clustering visualization
• workload_forecast.html — Time series forecast with anomalies

Metadata

• summary.json — Pipeline run statistics
• feature_store_preview_top10.csv — Sample feature data

🔗 Browse all artifacts

---

⚠️ Limitations

1. Data Anonymization

   • Flags indicate risk, not proof of wrongdoing
   • All investigations must follow due process

2. Missing External Data

   • No official camp calendars or event schedules
   • Missing population denominators for rate calculations
   • Recommendation: Integrate government event calendars (schools, health, welfare schemes)

3. Detector Inconsistency

   • Daily anomaly detector reported 0 anomalies
   • Monthly quantile detector flagged many cases
   • Action: Do not rely on single detector; use combined resolution checks

4. Temporal Scope

   • Limited to March–December 2025
   • Requires continuous retraining with new data

5. Causal Attribution

   • Statistical association ≠ causation
   • Field verification required for definitive conclusions

---

🚀 Future Work

Immediate Enhancements

• Integrate official event calendars (admissions, immunization, PDS deadlines)
• Add operator registry for cross-referencing
• Implement real-time alerting system
• Develop mobile audit app for field officers

Medium-Term Goals

• Deploy to state UIDAI dashboards with role-based access
• Build feedback loop: audit outcomes → label store → model retraining
• Add explainability module (SHAP values for individual cases)
• Extend to sub-district (tehsil/block) level granularity

Long-Term Vision

• Multi-modal anomaly detection (biometric quality, document types)
• Predictive resource allocation optimization
• Natural language query interface for policy analysts
• Integration with national identity management systems

---

🏆 Societal Impact

Pattern A: Workload Concentration

Evidence: Few districts (Murshidabad, Bengaluru Urban, etc.) account for national-scale volumes
Policy Use: Pre-authorize camps, negotiate scheduling with state offices

Pattern B: Event Signatures

Evidence: High burstiness (Gini >0.9) in Banas Kantha, Pashchim Champaran
Policy Use: Require advance notification, implement camp authorization protocols

Pattern C: Sustained vs Spike

Evidence: Murshidabad shows high excess but low Gini (sustained demand)
Policy Use: Differentiate scaling (multi-week campaigns) from audit (single-day batches)

Pattern D: Recurring Hotspots

Evidence: Districts flagged multiple months
Policy Use: Watchlist for root-cause analysis (operator behavior, local incentives)

Pattern E: Demographic Drivers

Evidence: High child-ratio in flagged months (12,270 of 13,877 in Murshidabad)
Policy Use: Coordinate with health/education departments, use pediatric audit checklists

---

📄 License

This project is developed for UIDAI's operational use. Data is anonymized and handled in compliance with privacy regulations. For access or collaboration inquiries, please contact the development team.

---

🤝 Contributing

This repository represents a submission to UIDAI's enrolment analytics initiative. For questions or suggestions:

• Author: S Aadarsh
• GitHub: @adarshvision1
• Kaggle: aadarshvision1
• Project: Kaggle Notebook

---

📊 Citation

If you use this work in research or policy documentation:

@misc{uidai_risk_engine_2026,
  author = {S. Aadarsh},
  title = {UIDAI Enrolment Risk Engine: Anomaly Detection for Identity Management},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/adarshvision1/UIDAI_7554}},
  note = {Team ID: UIDAI\_7554}
}

---

Built with 🧠 Machine Learning for 🇮🇳 Digital India

📊 View Notebook • 🔍 Explore Data • 📧 Contact