Add QuickSight dashboard, improve propensity signal, update README use case section

- Add scripts/create_dashboard.py: fully automated QuickSight dashboard (4 sheets)
  with Athena/Glue data layer, DIRECT_QUERY datasets, chart subtitles
- Update inference_handler.py: pass through contextual columns (campaign, device,
  category, purchase_amount, impressions, clicks) for dashboard segmentation
- Update generate_synthetic_data.py: embed campaign and category propensity
  multipliers so dashboard charts show meaningful variation across segments
- Update undeploy/undeploy.py: add QuickSight resource cleanup (dashboard,
  analysis, dataset, datasource, IAM inline policy)
- Update config.py: add QS_NOTIFICATION_EMAIL, make hardcoded values authoritative
  over env vars, improve validate() with require_qs_email flag
- Update README: rewrite opening and use case section to lead with business value;
  add Step 6 (QuickSight dashboard) to timing table and project structure;
  update output format table to reflect passthrough columns; add QuickSight
  region prerequisite note

Author: Diego Colombatto

.gitignore

@@ -50,3 +50,9 @@ uv-lockfile-tasks.md
 
 # Kiro
 .kiro/
+
+# Local test root directory
+local_test/
+
+# Working docs
+DASHBOARD_PROPOSAL.md

README.md

@@ -2,7 +2,11 @@
 
 [![License: MIT-0](https://img.shields.io/badge/License-MIT--0-yellow.svg)](https://opensource.org/licenses/MIT-0)
 
-Self-contained, reusable demo for **Customer Propensity Scoring** using AWS Clean Rooms ML with custom training and inference containers.
+Self-contained, reusable, and customizable demo showing how an **advertiser** and a **retailer** can jointly predict which customers are most likely to make a purchase — without either party ever sharing their raw data with the other.
+
+The advertiser contributes **ad engagement data** (impressions, clicks, time spent, device type, campaign) and the retailer contributes **purchase behavior data** (product categories, purchase amounts, site visits, conversion history). AWS Clean Rooms ML joins these datasets inside a secure collaboration, trains a propensity model on the combined signal, and scores every customer — all without exposing either party's underlying records.
+
+The output is a ranked list of customers by purchase propensity, visualized in an Amazon QuickSight dashboard that shows which campaigns, categories, and segments drive the highest conversion intent.
 
 This repo is a sample, to quickly get started with AWS Clean Rooms Custom ML models analysis; it's not meant for production usage AS-IS.
 
@@ -32,11 +36,15 @@ This repo is a sample, to quickly get started with AWS Clean Rooms Custom ML mod
 
 ## Use Case: Customer Propensity Scoring
 
-**Scenario:** An advertiser and a retailer want to collaborate on predicting which customers are most likely to convert (make a purchase) based on combined ad engagement and purchase behavior data. Neither party wants to share their raw data with the other.
+An **advertiser** knows which users engaged with their ads — but not whether those users actually bought anything. A **retailer** knows which users purchased — but not which ads influenced them. Neither party is willing to share their raw customer data with the other.
+
+By combining both datasets inside an AWS Clean Rooms collaboration, the model learns from the full picture: ad engagement signals from the advertiser and purchase behavior signals from the retailer. The result is a propensity score for every customer that neither party could have produced alone.
+
+**What the advertiser gains:** a ranked list of users to prioritise for ad targeting, based on actual purchase signals — not just clicks.
 
-**Solution:** AWS Clean Rooms ML enables both parties to contribute their data to a secure collaboration. AWS Clean Rooms joins the datasets on a shared key (`user_id`), trains a propensity model on the combined features, and runs inference — all without either party seeing the other's raw data.
+**What the retailer gains:** insight into which ad-exposed customers are most likely to buy, enabling smarter inventory planning and personalised offers.
 
-**Business Value:** The advertiser can identify high-propensity users to target with ad campaigns, while the retailer gains insight into which ad-exposed customers are most likely to purchase — enabling better ad spend allocation and personalized marketing.
+**What neither party gives up:** their raw customer data. AWS Clean Rooms enforces that the join happens inside the secure collaboration — no raw records cross the boundary.
 
 ---
 
@@ -51,6 +59,7 @@ This repo is a sample, to quickly get started with AWS Clean Rooms Custom ML mod
 | 3 | Build & Push Containers (CodeBuild) | ~7 min |
 | 4 | Setup Clean Rooms Infrastructure | ~31s |
 | 5 | Train Model & Run Inference | ~34 min |
+| 6 | Create QuickSight Dashboard (optional) | ~3 min |
 | **Total** | **End-to-end** | **~42 min** |
 
 ### Prerequisites
@@ -59,13 +68,16 @@ This repo is a sample, to quickly get started with AWS Clean Rooms Custom ML mod
 - AWS CLI configured with valid credentials
 - AWS account with AWS Clean Rooms ML access enabled
 
+> **Optional — QuickSight Dashboard (Step 6):** If you plan to run `scripts/create_dashboard.py`, your `AWS_REGION` must be a region where Amazon QuickSight is available. QuickSight, Athena, Glue, and S3 must all be in the same region — cross-region Athena connections are not supported by QuickSight. Supported regions include `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, `eu-west-2`, `eu-west-3`, `eu-central-1`, `eu-north-1`, `ap-northeast-1`, `ap-northeast-2`, `ap-southeast-1`, `ap-southeast-2`, `ap-south-1`, `ca-central-1`, and others. See the [full list](https://docs.aws.amazon.com/quicksight/latest/user/regions-qs.html). Also set `QS_NOTIFICATION_EMAIL` in `config.py` to a valid email address — this is used for QuickSight account registration and is validated at script startup.
+
 ### Step 0: Configure Your Account
 
 Edit `config.py` and set your values:
 
 ```python
-AWS_ACCOUNT_ID = "123456789012"   # Your 12-digit AWS account ID
-AWS_REGION     = "eu-north-1"     # Your preferred region
+AWS_ACCOUNT_ID        = "123456789012"        # Your 12-digit AWS account ID
+AWS_REGION            = "eu-north-1"           # Your preferred region
+QS_NOTIFICATION_EMAIL = "your@email.com"       # Optional: only needed for Step 6 (QuickSight)
 ```
 
 All scripts read from this single file — no other hardcoded values to change.
@@ -416,6 +428,14 @@ After successful inference, AWS Clean Rooms ML writes the output to the configur
 |--------|------|-------------|
 | propensity_score | float (0–1) | Predicted probability of conversion |
 | predicted_converter | int (0/1) | Binary prediction: 1 = likely converter |
+| ad_campaign_id | string | Ad campaign the record belongs to |
+| device_type | string | Device type (mobile, desktop, tablet, smart_tv) |
+| product_category | string | Product category browsed/purchased |
+| purchase_amount | float | Total purchase amount |
+| impressions | int | Number of ad impressions |
+| clicks | int | Number of ad clicks |
+
+> **Note:** `user_id` is never present in the output — it is the Clean Rooms join key and is excluded from the ML input channel by design. The passthrough contextual columns (`ad_campaign_id`, `device_type`, etc.) come from the pre-joined data already approved for the inference channel and are used to power the QuickSight dashboard segmentation.
 
 Example output rows:
 
@@ -478,6 +498,7 @@ scripts/
   build_and_push.py               ← Build containers via local Docker
   setup_cleanrooms.py             ← Create Glue, IAM, collaboration, ML config
   run_cleanrooms_ml.py            ← Create channels, train model, run inference
+  create_dashboard.py             ← Optional: create QuickSight dashboard (Step 6)
   test_training_local.py          ← Test training locally (no AWS needed)
   sagemaker_training_job.py       ← Optional: run training via SageMaker directly
   update_requirements.sh          ← Regenerate container requirements.txt from lockfile
@@ -612,12 +633,13 @@ The undeploy script removes all resources in reverse dependency order:
 
 1. **Clean Rooms ML** — inference jobs, trained models, ML input channels, algorithm associations, configured model algorithms
 2. **Clean Rooms** — ML configuration, table association analysis rules, table associations, configured tables, analysis rules, collaboration
-3. **AWS Glue** — tables and database (`cleanrooms_ml_demo`)
+3. **AWS Glue** — tables and database (`cleanrooms_ml_demo`), including dashboard tables (`inference_output`, `model_metrics`, `feature_importance`) if `create_dashboard.py` was run
 4. **Lake Formation** — permission grants for the data provider role
-5. **Amazon S3** — source and output buckets (empties all objects and versions first)
+5. **Amazon S3** — source and output buckets (empties all objects and versions first, including `dashboard-data/` CSVs)
 6. **Amazon ECR** — training and inference container repositories (including all images)
 7. **IAM** — all demo roles (`data-provider`, `model-provider`, `ml-config`, `query-runner`, `codebuild`, `sagemaker`)
 8. **CodeBuild** — build project and associated CloudWatch log groups
+9. **Amazon QuickSight** — dashboard, analysis, SPICE datasets, and Athena data source (if `create_dashboard.py` was run). The QuickSight account subscription itself is **not** deleted as it is account-wide.
 
 > **Note:** IAM roles are global (not region-scoped), so they only need to be deleted once regardless of how many regions were used. The script handles this gracefully — if a role was already deleted by a previous region's undeploy run, it skips it.
 

config.py

@@ -15,10 +15,22 @@
 """
 
 # ─── REQUIRED: Set these to your values ───────────────────
-# Prefer environment variables; fall back to placeholder for local dev.
+# Edit the values below directly. These are the authoritative settings.
+# Environment variables AWS_ACCOUNT_ID / AWS_REGION are only used as
+# fallback when the placeholder values below have not been changed.
 import os as _os_cfg
-AWS_ACCOUNT_ID = _os_cfg.environ.get("AWS_ACCOUNT_ID", "123456789012")
-AWS_REGION     = _os_cfg.environ.get("AWS_REGION", "us-east-1")
+
+_ACCOUNT_DEFAULT = "123456789012"
+_REGION_DEFAULT  = "eu-west-2"
+_EMAIL_DEFAULT   = "your-email@example.com"
+
+AWS_ACCOUNT_ID        = _ACCOUNT_DEFAULT if _ACCOUNT_DEFAULT != "123456789012" else _os_cfg.environ.get("AWS_ACCOUNT_ID", "123456789012")
+AWS_REGION            = _REGION_DEFAULT  if _REGION_DEFAULT  != "us-east-1"    else _os_cfg.environ.get("AWS_REGION",     "us-east-1")
+
+# ─── OPTIONAL: Required only for scripts/create_dashboard.py ──
+# Email address for QuickSight account registration and admin user.
+# Must be a valid address — QuickSight sends subscription notifications to it.
+QS_NOTIFICATION_EMAIL = _EMAIL_DEFAULT   if _EMAIL_DEFAULT   != "your-email@example.com" else _os_cfg.environ.get("QS_NOTIFICATION_EMAIL", "your-email@example.com")
 
 # ─── RUN ID (auto-generated, ensures unique bucket names) ─
 import os as _os
@@ -67,13 +79,16 @@ def _get_or_create_run_id():
 ROLE_QUERY_RUNNER   = f"{PREFIX}-query-runner-role"
 
 
-def validate():
+def validate(require_qs_email=False):
     """Call this at the start of any script to catch misconfiguration early."""
     errors = []
     if AWS_ACCOUNT_ID == "CHANGE_ME" or not AWS_ACCOUNT_ID.isdigit() or len(AWS_ACCOUNT_ID) != 12:
         errors.append(f"AWS_ACCOUNT_ID must be a 12-digit number, got: '{AWS_ACCOUNT_ID}'")
     if AWS_REGION == "CHANGE_ME" or not AWS_REGION:
         errors.append(f"AWS_REGION must be set, got: '{AWS_REGION}'")
+    if require_qs_email and QS_NOTIFICATION_EMAIL == "your-email@example.com":
+        errors.append("QS_NOTIFICATION_EMAIL must be set to a real email address in config.py "
+                      "(required for QuickSight account registration)")
     if errors:
         print("=" * 60)
         print("CONFIGURATION ERROR — edit config.py")

containers/inference/inference_handler.py

@@ -4,6 +4,17 @@
 """
 Inference handler for Customer Propensity Scoring model.
 Compatible with: local, SageMaker Batch Transform, Clean Rooms ML.
+
+Output columns:
+  - propensity_score      float (0-1)
+  - predicted_converter   int (0/1)
+  - ad_campaign_id        str   ─┐
+  - device_type           str    │  passthrough contextual columns
+  - product_category      str    │  (present in Clean Rooms pre-joined input,
+  - purchase_amount       float  │   no raw user identifiers re-introduced)
+  - impressions           int    │
+  - clicks                int   ─┘
+  - user_id               str   (only when present in input, e.g. local/SageMaker mode)
 """
 
 import os, json, logging, io
@@ -122,8 +133,21 @@ def predict(input_data, content_type="text/csv"):
         "propensity_score": np.round(probabilities, 4),
         "predicted_converter": predictions.astype(int),
     })
+
+    # Pass through contextual columns for dashboard segmentation.
+    # These come from the Clean Rooms pre-joined input — no raw user
+    # identifiers are re-introduced. user_id is only present in
+    # local/SageMaker mode (never in the Clean Rooms execution path).
+    PASSTHROUGH_COLS = [
+        "ad_campaign_id", "device_type", "product_category",
+        "purchase_amount", "impressions", "clicks",
+    ]
+    for col in PASSTHROUGH_COLS:
+        if col in df.columns:
+            result[col] = df[col].values
+
     if user_ids is not None:
         result.insert(0, "user_id", user_ids.values)
 
-    logger.info(f"Output shape: {result.shape}")
+    logger.info(f"Output shape: {result.shape}, columns: {list(result.columns)}")
     return result.to_csv(index=False)

data/generate_synthetic_data.py

@@ -34,6 +34,26 @@
 DEVICES = ["mobile", "desktop", "tablet", "smart_tv"]
 CATEGORIES = ["electronics", "clothing", "home_garden", "sports", "beauty", "grocery", "toys"]
 
+# Campaign effectiveness multipliers — drives visible variation in avg propensity by campaign
+CAMPAIGN_PROPENSITY_BOOST = {
+    "camp_holiday":        0.18,
+    "camp_summer_sale":    0.10,
+    "camp_back_to_school": 0.04,
+    "camp_spring":        -0.05,
+    "camp_clearance":     -0.14,
+}
+
+# Category affinity multipliers — drives visible variation in avg propensity by category
+CATEGORY_PROPENSITY_BOOST = {
+    "electronics":  0.16,
+    "sports":       0.08,
+    "home_garden":  0.03,
+    "clothing":    -0.02,
+    "toys":        -0.07,
+    "grocery":     -0.12,
+    "beauty":      -0.18,
+}
+
 BASE_DATE = datetime(2025, 1, 1)
 
 
@@ -53,13 +73,14 @@ def generate_advertiser_data():
         propensity = max(0.05, min(0.95, propensity))
 
         for campaign in random.sample(CAMPAIGNS, num_campaigns):
-            # Weaker propensity signal: more baseline randomness, less propensity-driven
-            impressions = max(1, int(random.randint(5, 40) + 10 * propensity))
+            # Apply campaign-level propensity boost so charts show meaningful variation
+            campaign_propensity = max(0.05, min(0.95, propensity + CAMPAIGN_PROPENSITY_BOOST[campaign]))
+            impressions = max(1, int(random.randint(5, 40) + 10 * campaign_propensity))
             device = random.choice(DEVICES)
             base_ctr = {"mobile": 0.08, "desktop": 0.05, "tablet": 0.06, "smart_tv": 0.03}[device]
-            ctr = base_ctr * (0.5 + 1.0 * propensity) * random.uniform(0.6, 1.5)
+            ctr = base_ctr * (0.5 + 1.0 * campaign_propensity) * random.uniform(0.6, 1.5)
             clicks = max(0, int(impressions * ctr))
-            time_per_click = random.uniform(5, 30) * (0.5 + 1.0 * propensity)
+            time_per_click = random.uniform(5, 30) * (0.5 + 1.0 * campaign_propensity)
             time_spent = round(clicks * time_per_click, 1) if clicks > 0 else 0
             event_date = random_date(BASE_DATE, BASE_DATE + timedelta(days=180))
 
@@ -95,7 +116,9 @@ def generate_retailer_data():
 
         num_categories = random.randint(1, 4)
         for category in random.sample(CATEGORIES, num_categories):
-            site_visits = max(1, int(random.randint(3, 12) + 8 * base_propensity))
+            # Apply category-level propensity boost so charts show meaningful variation
+            cat_propensity = max(0.05, min(0.95, base_propensity + CATEGORY_PROPENSITY_BOOST[category]))
+            site_visits = max(1, int(random.randint(3, 12) + 8 * cat_propensity))
 
             avg_price = {"electronics": 150, "clothing": 45, "home_garden": 65,
                          "sports": 55, "beauty": 30, "grocery": 25, "toys": 35}[category]