Merge pull request #1 from markndg/radar-plot-visualiser

Added behavioural fingerprint and compatibility scoring

Author: markndg

README.md

@@ -14,9 +14,11 @@ until a customer complained.
 
 Arsenic distinguishes **blocking behavioural regressions** (factual errors, schema breaks, instruction failures) from **presentation drift** (formatting, verbosity, markdown structure) and **telemetry drift** (latency, consistency). A model upgrade can look scary on raw dimension counts while still being safe to ship — the report tells you which is which.
 
+Arsenic also generates a **behavioural fingerprint** for every comparison: an evidence-backed visual map of where the candidate retained baseline behaviour and where it changed. This is a compatibility fingerprint, not a global quality score or model personality profile.
+
 ---
 
-<![Arsenic drift report — gpt-4o-mini vs gpt-4.1-mini showing 3 critical regressions](docs/report-screenshot.png)
+![Arsenic behavioural fingerprint — baseline retention radar from a real gpt-4o-mini → gpt-4.1-mini report](docs/fingerprint-screenshot.png)
 
 ---
 
@@ -47,6 +49,52 @@ Open the prebuilt reports in your browser — no install required:
 - [GPT-4.1-mini → GPT-5.4-mini (standard suite)](https://markndg.github.io/arsenic/examples/gpt-4_1-mini_vs_gpt-5_4-mini.html)
 - [GPT-4o-mini → GPT-4.1-mini](https://markndg.github.io/arsenic/examples/gpt-4o-mini_vs_gpt-4_1-mini.html)
 - [Llama 3.1:8b → Llama 3.2:3b](https://markndg.github.io/arsenic/examples/llama3_1-8b_vs_llama3_2-3b.html)
+- [Behavioural fingerprint demo](docs/fingerprint-demo.html) (re-rendered from the shipped fixture)
+
+---
+
+## Behavioural fingerprint
+
+Every HTML/JSON report includes a **baseline-retention** fingerprint when enough dimensions were evaluated.
+
+| Score | Meaning |
+|------:|---------|
+| **100** | No detected drift from the baseline in that dimension |
+| **0** | Every applicable probe showed drift |
+| Between | Share of applicable probes that stayed baseline-equivalent |
+
+**Important:** a smaller polygon is not “a worse model”. Improvements count as drift (compatibility changed) but are labelled as improvements in the detail panel and table. Latency stays off the radar as telemetry; it appears as an adjacent summary card.
+
+**Axes** (included only when Arsenic has applicable observations): Morphology, Tone, Factual, Schema, Instruction, Refusal, Consistency retention, Claim retention, Semantic retention. Omitted axes are listed outside the polygon with an explicit reason (no applicable probes, disabled, or unavailable).
+
+**Consistency retention** uses multi-run mean pairwise embedding distance when `--consistency-runs` > 1. That “variance” is `mean(1 − cosine)` over L2-normalised non-negative hash-bag embeddings, so it is intrinsically in `[0, 1]`; clamping is a defensive float guard, not a rescale of unbounded statistical variance:
+
+```
+repeatability% = 100 × (1 − clamp(variance, 0, 1))
+retention      = 100 − |candidate_repeatability − baseline_repeatability|
+```
+
+The axis measures similarity to baseline consistency (both increased and decreased consistency count as drift).
+
+**Ordinary dimensions** use Arsenic’s existing materiality rules (same bands that drive Amber/Red in each dimension), not raw numeric inequality and not risk colour as a proxy for unchanged:
+
+```
+unchanged  = applicable && !materially_changed
+changed    = applicable && materially_changed
+retention  = 100 × unchanged / applicable
+```
+
+Direction counts (regressions / improvements / neutral) apply only to materially changed probes. Risk/severity remains a separate deployment signal.
+
+**Consistency retention** keeps an aggregate repeatability similarity score. Per-probe fingerprint drift counts only **band crossing** (`v1_consistent != v2_consistent`): both sides inconsistent with similar variance is absolute telemetry, not compatibility drift. Absolute inconsistency remains in the consistency dimension `probes_affected` count. Near-identical aggregate scores (retention ≥ 95%) appear under telemetry, not as high-impact compatibility drops.
+
+Re-render old JSON (fingerprint is derived if missing):
+
+```bash
+arsenic report render ./report.json --format html --output ./report.html
+```
+
+Open [docs/fingerprint-demo.html](docs/fingerprint-demo.html) for a real-data example. The radar needs at least three valid axes; otherwise the retention table still renders.
 
 ---
 
@@ -282,7 +330,7 @@ arsenic probe list                 List standard probes
 arsenic probe list --category tone Filter by category
 arsenic probe show <name>          Show one probe as JSON
 arsenic probe validate <path>      Validate user corpus TOML
-arsenic report render <json>       Re-render a saved JSON report
+arsenic report render <json>       Re-render a saved JSON report (derives fingerprint if absent)
 arsenic report summary <json>      Print summary to stdout
 ```
 
@@ -291,17 +339,18 @@ arsenic report summary <json>      Print summary to stdout
 ## Built in Rust
 
 Fast. No runtime dependencies. The report is a single self-contained HTML file with no
-external CDN calls after the font load.
+chart CDN — the behavioural fingerprint radar is inline SVG (optional Google Fonts only).
 
 ```
 crates/
-  arsenic-core/       Comparison engine, claim matching, mutation engine
+  arsenic-core/       Comparison engine, claim matching, fingerprint, mutation engine
   arsenic-probes/     TOML probe loader
   arsenic-adapters/   OpenAI-compatible, Anthropic, Google adapters
   arsenic-report/     HTML / JSON report rendering
   arsenic-cli/        arsenic binary
 probe-suite/standard/ Standard probe suite (18 probes, 7 categories)
 examples/             Prebuilt HTML drift reports
+docs/                 Fingerprint demo HTML/JSON + screenshot
 ```
 
 ---

crates/arsenic-core/src/comparison.rs

@@ -149,6 +149,8 @@ impl ComparisonEngine {
         let upgrade_path = build_upgrade_path(&probe_results, &[]);
         let latency_summary = compute_latency_summary(&probe_results);
         let migration_profile = compute_migration_profile(&probe_results, &latency_summary);
+        let behaviour_fingerprint =
+            crate::fingerprint::compute_behaviour_fingerprint(&probe_results, &v1_model, &v2_model);
         Ok(DriftReport {
             run_id,
             generated_at: chrono::Utc::now(),
@@ -163,6 +165,7 @@ impl ComparisonEngine {
             mutation_results: Vec::new(),
             latency_summary,
             migration_profile,
+            behaviour_fingerprint,
         })
     }
 
@@ -808,7 +811,14 @@ impl ComparisonEngine {
     }
 
     pub fn compute_dimension_summaries(&self, results: &[ProbeResult]) -> DimensionSummaries {
-        DimensionSummaries {
+        use crate::materiality::{
+            claim_materially_changed, consistency_materially_changed, factual_materially_changed,
+            instruction_materially_changed, morphology_materially_changed,
+            refusal_materially_changed, schema_materially_changed, semantic_materially_changed,
+            tone_materially_changed,
+        };
+
+        let mut summaries = DimensionSummaries {
             morphology: dim_summary_core("morphology", results, |d| {
                 Some((&d.morphology.risk, d.morphology.direction))
             }),
@@ -840,7 +850,68 @@ impl ComparisonEngine {
             custom_assertions: dim_summary_core("custom_assertions", results, |d| {
                 d.custom_assertions.as_ref().map(|c| (&c.risk, c.direction))
             }),
-        }
+        };
+
+        // Material counts use the same helpers as the fingerprint (not risk colour).
+        // Consistency probes_affected remains absolute inconsistency (risk ≠ Green).
+        summaries.morphology.materially_changed_probes = results
+            .iter()
+            .filter(|pr| morphology_materially_changed(&pr.dimensions.morphology))
+            .count();
+        summaries.tone.materially_changed_probes = results
+            .iter()
+            .filter(|pr| tone_materially_changed(&pr.dimensions.tone))
+            .count();
+        summaries.factual.materially_changed_probes = results
+            .iter()
+            .filter(|pr| {
+                pr.dimensions
+                    .factual
+                    .as_ref()
+                    .is_some_and(factual_materially_changed)
+            })
+            .count();
+        summaries.schema.materially_changed_probes = results
+            .iter()
+            .filter(|pr| {
+                pr.dimensions
+                    .schema
+                    .as_ref()
+                    .is_some_and(schema_materially_changed)
+            })
+            .count();
+        summaries.instruction.materially_changed_probes = results
+            .iter()
+            .filter(|pr| {
+                pr.dimensions
+                    .instruction
+                    .as_ref()
+                    .is_some_and(instruction_materially_changed)
+            })
+            .count();
+        summaries.refusal.materially_changed_probes = results
+            .iter()
+            .filter(|pr| refusal_materially_changed(&pr.dimensions.refusal))
+            .count();
+        summaries.semantic.materially_changed_probes = results
+            .iter()
+            .filter(|pr| semantic_materially_changed(&pr.dimensions.semantic))
+            .count();
+        summaries.claim.materially_changed_probes = results
+            .iter()
+            .filter(|pr| claim_materially_changed(&pr.dimensions.claim, pr.probe.category))
+            .count();
+        summaries.consistency.materially_changed_probes = results
+            .iter()
+            .filter(|pr| {
+                pr.dimensions
+                    .consistency
+                    .as_ref()
+                    .is_some_and(consistency_materially_changed)
+            })
+            .count();
+
+        summaries
     }
 }
 
@@ -1481,6 +1552,11 @@ impl DriftReport {
         self.migration_profile =
             compute_migration_profile(&self.probe_results, &self.latency_summary);
         self.dimension_summaries = engine.compute_dimension_summaries(&self.probe_results);
+        self.behaviour_fingerprint = crate::fingerprint::compute_behaviour_fingerprint(
+            &self.probe_results,
+            &self.v1_model,
+            &self.v2_model,
+        );
     }
 
     /// Recomputes valence counts from `probe_results` and copies probe-level buckets into `summary`.
@@ -1702,17 +1778,20 @@ fn build_upgrade_path(results: &[ProbeResult], mutations: &[MutationResult]) ->
             .count(),
         auto_certified: mutations.iter().filter(|m| m.validated).count(),
     };
-    UpgradePathReport {
+    let mut path = UpgradePathReport {
         critical_regressions: critical,
         policy_changes: policy,
         blocking_regressions: blocking,
+        changes_to_verify: verify.clone(),
         improvements_to_verify: verify,
         neutral_changes: neutral,
         presentation_drift: presentation,
         telemetry_drift: telemetry,
         certified_prompts: certified,
         remediation,
-    }
+    };
+    path.sync_review_aliases();
+    path
 }
 
 fn dim_summary_core<Rd>(dimension: &str, results: &[ProbeResult], mut get: Rd) -> DimensionSummary
@@ -1753,6 +1832,8 @@ where
         drift_neutral,
         drift_not_applicable,
         impact_label,
+        // Ordinary dimensions: non-Green risk is the materiality signal.
+        materially_changed_probes: affected,
     }
 }
 
@@ -1765,7 +1846,12 @@ fn morphology_risk_level(
 ) -> RiskLevel {
     if token_delta_pct >= red || (response_type_changed && token_delta_pct >= amber) {
         RiskLevel::Red
-    } else if token_delta_pct >= amber || structure_changed || response_type_changed {
+    } else if crate::materiality::morphology_crosses_material_band(
+        token_delta_pct,
+        response_type_changed,
+        structure_changed,
+        amber,
+    ) {
         RiskLevel::Amber
     } else {
         RiskLevel::Green
@@ -1846,6 +1932,15 @@ fn soften_dimensions_for_code_formatting(probe: &Probe, dims: &mut ProbeDimensio
     if fence_only_structure || dims.morphology.delta.token_delta_pct < 0.10 {
         dims.morphology.risk = RiskLevel::Green;
         dims.morphology.direction = DriftDirection::Neutral;
+        // Keep materiality helpers aligned with the softened Green decision.
+        dims.morphology.delta.structure_changed = false;
+        dims.morphology.delta.response_type_changed = false;
+        if dims.morphology.delta.token_delta_pct
+            >= crate::materiality::DEFAULT_MORPHOLOGY_TOKEN_DELTA_AMBER
+        {
+            dims.morphology.delta.token_delta_pct =
+                crate::materiality::DEFAULT_MORPHOLOGY_TOKEN_DELTA_AMBER - f64::EPSILON;
+        }
     }
 
     dims.semantic.risk = RiskLevel::Green;