v5.2.0

Added

• Path patterns got a real syntax, and a guide page to go with it: Path Patterns. One position can now be a class of events instead of a single name — [a|b|c] (any of), [^a] / [^a|b] (anything but), . (any event) — which replaces the rename_events workaround for "any of these events", damaging the stream to ask one question. Quantifying a class with * turns it into a restricted gap, a run of events like this rather than one: add_to_cart->[^support_chat]*->purchase is "bought without ever contacting support", path_start->[^purchase]*->path_end is "never purchased". .* is the unrestricted case of the same construct, so the two compose. Works everywhere a pattern is accepted — Step Matrix / Step Sankey / Transition Graph path_pattern, the matches_pattern metric, truncate_paths anchors. Details: the syntax follows Python's regular expressions with one substitution, a token is an event rather than a character, so negation lives inside the brackets ([^a], never ^[a]) and members are separated by |. A class occupies one position and one ordinal, so at= / occurrence= / Step Matrix centring are unaffected — centring on [payment_error|checkout_bug] behaves as centring on an event, except that column 0 shows a distribution over the class rather than one event at 1.0; a gap is not a position and takes no ordinal, so they are unaffected by that too. . and [^...] never match path_start/path_end, mirroring regex, where . does not match a string boundary — which is what makes ".->product_view" mean "a product view that was not the path's first event", previously inexpressible. A restricted gap needs an anchor on each side, and ^/$ are not anchors: a path's boundaries already have names. Constructs whose scope would exceed one position ([^a->b], a->b|c->d, [a|b]*) are rejected with an explanation rather than half-supported

• An anchor is a spec, not just an event name. truncate_paths' start_anchor/end_anchor each accept an event name, a spec {"pattern", "at", "occurrence", "offset", "offset_side"}, or a list of either. pattern is a full -> pattern, so a window can open on the event that completes a sequence rather than on any occurrence of it; at picks which of its event names anchors; occurrence chooses "first" (default) or "last"; offset moves the bound by a number of events (10) or by time ("30m", snapping to the nearest event inside the window, clamping at the path's own boundary), and offset_side says which way a time offset rounds to a real event — "start" (forward) or "end" (backward). A list keeps the narrowest window the anchors imply — the latest start, the earliest end — which is how you write both "whichever comes first" (end_anchor=["purchase", {"pattern": "add_to_cart", "offset": 10}]) and a keep-whole fallback (end_anchor=["purchase", "path_end"], cutting converters at their purchase while keeping everyone else). The same spec is what add_events, Step Matrix, Step Sankey and get_conversion_rate take. Two notes on the newer keys. occurrence="all" is the third value, returning every position each token can occupy in some valid match — the union of what "first" and "last" pick — instead of choosing one; the rows then no longer describe a single match, so anything needing one position rejects it — truncate_paths raises rather than guessing which occurrence to cut at, and points at add_events; and offset_side only affects a time offset, since a step offset always lands on a row. truncate_paths still defaults to rounding a mark inward, keeping an exact hit inside the window on both sides

• add_events(anchor=...): a fourth mode that inserts an event at a position rather than at every occurrence of an event name — anchor={"pattern": "cart->[^cart]*->shipping_details", "at": "start"} names "the cart that checkout actually followed", not every cart. This is what makes such a position usable by the rest of the library: a pattern can describe it, but only an event name can be centred on by Step Matrix, counted by a funnel, or filtered on. With occurrence="all" it marks every attempt rather than one per path. One anchor per call, not a list — a list in truncate_paths is a fallback chain narrowing one window, and there is no window here to narrow, so two markers means two calls. path_col resolves the anchor per session rather than per user; a path where the anchor resolves nowhere simply gets no event, though a single anchor naming an unknown event is a hard error

• Step Matrix / Step Sankey accept anchor=: centre everything on one position instead of laying a pattern's parts out side by side. It reaches the two things path_pattern cannot say — which occurrence to centre on ({"pattern": "cart", "occurrence": "last"}), and an offset in events or in time — and yields a single block. Mutually exclusive with path_pattern (ADR-0008 rule 6); a path where the anchor does not resolve has no centre and is absent, so the anchor selects as well as centres. occurrence="all" is rejected here: several centres per path would let one path count more than once while every cell is a share of paths. A pattern typed into the widget sidebar replaces an anchor passed from Python rather than colliding with it

• get_conversion_rate(start_anchor, end_anchor, within=None): "if Y happened, how often does X follow — and is that different from the baseline?" as one call instead of a filter_paths(matches_pattern("Y->.*->X")) + get_metrics + .mean() composition. Returns a row per (start, end) pair with paths_with_start, converted, conversion_rate, base_rate and lift: the denominator ships in every row because 0.5 out of two paths and 0.5 out of five thousand are different claims, and the base rate because a rate that looks high only for an event that is common everywhere is not a finding — lift below 1 says the start event makes the outcome less likely. within expresses the window the composition could not: an int counts events, a duration counts time ("30m"), both measured from the start anchor and inclusive of the far edge. Both sides take anchor specs, so end_anchor="path_end", within=1 is an exit rate and start_anchor={"pattern": "path_start->catalog", "at": -1} restricts the question to the sessions that landed there; a list on either side is a fan-out into separate questions, one row per combination. The unit of observation is the path, not the occurrence: a path where Y happened three times counts once, so per-visit questions ("of 23,000 visits, how many were entrances") remain out of scope. Also exposed as an MCP agent tool with a conversion_rate playbook scenario — the first one that answers in numbers instead of registering a report tab, so its docstring and the system prompt both say to quote those numbers in backticks (which check_analysis exempts from the anchor-link rule) and to report the denominator and the lift rather than the rate alone. describe_tool()'s index gained an analysis_tools key for the same reason: a tool called directly is not a step you put in a preprocessors list

• add_segment(metric_bins=...): split paths into a segment by any per-path metric. {"metric": {...}, "edges": [5, 15], "segment_levels": ["short", "mid", "long"]}, or "quantiles": 4 for quartiles / "quantiles": [0.25, 0.75] for chosen cut points. Cut points are interior — N of them always give N+1 bins, so nothing falls outside the split (unlike pd.cut, whose out-of-range values become NaN, which a segment column cannot hold). segment_levels is optional; without it bins are named "[5, 15)" / q1..qN. Paths whose metric has no value get the level "undefined", which is not counted as a bin

• in_segment_bulk path metric: the in_segment membership check fanned out over whole segment columns, the way event_count_bulk fans out over events. {"metric": "in_segment_bulk", "metric_args": {"segment_name": "channel"}} gives one 0/1 column per level of channel; omitting segment_name too gives one column per level of every segment column, which is the one-liner for "put all my segments into this clustering feature set / overview". segment_levels (a list) narrows it back down, mode/threshold work exactly as in in_segment, and the columns are named in_segment_bulk_{segment}_{level}_{mode}. As with the other *_bulk metrics, an explicit empty list is rejected rather than read as the wildcard, and the metric cannot appear in a filter_paths/collapse_events condition, which needs one comparable value per path — use `in_segm...

v5.1.0

Added

• Transition graph: deterministic semantic node layout (new tools/graph_layout.py, adds a gensim dependency) — word2vec embedding of user trajectories, recursively clustered and mapped onto nested canvas regions so related events share a region. Deterministic across kernel restarts (fixed seed, single worker, process-independent hash, pinned corpus order). Used for new widgets, Reset layout (new toolbar button), HTML export, and MCP report tabs; manually arranged positions still always win
• Transition graph: per-node top-k edge filter as the new default ("Auto" mode, k strongest outgoing edges per node, adjustable stepper) with a toggle back to the manual weight-range slider; saved states and old exported HTML with a [min, max] filter keep working as manual mode
• Transition graph: contextual legend (bottom-left, collapsible) explaining node size / edge width / focus and diff colors, with a coverage indicator ("edges: X / Y (Z% of weight)") and interaction hints
• Transition graph: edge focus — clicking an edge dims everything else, fits the node pair, and shows the weight label; edge coloring moved to a toolbar button shown while an edge is focused
• Transition graph: route statistics — selecting a path (⌘click) shows a badge with stats for that exact contiguous route: unique paths (and share), traversal count, average per path, median / p95 route duration, or the Markov probability product of its edges; the default metric follows the current edge weight and is switchable in the badge. Backed by a new internal route-stats helper (strict contiguous matching, overlapping occurrences counted) — widget-only, not exposed on Eventstream
• Transition graph: ego view — with a node focused, a toolbar button expands its neighborhood into a modal mini-sankey: incoming transitions left, outgoing right, self-loops on both sides. The sides show shares, not the graph's edge weights: each source's share of the arrivals (proba_in) and each target's share of the exits (proba_out), with raw counts in the tooltip — the graph payload now carries sparse transition counts to make this exact under any edge weight. Clicking a neighbor re-centers the view on it; diff mode shows the displayed diff values with the red/blue code, and hovering a ribbon shows the same per-group breakdown tooltip as a graph edge. Works in exported HTML too — no kernel needed
• Transition graph: GraphView — serializable named visual presets (focus on a node/edge/path, filters, hidden events, viewport; never data parameters). Entry points: transition_graph(views=[...], view=...) kwargs rendered as pills with a Default reset, a "Copy view link" toolbar button, a #view=<base64url> URL fragment on exported HTML, [Tab:view=Name] links in MCP report analysis (plus a views= parameter on the MCP add_transition_graph tool). Node/edge analysis links now go through the same pipeline, so edge links in exported reports use the proper edge focus instead of the marching-ants animation

Changed

• Transition graph: focusing a node (search, exported-report links) now fits the node together with its neighborhood instead of zooming onto the node itself, so its edges stay inside the viewport
• Transition graph: in focus mode all incoming/outgoing edges are colored (violet/orange) regardless of weight — the confusing gray fallback for weak edges is gone
• Transition graph: edge labels are allocated adaptively — small graphs get every edge labeled, large ones a bounded share of the currently visible edges (tightening the filter labels more of what remains); was a fixed top-10
• Transition graph: the auto-layout is deterministic — the same graph renders the same picture every time (seeded PRNG around fcose)
• Transition graph: the manual edge filter slider is logarithmic for every weight type, spanning exactly the data range (0.5%–100% for probabilities, smallest nonzero weight to max otherwise) — no more dead track zones
• Transition graph: in diff mode edge thickness/opacity for probability weights is normalized against the largest |Δp| on the graph instead of the absolute 100%, so the strongest changes render thick instead of uniformly thin; diff self-loops keep their red/blue color in focus mode
• Lowered minimum supported Python version from 3.11 to 3.10
• Reorganized license files to clarify exact Apache-2.0, Notice file with copyright created

Fixed

• diff over a boolean or numeric segment failed with SegmentValueNotFoundError when the values arrived as strings (which is what the widget UI and MCP always send): 'false'/'5' now resolve back to the typed segment levels False/5
• Transition graph: probability edges below 1% were silently dropped (including |Δp| < 1pp in diff mode) — removed; hiding is now always explicit via the edge filter and reported by the coverage indicator
• Transition graph: a saved edge-weight filter was silently reset after any graph rebuild (e.g. switching the weight type) until the slider was touched
• Transition graph: the broken graph_layout backend compute (imported a module that didn't exist and silently returned nothing) is implemented; compute errors are now surfaced to the client instead of swallowed
• cluster_analysis_data() now applies the documented n_clusters="3-8" default instead of raising ValueError when it's omitted for the kmeans method; corrected the cluster_analysis/cluster_analysis_data docstrings, which falsely claimed features and overview_metrics default to per-event counts (#89)
• add_clusters()'s scaler argument now defaults to "minmax" instead of None, matching the AddClusters processor's own default and cluster_analysis_data()'s documented default. Previously, omitting scaler silently clustered on unscaled features, so add_clusters() could produce a different clustering than cluster_analysis_data() for the same features/n_clusters

v5.0.1

Changed

• Relicensed from the Retentioneering Software Non-Exclusive License to Apache-2.0
• Updated the PyPI package description, keywords, and classifiers for discoverability

Fixed

• Release workflow runs are now labeled Release <tag> instead of inheriting the
  triggering commit's message, so they're distinguishable from CI runs in the
  Actions list

v5.0.0

Complete rewrite of the library's core engine, compared to 3.3.0. The
pandas-based Eventstream, the iframe+CDN-loaded Transition Graph /
Preprocessing Graph widgets, and the params_model GUI-schema system are
gone, replaced by a DuckDB-backed Eventstream and a new generation of
anywidget-based interactive widgets. The JS visualization layer is now
open source and lives in this repository — it's built in CI and embedded
directly into the Python wheel, instead of being downloaded from a CDN at
runtime.

Breaking

• Requires Python 3.11+ (was 3.8–3.11 in 3.3.0)
• Eventstream is now backed by DuckDB/pyarrow instead of pandas; the old
  data_processor/preprocessor/data_processors_lib pipeline and the
  params_model/widget pydantic-based GUI-schema system have both been
  removed — data processors now take plain keyword arguments
• The interactive Preprocessing Graph (the visual, no-code pipeline builder)
  and the Cohorts, StatTests, and Sequences tools from 3.3.0 are not
  part of this release. There is no direct replacement yet — they may return
  in a future version
• TransitionGraph, StepMatrix, StepSankey, Funnel, and Clusters
  (now ClusterAnalysis) are reimplemented from scratch as anywidget
  components instead of iframe+CDN-loaded JS; the custom Jupyter kernel-comm
  backend (backend/, iframe postMessage bridge) they depended on has been
  removed

Eventstream API changes

Renamed or changed signature (same concept, different call shape):

• to_dataframe(copy=False) → to_dataframe(exclude_start_end=True)
• filter_events(func) → filter_events(keep=None, drop=None, func=None, sql=None)
  — func is now one of four alternative filtering modes
• add_start_end_events() → add_start_end_events(path_col=None)
• split_sessions(timeout, delimiter_events, delimiter_col, session_col, mark_truncated)
  → split_sessions(session_col, session_index_col, separator, start_event, end_event, timeout, path_col, event_col)
  — timeout now takes a duration string with an explicit unit ("30m") or
  a pandas.Timedelta; bare numbers are rejected
• truncate_paths(drop_before, drop_after, occurrence_before, occurrence_after, shift_before, shift_after)
  → truncate_paths(start_event, end_event, path_col=None, event_col=None)
• rename(rules: list[dict]) → rename_events(mapping: dict)
• collapse_loops(suffix, time_agg) → collapse_events(consecutive, event_groups, group_col, session_col, session_type_col, agg, path_col, event_col)
• describe()/describe_events() → describe() — single headless summary
  (schema, shape (event/path counts), date range, event frequency, and path
  length/duration distributions (mean/median/min/max/percentiles))

Removed, no equivalent in this release:
copy(), append_eventstream(), index_events(), add_custom_col(),
clusters() (the stateful fit()/extract_features() object), cohorts(),
stattests(), timedelta_hist(), user_lifetime_hist(),
event_timestamp_hist(), preprocessing_graph(), transition_matrix() as a
public method (the computation still happens internally inside
transition_graph_data()), sequences(), add_negative_events(),
add_positive_events(), drop_paths(), group_events(),
group_events_bulk(), label_cropped_paths(), label_lost_users(),
label_new_users(), pipe()

Added, no equivalent in 3.3.0:
schema/df properties, is_empty(), equals(), get_event_counts(),
fingerprint, get_segment_levels(), urls_to_events(), filter_paths()
(condition-tree based), get_metrics(), add_events(), add_segment(),
add_clusters() (a new one-shot processor, unrelated to 3.3.0's clusters()),
to_daily_states(), drop_segment(), edit_events(), drop_events(),
sample_paths(), transition_graph_data(), step_sankey_data() /
step_matrix_data(), funnel_data(),
segment_overview()/segment_overview_data(),
cluster_analysis()/cluster_analysis_data(), get_metric_distribution()

Naming conventions across the new API:

• one column vocabulary everywhere: path_col, event_col, timestamp_col,
  session_col, segment_col
• window anchors are always the start_event / end_event pair
  (truncate_paths, split_sessions, the time_between metric)
• the diff-mode sentinel for "every other segment value" is <REST>
• path metric names: has_event, matches_pattern, in_segment,
  first_event_time (plus length, duration, event_count,
  time_between, active_days); the complement_distance aggregation
• transition-graph edge weights: proba_out, proba_in, count,
  unique_paths, share_of_total, avg_per_path, time_median, time_q95
• sample_paths(n=, frac=) mirrors pandas.DataFrame.sample

Added

• VS Code-based environments (including Cursor) are fully supported now.
• Segment Overview — a new widget/tool with no equivalent in 3.3.0, for
  comparing metrics across segments
• MCP server (retentioneering.mcp.serve()) — exposes the eventstream
  to Claude and other MCP clients over SSE, with tools for adding widgets,
  managing a session baseline, validating analysis text, and exporting a
  multi-widget static HTML report with clickable cross-references
• ipywidgets is now a core dependency, so widgets work out of the box in
  plain JupyterLab
• schema.custom_cols now defaults to None instead of []: any DataFrame
  column not otherwise declared in the schema is added to it automatically,
  keeping it from being silently dropped by row-reshaping data processors
  (collapse_events, to_daily_states). Passing an explicit list — even
  [] — switches to strict mode: only schema-declared and listed columns
  are kept, everything else is excluded from the eventstream.
• rename_segment_levels(segment_col, mapping) — rename levels within an existing
  segment column (e.g. cluster labels produced by add_clusters, or messy raw
  segment data), analogous to rename_events but for segment columns
• cluster_analysis_data() and the Cluster Analysis widget now report best_params
  (chosen_params on the widget) — the concrete clustering parameters that produced
  the current result (e.g. the winning n_clusters from a silhouette grid search),
  so they can be passed straight to add_clusters to reproduce it
• Cluster Analysis widget: "Save Clusters" sidebar action — write the current
  clustering into the eventstream as a new segment column, optionally renaming
  cluster labels first. Choose either a copy-pasteable add_clusters(...) code
  snippet (stream stays untouched) or applying it in place immediately
• All widgets accept a state_file argument binding the full widget state
  (data and display parameters, plus widget-specific extras: the transition
  graph's node layout, event visibility, filters, and zoom; the step matrix's
  event visibility/pins, filters, row order, step window, and horizontal
  scroll; the step sankey's event count filter and horizontal scroll; the
  cluster analysis' cluster renames and active tab) to a JSON file: if the
  file exists the state is loaded from it, otherwise it is created, and every
  subsequent change is auto-saved. Explicitly passed arguments override the
  loaded state.
• The transition graph keeps its zoom/pan across recomputes (changing edge
  weight, diff, or path column no longer resets the viewport); step matrix
  and step sankey likewise keep their horizontal scroll and row order.

Fixed

• SQL injection in the funnel data processor's query builder — event names
  are now properly escaped

v5.0.0rc3

Complete rewrite of the library's core engine, compared to 3.3.0. The
pandas-based Eventstream, the iframe+CDN-loaded Transition Graph /
Preprocessing Graph widgets, and the params_model GUI-schema system are
gone, replaced by a DuckDB-backed Eventstream and a new generation of
anywidget-based interactive widgets. The JS visualization layer is now
open source and lives in this repository — it's built in CI and embedded
directly into the Python wheel, instead of being downloaded from a CDN at
runtime.

Breaking

• Requires Python 3.11+ (was 3.8–3.11 in 3.3.0)
• Eventstream is now backed by DuckDB/pyarrow instead of pandas; the old
  data_processor/preprocessor/data_processors_lib pipeline and the
  params_model/widget pydantic-based GUI-schema system have both been
  removed — data processors now take plain keyword arguments
• The interactive Preprocessing Graph (the visual, no-code pipeline builder)
  and the Cohorts, StatTests, and Sequences tools from 3.3.0 are not
  part of this release. There is no direct replacement yet — they may return
  in a future version
• TransitionGraph, StepMatrix, StepSankey, Funnel, and Clusters
  (now ClusterAnalysis) are reimplemented from scratch as anywidget
  components instead of iframe+CDN-loaded JS; the custom Jupyter kernel-comm
  backend (backend/, iframe postMessage bridge) they depended on has been
  removed

Eventstream API changes

Renamed or changed signature (same concept, different call shape):

• to_dataframe(copy=False) → to_dataframe(exclude_start_end=True)
• filter_events(func) → filter_events(keep=None, drop=None, func=None, sql=None)
  — func is now one of four alternative filtering modes
• add_start_end_events() → add_start_end_events(path_col=None)
• split_sessions(timeout, delimiter_events, delimiter_col, session_col, mark_truncated)
  → split_sessions(session_col, session_index_col, separator, start_event, end_event, timeout, path_col, event_col)
  — timeout now takes a duration string with an explicit unit ("30m") or
  a pandas.Timedelta; bare numbers are rejected
• truncate_paths(drop_before, drop_after, occurrence_before, occurrence_after, shift_before, shift_after)
  → truncate_paths(start_event, end_event, path_col=None, event_col=None)
• rename(rules: list[dict]) → rename_events(mapping: dict)
• collapse_loops(suffix, time_agg) → collapse_events(consecutive, event_groups, group_col, session_col, session_type_col, agg, path_col, event_col)
• describe()/describe_events() → describe() — single headless summary
  (schema, shape (event/path counts), date range, event frequency, and path
  length/duration distributions (mean/median/min/max/percentiles))

Removed, no equivalent in this release:
copy(), append_eventstream(), index_events(), add_custom_col(),
clusters() (the stateful fit()/extract_features() object), cohorts(),
stattests(), timedelta_hist(), user_lifetime_hist(),
event_timestamp_hist(), preprocessing_graph(), transition_matrix() as a
public method (the computation still happens internally inside
transition_graph_data()), sequences(), add_negative_events(),
add_positive_events(), drop_paths(), group_events(),
group_events_bulk(), label_cropped_paths(), label_lost_users(),
label_new_users(), pipe()

Added, no equivalent in 3.3.0:
schema/df properties, is_empty(), equals(), get_event_counts(),
fingerprint, get_segment_levels(), urls_to_events(), filter_paths()
(condition-tree based), get_metrics(), add_events(), add_segment(),
add_clusters() (a new one-shot processor, unrelated to 3.3.0's clusters()),
to_daily_states(), drop_segment(), edit_events(), drop_events(),
sample_paths(), transition_graph_data(), step_sankey_data() /
step_matrix_data(), funnel_data(),
segment_overview()/segment_overview_data(),
cluster_analysis()/cluster_analysis_data(), get_metric_distribution()

Naming conventions across the new API:

• one column vocabulary everywhere: path_col, event_col, timestamp_col,
  session_col, segment_col
• window anchors are always the start_event / end_event pair
  (truncate_paths, split_sessions, the time_between metric)
• the diff-mode sentinel for "every other segment value" is <REST>
• path metric names: has_event, matches_pattern, in_segment,
  first_event_time (plus length, duration, event_count,
  time_between, active_days); the complement_distance aggregation
• transition-graph edge weights: proba_out, proba_in, count,
  unique_paths, share_of_total, avg_per_path, time_median, time_q95
• sample_paths(n=, frac=) mirrors pandas.DataFrame.sample

Added

• VS Code-based environments (including Cursor) are fully supported now.
• Segment Overview — a new widget/tool with no equivalent in 3.3.0, for
  comparing metrics across segments
• MCP server (retentioneering.mcp.serve()) — exposes the eventstream
  to Claude and other MCP clients over SSE, with tools for adding widgets,
  managing a session baseline, validating analysis text, and exporting a
  multi-widget static HTML report with clickable cross-references
• ipywidgets is now a core dependency, so widgets work out of the box in
  plain JupyterLab
• schema.custom_cols now defaults to None instead of []: any DataFrame
  column not otherwise declared in the schema is added to it automatically,
  keeping it from being silently dropped by row-reshaping data processors
  (collapse_events, to_daily_states). Passing an explicit list — even
  [] — switches to strict mode: only schema-declared and listed columns
  are kept, everything else is excluded from the eventstream.
• rename_segment_levels(segment_col, mapping) — rename levels within an existing
  segment column (e.g. cluster labels produced by add_clusters, or messy raw
  segment data), analogous to rename_events but for segment columns
• cluster_analysis_data() and the Cluster Analysis widget now report best_params
  (chosen_params on the widget) — the concrete clustering parameters that produced
  the current result (e.g. the winning n_clusters from a silhouette grid search),
  so they can be passed straight to add_clusters to reproduce it
• Cluster Analysis widget: "Save Clusters" sidebar action — write the current
  clustering into the eventstream as a new segment column, optionally renaming
  cluster labels first. Choose either a copy-pasteable add_clusters(...) code
  snippet (stream stays untouched) or applying it in place immediately
• All widgets accept a state_file argument binding the full widget state
  (data and display parameters, plus widget-specific extras: the transition
  graph's node layout, event visibility, filters, and zoom; the step matrix's
  event visibility/pins, filters, row order, step window, and horizontal
  scroll; the step sankey's event count filter and horizontal scroll; the
  cluster analysis' cluster renames and active tab) to a JSON file: if the
  file exists the state is loaded from it, otherwise it is created, and every
  subsequent change is auto-saved. Explicitly passed arguments override the
  loaded state.
• The transition graph keeps its zoom/pan across recomputes (changing edge
  weight, diff, or path column no longer resets the viewport); step matrix
  and step sankey likewise keep their horizontal scroll and row order.

Fixed

• SQL injection in the funnel data processor's query builder — event names
  are now properly escaped

v5.0.0rc2

Complete rewrite of the library's core engine, compared to 3.3.0. The
pandas-based Eventstream, the iframe+CDN-loaded Transition Graph /
Preprocessing Graph widgets, and the params_model GUI-schema system are
gone, replaced by a DuckDB-backed Eventstream and a new generation of
anywidget-based interactive widgets. The JS visualization layer is now
open source and lives in this repository — it's built in CI and embedded
directly into the Python wheel, instead of being downloaded from a CDN at
runtime.

Breaking

• Requires Python 3.11+ (was 3.8–3.11 in 3.3.0)
• Eventstream is now backed by DuckDB/pyarrow instead of pandas; the old
  data_processor/preprocessor/data_processors_lib pipeline and the
  params_model/widget pydantic-based GUI-schema system have both been
  removed — data processors now take plain keyword arguments
• The interactive Preprocessing Graph (the visual, no-code pipeline builder)
  and the Cohorts, StatTests, and Sequences tools from 3.3.0 are not
  part of this release. There is no direct replacement yet — they may return
  in a future version
• TransitionGraph, StepMatrix, StepSankey, Funnel, and Clusters
  (now ClusterAnalysis) are reimplemented from scratch as anywidget
  components instead of iframe+CDN-loaded JS; the custom Jupyter kernel-comm
  backend (backend/, iframe postMessage bridge) they depended on has been
  removed

Eventstream API changes

Renamed or changed signature (same concept, different call shape):

• to_dataframe(copy=False) → to_dataframe(exclude_start_end=True)
• filter_events(func) → filter_events(keep=None, drop=None, func=None, sql=None)
  — func is now one of four alternative filtering modes
• add_start_end_events() → add_start_end_events(path_col=None)
• split_sessions(timeout, delimiter_events, delimiter_col, session_col, mark_truncated)
  → split_sessions(session_col, session_index_col, separator, start_event, end_event, timeout, path_col, event_col)
  — timeout now takes a duration string with an explicit unit ("30m") or
  a pandas.Timedelta; bare numbers are rejected
• truncate_paths(drop_before, drop_after, occurrence_before, occurrence_after, shift_before, shift_after)
  → truncate_paths(start_event, end_event, path_col=None, event_col=None)
• rename(rules: list[dict]) → rename_events(mapping: dict)
• collapse_loops(suffix, time_agg) → collapse_events(consecutive, event_groups, group_col, session_col, session_type_col, agg, path_col, event_col)
• describe()/describe_events() → describe() — single headless summary
  (schema, shape (event/path counts), date range, event frequency, and path
  length/duration distributions (mean/median/min/max/percentiles))

Removed, no equivalent in this release:
copy(), append_eventstream(), index_events(), add_custom_col(),
clusters() (the stateful fit()/extract_features() object), cohorts(),
stattests(), timedelta_hist(), user_lifetime_hist(),
event_timestamp_hist(), preprocessing_graph(), transition_matrix() as a
public method (the computation still happens internally inside
transition_graph_data()), sequences(), add_negative_events(),
add_positive_events(), drop_paths(), group_events(),
group_events_bulk(), label_cropped_paths(), label_lost_users(),
label_new_users(), pipe()

Added, no equivalent in 3.3.0:
schema/df properties, is_empty(), equals(), get_event_counts(),
fingerprint, get_segment_levels(), urls_to_events(), filter_paths()
(condition-tree based), get_metrics(), add_events(), add_segment(),
add_clusters() (a new one-shot processor, unrelated to 3.3.0's clusters()),
to_daily_states(), drop_segment(), edit_events(), drop_events(),
sample_paths(), transition_graph_data(), step_sankey_data() /
step_matrix_data(), funnel_data(),
segment_overview()/segment_overview_data(),
cluster_analysis()/cluster_analysis_data(), get_metric_distribution()

Naming conventions across the new API:

• one column vocabulary everywhere: path_col, event_col, timestamp_col,
  session_col, segment_col
• window anchors are always the start_event / end_event pair
  (truncate_paths, split_sessions, the time_between metric)
• the diff-mode sentinel for "every other segment value" is <REST>
• path metric names: has_event, matches_pattern, in_segment,
  first_event_time (plus length, duration, event_count,
  time_between, active_days); the complement_distance aggregation
• transition-graph edge weights: proba_out, proba_in, count,
  unique_paths, share_of_total, avg_per_path, time_median, time_q95
• sample_paths(n=, frac=) mirrors pandas.DataFrame.sample

Added

• VS Code-based environments (including Cursor) are fully supported now.
• Segment Overview — a new widget/tool with no equivalent in 3.3.0, for
  comparing metrics across segments
• MCP server (retentioneering.mcp.serve()) — exposes the eventstream
  to Claude and other MCP clients over SSE, with tools for adding widgets,
  managing a session baseline, validating analysis text, and exporting a
  multi-widget static HTML report with clickable cross-references
• ipywidgets is now a core dependency, so widgets work out of the box in
  plain JupyterLab
• schema.custom_cols now defaults to None instead of []: any DataFrame
  column not otherwise declared in the schema is added to it automatically,
  keeping it from being silently dropped by row-reshaping data processors
  (collapse_events, to_daily_states). Passing an explicit list — even
  [] — switches to strict mode: only schema-declared and listed columns
  are kept, everything else is excluded from the eventstream.
• rename_segment_levels(segment_col, mapping) — rename levels within an existing
  segment column (e.g. cluster labels produced by add_clusters, or messy raw
  segment data), analogous to rename_events but for segment columns
• cluster_analysis_data() and the Cluster Analysis widget now report best_params
  (chosen_params on the widget) — the concrete clustering parameters that produced
  the current result (e.g. the winning n_clusters from a silhouette grid search),
  so they can be passed straight to add_clusters to reproduce it
• Cluster Analysis widget: "Save Clusters" sidebar action — write the current
  clustering into the eventstream as a new segment column, optionally renaming
  cluster labels first. Choose either a copy-pasteable add_clusters(...) code
  snippet (stream stays untouched) or applying it in place immediately
• All widgets accept a state_file argument binding the full widget state
  (data and display parameters, plus widget-specific extras: the transition
  graph's node layout, event visibility, filters, and zoom; the step matrix's
  event visibility/pins, filters, row order, step window, and horizontal
  scroll; the step sankey's event count filter and horizontal scroll; the
  cluster analysis' cluster renames and active tab) to a JSON file: if the
  file exists the state is loaded from it, otherwise it is created, and every
  subsequent change is auto-saved. Explicitly passed arguments override the
  loaded state.
• The transition graph keeps its zoom/pan across recomputes (changing edge
  weight, diff, or path column no longer resets the viewport); step matrix
  and step sankey likewise keep their horizontal scroll and row order.

Fixed

• SQL injection in the funnel data processor's query builder — event names
  are now properly escaped

v5.0.0rc1

rc-release: 5.0.0rc1