feat(profiles): Mixpanel-style profiles page — perf, filters, search - #371
Open
ayushjhanwar-png wants to merge 12 commits into
Open
feat(profiles): Mixpanel-style profiles page — perf, filters, search#371ayushjhanwar-png wants to merge 12 commits into
ayushjhanwar-png wants to merge 12 commits into
Conversation
Profiles page overhaul for large projects (dashreels: ~80M profiles). Performance - getProfileList: recent-month partition window + do_not_merge_across_partitions _select_final → ~20s to ~1.6s. Falls back to full range if the window doesn't fill the page. - Exact profile-id search is a PK point lookup (~instant); fuzzy name/email ILIKE is the fallback scan. - Count uses uniq(id) (approx distinct users) instead of counting every ReplacingMergeTree version. Filters (v1, Mixpanel-style) - "Users who did event X [where event.property = Y] in a date range" — subquery over the raw events table (correct for anon/pre-identify; the MV would undercount). No event selected → filters apply as profile-property filters. - Date range lives inside the filter card next to the event selector and only shows for a real event selection. - The `['*']` "All Events" wildcard is stripped so it means "no filter" instead of an event literally named `*` (which returned 0 profiles). Count - Reflects the active filter (behavioral → distinct profiles who did the event; property → filtered uniq(id)) and mirrors the list's id-first search so an id-search count matches the row shown. - Prominent top-left, flush with the title/card, skeleton while (re)fetching. Columns / UI - Mixpanel columns: Name, Email, Distinct ID, Last seen, Country, Region, City (dropped Referrer/OS/Browser/Model). - Removed the Identified/Anonymous/Power-users tabs; loader shows during filter refetch.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughProfile listing now supports behavioral, property, search, event-count, and last-seen filters through tRPC and ClickHouse. The identified profiles view exposes these controls, updates table fields and loading behavior, and simplifies the surrounding route layout. ChangesProfile behavioral filtering and presentation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant IdentifiedProfiles
participant EventsFilters
participant profileRouterList
participant getProfileList
participant ClickHouse
IdentifiedProfiles->>EventsFilters: render event, count, and range controls
EventsFilters-->>IdentifiedProfiles: selected filters and date bounds
IdentifiedProfiles->>profileRouterList: submit profile list options
profileRouterList->>getProfileList: validate and forward options
getProfileList->>ClickHouse: execute behavioral or property-filtered query
ClickHouse-->>getProfileList: profiles and count
getProfileList-->>IdentifiedProfiles: update table and count
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/db/src/services/profile.service.ts`:
- Around line 312-316: Fix SQL injection in the fuzzy search branches of the
profile query and getProfileListCount by escaping the user-provided search value
before constructing the ILIKE pattern. Reuse the escaped LIKE pattern
consistently across email, first_name, and last_name conditions, while
preserving the existing fuzzy-search behavior.
- Around line 196-199: Update the column selection in eventFilterClauses so bare
f.name values resolve only through a fixed allowlist map of permitted event
columns; reject or skip unknown names instead of interpolating them directly
into SQL. Preserve the existing properties.* handling and IN-list construction,
while ensuring the mapped column is used in ${col} IN (...).
- Around line 183-185: Update the date-filter logic around formatClickhouseDate
in the profile query builder to validate startDate and endDate before formatting
them, ignoring malformed values so invalid inputs fall back to the default range
instead of throwing. Preserve the existing BETWEEN clause when both dates are
valid.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 12f50cbe-a44a-4b1d-866f-43a3e06f95e8
📒 Files selected for processing (7)
apps/start/src/components/events/filters/events-filters.tsxapps/start/src/components/profiles/table/columns.tsxapps/start/src/components/profiles/table/index.tsxapps/start/src/routes/_app.$organizationId.$projectId.profiles._tabs.identified.tsxapps/start/src/routes/_app.$organizationId.$projectId.profiles._tabs.tsxpackages/db/src/services/profile.service.tspackages/trpc/src/routers/profile.ts
…ingle date control - Sortable "Last seen" (created_at) column header — click to flip asc/desc (default desc). URL state via useProfilesSort. - Optional "Last seen" window (hour-precise) reusing the app's calendar date-range popover with an opt-in withTime flag (From/To time inputs). Null by default so nothing is filtered until set; Clear resets to all-time. - Single date control: removed the Today/Yesterday/7D/30D/3M/Custom segmented picker; the Last-seen window now bounds both the profile list (created_at) and the behavioral event subquery. - Timezone fix: the picker gives a naive wall-clock the user reads in the org timezone. Interpret it there via toDateTime(str, tz) (tz from getOrganizationByProjectIdCached) instead of new Date().toISOString(), which used the server TZ and shifted results. created_at stays raw so partition pruning holds. Perf (dashreels, 98.7M profiles): bounded tz-aware range ~0.4-1.4s, prunes to the month partition(s); unbounded ASC ~10s (full FINAL dedup) — the date range is the fast path. withTime defaults off, so the only other CustomDateRangePopover consumer (time-window-picker) is unchanged. Profiles page only.
Replace the last-seen window control with a dedicated Mixpanel-style picker (LastSeenPicker): - Fixed / Since mode sidebar. - Editable Starts/Ends datetime fields — free-type "Jun 4, 2026, 01:00 PM" (AM/PM guaranteed via hh:mm a regardless of locale), commit on blur/Enter, invalid reverts. Two-way synced with the calendar; changing the day keeps the typed time. - Dual-month calendar with range highlight. - "Enable time ranges" toggle: off = day-granular, on = hour-precise. Self-contained to profiles — reverted the withTime addition to the shared custom-date-range-popover (now byte-identical to origin/main); the only other consumer (time-window-picker) is unchanged.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/start/src/components/profiles/last-seen-picker.tsx`:
- Around line 59-71: Update the persisted date initialization and
synchronization in the picker component to parse startDate and endDate with
date-fns using the existing DB_FORMAT and a current-date reference, matching the
approach in last-seen-range.tsx. Replace both new Date(startDate) and new
Date(endDate) usages while preserving the existing undefined handling and mode
logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 092be05c-d41e-4af0-8200-b6e511986192
📒 Files selected for processing (7)
apps/start/src/components/profiles/last-seen-picker.tsxapps/start/src/components/profiles/last-seen-range.tsxapps/start/src/components/profiles/table/columns.tsxapps/start/src/hooks/use-profiles-sort.tsapps/start/src/routes/_app.$organizationId.$projectId.profiles._tabs.identified.tsxpackages/db/src/services/profile.service.tspackages/trpc/src/routers/profile.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- apps/start/src/components/profiles/table/columns.tsx
- packages/db/src/services/profile.service.ts
… operators Route the Profiles "who did event X where property=Y OP N times" filter to profile_event_property_summary_v2 for allowlisted projects (~350ms vs multi- second raw events scan). Non-allowlisted projects fall back to raw events. - Two-step: rank in v2 by maxMerge(last_event_time), hydrate the 50-row page via getProfiles (no FINAL — 2.1s -> 190ms). - Exact-window clamp on last_event_time so "Last seen" is strictly within the picked window (fixes the UTC-day 05:29 edge artefact). - UTC-day alignment on the event_date bound (fixes ~29% undercount at the window's trailing UTC day). - "did event OP N times": eq/ne/gt/gte/lt/lte/between/notBetween on countMerge (v2) / count() (fallback), list + count consistent. - Guard: v2 only for exactly one properties.* filter (name-only and multi-key route to events). OR across events/values on the fast path. - Page shows all profiles (anon + identified) so base and behavioural count share one population. - UI: "Profiles who did" label, operator control, 15-day default window, exact-timestamp Last seen. - CLICKHOUSE_MAX_EXECUTION_TIME cap (default 60s) on the app CH client. Env: PROFILES_BEHAVIORAL_V2_PROJECTS, PROFILES_BEHAVIORAL_V2_START_DATE, CLICKHOUSE_MAX_EXECUTION_TIME. Doc: docs/profiles-v2-behavioral-filter.md.
…date parse Review fixes on the profiles behavioural filter: - SECURITY (critical): escape `search` in the fuzzy ILIKE clause (list + count) with sqlstring.escape — was interpolated raw, allowing SQL injection. - SECURITY (major): reject non-identifier bare event-filter names in eventFilterClauses before interpolating (column names can't be string-escaped). - search + behavioural now applies the exact-window clamp + count threshold via buildBehavioralV2Subquery, so results match the count (was silently dropped). - last-seen-picker: parse persisted yyyy-MM-dd HH:mm:ss with date-fns instead of new Date() (Safari returned Invalid Date, leaving the picker empty). - drop the unrelated CLICKHOUSE_MAX_EXECUTION_TIME cap (separate concern).
A name/email/id search must find a profile regardless of when it was last seen, but the created_at window (incl. the 15-day default) was still applied during search — so searching only matched profiles seen in that window. Skip the window whenever a search term is present, in both the list and the count (so they stay consistent). Behavioural time bounds are unaffected.
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
docs/profiles-v2-behavioral-filter.md (1)
77-79: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd a language to the fenced block (markdownlint MD040).
-``` +```sql HAVING maxMerge(last_event_time) BETWEEN <startInstant> AND <endInstant><details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@docs/profiles-v2-behavioral-filter.mdaround lines 77 - 79, Update the
fenced SQL example containing the HAVING maxMerge(last_event_time) query to
specify the sql language identifier after the opening fence, preserving the
query content unchanged.</details> <!-- cr-comment:v1:c3d449d92d369a840bc5faa9 --> _Source: Linters/SAST tools_ </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.Inline comments:
In @.env.example:
- Around line 55-63: Update the behavioral filter v2 routing description in
.env.example to state that exactly one properties.* filter is required, not at
most one, and align the reported latency with the ~350 ms figure documented in
docs/profiles-v2-behavioral-filter.md and the implementation comment.In
@apps/start/src/hooks/use-profiles-sort.ts:
- Around line 58-65: Constrain the countVal and countVal2 query-state values to
non-negative integers instead of using unrestricted parseAsInteger. Update the
useQueryState definitions (or normalize both operands before constructing
eventCount) so negative URL values cannot produce negative
eventCount.value/value2 or bypass the minimum event-count requirement.In
@packages/db/src/services/profile.service.ts:
- Around line 469-513: Propagate the explicit last-seen window through all
behavioral list and count paths instead of applying it only in fallback
handling. Update getBehavioralV2ProfileList and its hydration query to accept
and enforce lastSeenStart/lastSeenEnd, and update getProfileListCount’s
behavioral branches to use the same window predicates; ensure allowlisted and
fallback results and counts remain consistent.- Around line 864-917: Update the behavioral count branch around
canRouteBehavioralToV2 and the events fallback to apply the same search-based
id/fuzzy predicate used by the list before counting profiles. Ensure both the V2
and events SQL paths AND this predicate with their existing behavioral
conditions, grouping, and thresholds so counts match filtered rows.- Around line 295-310: Validate PROFILES_BEHAVIORAL_V2_START_DATE immediately
after construction and make canRouteBehavioralToV2 return false when the
configured date is invalid. Ensure malformed environment values fail closed
rather than allowing the queryStart coverage comparison to pass.
Nitpick comments:
In@docs/profiles-v2-behavioral-filter.md:
- Around line 77-79: Update the fenced SQL example containing the HAVING
maxMerge(last_event_time) query to specify the sql language identifier after the
opening fence, preserving the query content unchanged.</details> <details> <summary>🪄 Autofix (Beta)</summary> Fix all unresolved CodeRabbit comments on this PR: - [ ] <!-- {"checkboxId": "4b0d0e0a-96d7-4f10-b296-3a18ea78f0b9"} --> Push a commit to this branch (recommended) - [ ] <!-- {"checkboxId": "ff5b1114-7d8c-49e6-8ac1-43f82af23a33"} --> Create a new PR with the fixes </details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: defaults **Review profile**: CHILL **Plan**: Pro Plus **Run ID**: `55a1ccde-6575-46be-ab05-ba1ef9770e12` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between e84542224f40eb86b354a56a1d93275ce2c98283 and 23e838f1c8d3edf5fb015f7f7164af0493f2f7c0. </details> <details> <summary>📒 Files selected for processing (13)</summary> * `.env.example` * `apps/start/src/components/column-created-at.tsx` * `apps/start/src/components/events/filters/events-filters.tsx` * `apps/start/src/components/profiles/event-count-filter.tsx` * `apps/start/src/components/profiles/last-seen-picker.tsx` * `apps/start/src/components/profiles/last-seen-range.tsx` * `apps/start/src/components/profiles/table/columns.tsx` * `apps/start/src/hooks/use-profiles-sort.ts` * `apps/start/src/routes/_app.$organizationId.$projectId.profiles._tabs.identified.tsx` * `docs/profiles-v2-behavioral-filter.md` * `packages/db/src/clickhouse/client.ts` * `packages/db/src/services/profile.service.ts` * `packages/trpc/src/routers/profile.ts` </details> <details> <summary>🚧 Files skipped from review as they are similar to previous changes (5)</summary> * apps/start/src/components/profiles/last-seen-range.tsx * apps/start/src/routes/_app.$organizationId.$projectId.profiles._tabs.identified.tsx * packages/trpc/src/routers/profile.ts * apps/start/src/components/profiles/last-seen-picker.tsx * apps/start/src/components/profiles/table/columns.tsx </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
…t parity Addresses the cloud + CodeRabbit review findings on the profiles filter: - All 15 filter operators (doesNotContain/startsWith/endsWith/regex/gt/lt/ gte/lte/isNull/isNotNull/…) via a shared operatorClause, in the v2, events and profile-property builders. Was: everything unhandled fell through to IN (e.g. doesNotContain returned the opposite audience). - profile.* filters (profile.id / profile.properties.*) now apply to the outer profiles query instead of being dropped — "did event X AND profile.id=Y" now actually filters by id (was returning all profiles). - Count/list parity: behavioural list no longer gates profiles.created_at (first-seen) by the window — the event subquery bounds time — so the count and rows agree (was 152 vs 8). Count now mirrors the list (behavioural membership + profile.* filters + search + window) via a shared subquery. - Pagination: offset = cursor (client already sends the row offset); removes the (page-1)*take*take that made page 2+ empty. - START_DATE guard: validate/anchor to UTC so a blank/garbled env falls back instead of silently disabling the coverage gate; parse queryStart as UTC too. - Nits: count input is a plain typed field (no spinner, no decimals → no 400); LastSeenPicker restores the time toggle on reopen; drop unused imports; fix a stale transformProfile comment.
…me + faster The events-fallback behavioural list sorted `profiles FINAL` by profiles.created_at (profile-record write time) — so "Last seen" showed stale record dates (e.g. Dec 2025) instead of when the profile did the event, and it fed the whole doer id-set into `profiles FINAL` (~20s on high-volume events). Give it the same two-step as v2: rank profiles by max(created_at) of the event INSIDE `events`, LIMIT one page, then hydrate. Now: - "Last seen" = the real last-event time on every no-search/no-profile-filter behavioural path (v2 or events), bounded exactly to the window. - No huge id-set into profiles FINAL — ranks+limits in events, hydrates 50.
…env doc - eventFilterClauses / profileAttrFilterClauses: allowlist bare column names (EVENT_FILTER_COLUMNS / PROFILE_FILTER_COLUMNS) instead of a loose identifier regex — blocks injection AND referencing arbitrary columns. - Clamp countVal/countVal2 to non-negative ints in the route so a hand-edited URL (?countVal=-1) can't 400 on the tRPC min(0) schema. - Fix stale .env.example guard doc: EXACTLY ONE properties.* filter, ~350ms.
… is picked
- No event ("All Events") → window defaults to ALL TIME (don't hide older
profiles on the plain list).
- Pick an event (e.g. showOpen) → auto-defaults to the last 15 days (a
"who did X" question is naturally time-bound); user can still change it.
- An explicit range always wins.
- Date control moved inline into the condition card, right next to the event +
count filter (Mixpanel-style); empty label reflects the effective default
("All time" vs "Last 15 days").
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Profiles-page overhaul for large projects (dashreels ≈ 80M profiles). Perf, Mixpanel-style behavioral/property filters, and search fixes.
Performance
getProfileListprunes to the recent monthly partition(s) +do_not_merge_across_partitions_select_final. Falls back to the full range only if the window doesn't fill the page.ILIKEis the fallback scan.uniq(id)(approx distinct users) instead of counting every ReplacingMergeTree version.Filters (v1, Mixpanel-style)
eventstable. Raw events (notprofile_event_summary_mv) is deliberate: the MV filtersprofile_id != device_idand would undercount anon/pre-identify users.filtersapply as profile-property filters on the profiles table.['*']"All Events" wildcard is stripped → means "no filter" instead of an event literally named*(which returned 0 profiles).Count
uniq(id)).Columns / UI
Notes / follow-ups
device_id/profile_id) resolution lands.🤖 Generated with Claude Code
Summary by CodeRabbit