Sync upstream #393 (session management) into develop

[mraj602-tohands]

What

Syncs upstream #393 — new session management into develop. Upstream moved session lifecycle out of the BullMQ delayed-job mechanism and into a sessionBuffer.ingest() abstraction in packages/db. This PR adopts that engine and re-grafts the fork's offline-first behavior on top.

Heads-up on diff size: this is an upstream merge, so the file count is large (~95 upstream files + 9 fork-side files). Review focus = the 9 fork-side files listed below; the rest is upstream's own work coming in via the merge.

Decisions baked in

Decision	Choice
Session engine	Adopt sessionBuffer.ingest (retire the fork's createSessionEndJob/getActiveSessionEndJob mechanism)
Past / offline events	Reconstruct real sessions — upstream deliberately suppresses sessions for events > 15 min old; the fork's offline-first SDKs (Flutter / LVGL) upload genuinely-past sessions that must appear in analytics
"Past" threshold	15 min — use upstream's isTimestampFromThePast flag as the single definition

These (and re-apply instructions for future syncs) are documented in FORK-PATCHES.md.

Fork-side files to review

• apps/api/src/utils/ids.ts — upstream edge derivation (getExistingSession + withinIdleWindow + deterministic bucket) + fork isTimestampFromThePast param that skips the live-session lookup for historical events. Load-bearing: without it, withinIdleWindow is true for old timestamps and a backfill would join the device's live session.
• apps/api/src/controllers/track.controller.ts — getTimestamp now merges upstream's 15-min isTimestampFromThePast with the fork's 5-day hard floor (throws 400). Threads the flag + eventTimeMs into getDeviceId and the queue payload. Keeps the batch envelope, dispatchEvent, and replay BACKCOMPAT.
• apps/api/src/controllers/event.controller.ts — deprecated /event: upstream's typed queueData + Kafka dual-path; threads isTimestampFromThePast + eventTimeMs.
• apps/worker/src/jobs/events.incoming-event.ts — adopts upstream's sessionBuffer.ingest live path; re-grafts the historical reconstruction arm (one session_start per (project, sessionId) bucket, Redis-lock dedup, no live-state touched) and preserves the __syncedAt stamp.
• packages/queue/src/queues.ts — re-added isTimestampFromThePast to the incoming-event payload. ⚠️ The auto-merge silently dropped this field (no conflict marker); the worker's reconstruction gating depends on it.
• Tests: apps/api/src/utils/ids.test.ts, apps/api/src/controllers/track.controller.test.ts, apps/worker/src/jobs/events.incoming-events.test.ts (rewritten for the new mechanism + reconstruction coverage).

Testing

• Full monorepo typecheck: ✓
• Worker suite: 27/27 (incl. upstream's new session-reaper / session-vacuum / create-session-end crons)
• ids.test.ts 6/6 · track.controller.test.ts 13/13 · track-batch.router.test.ts 43/43 (batch end-to-end through the resolved controller)
• Pre-existing / infra (not this PR): bot-detection suite setup hooks time out locally (assertions still pass 23/23); insights.router (ClickHouse integration) not run in the sandbox.

Before deploy

• Run pnpm migrate — the merge brings 2 new Prisma migrations (codegen already run for client types).
• New session crons + sessionBuffer may need ClickHouse schema/MVs applied locally.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added session lifecycle automation with idle session reaping and cleanup.
  • Introduced global filters for reports to apply filtering across all event series.
  • New Amplitude import provider for analytics data migration.
  • Redesigned billing plan picker interface.

• Improvements

  • Enhanced session boundary detection and timeout handling.
  • Improved retention cohort calculations with weighted average summaries.
  • Refined bot detection filtering.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR overhauls session lifecycle ingestion and cleanup, adds report global filters and new retention outputs, centralizes subscription-state-driven billing and onboarding flows, introduces Amplitude imports, tightens bot handling for authenticated traffic, and adds operational scripts, docs, and end-to-end coverage.

Changes

Session lifecycle and event ingestion

Layer / File(s)	Summary
API timestamp and session contracts FORK-PATCHES.md, apps/api/src/controllers/..., apps/api/src/utils/ids.ts, packages/queue/..., packages/validation/src/track.validation.ts, packages/sdks/web/src/index.ts, apps/public/content/docs/self-hosting/environment-variables.mdx	Historical timestamp classification, replay session forwarding, session id resolution, and Kafka queue payload contracts are updated across API, queue, SDK, and docs.
Session buffer and storage model packages/db/src/buffers/..., packages/db/src/services/event.service.ts, packages/db/scripts/*sessions*.ts	Session persistence moves to ingest/cleanup-based buffer handling with Redis blob indices, collapsing ClickHouse rows, and migration and inspection scripts.
Worker session lifecycle and Kafka flow apps/worker/src/jobs/..., apps/worker/src/utils/session-handler.ts, apps/worker/src/metrics.ts, apps/worker/src/boot-*.ts, docker-compose.yml	Workers ingest live sessions through the buffer, emit deterministic historical session starts, run reaper and vacuum crons, enqueue v2 close jobs, and expose related metrics and Kafka tooling.
Session end-to-end coverage apps/api/e2e/*, apps/api/package.json	New E2E harnesses validate session lifecycle, replay, identify, draining, Redis cleanup, and ClickHouse reconciliation through runnable scripts.

Reporting filters and retention queries

Layer / File(s)	Summary
Report filter contracts and persistence packages/constants/index.ts, packages/validation/src/*, packages/db/prisma/*, apps/start/src/components/report/reportSlice.ts, packages/db/src/services/reports.service.ts, packages/db/src/engine/normalize.ts, packages/trpc/src/routers/report.ts	Reports now store globalFilters, typed filter metadata, and merged normalized series across shared contracts, persistence, and client state.
Filter editing and query wiring apps/start/src/components/report/..., apps/start/src/components/ui/dropdown-menu.tsx, apps/start/src/modals/view-chart-users.tsx, packages/db/src/services/*filter*.ts, packages/db/src/services/chart.service.ts, packages/db/src/services/funnel.service.ts, packages/db/src/services/conversion.service.ts	UI filter editors support global filters and typed operators, and backend query builders apply typed casts and merged global filters across charts, funnels, and profile views.
Retention engine and tool outputs packages/db/src/services/retention.service.ts, packages/trpc/src/routers/chart.ts, test/retention-fixtures.ts, packages/mcp/src/tools/analytics/retention.ts, apps/api/src/agents/tools/base.ts	Retention queries use a new cohort engine and weighted-average output, with updated fixtures, tests, router wiring, and tool descriptions.

Subscription state, billing UI, and onboarding

Layer / File(s)	Summary
Subscription state and backend sync apps/api/src/controllers/webhook.controller.ts, packages/payments/src/*, packages/db/src/prisma-client.ts, packages/db/src/services/organization.service.ts, packages/trpc/src/routers/subscription.ts, packages/db/scripts/set-subscription-state.ts	Canonical subscription-state logic now drives webhook sync, Prisma computed fields, billing counters, subscription updates, and state-testing scripts.
Billing screens and prompts apps/start/src/components/organization/*, apps/start/src/modals/select-billing-plan.tsx, apps/start/src/components/sidebar-organization-menu.tsx, apps/start/src/routes/_app.$organizationId*.tsx	Billing pages, prompts, plan picking, badges, and route banners are rebuilt around shared subscription-state metadata.
Onboarding emails and queue cleanup apps/worker/src/jobs/cron.onboarding*.ts, packages/email/src/emails/*, apps/worker/src/boot-workers.ts, apps/worker/src/index.ts, apps/api/src/utils/graceful-shutdown.ts, packages/queue/src/queues.ts	Onboarding emails now use usage-aware payloads and data-dependent copy, and remaining misc queue wiring is removed from workers, shutdown, and queue definitions.

Amplitude import support

Layer / File(s)	Summary
Import contracts and entrypoints packages/validation/src/index.ts, apps/start/src/modals/add-import.tsx, apps/worker/src/jobs/import.ts, packages/importer/src/index.ts, packages/importer/src/providers/metadata.ts	Amplitude is added to import validation, UI selection, worker provider creation, provider metadata, and exporter entrypoints.
Amplitude provider and shared file parsing packages/importer/src/providers/amplitude.ts, packages/importer/src/providers/country-codes.ts, packages/importer/src/providers/umami.ts	A new Amplitude provider streams remote NDJSON with compression detection, normalizes event fields, and shares safer remote parsing and country normalization helpers.
Importer fixtures and coverage packages/importer/src/providers/__fixtures__/amplitude-export.json, packages/importer/src/providers/amplitude.test.ts	Fixture-backed tests cover Amplitude parsing, transformation, streaming, fallback ids, and country-code handling.

Bot detection, authenticated traffic, and small utilities

Layer / File(s)	Summary
Bot list regeneration and matching apps/api/scripts/get-bots*.ts, apps/api/src/bots/*	Allowlisted runtime tokens are stripped from generated bot regexes, and bot detection is exported and tested directly.
Authenticated request bot bypass apps/api/src/app.ts, apps/api/src/hooks/is-bot.hook*.ts, apps/api/src/utils/auth.ts	Client-secret-authenticated requests are marked on Fastify requests and bypass bot detection and bot-event creation.
Duplicate-event inspection utility packages/db/package.json, packages/db/scripts/find-duplicate-events.ts	A new script reports duplicate logical events in ClickHouse and can submit batched delete mutations for extra rows.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• Tohands-Private-Limited/openpanel#5: Also changes historical event/session handling, timestamp classification, and deduplicated session_start behavior in the tracking pipeline.

Poem

🐇 I hopped through queues and charts all day,
with sessions tucked in Redis hay.
New filters bloom, old bots behave,
billing signs now gently wave.
Amplitude joins the burrow bright,
and metrics hum into the night.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/sync-393-sessions