feat(packets): full-history archive browser + month-partition the mqtt_messages table

[SimmerV]

Overview

Makes the raw-packet archive (mqtt_messages) both queryable for users and
operationally scalable. Adds a full-history packet browser backed by keyset
pagination, and converts mqtt_messages (98%+ of the DB) to a month-RANGE
partitioned, lz4-compressed table so date-range queries prune to the relevant
months and maintenance runs at partition scale.

What's included

Packet archive browser

• GET /v1/packets: keyset/cursor pagination with absolute start/end
  (epoch) date-range filtering + q/topic filters; GET /v1/packets/{id}
  for per-packet deeplinks. Response: {messages, next_cursor}.
• Rewritten Logs page: infinite scroll, datetime-range pickers, quick-range
  pills, deeplinked single-packet banner, URL↔state sync.

mqtt_messages partitioning + lz4

• Fresh installs get a month-partitioned table (composite PK (id, created_at),
  payload lz4-compressed) via a guarded block in schema.sql.
• ensure_mqtt_partitions() rolls current + next 2 months' partitions forward
  (at startup and hourly from the renamed maintenance_loop).
• scripts/migrate-mqtt-partitioning.sh: one-time in-place conversion for
  existing DBs — single atomic transaction (rename → create → copy → verify row
  count → rebuild indexes → re-home sequence → reinstall trigger), with a 3×
  disk pre-flight and the original kept as mqtt_messages_old for rollback.

Enrichment / fixes

• New Meshview bulk-fetch enrichment provider (one GET, filtered locally);
  drops the broken bayme preset (legacy configs warn + skip).
• Fix: serialize datetime/date values in JSONB writes (previously raised and
  dropped the write).
• Env-overridable Postgres memory knobs in both compose files (safe small-host
  defaults).

Migration / ops notes

• Existing DBs: run bash scripts/migrate-mqtt-partitioning.sh once (stack
  up). Idempotent; transactional; keeps mqtt_messages_old until you drop it.
• Requires lz4: Postgres must be built with lz4 (PG 14+; the postgres:18
  image provides it).
• No data loss: mark_nodes_inactive_by_age ("prune") only flips an active
  flag; partitioning copies + row-count-verifies before commit.

Testing / verification

• pytest: 114 passed.
• Validated against a live 26.1 M-row migrated DB: table partitioned, payload
  compression = lz4, partition pruning confirmed (Subplans Removed), 0 rows
  in default, migration integrity intact, rollback copy retained.
• API: cursor pagination (no overlap/gaps across pages, handles live inserts),
  /v1/packets/{id} deeplink + 404, date-range into historical months.
• Confirmed the relkind fix end-to-end: app auto-creates the current+2
  partition on startup.

[Copilot]

Pull request overview

This PR makes the mqtt_messages packet archive both user-queryable (full-history browser with keyset pagination + deeplinks) and operationally scalable (monthly range partitioning + lz4 compression), while also improving enrichment provider handling and Postgres ops documentation.

Changes:

• Added a new /v1/packets keyset-paginated API (plus /v1/packets/{id}) and rewired the frontend Logs page to browse full history with infinite scroll, URL↔state sync, and packet deeplinks.
• Converted mqtt_messages to a month-RANGE partitioned table on fresh installs and added an in-place migration script for existing DBs; added runtime partition roll-forward.
• Hardened JSONB writes against datetime-containing payloads and updated enrichment provider configuration (Meshview bulk support; dead legacy providers dropped).

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
storage/db/postgres.py	Adds JSON serialization fallback, cursor pagination helpers, partition roll-forward, and packet archive query/deeplink support.
api/api.py	Adds /v1/packets + /v1/packets/{id} and updates node packets response to include cursors.
frontend/src/pages/Log.tsx	Rebuilds Logs into a full-history packet browser with infinite scroll, filters, and deeplink banner.
frontend/src/slices/apiSlice.ts	Adds RTK Query endpoints/types for packet archive pagination and single-packet fetch.
postgres/sql/schema.sql	Creates partitioned+compressed mqtt_messages on fresh installs and keeps legacy installs untouched.
scripts/migrate-mqtt-partitioning.sh	Provides one-time transactional migration of existing mqtt_messages into monthly partitions.
main.py	Renames/extends prune loop into maintenance loop and adds periodic partition roll-forward.
data_store.py	Removes presets, adds dead-legacy provider handling, and adds Meshview bulk enrichment path.
tests/test_postgres_helpers.py	Adds unit tests for cursor codec, JSON default serialization, page building, and partition spec generation.
tests/test_enrichment_providers.py	Updates provider-resolution tests for dead legacy providers and Meshview provider kind.
POSTGRES.md	Documents memory tuning knobs and the mqtt_messages partitioning/migration workflow.
docker-compose.yml	Adds env-overridable Postgres memory tuning settings.
docker-compose-dev.yml	Adds env-overridable Postgres memory tuning settings for dev compose.
API.md	Documents keyset pagination and deeplinks for packets endpoints.
config.toml.sample	Updates enrichment provider examples to Meshview bulk + explicit URL templates.
config.py	Updates default enrich provider list to empty (no shipped presets).
.env.sample	Adds sample Postgres memory tuning env vars.
postgres/sql/meshinfo.sql	Removes obsolete SQL file.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.