perf: cache, compress, and precompute API responses

[ashwinn-si]

Summary

The dataset here is generated once at boot from fixed seeds, so a given URL returns identical bytes for the life of a deployment. Nothing communicated that to clients or a CDN, and every response went out uncompressed — so each request paid full origin cost and full transfer cost for data that never changes.

This PR makes responses cacheable and compressed, moves per-request work to boot time, and removes redundant work from the filter loops. No response body changes.

Changes

Caching and compression

• Cache-Control on the API (s-maxage=86400) and on the HTML pages, with an ETag derived from the app version so replicas agree on validators and a restart doesn't invalidate CDN entries. Setting the ETag up front also skips express's per-response body hash and lets revalidation return a bodyless 304.
• Those headers are stripped at writeHead time on any non-200, so a 500 can never get pinned in a shared cache.
• compression middleware. API payloads shrink 75–88%, the HTML pages 99%.

Work moved off the request path

• /docs was calling fs.readFileSync + yaml.load on a 53 KB YAML file on every request. Parsed once at boot.
• /spec re-ran JSON.stringify(spec, null, 2) per render. Serialized once at boot.
• Both payloads are now embedded compact instead of pretty-printed, with < escaped so a value can't close the <script> tag early.
• express.static moved below the routers — every API request was doing a filesystem lookup against src/public/ on its way to a handler.
• PostHog was configured with flushAt: 1, flushInterval: 0, meaning one outbound HTTPS request per captured event. Now batched; the existing shutdown handler flushes the queue.

Filter loops

• Every list endpoint re-derived its filter values inside the .filter() callback. ?created_gte=2024-01-01 parsed the same date string 200 times per request; ?city=lax re-ran toUpperCase() per row. Normalized once before the scan, and date filters now compare numeric timestamps instead of allocating a Date per comparison.

Latent bugs fixed along the way

• js-yaml was required in src/index.js but never declared as a dependency — it resolved only by hoisting out of swagger-jsdoc's tree, so any dependency bump would have broken /docs in production.
• The Dockerfile HEALTHCHECK probes /health, which did not exist. Containers were permanently unhealthy. Added the route.
• The swagger glob resolved from the working directory rather than from the source file, so the spec built empty if the process started from anywhere but the repo root.

Verification

Captured responses for 24 filter and pagination combinations from the current main, then replayed the same requests against this branch. Byte-identical across all 24 — covering every domain, every filter type (string, numeric, date-range, array, full-text search), and both single and combined filters.

Also confirmed: all routes return 200, /spec still emits all 36 paths after the glob change, both embedded JSON payloads parse, and a genuine 500 gets Cache-Control: no-store while its 200 siblings stay cacheable.

Measurements

Localhost, keep-alive, 300 requests per path:

Path	Before	After
/docs	1.477 ms	0.388 ms	3.8×
/spec	0.847 ms	0.461 ms	1.8×
/hotels?per_page=50 (uncompressed)	0.199 ms	0.174 ms	1.1×
revalidation (304)	—	0.118 ms	0 B body

On compression, stated plainly: localhost is the worst case for it. Serving /hotels?per_page=50 gzipped costs 0.100 ms of CPU and measures slower over loopback (0.199 → 0.274 ms) because there's no network to save. What it buys is 32.6 KB per response — 2.6 ms of transfer on 100 Mbps, 26 ms on 4G, 130 ms on 3G. For a public API that's a 26×–1300× net win, but it is a real trade and worth naming rather than burying.

Payload sizes:

Response	Before	After (gzip)
/hotels?per_page=50	37.2 KB	4.5 KB	−88%
/billing/invoices (filtered)	13.1 KB	3.2 KB	−75%
/spec	65.3 KB	0.5 KB	−99%
/docs	81.2 KB	0.5 KB	−99%

The larger point is the caching, which these numbers don't capture: with Cache-Control set, repeat traffic is served by the CDN and never reaches origin at all.

Notes for reviewers

Two things worth a deliberate decision rather than a rubber stamp:

1. s-maxage=86400 means a deploy takes up to 24h to fully propagate through a CDN unless you purge on release. Lower it if you'd rather not add a purge step.
2. The ETag is keyed on the package.json version. Bump it on any change to the generated data, or warm clients will keep serving stale bodies from cache. If that's too easy to forget, an alternative is hashing the generated datasets at boot — costs a little startup time and gives up cross-replica ETag stability.

Out of scope

/billing/payments returns 500 on every request — a null dereference in generatePayment, likely from the lazy require used to break the payments/invoices circular dependency. This reproduces on main and is untouched here; it's what surfaced the error-caching guard during testing. Worth a separate PR.

[ashwinn-si]

@helloanoop @Its-treason @dcoomber @sid-bruno