Skip to content

CLAUDE.md

Latest commit

 

History

History
99 lines (64 loc) · 23.9 KB

CLAUDE.md

File metadata and controls

99 lines (64 loc) · 23.9 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

CCAS (Chess Club Admin System) is a Scala 3 application that pulls data from the Chess.com public API and stores it in a PostgreSQL database for chess club management tasks: tracking membership, analysing member performance, and scouting/recruiting players from other clubs. It includes a backend HTTP server with job scheduling for running these tasks.

Build & Test Commands

This is an SBT project (Scala 3.8.3, SBT 1.12.8).

  • Compile: sbt compile
  • Run all tests: sbt test
  • Run a single test suite: sbt "testOnly ccas.api.TestApiJsonParsing" (fully qualified object name)
  • Continuous compile on change: sbt ~compile
  • Interactive SBT shell: sbt then run commands without the sbt prefix

Tests use ZIO Test (ZIOSpecDefault). SQL tests require a running PostgreSQL instance with a ccas_test database (see src/test/resources/application.conf).

Architecture

Layers

The codebase has four main packages:

  1. ccas.api — Chess.com API models and client. Case classes model API JSON responses (e.g., ApiPlayer, ApiClub, ApiDailyMatch). Each API model companion object extends JsonDecoding[T] to provide a ZIO JSON decoder. API models are read-only data transfer objects; they are never written to the database directly.

  2. ccas.analysis — Domain tables and business logic. analysis.tables contains database-persisted entities: core (Player, PlayerSnapshot, Club, ClubAdmin, ClubMember, ClubMatch, ClubMatchBoard, ClubMatchGame), ref resolution (ClubMatchRef, PlayerMatchRef, PlayerTournamentRef, ClubRefSkip, PlayerRefSkip, UnresolvedBoardPlayer, UnresolvedMatchClub), recruitment (RecruitmentCriteria, RecruitmentAlias, RecruitmentBlacklist, RecruitmentRun, RecruitmentCandidate, PlayerRecruitmentCache), history crawl (HistoryMemberQuery, HistoryPendingMatch, HistoryRun), run tracking (MembershipRun), app-wide settings (AppSetting), and API diagnostics / caching (ApiFetchFailure, ApiResponseBody, ApiResponseCache, ClientConfig, ClientStats). analysis.apps contains runnable applications (MembershipApp, RecruitmentApp, RecruitmentCriteriaApp, RefApp, HistoryApp, StatsApp, ClubDataApp) and shared helpers (PlayerUpdater, UsernameRenameResolver, ClubSlugRenameResolver). BlacklistApp is invoked from the CLI or synchronously from BlacklistRoutes REST endpoints; it is not a JobKind and does not go through JobRunner (blacklist mutations are small enough to handle inline rather than as async jobs). RecruitmentCriteriaApp follows the same sync pattern (invoked from CLI or RecruitmentCriteriaRoutes): subcommands are set <club-slug> <alias> [--json <file>], show <club-slug> <alias>, list <club-slug>, sample. A "set" is a versioned insert — criteria rows are immutable and recruitment_alias's PK is (club_id, alias, since), so the same op handles first-set and later-change; RecruitmentApp reads newest-wins via selectLatest. The app dedups unchanged re-submits (skipping the insert when the capped incoming criteria equals the latest stored), uses RecruitmentAlias.upsert so a same-instant re-set repoints the row instead of colliding on the composite PK, requires the club to already exist locally (no ChessComClient dependency), and emits a per-field diff log on every save. Interactive CLI prompts pre-fill from the existing alias (or RecruitmentCriteria.defaultDaily for a new alias), preview a diff, and require a Save? [Y/n] confirmation; the CriteriaSpec DTO (RecruitmentCriteria minus criteria_id) is the shared wire shape used by the --json file path and the HTTP body. ClubDataApp refreshes club profile data (admins, member count, slug, latest match activity) for all known clubs (or a specific subset when invoked with slug arguments on the CLI) and is not exposed as an HTTP route — it runs from CLI or scheduler only, as a data-integrity job.

  3. ccas.server — Backend HTTP server with job execution and scheduling. server.jobs has JobRunner (async job execution via forked fibers), JobRun/JobSchedule (database entities). server.routes has zio-http route handlers for jobs, schedules, and health checks. server.scheduler has JobScheduler (polling-based scheduled job execution). Entry point is CcasServer extends ZIOAppDefault.

  4. ccas.utils — Shared infrastructure: HTTP client (client/), JSON traits (json/), SQL client and helpers (sql/), opaque type utilities (opaque/), and pretty-printing (prettyprinting/).

Key Patterns

Table entity pattern with Magnum: Each database entity (e.g., Club, Player) follows a consistent structure:

  • A case class annotated with @Table(PostgresDbType, SqlNameMapper.CamelToSnakeCase) and derives DbCodec
  • Its companion object provides static methods: createTable, selectAll, selectId, insert, insertBatch, update, upsert, etc.
  • All SQL operations wrap Magnum queries in PostgresClient.connectZIO (reads) or PostgresClient.transactZIO (writes/batches)
  • Complex queries use SqlLiteral for reusable column lists and raw SQL interpolation (sql"""...""")
  • Some entities also use Magnum's Repo[T, T, ID] or ImmutableRepo[T, ID] for standard CRUD

Type safety with opaque types: Domain IDs and constrained values use Scala 3 opaque types with companion traits (StringCompanion, StringKeyCompanion, IntCompanion, LongCompanion, DoubleCompanion) defined in ccas.utils.opaque. Each companion provides JsonCodec, DbCodec, and DeriveConfig instances, plus optional validation via validateRaw and normalization via normalize. StringKeyCompanion extends StringCompanion with JsonFieldEncoder/JsonFieldDecoder for types used as JSON object keys. Concrete types live in two places: ccas.api.misc.subtypes for Chess.com domain IDs (PlayerId, ClubId, Username, ClubUrlName, Elo, Percentage, etc.) and ccas.analysis.tables.subtypes for internal DB surrogate IDs (ApiResponseBodyId, ApiResponseCacheId), the latter with a stricter > 0L validator since BIGSERIAL keys start at 1.

Enum serialization: Enums extend EnumJson[T] for JSON (snake_case wire format, PascalCase in Scala) and/or EnumSql[T] for database persistence. Both provide codec instances via given. Some enums override jsonToEnum for non-standard Chess.com API mappings (e.g., "closed:fair_play_violations" → Fairplay).

JSON decoding trait: API model companions extend JsonDecoding[T], which wraps JsonDecoder with convenience methods (decodeZIO, string extensions). The derived decoder is provided via jsonDecoderDerived. API models use @jsonMemberNames(SnakeCase) for field mapping. Global decoders for Instant and URL are exported automatically.

ZIO effect types: Helper functions connectZIO and transactZIO (in ccas.utils.sql.PostgresClient companion) provide the bridge between Magnum's context-function-based API and ZIO effects, returning ZIO[PostgresClient, SQLException, A]. For multi-statement atomic operations, withTransaction runs multiple connectZIO calls in a single JDBC transaction (commits on success, rolls back on any failure or interruption) by sharing a proxied connection via a scoped PostgresClient. All three functions retry automatically on transient connection errors (SQLState 08xxx).

Username and club-slug rename recovery: Chess.com permits handle changes; the old username/slug 404s while the player/club is now reachable under a new name. The resolvers UsernameRenameResolver and ClubSlugRenameResolver (ccas.analysis.apps) recover the current canonical name when a 404 fires on a previously-known handle. Both follow a tiered strategy:

  • Tier A is a pure DB lookup (Player.selectByUsername + PlayerSnapshot.selectLatestPlayerIdByUsername for players; Club.selectId for clubs) that succeeds when our DB has already learned the rename through some other path.
  • Tier B falls back to a board-endpoint trick (PlayerMatchRef.findOrInfer → ApiMatchBoard → eliminate the opposing side's known canonical name; or Club.slugFromMatchRef → match team URL).
  • Tier C (ClubSlugRenameResolver only) closes the gap for clubs that have never played a match — where Tier B is a no-op. It loads stored ClubAdmin rows for the clubIdHint, fetches each non-tombstoned admin's /pub/player/{u}/clubs, filters out slugs we already know (those would have surfaced via Tier A) using Club.selectExistingSlugs, and returns the first slug whose verified ApiClub.clubId matches the hint. Bounded by the admin count and short-circuits on first hit; the typical ClubDataApp.refreshClub callsite passes Some(club.clubId) so Tier C is reachable.

Tier resolution uses ZIO.collectFirst so each tier executes only when prior tiers return None. A verification fetch confirms playerId / clubId matches the hint before returning. Players/clubs whose canonical name can't be discovered (no match refs, no snapshot history, no admin signal) are tombstoned with a sentinel _stale_<id> (set by PlayerUpdater.archiveAndUpdate for recycled-handle cases, or Club.resolveStaleSlug for fresh slug discovery failures) so the UNIQUE constraint slot is freed; tombstoned rows are skipped at iteration sites (e.g. HistorySeeding.seedFromMemberMatches) and rendered as <unknown player #<id>> / <unknown club #<id>> in user-facing output via Player.displayUsername / Club.displayName. The collision-safety of the _stale_<id> format is tracked in Sootopolis/ccas#21. Callers wire recovery via the withPlayerRenameRecovery and withClubSlugRenameRecovery extension methods on any 404-prone effect, or directly via UsernameRenameResolver.fetchOrRecover for the common "fetch ApiPlayer with rename fallback" pattern. Resolver internals never replace the caller's original 404 with a recovery-internal error — Tier A SQL exceptions and Tier B/C HTTP errors are debug-logged and silently fall through to "no rename inferred."

HTTP Client

ChessComClient wraps a custom zio-http Client layer (HttpClientLayer.live in ccas.utils.client) for making GET requests with automatic JSON decoding. All CCAS apps and CcasServer provide HttpClientLayer.live rather than Client.default so transport-level settings (gzip, connection-pool shape, future HTTP/2) live in one place. Features include:

  • Gate-based admission control with configurable concurrency limit (default 8 permits)
  • Adaptive rate limiting: EMA-based request spacing with configurable minimum delay floor, failure-window throttle-down on 429 responses, time-gated recovery tiers
  • Cloudflare challenge detection: immediate hard throttle to 1 permit on CF 403, independent of the failure window
  • Per-attempt failure recording in api_fetch_failure with deduplicated response bodies in api_response_body
  • Separate retry schedules for 429 (exponential backoff), Cloudflare 403 (fixed delay), and connection errors (exponential backoff); non-Cloudflare 403 and 404 are never retried
  • Requires CCAS_CONTACT_EMAIL environment variable for the User-Agent header
  • Accept-Encoding: gzip on every request; HttpClientLayer configures Decompression.NonStrict so Netty's HttpContentDecompressor decodes compressed bodies transparently. HTTP/2 is not yet available in zio-http 3.10.1 (tracked upstream at zio/zio-http#3473); HttpClientLayer is the single point of upgrade when it lands.
  • Batch fetching via getAll[T](urls) using ZIO.foreachPar capped at maxPermits — network-bound fibers already bottleneck at the gate, and the cap bounds cache-warm fan-outs against the Hikari connection pool.
  • The followRedirects(3) aspect's error handler returns 304 responses as-is (rather than failing on the missing Location header) so the cache's conditional-GET revalidation path works.

Response caching (getCacheable): Every successful fetch is persisted to api_response_cache (keyed by URL, with ETag / Last-Modified / Cache-Control: max-age / Content-Type metadata) and its body to api_response_body (SHA-256-deduped, so byte-identical responses across URLs share one row). ChessComClient.getCacheable[T](url) returns a CacheableResult[T] (ccas.utils.client.CacheableResult) with four variants:

  • Fresh — served from cache with no network call; entry was within Cache-Control: max-age.
  • Revalidated — conditional GET (If-None-Match / If-Modified-Since) returned 304 Not Modified; fetched_at is refreshed.
  • IdenticalBody — 200 OK with a byte-identical body (SHA-256 dedup kept the same body_id).
  • Changed — first fetch or actual content change; the new body replaces the cache row.

Each variant carries a lazy getValue: Task[T] so callers can branch on isUnchanged and skip downstream processing without triggering a body load or JSON decode. get[T] is a thin wrapper returning T via getCacheable[T](url).flatMap(_.getValue). CacheableResult exposes foldZIO(ifUnchanged)(ifChanged) and unlessUnchangedDiscard(zio) for branching without hand-rolling a pattern match; dispatched via final overrides on a sealed Unchanged[T] intermediate supertype over the three hit variants, so T is compiler-verified end-to-end (no erased @unchecked match). Current callers: HistoryProcessing.refreshSingleMatch (Unchanged → bump fetched_at; Changed → full rework) and HistorySeeding.seed{FromClubMatches,MatchesForPlayer,MatchesForPlayerAllClubs} (Unchanged → skip the INSERT pipeline but still stamp HistoryMemberQuery where required). Cache-Control: no-store responses are not cached; Cache-Control: no-cache is honoured by persisting the entry with a cleared max_age_seconds so every subsequent request revalidates (RFC 7234 §5.2.2.2). ETag values are stored in wire format ("..." / W/"...") and echoed back via Header.Custom("If-None-Match", …) to work around a quote-stripping bug in zio-http 3.10.1's Header.IfNoneMatch.ETags.render. Last-Modified is read via a raw-header lookup piped through ccas.utils.HttpDate.parse because Chess.com ships a non-RFC format (Thursday, 16-Apr-2026 23:13:22 GMT+0000, matching none of the three forms in RFC 7231 §7.1.1.1) that zio-http's typed Header.LastModified silently rejects; the parser tries Chess.com's shape first and falls back to IMF-fixdate / RFC 850 / asctime, and If-Modified-Since is always echoed in IMF-fixdate regardless of what was received. Empirically (2026-04-17) Chess.com's origin ignores If-Modified-Since regardless of format, so the conditional-GET path is in practice ETag-only. On a 304, ApiResponseCache.touch merges any refreshed Cache-Control / ETag / Last-Modified / Content-Type values from the 304 response using COALESCE for the validators (absent headers preserve stored values) and a MaxAgeUpdate ADT (Preserve / Clear / Overwrite(n)) for max-age covering the three wire-level distinctions (header absent → Preserve; no-cache → Clear; max-age=n → Overwrite(n)). Cache hits, 304 revalidations, and cache misses have dedicated counters on ClientStatsAccumulator (persisted to client_stats.cache_hits / cache_revalidations / cache_misses) so the requests series stays an honest indicator of Chess.com API load. The three HistoryApp skip sites also record per-run domain-level counters on history_run (refresh_match_unchanged / seed_club_matches_unchanged / seed_player_matches_unchanged) so the short-circuits' impact on downstream DB / decode work is visible separately from transport-level cache savings.

Cache retention: Tables.ensureTables calls ApiResponseCache.deleteBefore(now - retention) on every app startup, chained with ApiResponseBody.deleteOrphans in the same transaction. The window is the cache_retention_days setting read from the app_setting table via AppSetting.get(AppSetting.CacheRetentionDays) — a DB-owned value (the only tunable source) with a compiled-in default of 7 days used when the row is absent (fresh DB) or unparseable. There is no HOCON/env mirror; change it with SQL (INSERT … ON CONFLICT (key) DO UPDATE), effective on each process's next startup. In tests the startup sweep runs against a freshly-created empty cache, so the default never deletes mocked-old fixtures (those are inserted by tests after ensureTables). app_setting (key TEXT PK, value TEXT) is a generic single-row-per-key store for DB-owned app-wide policy — runtime-tunable without a redeploy and consistent across every process on one DB — kept orthogonal to client_config (per-process ChessComClient tuning). Typed access goes through AppSetting.Key[A] and the companion registry (AppSetting.CacheRetentionDays, AppSetting.all) — each key carries its default + string codec — so the stringly-typed table is confined to one place; AppSetting.all lists the keys for future discoverability. New keys reuse the same shape with no schema churn. No CLI/route surface yet — change values via SQL or a future admin endpoint. Mid-flight races are tolerated: a Fresh / Revalidated result whose body was pruned by another process falls through to a recursive network refetch via loadAndDecode's None-branch / JsonDecodingException recovery paths.

Deployment model: one CcasServer per database is the supported model. A CLI invocation alongside the server on the same DB is fine — the CLI builds no JobRunner, so it never mutates scheduled-job state; it only shares the Chess.com egress and runs the same idempotent ensureTables (incl. the cache sweep, hence cache_retention_days being DB-owned so server and CLI agree). Running two or more servers against one DB is unsupported / at-your-own-risk: JobRun.markOrphansAsFailed on startup fails all Running jobs with no instance-ownership filter (a second server kills the first's live jobs — #110), and the rate limiter is per-process (N servers ≈ N× the per-IP rate — #111). Scheduler double-fire is prevented (the idx_job_run_running_unique partial index + FOR UPDATE in JobRunner.submit reject the duplicate insert), and all seeders are idempotent (IF NOT EXISTS / WHERE NOT EXISTS / ON CONFLICT DO NOTHING). Multi-server hosting is tracked under #60 and gated on fixing #110 + #111.

Backend Server

CcasServer (ccas.server) is a zio-http server that exposes REST endpoints and runs background jobs:

  • Routes: HealthRoutes (health/readiness), JobRoutes (submit, query, and cancel jobs — POST /api/jobs/{jobId}/cancel interrupts a running job's fiber in this process (200 CancelResult, or 404 when no live job fiber exists here: unknown id, already-terminal, or owned by another instance), plus recruitment-result delivery: GET /api/jobs/{jobId}/recruitment/invited for a completed run's paste-ready invited usernames, GET .../recruitment/found for still-Deferred candidates, the mutating POST .../recruitment/confirm which flips that run's Deferred candidates to Invited in one transaction and returns ConfirmResult{marked, usernames}, and GET /api/recruitment/clubs/{slug}/latest/invited + GET /api/recruitment/runs/{runId}/invited for reporting a past run — these drive the ccas recruit --stdout / interactive-confirm / --report delivery modes, where an interactive scout sends autoConfirm=false so candidates stay Deferred until the operator confirms), ScheduleRoutes (CRUD for scheduled jobs), BlacklistRoutes (synchronous CRUD for RecruitmentBlacklist entries — delegates to BlacklistApp directly without going through JobRunner), RecruitmentCriteriaRoutes (synchronous POST /api/recruitment-criteria to set criteria, GET .../{slug}/{alias} to show, GET .../{slug} to list aliases — delegates to RecruitmentCriteriaApp). Route handlers use inline JSON codecs for request/response types. User-facing errors (BadRequestException, NotFoundException, ConflictException) form a sealed UserFacingError trait (ccas.utils.errors) that carries the HTTP status and a pre-encoded JSON body; RouteHelpers.withErrorHandling renders them uniformly (default body shape is {"error": "<message>"}, but .of[B: JsonEncoder](body, msg) companion constructors support structured payloads). Escaping HttpStatusExceptions from the Chess.com client render as 502. Anything else — including defects — collapses to a generic 500 with the full cause logged via ZIO.logErrorCause so operational visibility isn't lost; pure-interrupt causes are re-propagated so shutdown / client-disconnect noise stays out of the error log.
  • JobRunner: Trait-based (JobRunnerLive) async executor that forks fibers per job, tracks state in JobRun table (ULID IDs via ulid-creator). Marks orphaned running jobs as failed on startup. The submit method takes an Option[JobRunId] => RIO[..., Any] effect function, passing the generated job run ID so analysis apps can link their run records back via job_run_id. Job kinds: Recruitment, Membership, MatchRef, History, Stats, ClubData (the latter has no HTTP route — scheduler only). JobRunStatus is Running | Completed | Failed | Cancelled. cancel(id) interrupts the job's forked fiber (handles retained per-process in a runningFibers map registered before the fork so de-register can't race the register): it registers the id in a cancelRequested set then interruptForks the fiber, and the job's own .onInterrupt records Cancelled only if its id is in that set. This gates operator cancellation apart from the layerScope interrupt fired at every in-flight job on server shutdown — those are left Running so the next boot's markOrphansAsFailed records them as Failed/"Service restarted" (which also still covers hard crashes / SIGKILL, where finalizers never run). Cancellation is best-effort/asynchronous — an in-flight uninterruptible blocking JDBC statement runs to completion before the interrupt lands (markCancelled's WHERE status = Running guard makes the terminal write a no-op if the job reached Completed/Failed first). All completed_at writers (updateStatus, markCancelled, markOrphansAsFailed) take an Instant from the caller's Clock.instant rather than SQL NOW(), so every job timestamp comes from the one testable app clock, coherent with started_at. Being per-process, cancel inherits the single-server-per-DB assumption (#110/#111). The CLI surface is ccas cancel <job-id>; a job cancelled out from under an active ccas logs follow renders "was cancelled" and exits non-zero.
  • JobScheduler: Polling daemon (configurable interval) that checks enabled JobSchedule entries and submits due jobs to the JobRunner.
  • ServerTables: Ensures both analysis and server database tables exist on startup.

Database

Uses Magnum (com.augustnagro.magnum) for SQL access with PostgreSQL. PostgresClient (ccas.utils.sql) wraps a Magnum Transactor backed by HikariCP and adds connection-pool hardening (keepalive probes, validation queries, lazy initialization) and transient-error retry (exponential backoff on SQLState 08xxx). PostgresClient.live reads config from application.conf under the database prefix and provides a PostgresClient ZLayer; all app and server code depends on PostgresClient rather than Transactor directly. Custom DbCodec instances handle Instant (via TIMESTAMPTZ), URL, and List[String] (PostgreSQL arrays). Table names are derived from case class names via CamelToSnakeCase naming strategy. Server tables (JobRun, JobSchedule) reference clubs by club_id FK; route handlers resolve the slug from HTTP requests to a ClubId before submitting jobs. Analysis run tables (MembershipRun, RecruitmentRun, HistoryRun) have an optional job_run_id column linking back to the server-level job. Schema migrations for existing databases are managed via manual SQL scripts in the sql/ directory.

Test Data

API JSON test fixtures live in data/test/api/*.json. Tests in TestApiJsonParsing validate that these parse correctly into API model types.