Skip to content

feat(tui): add a Jobs section; fix task_size and db_uri passthrough - #29

Merged
justinrmiller merged 5 commits into
mainfrom
feat/tui-jobs-section
Jul 31, 2026
Merged

feat(tui): add a Jobs section; fix task_size and db_uri passthrough#29
justinrmiller merged 5 commits into
mainfrom
feat/tui-jobs-section

Conversation

@justinrmiller

Copy link
Copy Markdown
Collaborator

Adds a Jobs section to the TUI alongside Tables — and two silent config-passthrough bugs that surfaced while building it.

Jobs section

j lists recent jobs newest-first across every status, with a tally on the section header (Jobs — 1 RUNNING · 2 DONE). Selecting one shows the full record: status, target column, elapsed, a progress line, the launch config, every metric, and the complete event log — which, since geneva has no streaming log API, is the log. f follows a running job by re-reading the record every 3s and stops itself once the job goes terminal (the TUI equivalent of jobs tail).

0e2512d7b2034a1e97738b318ec169bb  DONE · video_clips.embedding · elapsed 0:00:10
progress: rows_committed 89/89 (100%)  tasks_completed 1/1 (100%)  fragments 1/1 (100%)

That progress line is curated on purpose: geneva attaches ~35 metrics to a backfill, most of them timers with a zero total. progress_summary() keeps only the ratio-shaped ones and prefers the three that read as progress. The full set stays in the scrollable detail below.

Record reading and formatting moved to core/jobs.py, so the jobs CLI and the TUI render identically and can't drift; ops/jobs.py sheds ~120 lines. The mode control also drops its auto entry — in an interactive app there's no command line recording what you picked, so the control should name the backend you're actually on.

fix: task_size was dropped on every local run

core/backfill.py kept task_size in the remote-only branch, on the premise that the local NativeTable API lacks it. That premise is false in geneva 0.14: NativeTable = Table is a plain alias, there's one backfill implementation, and task_size reaches run_ray_add_column(task_size=…) either way.

Verified by task count rather than by reading signatures — 40 rows, differing only in --task-size:

--task-size 8 --task-size 40
before 4 tasks 4 tasks
after 5 (⌈40/8⌉) 1 (⌈40/40⌉)

Before, both landed on geneva's default sizing; the flag did nothing. This affects every step going through backfill_column under --mode local — the laptop path, where an oversized task on a slow UDF is exactly what trips the stall watchdog. use_cpu_only_pool stays remote-only: it requests a CPU_ONLY_NODE Ray resource that a local cluster never advertises, so the tasks would sit unschedulable.

fix: a scheme-less db_uri silently became a local database

geneva reads any uri that isn't db://… as an on-disk database created relative to the working directory. So a bare --db-uri smoke gave you ./smoke/ while you believed you were on the cluster — the source of the stray database directories that were showing up as untracked files.

normalize_db_uri() now restores the scheme and logs it:

WARNING db_uri 'tts' has no scheme — using 'db://tts'. geneva would otherwise have
read it as an on-disk database and created ./tts here instead of connecting to the cluster.

Object-store uris (s3://, gs://, az://) and explicit filesystem paths pass through untouched, as does everything in local mode where db_uri is unused. The override was previously applied by assigning onto the returned Config at five separate call sites, each of which would have bypassed normalization — so --db-uri now resolves inside load_config() and no path can skip the check.

Also

  • .gitignore covers stray local database directories. Not by name — the name comes from config.yaml, which is itself gitignored — but by the structural markers every geneva database contains (__manifest/, …___system$geneva_jobs/). Verified no currently-tracked file becomes ignored.
  • The TUI's db_uri field is hidden outside enterprise mode, where geneva ignores it.

Testing

Coverage goes to 100% line and branch (from 97.06%), 305 tests passing, ty diagnostics unchanged at 22.

Beyond covering the new code, this adds regression guards for both fixes — task_size surviving the local backfill path, --db-uri reaching a generated CLI normalized, and all three ops CLIs forwarding db_uri_override (parametrized, so it can't silently regress in just one). It also closes five pre-existing partial branches in untouched modules, each a real edge rather than a number: even-division batching, no-jitter retry backoff, the runtime_session fallback for when geneva's private ray_cluster moves, skip_null_video=False, and deleting from an empty UDF library.

🤖 Generated with Claude Code

Adds a Jobs browser to the TUI alongside Tables, plus two silent
config-passthrough bugs found while building it.

Jobs section
- `j` lists recent jobs newest-first across every status, with a status
  tally on the section header. Selecting one shows the full record:
  status, target, elapsed, a curated progress line, launch config, every
  metric, and the complete event log (geneva has no streaming log API, so
  the record's append-only event list *is* the log).
- `f` follows a running job by re-reading the record every 3s, and stops
  itself once the job reaches a terminal state.
- Record reading/formatting moves to core/jobs.py so the `jobs` CLI and
  the TUI cannot drift; ops/jobs.py sheds ~120 lines.
- The mode control drops its `auto` entry: in an interactive app with no
  command line to record the choice, the control should name the backend
  you are actually on.

fix(backfill): forward task_size on local runs

core/backfill.py kept `task_size` in the remote-only branch, on the
premise that the local NativeTable API lacks it. In geneva 0.14
`NativeTable = Table` — a plain alias, one backfill implementation — and
task_size reaches `run_ray_add_column(task_size=...)` either way, so the
setting was silently dropped on every `--mode local` run. Verified with
40 rows: `--task-size 8` and `--task-size 40` both produced 4 tasks
before, and now produce 5 and 1. `use_cpu_only_pool` stays remote-only
(it requests a CPU_ONLY_NODE Ray resource local Ray never advertises).

fix(config): keep the db:// scheme on an enterprise db_uri

geneva reads any uri that is not `db://…` as an on-disk database created
relative to the working directory, so a scheme-less name silently gave
you ./<name>/ instead of the cluster — the source of stray database
directories at the repo root. normalize_db_uri() corrects it and logs at
WARNING; object-store uris and explicit paths pass through untouched.
`--db-uri` now resolves inside load_config() rather than by assignment at
five call sites, so no caller can bypass the check.

Also: gitignore stray local database directories by their structural
markers (the name comes from a gitignored config.yaml, so it cannot be
listed), and hide the TUI's db_uri field outside enterprise mode, where
geneva ignores it.

Coverage goes to 100% line and branch (from 97.06%), 305 tests. Beyond
the new code that includes regression guards for both fixes and five
pre-existing partial branches in untouched modules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
justinrmiller and others added 4 commits July 30, 2026 12:25
The comment described the flag as pinning appliers to the CPU-only node
pool without noting that it only ever applies to appliers requesting no
GPU, which reads as though it could keep a GPU UDF off the GPUs.

geneva branches on the UDF's num_gpus first and only reaches the
use_cpu_only_pool arm when that is zero (runners/ray/pipeline.py), and
the cpu-only resource is advertised only by node groups with num_gpus==0
(runners/ray/raycluster.py). So the flag keeps CPU work from squatting on
GPU nodes — it leaves GPU scheduling untouched.

Also reword the `lightweight` step's --help text, which is the
user-facing surface of the same ambiguity.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mode/config/db_uri controls only changed what the *next* read would connect
to; everything already on screen survived. Switching to enterprise left the
local job ids in the nav, the selected job id in `_current_job`, and the follow
timer polling — so the next read (a click, `r`, or the 3s poll) asked the
cluster for a job that only exists on disk and came back "not found".

The controls now define a target key, and moving it retargets the app: the
listings are dropped, the selected job/table and the follow poll are cleared,
and the section on screen is re-listed against the new backend (jobs if you
were reading job records, tables otherwise). Reads carry the epoch they started
in, so a slow scan that lands after a switch is discarded instead of repainting
the pane with the previous database's rows.

Also:
- An unusable target no longer takes the app down. `load_config` raises for
  enterprise-with-no-config.yaml, and that exception propagated out of the
  event handler into Textual's crash dialog; it now lands as a `⚠` leaf in the
  nav section, or on the pane the read would have filled.
- Refresh/`r` with nothing selected re-lists the section instead of doing
  nothing — the state a retarget leaves behind.
- The header names the backend you are on (`enterprise · db://prod`), and a
  genuine miss reads `no such job in the local database ./local_db` rather than
  a bare `not found`.
- The table/job load and filter paths funnel through `_open_table` /
  `_refresh_tables` / `_refresh_jobs` rather than repeating the
  build-cfg-and-dispatch dance at each call site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…yaml

test_tui_switching_mode_drops_the_other_backends_listings passed locally
and failed in CI. Switching the mode Select to enterprise re-lists the
browsed section, and enterprise mode requires config.yaml — which is
gitignored. On a machine that happens to have one the listing succeeds;
in CI (or a fresh clone) load_config raises and the nav gets a
`⚠ config file not found` leaf instead of the empty listing asserted.

Stub load_config in the three tests that switch backends, so they test
retarget semantics rather than the ambient filesystem. Verified from a
working directory with no config.yaml: 310 passed, was 1 failed.

The production behaviour is unchanged and correct — surfacing the config
error into the pane is what should happen when a backend can't be
configured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@justinrmiller
justinrmiller merged commit 2b914ff into main Jul 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants