Explore llama-server router mode as a llama-swap replacement

[tobocop2]

llama-server recently gained a router mode: start it with no --model and it acts as a supervisor that spawns each model as an isolated child process, routes OpenAI requests to the right child by the model field, and exposes load/unload endpoints. That overlaps heavily with what we run llama-swap for. This issue is context for a focused spike into whether the engine can shed a moving part; nothing here is committed work.

The short version

We currently vendor two helper binaries next to llama-server: llama-swap (a process supervisor and request router) and gguf-parser (a VRAM estimator). Router mode plausibly replaces llama-swap because it does the same supervise/spawn/route/unload job from inside the binary we already ship. It replaces nothing gguf-parser does, since the router runs whatever it is told and makes no placement decisions. So the honest framing is "maybe remove llama-swap", and a small contained experiment can settle it.

Background: why we ship two helper binaries

llama-swap is the babysitter. A llama-server process can only serve one model. We run several (chat, embed, sometimes vision and rerank, plus elastic embed replicas during ingest), so something has to start those processes, watch them, restart them, route requests to the right one, and shut down the ones we no longer need. Today that something is llama-swap, a third-party Go binary we build into the engine wheel.

gguf-parser is the measuring tape. Before anything starts, we have to decide which GPU each model lands on, how a big model splits across cards, and how much context fits. That means predicting VRAM use per model per device. gguf-parser reads a GGUF file and produces that estimate; our placement planner turns the estimates into a concrete layout.

Router mode is llama-server absorbing the babysitter job. Nothing in it measures or places, so the measuring tape stays either way.

What router mode offers (as of recent llama.cpp)

• --models-preset takes an INI file where each section is one model and the keys are that model's own command-line arguments. Per-model tensor-split, ctx-size, device, and n-gpu-layers are all expressible, which is the same per-role argv we currently render into llama-swap's JSON config (build_swap_config in providers/fleet/swap_config.py).
• --models-max N caps how many children run at once, evicting the least recently used past the cap.
• POST /v1/models/load and /unload start and stop children explicitly.
• Children run on ephemeral local ports behind one endpoint, one process per model, so a crash stays contained.
• Auto-discovery from a models directory also exists; we would not use it because the planner decides what runs.

Today's architecture (before)

Follow the numbers: the planner measures, renders a config, llama-swap spawns the servers, and the provider routes live requests. One llama-swap per role (since #477), so a placement change restarts only the roles that moved.

flowchart LR
    subgraph lilbee ["lilbee"]
        planner["placement planner"]
        gguf["gguf-parser"]
        provider["FleetProvider"]
        planner -->|"1: measure each GGUF"| gguf
    end
    subgraph swaps ["llama-swap, one per role"]
        sc["swap: chat"]
        se["swap: embed"]
    end
    subgraph fleet ["llama-server processes"]
        chat["chat<br/>split GPU 1+2"]
        e1["embed<br/>GPU 0"]
        e2["embed<br/>GPU 2"]
    end
    planner -->|"2: render JSON config"| sc
    planner -->|"2: render JSON config"| se
    sc -->|"3: spawn and watch"| chat
    se -->|"3: spawn and watch"| e1
    se -->|"3: spawn and watch"| e2
    provider -->|"4: chat requests"| sc
    provider -->|"4: embed requests"| se

Loading

With the router (after, in theory)

Same numbered flow. Steps 1 and 4 are untouched; step 2 renders an INI preset instead of JSON, and step 3 moves inside llama-server itself. The whole middle column disappears.

flowchart LR
    subgraph lilbee2 ["lilbee"]
        planner2["placement planner<br/>unchanged"]
        gguf2["gguf-parser<br/>unchanged"]
        provider2["FleetProvider"]
        planner2 -->|"1: measure each GGUF"| gguf2
    end
    router["llama-server<br/>router mode"]
    subgraph fleet2 ["child processes"]
        chat2["chat<br/>split GPU 1+2"]
        e21["embed<br/>GPU 0"]
        e22["embed<br/>GPU 2"]
    end
    planner2 -->|"2: render INI preset"| router
    router -->|"3: spawn and watch"| chat2
    router -->|"3: spawn and watch"| e21
    router -->|"3: spawn and watch"| e22
    provider2 -->|"4: all requests, routed by model id"| router

Loading

An open variant: one router total versus one router per role. One total is the bigger simplification; one per role preserves #477's surgical-reload behavior if preset changes turn out to require a router restart (question 2 below).

What it would and would not replace

Replaces llama-swap, plausibly. Everything we use llama-swap for maps onto a router feature on paper: process supervision, lazy spawn, routing by model id, per-model unload (how elastic ingest replicas shrink back), health. Dropping it removes a Go build from the engine wheel and one vendored dependency.

Does not replace gguf-parser. The router runs whatever the preset says. Deciding which cards a model lands on, the tensor split, and the context size that fits stays entirely ours.

Questions the spike must answer

1. Resident-fleet semantics. We need every query role pinned resident. The router documents LRU eviction past --models-max but no explicit pin. Setting the cap at or above fleet size should keep everything loaded; verify eviction can never bite mid-ingest when elastic replicas push the count up.
2. Surgical reload parity. Placement changes restart only the roles that moved #477 made placement changes restart only the moved roles. What happens to a running router when the preset changes: a full restart (llama-swap's exact disease), or can /load and /unload swap one child's config surgically? If preset changes force a router restart, we need one router per role to keep Placement changes restart only the roles that moved #477's behavior, and the simplification shrinks.
3. Readiness and observability parity. We lean on llama-swap's /running for per-role readiness and /logs/stream/{model} for upstream log capture (SwapManager and client.py). Map each onto router equivalents; child state signaling exists but the shapes differ.
4. Orphan reaping. Our crash-recovery story (per-owner state files, config-path-keyed reaping of dead owners' processes, pid-reuse guards in swap_manager.py) assumes we own the supervisor's children. Confirm the router's children die with it reliably on every platform we ship, or that the same state-file discipline can wrap it.
5. Maturity. llama-swap has years of multi-model production use; router mode is new. The current stack is battle-tested and just got faster (Fleet rebuild after a placement change takes over a minute #474, Placement changes restart only the roles that moved #477), so the bar is "clearly simpler with no capability loss", not "newer".

Suggested experiment

Contained and cheap: render a preset for a two-role fleet, run one router, and script the placement-change and crash-recovery scenarios against it. If it clears questions 1 through 4, a migration is a SwapManager-shaped adapter change, not an architecture change. The planner, the config diffing, and the provider routing all survive intact.