ollama-bench

Reproducible benchmark of local LLMs on Ollama. Measure speed (tok/s), the effect of num_predict, reasoning quality, and code quality on your own hardware.

Zero dependencies beyond the Python 3 standard library - it talks to Ollama over HTTP (/api/generate).

Results at a glance

Generation speed on an Apple M1 Max 64 GB running Ollama - one consistent run of bench_speed.py --think=false (warmup + median of 3, each model measured in isolation):

Model	Gen tok/s	Thinking
qwen3-coder:30b	62.0	off
north-mini-code-1.0	61.0	off
ornith:35b	51.0	off
qwen3.6:35b-a3b	50.8	off
gemma4:e4b	47.1	off
ornith:9b	34.8	off
phi4:14b	19.7	off
gpt-oss:20b	51.7 †	on

The MoE / distilled coders run their daily-driver regime with thinking off; the dense phi4:14b is the slow outlier at 19.7. † gpt-oss:20b cannot disable thinking, so its 51.7 is the raw generation rate including thinking tokens - on a short prompt it spends that whole budget reasoning before answering (0 visible chars here), so it is not directly comparable to the thinking-off rows. Absolute tok/s is hardware-specific (see Disclaimer); what reproduces across machines is the relations and orders of magnitude. Reproduce with python3 bench_speed.py --think=false qwen3-coder:30b north-mini-code-1.0 ornith:35b qwen3.6:35b-a3b gemma4:e4b ornith:9b phi4:14b gpt-oss:20b.

Requirements

• Python 3.9+
• A running Ollama server at http://localhost:11434 (override with OLLAMA_HOST)
• The models you want to test, pulled:

ollama pull qwen3-coder:30b
ollama pull gpt-oss:20b

What each script measures

Script	Measures
bench_speed.py	generation tok/s (eval_count/eval_duration), isolated, warmup + median of 3
bench_speed.py --big	time on a ~12k-token prompt (prompt eval + generation)
bench_numpredict.py	two gpt-oss runs at different num_predict (thinking overflow)
bench_reasoning.py	reasoning: GENERATES puzzle answers and saves them to a file (no grading)
grade_reasoning.py	LLM-as-judge grading of the saved answers (decoupled; one unambiguous correct answer per puzzle, judged by another model)
bench_coding.py	code quality: function generation (auto-tested by execution) + bug finding. --hard/--expert/--mutated task sets (--mutated = LeetCode classics with spec twists that defeat memorization)
bench_cost.py	energy kWh/1M output tokens (from tok/s and an assumed power draw)

Quick start

# full suite via run_bench.py (edit MODELS + RUNS_CODE inside run.sh; RUNS_CODE=100 = published campaign)
bash run.sh

# or individually. NOTE: bench_reasoning/bench_coding now REQUIRE the token budgets that
# run_bench.py otherwise supplies from models.json - pass them explicitly for direct calls.
python3 bench_speed.py qwen3-coder:30b gpt-oss:20b
python3 bench_speed.py --big qwen3-coder:30b
python3 bench_numpredict.py gpt-oss:20b
python3 bench_reasoning.py --runs 3 --num-predict=10000 qwen3-coder:30b gpt-oss:20b
python3 bench_coding.py --expert --num-predict=4000 --bug-num-predict=800 qwen3-coder:30b   # runs the generated code!
python3 bench_cost.py "qwen3-coder:30b=61.2" "devstral:24b=9.8"   # energy kWh/1M

bench_coding.py executes (exec) code generated by the model against local test cases - run it only on trusted local models.

Canonical pipeline: models.json + run_bench.py (single source of truth)

models.json pins, per model, the best sampling params AND the per-task invocation (think, num_predict, and scalar Ollama option overrides like temperature / top_p). run_bench.py reads it and runs each model with ITS canonical flags, so invocations are never hand-assembled (no forgotten --think flag, no guessed temperature, no drifting num_predict). It prints the exact command per model. This is the reproducible entry point - prefer it over calling bench_*.py directly.

python3 run_bench.py reasoning fleet --runs 10          # reasoning, all 8 best models, n=10
python3 run_bench.py code gemma-best --expert           # one model, expert code set
python3 run_bench.py code fleet --expert --runs 10      # full code-expert table, n=10 (one command)
python3 run_bench.py code fleet --mutated               # anti-memorization tasks
python3 run_bench.py speed fleet                        # tok/s, isolated, --think=false for all

--runs N (default 3) applies to both reasoning and code. Models: explicit names, fleet (8 main), or all. Per-task thinking comes from the manifest (--think=false for Qwen-distill code, --think=on for reasoning, --think=low|high for gpt-oss which cannot be disabled). Per-task sampling overrides also come from the manifest; currently this matters for gpt-oss-best reasoning (temperature=1.0) and north-best reasoning (top_p=0.95). Truncation detection: a TR! flag when done_reason=length.

Note on comparability: runs made before run_bench.py started forwarding per-task sampling overrides used the Modelfile defaults for those two reasoning profiles. Re-benchmark gpt-oss/north reasoning before comparing old and new answer files.

Thinking default: the bench_*.py scripts with NO --think flag default to think=false (explicit OFF) - NOT the model's own default. think=None (no flag at the API) is thinking-ON for some models (gemma4 E4B), which once contaminated an "OFF" baseline. So pass --think=on for thinking explicitly; --think=none only if you want the model default. (run_bench is always explicit.)

The configs/*.best.Modelfile are GENERATED from models.json by gen_modelfiles.py - edit the manifest, not the Modelfiles. gen_modelfiles.py --check is a pre-commit guard against drift. Build a model with ollama create qwen36-best -f configs/qwen36.best.Modelfile. Full default-vs-best table: MODELS_CONFIG.md.

For long or unattended runs, run_pinned.sh wraps run_bench.py and can pin the ollama systemd service to specific GPU(s) first (restoring afterwards) - useful on multi-GPU hosts, because a big model spilling to CPU is not just slower but measurably degrades quality (a model that ran fully on GPU scored ~0.5 higher on the expert set than the same model offloaded to CPU). Example: ./run_pinned.sh --gpus 0,1 --restore-gpus 1 --task code --set --expert --runs 100 --models fleet (--gpus needs a systemd ollama service + passwordless sudo; omit it to leave the GPU config alone).

Why num_predict matters (thinking overflow)

A thinking model generates "thinking tokens" before it answers. With too small a budget those tokens eat the whole allowance before the model reaches the actual answer. In the bench_numpredict.py run, raising gpt-oss from num_predict=1500 to 3000 grew the answer from 366 to 494 words (~42 s to ~49 s). On harder prompts the effect is worse than "shorter": the whole answer can vanish - at num_predict=1500, 0 visible words (done_reason=length, every token spent on thinking) until the budget is raised. Exact counts are prompt-dependent and noisy. This is why the gpt-oss best config sets num_predict 3000 (and the per-task code budget is 4000).

Repo structure

• models.json - single source of truth (params + per-task invocation/options per model)
• run_bench.py - canonical dispatcher (reads models.json); gen_modelfiles.py - generates configs/ from it
• bench_*.py, grade_reasoning.py, _common.py - measurement + grading scripts
• prompts_pl.json / prompts_en.json - all test tasks in Polish (default) and English. Pick with BENCH_PROMPTS=prompts_en.json (see "Language effect" below)
• mutated_candidates*.json, verify_mutated*.py - anti-memorization code tasks + their verification
• configs/ - 11 best Modelfiles (GENERATED from models.json)
• MODELS_CONFIG.md - default vs best params per model + thinking control
• legacy/ - old Modelfiles from early experiments; kept for provenance, NOT used currently

Run logs (log_*.txt, _*.log), helper scripts (_*.sh, _*.py), and raw results (results_*.json, answers_*.json, graded_*.json) are gitignored - regenerable.

Disclaimer

Numbers depend on hardware, Ollama version, and model quantization. The reference values were measured on an Apple M1 Max 64 GB. Yours will differ in absolute value - the point is to reproduce the orders of magnitude and the relations (qwen many times faster than a dense 24B; num_predict really does lengthen a thinking model's answer). Energy is computed, not metered: kWh = tok/s and an assumed 45 W, so the ranking is solid (slower model = proportionally more energy) but the absolute kWh scales with that 45 W estimate.