Skip to content

fix(benchmark-local): handle Ollama connection/response errors gracefully - #705

Open
Anuj04432 wants to merge 2 commits into
DietrichGebert:mainfrom
Anuj04432:fix/ollama-error-handling
Open

fix(benchmark-local): handle Ollama connection/response errors gracefully#705
Anuj04432 wants to merge 2 commits into
DietrichGebert:mainfrom
Anuj04432:fix/ollama-error-handling

Conversation

@Anuj04432

Copy link
Copy Markdown

What

call_ollama() in benchmarks/benchmark-local.py had no error handling around the HTTP request to Ollama or around parsing its response. Any failure there (Ollama not running, wrong model name, unexpected response shape) crashed the whole benchmark run with a raw Python traceback.

This PR wraps those two points in try/except and raises a clear RuntimeError instead:

  • urllib.error.URLError → Ollama unreachable (not running / wrong URL)
  • json.JSONDecodeError → Ollama responded with non-JSON
  • KeyError → response JSON didn't have the expected message.content shape (e.g. an error payload)

Why

This script is meant to be run locally against Ollama (benchmarks/benchmark-local.py --model llama3.2 --repeat 3), and "Ollama isn't running yet" or "wrong model name" are common, expected mistakes — not exceptional edge cases. Right now those mistakes produce a confusing stack trace instead of telling the user what to check.

Before / after

Before (Ollama not reachable):

```
Traceback (most recent call last):
File "benchmark-local.py", line 76, in call_ollama
with urllib.request.urlopen(req, timeout=180) as resp:
...
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
```

After:

```
RuntimeError: Could not reach Ollama at http://localhost:11434/ (model 'llama3.2'): <urlopen error [Errno 111] Connection refused>. Is Ollama running and is the model pulled?
```

Testing

  • python -m py_compile benchmarks/benchmark-local.py — passes
  • Manually verified both new error paths raise the expected RuntimeError with a clear message:
    • unreachable host → clear connection-refused message
    • malformed response shape (mocked) → clear "unexpected response shape" message

Scope

Small, self-contained fix — only touches call_ollama() and one import line. No behavior changes for the success path.

@sanmaxdev sanmaxdev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One error-path issue to address.

Comment thread benchmarks/benchmark-local.py
@Anuj04432
Anuj04432 requested a review from sanmaxdev August 10, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants