Running career-ops locally with Ollama on 16GB RAM

[mahindra89]

Hey everyone! I wanted to share my experience trying to run career-ops fully locally using Ollama, in case it helps others with similar hardware.

My Setup

• Windows 11
• 16GB RAM
• Ollama with local models
• Claude Code v2.1.114

What I Tried

llama3.1:8b — Got past the login screen but the model couldn't follow the skill files at all. /career-ops just gave generic chatbot responses instead of the proper command menu.

qwen2.5-coder:7b — Same issue. The model was actively hallucinating file contents instead of actually reading them. When I asked it to read modes/_shared.md, it made up fake content instead of reading the real file.

qwen2.5-coder:14b with 32k context (custom Modelfile) — Got further, but took nearly 3 minutes just to respond to a simple file check. Way too slow to be practical for multi-step agentic tasks like scan or auto-pipeline.

Key Findings for Local Setup

1. The slash command routing doesn't work reliably with smaller models. /career-ops scan triggers a generic response instead of loading the skill files. A workaround is to explicitly say "Read modes/_shared.md and modes/scan.md, then scan for offers" but even then smaller models hallucinate the file contents.

2. 16GB RAM is the hard limit. After Windows overhead you have ~10-12GB free, which means 14b models run but are extremely slow for agentic multi-step tasks. 32b+ models won't fit at all.

3. The scan mode specifically needs Playwright browser automation which requires very reliable tool use — something 7b and 14b models can't handle consistently.

4. The Ollama environment variable setup works fine — the login prompt is bypassed correctly with:

ANTHROPIC_AUTH_TOKEN=ollama
ANTHROPIC_API_KEY=""
ANTHROPIC_BASE_URL=http://localhost:11434

but it is not able to use the career ops functionality is there anyone who figured out a way to use Career ops with ollama and keeping everything 100% local?

[kenshln47]

You’re 100% hitting a memory swap wall. A 14b Q4 model is already eating ~8.5GB, and that 32k context window is bloating your KV cache by another few gigs. Once you factor in Windows 11 overhead and Playwright spinning up a headless browser for the scan task, your 16GB is toast. That 3-minute delay is just your OS paging to disk to keep up.

The hallucinations on 7b/8b models happen because Claude Code’s engine is hardcoded for Anthropic’s XML tool-calling schema (, etc.). Base Llama/Qwen models don’t naturally parse those triggers well, so they just default to "chatbot mode" and make up file contents instead of actually calling the read tool.

To make this work locally on your current setup:

Drop your num_ctx to 8192 in the Modelfile. It’ll stop the paging/lag, though you'll lose some long-context reasoning for big JDs.

Use LiteLLM as a proxy between Claude Code and Ollama. It maps Anthropic’s tool-calling requests into standard OpenAI formats that models like qwen2.5-coder actually understand. This usually fixes the hallucination issues where the model "fakes" reading a file.

Honestly, 16GB is barely enough to keep the OS and a browser happy, let alone an agentic loop with Playwright. If LiteLLM doesn't cut it, you're probably better off using something like Aider that's built for local LLMs from the ground up.

[santifer]

First: apologies for the months-late reply, @mahindra89 — this writeup deserved a response long ago, and it's genuinely one of the most useful local-AI data points we have. Thanks for the patience. Where things stand now: you're right that 8B models can't follow the skill files — they lose the thread on long structured instructions, it's not your setup. What's changed since April: the engine is much more CLI-agnostic (OpenCode/Gemini/Qwen lanes shipped; a generic agent-execution table for batch is landing via #834), which is the groundwork local-model support builds on. Ollama support is explicitly on the roadmap (the "Next" section of #156). Practical today on 16GB: Qwen-coder-class models at 14B+ follow skills noticeably better than llama3.1:8b if you want to re-experiment. When the agent table lands, an Ollama entry is the natural next step — and your benchmark here is exactly the kind of test it'll need.