Skip to content

llm-providers.md

Latest commit

 

History

History
388 lines (277 loc) · 11.4 KB

llm-providers.md

File metadata and controls

388 lines (277 loc) · 11.4 KB
Raw

LLM Provider Configuration

Sashiko supports multiple LLM providers. This guide covers setup for each one. Gemini is the default and simplest to configure; the others are alternatives you can swap in depending on your infrastructure and preferences.

For all providers, configuration lives in Settings.toml at the project root. Per-provider example files are available in docs/examples/.

Gemini (default)

The quickest way to get started.

cp docs/examples/Settings.example.toml Settings.toml
export LLM_API_KEY="your-gemini-api-key"

The example file sets provider = "gemini" and model = "gemini-3.1-pro-preview". Adjust the model name as needed.

You can also set any config value via environment variables using the SASHIKO prefix with __ (double underscore) as the separator:

export SASHIKO__AI__PROVIDER=gemini
export SASHIKO__AI__MODEL=gemini-3.1-pro-preview

Claude (API)

Uses Anthropic's Claude API directly.

Get an API key: https://console.anthropic.com/

Set credentials:

export ANTHROPIC_API_KEY="sk-ant-..."
# Or use the generic fallback:
export LLM_API_KEY="sk-ant-..."

Apply the example config:

cp docs/examples/Settings.claude.toml Settings.toml

What you get:

  • Automatic prompt caching (5-minute TTL) to reduce costs on repeated context
  • Full tool/function calling support for git operations
  • Automatic retry on rate limits and API overload
  • 200K context window (use max_input_tokens = 40000 for cost-conscious defaults)
  • Extended thinking via the thinking and effort settings in [ai.claude]

Claude Code CLI

Uses a local Claude Code installation as the completion backend. This uses your Claude Code subscription -- no per-token charge, no API key.

Prerequisites: Install Claude Code and sign in. Verify with claude --version.

Apply the example config:

cp docs/examples/Settings.claude-cli.toml Settings.toml

Model selection:

model accepts any identifier the CLI supports via --model -- aliases like opus or sonnet, or full names like claude-opus-4-7, claude-sonnet-4-6.

Context window sizes:

  • claude-opus-4-7: 1M tokens by default.
  • claude-sonnet-4-6 and claude-opus-4-6: 1M capable, but the CLI defaults to 200K. Append [1m] (e.g. claude-sonnet-4-6[1m]) to opt into the 1M variant.
  • claude-haiku-4-5: 200K only, no 1M variant.
  • Pre-thinking models (Claude 3.x) are rejected with HTTP 404.

What you get:

  • No API key needed -- uses Claude Code's subscription auth.
  • Stateless: spawns claude --print --output-format json per request. No tool access, no file access, no session reuse.
  • Prompt caching handled automatically by Claude Code.
  • effort controls the model's thinking budget. Opus 4.7 uses adaptive thinking; Sonnet 4.6 and Haiku 4.5 use extended thinking. If you need to pick the mode explicitly or disable thinking, use provider = "claude" (API) instead.
  • [ai.claude] settings (prompt_caching, thinking, etc.) apply only to the API provider above -- they are ignored on this path.

Note: Each review may spawn many CLI processes. Lower review.concurrency if you hit subscription rate limits.

GitHub Copilot CLI

Uses a local GitHub Copilot CLI installation as the completion backend. This uses your GitHub Copilot subscription -- no per-token charge, no API key.

Prerequisites:

  • copilot CLI installed and on $PATH
  • Authenticated session (run copilot once interactively to log in)

Apply the example config:

cp docs/examples/Settings.copilot-cli.toml Settings.toml

What you get:

  • model follows GitHub Copilot's catalog (e.g. claude-sonnet-4.5, gpt-5.5); pick a model your subscription has access to.
  • Sashiko invokes copilot with --disable-builtin-mcps, --no-custom-instructions, and --allow-all-tools so it acts as a pure text-completion-with-tools backend.
  • The prompt is sent via stdin (not -p) to avoid Linux's MAX_ARG_STRLEN cap (~128 KB per argv element).
  • max_interactions controls tool-call rounds before aborting.

Note: Each review may spawn many copilot processes. Lower review.concurrency if you hit subscription rate limits.

AWS Bedrock

Uses AWS Bedrock via the Converse API. Works with any Bedrock-hosted model (Claude, Llama, Mistral, etc.).

Prerequisites: Enable model access in the AWS Bedrock console for your desired model and region.

Set AWS credentials using any standard method:

# Option 1: Environment variables
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"

# Option 2: AWS CLI profile (~/.aws/credentials)
aws configure

Apply the example config:

cp docs/examples/Settings.claude-bedrock.toml Settings.toml

What you get:

  • Converse API -- works with any Bedrock-hosted model
  • No API key needed -- uses standard AWS IAM authentication
  • Cross-region inference profiles (e.g. us.anthropic.claude-*)
  • Full tool/function calling support

Google Cloud Vertex AI

Uses Claude models (and potentially others) via Google Cloud infrastructure. Requires building with --features vertex.

Prerequisites: Enable the Vertex AI API and model access in the Vertex AI Model Garden.

Authenticate:

gcloud auth application-default login

Set project and region:

export ANTHROPIC_VERTEX_PROJECT_ID="my-gcp-project"
export CLOUD_ML_REGION="us-east5"  # "global" endpoint not currently supported

Apply the example config:

cp docs/examples/Settings.claude-vertex.toml Settings.toml

What you get:

  • No API key needed -- uses Google Cloud Application Default Credentials
  • Global, multi-region, and regional endpoint support
  • 1M context window for Claude Opus 4.7/4.6 and Sonnet 4.6 on Vertex
  • Full tool/function calling and prompt caching support

Kiro CLI

Uses the local kiro-cli as a completion backend.

Prerequisites: Install kiro-cli and authenticate with KIRO_API_KEY or a browser login.

Apply the example config:

cp docs/examples/Settings.kiro-cli.toml Settings.toml

What you get:

  • Runs kiro-cli acp as a stateless completion backend
  • Kiro native tools are disabled by default; Sashiko's own tool protocol is used instead
  • An isolated temporary agent with a deny-all hook prevents accidental tool execution

Codex CLI

Uses a local Codex CLI (OpenAI) installation as the completion backend. This uses your Codex subscription -- no per-token charge, no API key.

Prerequisites: Install the codex CLI and authenticate.

Apply the example config:

cp docs/examples/Settings.codex-cli.toml Settings.toml

For OpenAI's coding-optimized model, use docs/examples/Settings.gpt-5-codex.toml instead (same backend, model = "gpt-5-codex").

What you get:

  • Runs codex exec --json --sandbox read-only as a stateless backend
  • Prompt sent via stdin to avoid ARG_MAX issues
  • No tool access -- sandbox is read-only

Devin CLI Setup

Sashiko can use a local Devin for Terminal install as a completion backend. This path uses your Devin subscription, so there is no per-token API charge and no API key to configure.

Prerequisites:

  • devin CLI installed and on $PATH
  • Authenticated session (run devin auth login once)

Update Settings.toml: Copy examples/Settings.devin-cli.toml to your Settings.toml and adjust as needed.

Notes:

  • model accepts any identifier devin --model accepts (e.g. opus, swe, gpt, codex). Leave it empty in [ai] to use the Devin default.
  • Sashiko invokes devin --print --prompt-file <tmp> per request. The prompt is written to a temp file which is passed to devin.
  • Each review may spawn many devin processes. Lower review.concurrency if you hit subscription rate limits.

Ollama

Ollama allows running LLMs locally.

Prerequisites: Install Ollama and pull your desired model (e.g., ollama pull deepseek-v3).

Apply the example config:

cp docs/examples/Settings.ollama.toml Settings.toml

What you get:

  • Private, local execution of LLMs
  • Support for reasoning models via the think setting
  • No API key or subscription required
  • context_window_size maps to Ollama's num_ctx

vLLM

vLLM serves local models behind an OpenAI-compatible API.

Prerequisites: Start a vLLM server, e.g.:

vllm serve Qwen/Qwen3-8B --max-model-len 32768 --host 0.0.0.0 --port 8000

Apply the example config:

cp docs/examples/Settings.vllm.toml Settings.toml

What you get:

  • Private, local execution of LLMs
  • Reasoning model support: <think> blocks and reasoning_content are separated from the answer automatically, and thinking can be toggled via the enable_thinking setting
  • Leaving max_tokens unset lets vLLM generate up to the remaining context, which is useful for servers running with a small --max-model-len
  • Optional guided_json setting to enforce JSON responses through guided decoding on backends that support it
  • Optional enable_tools setting to forward tool definitions; requires a server started with --enable-auto-tool-choice and --tool-call-parser
  • If the server was started with --api-key, export it as VLLM_API_KEY (or LLM_API_KEY)

Set context_window_size to match the server's --max-model-len.

OpenAI-Compatible Providers

Sashiko includes an OpenAI-compatible provider for endpoints that implement the OpenAI chat completions API.

Apply the example config:

cp docs/examples/Settings.openai-compat.toml Settings.toml

Adjust base_url to point to your provider's endpoint.

base_url may be either:

  • a shorthand such as http://localhost:8080/v1 (the /chat/completions suffix is appended automatically), or
  • the full chat completions URL, e.g. https://api.z.ai/api/coding/paas/v4/chat/completions. Use this form for providers whose path is not a recognised shorthand.

z.ai / Zhipu (glm-*) example:

z.ai exposes two OpenAI-compatible gateways with separate billing: a direct API (…/api/paas/v4/…, billed to the API resource package) and a coding-plan gateway (…/api/coding/paas/v4/…, billed to the coding-plan subscription). To review against the coding-plan quota, point base_url at the full coding endpoint and set LLM_API_KEY (or OPENAI_API_KEY):

[ai]
provider = "openai-compatible"
model = "glm-5.2"

[ai.openai_compat]
base_url = "https://api.z.ai/api/coding/paas/v4/chat/completions"
context_window_size = 128000
max_tokens = 16384

OrcaRouter example:

OrcaRouter is an OpenAI-compatible gateway to models from OpenAI, Anthropic, Google, and other providers behind a single endpoint and API key. Point base_url at https://api.orcarouter.ai/v1 and use a namespaced model id such as openai/gpt-4o-mini:

[ai]
provider = "openai-compatible"
model = "openai/gpt-4o-mini"

[ai.openai_compat]
base_url = "https://api.orcarouter.ai/v1"
context_window_size = 128000
max_tokens = 16384

For OpenAI's own API with an API key (rather than a self-hosted compatible endpoint), use docs/examples/Settings.openai-api.toml: set provider = "openai", model = "gpt-5.6-sol", and export OPENAI_API_KEY.