AI Gateway

Backend for Brave's AI Chat (Leo). An OpenAI-compatible API server that routes chat completions to configurable LLM backends (local, vLLM, LiteLLM, Bedrock, Anthropic) with support for MCP tools, deep research, and more.

Prerequisites

• Docker (including Compose) - used by make setup/start/test-docker.
• make - included by default on macOS and most Linux distros.
• A backend to actually generate responses, e.g. Ollama (the .env.example default) or LM Studio running locally - see Configuring models.

make start-native additionally requires Poetry and Python 3.12-3.14 on the host, since it runs outside Docker.

Quickstart

Use make to build the image and copy the example environment file:

make setup

.env defaults MODELS to a local Ollama instance (ollama pull qwen2.5:14b); edit it to point at a different backend if needed. If you edit the JSON values, keep them single-quoted (e.g. MODELS='{"...":"..."}') - unquoted double quotes get stripped when the variables are loaded into the shell, producing invalid JSON. Then start the server:

make start

Or run directly on the host instead of Docker (e.g. to use stdio MCP servers via a command not baked into the image, like NPX):

make start-native

Check that it's up:

curl http://127.0.0.1:8000/v1/models

This should return a JSON array containing your configured model(s). If it returns [], see Configuring models - a few metadata fields are required for a model to show up here, beyond what's needed to route requests to it.

For Brave Developers

Checkout internal setup for connecting a local browser build, premium access, and other Brave-internal instructions.

Testing

Build the test image:

make test-image

Run all the tests with docker-compose:

make test-docker

Additional Test Commands

or run a specific test

make test-docker TEST_ARG="test/aichat/serve/conversation_api_test.py::test_v1_conversation_max_tokens_config"

Instruct Syrupy to generate new snapshots based on new outputs

make test-docker TEST_ARG="--snapshot-update test/aichat/conversation/prompt_test.py test/aichat/conversation/retrieve/functions_test.py"

Roadmap

In addition to the ai-gateway repo, our roadmap includes open sourcing some other Brave built internal services used within ai-gateway. This includes the following:

• Brave Deep Research MCP
• Leo Private Analytics
• Search Orchestration MCP

Learn more

• Configuring models - backends, model metadata, common local-setup gotchas
• Configuring MCP Servers - MCP server config and custom handlers
• Premium model access - how premium vs. free requests are distinguished and what it affects
• Trying other endpoints - images, TTS, embeddings, passthrough, API-key chat completions, regression tests, local browser builds
• Deep Research - enabling the deep research MCP tool
• Bedrock - using AWS Bedrock-backed models
• Conversation Sharing - the Leo conversation-sharing endpoints
• Rate Limiting - rate limiting behavior and configuration

Attributions

This repo contains code from the OpenAI compatibility server of FastChat which is under an Apache 2.0 license.

Subsequent Brave modifications and original contributions are under the MPL 2.0 license.