Skip to content

Commit 672e2ce

Browse files
dcmcanddcmcand
authored
feat(dev): align local dev stack with AI Gateway v0.5 + key-manager UI dev mode (#115)
* feat(dev): align local dev stack with AI Gateway v0.5 and add key-manager UI dev mode Bump the dev/Makefile dependency stack to versions compatible with the bundled Envoy AI Gateway v0.5.0 (Envoy Gateway v1.6.7, Gateway API v1.4.0) and wire the AI Gateway ext_proc extension into Envoy Gateway at install time. On the previous versions (EG v1.3.0) a PassthroughModel reconciled to Ready but its upstream TLS was never programmed, so provider inference returned 503. Extend the dev manifests with the PassthroughModel RBAC, validating webhook, and shared-TLS issuance via the local self-signed ClusterIssuer, plus Makefile targets and an example model for the OpenRouter passthrough. Add an off-by-default dev mode to the key-manager: LLM_DEV_MODE bypasses auth and injects a fixed identity so the UI runs on a local cluster with no Keycloak. Exposed via keyManager.devMode in the Helm chart and enabled in the dev manifest, with a `make ui` port-forward target. Refs #113, #114 * fix(operator): emit BackendTLSPolicy as v1 for the PassthroughModel upstream Gateway API v1.4.0 (required by the bundled Envoy AI Gateway v0.5) graduates BackendTLSPolicy to v1 and no longer serves v1alpha3, so the operator's hardcoded v1alpha3 failed to apply on a version-aligned stack ("no matches for kind BackendTLSPolicy in gateway.networking.k8s.io/v1alpha3") and the passthrough upstream never got a TLS transport socket. Emit v1, which is the same spec shape. Refs #113 * fix(dev): grant key-manager RBAC to list passthroughmodels The key-manager watches PassthroughModels as well as LLMModels, but the dev manifest's llm-key-manager-models ClusterRole only granted llmmodels, so model sync failed ("cannot list passthroughmodels") and passthrough models never appeared in the UI. Matches the chart's key-manager role. Refs #114 * feat(dev): add `make run-dev` one-command UI dev environment with hot reload Frontend devs working on the key-manager UI now need only an OpenRouter key in dev/.env and `make run-dev`. The target idempotently brings up the kind cluster, operator, dev-mode key-manager, and three OpenRouter passthrough models, then port-forwards the key-manager and starts a hot-reloading UI dev server. - dev/uidev: a zero-dependency (stdlib-only) Go dev server that serves the UI static files from disk, proxies /api/* to the port-forwarded key-manager, and live-reloads the browser on file edits. The UI is plain static files, so no build step or npm is involved. - dev/run-dev.sh + `make run-dev`: orchestrates cluster/deploy/models/port-forward /UI server, loading OPENROUTER_API_KEY from a gitignored dev/.env. - dev/manifests/dev-models.yaml: three passthrough models so the UI list is populated. - docs/ui-development.md: frontend-dev guide (setup, editing, dev-mode auth, shipping changes, API table, troubleshooting), linked from getting-started. Refs #114 * fix(dev): harden local dev stack for Helm 4 and the webhook startup race Addresses review feedback on the local dev path: - Gateway API CRDs: apply with --server-side and pass --force-conflicts to the eg chart install, gated on Helm major version (Helm 4 needs it; Helm 3 has no server-side apply, skips present crds/, and rejects the flag). Fixes the CRD ownership conflict that blocked `make setup` on Helm 4. - make setup: guard `kind create cluster` so a partial setup is re-runnable. - operator.yaml: add a readinessProbe on the webhook port (9443) so the `make deploy` rollout waits until the webhook is serving. - run-dev.sh: bounded retry around the webhook-gated PassthroughModel apply (covers the residual endpoint-propagation window), and a per-run mktemp file for the port-forward log.
1 parent 19e6087 commit 672e2ce

23 files changed

Lines changed: 866 additions & 46 deletions

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
11
CLAUDE.md
22
docs/superpowers/
33
.claude/
4+
5+
# Local dev secrets (OPENROUTER_API_KEY, etc.)
6+
.env

‎charts/nebari-llm-serving/templates/key-manager-deployment.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,17 @@ spec:
4848
- name: LLM_OIDC_USERINFO_URL
4949
value: {{ .Values.keyManager.oidcUserinfoURL | quote }}
5050
{{- end }}
51+
{{- if .Values.keyManager.devMode.enabled }}
52+
# Dev mode: bypasses auth and injects a fixed identity so the UI
53+
# works on a local cluster with no Keycloak. Never enable in a real
54+
# deployment. See nebari-dev/llm-serving-pack#114.
55+
- name: LLM_DEV_MODE
56+
value: "true"
57+
- name: LLM_DEV_USER
58+
value: {{ .Values.keyManager.devMode.user | quote }}
59+
- name: LLM_DEV_GROUPS
60+
value: {{ .Values.keyManager.devMode.groups | join "," | quote }}
61+
{{- end }}
5162
resources:
5263
limits:
5364
cpu: 200m

‎charts/nebari-llm-serving/values.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,16 @@ envoyAIGateway:
6262

6363
keyManager:
6464
enabled: true
65+
# devMode bypasses the key-manager's auth and injects a fixed identity so the
66+
# UI can run on a local cluster with no Keycloak / gateway OIDC layer. Off by
67+
# default; never enable it in a real deployment. See
68+
# nebari-dev/llm-serving-pack#114. Pair it with a direct port-forward to the
69+
# key-manager Service so the gateway OIDC enforcement is bypassed too.
70+
devMode:
71+
enabled: false
72+
user: dev
73+
groups:
74+
- llm
6575
image:
6676
repository: ghcr.io/nebari-dev/nebari-llm-serving-pack/key-manager
6777
# tag defaults to .Chart.AppVersion when empty so the chart version and

‎dev/.env.example‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copy to dev/.env and fill in. dev/.env is gitignored.
2+
#
3+
# OpenRouter API key (https://openrouter.ai/keys). Used as the upstream
4+
# provider credential for the PassthroughModels that `make run-dev` deploys.
5+
OPENROUTER_API_KEY=

‎dev/Makefile‎

Lines changed: 59 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,58 @@
11
CLUSTER_NAME ?= llm-serving-test
22

3-
.PHONY: setup teardown build-images load-images deploy deploy-operator deploy-key-manager apply-test-model logs-operator logs-key-manager clean help
3+
# Dependency versions. These move together: Envoy AI Gateway v0.5.x requires
4+
# Envoy Gateway v1.6.x and Gateway API v1.4.0 (see
5+
# https://aigateway.envoyproxy.io/docs/compatibility/). Bump them as a set
6+
# when upgrading the AI Gateway.
7+
CERT_MANAGER_VERSION ?= v1.17.2
8+
GATEWAY_API_VERSION ?= v1.4.0
9+
GIE_VERSION ?= v1.4.0
10+
ENVOY_GATEWAY_VERSION ?= v1.6.7
11+
AI_GATEWAY_VERSION ?= v0.5.0
12+
13+
# Helm 4 applies the eg chart's bundled Gateway API CRDs (crds/) server-side,
14+
# which conflicts with the standalone install in `setup` (owned by a different
15+
# field manager). Force ownership so the chart wins. Helm 3 has no server-side
16+
# apply and skips already-present crds/, so it neither needs nor accepts this
17+
# flag - keep it empty there. Empty when helm is missing (setup fails later).
18+
HELM_MAJOR := $(shell helm version --template '{{.Version}}' 2>/dev/null | sed -E 's/^v?([0-9]+).*/\1/')
19+
HELM_FORCE_CONFLICTS := $(if $(filter-out 1 2 3,$(HELM_MAJOR)),--force-conflicts,)
20+
21+
.PHONY: setup teardown build-images load-images deploy deploy-operator deploy-key-manager apply-test-model apply-passthrough-model create-openrouter-secret run-dev ui logs-operator logs-key-manager clean help
422

523
help: ## Show this help
624
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n\nTargets:\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)
725

26+
run-dev: ## One-command UI dev environment: cluster + models + port-forward + hot-reload UI server (needs dev/.env with OPENROUTER_API_KEY)
27+
./run-dev.sh
28+
829
setup: ## Create kind cluster and install dependencies
9-
kind create cluster --name $(CLUSTER_NAME)
30+
# Idempotent: skip creation if the cluster already exists so a setup that
31+
# died midway can be re-run without `make teardown` first.
32+
@kind get clusters | grep -qx $(CLUSTER_NAME) || kind create cluster --name $(CLUSTER_NAME)
1033
# cert-manager
11-
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.2/cert-manager.yaml
34+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$(CERT_MANAGER_VERSION)/cert-manager.yaml
1235
kubectl -n cert-manager rollout status deployment/cert-manager-webhook --timeout=120s
13-
# Gateway API CRDs
14-
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
15-
# GIE CRDs (v1.4.0, includes graduated k8s.io group)
16-
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.4.0/manifests.yaml
17-
# Envoy Gateway
18-
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.3.0 -n envoy-gateway-system --create-namespace
19-
kubectl -n envoy-gateway-system rollout status deployment/envoy-gateway --timeout=120s
20-
# Envoy AI Gateway
21-
helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version v0.5.0 -n envoy-ai-gateway-system --create-namespace
22-
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm --version v0.5.0 -n envoy-ai-gateway-system
36+
# Gateway API CRDs. Server-side apply so the field ownership matches the
37+
# eg chart's own SSA on Helm 4, letting it cleanly take co-ownership with
38+
# --force-conflicts (see HELM_FORCE_CONFLICTS above) instead of erroring.
39+
kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api/releases/download/$(GATEWAY_API_VERSION)/standard-install.yaml
40+
# GIE CRDs (includes the graduated inference.networking.k8s.io group)
41+
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/$(GIE_VERSION)/manifests.yaml
42+
# Envoy AI Gateway first: envoy-gateway's extensionManager below points at
43+
# the ai-gateway-controller XDS service, so bring it up before reconfiguring
44+
# envoy-gateway to avoid noisy connection-refused logs during translation.
45+
helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version $(AI_GATEWAY_VERSION) -n envoy-ai-gateway-system --create-namespace
46+
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm --version $(AI_GATEWAY_VERSION) -n envoy-ai-gateway-system
2347
kubectl -n envoy-ai-gateway-system wait --timeout=120s deployment/ai-gateway-controller --for=condition=Available
24-
# LLMModel CRD
48+
# Envoy Gateway, wired with the AI Gateway ext_proc extension (enableBackend,
49+
# extensionManager, backendResources). Without this the per-model routing
50+
# layer 404s and passthrough upstreams never get a TLS transport socket.
51+
helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm --version $(ENVOY_GATEWAY_VERSION) -n envoy-gateway-system --create-namespace $(HELM_FORCE_CONFLICTS) -f eg-extension-values.yaml
52+
kubectl -n envoy-gateway-system rollout status deployment/envoy-gateway --timeout=120s
53+
# CRDs (served LLMModels and external-provider PassthroughModels)
2554
kubectl apply -f ../charts/nebari-llm-serving/crds/llmmodel-crd.yaml
55+
kubectl apply -f ../charts/nebari-llm-serving/crds/passthroughmodel-crd.yaml
2656
# Test Gateways
2757
kubectl apply -f gateways.yaml
2858

@@ -45,13 +75,26 @@ deploy-operator: ## Deploy operator
4575
kubectl apply -f manifests/webhook.yaml
4676
kubectl -n llm-operator-system rollout status deployment/llm-operator --timeout=60s
4777

48-
deploy-key-manager: ## Deploy key manager
78+
deploy-key-manager: ## Deploy key manager (dev mode: auth bypassed, dev identity injected)
4979
kubectl apply -f manifests/key-manager.yaml
5080
kubectl -n llm-operator-system rollout status deployment/llm-key-manager --timeout=60s
5181

52-
apply-test-model: ## Apply test model
82+
apply-test-model: ## Apply served test model (mock vLLM)
5383
kubectl apply -f manifests/test-model.yaml
5484

85+
create-openrouter-secret: ## Create the OpenRouter provider credential (OPENROUTER_API_KEY env var required)
86+
@test -n "$(OPENROUTER_API_KEY)" || { echo "set OPENROUTER_API_KEY"; exit 1; }
87+
kubectl -n llm-operator-system create secret generic openrouter-api-key \
88+
--from-literal=apiKey="$(OPENROUTER_API_KEY)" \
89+
--dry-run=client -o yaml | kubectl apply -f -
90+
91+
apply-passthrough-model: ## Apply the OpenRouter PassthroughModel (run create-openrouter-secret first)
92+
kubectl apply -f manifests/passthrough-test-model.yaml
93+
94+
ui: ## Port-forward the key-manager UI to http://localhost:8080 (dev mode, no login)
95+
@echo "key-manager UI at http://localhost:8080 (dev mode injects user 'dev')"
96+
kubectl -n llm-operator-system port-forward svc/llm-key-manager 8080:8080
97+
5598
logs-operator: ## Tail operator logs
5699
kubectl -n llm-operator-system logs -f deployment/llm-operator
57100

‎dev/eg-extension-values.yaml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Envoy Gateway Helm values that wire in the Envoy AI Gateway ext_proc
2+
# extension. Mirrors docs/install-production.md section 6.1, which sources
3+
# these from envoyproxy/ai-gateway's manifests/envoy-gateway-values.yaml.
4+
#
5+
# - extensionApis.enableBackend lets HTTPRoutes reference InferencePool.
6+
# - extensionManager points envoy-gateway at the AI Gateway controller's XDS
7+
# extension server (port 1063) so it inserts the ext_proc filter.
8+
# - backendResources lists the non-builtin backend kinds the extension handles.
9+
config:
10+
envoyGateway:
11+
gateway:
12+
controllerName: gateway.envoyproxy.io/gatewayclass-controller
13+
extensionApis:
14+
enableEnvoyPatchPolicy: true
15+
enableBackend: true
16+
extensionManager:
17+
hooks:
18+
xdsTranslator:
19+
translation:
20+
listener: { includeAll: true }
21+
route: { includeAll: true }
22+
cluster: { includeAll: true }
23+
secret: { includeAll: true }
24+
post:
25+
- Translation
26+
- Cluster
27+
- Route
28+
service:
29+
fqdn:
30+
hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
31+
port: 1063
32+
backendResources:
33+
- group: inference.networking.k8s.io
34+
kind: InferencePool
35+
version: v1

‎dev/manifests/dev-models.yaml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Three OpenRouter passthrough models for the local dev environment, applied by
2+
# `make run-dev` so the key-manager UI has a populated model list to work with.
3+
# Each is its own PassthroughModel so the UI lists three distinct models.
4+
#
5+
# Note: API keys are currently gateway-scoped, not model-scoped (see
6+
# nebari-dev/llm-serving-pack#116), so a key minted for one of these works for
7+
# all of them. That does not affect UI development.
8+
apiVersion: llm.nebari.dev/v1alpha1
9+
kind: PassthroughModel
10+
metadata:
11+
name: claude-sonnet-45
12+
namespace: llm-operator-system
13+
spec:
14+
provider:
15+
hostname: openrouter.ai
16+
schemaVersion: api/v1
17+
credentialSecretName: openrouter-api-key
18+
models:
19+
catchAll: false
20+
declared:
21+
- anthropic/claude-sonnet-4.5
22+
access:
23+
groups:
24+
- llm
25+
---
26+
apiVersion: llm.nebari.dev/v1alpha1
27+
kind: PassthroughModel
28+
metadata:
29+
name: gemini-25-flash
30+
namespace: llm-operator-system
31+
spec:
32+
provider:
33+
hostname: openrouter.ai
34+
schemaVersion: api/v1
35+
credentialSecretName: openrouter-api-key
36+
models:
37+
catchAll: false
38+
declared:
39+
- google/gemini-2.5-flash
40+
access:
41+
groups:
42+
- llm
43+
---
44+
apiVersion: llm.nebari.dev/v1alpha1
45+
kind: PassthroughModel
46+
metadata:
47+
name: llama-33-70b
48+
namespace: llm-operator-system
49+
spec:
50+
provider:
51+
hostname: openrouter.ai
52+
schemaVersion: api/v1
53+
credentialSecretName: openrouter-api-key
54+
models:
55+
catchAll: false
56+
declared:
57+
- meta-llama/llama-3.3-70b-instruct
58+
access:
59+
groups:
60+
- llm

‎dev/manifests/key-manager.yaml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ metadata:
1111
name: llm-key-manager-models
1212
rules:
1313
- apiGroups: ["llm.nebari.dev"]
14-
resources: ["llmmodels"]
14+
resources: ["llmmodels", "passthroughmodels"]
1515
verbs: ["get", "list", "watch"]
1616
---
1717
apiVersion: rbac.authorization.k8s.io/v1
@@ -91,6 +91,16 @@ spec:
9191
value: "IdToken"
9292
- name: LLM_LISTEN_ADDR
9393
value: ":8080"
94+
# Dev mode: there is no Keycloak or gateway OIDC layer on the kind
95+
# cluster, so bypass auth and inject a fixed identity. The UI and
96+
# /api/* then work behind a plain `make ui` port-forward. Never set
97+
# this in a real deployment. See nebari-dev/llm-serving-pack#114.
98+
- name: LLM_DEV_MODE
99+
value: "true"
100+
- name: LLM_DEV_USER
101+
value: "dev"
102+
- name: LLM_DEV_GROUPS
103+
value: "llm"
94104
resources:
95105
limits:
96106
cpu: 200m

‎dev/manifests/operator.yaml‎

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ rules:
2929
- apiGroups: ["llm.nebari.dev"]
3030
resources: ["llmmodels/finalizers"]
3131
verbs: ["update"]
32+
- apiGroups: ["llm.nebari.dev"]
33+
resources: ["passthroughmodels"]
34+
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
35+
- apiGroups: ["llm.nebari.dev"]
36+
resources: ["passthroughmodels/status"]
37+
verbs: ["get", "update", "patch"]
38+
- apiGroups: ["llm.nebari.dev"]
39+
resources: ["passthroughmodels/finalizers"]
40+
verbs: ["update"]
3241
- apiGroups: ["apps"]
3342
resources: ["deployments"]
3443
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
@@ -48,17 +57,30 @@ rules:
4857
resources: ["inferencepools"]
4958
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
5059
- apiGroups: ["aigateway.envoyproxy.io"]
51-
resources: ["aigatewayroutes"]
60+
resources: ["aigatewayroutes", "aiservicebackends", "backendsecuritypolicies"]
5261
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
5362
- apiGroups: ["gateway.envoyproxy.io"]
54-
resources: ["securitypolicies"]
63+
resources: ["securitypolicies", "backends"]
64+
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
65+
# PassthroughModel provider TLS validation toward the external provider.
66+
- apiGroups: ["gateway.networking.k8s.io"]
67+
resources: ["backendtlspolicies"]
5568
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
5669
- apiGroups: ["gateway.networking.k8s.io"]
5770
resources: ["referencegrants"]
5871
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
5972
- apiGroups: ["monitoring.coreos.com"]
6073
resources: ["podmonitors"]
6174
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
75+
# Shared-TLS reconciler: issues the cert covering llm/llm-internal via the
76+
# selfsigned-issuer ClusterIssuer (see cert-manager-config.yaml) and patches
77+
# HTTPS listeners onto the shared Gateways.
78+
- apiGroups: ["cert-manager.io"]
79+
resources: ["certificates"]
80+
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
81+
- apiGroups: ["gateway.networking.k8s.io"]
82+
resources: ["gateways"]
83+
verbs: ["get", "list", "watch", "update", "patch"]
6284
---
6385
apiVersion: rbac.authorization.k8s.io/v1
6486
kind: ClusterRoleBinding
@@ -95,7 +117,7 @@ spec:
95117
imagePullPolicy: Never
96118
env:
97119
- name: LLM_BASE_DOMAIN
98-
value: "llm.local"
120+
value: "local"
99121
- name: LLM_EXTERNAL_GATEWAY_NAME
100122
value: "nebari-gateway"
101123
- name: LLM_EXTERNAL_GATEWAY_NAMESPACE
@@ -104,8 +126,13 @@ spec:
104126
value: "nebari-internal-gateway"
105127
- name: LLM_INTERNAL_GATEWAY_NAMESPACE
106128
value: "envoy-gateway-system"
129+
# Issue the shared llm.local / llm-internal.local cert via the
130+
# local self-signed ClusterIssuer (cert-manager-config.yaml) so the
131+
# operator can patch HTTPS listeners onto the gateways with no ACME.
132+
- name: LLM_CLUSTER_ISSUER_NAME
133+
value: "selfsigned-issuer"
107134
- name: LLM_OIDC_ISSUER_URL
108-
value: "https://keycloak.llm.local/realms/nebari"
135+
value: "https://keycloak.local/realms/nebari"
109136
- name: LLM_OIDC_GROUPS_CLAIM
110137
value: "groups"
111138
- name: ENABLE_WEBHOOKS
@@ -120,6 +147,16 @@ spec:
120147
- containerPort: 9443
121148
name: webhook-server
122149
protocol: TCP
150+
# The validating webhook binds 9443 only after controller-runtime
151+
# loads the mounted serving cert. Gate readiness on that port so
152+
# `kubectl rollout status` in `make deploy` does not return before the
153+
# webhook can accept PassthroughModel creates (the manager's /readyz
154+
# is only a ping and goes green before the webhook is serving).
155+
readinessProbe:
156+
tcpSocket:
157+
port: 9443
158+
initialDelaySeconds: 2
159+
periodSeconds: 2
123160
volumeMounts:
124161
- name: cert
125162
mountPath: /tmp/k8s-webhook-server/serving-certs

‎dev/manifests/passthrough-test-model.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# OpenRouter PassthroughModel for the local dev cluster.
2+
#
3+
# Prereq: the provider credential Secret. Create it with:
4+
# make create-openrouter-secret OPENROUTER_API_KEY=sk-or-v1-...
5+
#
6+
# Once reconciled (kubectl -n llm-operator-system get passthroughmodel), reach
7+
# it through the gateway. The external endpoint uses API-key auth, so inject a
8+
# client key into the api-keys Secret to skip the key-manager:
9+
# kubectl -n llm-operator-system patch secret openrouter-api-keys --type merge \
10+
# -p '{"stringData":{"localtester":"sk-localtest-abc123"}}'
11+
# kubectl -n envoy-gateway-system port-forward svc/envoy-...-nebari-gateway 8443:443 &
12+
# curl -k https://llm.local:8443/v1/chat/completions \
13+
# --resolve llm.local:8443:127.0.0.1 \
14+
# -H "Authorization: Bearer sk-localtest-abc123" -H "Content-Type: application/json" \
15+
# -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'
16+
apiVersion: llm.nebari.dev/v1alpha1
17+
kind: PassthroughModel
18+
metadata:
19+
name: openrouter
20+
namespace: llm-operator-system # per #59 lives in the operator namespace
21+
spec:
22+
provider:
23+
hostname: openrouter.ai
24+
# OpenRouter serves the OpenAI API under /api/v1.
25+
schemaVersion: api/v1
26+
credentialSecretName: openrouter-api-key
27+
models:
28+
catchAll: true
29+
declared:
30+
- openai/gpt-4o-mini
31+
access:
32+
groups:
33+
- llm

0 commit comments

Comments
 (0)