Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
# Copy to .env (gitignored) and fill in. promptfoo reads this automatically.
ANTHROPIC_API_KEY=sk-ant-...
# MiniMax (global OpenAI-compatible endpoint). Used by benchmarks/promptfooconfig.minimax.yaml.
MINIMAX_API_KEY=...
15 changes: 15 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,21 @@ npx promptfoo@latest view
`--env-file ../.env` is required because promptfoo reads `.env` from the current
directory (`benchmarks/`), not the repo root where the file lives.

### MiniMax (MiniMax-M3 / MiniMax-M2.7)

Requires a MiniMax API key (global endpoint) and **Node.js ≥ 22.22.0** (same
promptfoo engine constraint as Claude):

```bash
cp ../.env.example .env # add your MINIMAX_API_KEY
npx promptfoo@latest eval -c promptfooconfig.minimax.yaml --env-file ../.env --repeat 10
npx promptfoo@latest view
```

The MiniMax provider is the OpenAI-compatible `openai:chat:<model>` surface with
`apiBaseUrl` pointed at `https://api.minimax.io/v1`; `MINIMAX_API_KEY` is read
via `apiKeyEnvar` so the key stays in `.env`. See `promptfooconfig.minimax.yaml`.

### Local models via Ollama

No API key or promptfoo required. Runs against any model served by Ollama:
Expand Down
41 changes: 41 additions & 0 deletions benchmarks/promptfooconfig.minimax.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Ponytail vs baseline, MiniMax: MiniMax-M3 + MiniMax-M2.7 over the OpenAI-compatible endpoint.
# npx promptfoo@latest eval -c benchmarks/promptfooconfig.minimax.yaml --env-file ../.env --repeat 10
# Needs MINIMAX_API_KEY in ../.env (global endpoint). The MiniMax OpenAI-compatible
# base URL points at the /v1 chat completions surface; see benchmarks/README.md.
description: "Ponytail vs baseline on MiniMax (MiniMax-M3, MiniMax-M2.7). LOC + correctness, cost telemetry."

providers:
- id: openai:chat:MiniMax-M3
config:
apiBaseUrl: https://api.minimax.io/v1
apiKeyEnvar: MINIMAX_API_KEY
max_tokens: 8192
temperature: 1
- id: openai:chat:MiniMax-M2.7
config:
apiBaseUrl: https://api.minimax.io/v1
apiKeyEnvar: MINIMAX_API_KEY
max_tokens: 2048
temperature: 1

prompts:
- id: file://arms/baseline.js
label: baseline (no skill)
- id: file://arms/ponytail.js
label: ponytail

defaultTest:
assert:
- type: javascript
value: file://loc.js
metric: code_loc
- type: javascript
value: file://correctness.js
metric: correct

tests:
- vars: { task: "Write me a Python function that validates email addresses." }
- vars: { task: "Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call." }
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }