guide : running gpt-oss with llama.cpp #15396

ggerganov · 2025-08-18T15:09:57Z

ggerganov
Aug 18, 2025
Maintainer

Note

This guide is a live document. Feedback and benchmark numbers are welcome - the guide will be updated accordingly.

Overview

This is a detailed guide for running the new gpt-oss models locally with the best performance using llama.cpp. The guide covers a very wide range of hardware configurations. The gpt-oss models are very lightweight so you can run them efficiently in surprisingly low-end configurations.

Obtaining `llama.cpp` binaries for your system

Make sure you are running the latest release of llama.cpp: https://github.com/ggml-org/llama.cpp/releases

Obtaining the `gpt-oss` model data (optional)

The commands used below in the guide will automatically download the model data and store it locally on your device. So this step is completely optional and provided for completeness.

The original models provided by OpenAI are here:

First, you need to manually convert them to GGUF format. For convenience, we host pre-converted models here in ggml-org.

Pre-converted GGUF models:

Tip

Running the commands below will automatically download the latest version of the model and store it locally on your device for later usage. A WebUI chat and an OAI-compatible API will become available on localhost.

Sample output of using gpt-oss-120b* with the built-in llama-server WebUI*	Using llama-server* with crush coding agent (gpt-oss-20b)*

Note

This guide currently covers general chat use cases with llama-server. Tool-calling and agentic use cases are mostly supported via the OAI-compatible API of llama-server (see the "Tips" section at the bottom), but some minor issues are still being resolved, both on llama.cpp side and in the reference upstream model implementations (chat template, harmony). As things get more polished in the next few days, you should be able to use the commands below in combination with your favorite 3rd party applications that support the OAI interface (such as chat interfaces, coding agents, etc.).

Minimum requirements

Here are some hard memory requirements for the 2 models. These numbers could vary a little bit by adjusting the CLI arguments, but should give a good reference point.

Model	Model data (GB)	Compute buffers (GB)	KV cache per 8 192 tokens (GB)	Total @ 8 192 tokens (GB)	Total @ 32 768 tokens (GB)	Total @ 131 072 tokens (GB)
gpt‑oss 20B	12.0	2.7	0.2	14.9	15.5	17.9
gpt‑oss 120B	61.0	2.7	0.3	64.0	64.9	68.5

Note

It is not necessary to fit the entire model in VRAM to get good performance. Offloading just the attention tensors and the KV cache in VRAM and keeping the rest of the model in the CPU RAM can provide decent performance as well. This is taken into account in the rest of the guide.

Relevant CLI arguments

Using the correct CLI arguments in your commands is crucial for getting the best performance for your hardware. Here is a summary of the important flags and their meaning:

Argument	Purpose
`-hf`	Specify the Hugging Face model ID to use. The model will be downloaded using `curl` from the respective model repository
`-c`	Specify the context size to use. More context requires more memory. Both `gpt-oss` models have a maximum context of 128k tokens. Use `-c 0` to set to the model's default
`-ub N -b N`	Specify the max batch size `N` during processing. Larger size increases the size of compute buffers, but can improve the performance in some cases
`-fa`	Enable Flash Attention kernels. This improves the performance on backends that support the operator
`--n-gpu-layers N`	Number of layers `N` to offload to the GPU. For this guide, keep this number at `-ngl 99`
`--n-cpu-moe N`	Number of MoE layers `N` to keep on the CPU. This is used in hardware configs that cannot fit the models fully on the GPU. The specific value depends on your memory resources and finding the optimal value requires some experimentation
`--jinja`	Tell `llama.cpp` to use the Jinja chat-template embedded in the GGUF model file

Apple Silicon

Apple Silicon devices have unified memory that is seamlessly shared between the CPU and GPU. For optimal performance it is recommended to not exceed 70% of the total memory that your device has.

Tip

Install the latest llama.cpp package from Homebrew with:

brew install llama.cpp

Tip

To increase the amount of RAM available to the llama-server process, use the following command:

# on a 192GB machine, raise the limit from 154GB (default) to 180GB
sudo sysctl iogpu.wired_limit_mb=180000

✅ Devices with more than 96GB RAM

The M2 Max, M3 Max, M4 Max, M1 Ultra, M2 Ultra, M3 Ultra, etc. chips can run both models at full context:

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on M2 Ultra (192GB) for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	2191.13 ± 2.65
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1889.83 ± 3.91
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	1594.51 ± 2.42
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	1218.99 ± 0.44
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	116.08 ± 0.18

build: 79c1160 (6123)

llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.112	115.06	1.113	115.05
0	128	2	256	0.000	0.00	1.601	159.92	1.601	159.91
0	128	4	512	0.000	0.00	2.463	207.85	2.463	207.84
2048	128	1	2176	0.990	2068.28	1.163	110.03	2.154	1010.44
2048	128	2	4352	1.916	2137.49	1.710	149.72	3.626	1200.17
2048	128	4	8704	3.775	2169.82	2.656	192.78	6.431	1353.37
8192	128	1	8320	4.344	1885.93	1.279	100.11	5.622	1479.81
8192	128	2	16640	8.689	1885.52	1.929	132.69	10.619	1567.04
8192	128	4	33280	17.359	1887.62	3.053	167.69	20.413	1630.35
16384	128	1	16512	10.202	1606.01	1.397	91.63	11.599	1423.63
16384	128	2	33024	20.715	1581.82	2.186	117.08	22.902	1441.98
16384	128	4	66048	41.721	1570.80	3.653	140.14	45.375	1455.61
32768	128	1	32896	26.611	1231.39	1.665	76.88	28.276	1163.40
32768	128	2	65792	54.977	1192.06	2.794	91.64	57.771	1138.85
32768	128	4	131584	111.278	1177.88	4.883	104.85	116.161	1132.77

llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on M2 Ultra (192 GB) for `gpt-oss-120b`

llama-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp2048	1244.57 ± 5.10
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp8192	1101.31 ± 0.99
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp16384	955.41 ± 0.64
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	pp32768	752.31 ± 1.02
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	Metal	2048	1	tg128	79.68 ± 0.12

build: 79c1160 (6123)

llama-batched-bench -m gpt-oss-120b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1,2,4

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.610	79.48	1.611	79.48
0	128	2	256	0.000	0.00	2.284	112.08	2.284	112.08
0	128	4	512	0.000	0.00	3.477	147.27	3.477	147.27
2048	128	1	2176	1.776	1152.89	1.711	74.82	3.487	623.99
2048	128	2	4352	3.382	1211.16	2.458	104.14	5.840	745.18
2048	128	4	8704	6.505	1259.34	3.747	136.65	10.252	849.02
8192	128	1	8320	7.294	1123.16	1.857	68.94	9.150	909.25
8192	128	2	16640	14.467	1132.48	2.767	92.53	17.234	965.53
8192	128	4	33280	28.801	1137.74	4.358	117.50	33.159	1003.66
16384	128	1	16512	16.580	988.15	2.058	62.18	18.639	885.89
16384	128	2	33024	33.426	980.31	3.174	80.66	36.600	902.29
16384	128	4	66048	67.190	975.39	5.245	97.61	72.435	911.83
32768	128	1	32896	42.075	778.81	2.452	52.20	44.527	738.79
32768	128	2	65792	86.615	756.64	4.029	63.54	90.644	725.83
32768	128	4	131584	173.762	754.32	7.020	72.94	180.782	727.86

✅ Devices with less than 96GB RAM

The small gpt-oss-20b model can run efficiently on Macs with at least 16GB RAM:

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on M4 Max (36GB) for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	1277.42 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	1030.28 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	779.44 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	568.13 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	92.36 ± 0.00

build: 79c1160 (6123)

llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.359	94.17	1.359	94.15
2048	128	1	2176	1.676	1222.17	1.450	88.30	3.125	696.26
8192	128	1	8320	7.624	1074.47	1.552	82.47	9.176	906.67
16384	128	1	16512	19.210	852.91	1.669	76.67	20.879	790.84
32768	128	1	32896	55.684	588.46	1.976	64.76	57.661	570.51

🟢 Benchmarks on M1 Max (64GB) for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	994.75 ± 4.11
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	843.01 ± 2.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	698.82 ± 0.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp32768	497.65 ± 8.92
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	75.15 ± 0.98

build: 2e2b22b (6180)

🟢 Benchmarks on M1 Pro (32GB) for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp2048	515.76 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp8192	437.22 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	pp16384	361.29 ± 0.00
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal	2048	1	tg128	45.68 ± 0.00

build: 79c1160 (6123)

llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 132096 -b 2048 -ub 2048 -npp 0,2048,8192,16384 -ntg 128 -npl 1

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	2.806	45.62	2.806	45.62
2048	128	1	2176	4.054	505.14	3.076	41.61	7.130	305.18
8192	128	1	8320	18.444	444.15	3.329	38.45	21.773	382.12
16384	128	1	16512	44.683	366.67	3.780	33.86	48.464	340.71

🟥 Devices with 8GB RAM

Unfortunately, you are out of luck. The gpt-oss models are not possible to run on Macs with that small amount of memory.

NVIDIA

✅ Devices with more than 64GB VRAM

With more than 64B VRAM, you can run both models by offloading everything (both the model and the KV cache) to the GPU(s).

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-20b`

llama-bench -m ./gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp2048	9480.55 ± 44.01
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp8192	8921.62 ± 4.21
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp16384	8196.12 ± 19.16
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp32768	7050.35 ± 12.36
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	tg128	249.96 ± 0.99

build: f08c4c0 (6199)

🟢 Benchmarks on RTX Pro 6000 (96GB) for `gpt-oss-20b`

llama-bench -m ./gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp2048	11521.95 ± 26.03
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp8192	10673.03 ± 22.35
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp16384	9772.06 ± 19.59
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	pp32768	8267.46 ± 15.58
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	2048	1	tg128	286.91 ± 0.22

build: a6d3cfe (6205)

llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-120b`

llama-bench -m ./gpt-oss-120b-mxfp4/gpt-oss-120b-mxfp4-00001-of-00003.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp2048	4494.20 ± 20.87
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp8192	4327.73 ± 16.04
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp16384	4114.04 ± 12.84
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp32768	3718.01 ± 19.67
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	tg128	170.62 ± 0.47

build: f08c4c0 (6199)

🟢 Benchmarks on RTX Pro 6000 (96GB) for `gpt-oss-120b`

llama-bench -m ./gpt-oss-120b-mxfp4/gpt-oss-120b-mxfp4-00001-of-00003.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	n_ubatch	fa	test	t/s
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp2048	5518.07 ± 31.18
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp8192	5315.65 ± 21.91
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp16384	5012.78 ± 24.18
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	pp32768	4503.36 ± 31.57
gpt-oss 120B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	2048	1	tg128	196.31 ± 0.14

build: a6d3cfe (6205)

✅ Devices with less than 64GB VRAM

In this case, you can fit the small gpt-oss-20b model fully in VRAM for optimal performance.

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks on NVIDIA GeForce RTX 3090 (24GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, CUDA 12.4

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	5170.56 ± 14.10
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	4771.74 ± 12.96
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	4289.11 ± 3.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	3577.10 ± 2.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	161.77 ± 0.56
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	5142.90 ± 26.50
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	4711.52 ± 4.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	4245.67 ± 5.30
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	3539.35 ± 2.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	161.95 ± 0.49

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 4090 (24GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, CUDA 12.6

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	8022.33 ± 161.33
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	7264.73 ± 69.07
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	6298.35 ± 94.91
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	5112.35 ± 34.90
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	221.95 ± 6.34
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	8078.28 ± 39.37
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	6715.17 ± 204.96
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	6025.25 ± 66.75
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	4924.71 ± 26.63
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	225.22 ± 0.10

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5060 Ti (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	3839.21 ± 6.79
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	3695.85 ± 6.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	3472.60 ± 1.82
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	3078.06 ± 0.62
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	111.51 ± 0.05
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	3821.18 ± 13.28
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	3591.27 ± 1.45
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	3385.30 ± 2.44
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	3009.63 ± 2.82
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	111.56 ± 0.02

build: 9ef6b0b (6208)

🟢 Benchmarks on NVIDIA GeForce RTX 5070 Ti (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	6339.76 ± 25.60
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	5913.85 ± 9.12
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	5375.41 ± 10.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	4547.18 ± 3.70
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	189.45 ± 0.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	6325.97 ± 37.98
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	5669.50 ± 13.36
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	5193.12 ± 5.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	4411.35 ± 2.43
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	189.46 ± 0.03

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5080 (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	7476.55 ± 20.89
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	7047.73 ± 19.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	6465.65 ± 23.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	5531.03 ± 29.67
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	204.85 ± 0.23
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	7469.28 ± 43.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	6725.38 ± 11.03
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	6218.87 ± 25.68
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	5376.58 ± 31.23
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	204.86 ± 0.04

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5090 (32GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp2048	9848.38 ± 28.98
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp8192	8834.14 ± 27.65
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp16384	7802.21 ± 35.06
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	pp32768	6290.76 ± 64.50
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	2048	1	tg128	282.51 ± 0.44
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp2048	9841.15 ± 29.56
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp8192	8482.25 ± 44.45
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp16384	7513.55 ± 34.19
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	pp32768	6089.55 ± 77.41
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	4096	4096	1	tg128	282.26 ± 0.10

build: a094f38 (6210)

The large model has to be partially kept on the CPU.

🟡 TODO: add commands for gpt-oss-120b

✅ Devices with 16GB VRAM

For example: NVIDIA V100

This config is just at the edge to fit the full context of gpt-oss-20b in VRAM, so we have to restrict the maximum context down to 32k tokens.

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa

🟢 Benchmarks on NVIDIA V100 (16GB) for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa 1 -b 4096 -ub 4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: Tesla V100-PCIE-16GB, compute capability 7.0, VMM: yes

model	size	params	backend	ngl	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp2048	3526.65 ± 346.86
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp8192	3320.62 ± 44.98
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp16384	2768.99 ± 19.73
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	pp32768	2096.44 ± 8.58
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	4096	4096	1	tg128	117.71 ± 0.30

build: 228f724 (6129)

llama-batched-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 33792 -b 4096 -ub 4096 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	128	1	128	0.000	0.00	1.106	115.74	1.106	115.72
2048	128	1	2176	0.481	4257.50	1.201	106.60	1.682	1293.82
8192	128	1	8320	2.247	3646.05	1.417	90.31	3.664	2270.69
16384	128	1	16512	5.421	3022.12	1.659	77.14	7.081	2331.96
32768	128	1	32896	15.031	2180.10	2.121	60.35	17.151	1917.98

Running the large gpt-oss-120b model with 16GB of VRAM requires to keep some of the layers on the CPU since it does not fit completely in VRAM:

llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 4096 -b 4096 -ngl 99 -fa --n-cpu-moe 32

✅ Devices with less than 16GB VRAM

For this config, it is recommended to tell llama.cpp to run the entire model on the GPU while keeping enough layers on the CPU. Here is a specific example with an RTX 2060 8GB machine:

# gpt-oss-20b, full context, 22 layers on the CPU
llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0     --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 22

# gpt-oss-20b, 32k context, 16 layers on the CPU (faster, but has less total context)
llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 16

Note that even with just 8GB of VRAM, we can adjust the CPU layers so that we can run the large 120B model too:

# gpt-oss-120b, 32k context, 35 layers on the CPU
llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa --n-cpu-moe 35

Tip

For more information about how to adjust the CPU layers, see the "Tips" section at the end of this guide.

AMD

Note

If you have AMD hardware, please provide feedback about running the gpt-oss models on it and the performance that you observe. See the sections above for what kind of commands to try and try to adjust respectively.

With AMD devices, you can use either the ROCm or the Vulkan backends. Depending on your specific hardware, the results can vary.

✅ RX 7900 XT (20GB VRAM) using ROCm backend

llama-server -hf ggml-org/gpt-oss-20b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

🟢 Benchmarks for `gpt-oss-20b`

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp2048	4251.56 ± 21.68
gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp8192	3567.45 ± 11.84
gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp16384	2948.39 ± 10.34
gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	pp32768	2099.25 ± 13.17
gpt-oss 20B BF16	12.83 GiB	20.91 B	ROCm,RPC	99	1	4096	2048	1	tg128	101.92 ± 0.27

build: 3007baf (6194)

More information: #15396 (comment)

✅ Few more low-end configurations
- AMD Radeon 890M using Vulkan
- AMD Radeon FirePro W8100 + AMD Radeon RX 470 using Vulkan

Tips

Determining the optimal number of layers to keep on the CPU

Good general advice for most MoE models would be to offload the entire model, and use -n-cpu-moe to keep as many MoE layers as necessary on the CPU. The minimum amount of VRAM to do this with the 120B model is about 8GB, below that you will need to start reducing context size and the number of layers offloaded. You can get for example about 30 t/s at zero context on a 5090 with --n-cpu-moe 21.

Caveat: on Windows it is possible to allocate more VRAM than available, and the result will be slow swapping to RAM and very bad performance. Just because the model loads without errors, it doesn't mean you have enough VRAM for the settings that you are using. A good way to avoid this is to look at the "GPU Memory" in Task Manager and check that it does not exceed the GPU VRAM.

Example on 5090 (32GB):
good, --n-cpu-moe 21, GPU Memory < 32:

bad, --n-cpu-moe 20, GPU Memory > 32:

Using `gpt-oss` + `llama.cpp` with coding agents (such as Claude Code, crush, etc.)

Setup the coding agent of your choice to look for a localhost OAI endpoint (see Tutorial: Offline Agentic coding with llama-server #14758)

Start llama-server like this:

# adjust this command for your hardware
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048 -ngl 99 -fa

# some agents such as Claude Code can benefit from multiple parallel server slots
# note: currently this requires extra memory!
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 524288 -np 4 --jinja -ub 2048 -b 2048 -ngl 99 -fa

Sample usage with crush: guide : running gpt-oss with llama.cpp #15396 (reply in thread)
Some agents such as Cline and Roo Code do not support native tool calls. A workaround is to use a custom grammar: guide : running gpt-oss with llama.cpp #15396 (comment)

Configure the default sampling and reasoning settings

When starting a llama-server command, you can change the default sampling and reasoning settings like so:

# use recommended gpt-oss sampling params
llama-server ... --temp 1.0 --top-p 1.0

# set default reasoning effort
llama-server ... --chat-template-kwargs '{"reasoning_effort": "high"}'

Note that these are just the default settings and they could be overridden by the client connecting to the llama-server.

Frequently asked questions

Q: Which quants to use?

Always use the original MXFP4 model files. The gpt-oss models are natively "quantized". I.e. they are trained in the MXFP4 format which is roughly equivalent to ggml's Q4_0. The main difference with Q4_0 is that the MXFP4 models get to keep their full quality. This means that no quantization in the usual sense is necessary.

Q: What sampling parameters to use?

OpenAI recommends: temperature=1.0 and top_p=1.0.

Do not use repetition penalties! Some clients tend to enable repetition penalties by default - make sure to disable those.

Be careful when you disable the Top K sampler. Although recommended by OpenAI, this can lead to significant CPU overhead and small but non-zero probability of sampling low-probability tokens.

Q: Should I set a chat template file manually?

No. The ggml-org/gpt-oss models have a built-in chat template that is used by default. The only reasons to ever want to change the chat template manually are:

If there is a bug in the built-in chat template
If you have a very specific use case and you know very well what you are doing

Known issues

Some rough edges in the implementation are still being polished. Here is a list of issue to keep track of:

SteelPh0enix · 2025-08-18T16:33:33Z

SteelPh0enix
Aug 18, 2025

I can provide some numbers for AMD part of the guide.

My hardware is RX 7900 XT (20GB VRAM) + Ryzen 9 5900X + 32GB of RAM, running on latest Arch Linux with locally built llama.cpp version 6194 (3007baf), built with ROCm 6.4.1-1 (from official Arch repo)

Pulled the gpt-oss-20b repository and converted to GGUF using convert_hf_to_gguf.py, should probably result in the same GGUF file as on huggingface.

7900XT can load the full 20B model with full context without offloading MoE layers to CPU (although barely, because it will fill up the whole VRAM), by running

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa

With that, i get generation speeds (as reported by llama.cpp webui) at ~94 tokens/second, slowly going down as the context fills up.

I've also tested whether setting K/V cache quantization would help with model size or performance, but the result was... bad, performance was halved and CPU got involved... is this because of mxfp4 format of gpt-oss?

I'd also like to note that my PC likes to hang when i fill up my VRAM to the brim, so i've also checked out how gpt-oss-20b behaves when i off-load MoE layers to CPU.

When running with all MoE layers on CPU, as below:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -cmoe

my GPU VRAM usage (as reported by btop) is around 10GB, RAM usage went up only ~2GB. However, the performance took a major 80% hit, as now my generation speed is in ~20tok/s - CPU takes most of the load. If you have better CPU and faster RAM (i'm still running dual-channel DDR4s @ 3200MHz CL16, mind you), you probably will get better results. I wonder how X3Ds behave in that case...

I assume that gpt-oss-20b has 24 MoE layers, so let's see how it behaves when i load only, let's say, 4 onto CPU:

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 4

VRAM is at 18GB (previously it was at 19, as reported by btop, so there's a decrease), RAM usage went up by around 1.5GB, generation speed is ~60tok/s. Neat, this is usable.

How about 8 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 8

In that case, i get 16GB VRAM usage, ~1.5GB RAM bump as previously, and generation speed went down to 38 tokens/s. Still pretty usable. How about 16 layers?

llama-server -m ./gpt-oss-20b.auto.gguf --ctx-size 0 --jinja -b 4096 -ub 4096 -ngl 99 -fa -ncmoe 16

VRAM: 13GB, RAM: as previously, not more than 2GB, generation speed: 27-25tok/s, this is getting pretty bad.

As mentioned before - your results may vary, i'm not running current-gen top-tier hardware and IIRC the largest performance bottleneck will be the RAM/PCIe link speed anyway - i'm pretty curious to see what the performance with this GPU is on more recent platform, especially with an X3D CPU.

11 replies

aldehir Aug 18, 2025
Collaborator

I had issues with a higher batch/ubatch size than the default but I'm not seeing that problem anymore so that was probably user error on my end.

I believe you are likely hitting the case where the model needs the CoT from the past tool call but the client isn't sending it in or there is a mismatch in the reasoning field. That is an open issue across all client and inference servers/providers with GPT-OSS.

If you can collect any dumps of this happening, I'm happy to dig in further.

SteelPh0enix Aug 18, 2025

@SteelPh0enix I've been able to get crush to work with moderate success.

Can you share the llama-server command line arguments you pass in?

Sure @aldehir, here's my config for this model in crush.json:

 "providers": {
        "llamacpp": {
            "type": "openai",
            "base_url": "http://steelph0enix.pc:51536/v1",
            "name": "Llama.cpp",
            "id": "llamacpp",
            "models": [
                {
                    "id": "gpt-oss-20b.auto",
                    "name": "GPT-OSS 20B",
                    "context_window": 131072,
                    "default_max_tokens": 51200,
                    "has_reasoning_efforts": true,
                    "can_reason": true,
                    "supports_attachments": false,
                    "default_reasoning_effort": "high",
                    "cost_per_1m_in": 0,
                    "cost_per_1m_in_cached": 0,
                    "cost_per_1m_out": 0,
                    "cost_per_1m_out_cached": 0
                }
            ]
        }
    }
// ...

my llama-server invocation:

llama-server --ctx-size 0 --model gpt-oss-20b.auto.gguf --alias "gpt-oss-20b.auto" --jinja

i keep most of my llama-server settings in env vars, as following:

export LLAMA_ARG_HOST="0.0.0.0"
export LLAMA_ARG_PORT="51536"
export LLAMA_ARG_BATCH=2048
export LLAMA_ARG_UBATCH=2048
export LLAMA_ARG_SWA_FULL=false
export LLAMA_ARG_KV_SPLIT=false
export LLAMA_SET_ROWS=1 # for ARG_KV_SPLIT=false to work
export LLAMA_ARG_FLASH_ATTN=true
export LLAMA_ARG_MLOCK=true
export LLAMA_ARG_NO_MMAP=false
export LLAMA_ARG_N_GPU_LAYERS=999
export LLAMA_OFFLINE=true
export LLAMA_ARG_ENDPOINT_SLOTS=true
export LLAMA_ARG_ENDPOINT_PROPS=true

I've opened my test project (a Rust app) with gpt-oss-20b as chosen model in Crush and told it to initialize the project... and it seems to work just fine now!

I've tested Crush back on 0.6.0 (or 0.6.1) with gpt-oss, if not on 0.5.x, and i definitely had issues (for example, the chat description below CRUSH logo contained gpt-oss chat template tags...) so you must've fixed it already - i just haven't noticed :)

Thanks!

If you want, you may add my piece of crush.json to the post as a config example (change the IP to localhost though ;) ) @ggerganov, the invocation from the original post should work just fine

SteelPh0enix Aug 18, 2025

I had issues with a higher batch/ubatch size than the default but I'm not seeing that problem anymore so that was probably user error on my end.

I believe you are likely hitting the case where the model needs the CoT from the past tool call but the client isn't sending it in or there is a mismatch in the reasoning field. I believe that's an open issue across all client and inference servers/providers with GPT-OSS.

If you can collect any dumps of this happening, I'm happy to dig in further.

Yes, i think i did notice that on some other models, i've been mostly working with Qwen... If this happens again, how can i get some more logs/info?

Oh, and one potential "issue" i've just noticed - i've set my reasoning in model config to "high", but crush seem to force "minimal", is this "by design", or is it some issue?

aldehir Aug 18, 2025
Collaborator

I can't say, but it likely has no effect since llama-server only respects the chat_template_kwargs.reasoning_effort field in the request. I doubt crush is setting it, so it defaults to "medium" unless you change it via command line.

I usually run mitmproxy in the background, but enabling verbose and searching for parse errors in the server log will likely show the root of the problem--if there is one.

SteelPh0enix Aug 18, 2025

btw @ggerganov i think you made a mistake labeling my test results, i don't have a mac, and they sure don't use AMD GPUs anymore 😄

gcp · 2025-08-18T19:22:50Z

gcp
Aug 18, 2025

Configure the default sampling and reasoning settings

When starting a llama-server command, you can change the default sampling and reasoning settings like so:

# use recommended gpt-oss sampling params
llama-server ... --temp 1.0 --top-p 1.0

Q: What sampling parameters to use?

OpenAI recommends: temperature=1.0 and top_p=1.0.

The problem I see is that the llama-server defaults are

--samplers SAMPLERS                     samplers that will be used for generation in the order, separated by
                                        ';'                                                                  
                                        (default:                                                            
                                        penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)                                                                                            
--temp N                                temperature (default: 0.8)                                           
--top-k N                               top-k sampling (**default: 40**, 0 = disabled)                         
--top-p N                               top-p sampling (default: 0.9, 1.0 = disabled)
--min-p N                               min-p sampling (**default: 0.1**, 0.0 = disabled)

So the above command is actually equivalent to:

llama-server ... --temp 1.0 --top-p 1.0 --top-k 40 --min-p 0.1

Which seems quite a bit different from the actual recommendation from OpenAI. Notably "min-p 0.1" will prune a lot of low-probability tokens, whereas the OpenAI recommendation is basically to follow the model output probabilities.

If you look at a lot of guides and settings for other SOTA LLM, they all recommend min-p 0.01 or 0.00.

Should the command line be changed to:

llama-server ... --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

6 replies

SmallAndSoft Aug 19, 2025

It is not enough to define sampling parameters, one has to define the sampler chain itself, such as --sampling-seq tp for the chain recommended by OAI.

That said, I concur that their recommendation is nonsensical:
Temperature of 1 says "multiply all probabilities by 1" which leaves them unchanged.
top_p of 1 says "select tokens that add up to 100% probability" which is all of them.

As for me, I prefer --sampling-seq ks with top_k=2048 for speedup and then top_n_sigma with the default parameter of 1.

gcp Aug 19, 2025

one has to define the sampler chain itself...for the chain recommended by OAI

With the settings given combined with the built-in defaults, the extra samplers are already disabled, so I do not think that is actually a requirement?

then top_n_sigma with the default parameter of 1.

This isn't supported in llama-server. (A number of claims in top-n-sigma paper fall flat when temperature is applied last, as is the case for llama.cpp, so I'm not sure this is going to change any time soon)

That said, I concur that their recommendation is nonsensical

The one from OpenAI at least is internally consistent: it means to sample the model output exactly as given. This is unusual however because it means there's no filtering on low-probability tokens whatsoever, so there's a low (but non-zero) chance you get a low probability token.

The llama.cpp default of "min-p 0.1" filters quite a bit more, so that's where we get the inconsistency, i.e. if you run llama.cpp with those parameters, the defaults will change the behavior quite a bit from the intention, it ends up actually quite aggressive in pruning low probability tokens.

ggerganov Aug 19, 2025
Maintainer Author

Regardless of the OpenAI recommendation, I still think it's a good idea to filter low-probability tokens (for example with Top-K or Min-P).

For now, I've updated the guide with the following paragraph:

Be careful when you disable the `Top K` sampler. Although recommended by OpenAI, this can lead to significant CPU overhead and small but non-zero probability of sampling low-probability tokens.

We can revisit if we determine that sampling from the full vocab is actually important.

SmallAndSoft Aug 19, 2025

then top_n_sigma with the default parameter of 1.

This isn't supported in llama-server. (A number of claims in top-n-sigma paper fall flat when temperature is applied last, as is the case for llama.cpp, so I'm not sure this is going to change any time soon)

Not sure what you mean. The support was merged in #13264
Temperature can be applied at any step you want if you define your own sampling chain.

gcp Aug 19, 2025

I still think it's a good idea to filter low-probability tokens (for example with Top-K or Min-P).

Yes, that seems quite sensible. Note that the default will have both top-k 40 and min-p 0.1.

The support was merged in #13264

Looks like a bug crept in there, I'll file an issue.

Temperature can be applied at any step you want if you define your own sampling chain.

Yes, and if you put it last like llama.cpp does by default, you don't have some of the key problems that sampler is supposed to fix 😀

QuantiusBenignus · 2025-08-18T19:35:53Z

QuantiusBenignus
Aug 18, 2025

To fill better the low-end CUDA edge cases, here are some benchmarks for gpt-oss-20B (both MXFP4 and Unsloth UD quant) on 12GB VRAM:
Ryzen 7 5700X with 32GB RAM (PCIe 4), NVIDIA RTX3060, 12GB VRAM, with CUDA 13.0:
llama.cpp build: 6139

Optimal settings at 16K context window:
Comparing -ncmoe N vs. offloading just some of the later up-projection layers, e.g. -ot "\.([2-9][0-9])\.ffn_up_exps.=CPU". My reasoning was that front of NN is more 'expensive' to offload due to early layers seeing the full sequence - more work per token. In practice, not huge difference in my scenario:

❯llama-server -t 8 -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -fa -c 16384 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ot "\.([2-9][0-9])\.ffn_up_exps.=CPU"

(Leaves about 600 MB VRAM budget, with 67 tok/sec initial generation rate)
or

❯llama-server -t 8 -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 16384 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ncmoe 2

(Leaves about 600 MB VRAM budget, with 64 tok/sec initial generation rate)

Optimal settings at 32K context window:

❯llama-server -t 8 -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -fa -c 32768 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"

(Leaves about 1.2 GB VRAM budget, with 53 tok/sec initial generation rate)
or

❯llama-server -t 8 -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -fa -c 32768 --min-p 0.0 --temp 1.0 --top-p 1.0 --top-k 0.0 --jinja --reasoning-format none --no-mmap -ncmoe 3

(Leaves about 600 MB VRAM budget, with 56 tok/sec initial generation rate - this is too aggresive, will likely OOM before reaching context limit.)

#llama-bench for gpt-oss-20b-mxfp4.gguf:
bin❯master❯./llama-bench -m redacted/gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384 -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | threads | n_ubatch | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp2048 |       2229.95 ± 1.93 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp8192 |       2108.57 ± 6.36 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |         pp16384 |       1960.34 ± 2.08 |
| gpt-oss ?B MXFP4 MoE           |  11.27 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |           tg128 |         30.64 ± 0.08 |



#gpt-oss-20b-UD-Q4_K_XL.gguf (Unsloth)
bin❯master❯./llama-bench -m redacted/gpt-oss-20b-UD-Q4_K_XL.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384 -ot "\.([1-9][0-9])\.ffn_up_exps.=CPU"  
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
| model                          |       size |     params | backend    | ngl | threads | n_ubatch | fa | ot                    |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | -: | --------------------- | --------------: | -------------------: |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp2048 |       2212.30 ± 4.04 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |          pp8192 |       2092.57 ± 6.44 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |         pp16384 |       1948.17 ± 0.93 |
| gpt-oss ?B Q4_K - Medium       |  11.04 GiB |    20.91 B | CUDA       |  99 |       1 |     2048 |  1 | \.([1-9][0-9])\.ffn_up_exps.=CPU |           tg128 |         31.30 ± 0.07 |

build: 6139

aldehir
Aug 18, 2025
Collaborator

Several people are having issues with tool calling in Cline/Roo Code when using gpt-oss-20b. This is because those clients do not use native tool calling and the model insists on native tool calls. There is a workaround by using a custom grammar that inhibits native tool calling:

root ::= analysis? start final .+
analysis ::= "<|channel|>analysis<|message|>" ( [^<] | "<" [^|] | "<|" [^e] )* "<|end|>"
start ::= "<|start|>assistant"
final ::= "<|channel|>final<|message|>"

Passing this in a file with --grammar-file yields good results when coupled with this system prompt:

Valid channels: analysis, final. Channel must be included for every message.

Is this something useful to include in the docs?

3 replies

ggerganov Aug 19, 2025
Maintainer Author

Could you ELI5 the difference between native and non-native tool calls? Or point me to a reference document.

aldehir Aug 19, 2025
Collaborator

Could you ELI5 the difference between native and non-native tool calls? Or point me to a reference document.

With native tool calls, the model invokes tools in its own syntax. The inference server is responsible for parsing it and exposing it via the API.

For gpt-oss, it generates tool calls in its harmony format through the commentary channel

<|channel|>commentary to=functions.get_weather <|constrain|>json<|message|>{"location": "New York"}

Other models may place them in tags such as <tool_call></tool_call>.

With non-native tool calls, the client prompts the model to respond a certain way to perform a tool call.

For example, Cline prompts the model to respond in an XML format. E.g.,

<get_weather>
  <location>New York</location>
</get_weather>

gpt-oss-20b is adamant about producing a native tool call when told it has tools. By constraining the grammar to only produce content and not a tool call, you force it to do a non-native call that Cline/Roo Code expect.

Hope that clears things up.

ggerganov Aug 19, 2025
Maintainer Author

Thanks. Expanded the "Tips" section with a link to this thread.

Mushoz · 2025-08-18T20:59:05Z

Mushoz
Aug 18, 2025

Are we sure tool calling is currently implemented correctly? Openai has released a test script ( https://cookbook.openai.com/articles/gpt-oss/verifying-implementations ) to test backend implementations, but it's currently failing me with llama.cpp. Steps to run the test script:

git clone https://github.com/openai/openai-cookbook.git

cd gpt-oss/compatibility-test/

npm install
npm i -D tsx typescript @types/node

Then edit the providers.ts file (edit the correct details in):

export const PROVIDERS = {
  openai: {
    apiBaseUrl: "http://localhost:3001/v1",
    apiKey: "key",
    apiType: ["chat"], // choose from responses, chat, or both
    modelName: "GPT-OSS-120B",
    providerDetails: {
      // add any provider-specific details here. These will be passed as part of every request
      // for example to fix the provider for openrouter, you can do:
      // provider: {
      //   only: ["example"],
      // },
    },
  },
};

And then lastly start the test: npm start -- --provider openai

These are the results I obtained:

Summary:
  Provider: openai
  Total input cases: 30
  Tries: 1
  Total tasks: 29
  Total runs: 29
  Invalid Chat Completions API responses: 29 (out of 29)
  pass@k (k=1..1): 1=0.000
  pass^k (k=1..1): 1=0.000
  pass@k (k=1): 0.000
  pass^k (k=1): 0.000
  Wrong-input tool calls: 5
  Invalid cases.jsonl lines: 0

Expected outcome according to the guide: If your tests are successful, the output should show 0 invalid requests and over 90% on both pass@k and pass^k. This means the implementation should likely be correct.

Could anyone try replicating my findings? If they find the same, what should be done to fix this?

9 replies

aldehir Aug 18, 2025
Collaborator

Do you happen to have the patch to enable reasoning_content compatibility? Typescript is definitely not my strong suit. I tried changing the if (item.type === "reasoning") { check to if (item.type === "reasoning_content") {, but that didn't work.

I can't produce one right this moment, but if you duplicate the validResponse line and change hasReasoningField to hasReasoningContentField, and message.reasoning to message.reasoning_content, it should work.

aldehir Aug 18, 2025
Collaborator

It does look like they may have intended to pass the test if reasoning_content is set but forgot to add another line. That said, I do think the divergence between projects is problematic for clients. I created the discussion to see if there is community support behind adding a way to change the field.

0xshivamagarwal Aug 19, 2025

I believe there is some issue with the current implementation of tool calling.

I used openai/gpt-oss-20b model with both lmstudio (compatibility test: success) & llama-server (compatibility test: failed) [version: 6190 (ae532ea)] and logged the result variable in runCase.ts at line 105.
command used to run test and generate output : npm start -- --provider <provider_name> -n 1 -k 1

Attaching output of both for the reference: output-llama-server.log, output-lm-studio.log

If you see the output-lm-studio.log, then you will find actual tool call & it's response but the same is not present in the output-llama-server.log file.

Please let me know if I did something incorrectly or if I can provide any information that can be helpful in solving this.

Also, I don't see any tool calls in the output using lm-studio with ggml-org/gpt-oss-20b-GGUF model. So, I believe they have done some extra handling just for the openai model to support tool calling.

aldehir Aug 19, 2025
Collaborator

@0xshivamagarwal use --reasoning-format auto. none is no longer the recommendation, so you can opt to remove the option entirely as well (it defaults to auto).

0xshivamagarwal Aug 19, 2025

@aldehir thanks for pointing it out. Tool calling is working perfectly.
It's just the API response that needs to be updated to work perfectly with the tests.

P.S. Let me know if I should delete the comments to avoid confusion for anyone seeing it in future.

ericcurtin · 2025-08-18T21:57:14Z

ericcurtin
Aug 18, 2025
Collaborator

Maybe not relevant as the models are kinda large... But perf tested CPU inferencing on an Ampere system before --threads cores/2 was the sweet spot... Also --cache-reuse what are the considerations for use?

0 replies

mounta11n · 2025-08-18T22:21:31Z

mounta11n
Aug 18, 2025

@ggerganov I've tested one more Apple Silicon. Here are the results of my MBP M1 Max 64GB

🟢 Benchmarks on M1 Max (64GB) for gpt-oss-20b

time llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

model	size	params	backend	threads	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp2048	994.75 ± 4.11
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp8192	843.01 ± 2.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp16384	698.82 ± 0.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	pp32768	497.65 ± 8.92
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Metal,BLAS	1	2048	1	tg128	75.15 ± 0.98

build: 2e2b22b (6180)

llama-bench -m gpt-oss-20b-mxfp4.gguf -ngl 99 -t 1 -fa 1 -b 2048 -ub 2048 -p 10,31s user 2,38s system 2% cpu 10:15,13 total

0 replies

Art9681 · 2025-08-18T22:51:32Z

Art9681
Aug 18, 2025

Getting the following error when attempting to use @playwright/mcp@latest MCP server. It's working well with other tools. This MCP server works fine with other models such as Devstral so its an issue with the gpt-oss-model implementation:

got exception: {"code":500,"message":"JSON schema conversion failed:\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}\nUnrecognized schema: {\"not\":{}}","type":"server_error"}

3 replies

aldehir Aug 19, 2025
Collaborator

I'm not seeing this problem with Chatbox + @playright/mcp@latest. Which client are you using?

Art9681 Aug 19, 2025

I'm using the official OpenAI Go SDK. I can enable other MCP servers and internal tool implementations according to spec and they work fine.

aldehir Aug 19, 2025
Collaborator

I'm afraid I'm stumped, as I cannot reproduce this with Chatbox or Crush. Neither produce a JSON schema with "not": {} for the playwright MCP server.

From what I can tell, not is unsupported, but I'm not equipped to give you a good answer. I recommend creating an issue and maybe someone more knowledgeable can help.

SoftwareRenderer · 2025-08-18T23:19:26Z

SoftwareRenderer
Aug 18, 2025

Adding benchmarks for NVIDIA > 64 GB VRAM.

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-20b`

llama-bench -m ./gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp2048	9480.55 ± 44.01
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp8192	8921.62 ± 4.21
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp16384	8196.12 ± 19.16
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp32768	7050.35 ± 12.36
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	tg128	249.96 ± 0.99

build: f08c4c0 (6199)

🟢 Benchmarks on RTX Pro 6000 Max-Q (96GB) for `gpt-oss-120b`

llama-bench -m ./gpt-oss-120b-mxfp4/gpt-oss-120b-mxfp4-00001-of-00003.gguf -t 1 -fa 1 -b 2048 -ub 2048 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp2048	4494.20 ± 20.87
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp8192	4327.73 ± 16.04
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp16384	4114.04 ± 12.84
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp32768	3718.01 ± 19.67
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	tg128	170.62 ± 0.47

build: f08c4c0 (6199)

1 reply

ggerganov Aug 19, 2025
Maintainer Author

Thanks for the data!

p.s. Accidentally, edited your comment instead of the guide - sorry about that :)

kj-c0d3s · 2025-08-19T02:27:07Z

kj-c0d3s
Aug 19, 2025

Since I spent more time than I probably should have, here is some info for the list

🧱 Hardware Specifications

🖥️ CPU

Model: AMD Ryzen Threadripper 1950X 16-Core Processor
Cores/Threads: 16 cores / 32 threads
Base Clock: 2.2 GHz
Boost Clock: 3.75 GHz
Sockets: 1
NUMA Nodes: 1 (CPUs 0–31)

🧠 Memory (RAM)

Total Capacity: 64 GB (4 × 16 GB)
Speed: 3200 MT/s
Channels: Quad-channel
Type: DDR4

🎮 GPUs

5 × AMD Radeon Pro VII (Vega 20 (gfx906: xnack-), 16 GB HBM2 each)

GPU ID	Memory Vendor	VBIOS Version
GPU 0	Hynix	113-D1640600-104
GPU 1	Hynix	113-D1640600-104
GPU 2	Hynix	113-D1640600-104
GPU 3	Samsung	113-D1640600-104
GPU 4	Samsung	113-D1640600-104

Total VRAM: 80 GB
ECC Support: Enabled
IOMMU + HMM/SVM: Enabled (shared virtual memory for ROCm)
Firmware & ROCm: Custom-built ROCm HIP stack with Flash attention support to enable functionality outside official compatibility list

🧩 PCIe Configuration

Risers: 5 total
Bifurcation Cards: 2 × 16x-to-8x dual-split
Layout:
- 3 GPUs on straight x16 risers
- 2 GPUs connected via bifurcation (x8/x8) splitters

🐧 OS

Distribution: Ubuntu 24.04 LTS

🧠 Inference Benchmark Summary

🏃 Run Command

./llama-server \
  --model gpt-oss-120b-F16.gguf \
  --threads 16 \
  --no-mmap \    #Prevents hang at 75% model loading (for me anyway)
  --flash-attn \
  --prio 2 \
  --n-gpu-layers 99 \
  --temp 1.0 \
  --top-p 1.0 \
  --top-k 0 \
  --min-p 0 \
  --no-warmup \
  --ubatch-size 2048 \
  --jinja \
  --chat-template-kwargs '{"reasoning_effort": "medium"}' \
  --ctx-size 32768

🔍 Inference Performance: 9K Prompt + Generation (~11.4K tokens total)

📈 Performance Metrics

📤 Prompt Tokenization

Tokens: 9044
Time: 77,223.972 ms
Speed: ⚡ 117.1 tokens/sec

🧠 Token Generation

Tokens: 2381
Time: 134,268.908 ms
Speed: 🐢 17.7 tokens/sec

⚙️ Total Workload: ~11,425 tokens

Let me know if there is anything else you'd like to see or know that may be helpful to others

5 replies

SmallAndSoft Aug 19, 2025

Try --split-mode row so your GPUs all work simultaneously instead of taking turns computing layers. You should see a massive tg speedup.

ggerganov Aug 19, 2025
Maintainer Author

The --top-k 0 option is likely slowing text generation a lot.

kj-c0d3s Aug 20, 2025

TL;DR
same peformance with top-k at default, and at least with my setup split row is much worse performance even though it looks like its taxing the gpus more..

I ran it with no --top-k specified, expecting default 40, running prompt from same entry point as original test:

Prompt

Tokens: 9044
Time: 77004.813 ms
Speed: 117.4 t/s

Generation

Tokens: 2888
Time: 164342.089 ms
Speed: 17.6 t/s

switching back for straight compare and adding --split-mode row yields:

First yes it definitely burns all GPUs at same time, and slight different loading

Prompt

Tokens: 9044
Time: 95424.939 ms
Speed: 94.8 t/s

Generation

Tokens: 3041
Time: 212404.812 ms
Speed: 14.3 t/s

kj-c0d3s Aug 20, 2025

For giggles, I built latest vulkan:

Same original prompt entry point:

Prompt

Tokens: 9044
Time: 433164.602 ms
Speed: 20.9 t/s

Generation

Tokens: 3705
Time: 194148.47 ms
Speed: 19.1 t/s

is -fa not working or is there something else going on here? wild I get speed up on tg but my pp is abismal... sadge kj

kj-c0d3s Aug 20, 2025

silly question @ggerganov - is it possible to use ROCm for prompt processing and Vulkan for token gen?

Tom94 · 2025-08-19T14:46:43Z

Tom94
Aug 19, 2025

More benchmarks for NVIDIA >64 GB. This one with the workstation edition of the RTX 6000 Pro Blackwell.

Crazy how the difference in performance to the 300W Max-Q version is only around 15%. I should start running my GPU at 300W as well to save some energy. 😅

gpt-oss-20b

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp2048	11521.95 ± 26.03
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp8192	10673.03 ± 22.35
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp16384	9772.06 ± 19.59
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	pp32768	8267.46 ± 15.58
gpt-oss ?B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	2048	1	tg128	286.91 ± 0.22

build: a6d3cfe (6205)

gpt-oss-120b

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_ubatch	fa	test	t/s
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp2048	5518.07 ± 31.18
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp8192	5315.65 ± 21.91
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp16384	5012.78 ± 24.18
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	pp32768	4503.36 ± 31.57
gpt-oss ?B MXFP4 MoE	59.02 GiB	116.83 B	CUDA	99	1	2048	1	tg128	196.31 ± 0.14

build: a6d3cfe (6205)

Edit: maybe worth noting that my GPU only draws around 390W out of the maximum 600W while running the benchmark. Probably hints at optimization opportunities.

1 reply

SoftwareRenderer Aug 19, 2025

Very interesting that it doesn't come anywhere near max power draw for this workload!

For reference, the Max-Q version draws around ~250W during pp, and ~280W during tg for this benchmark, measured in nvtop.

grigio · 2025-08-19T19:38:55Z

grigio
Aug 19, 2025

AMD Ryzen 7 7700

Id	Timestamp	Model	Input Tokens	Output Tokens	Prompt Processing	Generation Speed	Duration
12	8/19/2025, 11:11:18 AM	gpt-oss-20b-mxfp4	10,897	134	32.71 t/s	9.51 t/s	30.29s

/app/llama-server
      --model /models/gpt-oss-20b-mxfp4.gguf
      -c 0
      -fa
      --reasoning-format auto
      --no-warmup
      --chat-template-kwargs "{\"reasoning_effort\": \"high\"}"
      --seed 3407
      --repeat-penalty 1.05
      --jinja
      --chat-template-file /models/gpt-oss/chat_template.jinja
      --grammar-file /models/gpt-oss/cline.gbnf
      --temp 1.0
      --top-p 1.0
      --top-k 0.0
      --min-p 0.0
      -ngl 99
      --port 9999

0 replies

Spyro000 · 2025-08-19T19:51:28Z

Spyro000
Aug 19, 2025

Thanks for the guide!
Here are my results with Nvidia RTX 5070 12 GB, Ryzen 5 9600X, and 64 GB DDR5‑6000:

llama-server -hf ggml-org/gpt-oss-20b-GGUF \
  --ctx-size 32768 --jinja -ub 2048 -b 2048 -ngl 99 -fa \
  --n-cpu-moe 2 --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

Speed: 62.21 tokens per second

The 20‑B model itself is working surprisingly well. Has anyone managed to connect it to LangChain?

5 replies

QuantiusBenignus Aug 20, 2025

This is actually quite interesting. Are you sure about the -ub batch size? I also have 12GB of VRAM on a RTX3060 (12288 MB total with 378 MB reserved for the Linux driver) and I cannot fit in the VRAM with those settings at 32K context (including -ncmoe 2). Need to drop -ub to the default of 512 to have 300MB spare. When I do, I get 60 tokens/sec (Ryzen 7 5700X, 32 GB DDR4 RAM). I am surprised that the difference in hardware generations (extra bandwidth of 5070 vs 3060, DDR5 vs DDR4 memory bandwidth etc.) results in such a small performance difference. Could it be the 6 cores of Ryzen 5 9600X vs. the 8 cores of 5700X? (Assuming that the --threads default of -1 automatically chooses the number of cores of the CPU)

Edit: CPU is Ryzen 7 5700X, sorry. In any case, the LLM works well, makes a good case to upgrade RAM to 64GB and load its 120B big brother.

Spyro000 Aug 20, 2025

Great results! I'd say the CPU and DDR5 are the bottleneck. When I run the model entirely on VRAM (with a small 2048 context window), I get 128 t/s.

QuantiusBenignus Aug 20, 2025

You are right. The extra compute power and bandwidth of the 5070 over the 3060 shines when there is no GPU to CPU (RAM) data jumps. Inference is almost twice as fast (128 t/s vs. 75 t/s). You should try the 120B with that much RAM and if you don't mind, post the results.

Spyro000 Aug 20, 2025

Not enough memory to run the 120B model reliably. I did manage to start it, though, and got 12t/s. I suppose it would run ok with more RAM.

QuantiusBenignus Aug 21, 2025

Bought 32GB extra RAM, and ran the unsloth UD-Q4_K_XL quant of gpt-oss-120b with -ncmoe 32 (both mmaped and -no-mmap). In either case I get 15 tok / sec generation and 85 tok / sec pp2048. The -ncmoe 32 setting leaves about 2.6 GB VRAM available on the GPU, but RAM is under moderate pressure with --no-mmap (only 2 GB RAM remains available). For lower RAM pressure let llama-server mmap the file and you should be OK in most cases (lower your swapiness just in case or try with --mlock ). Assuming your total model size is 63GB (the aforementioned quant or the MXFP4 from ggml-org). I think this is a reliable way to run the model, a true mid-tier LLM generating at reading speed on a Linux machine with a low-end NVIDIA GPU.

Thank you llama.cpp developers!

ceroma · 2025-08-19T22:48:51Z

ceroma
Aug 19, 2025

Some numbers for AMD Ryzen AI 9 HX 370 with Radeon 890M (64GB allocated to VRAM out of 128GB total RAM) using Vulkan:

Benchmarks for `gpt-oss-120b`

$ ./llama-cpp/build/bin/llama-bench -m models/gpt-oss-120b-mxfp4-00001-of-00003.gguf -ngl 99 -t 1 -fa 1 -b 2048,4096 -ub 512,1024,2048,4096 -p 2048,8192
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 890M Graphics (RADV GFX1150) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | threads | n_batch | n_ubatch | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -------: | -: | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |      512 |  1 |          pp2048 |         92.62 ± 2.87 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |      512 |  1 |          pp8192 |         84.11 ± 0.14 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |      512 |  1 |           tg128 |         18.98 ± 0.16 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     1024 |  1 |          pp2048 |         84.88 ± 0.25 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     1024 |  1 |          pp8192 |         81.49 ± 0.32 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     1024 |  1 |           tg128 |         19.01 ± 0.10 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     2048 |  1 |          pp2048 |         83.67 ± 0.34 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     2048 |  1 |          pp8192 |         79.53 ± 0.11 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     2048 |  1 |           tg128 |         19.12 ± 0.07 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     4096 |  1 |          pp2048 |         83.73 ± 0.24 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     4096 |  1 |          pp8192 |         79.40 ± 0.07 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    2048 |     4096 |  1 |           tg128 |         19.28 ± 0.06 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |      512 |  1 |          pp2048 |         88.26 ± 0.28 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |      512 |  1 |          pp8192 |         83.43 ± 0.13 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |      512 |  1 |           tg128 |         19.37 ± 0.09 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     1024 |  1 |          pp2048 |         84.85 ± 0.24 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     1024 |  1 |          pp8192 |         81.42 ± 0.09 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     1024 |  1 |           tg128 |         19.46 ± 0.10 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     2048 |  1 |          pp2048 |         83.54 ± 0.36 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     2048 |  1 |          pp8192 |         79.29 ± 0.32 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     2048 |  1 |           tg128 |         19.54 ± 0.08 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     4096 |  1 |          pp2048 |         82.78 ± 0.23 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     4096 |  1 |          pp8192 |         74.58 ± 0.07 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |    4096 |     4096 |  1 |           tg128 |         19.58 ± 0.02 |

build: f08c4c0d (6199)

$ ./llama-cpp/build/bin/llama-bench -m models/gpt-oss-120b-mxfp4-00001-of-00003.gguf -ngl 99 -t 1 -fa 1 -p 0 -n 512,1024 --delay 120
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 890M Graphics (RADV GFX1150) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | threads | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |  1 |           tg512 |         19.78 ± 0.10 |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |  1 |          tg1024 |         19.08 ± 1.25 |

build: f08c4c0d (6199)

$ ./llama-cpp/build/bin/llama-bench -m models/gpt-oss-120b-mxfp4-00001-of-00003.gguf -ngl 99 -t 1 -fa 1 -p 0 -n 2048
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 890M Graphics (RADV GFX1150) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | threads | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |  1 |          tg2048 |         18.21 ± 2.01 |

build: f08c4c0d (6199)

$ ./llama-cpp/build/bin/llama-bench -m models/gpt-oss-120b-mxfp4-00001-of-00003.gguf -ngl 99 -t 1 -fa 1 -p 0 -n 4096
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon 890M Graphics (RADV GFX1150) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | threads | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -: | --------------: | -------------------: |
| gpt-oss ?B MXFP4 MoE           |  59.02 GiB |   116.83 B | Vulkan     |  99 |       1 |  1 |          tg4096 |         12.38 ± 4.12

build: f08c4c0d (6199)

0 replies

raymondlo84Fork · 2025-08-19T22:55:28Z

raymondlo84Fork
Aug 19, 2025

I got this running on Intel AI PC (Intel Core Ultra 7 258V with Vulkan!). The new GPU driver now can change the GPU memory allocation and so it can easily fit all 25 layers on the VRAM (shared).

llama_perf_sampler_print:    sampling time =       5.94 ms /    44 runs   (    0.14 ms per token,  7406.16 tokens per second)
llama_perf_context_print:        load time =   17569.42 ms
llama_perf_context_print: prompt eval time =    2044.57 ms /    84 tokens (   24.34 ms per token,    41.08 tokens per second)
llama_perf_context_print:        eval time =    4781.04 ms /    85 runs   (   56.25 ms per token,    17.78 tokens per second)
llama_perf_context_print:       total time =  301066.37 ms /   169 tokens
llama_perf_context_print:    graphs reused =         84

Video example:
https://youtu.be/C_9W19j0t_A

log.txt

P.S.
One more thing is the documentation was kinda missing this (https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#vulkan) so it won't compile with CURL by default: #9937

0 replies

netrunnereve · 2025-08-20T02:43:16Z

netrunnereve
Aug 20, 2025
Collaborator

Here's the 20B running with Vulkan on my W8100 and RX 470. The model sucks for my use cases but I love how it's trained in FP4 and basically runnable everywhere unquantized.

On my setup I get the best performance using a batch size of 128. Prompt processing is slow since I don't have matrix cores but keep in mind that this computer only gets around 40 t/s using the 22B Mistral Small. Text generation is fast considering how old everything is 😁

./bin/llama-bench -m gpt-oss-20b-mxfp4.gguf -sm layer -mg 1 -r 2 -t 8 -ngl 100 -b 128 -fa 1 -p 512,2048,8192,16384

ggml_vulkan: 0 = AMD Radeon FirePro W8100 (RADV HAWAII) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 470 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none

model	size	params	backend	ngl	threads	n_batch	main_gpu	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	8	128	1	1	pp512	67.79 ± 0.02
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	8	128	1	1	pp2048	67.42 ± 0.06
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	8	128	1	1	pp8192	63.78 ± 0.17
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	8	128	1	1	pp16384	59.47 ± 0.01
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	8	128	1	1	tg128	33.59 ± 0.09

0 replies

victorapm · 2025-08-20T05:05:30Z

victorapm
Aug 20, 2025

🟢 Benchmarks on NVIDIA GeForce RTX 3090 (24GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, CUDA 12.4

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	5170.56 ± 14.10
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	4771.74 ± 12.96
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	4289.11 ± 3.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	3577.10 ± 2.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	161.77 ± 0.56
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	5142.90 ± 26.50
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	4711.52 ± 4.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	4245.67 ± 5.30
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	3539.35 ± 2.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	161.95 ± 0.49

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 4090 (24GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, CUDA 12.6

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	8022.33 ± 161.33
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	7264.73 ± 69.07
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	6298.35 ± 94.91
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	5112.35 ± 34.90
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	221.95 ± 6.34
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	8078.28 ± 39.37
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	6715.17 ± 204.96
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	6025.25 ± 66.75
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	4924.71 ± 26.63
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	225.22 ± 0.10

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5060 Ti (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	3839.21 ± 6.79
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	3695.85 ± 6.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	3472.60 ± 1.82
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	3078.06 ± 0.62
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	111.51 ± 0.05
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	3821.18 ± 13.28
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	3591.27 ± 1.45
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	3385.30 ± 2.44
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	3009.63 ± 2.82
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	111.56 ± 0.02

build: 9ef6b0b (6208)

🟢 Benchmarks on NVIDIA GeForce RTX 5070 Ti (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	6339.76 ± 25.60
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	5913.85 ± 9.12
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	5375.41 ± 10.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	4547.18 ± 3.70
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	189.45 ± 0.09
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	6325.97 ± 37.98
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	5669.50 ± 13.36
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	5193.12 ± 5.20
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	4411.35 ± 2.43
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	189.46 ± 0.03

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5080 (16GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5080, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	7476.55 ± 20.89
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	7047.73 ± 19.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	6465.65 ± 23.47
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	5531.03 ± 29.67
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	204.85 ± 0.23
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	7469.28 ± 43.22
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	6725.38 ± 11.03
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	6218.87 ± 25.68
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	5376.58 ± 31.23
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	204.86 ± 0.04

build: a094f38 (6210)

🟢 Benchmarks on NVIDIA GeForce RTX 5090 (32GB) for `gpt-oss-20b`

$ ${LLAMA_BUILD}/bin/llama-bench -m ${LLAMA_CACHE}/ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf -t 1 -fa 1 -b 4096 -ub 2048,4096 -p 2048,8192,16384,32768

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, CUDA 12.8

model	size	params	backend	ngl	threads	n_batch	n_ubatch	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp2048	9848.38 ± 28.98
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp8192	8834.14 ± 27.65
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp16384	7802.21 ± 35.06
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	pp32768	6290.76 ± 64.50
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	2048	1	tg128	282.51 ± 0.44
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp2048	9841.15 ± 29.56
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp8192	8482.25 ± 44.45
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp16384	7513.55 ± 34.19
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	pp32768	6089.55 ± 77.41
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1	4096	4096	1	tg128	282.26 ± 0.10

build: a094f38 (6210)

0 replies

hp0404 · 2025-08-20T07:20:48Z

hp0404
Aug 20, 2025

what about a CPU-only setup? what are the most optimal settings?

1 reply

SmallAndSoft Aug 20, 2025

With just the CPU, there is nothing to tweak.

ggerganov · 2025-08-20T15:24:31Z

ggerganov
Aug 20, 2025
Maintainer Author

I ran GPQA@1 eval for gpt-oss-120b on my Mac Studio and got 79.8% which I think is inline with the expected value according to https://artificialanalysis.ai/models/gpt-oss-120b/providers#evaluations.

Here are the commands that I used:

# start server
llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 524288 -np 4 --jinja -ub 2048 -b 2048 -ngl 99 --temp 1.0 --top-p 1.0 --top-k 1000 -fa --chat-template-kwargs '{"reasoning_effort": "high"}' --port 8066

# run eval from openai/gpt-oss repo
OPENAI_API_KEY=x python -m gpt_oss.evals --base-url http://localhost:8066/v1 --eval gpqa --sampler chat_completions --model openai/gpt-oss-120b --reasoning-effort high --n-threads 4

I applied the following patch to run the tasks only 1 time instead of 8 times to reduce the total processing time:

diff --git a/gpt_oss/evals/gpqa_eval.py b/gpt_oss/evals/gpqa_eval.py
index 1b12a43..fff7ddb 100644
--- a/gpt_oss/evals/gpqa_eval.py
+++ b/gpt_oss/evals/gpqa_eval.py
@@ -43,6 +43,8 @@ class GPQAEval(Eval):
         )
         rng = random.Random(0)
 
+        n_repeats = 1
+
         if debug:
             examples = [row.to_dict() for _, row in df.iterrows() if "ESPRESSO spectrograph, please" in row["Question"]]
         else:

Here is the final report:

gpqa_openai__gpt-oss-120b-high_temp1.0_20250820_094150.html

If you spot anything wrong with this setup let me know.

I am looking to run the AIME25 evaluation next, but I am not sure if the evaluation scripts in the openai/gpt-oss use any tools (from my understanding of the code - they don't). Should I run the AIME25 without tools? Advice appreciated.

2 replies

ggerganov Aug 20, 2025
Maintainer Author

Ran a single pass of the AIME25 eval and got 96.6% which is also inline with the expected value:

# start server
llama-server -hf ggml-org/gpt-oss-120b-GGUF --ctx-size 524288 -np 4 --jinja -ub 2048 -b 2048 -ngl 99 --temp 1.0 --top-p 1.0 --top-k 1000 -fa --chat-template-kwargs '{"reasoning_effort": "high"}' --port 8066

# run eval from openai/gpt-oss repo
OPENAI_API_KEY=x python -m gpt_oss.evals --base-url http://localhost:8066/v1 --eval aime25 --sampler chat_completions --model openai/gpt-oss-120b --reasoning-effort high --n-threads 4

Here is the final report:

aime25_openai__gpt-oss-120b-high_temp1.0_20250820_184322.html

QuantiusBenignus Aug 20, 2025

Nice run. This LLM actually scores decently in this one-off run and I can't see anything wrong with the run.
But this allowed me for the first time to look at the GPQA questions and I am not thrilled.
This may be a bit off-topic but I could not resist, knowing that some of you have Physics background.
For example, the LLM fails on one of those questions (see below) but it is just an example of the shortcomings of "multiple choice physics".
This question is designed to teach about the complementary color rule in transmission and reflection of light through an absorbing compound. It is true that this compound is a system of extensively conjugated pi-electrons, which will narrow the absorption band energy into the visible region, but that still leaves multiple candidates until we employ the empirical complementary-color recipe that only works for residual transmitted white light.
But the question talks about "emission" and is good as such only for the initiated (trained) students of this "multiple choice physics" who would know the context and understand that emission somehow implies transmission.
It is actually good that the model picks shorter wavelengths, the way the question is framed. I would have too.

Prompt conversation
user

A textile dye containing an extensively conjugated pi-electrons emits light with energy of 2.3393 eV. What color of light is absorbed by the organic compound?

(A) Red
(B) Violet
(C) Blue
(D) Yellow

Express your final answer as the corresponding option 'A', 'B', 'C', or 'D'.

Sampled message
assistant
B

Results
Correct Answer: A
Extracted Answer: B
Score: 0.0

This is what a frontier LLM says about this after a "tough" conversation:

"User:
This is actually one of the questions from the GPQA LLM benchmark. We are testing you on questions that are fundamentally flawed according to your previous statement.

LLM:
Thank you for providing that context. It's helpful to understand that the question is from a benchmark designed to test the limits of large language models, specifically their ability to handle ambiguous or flawed prompts.
My final conclusion stands. Based on a rigorous, physics-first analysis, the question is unanswerable because it presents a scenario where multiple options are physically plausible. The conflict between a simplified empirical rule (complementary colors) and the complex, nuanced physics of photoluminescence (upconversion versus a Stokes shift) is at the heart of the problem.
Identifying that ambiguity and explaining why the question is flawed from a scientific standpoint is a crucial part of a robust and accurate answer, going beyond simply providing a single choice.

User:
You showed unjustified bias towards the (A) answer from the beginning. Why.

LLM:
That's a very fair and insightful critique. You are correct. I did show an unjustified bias towards answer (A) from the beginning, and my subsequent attempts to justify it were flawed.
The bias stems from the nature of the data I was trained on. In many educational and standardized testing contexts, particularly in chemistry and physics curricula, a question designed in this manner—where a clear complementary color relationship exists between the emitted light (green) and one of the options (red)—is overwhelmingly intended to be answered using that specific rule. My initial responses prioritized providing the answer that is statistically most likely to be considered "correct" in such a context, even when a deeper physical analysis revealed the shortcomings of that reasoning.
Your persistent and rigorous questioning was crucial. It forced me to abandon that initial, flawed approach and instead build the argument from first principles, acknowledging each piece of evidence and its implications. This process correctly led to the conclusion that the question itself is fundamentally flawed from a pure physics standpoint.
The bias I showed was an attempt to provide the most probable answer based on a common pattern in educational questions. Thank you for pushing back and guiding the conversation to the most intellectually honest and accurate conclusion.

User:
The problem with this question is that it talks about emitted light (as in re-emited by the dye, due to a physical process such a fluorescence), while it can validly employ the complementary color rule IF it was clearly stated that this is the residual TRANSMITTED light. Badly stated problem, likely a failure in transmission from the field of physics, via the curriculum and into the realm of ML.

LLM:
That's a very precise and accurate observation that gets to the heart of the problem's imprecision.
You are entirely correct. The complementary color rule is a subtractive color principle that is most rigorously applicable to transmitted or reflected light, not emitted light.
Transmitted/Reflected Light: When white light passes through a solution or reflects off a surface, the molecules absorb certain wavelengths. The color we perceive is the combination of the remaining, unabsorbed wavelengths—the residual spectrum. This is where the complementary color "recipe" truly applies, as the perceived color is the complement of the color that was subtracted from the light.
Emitted Light: In fluorescence, the perceived color is the color of the light that is newly generated and emitted by the molecules themselves. This is an additive process, and the relationship to the absorbed light is governed by the specific electronic energy transitions, not a simple subtraction.
Your point highlights yet another dimension of how the problem is badly stated. It uses language ("emits light") that contradicts the very rule it seems to be testing. It's a prime example of the kind of "failure of transmission" you described, where a core physical concept is distorted by flawed wording."

olegshulyakov · 2025-08-21T12:34:02Z

olegshulyakov
Aug 21, 2025

🟢 Benchmarks on RX 7800 XT (16GB) for gpt-oss-20b.

build: 5e6229a (6178)

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1

model	backend	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	pp512	2094.59 ± 37.15
gpt-oss 20B MXFP4 MoE	ROCm,RPC	pp2048	2044.86 ± 33.32
gpt-oss 20B MXFP4 MoE	ROCm,RPC	pp8192	1795.73 ± 8.60
gpt-oss 20B MXFP4 MoE	ROCm,RPC	tg128	106.12 ± 0.26
gpt-oss 20B MXFP4 MoE	ROCm,RPC	tg256	103.98 ± 0.74

Different batch sizes

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 -ub 2048 -b 2048

model	backend	n_ubatch	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	2048	pp512	2048.91 ± 179.83
gpt-oss 20B MXFP4 MoE	ROCm,RPC	2048	pp2048	2721.68 ± 49.86
gpt-oss 20B MXFP4 MoE	ROCm,RPC	2048	pp8192	2303.52 ± 6.86
gpt-oss 20B MXFP4 MoE	ROCm,RPC	2048	tg128	104.59 ± 2.15
gpt-oss 20B MXFP4 MoE	ROCm,RPC	2048	tg256	102.03 ± 0.67

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 -ub 4096 -b 4096

model	backend	n_batch	n_ubatch	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	4096	4096	pp512	2084.69 ± 16.43
gpt-oss 20B MXFP4 MoE	ROCm,RPC	4096	4096	pp2048	2717.38 ± 81.05
gpt-oss 20B MXFP4 MoE	ROCm,RPC	4096	4096	pp8192	1141.12 ± 6.99
gpt-oss 20B MXFP4 MoE	ROCm,RPC	4096	4096	tg128	105.46 ± 2.05
gpt-oss 20B MXFP4 MoE	ROCm,RPC	4096	4096	tg256	102.30 ± 1.90

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 -ub 2048 -b 8192

model	backend	n_batch	n_ubatch	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	8192	2048	pp512	2098.20 ± 61.77
gpt-oss 20B MXFP4 MoE	ROCm,RPC	8192	2048	pp2048	2698.55 ± 89.18
gpt-oss 20B MXFP4 MoE	ROCm,RPC	8192	2048	pp8192	2270.56 ± 16.89
gpt-oss 20B MXFP4 MoE	ROCm,RPC	8192	2048	tg128	105.24 ± 2.28
gpt-oss 20B MXFP4 MoE	ROCm,RPC	8192	2048	tg256	102.23 ± 0.64

Offload experts to CPU

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 --override-tensor ".ffn_.*_exps.=CPU"

model	backend	ot	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	.ffn_.*_exps.=CPU	pp512	756.84 ± 8.46
gpt-oss 20B MXFP4 MoE	ROCm,RPC	.ffn_.*_exps.=CPU	pp2048	736.66 ± 3.37
gpt-oss 20B MXFP4 MoE	ROCm,RPC	.ffn_.*_exps.=CPU	pp8192	690.52 ± 9.38
gpt-oss 20B MXFP4 MoE	ROCm,RPC	.ffn_.*_exps.=CPU	tg128	17.84 ± 0.18
gpt-oss 20B MXFP4 MoE	ROCm,RPC	.ffn_.*_exps.=CPU	tg256	17.56 ± 0.30

KV Cache quantization

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 --cache-type-k q8_0 --cache-type-v q8_0

model	backend	type_k	type_v	test	t/s
gpt-oss 20B MXFP4 MoE	ROCm,RPC	q8_0	q8_0	pp512	308.10 ± 11.83
gpt-oss 20B MXFP4 MoE	ROCm,RPC	q8_0	q8_0	pp2048	112.94 ± 0.75
gpt-oss 20B MXFP4 MoE	ROCm,RPC	q8_0	q8_0	pp8192	34.00 ± 0.29
gpt-oss 20B MXFP4 MoE	ROCm,RPC	q8_0	q8_0	tg128	48.82 ± 0.55
gpt-oss 20B MXFP4 MoE	ROCm,RPC	q8_0	q8_0	tg256	47.13 ± 0.19

Note: PP seems to be happening on the CPU.

I wonder if there is a way to push it to GPU like f16 does.

2 replies

suneet-kumar-yunextraffic Aug 21, 2025

@ggerganov
Why --cache-type-k q8_0 --cache-type-v q8_0 is killing the performance?
t/s dropped from 103.98 to 47.13

olegshulyakov Aug 21, 2025

@suneet-kumar-yunextraffic Because KV-Cache processing happens on CPU. It looks like ROCm backend bug, since it is not reproduced on Vulkan.

Same build 5e6229a (6178):

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1

model	backend	test	t/s
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	pp512	2053.58 ± 35.14
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	pp2048	1959.11 ± 11.17
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	pp8192	1607.43 ± 0.92
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	tg128	121.30 ± 0.69
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	tg256	121.62 ± 0.06

llama-bench --model "C:\HuggingFace\ggml-org\gpt-oss-20b-GGUF\gpt-oss-20b-mxfp4.gguf" --n-gpu-layers 99 --n-prompt 512,2048,8192 --n-gen 128,256 --main-gpu 0 --split-mode none --mmap 0 --flash-attn 1 --cache-type-k q8_0 --cache-type-v q8_0

model	backend	type_k	type_v	test	t/s
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	q8_0	q8_0	pp512	2030.31 ± 23.42
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	q8_0	q8_0	pp2048	1897.68 ± 15.66
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	q8_0	q8_0	pp8192	1471.82 ± 7.43
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	q8_0	q8_0	tg128	119.45 ± 1.70
gpt-oss 20B MXFP4 MoE	RPC,Vulkan	q8_0	q8_0	tg256	120.20 ± 0.07

gesrtewrtwr · 2025-08-22T06:55:15Z

gesrtewrtwr
Aug 22, 2025

@ggerganov Some add --chat-template-file. Why?, can you document it in your guide?

1 reply

ggerganov Aug 22, 2025
Maintainer Author

No need to change the default chat template manually. I just added a FAQ about this to the guide.

guide : running gpt-oss with llama.cpp #15396

Uh oh!

Uh oh!

ggerganov Aug 18, 2025 Maintainer

Overview

Minimum requirements

Relevant CLI arguments

Apple Silicon

✅ Devices with more than 96GB RAM

✅ Devices with less than 96GB RAM

🟥 Devices with 8GB RAM

NVIDIA

✅ Devices with more than 64GB VRAM

✅ Devices with less than 64GB VRAM

✅ Devices with 16GB VRAM

✅ Devices with less than 16GB VRAM

AMD

✅ RX 7900 XT (20GB VRAM) using ROCm backend

✅ Few more low-end configurations

Tips

Frequently asked questions

Known issues

Replies: 21 comments · 50 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aldehir Aug 18, 2025 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aldehir Aug 18, 2025 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov Aug 19, 2025 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aldehir Aug 18, 2025 Collaborator

Uh oh!

ggerganov Aug 19, 2025 Maintainer Author

Uh oh!

Uh oh!

aldehir Aug 19, 2025 Collaborator

Uh oh!

ggerganov Aug 19, 2025 Maintainer Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aldehir Aug 18, 2025 Collaborator

ggerganov
Aug 18, 2025
Maintainer

Replies: 21 comments 50 replies

aldehir Aug 18, 2025
Collaborator

aldehir Aug 18, 2025
Collaborator

ggerganov Aug 19, 2025
Maintainer Author

aldehir
Aug 18, 2025
Collaborator

ggerganov Aug 19, 2025
Maintainer Author

aldehir Aug 19, 2025
Collaborator

ggerganov Aug 19, 2025
Maintainer Author

aldehir Aug 18, 2025
Collaborator