tokens in gbnf

[dstoc]

I wrote this grammar: https://gist.github.com/dstoc/ab58a1829b3f504c64f08bee5e8c6ea6

But there are some problems:

not-end ::=
    [^<] |
    "<" [^|] |
    "<|" [^e] |
    "<|e" [^n] |
    "<|en" [^d] |
    "<|end" [^|] |
    "<|end|" [^>]

1. I think not-end, prevents the model from outputting <|end|> using different tokens, or ending with a prefix like <|en
2. I want to prevent other special tokens, not just <|end|>, but the grammar becomes unwieldy

Is there a better way to structure this?

Is there a way to incorporate token ids into the grammar?

e.g. something like:

not-end ::= [^\t200007]

[dstoc]

It seems like this is possible in llguidance! I will test the integration in llama.cpp

[dstoc]

seems something is not wired up correctly with gpt-oss and special tokens?

llg error: at 8(11): unknown special token: "<|channel|>"; following special tokens are available: ⟦<|return|>‧<|reserved_200017|>‧<|reserved_200016|>‧<|reserved_200015|>‧<|reserved_200014|>‧<|reserved_200013|>‧<|reserved_200011|>‧<|reserved_200010|>‧<|reserved_200009|>‧<|reserved_200004|>‧<|reserved_200001|>‧<|reserved_200000|>‧≺EOS≻‧<|endofprompt|>‧<|call|>⟧

[aldehir]

This might be a result of:

llama.cpp/src/llama-vocab.cpp

Lines 2343 to 2347 in 79bc429

	for (const auto & t : token_to_id) {
	if (t.first == "<|channel|>" || t.first == "<|message|>" || t.first == "<|start|>" || t.first == "<|constrain|>") {
	id_to_token[t.second].attr = LLAMA_TOKEN_ATTR_USER_DEFINED;
	}
	}

Without this, llama-cli doesn't properly work with gpt-oss. It might be affecting how it works with llguidance.

[dstoc]

% ./gguf-tokens.py ggml-org_gpt-oss-20b-GGUF_gpt-oss-20b-mxfp4.gguf
199998  [CONTROL     ] '<|startoftext|>' (bos)
199999  [CONTROL     ] '<|endoftext|>' (padding)
200000  [CONTROL     ] '<|reserved_200000|>'
200001  [CONTROL     ] '<|reserved_200001|>'
200002  [CONTROL     ] '<|return|>' (eos)
200003  [CONTROL     ] '<|constrain|>'
200004  [CONTROL     ] '<|reserved_200004|>'
200005  [CONTROL     ] '<|channel|>'
200006  [CONTROL     ] '<|start|>'
200007  [CONTROL     ] '<|end|>'
200008  [CONTROL     ] '<|message|>'
200009  [CONTROL     ] '<|reserved_200009|>'
200010  [CONTROL     ] '<|reserved_200010|>'
200011  [CONTROL     ] '<|reserved_200011|>'
200012  [CONTROL     ] '<|call|>'
200013  [CONTROL     ] '<|reserved_200013|>'
200014  [CONTROL     ] '<|reserved_200014|>'
200015  [CONTROL     ] '<|reserved_200015|>'
200016  [CONTROL     ] '<|reserved_200016|>'
200017  [CONTROL     ] '<|reserved_200017|>'
200018  [CONTROL     ] '<|endofprompt|>'

[dstoc]

Only size == 0 tokens are special?

llama.cpp/common/llguidance.cpp

Lines 172 to 175 in 61bdfd5

	if (size != 0) {
	*dp = '\xff'; // special token prefix marker
	size += 1;
	}

[dstoc]

Sent a fix: #15837