LlamaCPP server does not follow the SO specification

[KanishkNavale]

Python Code:

from openai import OpenAI
from pydantic import BaseModel, Field

from typing import List


class SO(BaseModel):
    reasoning_steps: List[str] = Field(
        default_factory=list, description="Resoning steps leading to an answer"
    )
    final_answer: str = Field(..., description="Final answer")


prompt = "John has 10 apples. He gives away 4 and then receives 5 more. How many apples does he have?"

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="None",
)

response = client.chat.completions.create(
    model="None",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_schema", "schema": SO.model_json_schema()},
    stream=True,
)

for token in response:
    token = token.choices[0].delta.content
    print(token, end="") if token else None

Llama Server in Docker:

services:
  inference_server:
    image: ghcr.io/ggerganov/llama.cpp:server-cuda
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]
    network_mode: "host"
    environment:
      LLAMA_ARG_HF_REPO: bartowski/Qwen2.5-7B-Instruct-1M-GGUF
      LLAMA_ARG_HF_FILE: Qwen2.5-7B-Instruct-1M-Q8_0.gguf
      LLAMA_ARG_N_GPU_LAYERS: 999
      LLAMA_ARG_FLASH_ATTN: True
      LLAMA_ARG_CTX_SIZE: 8192
      LLAMA_ARG_N_PARALLEL: 1
      LLAMA_ARG_PORT: 8080
      LLAMA_ARG_HOST: localhost

Output: {"1": "6+5", "answer": 11}

Moreover, the outputs have never been consistent.
However, llama-cpp-python gives you consistent outputs with grammar.
llama-cpp-python working example: https://github.com/KanishkNavale/llamaso

Could you please help me to fix this?

[KanishkNavale]

No issues. I solved it