Pulldown to select the LLM not working

[fmandelbaum]

Hello, thanks for all your hard work on running bolt.new locally.

I've followed the instructions on https://www.youtube.com/watch?v=31ivQdydmGg, and after copying my .env.local to /app on the docker container, I have it running with no warnings. My idea is to use a locally running Ollama, and I have (and don't intend to have) any API key for any external AI service.

So, I select Ollama in the pulldown, select my ollama-3.2 "small" model (my laptop hasn't one of those newer M chips...), and I see an error on the UI no matter which prompt I use. Checking the docker logs for the container I see that it's apparently always trying to use the 1st model (Anthropic), no matter which model I set using the pulldowns.

I'd really appreciate if this is fixed, like I said, and like you promote on your videos, I want to do a fully local thing, without the need to have any AI chat with any external API.

Thanks in advance.

Best regards.

[yieme]

I think bolt.new needs something like a 32K context window. As I recall Ollama defaults to something like 2K.

• Verify your model's maximum context length is at least 32768, ex: ollama show qwen2.5-coder:7b-instruct-q4_K_M

• Create Ollama Modelfile

FROM qwen2.5-coder:7b-instruct-q4_K_M
PARAMETER num_ctx 32768

• Create a variant with the context length, ex: ollama create -f Modelfile qwen2.5-coder:7b-instruct-32k_num_ctx-q4_K_M

• Try your 32K context length version of your model to see if that helps.

Note: Some models repeat a lot with larger num_ctx