ollam isn't working with autogen

[barshag]

Tried this code:

import autogen
from autogen import AssistantAgent, UserProxyAgent

config_list = [
    {
        
        "model":"openhermes2.5-mistral",

        "base_url": "http://localhost:11434/",
        "api_key" : "NULL",
        
    }
]

llm_config = {
    # "request_timeout" : 800,
    "config_list" : config_list
}

assistant = AssistantAgent(
    "assistant",
    llm_config = llm_config
)

user_proxy = UserProxyAgent(
    "user_proxy",
    code_execution_config = {
        "work_dir" : "coding"
    }
)

user_proxy.initiate_chat(
    assistant,
    message ="What is the name of the model you are based on?"
)

Got:
_base_client.py", line 885, in _request
raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: 404 page not found

[rickyloynd-microsoft]

What's ollam?

[ryanSphereOne]

@rickyloynd-microsoft https://github.com/jmorganca/ollama

@barshag not ollama but this might help https://www.youtube.com/watch?v=10FCv-gCKug

[abc-africa]

Hello @barshag I have got it working by using a combination of ollama.ai and litellm.ai here are the steps

Step 1: install ollama.ai by following the link ollama install

curl https://ollama.ai/install.sh | sh

Step 2: Run the selected model, i have choosen llama2

ollama run llama2

Step 3: Install litellm.ai by following the link litellm intall

pip install litellm

Step 4: Run litellm by specifying the model served by ollama in Step 2

litellm --model ollama/llama2

Step 5: Modified version of your code using the url output from Step 4

import autogen
config_list = [
	{
		"api_base": "http://127.0.0.1:8000",
		"api_key" : "NULL",
	}
]
llm_config = {
	"request_timeout" : 800,
	"config_list" : config_list
}

assistant = AssistantAgent(
    "assistant",
    llm_config = llm_config
)

user_proxy = UserProxyAgent(
    "user_proxy",
    code_execution_config = {
        "work_dir" : "coding"
    }
)

user_proxy.initiate_chat(
    assistant,
    message ="What is the name of the model you are based on?"
)

Happy AutoGenerating!🚀

[iplayfast]

I've found that request_timeout gives an error.
I've been able to autogenerate code, but it seems to not actually make it back from litellm. I can see it in litellm output, but it doesn't show up on disk.

[abc-africa]

Hello @iplayfast
Sureit sometimes timeout which i think is litellm issue.
On the other hand code not writing on a disk i think this is the issue dealing with autogen, i am still investigating may be @rickyloynd-microsoft can help us on this.

[rickyloynd-microsoft]

may be @rickyloynd-microsoft can help us on this.

I haven't used ollama or litellm, but I see plenty of discussion about them on our Discord channel.

[Donno191]

It would be nice to use ollama without litellm

[lestan]

Using AutoGen with Ollama is very easy now that Ollama support's OpenAI compatible API. No need for LiteLLM.

Here's a gist I created and tested with AutoGen version 0.2.19.

I'm using miqu model as I found that to work best. Also tested with Mixtral 8x7b.

AutoGen AI Agents with Local LLM served via Ollama

[ekzhu]

@lestan would you like to contribute your example as part of a topic category on AutoGen's support for open models? I believe @marklysze is interested in this as well.

[Donno191]

@lestan that worked for me THANKS

[zacscy]

In case anyone else runs into the same issue with Ollama and AutoGen or AutoGen Studio, what got it working for me was to add "/v1" to the base_url, that is:

"base_url": "http://127.0.0.1:11434/v1"

Might work with "base_url": "http://localhost:11434/v1" as well.