Implemented OpenRouter API key usage with any model, set Gemini 2.5 Flash default, created natural-language agent to parse chat. #33

rio-codes · 2025-08-09T06:06:05Z

Many changes have been made.

Regarding OpenRouter, the embeddings model still requires OpenAI, so I had to implement both API keys, with the possibility for the user to provide their own for either one.
I implemented logic including an embeddings_needed Boolean parameter to determine whether a command requires the embeddings model or not, and the function will provide the correct API key to the langchain function call based on the nature of the command.
I edited the Streamlit UI so that you can enter an OpenRouter key as well as a custom model.
I set the default model to google/gemini-2.5-flash and tested with it, it was able to complete all DDG functions. I also tried with an alternate model from Mistral.
I created the command chooser utility in command_chooser.py with the function get_raw_command that accurately determines the proper command to use given any user input. It is quite robust and selected an appropriate command with a variety of different scenarios. It uses the /kb command to check the topic of the selected collection before choosing a command so it knows whether to create a new collection or not.
I tested thoroughly with many different prompts of differing topics and specificity.
I did encounter a bug near the end that appears to be unrelated to any of the changes I made, after running /re hs 4:

ValueError: Expected IDs to be a non-empty list, got 0 IDs in get.

File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 128, in exec_func_with_error_handling
    result = func()
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 669, in code_to_exec
    exec(code, module.__dict__)  # noqa: S102
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/midbar/Projects/docdocgo-core/streamlit_app.py", line 370, in <module>
    llm_raw_command: dict[str, str] = get_raw_command(full_query, chat_state)
                                      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/midbar/Projects/docdocgo-core/agents/command_chooser.py", line 105, in get_raw_command
    details = get_bot_response(chat_state)
File "/home/midbar/Projects/docdocgo-core/docdocgo.py", line 124, in get_bot_response
    return chat_chain.invoke(
           ~~~~~~~~~~~~~~~~~^
        {
        ^
    ...<6 lines>...
        }
        ^
    )
    ^
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/langchain/chains/base.py", line 167, in invoke
    raise e
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/langchain/chains/base.py", line 157, in invoke
    self._call(inputs, run_manager=run_manager)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/midbar/Projects/docdocgo-core/components/chat_with_docs_chain.py", line 192, in _call
    docs = self.retriever.get_relevant_documents(
        standalone_query,
        # callbacks=_run_manager.get_child(),
        **search_kwargs,
    )
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/langchain_core/_api/deprecation.py", line 191, in warning_emitting_wrapper
    return wrapped(*args, **kwargs)
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/langchain_core/retrievers.py", line 411, in get_relevant_documents
    return self.invoke(query, config, **kwargs)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/langchain_core/retrievers.py", line 259, in invoke
    result = self._get_relevant_documents(
        input, run_manager=run_manager, **_kwargs
    )
File "/home/midbar/Projects/docdocgo-core/components/chroma_ddg_retriever.py", line 158, in _get_relevant_documents
    rsp = self.vectorstore.collection.get(unique_parent_ids)
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/chromadb/api/models/Collection.py", line 124, in get
    get_request = self._validate_and_prepare_get_request(
        ids=ids,
    ...<2 lines>...
        include=include,
    )
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/chromadb/api/models/CollectionCommon.py", line 95, in wrapper
    return func(self, *args, **kwargs)
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/chromadb/api/models/CollectionCommon.py", line 251, in _validate_and_prepare_get_request
    validate_ids(ids=unpacked_ids)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/home/midbar/Projects/docdocgo-core/.venv/lib/python3.13/site-packages/chromadb/api/types.py", line 701, in validate_ids
    raise ValueError(f"Expected IDs to be a non-empty list, got {len(ids)} IDs")

If you would like to employ my services to troubleshoot this bug as well I would be glad to assist.

Since this PR encompasses the content of the previous PR, I will close that one and leave a single PR for your approval. Thank you!

rio-codes · 2025-08-11T04:21:22Z

After the most recent commit, everything is working smoothly and the default mode is implemented properly throughout the codebase. I am able to switch easily between heatseek and collections, and the collection summary is only fetched once. All lingering issues with the distinction between OpenAI and OpenRouter API keys have been resolved.

utils/chat_state.py

reasonmethis · 2025-08-13T02:15:34Z

agents/command_chooser.py

+
+    # Get details on the current collection 
+    print("Getting details on", chat_state.collection_name)
+    if chat_state.collection_name not in coll_summary_query:


coll_summary_query is initialized as empty above. More importantly, why waste time and money summarizing when we haven't yet checked if the query already starts with a command string?

It should only check for the summary once for each collection and store it in the coll_summary_query dictionary. Whether it starts with a command string or not, each collection should be summarized at least once.

I have modified the code so it only initializes coll_summary_query if it doesn't exist here: 11d1441

reasonmethis · 2025-08-13T02:18:13Z

agents/command_chooser.py

+    if chat_state.collection_name not in coll_summary_query:
+        coll_summary_query[chat_state.collection_name] = ""
+        summary_prompt = "/kb Can you summarize in one sentence the contents of the current collection?"
+        summary_llm = get_llm(chat_state.bot_settings, chat_state,chat_state.openrouter_api_key,embeddings_needed=False)


Ideally, the agent shouldn't waste resources on this unless it's the first time in the conversation when it needs to know what the collection is about. Most of the time the description could be cached.

I think every collection should be summarized at least once, I think it's essential for the new natural language parsing feature of this app. It can't decide which command to use if it doesn't know anything about the collection. I don't think it's a waste of resources to have that summary stored in the variable. When you say cached, do you think this variable works as a cache or is something more involved necessary?

reasonmethis · 2025-08-13T02:20:11Z

agents/researcher.py

+    else:
+        research_params.num_iterations_left = 1


This else clause makes no sense

research_params was sometimes not getting initialized, and it was looking for num_iterations_left and causing an exception. This ensures that this variable is set even if research_params doesn't get initialized for some reason.

api.py

reasonmethis · 2025-08-13T02:27:56Z

api.py

                return ChatResponseData(
                    content="Apologies, you can customize your model settings (e.g. model name, "
-                    "temperature) only when using your own OpenAI API key."
+                    "temperature) only when using your own OpenRouter API key."


Way more people have an openai key than an openrouter key, so if I understand correctly this is a regression. I thought we would allow someone to provide just an openai key and use the app fully, with any openai model. And if they provide an openrouter key then they can use any model.

But it looks like the openrouter key is now king - the user id is tied to it.

In that case, I completely misunderstood your intent. I thought the idea was to make gemini 2.5 flash the default model via openrouter, phase out openai models, and use openrouter for any other model. I thought the only reason to retain openai functionality was because you can't use embeddings models via openrouter. Would you like me to change this and allow any openai model if an openai key is provided?

components/chroma_ddg_retriever.py

streamlit_app.py

utils/type_utils.py

…ore into dev-command-chooser

Co-authored-by: Dmitriy Vasilyuk <[email protected]>

…returned

reasonmethis

In that case, I completely misunderstood your intent. I thought the idea was to make gemini 2.5 flash the default model via openrouter, phase out openai models, and use openrouter for any other model. I thought the only reason to retain openai functionality was because you can't use embeddings models via openrouter. Would you like me to change this and allow any openai model if an openai key is provided?

Yes, otherwise it would be a big regression in UX. That would prevent the necessity for the user to provide two separate keys, which the majority of users would find unnecessarily burdensome. Since an OpenAI key is required in any case, it should continue being "king" (the user id should be tied to it) and the OpenRouter key can just be optional.

rio-codes added 15 commits June 23, 2025 14:04

fix typo in env variable in dev docs

8b1234c

Merge branch 'main' of github.com:rio-codes/docdocgo-core

7e172f4

Merge branch 'main' of github.com:rio-codes/docdocgo-core

87cc262

updating gitignore

19252db

changing all references to OpenAI API key to OpenRouter

ae7eab3

openai is required for embeddings, re-implementing

be0035a

more reversions to openai where embedding is needed

258f5c8

reverting announcement since it is for old version

9cc899f

changed streamlit UI to include OpenRouter

127614a

fixing openrouter model setting

bb11ad5

model name is now obtained from settings

e8828c9

various changes and beginning of new function code

e230fb7

uploading progress but still troubleshooting

4c4016e

more fixes to command chooser, OpenRouter migration

6c03c1a

small formatting modifications, change to env example

393adce

rio-codes requested a review from reasonmethis as a code owner August 9, 2025 06:06

This was referenced Aug 9, 2025

Added OpenRouter API acccess for google/gemini-2.5-flash model #30

Closed

Switch to OpenRouter and enable Gemini 2.5 flash as default #28

Open

Have LLM select correct command based on user prompt in natural language #29

Open

rio-codes added 4 commits August 9, 2025 16:49

added default mode, fixed callbacks bug, adjusted prompt

d1f10cd

implementing new default chat mode, cached summaries, and other fixes

677d9c6

quick commit of file that should have been saved

a256890

quick commit of file that should have been saved

6033dea

reasonmethis reviewed Aug 13, 2025

View reviewed changes

rio-codes and others added 5 commits August 13, 2025 00:02

removed all references to embeddings_needed

e0781d1

Merge branch 'dev-command-chooser' of github.com:rio-codes/docdocgo-c…

a8d6f22

…ore into dev-command-chooser

Only initialize coll_summary_query as blank if it doesn't exist

11d1441

Update components/chroma_ddg_retriever.py

6d52d0f

Co-authored-by: Dmitriy Vasilyuk <[email protected]>

removed unnecessary code that would not be reached if no chunks were …

19bcd22

…returned

rio-codes added 3 commits August 13, 2025 00:44

removed unneccesary pwd check

50cd78a

simplifying logic to collapse API key fields if LLM response

2707f8e

changed all instances of DEFAULT_CHAT_COMMAND_ID to AUTO_COMMAND_ID

7720040

reasonmethis reviewed Aug 30, 2025

View reviewed changes

Uh oh!

Implemented OpenRouter API key usage with any model, set Gemini 2.5 Flash default, created natural-language agent to parse chat. #33

Are you sure you want to change the base?

Implemented OpenRouter API key usage with any model, set Gemini 2.5 Flash default, created natural-language agent to parse chat. #33

Uh oh!

Conversation

rio-codes commented Aug 9, 2025

Uh oh!

rio-codes commented Aug 11, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reasonmethis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants