Project Direction - Features, improvements, etc

[neodon]

I'm considering contributing to the project. I was wondering if you could provide some insights on where you see it going. Do you have particular features or improvements in mind? I have some ideas, but they're fairly specific to my workflows and quirks. I imagine there are a lot of use cases I'm unaware of.

Some things I've been looking for in particular:

• Chat interface with rich export options so I can share conversations (e.g. single HTML file, PDF, full-height SVG or PNG). Maybe at some point the ability to share a URL for a read-only view of a conversation.
• Auto-discovery of models available for providers that support it such as OpenRouter instead of needing to wait for new models to be added to the app. Perhaps not importing each provider's entire model list, but allowing a search by model name and importing just the ones I want. In addition to the curated model lists you have now if they are reasonable to maintain.
• Transparency of what's being sent and received for the API and providers.
• Responses API support for better interleaving of tool calls and thinking (affects output quality quite a bit - seems to be a more visible issue lately).
• Memory support via vector indexing and search. I want to be able to import the latest documentation and articles for things I use and expose them via tool calls. Right now my chats contain repetitive web searches for docs with results that fill up the context with irrelevant data. Context7 is closest to what I'm thinking, but the UX and quality of results aren't great. Also everything is stored on their servers.

I'd also be interested in your thoughts around the directions/features of Open Web UI and LLM Gateway (and any other projects you think of) compared to this project.

[mythz]

Do you have particular features or improvements in mind?

Plugin System

First thing I want to do is add is a plugin system (ala ComfyUI custom nodes) where you can clone a github repo to ~/.llms/plugins/ and restart the server to extend both server routes and serve custom UIs. I would like to keep the core ChatGPT like functionality baked into the core /ui and anything opinionated that goes beyond this (or anything requiring additional dependencies) in plugins so they can be easily disabled/replaced/etc.

It would also allow others be able to develop, prototype and share functionality easier.

First step is to design how plugin system would work.

Installation:

Should be able to copy anyones git repo into the plugins folder, e.g:

git clone https://github.com/username/repo.git ~/.llms/plugins/repo

So to remove the feature they'd only need to delete the folder.

Requirements, should be able to:

• extend the server aiohttp Web Application routes
• access to provider configuration, clients, etc
• serve their own static ui files
• override built-in components
• Make room in the UI for plugins/extensions (ala VS Code)
  • icons in toolbar (below Model/Prompt selector)
  • vertical icons on the right?
  • status bar?

Convention for storing user data outside the plugins folder, e.g ~/users/default or ~/users/<github-user>

File Storage

Add a file/sha storage feature for attachments, that you raised earlier so we can keep image previews / original attachments in the UI.

Streaming

It would be nice to show streaming of responses from providers that support it.

llama.cpp/llamaswap

Explicit support for llama.cpp and auto discovery of llama-swap models. Maybe vLLM as well if there's demand for it, although I've never used it personally.

Static Website

Adding docs for all features in 1 long README isn't feasible, and will eventually need a static docs website.

Tool call support

Nice to have as we don't have any, as I have no idea what that would look like - copying what someone who does this well would be a good start.

Gemini RAG

Other than that the only feature I would like to build personally atm (after implementing a plugin system) is support for File Search in Gemini where I can select a folder to upload all files to Gemini and be able to query my knowledge base. No idea how that would look/work yet, but it's a feature I'd like to have. It's aggressively priced so I expect this to be very popular for querying knowledge bases.

Chat interface with rich export options

With the export options, given we have optional GitHub OAuth support I think publishing to a gist makes the most sense. But what to export is unclear.

Auto-discovery of models available for providers that support it such as OpenRouter

Not sure about this, it currently uses user-defined model ids so requests can be cascaded to other configured providers if they fail. There's no universal API to discover models/pricing so this would be unique and specific to each provider. As such I think this would be good to add this as a OpenRouter plugin which we could bundle by default.

Transparency of what's being sent and received for the API and providers.

This is available via --verbose flag:

llms --serve 8000 --verbose

Responses API support for better interleaving of tool calls

Maybe something that could fit in Tool support? Not currently on my radar.

Memory support via vector indexing and search.

Similar to the feature I'd like to build with Gemini File Search, Should be a plugin. A lot of people are going to want an all-local/privacy solution.

I'd also be interested in your thoughts around the directions/features of Open Web UI and LLM Gateway

I don't spend anytime thinking about anything I don't use. Not interested in chasing features others are doing that I don't use personally. Happy to copy good ideas if I find them useful but I definitely don't want to be relying on anything that's VC funded which eventually leads into enshittification, as such happy to maintain my own tool that I have complete control of. I primarily want a good AI Assistance experience that I can use for local/cloud LLMs in the same UI. I'd like to keep the core functionality in a single file with minimal deps that's an easy drop-in any Python project (e.g. ComfyUI Custom Node) to access AI Features.

Any other features should be delivered via plugins that users can easily enable/disable.