gptel-model and gptel-backend, why do we need these two?

[ryanobjc]

The README.md notes that gptel-model is how you set the default model. Except it really isn't entirely true.

If you attempt to set the gptel-model to one that isn't hosted by the default gptel-backend, then it doesn't really do anything.

The code didn't use to be like this.

Now you have to set both gptel-model and gptel-backend in concert with each other, also gptel-backend isn't something that can be customized since it has to be a complex record that comes from another live data structure.

In terms of a user interface, this is just confusing and non-ideal. The extra step of configuring a gptel-backend to match the model.

I run in to this because I want my default model to be sonnet 3.7, and I don't want to fiddle with gptel-menu, cancel that, then run the 'gptel' command to switch my model. I've adjusted my init.el now, but this is internal details of the workings of gptel leaking to the end user configuration.

[karthink]

If you attempt to set the gptel-model to one that isn't hosted by the default gptel-backend, then it doesn't really do anything.

Yeah, this is true.

The code didn't use to be like this.
Now you have to set both gptel-model and gptel-backend in concert with each other,

It's been like this for as long as gptel has supported more than just ChatGPT, so going back to Nov 2023 or earlier. Setting both independently is required for reasons I explain below.

also gptel-backend isn't something that can be customized since `it has to be a complex record that comes from another live data structure.

The inability to customize gptel-backend is a problem, but it can be fixed quite easily by providing a custom setter. It's just been a low priority because no one's complained about not being able to run customize-variable on gptel-backend yet. My time for gptel is limited and I have to focus on features/problems that have a big impact. Is using the customize interface to set gptel-backend something you actually want to do, or are you bringing it up as a hypothetical?

I run in to this because I want my default model to be sonnet 3.7, and I don't want to fiddle with gptel-menu, cancel that, then run the 'gptel' command to switch my model.

You don't have to use gptel-menu to use gptel. In fact the code for the transient menus is never even loaded if you don't call it. Ditto for most features of gptel that you're not interested in, like gptel-context or gptel-rewrite.

I've adjusted my init.el now, but this is internal details of the workings of gptel leaking to the end user configuration.
...
In terms of a user interface, this is just confusing and non-ideal. The extra step of configuring a gptel-backend to match the model.

Please suggest an alternative based on the information below.

---

The reason that the model and the backend need to be set independently is because they are independent settings -- the same model can be supported by multiple backends, and each backend can support multiple models. For instance, The model deepseek-r1-distill-llama-70b can be running on Ollama, Openrouter, Groq, Together.ai or any number of additional providers/backends. This is also true of Sonnet 3.7 -- many providers can act as proxies to this Anthropic model. The same model can behave differently depending on the backend settings. For example, Sonnet 3.7 can be used in thinking mode or regular mode depending on the gptel-anthropic backend settings in effect. Picking the model alone thus does not tell gptel where to send the request, what API key to use, what format the response will be in and so on.

The reason that the backend cannot be specified as a string is because there is no guarantee that the live data structure it represents will exist. Someone needs to create this live data structure, and it can't be gptel. What I mean is that if you want to just do

(setq gptel-backend "Claude")

and have it work, you would still need

(gptel-make-anthropic "Claude" :key ... :stream t)

in your configuration. So it's actually more configuration now -- two statements instead of one. gptel cannot ship with predefined backends for all possible providers, because there are too many of them and too many ways to configure each of them. (Look at the :request-params, :curl-args and other slots of gptel-make-anthropic for an example.)

Even under these limitations it can be made more user-friendly, but I'd need to know what that means exactly.

[ryanobjc]

First off, thanks for writing such great response to my issue. I really appreciate it!

I think the way I would approach this is to work backwards from the ideal user experience, then identify how it would work. I see your point about models being accessible via multiple backends.

When I use gptel-menu I see my model choice as being essentially "provider:model-name". Extending that to the gptel-model variable makes sense to me. I'd probably want to avoid the use of strings since it's a bit of an emacs anti-pattern, symbols preferred. But we really want a multi-level symbol here, so we could do something like

(set-default gpt-model '(claude . claude-3-7-sonnet-20250219))

Basically if the gpt-model is a symbol, do the current code. If it's a cons, then use the (car) as the backend and the (cdr) as the model within said backend.

We could keep gpt-model backwards compatible and also extend it a trivial manner that would make sense to a lot of people.

If this makes sense to you, I'd be more than happy to work on it.

[pabl0]

The reason that the model and the backend need to be set independently is because they are independent settings -- the same model can be supported by multiple backends, and each backend can support multiple models. For instance, The model deepseek-r1-distill-llama-70b can be running on Ollama, Openrouter, Groq, Together.ai or any number of additional providers/backends. This is also true of Sonnet 3.7 -- many providers can act as proxies to this Anthropic model. The same model can behave differently depending on the backend settings. For example, Sonnet 3.7 can be used in thinking mode or regular mode depending on the gptel-anthropic backend settings in effect. Picking the model alone thus does not tell gptel where to send the request, what API key to use, what format the response will be in and so on.

One interesting thing I noticed while working on #706: if I have gpt-4o defined as model provided by Github, it will inherit all the model parameters from openai, which is obviously confusing since I don't think the prices are identical.

If however I override the values like this:

  (gptel-make-openai "Github"
    :host "models.inference.ai.azure.com"
    :endpoint "/chat/completions?api-version=2024-05-01-preview"
    :stream t
    :key #'gptel-api-key-from-auth-source
    :models '((gpt-4o :description "gpt-4o as provided by GH" :output-cost 0.0) deepseek-r1))

then the parameters will be overridden in the openai (ChatGPT:gpt-4o) model as well. Basically currently you cannot have different pricing of same models from different providers, which isn't ideal.

Any ideas if this could be improved? This does not affect actually using the model, but it would still be nice if the tab completion in -m would show the correct pricing info (and describe-gptel-model if that gets merged). (Not sure if other aspect than pricing will vary between providers with same model?)

[karthink]

I'd probably want to avoid the use of strings since it's a bit of an emacs anti-pattern, symbols preferred.

I'm not against using symbols for backend names, but why are they an anti-pattern? We don't do enough comparisons for the eq/equal performance difference to be an issue. There is no plan to use symbol properties to store backend information. I'm not aware of other reasons.

I think the way I would approach this is to work backwards from the ideal user experience, then identify how it would work.

As I see it, the ideal user experience would be the following:

(setq gptel-model 'claude-3-sonnet-20240229
      gptel-backend "Claude")

with no other configuration required. But I don't think that's possible.

When I use gptel-menu I see my model choice as being essentially "provider:model-name". Extending that to the gptel-model variable makes sense to me. I'd probably want to avoid the use of strings since it's a bit of an emacs anti-pattern, symbols preferred. But we really want a multi-level symbol here, so we could do something like

(set-default gpt-model '(claude . claude-3-7-sonnet-20250219))

Basically if the gpt-model is a symbol, do the current code. If it's a cons, then use the (car) as the backend and the (cdr) as the model within said backend.

Instead of setting two variables (gptel-model and gptel-backend) to a symbol/string, you'd be setting one variable, but to a more complicated structure. This is a sideways move at best, as it's going to complicate the code quite a bit without much payoff.

But the main issue is that this does not fix the main problem, which is that you still need to define the claude backend yourself. You'll still need this in your configuration:

(gptel-make-claude 'claude :stream t :key ...) ;Assuming backend name is a symbol here

with much more code required to define other backends like Ollama or Groq.

(This is annoying, but at least the configuration for each backend is encapsulated in the structure. It's less annoying than having about eighty individual user options like gptel-anthropic-api-key, gptel-deepseek-api-key, gptel-anthropic-curl-args, gptel-deepseek-curl-args and so on. You can also have multiple backends of the same type with different settings this way.)

In terms of a user interface, this is just confusing and non-ideal. The extra step of configuring a gptel-backend to match the model.

Going back to this, you'd still need to match the backend and the model if you made the model a cons.

---

I see two actionable improvements here:

1. Allow the backend to be specified by a handle (i.e. symbol/string), as in (setq gptel-backend "Claude"). All references to gptel-backend in the code will have to be replaced by

   (if (stringp gptel-backend)
       (gptel-get-backend gptel-backend)
     gptel-backend)

2. Allow the backend struct to be customized. This is probably not necessary if we implement 1.

[karthink]

Basically currently you cannot have different pricing of same models from different providers, which isn't ideal.

Any ideas if this could be improved?

Yes, this is because model data is stored as symbol properties, and you can't have two symbols with the same name. I was aware of this limitation when I implemented it, and just punted the problem into the future. This can be fixed quite easily since the model property interface is abstracted. We use gptel--model-name, gptel--model-capabilities etc to get model properties, and the model metadata representation can be changed along with the implementations of these getters. However I don't think this is worth doing yet. We can come back to it when model name collisions become a real problem.

For future reference, the easiest way to fix it is to store the model name as a symbol property instead of calling (symbol-name gptel-model). Then you can have different symbols with the same model name.

[Gavinok]

Something worth mentioning is the possibility of multiple backends. As of now I am mixing my use of OpenAI's API and Anthropic and as far as I can tell I need to set the backend twice in order to be able to switch between models on both platforms.

[karthink]

as far as I can tell I need to set the backend twice in order to be able to switch between models on both platforms

@Gavinok Can you explain what exactly you're trying to do? You can have different backends active in different buffers, so I'm not sure what you mean here.

[Gavinok]

How exactly do you configure gptel to utilize multiple backends? As I understand it you need to (setq gptel-backend (...)) so how exactly are you supposed to configure more than one?

[karthink]

How exactly do you configure gptel to utilize multiple backends? As I understand it you need to (setq gptel-backend (...)) so how exactly are you supposed to configure more than one?

Interactively, set the scope before setting an option:

Available scopes are global (set globally), buffer (set for this buffer) and oneshot (set for next request only)

Via elisp, use setq-local instead of setq:

(setq-local gptel-backend (gptel-get-backend "Claude")
            gptel-model   'claude-3-5-sonnet-20240620)

[Gavinok]

hmm, I guess that could work as a solution though I often will switch back and forth between claude and gpt-4o-min in the same buffer. I realized that there is already gptel--known-backends. This is close to what I am asking for since I simply want to define multiple backends that I can switch between using gptel-menu. So I could just set gptel--known-backends myself

[Gavinok]

really all I want is a way to register and configure multiple backends in my init.el

[karthink]

really all I want is a way to register and configure multiple backends in my init.el

You can register as many backends as you want.

(gptel-make-backend "backend1" ...)
(gptel-make-backend "backend2" ...)
...

Only one backend is active at a time in a given buffer, it's the value of gptel-backend. You can switch between backends using the transient menu or by using setq/setq-local.

You don't need to use gptel--known-backends, it's meant for internal use.

[Gavinok]

Omg that makes sense. I for whatever reason didn't realize that was registering each of the backends. Thanks for the clarification