Support for remote models

[adityaprakash-work]

I really like guidance's API. However, remote models don't seem to work with most of the useful APIs like gen, json, select etc. For gen I usually have to break the assistant block into one assistant block having lm += gen(...) and the rest. I was wondering how others are approaching this problem. I don't want to use workarounds because it defeats the whole purpose of using guidance in the first place. Can't we build something, an intermediate layer that emulates a local model using a remote one? Of course I am willing to accept the additional latency because my projects will have the option to swap the model with an actually local one and I wouldn't have to write 2 separate cases; remote and local.

What I am looking for:

1. Pointers to high level concepts within the code explaining guidance's inability to work with remote models.
2. Current status of ideation.
3. Even a 'trick' will do. As I said, I am willing to tradeoff speed because my primary concern is the development process. I want to use the same code for both remote and local models so that the project as a whole is less redundant to develop and has the option to switch models in case.

[Harsha-Nori]

This is a great call-out @adityaprakash-work. Have been doing some thinking on this, and I think something like what you propose is possible in the near future. @hudson-ai is leading some significant guidance internals refactoring over the next few weeks, which will make it much easier for us to try out innovations like this.

[hudson-ai]

@adityaprakash-work we've made some progress on this in the latest release (still more to do 😜). But we now support both unconstrained gen and json, so long as it's (1) in an assistant role block and (2) the first thing in the role block.