InferenceClient: allow passing a pydantic model as response_format
#2646
Labels
enhancement
New feature or request
Comments
|
adding to that, I think It's common to pass Pydantic objects to structure outputs - for example, OpenAI client supports this too (passing a Pydantic object to constrain the output format). Example from their documentation: from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()
class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-06",
messages=[
{"role": "system", "content": "Extract the event information."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
)
event = completion.choices[0].message.parsed |
|
Cool to know about the |
This was referenced Oct 31, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(issue opened after discussion with @lhoestq)
In
InferenceClient.chat_completion, one can pass aresponse_formatwhich constraint the output format. It must be either a regex or a json schema. A usual use case is to have a dataclass or a Pydantic model and you want the LLM to generate an instance of that class. This can currently be done like this:It would be good to either:
client.chat_completion(..., response_format=MyCustomModel)and handle the serialization automatically before making the http call. If we do so, pydantic shouldn't be a dependency.Note: the same should be done for
client.text_generation(..., grammar=...).Note: it seems that it's also possible to handle simple dataclasses with something like this. Unsure if it's worth the hassle though. If we add that, we should not add a dependency, simply copy the code + license into a submodule given how tiny and unmaintained the code is.
The text was updated successfully, but these errors were encountered: