PDL model functions

[eloycoto]

There is a way to define functions like this as the model parameter:

description: "Get current stock"
text:
  - model: ollama/llama3.2:1b
    parameters:
        temperature: 0.3
        functions:
        - '{ "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }'
    input: |
      array:
      - role: user
        content: |
          Could you tell me the weather in NY city please?

Is there any way to define this functions as PDL functions? Like, I have:

defs:
  getMarkdownFromURL:
    function:
      url: str
    return:
    - lang: python
      code: |
        import requests
        import html2text
        response = requests.get('${url}')
        result = html2text.html2text(str(response.content))

Is there any way that I say:

 - model: ollama/llama3.2:1b
    parameters:
        temperature: 0.3
        functions:
        - getMarkdownFromURL

This allow to reuse and compose scripts using the same tools/functions available in different PDL scripts.

[eloycoto]

Btw: request to ollama is like this:

{"model": "llama3.2:1b", "prompt": "### User:\narray:\n- role: user\n  content: |\n    Could you tell me the weather in NY city please?\n\n\n### System:\nProduce JSON OUTPUT ONLY! Adhere to this format {\"name\": \"function_name\", \"arguments\":{\"argument_name\": \"argument_value\"}} The following functions are available to you:\n{ \"type\": \"function\", \"function\": { \"name\": \"get_current_weather\", \"description\": \"Get the current weather in a given location\", \"parameters\": { \"type\": \"object\", \"properties\": { \"location\": { \"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\", }, \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}, }, \"required\": [\"location\"], }, }, }\n\n\n", "options": {"temperature": 0.3}, "stream": false, "format": "json"}

And response like this:

{"model":"llama3.2:1b","created_at":"2024-11-07T16:26:34.060688639Z","response":"{ \"type\": \"function\", \"function\": { \"name\": \"get_current_weather\", \"parameters\": { \"location\": \"New York City, NY\", \"unit\": \"celsius\" } } }","done":true,"done_reason":"stop","context":[],"total_duration":2551073970,"load_duration":15137079,"prompt_eval_count":195,"prompt_eval_duration":62000000,"eval_count":44,"eval_duration":2472000000}

[vazirim]

@eloycoto Thanks for posting this thread. It's a great question, it would mean taking an existing PDL function and extracting its signature to send to the model. We can work on this feature.

To help us with it, do you mind sending a version of the first code snippet above that works for you? I am getting a weird error from the model on that. Thanks!

[eloycoto]

Got an error too, it's maybe because it should translate functions to tools dictionary as said in the doc?

https://docs.litellm.ai/docs/completion/function_call

functions is not a supported param:
https://github.com/BerriAI/litellm/blob/a3baec081bd08c5aaa725050b8e872f1a9a78991/litellm/llms/OpenAI/openai.py#L172

[eloycoto]

made a quick check, If I do this:

diff --git a/src/pdl/pdl_llms.py b/src/pdl/pdl_llms.py
index 7ebe19a..3df2ebe 100644
--- a/src/pdl/pdl_llms.py
+++ b/src/pdl/pdl_llms.py
@@ -157,6 +157,9 @@ class LitellmModel:
             )
         if parameters.get("mock_response") is not None:
             litellm.suppress_debug_info = True
+
+        parameters["tools"] = [json.loads(parameters.get("functions")[0])]
+        # import ipdb; ipdb.set_trace()
         response = completion(
             model=model_id, messages=messages, stream=False, **parameters
         )

The request an response from the API looks good, but tool_usage are not used anywhere, or I do not think so!

T +0.001553 192.168.2.193:41918 -> 192.168.2.5:11434 [AP] #153
{"messages": [{"role": "user", "content": "array:\n- role: user\n  content: |\n    Could you tell me the weather in NY city please?\n"}], "model": "llama3.2:3b", "functions": ["{\"type\": \"function\", \"function\": {\"name\": \"get_current_weather\", \"description\": \"Get the current weather in a given location\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"}, \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}}, \"required\": [\"location\"]}}}"], "stream": false, "temperature": 0.3, "tools": [{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}}]}
##
T +2.051197 192.168.2.5:11434 -> 192.168.2.193:41918 [AP] #155
HTTP/1.1 200 OK'
Content-Type: application/json'
Date: Fri, 08 Nov 2024 17:55:08 GMT'
Content-Length: 462'
'
{"id":"chatcmpl-350","object":"chat.completion","created":1731088508,"model":"llama3.2:3b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"","tool_calls":[{"id":"call_jqzzslnw","type":"function","function":{"name":"get_current_weather","arguments":"{\"location\":\"New York, NY\",\"unit\":\"fahrenheit\"}"}}]},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":207,"completion_tokens":28,"total_tokens":235}}

####

[vazirim]

@eloycoto you could directly pass the JSON object in the functions field in the code above (or even write that in yaml). I tried that and still getting an error. Debugging now...

[vazirim]

@eloycoto I see what you mean in the LiteLLM code, it indeed looks like it's dropping functions parameter. I am going to make this change in my own fork of LiteLLM and see if this works. We can then submit a PR to LiteLLM.

[eloycoto]

Checked with tools parameters instead of functions(deprecated) to use tools, and I got this:

https://github.com/IBM/prompt-declaration-language/blob/main/src/pdl/pdl_ast.py#L206

PDL:

description: "Get current stock"
text:
  - model: openai/llama3.2:1b
    parameters:
        temperature: 0.3
        tools:
          - { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } }
    input: |
      array:
      - role: user
        content: |
          Could you tell me the weather in NY city please?

HTTP request & response

T 127.0.0.1:47202 -> 127.0.0.1:11434 [AP] #86
POST /v1/chat/completions HTTP/1.1.
Host: 127.0.0.1:11434.
Accept-Encoding: gzip, deflate, br.
Connection: keep-alive.
Accept: application/json.
Content-Type: application/json.
User-Agent: OpenAI/Python 1.52.1.
X-Stainless-Lang: python.
X-Stainless-Package-Version: 1.52.1.
X-Stainless-OS: Linux.
X-Stainless-Arch: x64.
X-Stainless-Runtime: CPython.
X-Stainless-Runtime-Version: 3.12.7.
Authorization: Bearer ollama.
X-Stainless-Async: false.
X-Stainless-Raw-Response: true.
x-stainless-retry-count: 0.
Content-Length: 536.
.

T 127.0.0.1:47202 -> 127.0.0.1:11434 [AP] #88
{"messages": [{"role": "user", "content": "array:\n- role: user\n  content: |\n    Could you tell me the weather in NY city please?\n"}], "model": "llama3.2:1b", "stream": false, "temperature": 0.3, "tools": [{"name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}]}

T 127.0.0.1:11434 -> 127.0.0.1:47202 [AP] #92
HTTP/1.1 200 OK.
Content-Type: application/json.
Date: Sat, 09 Nov 2024 20:20:59 GMT.
Content-Length: 427.
.
{"id":"chatcmpl-857","object":"chat.completion","created":1731183659,"model":"llama3.2:1b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"","tool_calls":[{"id":"call_9jubkctu","type":"function","function":{"name":"weather","arguments":"{\"required\":[],\"type\":\"\"}"}}]},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":153,"completion_tokens":149,"total_tokens":302}}

Looks like the problem is when the llm Response has tool_calls, pdl do nothing with it.

For me, tackling this problem will be:

1. Checking if here the tools can be called, if so, how
   

   prompt-declaration-language/src/pdl/pdl_llms.py

   Line 164 in a61c3af

   if msg.content is None:

2. Check how to create the tool API json in the yaml from a given function.

[vazirim]

@eloycoto This has been fixed now in the main branch (#178).

Here's an example. The modelResponse field captures the raw output of the model, which contains tool calls.

text:
  - Could you tell me the weather in Tokyo please?
  - model: ollama/llama3.2:1b
    modelResponse: out
    parameters:
      temperature: 0
      tool_choice: auto
      tools:
      - type: function
        function: 
          name: get_current_weather
          description: Get the current weather in a given location
          parameters:
            type: object
            properties:
              location:
                type: string
                description: The city and state, e.g. San Francisco, CA
              unit:
                type: string
                enum:
                - celsius
                - fahrenheit
              required:
                - location
  - ${ out }

We are working on making this easy to consume on the PDL side (i.e. make it easy to do the tool call here, and call the model again with the results).

[eloycoto]

Many thanks! Ok, but the tool call is not appended into the call, no? Like, how can I execute the function call?

[vazirim]

@eloycoto We are working on it.

[eloycoto]

awesome, if something ping me, I might able to contribute a bit!

[vazirim]

@eloycoto PR #184 (just merged) contains a fix to allow richer messages. So the following example now works for obtaining tool calls from the LLM, calling them locally, and returning the responses back to the LLM:

defs:
  get_current_weather:
    function:
      args: str
    return: 
      defs: 
        arguments:
          text: ${ args }
          parser: json 
      if: ${ arguments.location == "Tokyo"}
      then: The temperature in Tokyo is 20 celsius.
text:
  - "Could you tell me the weather in Tokyo please?\n"
  - role: assistant
    model: ollama/llama3.2:1b
    modelResponse: out
    parameters:
      temperature: 0
      tool_choice: auto
      tools:
      - type: function
        function: 
          name: get_current_weather
          description: Get the current weather in a given location
          parameters:
            type: object
            properties:
              location:
                type: string
                description: The city and state, e.g. San Francisco, CA
              unit:
                type: string
                enum:
                - celsius
                - fahrenheit
              required:
                - location
  - def: tool
    data: ${ out[0].choices[0].delta.tool_calls[0] }
    contribute: []
  - "\n"
  - call: ${ tool.function.name }
    def: functionRes
    contribute:
    - context: 
        value:
          - role: tool
            tool_call_id: ${ tool.id }
            name: ${ tool.function.name }
            content: ${ functionRes }
    args: 
      args: ${ tool.function.arguments }
  - model: ollama/llama3.2:1b
    parameters:
      temperature: 0

The ContributeValue in the function call allows the user to contribute a rich message containing the function response to the background context.

In the future, we can make this more seamless with the following abstractions to help with the following:

• Providing the signature of the tool in the first model call
• Obtaining the tool call from the model response (which is a different expression based on streaming or not)
• Knowing what message to add to the context with the function response.

[eloycoto]

Awesome! I'll play with this in a couple of weeks, Thanks!