[AQUA] Adding handler for streaming inference predict endpoint

[kumar-shivam-ranjan]

Description

This PR is intended to enhance the Model deployment inference experience within AQUA. With the introduction of new predict endpoint (/predictWithResponseStream) with streaming support , AQUA can now use this native API to provide streaming inference experience to users.

Sample payload

CURL

curl --location 'http://localhost:8888/aqua/inference/stream/<MD-OCID>' \
--header 'Content-Type: application/json' \
--data '{
    "max_tokens": 1024,
    "temperature": 0.5,
    "prompt": "what are some good skills deep learning expert. Give us some tips on how to structure interview with some coding example?",
    "top_p": 0.4,
    "top_k": 100,
    "model": "odsc-llm",
    "frequency_penalty": 1,
    "presence_penalty": 1,
    "stream": true
}'

Unit tests

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[mrDzurb]

I didn't know that we use it at all

[kumar-shivam-ranjan]

It was added before we had kernel messaging solution for inference. Since then it has been lying orphaned in code.
Now we can use this handler to expose streaming inference API which our client (AQUA UI) can use for inference instead of keeping the script at UI.

[mrDzurb]

Got it, let's use it but with AQUA Client.
I've added this PR to update the docs. I think In UI playground it would be useful to add a checkbox where users can choose if they want to use streaming or not. By default it can use streaming end-point.

[mrDzurb]

There is AQUA client, that can be utilized.
https://accelerated-data-science.readthedocs.io/en/latest/user_guide/large_language_model/aqua_client.html#usage

[kumar-shivam-ranjan]

I see. yeah we can use aqua client instead of model deployment client for streaming inference support. will update. Thanks

[kumar-shivam-ranjan]

Updated

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[mrDzurb]

NIT: I think we probably don't need it on the API level, having this logic in handler would be sufficient.

[kumar-shivam-ranjan]

I might not have properly understood your statement ..
but we have business logic based on predict endpoint type like text/chat completions and even endpoint override feature in this method.

Do you mean adding this logic inside deployment_handler.py ?

[mrDzurb]

Yes, I mean do we really need this logic to be placed in the deployment.py? Will it be used outside of the UI? If not, then probably would be better to move this logic to the handlers.

[kumar-shivam-ranjan]

Updated. Thanks

[VipulMascarenhas]

can we accept chat_template as well since we want to support it?

[kumar-shivam-ranjan]

yes it will be accepted implicitly. Chat template will be optional but still still user can provide it and the code will accept and pass it to aqua client.

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage:

[github-actions]

📌 Cov diff with main:

📌 Overall coverage: