feat(llm): support async/streaming output mode in api layer

[chiruu12]

close #177

[chiruu12]

@imbajin thx for the suggestion will do so!

[imbajin]

@imbajin sir I am having a little problem locating pyhugegraph can you tell me where it is ?

U could ignore it (not related to current PR)

BTW, py-client here-> https://github.com/apache/incubator-hugegraph-ai/tree/main/hugegraph-python-client

[chiruu12]

@imbajin shall I commit the changes that I made?

[chiruu12]

@imbajin I have separated the files for stream_api and config too and tried to make the format better for passing the pylint test too I had to manually do it any way we can do it using a script? whenever i am running the script it is telling me I have to do it manually

[imbajin]

@imbajin I have separated the files for stream_api and config too and tried to make the format better for passing the pylint test too I had to manually do it any way we can do it using a script? whenever i am running the script it is telling me I have to do it manually

Not quite sure what you mean when you say the script content? Should Pytest be able to specify multiple tests at once without having to execute them separately multiple times? Can you provide specific explanations for it?

[chiruu12]

@imbajin I have separated the files for stream_api and config too and tried to make the format better for passing the pylint test too I had to manually do it any way we can do it using a script? whenever i am running the script it is telling me I have to do it manually

Not quite sure what you mean when you say the script content? Should Pytest be able to specify multiple tests at once without having to execute them separately multiple times? Can you provide specific explanations for it?

sir I have been going through them manually one by one, which is pretty time-consuming. Is there a better approach I'm missing? Maybe a tool or command that can automatically fix some of the common pylint issues instead of just listing them? I'm not very experienced with pylint and wondered if there's a more efficient way to handle this than the manual corrections I've been doing.
Like I have contributed to openvino once and they told me to run a small command using black and then the issue with spacing was resolved automatically

and I saw that there are a few errors which are coming like the
FIXME: line 31: E0702: Raising dict while only classes or instances are allowed (raising-bad-type)
This error stems from admin_api.py class

[imbajin]

sir I have been going through them manually one by one, which is pretty time-consuming. Is there a better approach I'm missing? Maybe a tool or command that can automatically fix some of the common pylint issues instead of just listing them? I'm not very experienced with pylint and wondered if there's a more efficient way to handle this than the manual corrections I've been doing. Like I have contributed to openvino once and they told me to run a small command using black and then the issue with spacing was resolved automatically

and I saw that there are a few errors which are coming like the FIXME: line 31: E0702: Raising dict while only classes or instances are allowed (raising-bad-type) This error stems from admin_api.py class

If you're saying that repairing pylint is troublesome, you can ignore it for now and I'll help to fix it together. (Focus on your code logic)

As for automated Lint repair tools, I'm not very familiar with them. If there are any suitable ones, could recommend it and introduce them into our project, which would be even better

[chiruu12]

sir I have been going through them manually one by one, which is pretty time-consuming. Is there a better approach I'm missing? Maybe a tool or command that can automatically fix some of the common pylint issues instead of just listing them? I'm not very experienced with pylint and wondered if there's a more efficient way to handle this than the manual corrections I've been doing. Like I have contributed to openvino once and they told me to run a small command using black and then the issue with spacing was resolved automatically
and I saw that there are a few errors which are coming like the FIXME: line 31: E0702: Raising dict while only classes or instances are allowed (raising-bad-type) This error stems from admin_api.py class

If you're saying that repairing pylint is troublesome, you can ignore it for now and I'll help to fix it together.

As for automated Lint repair tools, I'm not very familiar with them. If there are any suitable ones, could recommend it and introduce them into our project, which would be even better

Thank you so much sir I really appreciate your guidance. I'm still getting up to speed with best practices, but I'm confident about handling the code logic correctly. If any issues arise, I'll make sure to work on them swiftly.

[chiruu12]

If you're saying that repairing pylint is troublesome, you can ignore it for now and I'll help to fix it together. (Focus on your code logic)

As for automated Lint repair tools, I'm not very familiar with them. If there are any suitable ones, could recommend it and introduce them into our project, which would be even better

Sir I check the repo I was telling you about shall I share the script that the used?
I think they used black and ruff

[imbajin]

Sir I check the repo I was telling you about shall I share the script that the used? I think they used black and ruff

Okay, regarding the issue of lint and automatic formatting, we can create a separate PR or issue for management, and we will keep the current PR independent

[chiruu12]

@imbajin Sir please check the pr and let me know if there is any changes that I need to make

[imbajin]

@imbajin Sir please check the pr and let me know if there is any changes that I need to make

Fine, we add a separate async_streaming_generate in the latest change/commit, seems we could use it to provide the aysnc_streaming api?

refer

incubator-hugegraph-ai/hugegraph-llm/src/hugegraph_llm/models/llms/base.py

Line 51 in 7ae5d6f

async def agenerate_streaming(

PS: Remember pull code first~

[imbajin]

Hi, could tell me how u test the APIs? By directly request them?

The gradio UI loss the API link now

Before:

Now:

Maybe refer here: (Or Gradio's mount doc?)

incubator-hugegraph-ai/hugegraph-llm/src/hugegraph_llm/demo/rag_demo/app.py

Line 182 in 7ae5d6f

app = gr.mount_gradio_app(

[imbajin]

'await' operator can't be used on list?

[imbajin]

so as here

[imbajin]

if we use a async api call, seems we should also update the usage in app.py?

still not work as expected (Also note admin_api need used in the same way

[chiruu12]

I will try to resolve the issues and then I will ping you again sir!

[imbajin]

@imbajin Sir please check the pr and let me know if there is any changes that I need to make

Fine, we add a separate async_streaming_generate in the latest change/commit, seems we could use it to provide the aysnc_streaming api?

refer

incubator-hugegraph-ai/hugegraph-llm/src/hugegraph_llm/models/llms/base.py

Line 51 in 7ae5d6f

async def agenerate_streaming(

PS: Remember pull code first~

@chiruu12 Also note the stream_http_api is also not bounded to gradio, and it seems that the design may need to be improved. (and remember to use agenerate_streaming() function if we can)

And it's better to follow the design of LLM such as OpenAI to provide a stream parameter instead of adding a separate API path. The internal logic can be implemented separately in two functions to provide a unified interface for the user layer

[chiruu12]

@imbajin Sir please check the pr and let me know if there is any changes that I need to make

Fine, we add a separate async_streaming_generate in the latest change/commit, seems we could use it to provide the aysnc_streaming api?
refer

incubator-hugegraph-ai/hugegraph-llm/src/hugegraph_llm/models/llms/base.py

Line 51 in 7ae5d6f

async def agenerate_streaming(

PS: Remember pull code first~

@chiruu12 Also note the stream_http_api is also not bounded to gradio, and it seems that the design may need to be improved. (and remember to use agenerate_streaming() function if we can)

And it's better to follow the design of LLM such as OpenAI to provide a stream parameter instead of adding a separate API path. The internal logic can be implemented separately in two functions to provide a unified interface for the user layer

understood sir I will try to get this done as soon as my exams are over and then maybe we can work on the other issue too!
Also had made an elementary demo for the agent. Let me know if you will get some time to review it too.