Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(llm) : Streaming output generation and update gradio to latest version #197

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

Aryankb
Copy link
Contributor

@Aryankb Aryankb commented Mar 7, 2025

Fixes #173
Fixes:-

  • Enabled streaming output generation in rag_block -> rag_answer, can be applied to all other output boxes of gradio
  • updated gradio to 5.20.0, no breaking changes found.
  • exploring more gradio features, will try to update them with future issues.

Modifications:-
used gr.update() in rag_block to generate streaming output.

How to Test:-
Run python3 -m hugegraph_llm.demo.rag_demo.app in CLI
click Answer question under RAG and user functions.
the output is streaming. (though it is not async. the final output is streaming after collecting all outputs)
Screencast from 03-07-2025 01:13:46 PM.webm

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Mar 7, 2025
@github-actions github-actions bot added the llm label Mar 7, 2025
@dosubot dosubot bot added the dependencies Pull requests that update a dependency file label Mar 7, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Mar 7, 2025
@Aryankb
Copy link
Contributor Author

Aryankb commented Mar 7, 2025

@imbajin
i tried my best to have async streaming. but due to many different blocking functions, not able to do it.

@imbajin
Copy link
Member

imbajin commented Mar 7, 2025

i tried my best to have async streaming. but due to many different blocking functions, not able to do it.

@Aryankb Thank you for your attempt, we already implement a partially less elegant asynchronous streaming output by modifying the previous version.

We can keep this implementation first and let the current PR only upgrade the gradio version. If possible, a simple list of the useful features of 4.x -> 5.x for us is better.

@Aryankb
Copy link
Contributor Author

Aryankb commented Mar 7, 2025

@imbajin okay, so i just need to list the possible changes (useful features) or need to make changes in gradio?

Also,
After which PR merge, async streaming was working? Because I fetched the main branch yesterday, it was not working, so i made these changes.
And now it is streaming but not async (after my changes)

@imbajin
Copy link
Member

imbajin commented Mar 7, 2025

@imbajin okay, so i just need to list the possible changes (useful features) or need to make changes in gradio?

Also, After which PR merge, async streaming was working? Because I fetched the main branch yesterday, it was not working, so i made these changes. And now it is streaming but not async (after my changes)

refer #190 (Added async func calls to the entire path, but some places have TODO)

After introducing the async workflow framework, we should make all current operator/step calls async. Currently, there are many synchronous places, so it is very troublesome to change them

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file llm size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update FE from 4.x -> latest (5.xx) & check compatibility carefully
2 participants