Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(llm): enable tasks concurrency configs in Gradio #188

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

fix(llm): enable tasks concurrency configs in Gradio #188

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e4b430c
[Feature] #185: Improvement of Prompt Configuration for rag_demo Grap…
Kryst4lDem0ni4s Mar 2, 2025
50678a7
feat(rag_demo): add task ID generation to enabled concurrency with Gr…
Kryst4lDem0ni4s Mar 2, 2025
50a8966
[Feature] #185: Improvement of Prompt Configuration for rag_demo Grap…
Kryst4lDem0ni4s Mar 2, 2025
885514d
Merge branch 'gradioconcurrency' of https://github.com/Kryst4lDem0ni4…
Kryst4lDem0ni4s Mar 2, 2025
d65881a
Merge branch 'main' into gradioconcurrency
imbajin Mar 3, 2025
418df29
move decorator
imbajin Mar 3, 2025
2e11b1f
fix #176: refactor(demo) switched gr.Blocks() concurrency to gr.Butto…
Kryst4lDem0ni4s Mar 3, 2025
c1aa28f
fix: improve exception handling in HGraphConfig for API version retri…
Kryst4lDem0ni4s Mar 3, 2025
ddd3f46
refactor: enhance task ID logging and improve exception handling in H…
Kryst4lDem0ni4s Mar 3, 2025
c684d45
Merge branch 'main' into gradioconcurrency
imbajin Mar 6, 2025
75abdf9
Merge branch 'main' into gradioconcurrency
imbajin Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 62 additions & 51 deletions hugegraph-llm/src/hugegraph_llm/demo/rag_demo/rag_block.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@

from hugegraph_llm.config import resource_path, prompt, huge_settings, llm_settings
from hugegraph_llm.operators.graph_rag_task import RAGPipeline
from hugegraph_llm.utils.decorators import with_task_id
from hugegraph_llm.utils.log import log


@with_task_id
def rag_answer(
text: str,
raw_answer: bool,
Expand Down Expand Up @@ -126,62 +128,68 @@ def create_rag_block():
gr.Markdown("""## 1. HugeGraph RAG Query""")
with gr.Row():
with gr.Column(scale=2):
inp = gr.Textbox(value=prompt.default_question, label="Question", show_copy_button=True, lines=3)
with gr.Blocks().queue(max_size=20, default_concurrency_limit=5):
inp = gr.Textbox(value=prompt.default_question, label="Question", show_copy_button=True, lines=3)

# TODO: Only support inline formula now. Should support block formula
gr.Markdown("Basic LLM Answer", elem_classes="output-box-label")
raw_out = gr.Markdown(elem_classes="output-box", show_copy_button=True, latex_delimiters=[{"left":"$", "right":"$", "display":False}])
gr.Markdown("Vector-only Answer", elem_classes="output-box-label")
vector_only_out = gr.Markdown(elem_classes="output-box", show_copy_button=True, latex_delimiters=[{"left":"$", "right":"$", "display":False}])
gr.Markdown("Graph-only Answer", elem_classes="output-box-label")
graph_only_out = gr.Markdown(elem_classes="output-box", show_copy_button=True, latex_delimiters=[{"left":"$", "right":"$", "display":False}])
gr.Markdown("Graph-Vector Answer", elem_classes="output-box-label")
graph_vector_out = gr.Markdown(elem_classes="output-box", show_copy_button=True, latex_delimiters=[{"left":"$", "right":"$", "display":False}])
# TODO: Only support inline formula now. Should support block formula
gr.Markdown("Basic LLM Answer", elem_classes="output-box-label")
raw_out = gr.Markdown(elem_classes="output-box", show_copy_button=True,
latex_delimiters=[{"left": "$", "right": "$", "display": False}])
gr.Markdown("Vector-only Answer", elem_classes="output-box-label")
vector_only_out = gr.Markdown(elem_classes="output-box", show_copy_button=True,
latex_delimiters=[{"left": "$", "right": "$", "display": False}])
gr.Markdown("Graph-only Answer", elem_classes="output-box-label")
graph_only_out = gr.Markdown(elem_classes="output-box", show_copy_button=True,
latex_delimiters=[{"left": "$", "right": "$", "display": False}])
gr.Markdown("Graph-Vector Answer", elem_classes="output-box-label")
graph_vector_out = gr.Markdown(elem_classes="output-box", show_copy_button=True,
latex_delimiters=[{"left": "$", "right": "$", "display": False}])

answer_prompt_input = gr.Textbox(
value=prompt.answer_prompt, label="Query Prompt", show_copy_button=True, lines=7
)
keywords_extract_prompt_input = gr.Textbox(
value=prompt.keywords_extract_prompt,
label="Keywords Extraction Prompt",
show_copy_button=True,
lines=7,
)
with gr.Column(scale=1):
with gr.Row():
raw_radio = gr.Radio(choices=[True, False], value=False, label="Basic LLM Answer")
vector_only_radio = gr.Radio(choices=[True, False], value=False, label="Vector-only Answer")
with gr.Row():
graph_only_radio = gr.Radio(choices=[True, False], value=True, label="Graph-only Answer")
graph_vector_radio = gr.Radio(choices=[True, False], value=False, label="Graph-Vector Answer")

def toggle_slider(enable):
return gr.update(interactive=enable)
answer_prompt_input = gr.Textbox(
value=prompt.answer_prompt, label="Query Prompt", show_copy_button=True, lines=7
)
keywords_extract_prompt_input = gr.Textbox(
value=prompt.keywords_extract_prompt,
label="Keywords Extraction Prompt",
show_copy_button=True,
lines=7,
)

with gr.Column():
with gr.Column(scale=1):
with gr.Row():
online_rerank = llm_settings.reranker_type
rerank_method = gr.Dropdown(
choices=["bleu", ("rerank (online)", "reranker")] if online_rerank else ["bleu"],
value="reranker" if online_rerank else "bleu",
label="Rerank method",
)
example_num = gr.Number(value=2, label="Template Num (0 to disable it) ", precision=0)
graph_ratio = gr.Slider(0, 1, 0.6, label="Graph Ratio", step=0.1, interactive=False)
raw_radio = gr.Radio(choices=[True, False], value=False, label="Basic LLM Answer")
vector_only_radio = gr.Radio(choices=[True, False], value=False, label="Vector-only Answer")
with gr.Row():
graph_only_radio = gr.Radio(choices=[True, False], value=True, label="Graph-only Answer")
graph_vector_radio = gr.Radio(choices=[True, False], value=False, label="Graph-Vector Answer")

graph_vector_radio.change(
toggle_slider, inputs=graph_vector_radio, outputs=graph_ratio
) # pylint: disable=no-member
near_neighbor_first = gr.Checkbox(
value=False,
label="Near neighbor first(Optional)",
info="One-depth neighbors > two-depth neighbors",
)
custom_related_information = gr.Text(
prompt.custom_rerank_info,
label="Query related information(Optional)",
)
btn = gr.Button("Answer Question", variant="primary")
def toggle_slider(enable):
return gr.update(interactive=enable)

with gr.Column():
with gr.Row():
online_rerank = llm_settings.reranker_type
rerank_method = gr.Dropdown(
choices=["bleu", ("rerank (online)", "reranker")] if online_rerank else ["bleu"],
value="reranker" if online_rerank else "bleu",
label="Rerank method",
)
example_num = gr.Number(value=2, label="Template Num (0 to disable it) ", precision=0)
graph_ratio = gr.Slider(0, 1, 0.6, label="Graph Ratio", step=0.1, interactive=False)

graph_vector_radio.change(
toggle_slider, inputs=graph_vector_radio, outputs=graph_ratio
) # pylint: disable=no-member
near_neighbor_first = gr.Checkbox(
value=False,
label="Near neighbor first(Optional)",
info="One-depth neighbors > two-depth neighbors",
)
custom_related_information = gr.Text(
prompt.custom_rerank_info,
label="Query related information(Optional)",
)
btn = gr.Button("Answer Question", variant="primary")

btn.click( # pylint: disable=no-member
fn=rag_answer,
Expand All @@ -200,6 +208,9 @@ def toggle_slider(enable):
example_num,
],
outputs=[raw_out, vector_only_out, graph_only_out, graph_vector_out],
# queue=True, # Enable queueing for this event
# concurrency_limit=5, # Maximum of 5 concurrent executions
Copy link
Member

@imbajin imbajin Mar 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe the two lines 211~212 modification is enough for now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright I'll modify it to incorporate the buttons for now.

concurrency_id="rag_answer_task"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so as here

)

gr.Markdown(
Expand Down
11 changes: 11 additions & 0 deletions hugegraph-llm/src/hugegraph_llm/utils/decorators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,14 @@ def wrapper(*args: Any, **kwargs: Any) -> Any:
log.debug("%s QPS: %f/s", args[0].__class__.__name__, qps)
return result
return wrapper

# A decorator to wrap functions with a task id generation and logging
def with_task_id(func):
def wrapper(*args, **kwargs):
import uuid
task_id = str(uuid.uuid4())
log.info(f"New task created with id: {task_id}")
# Optionally, you could also pass the task_id to the function if needed:
# kwargs['task_id'] = task_id
return func(*args, **kwargs)
return wrapper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this decorator still useful now or in the future? Is the main purpose of logging newly created tasks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this one is useful, it's for generating concurrency ids, while the other is a more static general id useful for logging references (the one in the button) and can be modified or removed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this one is useful, it's for generating concurrency ids, while the other is a more static general id useful for logging references (the one in the button) and can be modified or removed.

We can keep it, but I still don't understand the purpose of recording this taskID separately in the click function? It seems that there is no significant difference even if I don't use this decorator and record it?

BTW, our logs are already too numerous and need to be streamlined as much as possible 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, I'll try to streamline the logs next.
And you're right, there isnt any significant usage of the click function's taskID, it is merely for identifying the ongoing process so that it is clearer to the dev, which process' ID was being generated, but we can reduce this too if it is unneccesary. As for the decorator, the task_id = str(uuid.uuid4()) is being used to identify any concurrent processes being run, so that we can differentiate ongoing processes per task_id for further debugging as per #176 . I hope this is useful for further development and scaling

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, I'll try to streamline the logs next.
And you're right, there isnt any significant usage of the click function's taskID, it is merely for identifying the ongoing process so that it is clearer to the dev, which process' ID was being generated, but we can reduce this too if it is unneccesary. As for the decorator, the task_id = str(uuid.uuid4()) is being used to identify any concurrent processes being run, so that we can differentiate ongoing processes per task_id for further debugging as per #176 . I hope this is useful for further development and scaling

I haven't found out how it works in my local tests. Can you give me an example of how to use it?

11 changes: 8 additions & 3 deletions hugegraph-python-client/src/pyhugegraph/utils/huge_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
# under the License.

import re
import sys
import traceback
from dataclasses import dataclass, field
from typing import List, Optional
Expand Down Expand Up @@ -63,8 +64,12 @@ def __post_init__(self):
)

except Exception as e: # pylint: disable=broad-exception-caught
traceback.print_exception(e)
self.gs_supported = False
log.warning(
try:
traceback.print_exception(e)
self.gs_supported = False
except Exception as e:
exc_type, exc_value, tb = sys.exc_info()
traceback.print_exception(exc_type, exc_value, tb)
log.warning(
"Failed to retrieve API version information from the server, reverting to default v1."
)
Loading