Skip to content

Conversation

0fism
Copy link

@0fism 0fism commented Oct 8, 2025

  • Add QUERY_BINDING, QUERY_MODEL, QUERY_BINDING_HOST, QUERY_BINDING_API_KEY env vars
  • Create separate LLM function for query operations
  • Support all LLM providers (openai, azure_openai, ollama, lollms, aws_bedrock)
  • Enable cost optimization: powerful model for queries, economical for extraction

Description

This PR adds support for using separate LLM configurations for query/retrieval operations versus entity extraction, enabling cost optimization by using powerful models for queries while keeping extraction on economical models.

Related Issues

Addresses the need for cost optimization in LightRAG deployments where users want to use different LLM models for different operations (extraction vs. query).
closes #1382

Changes Made

  • Added query-specific environment variables in lightrag/api/config.py: QUERY_BINDING, QUERY_MODEL, QUERY_BINDING_HOST, QUERY_BINDING_API_KEY
  • Created create_query_llm_func() in lightrag/api/lightrag_server.py to initialize query-specific LLM with support for all providers (openai, azure_openai, ollama, lollms, aws_bedrock)
  • Enhanced condition logic to create separate query LLM when either binding OR model differs from extraction LLM

Checklist

  • Changes tested locally
  • [V] Code reviewed
  • Documentation updated (if necessary)
  • Unit tests added (if applicable)

Additional Notes

[Add any additional notes or context for the reviewer(s).]

@0fism 0fism force-pushed the feature/query-specific-llm branch from 1b01848 to 88b8ba6 Compare October 10, 2025 14:19
@0fism
Copy link
Author

0fism commented Oct 14, 2025

git commit --amend --no-edit
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
fix requirements.txt.................................(no files to check)Skipped
ruff-format..............................................................Passed
ruff.....................................................................Passed

- Add QUERY_BINDING, QUERY_MODEL, QUERY_BINDING_HOST, QUERY_BINDING_API_KEY env vars
- Create separate LLM function for query operations
- Support all LLM providers (openai, azure_openai, ollama, lollms, aws_bedrock)
- Enable cost optimization: powerful model for queries, economical for extraction
@0fism 0fism force-pushed the feature/query-specific-llm branch from 88b8ba6 to 4c81706 Compare October 16, 2025 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant