Fix EmbeddingService blocking event loop by offloading encode to thread

[dolliecoder]

Fixes #261
Problem:
EmbeddingService.get_embedding() and get_embeddings() call SentenceTransformer.encode() inside async methods.
Since encode() is synchronous and CPU-heavy, it blocks the asyncio event loop and prevents concurrent requests.

Solution:
Moved encode calls to asyncio.to_thread() using a small synchronous helper method.

Result:
Embedding generation no longer blocks the event loop.
Improves concurrency without changing public API or behavior.

Summary by CodeRabbit

• Refactor
  • Improved performance and responsiveness of the embedding service for both single and batch embedding operations.

[coderabbitai]

📝 Walkthrough

Walkthrough

The EmbeddingService now offloads blocking model.encode() operations to a thread pool using asyncio.to_thread(), preventing synchronous encoding calls from blocking the event loop and allowing concurrent requests to proceed without stalling.

Changes

Cohort / File(s)	Summary
EmbeddingService Threading backend/app/services/embedding_service/service.py	Added _encode_sync() helper method to execute synchronous model encoding. Updated get_embedding() and get_embeddings() to invoke _encode_sync() via asyncio.to_thread(), offloading CPU-bound operations to a worker thread while maintaining async control flow.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant EventLoop as Event Loop
    participant ThreadPool as Thread Pool
    participant Model as Model.encode()
    
    rect rgba(100, 150, 200, 0.5)
    Note over Client,Model: Before: Blocking
    Client->>EventLoop: await get_embedding()
    EventLoop->>Model: model.encode() [BLOCKING]
    Note over EventLoop: ⚠ Other requests stall
    Model-->>EventLoop: result
    EventLoop-->>Client: embedding
    end
    
    rect rgba(100, 200, 100, 0.5)
    Note over Client,Model: After: Non-blocking
    Client->>EventLoop: await get_embedding()
    EventLoop->>ThreadPool: asyncio.to_thread(_encode_sync)
    Note over EventLoop: ✓ Event loop free
    ThreadPool->>Model: model.encode() [in thread]
    rect rgba(150, 200, 255, 0.5)
    Note over EventLoop: Other requests run
    end
    Model-->>ThreadPool: result
    ThreadPool-->>EventLoop: result
    EventLoop-->>Client: embedding
    end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 Hops through threads with glee,
No more blocking the loop, you see,
Async embedding runs free,
While concurrency flows with tea,
Event loop stays spry as can be! 🧵✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main objective of the PR—fixing the EmbeddingService blocking event loop by offloading encode to a thread, which aligns with the core change.
Linked Issues check	✅ Passed	The implementation meets all primary objectives: uses asyncio.to_thread for non-blocking encode calls, adds synchronous helper _encode_sync, preserves public async API, and converts tensors to lists.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fixing the event loop blocking issue in EmbeddingService; no unrelated modifications are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[vaishcodescape]

LGTM !

[dolliecoder]

LGTM !

@vaishcodescape do you think its ready to be merged?