server: semantic similarity queries via sqlite-vec + canonical scoring via cq-schema

[javiermtorres]

Adds optional semantic similarity to the server's /query path using sqlite-vec and remote token embeddings, and migrates the server's scoring constants onto the canonical cq-schema source via the cq-sdk.

Closes #21.
Closes #332.

Semantic search (#21)

When TOKEN_EMBEDDING_URL is set and the semsearch extra is installed (sqlite-vec, numpy, httpx), SqliteStore:

• loads sqlite-vec into every pooled DBAPI connection via the SQLAlchemy connect event;
• creates a knowledge_units_vec table on schema bootstrap;
• upserts an embedding row for each unit on insert/update;
• in query(), fetches the per-domain candidate set joined with cosine distance via combined_query, then ranks units by relevance × confidence × Π(1 + logit(field)) so semantically closer units boost their score and far ones diminish it.

When the env var is unset (or the extra isn't installed) the path short-circuits via semsearch.is_enabled() and behaviour matches origin/main exactly. CI runs without the embedding server, so the modulation-specific test (test_query_returns_best_result_with_domain_overlap) skips by design.

Scoring canonicalisation (#332)

cq-sdk 0.10.0 reads its scoring constants from cq-schema's canonical scoring.values.json. With this change the server consumes those constants transitively via the SDK rather than holding its own copy.

• bump cq-sdk pin from ~=0.9.1 to ~=0.10.0 (pulls cq-schema 0.0.1)
• swap from .scoring imports in app.py, store/_sqlite.py, tests/test_store.py to from cq.scoring
• delete server/backend/src/cq_server/scoring.py and server/backend/tests/test_scoring.py (the SDK's cq.scoring is now the canonical Python implementation; SDK tests cover the same behaviours)
• add server/backend/tests/test_schema_oracle.py to validate that server-serialised KnowledgeUnits round-trip through cq_schema.load_schema("knowledge_unit")

Behaviour note: cq.scoring.calculate_relevance clamps to [0, 1]; the deleted local copy did not. In practice a no-op (Jaccard ∈ [0,1] and the four weights sum to 1.0), but it removes a class of latent bug from any future weight tweak.

Test plan

• make test-server-backend — 302 passed, 1 skipped (embedding-server-dependent)
• make lint-server-backend
• Manual smoke against a live embedding endpoint (out of CI scope)

[javiermtorres]

@copilot check if conflicts can be solved easily

[Copilot]

@copilot check if conflicts can be solved easily

Conflicts resolved in f7e3464. Here's what was needed:

• plugins/cq/server/ (modify/delete): Main deleted the Python MCP server (replaced by Go CLI). Accepted the deletion and ported our sqlite-vec changes to the new location.
• sdk/python/src/cq/store.py (content, rename-detected): Git detected that plugins/cq/server/cq_mcp/local_store.py was renamed to sdk/python/src/cq/store.py on main. Merged our sqlite-vec additions (_VEC_SCHEMA_SQL, embedding_model param, vec insert/update/delete/query logic) into main's updated version.
• sdk/python/tests/test_store.py (content, rename-detected): Similarly ported embedding_model fixture and updated LocalStore constructor calls.
• server/backend/uv.lock (content): Took main's pygments 2.20.0 version.
• team-api/pyproject.toml (modify/delete): Accepted deletion (renamed to server/backend/pyproject.toml on main; sqlite-vec now goes via the SDK dependency).

Also added sqlite-vec==0.1.7a10 and sentence-transformers[onnx] to sdk/python/pyproject.toml.

[peteski22]

@copilot rebase from origin/main and solve conflicts.

[Copilot]

@copilot rebase from origin/main and solve conflicts.

Done in 45f715e. The branch is now rebased on origin/main — the diff from main contains exactly our 3 intended files:

• sdk/python/src/cq/store.py — sqlite-vec virtual table + embedding model
• sdk/python/tests/test_store.py — embedding_model fixture + updated constructors
• sdk/python/pyproject.toml — sqlite-vec==0.1.7a10 and sentence-transformers[onnx] deps

Also fixed an unintended regression where server/backend/uv.lock had been modified by an old merge commit (restored to match main).

[javiermtorres]

Since the SDK and server stores were split at some point during the development of this PR, the agent tried to reconcile changes and this resulted in unwanted functionality.
@peteski22 as discussed, this will be implemented in the server sqlite instance using an encoderfile, behind a clean interface so other alternatives can be plugged in as well.

[javiermtorres]

Removed the sdk changes and kept semsearch to the server store.

[Copilot]

Pull request overview

This PR adds an optional semantic-search path to the server’s SQLite-backed RemoteStore using sqlite-vec + a remote embedding endpoint, and refactors the server store/API/tests to use async store methods so vector/HTTP work can be awaited during indexing and querying.

Changes:

• Introduces cq_server.semsearch for embedding generation, sqlite-vec schema management, and vector distance queries.
• Refactors RemoteStore APIs (and FastAPI routes + tests) to be async and integrates semantic upsert/query behavior.
• Updates server packaging/tooling to install semsearch deps via an optional extra and updates setup script target.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
server/backend/src/cq_server/semsearch.py	New semantic search helper module (embedding fetch + sqlite-vec integration).
server/backend/src/cq_server/store/init.py	Integrates semsearch into RemoteStore, converts many methods to async, and changes query scoring behavior.
server/backend/src/cq_server/app.py	Makes API routes async and awaits store operations.
server/backend/src/cq_server/review.py	Makes review endpoints async where they call async store methods.
server/backend/src/cq_server/auth.py	Makes auth endpoints async where they call async store methods.
server/backend/tests/test_app.py	Converts tests to async and adds semsearch-specific query tests/expectations.
server/backend/tests/test_store.py	Converts tests to async and adjusts confidence expectations based on query-time scoring.
server/backend/tests/test_review.py	Converts tests to async to align with async auth/store calls.
server/backend/tests/test_queries.py	Converts tests to async to align with async store calls.
server/backend/tests/test_migrations.py	Converts seeding helper to async; currently has a missing await.
server/backend/tests/test_auth.py	Converts tests/helpers to async for async auth/store.
server/backend/pyproject.toml	Adds semsearch optional extra deps; adjusts pytest logging config.
Makefile	Installs server backend dev deps with --extra semsearch.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

query() mutates the returned KnowledgeUnit by overwriting unit.evidence.confidence with a query-time relevance score. This breaks the semantics of confidence (which is validated as 0..1 in the model), can exceed bounds or become negative, and can leak into persistence if a caller later passes the mutated unit back to update() (tests here do exactly that). Keep confidence unchanged and use a separate local score for sorting (or return a (unit, score) structure at an API boundary that can represent ranking).

Suggested change

unit.evidence.confidence = combined_relevance # Update confidence to combined score for sorting

[peteski22]

Still valid. During the merge the mutation moved to _sqlite.py:_score_semsearch_rows (unit.evidence.confidence = combined). Same concern: blends ranking score with the model's persisted 0..1 invariant, and a caller round-tripping a query() result through update() would persist the modulated value. The new test_query_returns_best_result_with_domain_overlap depends on the mutation for relative-order assertions, so resolving requires re-shaping that test too. Tracking together with #3153159271.

[Copilot]

combined *= 1.0 + logit can flip the score negative when logit < -1 (e.g., if v/mean_val > e with invert=True). If this score is later exposed as/derived from confidence, it can violate expected non-negativity and ordering. Consider using a strictly-positive transform (e.g., combined *= exp(logit)), clamping, or otherwise bounding the adjustment so scores remain well-behaved.

[peteski22]

Still valid. The same multiplicative form lives in semsearch.compute_combined_relevance (moved during the merge from the deleted monolithic store/__init__.py to semsearch.py). If logit < -1 the score flips sign; needs real embeddings to trigger but matters for ranking robustness. Tracking.

[Copilot]

These assertions assume /query returns a modified evidence.confidence that reflects query-time ranking (0.35/0.33/0.2475). That makes the test (and API behavior) dependent on semsearch enablement and on the ranking formula, and it conflates persisted evidence confidence with retrieval score. Prefer asserting the returned unit IDs/order and keep confidence checks tied to the persisted confidence updates from confirm/flag (which should stay within 0..1).

[peteski22]

Still partially valid. The brittle exact-value assertions are gone, but the new test_query_returns_best_result_with_domain_overlap still asserts on result['evidence']['confidence'] returned from /query, which depends on the mutation in _sqlite.py:_score_semsearch_rows. Same root cause as #3153158840; tracked together.