Remove the ChromaDB pipeline (resolves critical CVE #455 + #401) - #498
Merged
Conversation
The DIY ChromaDB semantic-search pipeline is being replaced by OLS4 embeddings + linkml-store + local GPU embeddings (#364). chromadb carries the unpatchable critical CVE #455 (no upstream fix), so removal is the only resolution; it also drops diskcache (#401). Removed: - chromadb dependency from the notebooks extra (uv.lock no longer contains chromadb or diskcache) - metpo/database/ (audit/combine/filter/migrate chromadb tools) - metpo/pipeline/chromadb_semantic_mapper.py, embed_ontology_to_chromadb.py - metpo/scripts/find_best_definitions.py, iterative_definition_improvement.py (the two chromadb-coupled definition tools; to be reworked on OLS4/linkml-store) - 7 chromadb-coupled CLI entry points; cli_common.chroma_path_option - the root Makefile's "Ontology Alignment Pipeline" section + 3 chromadb-dependent definition-report targets - notebooks/query_chromadb.ipynb (+ stale checkpoints) Kept: the non-chromadb pipeline/analysis scripts (analyze_matches, analyze_coherence, fetch_ontology_names, categorize_ontologies, the chromadb-free definition scripts) and notebooks/assess_ontology_by_api_search.ipynb (the OLS-API search notebook, to be reformulated as a Click CLI as the replacement). Not touched: the ontology build (src/ontology/Makefile) and CI — chromadb was never part of either. make test, ruff, and the 35-test suite pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced Jun 5, 2026
Closed
Closed
…db tools) Small, dependency-light CLI so the definition/synonym/mapping-finding capability isn't lost with the chromadb removal: OLS4 lexical search for candidates + local free embeddings (Ollama, parameterized via --model/--embed-url) for ranking, with a --no-embeddings lexical fallback. Output gives mapping CURIE, label (synonym), and definition (source) per match. Explicitly marked a STOPGAP in the module docstring: it duplicates capabilities of OAK/oaklib (#194) and linkml-store (#364) and should migrate to them, not grow. Tracked in those issues. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eplacement Same behavior, reorganized: extracted load_terms / rank_candidates / write_matches helpers (fixes PLR0912 too-many-branches), Path.open + context-managed output (PTH123/SIM115), zip(strict=False) (B905). Dropped the 'stopgap' framing per direction: this IS the chromadb replacement; OAK (#194) + linkml-store (#364) are the acknowledged future direction, not a blocking migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
# Conflicts: # pyproject.toml # uv.lock
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
Makefile:65
make helplists targets that no longer exist (e.g.help-alignment,generate-non-ols-tsvs,scan-manifest). Since those targets were removed, the help output is now misleading and suggests workflows that will fail when invoked.
@echo "Ontology Alignment Pipeline:"
@echo " make help-alignment - Show detailed alignment pipeline help"
@echo ""
@echo "External Ontology Downloads:"
@echo " make download-external-bioportal-ontologies - Download non-OLS ontologies"
@echo " make generate-non-ols-tsvs - Extract terms for embeddings"
@echo " make scan-manifest - Update ontology manifest"
@echo ""
- load_terms: skip the ROBOT template spec row (row 2, label 'LABEL') so it is not queried as a real term. - ols_candidates: catch URLError/timeout/JSON errors and skip the term instead of aborting the whole run on a transient OLS outage. - write_matches / load_terms: open files with explicit encoding='utf-8' so non-ASCII labels/definitions are locale-independent. - pyproject: drop 'openai' from the notebooks extra (nothing in notebooks/ imports it after the ChromaDB demo removal); refresh uv.lock. - Makefile: remove help text and epilogue lines pointing at targets this PR deleted (help-alignment, generate-non-ols-tsvs, scan-manifest, view-logs). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Author
|
Re the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of the ChromaDB re-scope (#364): remove the DIY ChromaDB semantic-search pipeline (it carries the unpatchable critical CVE #455, no upstream fix → removal is the resolution; also drops
diskcache/ #401), and add a small stopgap replacement so the definition/synonym/mapping-finding capability isn't lost.Security fix
uv.lockafter this change has 0chromadband 0diskcache. Resolves #455, #401.Removed
chromadbdep + themetpo/database/dir + the 2 chromadb pipeline files + 2 chromadb-coupled definition scripts + 7 entry points +cli_common.chroma_path_optionMakefile"Ontology Alignment Pipeline" + "Non-OLS Embedding" sections + 3 chromadb-dependent definition-report targetsnotebooks/query_chromadb.ipynb(+ stale checkpoints)Added (stopgap)
cross-ontology-search(metpo/pipeline/cross_ontology_search.py): OLS4 lexical search for candidates + free local embeddings (Ollama,--model/--embed-urlparameterized;--no-embeddingslexical fallback) for ranking. Output per match: external CURIE (mapping), label (synonym), definition (source). No chromadb, no OpenAI, no 300GB OLS SQLite.index/find-matcheswith local embeddings via thellmlibrary; OAK for ontology access + SSSOM + similarity; OLS4llm_searchserver-side) is recorded on Re-scope: replace the DIY ChromaDB pipeline (OLS4 embeddings + linkml-store + local GPU) #364 and add oak annotate capabilities to METPO's utility codebase #194.Kept
notebooks/assess_ontology_by_api_search.ipynb;download-external-bioportal-ontologies(reusable, embedding-agnostic)CI / build impact: none
chromadb was never in the ontology build or any workflow.
make test, the 35-test suite, and the package build pass.Resolves #455, #401. Advances #364, #441, #194.