Sync ChromaDB with Repository Ontology Changes

[turbomam]

Summary

ChromaDB semantic mapping database (chroma_ols20_nonols4) contains MEO and MISO ontologies that were removed from the repository in #311. This creates an inconsistency between the database and the current ontology configuration.

Current State

ChromaDB contains (24 ontologies, 452,942 embeddings):

• meo: 2,499 embeddings (Metagenome and Environment Ontology) ❌ REMOVED in Remove Unused Non-OLS BioPortal Ontologies #311
• miso: 387 embeddings (Microbial Isolation Source Ontology) ❌ REMOVED in Remove Unused Non-OLS BioPortal Ontologies #311
• d3o: 283 embeddings ✅ Kept
• n4l_merged: 454 embeddings ✅ Kept
• omp: 2,440 embeddings ✅ Kept
• 19 OLS ontologies ✅

Repository state (post #311):

• Non-OLS downloads: D3O, MPO, OMP
• MEO and MISO removed (not referenced in METPO, low ROI)

Issues

1. Misleading Database Name

chroma_ols20_nonols4 implies:

• 20 OLS ontologies ✅ Correct
• 4 non-OLS ontologies ❌ Actually has 4, but wrong ones (includes meo/miso, missing mpo)

Should be: d3o, n4l_merged, omp, mpo (mpo not in ChromaDB either)

2. Semantic Mapping Uses Removed Ontologies

Queries against ChromaDB will still match MEO/MISO terms, inconsistent with repository decision to remove them.

Potential Actions

Option A: Rebuild ChromaDB (Clean State)

Remove from ChromaDB:

• meo (2,499 embeddings)
• miso (387 embeddings)

Add to ChromaDB:

• mpo (320 terms) - if we're keeping it downloaded

Result: ChromaDB matches repository downloads exactly

Cost: Re-embedding 320 MPO terms ($0.01), rebuild database

Option B: Remove MPO from Downloads

If MPO truly has 0 METPO references and isn't in ChromaDB, why keep it?

Current non-OLS status:

• OMP: 95 METPO refs ✅ Keep
• D3O: 0 refs, in ChromaDB ✅ Keep (strategic reserve)
• MPO: 0 refs, NOT in ChromaDB ❓ Remove?

Option C: Archive Current ChromaDB

Rename chroma_ols20_nonols4 → chroma_ols20_nonols4_legacy_2025-10-31

• Preserves historical evaluation results
• Build new database matching current state

Recommendation

Option A + B combined:

1. Remove MPO from Makefile (0 refs, not in ChromaDB, no value)
2. Rebuild ChromaDB without MEO/MISO
3. Final state: ChromaDB contains d3o, n4l_merged, omp (matches downloads)

Related

• Remove Unused Non-OLS BioPortal Ontologies #311 - Removed MEO, MISO, and 8 other unused ontologies
• docs/ONTOLOGY_SELECTION_SUMMARY.md - Original evaluation methodology

Notes

• D3O/MISO IRIs: Both use https://purl.dsmz.de/schema/ (DSMZ ontologies)
• MEO removed because: ENVO is preferred for environmental terms (4 ENVO refs in METPO)
• MISO removed because: Isolation source ontology, 0 METPO references, ROI 15.50

[turbomam]

See #364 for a broader re-evaluation of the ChromaDB pipeline. The sync question here is still valid for the interim, but if we migrate to OLS4 API + linkml-store (#364), the local ChromaDB collections would be retired or drastically reduced (only BioPortal-only ontologies).

[turbomam]

Superseded: the ChromaDB pipeline is being removed (PR #498 / decision #364), so 'sync ChromaDB with repo ontology changes' is moot. The replacement (OLS4 /llm_search + linkml-store) re-queries live, so there's no local DB to keep in sync.