feat(search): make AI Governance Studio assets vector-searchable

[pmbrull]

What

Makes the AI Governance Studio's asset entities reachable by vector and hybrid search. Today none of them is — llmModel, aiApplication, promptTemplate, mcpServer, aiGovernancePolicy and aiGovernanceFramework are indexed, but a semantic search for "which LLM models do we use?" returns nothing.

Why it was invisible

Vector search is gated in three independent places, and the AI entities were missing from all three. Any one of them alone is enough to make an asset unsearchable, and each fails silently.

Gate	Where	Effect when missing
Write path	AvailableEntityTypes.LIST	OpenSearchBulkSink.shouldEmbed / VectorEmbeddingHandler skip the entity, so no vector is ever computed
Read path	dataAssetEmbeddings parent alias in indexMapping.json	VectorIndexService.VECTOR_EMBEDDING_ALIAS is the query target; a non-member index cannot be returned even if its docs carry vectors
Index schema	fingerprint field in the type's index mapping file	OsUtils.addKnnVectorSettings detects embedding support by the presence of fingerprint and returns early without it — the embedding knn_vector field is never added to the index

The third one is the least obvious and the most consequential: without fingerprint the index physically has nowhere to store a vector, so fixing only the first two would still produce nothing.

Changes

1. AvailableEntityTypes.LIST — added the six types. This is the single write-path source of truth, consumed by OpenSearchBulkSink, VectorEmbeddingHandler, RecreateWithEmbeddings.coversAllVectorTypes, and OpenSearchVectorService's staged-chunk expected-type set, so one edit flows to all of them.

2. indexMapping.json — added dataAssetEmbeddings to the six entries' parentAliases. Members go 22 → 28. Deliberately not added to dataAsset, which drives Explore and has a much wider blast radius.

3. Index mapping files (6 types × 4 languages = 24 files) — added the fields VectorDocBuilder.buildEmbeddingFields writes on every entity doc: fingerprint, textToEmbed, textToLLMContext, chunkIndex, chunkCount, parentId. Copied verbatim from existing members.

4. AvailableEntityTypesConsistencyTest (new) — pins the three gates together.

Scope: which types, and why not the rest

In: llmModel, aiApplication, promptTemplate, mcpServer, aiGovernancePolicy, aiGovernanceFramework.

Out, deliberately:

• agentExecution, mcpExecution — execution records, reached by drilling into a parent, not searched by name. (agentExecution also has no ES index at all today; separate issue.)
• aiFrameworkControl, auditReport — leaves of a framework / generated artifacts, same reasoning.
• llmService, mcpService — no service type is in dataAssetEmbeddings; keeping that convention.

Pre-existing gap this surfaced

page and contextMemory are vector-indexable and VectorDocBuilder.buildEmbeddingFields writes textToLLMContext onto their docs, but neither mapping declared the field — Elasticsearch was creating it via dynamic mapping. Now declared explicitly as text, matching the dynamic default and every other member, so no behavior change. Flagging it because it is outside the AI-entity scope: happy to split it out if preferred.

Testing

AvailableEntityTypesConsistencyTest asserts:

• AvailableEntityTypes.SET equals the dataAssetEmbeddings alias membership exactly, in both directions, with a message naming the drifting types ("embedded but not searchable" vs "in the alias but never embedded").
• Every vector-indexable type declares all six embedding fields, in every language variant present.

Verified the test is not vacuous: removing "llmModel" from AvailableEntityTypes fails it with types in the alias but never embedded: [llmmodel], and the field assertion is what caught the page/contextMemory gap above.

mvn -pl openmetadata-service test -Dtest='AvailableEntityTypesConsistencyTest,IndexMappingVersionTrackerTest,\
IndexMappingNestedFieldConsistencyTest,IndexAnalyzerMappingTest,RecreateWithEmbeddingsTest,VectorDocBuilder*Test,\
VectorEmbeddingHandlerTest,OpenSearchVectorService*Test,ElasticSearchVectorServiceTest,DefaultRecreateHandlerTest'

Tests run: 196, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

Upgrade note

Existing clusters need a search reindex for the six AI types to pick up the new knn_vector mapping and get embeddings written. Until then they behave exactly as they do today (indexed, keyword-searchable, absent from vector results) — no regression, just no improvement.

Follow-ups (not in this PR)

• agentExecution has no search index at all, though it carries modelCalls, toolCalls, dataAccessed and complianceChecks. mcpExecution is indexed but its only parent alias is mcpServer, so it is outside all too.
• aiApplication.dataSources / knowledgeBases / primaryModel are entityReferenceLists, not lineage edges — AIApplicationRepository creates no lineage relationships, so "which AI applications read this table?" is unanswerable in either direction.

Consumer-side context: open-metadata/ai-platform#975.

🤖 Generated with Claude Code

Greptile Summary

The PR makes six AI Governance Studio entity types eligible for vector and hybrid search by aligning embedding generation, alias membership, and index mappings.

• Adds the AI entity types to the vector write-path allowlist.
• Adds their indices to the shared dataAssetEmbeddings read alias.
• Declares embedding and chunk metadata fields across all supported language mappings.
• Adds a consistency test to prevent the three vector-search gates from drifting.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/utils/AvailableEntityTypes.java	Adds the six AI Governance Studio entity types to the centralized vector-indexing allowlist.
openmetadata-service/src/test/java/org/openmetadata/service/search/vector/utils/AvailableEntityTypesConsistencyTest.java	Adds coverage ensuring the write allowlist, read alias, and localized embedding mappings remain synchronized.
openmetadata-spec/src/main/resources/elasticsearch/indexMapping.json	Adds the six AI entity indices to the vector-search alias without adding them to the broader dataAsset alias.
openmetadata-spec/src/main/resources/elasticsearch/en/ai_application_index_mapping.json	Representative AI mapping now declares the fields required for vector documents and chunk metadata.
openmetadata-spec/src/main/resources/elasticsearch/en/llm_model_index_mapping.json	Adds the vector-document fields needed to store and retrieve LLM model embeddings.
openmetadata-spec/src/main/resources/elasticsearch/en/mcp_server_index_mapping.json	Adds the vector-document fields needed to store and retrieve MCP server embeddings.
openmetadata-spec/src/main/resources/elasticsearch/en/context_memory_search_index.json	Explicitly declares the existing textToLLMContext document field.
openmetadata-spec/src/main/resources/elasticsearch/en/knowledge_page_search_index.json	Explicitly declares the existing textToLLMContext document field.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  E[AI governance entity] --> W[AvailableEntityTypes write gate]
  W --> B[Vector embedding generation]
  B --> M[Index mapping with embedding fields]
  M --> A[dataAssetEmbeddings alias]
  A --> Q[Vector or hybrid search]
  T[Consistency test] -. validates .-> W
  T -. validates .-> M
  T -. validates .-> A

Loading

_{Reviews (5): Last reviewed commit: "Merge branch 'main' into pmbrull/ai-enti..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Metadata platform service

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 22b3413a2cfba82ca8fb55f011db2bfc6000a4a4 in Playwright run 32627354112, attempt 1.

✅ 1279 passed · ❌ 0 failed · 🟡 4 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 51m 16s

⏱️ Max setup 4m 31s · max shard execution 18m 45s · max shard-job elapsed before upload 21m 45s · reporting 10s

🌐 200.40 requests/attempt · 2.13 app boots/UI scenario · 14.49% common-shard skew

Optimization targets still in progress:

• Browser traffic was 200.4 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.13 per UI scenario (2787 boots / 1310 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	138	0	0	0	0	0
✅ Shard chromium-02	143	0	0	0	0	0
🟡 Shard chromium-03	153	0	1	0	0	0
🟡 Shard chromium-04	168	0	1	0	0	0
🟡 Shard chromium-05	180	0	1	0	0	0
✅ Shard chromium-06	153	0	0	0	0	0
🟡 Shard chromium-07	155	0	1	0	0	0
✅ Shard data-asset-rules-01	61	0	0	0	0	0
✅ Shard domain-isolation-01	16	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard import-export-01	7	0	0	0	0	0
✅ Shard ingestion-01	2	0	0	0	0	0
✅ Shard reindex-01	28	0	0	0	0	0
✅ Shard search-01	12	0	0	0	0	0
✅ Shard search-rbac-01	29	0	0	0	0	0

🟡 4 flaky test(s) (passed on retry)

• Features/Table.spec.ts › should persist page size (shard chromium-03, 1 retry)
• Pages/Glossary.spec.ts › Approve and reject glossary term from Glossary Listing (shard chromium-04, 1 retry)
• Pages/ExplorePageRightPanel_KnowledgeCenter.spec.ts › Should remove user owner for knowledgeCenter (shard chromium-05, 1 retry)
• Pages/Entity.spec.ts › User as Owner with unsorted list (shard chromium-07, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved

Enables vector and hybrid search for AI Governance Studio assets by updating write-path entity lists, index mapping aliases, and schema definitions across all languages. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud