feat: implement concept graph search with depth-2 BFS expansion #349
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| import type { ISdk } from "iii-sdk"; | ||
| import type { StateKV } from "../state/kv.js"; | ||
| import type { Memory } from "../types.js"; | ||
| import { KV } from "../state/schema.js"; | ||
| import { recordAudit } from "./audit.js"; | ||
| import { logger } from "../logger.js"; | ||
|
|
||
| const CONFIG_KEY = "concept-backfill-done"; | ||
|
|
||
| export function registerConceptBackfillFunction(sdk: ISdk, kv: StateKV): void { | ||
| sdk.registerFunction( | ||
| "mem::concept-backfill", | ||
| async () => { | ||
| const flag = await kv.get<{ done: boolean }>(KV.config, CONFIG_KEY); | ||
| if (flag?.done) { | ||
| return { success: true, skipped: true, reason: "already completed" }; | ||
| } | ||
|
Comment on lines
+14
to
+17
There was a problem hiding this comment. Backfill can be permanently marked “done” after partial failure and can double-run concurrently. Line 36 swallows upsert failures, Line 39 still counts them as processed, and Line 42 sets Suggested direction- const flag = await kv.get<{ done: boolean }>(KV.config, CONFIG_KEY);
+ const flag = await kv.get<{ done: boolean; inProgress?: boolean }>(KV.config, CONFIG_KEY);
if (flag?.done) {
return { success: true, skipped: true, reason: "already completed" };
}
+ if (flag?.inProgress) {
+ return { success: true, skipped: true, reason: "already running" };
+ }
+ await kv.set(KV.config, CONFIG_KEY, { done: false, inProgress: true });
const memories = await kv.list<Memory>(KV.memories);
@@
- let processed = 0;
+ let processed = 0;
+ let failed = 0;
@@
- await Promise.all(
- batch.map((m) =>
- sdk
- .trigger({
- function_id: "mem::concept-edge-upsert",
- payload: { concepts: m.concepts },
- })
- .catch(() => {}),
- ),
- );
- processed += batch.length;
+ const settled = await Promise.allSettled(
+ batch.map((m) =>
+ sdk.trigger({
+ function_id: "mem::concept-edge-upsert",
+ payload: { concepts: m.concepts },
+ }),
+ ),
+ );
+ for (const r of settled) {
+ if (r.status === "fulfilled") processed += 1;
+ else failed += 1;
+ }
}
- await kv.set(KV.config, CONFIG_KEY, { done: true, completedAt: new Date().toISOString() });
+ const completedAt = new Date().toISOString();
+ await kv.set(KV.config, CONFIG_KEY, {
+ done: failed === 0,
+ inProgress: false,
+ completedAt,
+ failed,
+ });Also applies to: 29-43 🤖 Prompt for AI Agents |
||
|
|
||
| const memories = await kv.list<Memory>(KV.memories); | ||
| const eligible = memories.filter( | ||
| (m) => m.isLatest !== false && m.concepts && m.concepts.length >= 2, | ||
| ); | ||
|
|
||
| let processed = 0; | ||
| let errors = 0; | ||
| const batchSize = 50; | ||
|
|
||
| for (let i = 0; i < eligible.length; i += batchSize) { | ||
| const batch = eligible.slice(i, i + batchSize); | ||
| const results = await Promise.allSettled( | ||
| batch.map((m) => | ||
| sdk.trigger({ | ||
| function_id: "mem::concept-edge-upsert", | ||
| payload: { concepts: m.concepts }, | ||
| }) | ||
| ), | ||
| ); | ||
| for (const res of results) { | ||
| if (res.status === "rejected") errors++; | ||
| else processed++; | ||
| } | ||
| } | ||
|
|
||
| if (errors > 0) { | ||
| throw new Error(`Concept backfill failed to process ${errors} items.`); | ||
| } | ||
|
|
||
| await kv.set(KV.config, CONFIG_KEY, { done: true, completedAt: new Date().toISOString() }); | ||
|
|
||
| try { | ||
| await recordAudit(kv, "concept_backfill", "mem::concept-backfill", [], { | ||
| memoriesProcessed: processed, | ||
| totalMemories: memories.length, | ||
| }); | ||
| } catch {} | ||
|
|
||
| logger.info("Concept backfill completed", { | ||
| processed, | ||
| total: eligible.length, | ||
| }); | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| return { success: true, processed, total: memories.length }; | ||
| }, | ||
| ); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| import type { ISdk } from "iii-sdk"; | ||
| import type { StateKV } from "../state/kv.js"; | ||
| import type { ConceptEdge } from "../types.js"; | ||
| import { KV, fingerprintId } from "../state/schema.js"; | ||
| import { recordAudit } from "./audit.js"; | ||
|
|
||
| function reinforceEdge(edge: ConceptEdge): void { | ||
| const now = new Date().toISOString(); | ||
| edge.reinforcements++; | ||
| edge.strength = Math.min( | ||
| 1.0, | ||
| edge.strength + 0.1 * (1 - edge.strength), | ||
| ); | ||
| edge.lastSeenAt = now; | ||
| } | ||
|
|
||
| function generatePairs(concepts: string[]): Array<[string, string]> { | ||
| const normalized = [...new Set(concepts.map((c) => c.toLowerCase().trim()).filter(Boolean))]; | ||
| const pairs: Array<[string, string]> = []; | ||
| for (let i = 0; i < normalized.length; i++) { | ||
| for (let j = i + 1; j < normalized.length; j++) { | ||
| const [a, b] = normalized[i] < normalized[j] | ||
| ? [normalized[i], normalized[j]] | ||
| : [normalized[j], normalized[i]]; | ||
| pairs.push([a, b]); | ||
| } | ||
| } | ||
| return pairs; | ||
| } | ||
|
|
||
| export function registerConceptEdgesFunctions(sdk: ISdk, kv: StateKV): void { | ||
| sdk.registerFunction( | ||
| "mem::concept-edge-upsert", | ||
| async (data: { concepts?: string[] }) => { | ||
| if (!data.concepts || !Array.isArray(data.concepts) || data.concepts.length < 2) { | ||
| return { success: false, error: "at least 2 concepts required" }; | ||
| } | ||
|
|
||
| const pairs = generatePairs(data.concepts); | ||
| if (pairs.length === 0) { | ||
| return { success: true, created: 0, reinforced: 0 }; | ||
| } | ||
|
|
||
| const now = new Date().toISOString(); | ||
| let created = 0; | ||
| let reinforced = 0; | ||
|
|
||
| const edgeOps = pairs.map(async ([from, to]) => { | ||
| const id = fingerprintId("ce", `${from}|${to}`); | ||
| const existing = await kv.get<ConceptEdge>(KV.conceptEdges, id); | ||
|
|
||
| if (existing) { | ||
| reinforceEdge(existing); | ||
| await kv.set(KV.conceptEdges, id, existing); | ||
| reinforced++; | ||
| } else { | ||
| const edge: ConceptEdge = { | ||
| id, | ||
| from, | ||
| to, | ||
| strength: 0.5, | ||
| reinforcements: 0, | ||
| lastSeenAt: now, | ||
| createdAt: now, | ||
| }; | ||
| await kv.set(KV.conceptEdges, id, edge); | ||
| created++; | ||
| } | ||
| }); | ||
|
|
||
| await Promise.all(edgeOps); | ||
|
|
||
| try { | ||
| await recordAudit(kv, "concept_edge_upsert", "mem::concept-edge-upsert", [], { | ||
| pairs: pairs.length, | ||
| created, | ||
| reinforced, | ||
| }); | ||
| } catch {} | ||
|
|
||
| return { success: true, created, reinforced }; | ||
| }, | ||
| ); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| import type { ISdk } from "iii-sdk"; | ||
| import type { StateKV } from "../state/kv.js"; | ||
| import type { ConceptEdge, Memory } from "../types.js"; | ||
| import { KV } from "../state/schema.js"; | ||
|
|
||
| const MAX_BFS_DEPTH = 2; | ||
| const MAX_NEIGHBORS_PER_NODE = 10; | ||
|
|
||
| function decayedStrength(edge: ConceptEdge): number { | ||
| const timestamp = new Date(edge.lastSeenAt).getTime(); | ||
| if (Number.isNaN(timestamp)) return 0.05; | ||
| const daysSinceLastSeen = (Date.now() - timestamp) / (1000 * 60 * 60 * 24); | ||
| const decay = edge.strength * 0.05 * (daysSinceLastSeen / 7); | ||
| return Math.max(0.05, edge.strength - decay); | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| export function registerConceptGraphSearchFunction(sdk: ISdk, kv: StateKV): void { | ||
| sdk.registerFunction( | ||
| "mem::concept-graph-search", | ||
| async (data: { concepts: string[]; depth?: number; limit?: number }) => { | ||
| if (!data.concepts || data.concepts.length === 0) { | ||
| return { success: false, error: "concepts array is required" }; | ||
| } | ||
|
|
||
| const depth = data.depth ?? 2; | ||
| if (!Number.isInteger(depth) || depth < 1 || depth > MAX_BFS_DEPTH) { | ||
| return { | ||
| success: false, | ||
| error: "depth_out_of_range", | ||
| message: `BFS depth must be an integer between 1 and ${MAX_BFS_DEPTH}, got ${depth}`, | ||
| }; | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| const limit = Math.max(1, Math.min(data.limit ?? 20, 100)); | ||
| const allEdges = await kv.list<ConceptEdge>(KV.conceptEdges); | ||
|
|
||
| const adjacency = new Map<string, Array<{ concept: string; strength: number }>>(); | ||
| for (const edge of allEdges) { | ||
| const strength = decayedStrength(edge); | ||
| if (strength <= 0.05) continue; | ||
|
|
||
| if (!adjacency.has(edge.from)) adjacency.set(edge.from, []); | ||
| if (!adjacency.has(edge.to)) adjacency.set(edge.to, []); | ||
| adjacency.get(edge.from)!.push({ concept: edge.to, strength }); | ||
| adjacency.get(edge.to)!.push({ concept: edge.from, strength }); | ||
| } | ||
|
|
||
| const seedConcepts = data.concepts.map((c) => c.toLowerCase().trim()); | ||
| const visited = new Set<string>(); | ||
| const conceptScores = new Map<string, number>(); | ||
|
|
||
| let frontier = new Set<string>(); | ||
| for (const seed of seedConcepts) { | ||
| visited.add(seed); | ||
| conceptScores.set(seed, 1.0); | ||
| frontier.add(seed); | ||
| } | ||
|
|
||
| for (let d = 0; d < depth; d++) { | ||
| const nextFrontier = new Set<string>(); | ||
| for (const current of frontier) { | ||
| const neighbors = adjacency.get(current) || []; | ||
| const sorted = neighbors | ||
| .filter((n) => !visited.has(n.concept)) | ||
| .sort((a, b) => b.strength - a.strength) | ||
| .slice(0, MAX_NEIGHBORS_PER_NODE); | ||
|
|
||
| for (const neighbor of sorted) { | ||
| if (visited.has(neighbor.concept)) continue; | ||
| visited.add(neighbor.concept); | ||
|
|
||
| const parentScore = conceptScores.get(current) || 0; | ||
| conceptScores.set(neighbor.concept, parentScore * neighbor.strength); | ||
| nextFrontier.add(neighbor.concept); | ||
| } | ||
| } | ||
| frontier = nextFrontier; | ||
| } | ||
|
|
||
| const expandedConcepts = [...conceptScores.keys()]; | ||
|
|
||
| const allMemories = await kv.list<Memory>(KV.memories); | ||
| const results: Array<{ memoryId: string; score: number; matchedConcepts: string[] }> = []; | ||
|
|
||
| for (const memory of allMemories) { | ||
| if (memory.isLatest === false) continue; | ||
| const memoryConcepts = memory.concepts.map((c) => c.toLowerCase()); | ||
| const matched = memoryConcepts.filter((c) => expandedConcepts.includes(c)); | ||
| if (matched.length === 0) continue; | ||
|
|
||
| let score = 0; | ||
| for (const mc of matched) { | ||
| score += conceptScores.get(mc) || 0; | ||
| } | ||
| score = score / matched.length; | ||
|
|
||
| results.push({ | ||
| memoryId: memory.id, | ||
| score, | ||
| matchedConcepts: matched, | ||
| }); | ||
| } | ||
|
|
||
| results.sort((a, b) => b.score - a.score); | ||
|
|
||
| return { | ||
| success: true, | ||
| results: results.slice(0, limit), | ||
| expandedConcepts, | ||
| depth, | ||
| }; | ||
| }, | ||
| ); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MCP tool count inconsistent with README and plugin.json.
AGENTS.md shows 45 MCP tools, but README.md and plugin.json both show 52 MCP tools. Per the review stack context, Layer 1 should update documentation to reflect 52 tools.
📝 Suggested fix
Based on learnings: When adding or removing MCP tools, update all six locations: tools-registry.ts, server.ts, triggers/api.ts, index.ts, test/mcp-standalone.test.ts, README.md, and plugin/.claude-plugin/plugin.json.
📝 Committable suggestion
🤖 Prompt for AI Agents