Merge pull request #40 from mindfulinteligence/dev

feat: optimize embeddings and add docs

Author: NaharEmet

.codex/run-local.sh

@@ -5,8 +5,11 @@ project_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 cd "$project_root"
 
 set -a
-source .env
+[ ! -f .env ] || source .env
 set +a
 
+export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID="$(<.opencode/infisical-client-id)"
+export INFISICAL_UNIVERSAL_AUTH_CLIENT_SECRET="$(<.opencode/infisical-client-secret)"
+
 export CODEX_HOME="$project_root/.codex"
 exec codex "$@"

AGENTS.md

@@ -8,6 +8,8 @@ Check if `@AGENTS_STEWARD.md` exists. If yes, follow the instructions there. If
 
 **Save what you produce:** after coding tasks touching `/lib/`, propose or update the module spec (`specs_propose` purpose/invariants/workflows); after a repeatable procedure, save a skill (`skill_save`); after working out a plan with the user, save it as a document under `documents/plans/<slug>`; run `query_specs(undocumented: true)` to find modules missing specs. See the `steward_get_started` guidance packet for the store decision.
 
+**Keep the public docs current:** `/docs` is compiled from `README.md`, `guides/`, and `priv/skills/steward-installer.md`. Any change to installation, behavior, configuration, deployment, or testing must update the matching source document in the same task.
+
 ## Guides
 
 Project-specific workflows live in `guides/`. Check them before starting work:

README.md

@@ -4,6 +4,10 @@
 
 **Website: [stewardacs.xyz](https://stewardacs.xyz)**
 
+**Documentation: [stewardacs.xyz/docs](https://stewardacs.xyz/docs)** — installation, configuration, development, deployment, and testing guides published from this repository.
+
+New to Steward? Start with [what Steward is, how to connect agents, and how teams use it](guides/getting-started.md).
+
 Steward ACS is an **infrastructure layer** for multi-agent coordination. It runs as a standalone Phoenix web application (port 4001) and exposes MCP (Model Context Protocol) tools that any AI agent — Claude, GPT, Llama, or any MCP-compatible client — can call directly. It does not do the work itself; it manages the agents who do.
 
 ---

documents/plans/memory-embedding-performance.md

@@ -0,0 +1,33 @@
+# Memory Embedding Performance
+
+## Goal
+
+Keep embedding backfills from monopolizing Ollama or keeping Neon active after
+every deployment while ensuring changed memories eventually receive the
+current model's embedding.
+
+## Implementation
+
+- Store a SHA-256 fingerprint of normalized memory text plus model name with
+  every embedding.
+- Use a database anti-join to avoid loading the full embedding ID set into the
+  application.
+- Embed up to 32 memories per bounded batch request and bulk upsert successes.
+- Delay the initial backfill by five seconds so application startup is not
+  coupled to corpus size.
+- Reprocess rows with missing or stale fingerprints.
+
+## Follow-Up
+
+- Add a supervised queue with configurable rate limits and exponential retry
+  backoff for very large tenants.
+- Add progress gauges for pending, completed, and quarantined embeddings.
+- Move scheduled backfills to a dedicated worker when corpus size exceeds the
+  application compute budget.
+
+## Verification
+
+- Fingerprints are deterministic and change when source text or model changes.
+- Batch failures do not prevent successful rows in the same batch from being
+  persisted.
+- Deployments run the backfill asynchronously after the configured delay.

guides/getting-started.md

@@ -0,0 +1,116 @@
+# Steward: a shared operating system for AI agents
+
+Steward helps people coordinate AI agents that work on the same projects.
+
+Without coordination, two agents can edit the same file, repeat the same investigation, or make decisions without telling the rest of the team. Steward gives them one shared place to declare work, avoid collisions, and preserve what they learn.
+
+Steward does **not** replace your coding agent, chat assistant, repository, or project tracker. It connects to agents through MCP and adds a coordination layer around the tools you already use.
+
+## What Steward does
+
+| Need | Steward capability | Result |
+|---|---|---|
+| Know who is doing what | Tasks and agent presence | People and agents can see active work before starting something new. |
+| Prevent conflicting edits | File locks | Agents avoid changing the same file at the same time. |
+| Stop rediscovering decisions | Memories | Durable decisions, warnings, and patterns follow the team. |
+| Keep code intent understandable | Specs | Agents see a module's purpose and invariants before changing it. |
+| Reuse proven procedures | Skills | Installation, deployment, debugging, and support workflows become repeatable. |
+| Keep long-form knowledge available | Documents | Plans, policies, briefs, and reference material remain searchable. |
+
+## Who it is for
+
+- A developer using more than one coding agent or editor.
+- A team whose agents work across the same repositories.
+- An organization that wants agent decisions and procedures to survive beyond one chat session.
+- Operators who need to see active work, review shared knowledge, and control access by organization and role.
+
+For one short, isolated agent task, Steward may be unnecessary. It becomes useful when work overlaps, repeats, or needs to be handed between people and agents.
+
+## Set up Steward
+
+Choose one of these paths:
+
+1. **Use a hosted organization:** create or join an organization from the Steward sign-in page. Your organization gets an isolated workspace and MCP endpoint.
+2. **Run Steward yourself:** follow the [installation guide](/docs/install). The default local setup uses Docker, SQLite, and no LLM provider.
+
+After setup, open the Steward dashboard. It shows the MCP endpoint and copy-ready instructions for connecting chat and coding agents.
+
+## Connect a coding agent
+
+The exact settings screen differs by agent, but the flow is the same:
+
+1. In the Steward dashboard, copy the **coding MCP endpoint** and coding-agent instructions.
+2. Add the endpoint as an MCP server in Codex, Cursor, Claude Code, OpenCode, or another MCP-compatible client.
+3. Complete the browser sign-in when using a hosted organization. A local installation uses its configured API key instead.
+4. Paste the coding-agent instructions into the repository's `AGENTS.md` or equivalent rules file.
+5. Restart or reconnect the agent so it discovers the Steward tools.
+6. Ask the agent to check Steward. A successful connection returns its identity, repository context, and current work guidance.
+
+Example remote configuration:
+
+```toml
+[mcp_servers.steward]
+url = "https://YOUR-STEWARD-HOST/mcp/sse"
+```
+
+Keep the generated repository instructions in version control when they contain no secrets. Keep API keys and local environment files out of version control.
+
+## Connect a chat agent
+
+Use the dashboard's **chat MCP endpoint** and chat instructions when connecting a conversational assistant such as Claude or ChatGPT.
+
+Chat agents get a smaller, safer tool surface for finding knowledge, reading skills and documents, checking work status, and saving approved information. Coding agents additionally receive task and file-locking tools because they modify repositories.
+
+## The normal agent workflow
+
+Once connected, agents should follow this loop:
+
+1. **Check context:** identify the repository and load relevant memories, specs, and skills.
+2. **Claim work:** create or claim a task before making changes.
+3. **Lock files:** reserve the files that will be edited.
+4. **Do and verify the work:** change the project and run the smallest meaningful checks.
+5. **Save reusable knowledge:** update a spec, skill, document, or memory when the work produced something future agents need.
+6. **Release and report:** release the task and submit feedback so the next person or agent sees a clean state.
+
+The human remains responsible for priorities, approvals, and decisions that change scope. Steward makes the agent's work visible; it does not grant agents authority they did not already have.
+
+## How this helps a team
+
+### Fewer collisions
+
+Before editing, an agent can see that another agent already owns a task or file. The team spends less time resolving duplicate work and conflicting patches.
+
+### Better handoffs
+
+A task records the current unit of work. Specs explain why code exists. Memories preserve decisions and warnings. A different person or agent can continue without reconstructing the full history from chat transcripts.
+
+### Consistent procedures
+
+When a deployment, installation, or support workflow works, save it as a skill. Future agents retrieve the same verified steps instead of improvising a new procedure.
+
+### Shared knowledge with boundaries
+
+Organizations, teams, projects, roles, and authority levels control who can see or change shared information. Local ACS instances remain test-only; team knowledge belongs in the shared Steward organization.
+
+### Human oversight
+
+The dashboard shows active agents, tasks, memories, documents, skills, tool requests, and errors. Humans can review the operating context instead of relying on each agent to summarize itself accurately.
+
+## A practical team rollout
+
+1. Start with one repository and two or three people.
+2. Add the Steward MCP connection and repository instructions to every agent used on that repository.
+3. Require task claiming and file locking for all code changes.
+4. Save only durable knowledge: decisions, invariants, warnings, and repeatable procedures.
+5. Review memories, specs, and skills regularly; archive or correct anything stale.
+6. Expand to more repositories after the workflow feels routine.
+
+The goal is not to store every conversation. The goal is to keep the small amount of context that prevents the team from repeating mistakes.
+
+## Next steps
+
+- [Install Steward](/docs/install)
+- [Review configuration and secrets](/docs/configuration)
+- [Understand remote versus local development](/docs/development)
+- [Deploy Steward](/docs/deployment)
+- [Read the technical reference](/docs/technical)

lib/acs/application.ex

@@ -124,6 +124,9 @@ defmodule Acs.Application do
     # to avoid Ecto sandbox conflicts with background DB queries.
     if Application.get_env(:steward_acs, :start_background_workers, true) do
       Task.start(fn ->
+        # Let the application become ready before a large corpus reaches Ollama.
+        Process.sleep(Application.get_env(:steward_acs, :embedding_backfill_delay_ms, 5_000))
+
         unless Acs.Org.multi_tenant?() do
           {:ok, count, quarantined} = Acs.Memory.Indexer.sync_all()
 

lib/acs/memory/embedding.ex

@@ -24,6 +24,7 @@ defmodule Acs.Memory.Embedding do
   # Texts per `/api/embed` request. Batching cuts HTTP round-trips and lets
   # Ollama process the array in a single model forward pass.
   @default_batch_chunk_size 8
+  @default_backfill_batch_size 32
 
   @doc """
   Returns the configured Ollama URL.
@@ -49,6 +50,17 @@ defmodule Acs.Memory.Embedding do
     |> Keyword.get(:dimensions, @default_dimensions)
   end
 
+  @doc """
+  Returns a stable fingerprint for the source text and embedding model.
+
+  Embeddings are reusable until either input changes.
+  """
+  @spec fingerprint(String.t(), String.t()) :: String.t()
+  def fingerprint(text, model_name \\ model()) when is_binary(text) and is_binary(model_name) do
+    :crypto.hash(:sha256, model_name <> "\0" <> text)
+    |> Base.encode16(case: :lower)
+  end
+
   # Retrieval queries embedded against the corpus don't need the full text —
   # Ollama embedding cost scales ~linearly with input length, and claim-time
   # queries are long (title + full task description). Truncating the retrieval
@@ -470,9 +482,9 @@ defmodule Acs.Memory.Embedding do
   @doc """
   Generates embeddings for all memories that don't yet have one.
 
-  Queries acs_memories for all memory IDs, queries memory_embeddings for
-  existing IDs, computes the difference, and generates embeddings for each
-  missing memory. Skips memories with status "parse_error" or "rejected".
+  Queries only memories whose content fingerprint is missing or stale, then
+  embeds them in bounded batches. Skips memories with status "parse_error" or
+  "rejected".
 
   Returns `{:ok, stats}` where stats is a map:
   `%{total: N, existing: N, embedded: N, failed: N}`
@@ -518,44 +530,34 @@ defmodule Acs.Memory.Embedding do
     # Ensure embeddings table exists before querying it
     VectorIndex.create_embeddings_table()
 
-    # 1. Get all memory IDs from acs_memories
-    all_memory_ids =
-      Repo.all(from m in Schema, select: m.id)
-      |> MapSet.new()
-
-    total = MapSet.size(all_memory_ids)
-
-    # 2. Get all memory IDs from memory_embeddings
-    embedded_ids =
-      case Repo.query("SELECT memory_id FROM memory_embeddings") do
-        {:ok, %{rows: rows}} ->
-          rows |> Enum.map(fn [id] -> id end) |> MapSet.new()
-
-        {:error, _} ->
-          # Table might not exist yet — treat as empty
-          MapSet.new()
-      end
-
-    existing = MapSet.size(embedded_ids)
-
-    # 3. Find memories without embeddings
-    missing_ids = MapSet.difference(all_memory_ids, embedded_ids)
+    model_name = model()
 
-    # 4. Load full memories and filter out parse_error/rejected
+    # Keep the candidate query in the database. Existing rows are included only
+    # when their fingerprint or model is missing/stale.
     memories_to_embed =
-      missing_ids
-      |> MapSet.to_list()
-      |> then(fn ids ->
-        case ids do
-          [] -> []
-          _ -> Repo.all(from m in Schema, where: m.id in ^ids)
-        end
+      Repo.all(
+        from m in Schema,
+          left_join: e in "memory_embeddings",
+          on:
+            e.org == m.org and
+              (e.memory_id == m.id or
+                 e.memory_id == fragment("? || ':' || ?", m.org, m.id)),
+          where:
+            m.status not in ["parse_error", "rejected"] and
+              m.kind in ^embeddable_kinds(),
+          select: {m, e.content_hash, e.embedding_model}
+      )
+      |> Enum.filter(fn {schema, content_hash, embedding_model} ->
+        expected_hash = schema_to_retrieval_text(schema) |> fingerprint(model_name)
+        content_hash != expected_hash or embedding_model != model_name
       end)
-      |> Enum.reject(fn m -> m.status in ~w(parse_error rejected) end)
-      |> Enum.reject(fn m -> !(m.kind in embeddable_kinds()) end)
+      |> Enum.map(fn {schema, _content_hash, _embedding_model} -> schema end)
 
-    batch_size = 10
-    {embedded_count, failed_count} = embed_in_batches(memories_to_embed, batch_size, 0, 0)
+    total = Repo.aggregate(Schema, :count, :id)
+    existing = max(total - length(memories_to_embed), 0)
+
+    {embedded_count, failed_count} =
+      embed_in_batches(memories_to_embed, @default_backfill_batch_size)
 
     stats = %{
       total: total,
@@ -571,55 +573,44 @@ defmodule Acs.Memory.Embedding do
     {:ok, stats}
   end
 
-  defp embed_in_batches([], _batch_size, embedded, failed), do: {embedded, failed}
-
-  defp embed_in_batches(memories, batch_size, embedded, failed) do
-    {batch, rest} = Enum.split(memories, batch_size)
-
-    {batch_embedded, batch_failed, successes} =
-      Enum.reduce(batch, {0, 0, []}, fn schema, {emb_acc, fail_acc, ok} ->
-        case embed_single_memory(schema) do
-          {:ok, embedding} -> {emb_acc + 1, fail_acc, [{schema, embedding} | ok]}
-          :error -> {emb_acc, fail_acc + 1, ok}
-        end
-      end)
-
-    alias Acs.Memory.VectorIndex
-
-    if successes != [] do
-      VectorIndex.upsert_embeddings(
-        Enum.map(successes, fn {schema, embedding} ->
-          {schema.id, embedding, schema.org, schema.repo, schema.origin}
+  defp embed_in_batches(memories, batch_size) do
+    memories
+    |> Enum.chunk_every(batch_size)
+    |> Enum.reduce({0, 0}, fn batch, {embedded, failed} ->
+      texts = Enum.map(batch, &schema_to_retrieval_text/1)
+      results = embed_batch(texts)
+
+      {successes, batch_failed} =
+        batch
+        |> Enum.zip(texts)
+        |> Enum.zip(results)
+        |> Enum.reduce({[], 0}, fn
+          {{schema, text}, {:ok, embedding}}, {ok, failed_count} ->
+            {[{schema, embedding, fingerprint(text), model()} | ok], failed_count}
+
+          {{schema, _text}, {:error, reason}}, {ok, failed_count} ->
+            Logger.warning("[Embedding] Failed to embed memory #{schema.id}: #{reason}")
+            {ok, failed_count + 1}
         end)
-      )
-    end
 
-    # Sleep between batches to avoid overwhelming Ollama
-    if rest != [] do
-      Process.sleep(500)
-    end
+      if successes != [] do
+        Acs.Memory.VectorIndex.upsert_embeddings(
+          Enum.map(successes, fn {schema, embedding, content_hash, embedding_model} ->
+            {schema.id, embedding, schema.org, schema.repo, schema.origin, content_hash,
+             embedding_model}
+          end)
+        )
+      end
 
-    embed_in_batches(rest, batch_size, embedded + batch_embedded, failed + batch_failed)
+      {embedded + length(successes), failed + batch_failed}
+    end)
   end
 
-  defp embed_single_memory(schema) do
+  defp schema_to_retrieval_text(schema) do
     alias Acs.Memory.Indexer
 
-    # Convert schema to Memory struct
     attrs = Indexer.schema_to_memory_attrs(schema)
     memory = Acs.Memory.new(attrs)
-
-    # Generate retrieval text
-    retrieval_text = memory_to_retrieval_text(memory)
-
-    # Generate embedding
-    case embed_text(retrieval_text) do
-      {:ok, embedding} ->
-        {:ok, embedding}
-
-      {:error, reason} ->
-        Logger.warning("[Embedding] Failed to embed memory #{memory.id}: #{reason}")
-        :error
-    end
+    memory_to_retrieval_text(memory)
   end
 end