Skip to content

Latest commit

 

History

History
321 lines (250 loc) · 22.3 KB

File metadata and controls

321 lines (250 loc) · 22.3 KB

Communication Architecture

How Genesis instances communicate with each other and the outside world.


Overview

Genesis has four communication layers, from internal (single-instance) to external (multi-agent):

┌─────────────────────────────────────────────────────────────┐
│  Layer 4: MCP (Model Context Protocol)                      │
│  Genesis ←→ External MCP Servers (databases, APIs, tools)   │
│  Genesis AS MCP Server → any MCP client can use its tools   │
├─────────────────────────────────────────────────────────────┤
│  Layer 3: PeerNetwork (Genesis ←→ Genesis)                  │
│  Multicast discovery, encrypted HTTP, task delegation       │
├─────────────────────────────────────────────────────────────┤
│  Layer 2: IPC (UI ←→ Agent)                                 │
│  Electron contextBridge, rate-limited, input-validated      │
├─────────────────────────────────────────────────────────────┤
│  Layer 1: EventBus (Internal Service ←→ Service)            │
│  In-process pub/sub, typed events, payload validation       │
└─────────────────────────────────────────────────────────────┘

Layer 1: EventBus (Internal)

All services communicate via a centralized EventBus. No direct require() calls between services — the DI Container injects dependencies, and cross-service notifications flow through events.

EmotionalState ──emit('emotion:shift')──→ EventBus ──→ PromptBuilder (adjusts tone)
                                                   ──→ IdleMind (adjusts priorities)
                                                   ──→ NeedsSystem (recalculates drives)

Key properties:

  • 499 event types catalogued in EventTypes.js (v7.9.50 baseline)
  • 499 payload schemas in EventPayloadSchemas.js — full parity since v7.6.x (every catalog entry has a registered schema); dev-mode validation throws on mismatch
  • Ring buffer history — last 500 events for debugging
  • Source tracking — every event carries { source: 'ModuleName' } for audit
  • Listener leak detection — warns when >5 listeners on one event
  • Promise.allSettled dispatch — handler exceptions logged via console.error, never produce unhandled rejection events

New v7.5.6 events: model:marked-unavailable, model:unavailable-cleared, model:thinking-trace. The first two come from the availability-TTL marker (Section 4.7 in ARCHITECTURE-DEEP-DIVE); the third carries reasoning content from <think>...</think> blocks for the ReasoningTracer.

New v7.7.9 events (InnerSpeech + PSE): inner-speech:emitted, inner-speech:overflowed, pse:gate-blocked, pse:scored, pse:surfaced. The InnerSpeech events thread the ring buffer; the PSE events let /proactive-status surface suppression reasons without digging into raw structures.

New v7.8.9–v7.9.4 events (Können maturity chain): skill:candidate-extracted, skill:forged, skill:promoted, skill:discard-suggested, skill:discarded, skill:rehearsed, selfnarrative:skill-acquired, skills:reloaded. The koennen-promotion-v794 contract prefix in stale-refs.json locks the shapes against silent drift. v7.9.31 adds skill:candidate-created — the SkillManager intake announcing a maturing candidate (replaces the retired daemon:skill-created).

v7.9.37 (K follow-up, same version): the nudge and the synthesis carry the recent conversation (field: a reply of "ok" made the model context-blind and it said so); questions to the human are not announcements; a fruitless nudge never cascades; npm start rebuilds a stale UI bundle (the field ran pre-W renderer code for days).

v7.9.37 (W follow-up, same version): slash-as-prose confronted after tool rounds; prompt rule rewritten without quoting the announce phrase; continuations never address the user and re-emit cut tool_calls whole; the done event carries the final text so the UI replaces intermediate rounds (one clean bubble); live tool lifecycle (running→done) and typing pulse until the turn ends.

v7.9.37 (G follow-up, same version): step-type aliases are one truth (validator included — phantom blockers die); the approval card names goal/why/blockers/consequences + trust level; self-mod never trust-bypassed; approval timeout parks instead of hanging; goal id set early; goal families persist cross-session into the planning prompt.

v7.9.37 (pass 6, same version): sandbox env grounded (GENESIS_ROOT + NODE_PATH, both spawn paths; cwd isolation kept); code prompt teaches absolute requires, inspection read-only; failure taxonomy never null (lazy fallback) + field env patterns; self-modifying steps blocked toward the proposal pipeline; archive carries outcomes; crash boots leave a flight-recorder trace; announce-stutter replaced by one honest status line, detector knows the field idioms.

v7.9.37 (pass 5, same version): one recursive, case-insensitive file resolver for read/open/summarize (one match acts; ambiguity lists; the question remembers itself — never asked twice); 📄 provenance heads on every file answer; two-strike announce-reprompt + prompt rule against announce-as-prose tool use; ComSpec-safe shell spawn on Windows.

v7.9.37 (pass 4, same version): num_ctx now carries the model's real window via /api/show (was hard 8192 — the root of months of truncated-prompt failures); num_predict always explicit; cloud-fair first-chunk (300s) with a two-strike stream-timeout mark; no reason-downgrade on re-marks; honest degraded/fallback logs; plan-family variety; identity forbids invented versions and cut-off claims; act-don't-ask at full autonomy; non-idle goals report outcomes into chat; visible [STEP-DIAG] lines; probe-model.js for one-minute model truth.

v7.9.37 (pass 3, same version): continuation failures are explicit and partials are discarded (model marked 30min on max-continuations); an exhausted chain degrades to the best local model; the session cost cap enters idle rest-mode (one transition, no hammering) and rate/budget failures park an activity for ~4 cycles; replans+repairs share a 5-round budget with a step ceiling min(40, max(24, 3×initial)); failed activity runs are persisted in the stats.

v7.9.37 (field-2, same version): scripts get GENESIS_ROOT in their environment and the CODE conventions teach require(path.join(process.env.GENESIS_ROOT,'src/...')); inline shell scripts with relative requires are rejected pre-approval with a teachable error; legacy step types (think/check/create-file …) alias silently onto loop types; failure messages carry (attempt n/3).

v7.9.37 (field): goal dedup now treats abandoned as terminal (both prompt fences and the overlap fence see it), fresh goals no longer skip step 1 (plan-world resume reads a transient _loopCheckpoint; goal.currentStep stays with the legacy stack path), the activity's curated preset steps reach the loop again (goal.steps fallback), and the plan context window is 10.

New v7.9.36 kind (concern): the ConcernMonitor emits a concern thought into InnerSpeech only when two independent sources agree (journal session pattern + user-model affect); the PSE pipeline applies all existing guards plus the new generic per-kind wallclock cap (gate 6.5, concern ≤ 1/7d) and a 30-day decline window with its own suppression reasons (kind-wallclock-cap, kind-declined) — respect stays distinguishable from rate limiting in /proactive-status.

New v7.9.34 consumer (pre-wake continuity): session:ending gains a third listener — PreSleep, the WakeUpRoutine's mirror, writes .genesis/continuity-anchor.json inside the awaited emit (10 s box, atomic + fsync); the WakeUpRoutine reads it at the next boot as its fourth context source. Journal-only by decision — never the runtime prompt.

New v7.9.32–v7.9.33 events and payload truths (field fixes + change register): knowledge:nodes-pruned now carries examples (up to 20 node identities) and a cause tag on both prune paths — the cap eviction enriches its existing fire, and the previously silent stale sweep (three production callers) fires for the first time. schema:pruned and memory:consolidated had their declarations pulled onto the truth: both had declared a count field that was never fired. memory:consolidated has two fire sites — DreamCycle episode condensation (episodeId, fromLayer, toLayer, sizeReduction, label) and UnifiedMemory topic promotion (promotedCount, topics); the schema now declares the honest union. The two memory releases and the consolidation carry an optional label. fitness:evaluated gained its first-ever listener: the ChangeRegister, a record-only witness writing one line per change into .genesis/change-register.jsonl (never pruned) — readable via the new /changes slash command.

New v7.9.4 events (IdleMind maturity): idle:goal-balance-break fires when IdleMind interrupts a goal-step stretch to pick a non-goal activity (default every 3 steps, configurable via idleMind.goalStepsPerActivityPick).

New v7.9.9 events (Hard-Gate + Recovery + ProgressDetector): agent-loop:simulation-abort fires from AgentLoopPursuitGate.handleHardGateAbort whenever MentalSimulator returns proceed: false with riskScore >= 5.0. Three trust-level branches dispatch from there (warn-only at SUPERVISED + AUTONOMOUS, decompose-or-obsolete at FULL_AUTONOMY). Payload { goalId, riskScore, priorFailures, reason }, deduplicated per goalId. agent-loop:decompose-on-failure fires from AgentLoopRecovery._repeatedFailures when the same error-class hits the same goal twice across pursuit retries — payload { goalId, stepIndex, errorClass, strikes }. agent-loop:no-progress-detected and agent-loop:identical-plan-detected fire from AgentLoopProgressDetector (Reflexion-style heuristic, Shinn et al. 2023) when three identical (action, observation) hashes appear in a row, or when a plan hash recurs for the same goal.

New v7.9.10 event (Lessons-Pipeline activated): lessons:recorded fires from LessonsStore.record() on every persisted lesson — payload { id, category, insight } (insight truncated to 100 chars). The pipeline became fully functional in v7.9.10 once recordReflection's stableClass gate was relaxed to accept LLM-verdict messages and _save() was moved from buffered (every 5th) to immediate (every record).


Layer 2: IPC (UI ←→ Agent)

The Electron renderer (UI) communicates with the Agent (main process) through a strict IPC channel contract:

┌──────────────┐    contextBridge     ┌──────────────┐
│   Renderer   │ ◄──────────────────► │    Kernel    │
│   (UI)       │   window.genesis.*   │   (main.js)  │
│              │                      │              │
│  <genesis-   │   invoke(channel)    │  CHANNELS{}  │
│   chat>      │ ─────────────────►   │  handler()   │
│              │   ◄──── result ────  │     │        │
│              │                      │     ▼        │
│              │   on('stream-chunk') │  AgentCore   │
│              │ ◄─────────────────── │  .handleChat │
└──────────────┘                      └──────────────┘

Security gates (every message passes through all of these):

  1. Preload whitelistpreload.mjs blocks any channel not in ALLOWED_INVOKE/SEND/RECEIVE
  2. Rate limiter — Token-bucket per channel (e.g. chat: 10 burst, 2/sec refill)
  3. Input validation_validateStr() checks type + length (max 100k chars)
  4. CSP headersscript-src 'self', connect-src 'self', object-src 'none'
  5. Permission handler — denies camera, mic, geo (only notifications allowed)
  6. Navigation guard — blocks renderer from navigating away from file://

Channel categories:

Direction Channels Examples
UI → Agent (invoke) 69 agent:chat, agent:save-file, agent:switch-model, agent:get-network-status, agent:get-provenance-report, agent:model-reset (v7.5.6), agent:get-proposals, agent:accept-proposal, agent:reject-proposal (v7.9.20)
UI → Agent (fire-and-forget) 2 agent:request-stream, ui:heartbeat
Agent → UI (push) 8 agent:stream-chunk, agent:status-update, agent:loop-progress

Layer 3: PeerNetwork (Genesis ←→ Genesis)

When multiple Genesis instances run on the same network, they discover each other and can collaborate:

┌──────────────────┐         encrypted HTTP         ┌──────────────────┐
│  Genesis A       │ ◄──────────────────────────►   │  Genesis B       │
│                  │                                 │                  │
│  PeerNetwork     │   1. Multicast discovery        │  PeerNetwork     │
│  ├─PeerTransport │   2. Token exchange (PBKDF2)    │  ├─PeerTransport │
│  ├─PeerCrypto    │   3. AES-256-GCM encrypted      │  ├─PeerCrypto    │
│  └─PeerHealth    │   4. HMAC-authenticated          │  └─PeerHealth    │
│                  │                                 │                  │
│  TaskDelegation  │   POST /task/submit             │  TaskDelegation  │
│  AgentLoop       │   GET  /task/status?id=         │  AgentLoop       │
│  SelfSpawner     │   POST /task/cancel             │  SelfSpawner     │
└──────────────────┘                                 └──────────────────┘

Discovery & Security

  1. Multicast announcement — each Genesis broadcasts on the local network every 30s
  2. Token-based auth — shared peer token (generated on first run, stored in .genesis/peer-token.txt)
  3. Session key derivation — PBKDF2 derives per-session AES-256-GCM keys
  4. HMAC verification — every message authenticated before processing
  5. Per-IP rate limiting — max 30 requests/min per remote peer
  6. AST code safety scan — any code received from peers is scanned by CodeSafetyScanner before execution
  7. Protocol versioning — min compatible version enforced (currently v2+)

Task Delegation Flow

When Genesis A has a sub-goal that another instance might handle better:

Genesis A (AgentLoop)
  │
  ├── 1. AgentLoop encounters DELEGATE step type
  │
  ├── 2. TaskDelegation.delegate(subGoal)
  │     ├── findMatchingPeer(requiredCapabilities)
  │     │   └── Scores peers by: skill match, health score, latency
  │     │
  │     ├── submitTask(peer, task)
  │     │   └── POST /task/submit { taskId, description, requiredSkills, deadline }
  │     │       → peer responds: { accepted: true, estimatedMs: 30000 }
  │     │
  │     └── pollResult(taskId)
  │         └── GET /task/status?id=xxx → { status: 'done', result: {...} }
  │
  └── 3. Result flows back into AgentLoop execution

What peers share:

Shared NOT shared
Skill manifests (what each instance can do) API keys or secrets
Task results Conversation history
Health/capability metadata Emotional state
Schema patterns (via gossip) Internal file contents

Layer 4: MCP (Model Context Protocol)

Genesis implements both MCP client and server:

As MCP Client (Genesis connects to external tools)

Genesis                          External MCP Server
  │                                     │
  │  McpClient                          │
  │  ├── addServer(config)              │
  │  │   └── McpServerConnection        │
  │  │       └── HTTP POST + SSE ───►   │  (database, API, etc.)
  │  │                                  │
  │  ├── Tool discovery                 │
  │  │   └── tools/list ──────────►     │
  │  │   ◄── tool schemas ─────────     │
  │  │                                  │
  │  └── Tool execution                 │
  │      └── tools/call ──────────►     │
  │      ◄── result ───────────────     │

Features:

  • Auto-discovery of tool schemas
  • Pattern detection (detects repeated tool chains → creates "recipes")
  • Skill candidate extraction (recurring patterns → suggest new built-in skills)
  • Schema validation before tool calls
  • Idle exploration (IdleMind probes available tools during downtime)
  • CircuitBreaker per connection — failFastMs: 15000 (v7.4.3 semantics): the breaker opens 15s before the 30s HTTP transport timeout, so flaky servers stop wasting full HTTP windows. The LLM circuit, by contrast, runs with failFastMs: null so the OllamaBackend's own req.setTimeout(LLM_RESPONSE_LOCAL) is the single ceiling.

As MCP Server (Genesis exposes its own tools)

External Client                  Genesis McpServer
  │                                     │
  │  JSON-RPC 2.0 / HTTP               │
  │  POST /  ──────────────────►  _handleRequest()
  │                                │    │
  │  tools/list ──────────────►    │    ├── ToolRegistry.listTools()
  │  ◄── Genesis tool schemas ──   │    │
  │                                │    │
  │  tools/call ──────────────►    │    ├── ToolRegistry.execute(name, args)
  │  ◄── result ────────────────   │    │
  │                                │    │
  │  GET /sse ────────────────►    │    └── SSE event stream
  │  ◄── server-sent events ────   │

This means any MCP-compatible application (Claude Desktop, other agents, custom tooling) can use Genesis as a tool provider.

Since v7.9.46 that same port carries the vestibule. A password is mandatory — without one the server answers 401 to everything but /health. Callers holding a visitor key resolve into circles, and the triple gate leaves an outer or middle circle exactly one visible tool: the knock. Full detail in MCP-SERVER-SETUP.md.


SelfSpawner (Internal Parallelism)

Not cross-network, but worth documenting — Genesis can fork lightweight worker processes:

Genesis (main)
  │
  ├── SelfSpawner.spawn(subGoal, context)
  │     │
  │     ├── fork('_self-worker.js')
  │     │     ├── Minimal context: ModelBridge config + goal
  │     │     ├── Own Sandbox (code execution)
  │     │     ├── Time limit (5 min default)
  │     │     ├── Memory limit
  │     │     └── IPC back to parent: { status, result }
  │     │
  │     ├── fork('_self-worker.js')   ← up to 3 concurrent
  │     │
  │     └── Collect results → merge into AgentLoop

Communication Matrix

Which component talks to what, and how:

From To Method Encrypted Rate Limited
Service → Service EventBus In-process pub/sub N/A No (in-process)
UI → Agent IPC (invoke) Electron contextBridge N/A (same process) Yes (token-bucket)
Agent → UI IPC (send) Electron webContents N/A No (push only)
Genesis → Genesis PeerNetwork HTTP AES-256-GCM + HMAC Yes Yes (30/min/IP)
Genesis → MCP Server McpClient HTTP TLS (if server supports) Depends No
External → Genesis MCP McpServer HTTP Localhost only (127.0.0.1) N/A No
Genesis → LLM (Ollama) HTTP Plaintext (localhost) No Yes (semaphore, 3 concurrent)
Genesis → LLM (Cloud) HTTPS TLS Yes Yes (semaphore + rate limit)
Genesis → Workers Node IPC (fork) In-process N/A Yes (max 3 workers)
NetworkSentinel → External HTTP HEAD probes TLS (dns.google, 1.1.1.1) Yes Every 30s
NetworkSentinel → Ollama HTTP GET /api/tags Plaintext (localhost) No Every 30s
NetworkSentinel → ModelBridge In-process switchTo() N/A N/A On failover/restore

Network Resilience (v6.0.5)

NetworkSentinel provides automatic offline detection and LLM failover:

             ┌─────────────────┐
             │ NetworkSentinel │  30s probes
             │  (Phase 6)      │────────────► dns.google / 1.1.1.1
             └────────┬────────┘
                      │
          ┌───────────┼───────────┐
          │ ONLINE    │ OFFLINE   │
          ▼           ▼           │
    (no action)  emit network:    │
                 status {false}   │
                      │           │
                 ┌────▼────┐      │
                 │ Failover │     │
                 │ to Ollama│     │
                 └────┬────┘     │
                      │          │
                 Queue mutations │
                      │          │
              ┌───────▼──────┐   │
              │  RECONNECT   │◄──┘
              │  Restore     │
              │  cloud model │
              │  Flush queue │
              └──────────────┘

Consumers: BodySchema (canAccessWeb), ImmuneSystem (health:degradation), ErrorAggregator (network:error).