feat(sdk): honor repo agency.toml MCP servers in worker sessions

Borrow the 1ES Agency CLI's agency.toml -> engine MCP resolution so a
PilotSwarm worker running inside a repo checkout (e.g. DsMainDev) surfaces
that repo's declared MCP servers to the Copilot SDK createSession({ mcpServers }).

- New packages/sdk/src/agency-config.ts: faithful subset port of agency's
  [mcps] resolution. Reads agency.toml [mcps.servers], plus (per its
  include_mcps_from_workspace / include_mcps_from_vscode flags) the repo's
  .mcp.json (default-on) and .vscode/mcp.json (opt-in, templated entries
  skipped), merged in agency's documented order (later wins). Maps each
  entry to MCPServerConfig via the same command|url disambiguation as
  agency's McpServer flat-format deserialize. Flags unsupported sections
  (plugins/marketplaces/remote_config/builtins/profiles). Never throws.
  Comments cite the upstream agency source files
  (https://dev.azure.com/1esgitops/_git/agency) since code may diverge.
- session-manager.ts: resolve agency MCP servers at the session working dir
  and layer them UNDER platform-provisioned servers (platform wins on name
  conflict, preserving real auth). Add agency.toml details to the GHCP-SDK
  createSession diagnostics log.
- Add smol-toml dependency for TOML parsing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 26f310eb-4680-4102-a8aa-f8d041940ab4

Author: andrewkcchung

package-lock.json

@@ -85,7 +85,8 @@
         "@opentelemetry/api": "^1.9.0",
         "duroxide": "^0.1.27",
         "file-type": "^20.5.0",
-        "pg": "^8.18.0"
+        "pg": "^8.18.0",
+        "smol-toml": "^1.7.1"
     },
     "peerDependencies": {
         "pilotswarm-horizon-store": "0.5.37"

packages/sdk/package.json

@@ -0,0 +1,314 @@
+// agency-config.ts
+// -----------------------------------------------------------------------------
+// Faithful (subset) port of how the 1ES **Agency** CLI resolves an `agency.toml`
+// file found in a target repo into engine (GitHub Copilot) session config. When
+// a PilotSwarm worker runs inside a repo checkout (e.g. DsMainDev), it should
+// honor that repo's `agency.toml` the same way the Agency pipeline task would —
+// primarily by surfacing the repo-declared MCP servers to the Copilot SDK
+// `createSession({ mcpServers })` call.
+//
+// This is intentionally a SUBSET. Agency does far more (remote_config layering,
+// profiles, plugin acquisition/marketplaces, builtin MCP machinery, auth
+// proxying). First pass covers the piece that is (a) most commonly present in
+// target repos and (b) maps cleanly onto the SDK: the `[mcps]` section.
+//
+// Source-of-truth references (code may diverge — re-check against upstream):
+//   Repo:   https://dev.azure.com/1esgitops/_git/agency  (local: C:\src\agency)
+//   Schema: docs/agency/CLI/agency-config-schema.md
+//   Struct: client/config/src/agency_config.rs
+//             - AgencyConfigCore / AgencyConfig (top-level, profiles)
+//             - McpsConfig (the `[mcps]` table; builtins/servers/include_* flags)
+//   MCP:    client/config/src/custom_agent/mcp.rs
+//             - McpServer enum (Local / SSE / HTTP) + its custom flat-format
+//               Deserialize impl (command|url disambiguation, `type` handling)
+//             - load_workspace_mcp_config (<root>/.mcp.json)
+//             - load_vscode_mcp_config    (<root>/.vscode/mcp.json)
+//   Merge:  client/agency/src/mcp_config.rs
+//             - merge_all_mcp_sources (source precedence order)
+//             - write_mcp_config_to_temp_file (final shape handed to the engine
+//               as `--additional-mcp-config @file`; SDK equivalent = mcpServers)
+// -----------------------------------------------------------------------------
+
+import fs from "node:fs";
+import path from "node:path";
+import { parse as parseToml } from "smol-toml";
+import type { MCPServerConfig } from "@github/copilot-sdk";
+
+/** Result of resolving a repo's agency.toml MCP configuration. */
+export interface AgencyMcpResolution {
+    /** True when an agency.toml was found and parsed at the root. */
+    found: boolean;
+    /** Absolute path to the agency.toml that was read, when found. */
+    configPath?: string;
+    /** Resolved MCP servers, keyed by server name (SDK createSession shape). */
+    mcpServers: Record<string, MCPServerConfig>;
+    /** For each resolved server name, which source it came from (for diagnostics). */
+    sources: Record<string, string>;
+    /** Non-fatal issues (skipped/invalid entries, both-command-and-url, etc.). */
+    warnings: string[];
+    /**
+     * agency.toml sections that were present but are NOT yet honored by this
+     * subset port (plugins, marketplaces, remote_config, builtins, profiles…).
+     * Surfaced so operators can see what was ignored.
+     */
+    unsupported: string[];
+}
+
+// Agency defaults (client/config/src/agency_config.rs McpsConfig):
+//   include_mcps_from_vscode    -> false (opt-in)
+//   include_mcps_from_workspace -> true  (opt-out)
+const DEFAULT_INCLUDE_MCPS_FROM_VSCODE = false;
+const DEFAULT_INCLUDE_MCPS_FROM_WORKSPACE = true;
+
+// Process-lifetime cache keyed by absolute root dir. A PilotSwarm worker is
+// bounded to one checkout, so the resolved config is stable for the process;
+// caching avoids re-reading/parsing on every per-turn createSession call.
+const _cache = new Map<string, AgencyMcpResolution>();
+
+function emptyResolution(): AgencyMcpResolution {
+    return { found: false, mcpServers: {}, sources: {}, warnings: [], unsupported: [] };
+}
+
+/**
+ * Resolve `<rootDir>/agency.toml` (and, per its flags, the repo's `.mcp.json` /
+ * `.vscode/mcp.json`) into a set of MCP servers for the Copilot SDK.
+ *
+ * Mirrors the repo-relevant slice of agency's `merge_all_mcp_sources`
+ * (client/agency/src/mcp_config.rs). Precedence (later wins) for the sources we
+ * cover: `.vscode/mcp.json` (opt-in) -> `.mcp.json` (default-on) ->
+ * `agency.toml [mcps.servers]`. Platform-provisioned PilotSwarm servers are
+ * layered on top by the caller and always win on name conflicts (they carry
+ * real auth) — see session-manager.ts.
+ *
+ * Never throws: parse/read failures degrade to warnings and an empty/partial
+ * result so a malformed repo file can never break session creation.
+ */
+export function resolveAgencyMcpServers(rootDir: string): AgencyMcpResolution {
+    const root = path.resolve(rootDir || ".");
+    const cached = _cache.get(root);
+    if (cached) return cached;
+
+    const result = emptyResolution();
+
+    // NOTE: Agency discovers agency.toml globally-first then cwd-last across
+    // parent dirs (list_agency_config_files in agency_config.rs). For the worker
+    // we only consider the checkout root — the single repo the session runs
+    // against — which is the layer that matters for repo-declared MCP servers.
+    const configPath = path.join(root, "agency.toml");
+    let rawConfig: string;
+    try {
+        if (!fs.existsSync(configPath)) {
+            _cache.set(root, result);
+            return result;
+        }
+        rawConfig = fs.readFileSync(configPath, "utf8");
+    } catch (err) {
+        result.warnings.push(`failed to read ${configPath}: ${errMsg(err)}`);
+        _cache.set(root, result);
+        return result;
+    }
+
+    let parsed: Record<string, unknown>;
+    try {
+        parsed = parseToml(rawConfig) as Record<string, unknown>;
+    } catch (err) {
+        // Agency rejects unknown fields / malformed TOML (deny_unknown_fields).
+        // We are more lenient: a parse failure yields no servers + a warning
+        // rather than aborting the session.
+        result.warnings.push(`failed to parse ${configPath} as TOML: ${errMsg(err)}`);
+        _cache.set(root, result);
+        return result;
+    }
+
+    result.found = true;
+    result.configPath = configPath;
+
+    // Flag top-level sections we do not (yet) honor, so nothing is silently lost.
+    // (agency_config.rs AgencyConfigCore top-level keys.)
+    for (const key of ["plugins", "marketplaces", "remote_config", "org_config", "profiles", "review", "ring", "telemetry"]) {
+        if (parsed[key] !== undefined) result.unsupported.push(key);
+    }
+
+    const mcps = (parsed.mcps ?? {}) as Record<string, unknown>;
+
+    // Agency also has [mcps.builtins] (BuiltinMcpEntry) — the `ado` builtin, etc.
+    // Those need agency-specific machinery (spawning the agency binary as an MCP
+    // proxy, auth). Not portable here; flag and skip.
+    if (mcps.builtins !== undefined && Object.keys(mcps.builtins as object).length > 0) {
+        result.unsupported.push("mcps.builtins");
+    }
+
+    const includeVscode = readBoolFlag(mcps.include_mcps_from_vscode, DEFAULT_INCLUDE_MCPS_FROM_VSCODE);
+    const includeWorkspace = readBoolFlag(mcps.include_mcps_from_workspace, DEFAULT_INCLUDE_MCPS_FROM_WORKSPACE);
+
+    // Apply sources in agency's documented order; later sources overwrite
+    // earlier ones on name collision (matches merge_all_mcp_sources sequencing:
+    // .vscode -> .mcp.json -> agency mcps.servers).
+    if (includeVscode) {
+        mergeExternalMcpFile(path.join(root, ".vscode", "mcp.json"), ".vscode/mcp.json", result, /*skipTemplated*/ true);
+    }
+    if (includeWorkspace) {
+        mergeExternalMcpFile(path.join(root, ".mcp.json"), ".mcp.json", result, /*skipTemplated*/ false);
+    }
+
+    const servers = (mcps.servers ?? {}) as Record<string, unknown>;
+    for (const [name, entry] of Object.entries(servers)) {
+        const mapped = mapMcpEntry(name, entry, "agency.toml [mcps.servers]", result);
+        if (mapped) {
+            result.mcpServers[name] = mapped;
+            result.sources[name] = "agency.toml [mcps.servers]";
+        }
+    }
+
+    _cache.set(root, result);
+    return result;
+}
+
+/**
+ * Read a Copilot/VS Code MCP JSON file (`{ "mcpServers"|"servers": {...} }`) and
+ * merge its entries. Mirrors custom_agent::load_workspace_mcp_config /
+ * load_vscode_mcp_config plus the lenient VS Code handling in
+ * client/agency/src/mcp_config.rs (load_vscode_mcp_lenient), which skips entries
+ * that reference un-expandable `${...}` template variables.
+ */
+function mergeExternalMcpFile(
+    filePath: string,
+    label: string,
+    result: AgencyMcpResolution,
+    skipTemplated: boolean,
+): void {
+    let text: string;
+    try {
+        if (!fs.existsSync(filePath)) return;
+        text = fs.readFileSync(filePath, "utf8");
+    } catch (err) {
+        result.warnings.push(`failed to read ${label}: ${errMsg(err)}`);
+        return;
+    }
+
+    let obj: Record<string, unknown>;
+    try {
+        obj = JSON.parse(stripJsonComments(text)) as Record<string, unknown>;
+    } catch (err) {
+        result.warnings.push(`failed to parse ${label} as JSON: ${errMsg(err)}`);
+        return;
+    }
+
+    // Copilot's `.mcp.json` uses "mcpServers"; VS Code's mcp.json uses "servers".
+    // Agency's McpConfig serde accepts both (rename/alias in custom_agent/mcp.rs).
+    const servers = (obj.mcpServers ?? obj.servers ?? {}) as Record<string, unknown>;
+    for (const [name, entry] of Object.entries(servers)) {
+        if (skipTemplated && containsTemplate(entry)) {
+            result.warnings.push(`${label}: skipped server '${name}' (references un-expandable \${...} template)`);
+            continue;
+        }
+        const mapped = mapMcpEntry(name, entry, label, result);
+        if (mapped) {
+            result.mcpServers[name] = mapped;
+            result.sources[name] = label;
+        }
+    }
+}
+
+/**
+ * Map one raw MCP entry to the SDK `MCPServerConfig`. Faithfully follows the
+ * command|url disambiguation in agency's McpServer flat-format Deserialize
+ * (client/config/src/custom_agent/mcp.rs):
+ *   - has `command`, no `url`            -> Local  (stdio)
+ *   - has `url`, type == "http"          -> HTTP
+ *   - has `url`, type == "sse" | (unset) -> SSE   (unset defaults to SSE upstream)
+ *   - both command and url               -> error (we warn + skip)
+ *   - neither                            -> error (we warn + skip)
+ */
+function mapMcpEntry(
+    name: string,
+    entry: unknown,
+    source: string,
+    result: AgencyMcpResolution,
+): MCPServerConfig | undefined {
+    if (!entry || typeof entry !== "object") {
+        result.warnings.push(`${source}: server '${name}' is not a table/object; skipped`);
+        return undefined;
+    }
+    const e = entry as Record<string, unknown>;
+    const command = typeof e.command === "string" ? e.command : undefined;
+    const url = typeof e.url === "string" ? e.url : undefined;
+    const type = typeof e.type === "string" ? (e.type as string).toLowerCase() : undefined;
+
+    if (command && url) {
+        result.warnings.push(`${source}: server '${name}' specifies both 'command' and 'url'; skipped`);
+        return undefined;
+    }
+
+    // Shared optional fields (agency CommonConfig: tools/timeout/env; local: env).
+    const tools = Array.isArray(e.tools) ? (e.tools as unknown[]).filter((t) => typeof t === "string") as string[] : undefined;
+    const timeout = typeof e.timeout === "number" ? e.timeout : undefined;
+
+    if (command) {
+        const args = Array.isArray(e.args) ? (e.args as unknown[]).map(String) : undefined;
+        const env = isStringMap(e.env) ? (e.env as Record<string, string>) : undefined;
+        const cwd = typeof e.cwd === "string" ? e.cwd : undefined; // agency LocalConfig.cwd
+        const cfg: MCPServerConfig = {
+            type: "stdio",
+            command,
+            ...(args && { args }),
+            ...(env && { env }),
+            ...(cwd && { workingDirectory: cwd }),
+            ...(tools && { tools }),
+            ...(timeout !== undefined && { timeout }),
+        };
+        return cfg;
+    }
+
+    if (url) {
+        // Unknown non-http/sse `type` is an error upstream; we warn + skip.
+        if (type && type !== "http" && type !== "sse") {
+            result.warnings.push(`${source}: server '${name}' has unknown type '${type}'; skipped`);
+            return undefined;
+        }
+        const headers = isStringMap(e.headers) ? (e.headers as Record<string, string>) : undefined;
+        const cfg: MCPServerConfig = {
+            type: type === "http" ? "http" : "sse", // unset defaults to SSE (agency)
+            url,
+            ...(headers && { headers }),
+            ...(tools && { tools }),
+            ...(timeout !== undefined && { timeout }),
+        };
+        return cfg;
+    }
+
+    result.warnings.push(`${source}: server '${name}' has neither 'command' nor 'url'; skipped`);
+    return undefined;
+}
+
+/** Deep scan for an un-expandable `${...}` template token in an entry. */
+function containsTemplate(entry: unknown): boolean {
+    try {
+        return /\$\{[^}]+\}/.test(JSON.stringify(entry));
+    } catch {
+        return false;
+    }
+}
+
+function readBoolFlag(v: unknown, dflt: boolean): boolean {
+    return typeof v === "boolean" ? v : dflt;
+}
+
+function isStringMap(v: unknown): v is Record<string, string> {
+    return !!v && typeof v === "object" && !Array.isArray(v)
+        && Object.values(v as object).every((x) => typeof x === "string");
+}
+
+/** Tolerate JSONC (// and /* *​/ comments) as VS Code allows in mcp.json. */
+function stripJsonComments(text: string): string {
+    // Remove block comments, then line comments. Conservative: does not attempt
+    // to preserve comment-like sequences inside strings, which is acceptable for
+    // MCP config files (they don't contain such literals in practice).
+    return text
+        .replace(/\/\*[\s\S]*?\*\//g, "")
+        .replace(/(^|[^:])\/\/.*$/gm, "$1");
+}
+
+function errMsg(err: unknown): string {
+    return err instanceof Error ? err.message : String(err);
+}

packages/sdk/src/agency-config.ts

@@ -25,6 +25,7 @@ import { buildKnowledgePromptBlocks, loadKnowledgeIndexFromFactStore, buildEnhan
 import { composeStructuredSystemMessage, extractPromptContent, mergePromptSections } from "./prompt-layering.js";
 import { buildPromptLayersEventPayload, type PromptLayerDescriptor } from "./prompt-layers.js";
 import { approvePermissionForSession } from "./permissions.js";
+import { resolveAgencyMcpServers } from "./agency-config.js";
 import { readSnapshotMarker, supportsVersionedSnapshots, writeSnapshotMarker } from "./snapshot-protocol.js";
 import fs from "node:fs";
 import path from "node:path";
@@ -1265,7 +1266,21 @@ export class SessionManager {
         const boundAgentMcpServers = effectiveSerializableConfig.boundAgentName
             ? this.workerDefaults.agentMcpServers?.[effectiveSerializableConfig.boundAgentName]
             : undefined;
+
+        // agency.toml (capability-profiles / repo-native): when the session runs
+        // against a repo checkout that ships an `agency.toml`, borrow the MCP
+        // servers it declares — the same thing the 1ES Agency pipeline task would
+        // surface to the engine. See agency-config.ts for the faithful mapping and
+        // the upstream source references (https://dev.azure.com/1esgitops/_git/agency).
+        // Root = the per-session workingDirectory when supplied, else the worker
+        // process cwd (the serve harness chdir's into the enlistment checkout).
+        // Precedence: repo-declared agency servers form the BASE; platform-
+        // provisioned servers (base + bound-agent) are layered on top and win on
+        // name conflicts because they carry real auth/proxying.
+        const agencyRoot = config.workingDirectory ?? process.cwd();
+        const agencyMcp = resolveAgencyMcpServers(agencyRoot);
         const effectiveMcpServers = {
+            ...agencyMcp.mcpServers,
             ...(this.workerDefaults.baseMcpServers ?? {}),
             ...(boundAgentMcpServers ?? {}),
         };
@@ -1373,6 +1388,14 @@ export class SessionManager {
                     skillDirectories: skillDirsProbe,
                     customAgentsCount: Array.isArray(sessionConfig.customAgents) ? sessionConfig.customAgents.length : 0,
                     mcpServerNames: Object.keys(effectiveMcpServers),
+                    agencyToml: {
+                        found: agencyMcp.found,
+                        configPath: agencyMcp.configPath ?? "(none)",
+                        serverNames: Object.keys(agencyMcp.mcpServers),
+                        sources: agencyMcp.sources,
+                        unsupportedSections: agencyMcp.unsupported,
+                        warnings: agencyMcp.warnings,
+                    },
                     githubProbeWorkingDir: probeGithub(effWorkingDir),
                     githubProbeCwd: probeGithub(process.cwd()),
                 }),