AI reliability engineering for production agents and LLM applications.
AI systems can pass conventional evaluations and still fail in real use. Instructions get dropped. Tools appear to succeed when they did not. Retrieved context changes meaning. Memory drifts. Policies weaken across long workflows. Evidence no longer explains why an action happened.
Hermes Labs finds these silent failures and engineers them out in the layer where they occur: prompts, tool interfaces, retrieval, memory, policies, runtime controls, and evidence.
Hermes Labs is an AI reliability engineering studio for product and engineering teams shipping agents and LLM applications whose behavior must remain dependable, inspectable, and reconstructable under real use.
- Diagnose. We review prompts, tools, scaffolds, configurations, memory, retrieval, and traces; run controlled adversarial probes; and deliver prioritized findings with reproducible failure cases and concrete fixes.
- Harden. We design and integrate runtime controls, anti-fabrication safeguards, context-integrity protections, policy gates, and offline-verifiable evidence inside the stack you already operate.
Bring us a system and a symptom →
In agent systems, language is part of the runtime.
System prompts, tool descriptions, retrieved context, memory, summaries, policies, and evaluation criteria do not merely describe a system. They condition what it notices, chooses, remembers, and does.
We treat this operational layer as an engineering surface: something that can be inspected before deployment, tested under adversarial conditions, controlled at runtime, and verified after an action occurs.
That is where systems can remain technically healthy while silently doing the wrong thing.
Start here. The active public core is nine repositories, each making one part of our approach inspectable and useful on its own. Hermes Labs engagements apply, integrate, and harden these methods in production systems. Per-tool evidence boundaries are documented on the open-source catalog.
| Tool | Reliability role | Start |
|---|---|---|
| lintlang 0.4.1 | Static analysis for agent configurations, tool descriptions, and system prompts. Zero LLM calls. Used in Character AI's Larch CI. | pip install lintlang==0.4.1 |
| hermeneutic 0.1.8 | Mine correction triples from chat logs and gate the next response before the same drift ships twice. | pip install hermeneutic==0.1.8 |
| fidelis 0.0.93 | Local-first agent memory that returns your original passages verbatim, with no LLM call in the default retrieval path. | pip install fidelis-memory==0.0.93 — repo, import name, and CLI stay fidelis; the unrelated PyPI package named fidelis is not ours |
| little-canary 0.3.3 | Input-side prompt-injection detection through sacrificial canary-model probes. | pip install little-canary==0.3.3 |
| hermes-rubric 1.1.0 | Evidence-first structured scoring for AI artifacts, with every dimension tied to quoted evidence. | pip install hermes-rubric==1.1.0 |
| hermes-blind 0.1.4 | Recover the original goal of a long Claude Code or Codex session from its own logs, as a compact reorientation anchor. | pip install hermes-blind==0.1.4 |
| agent-kickstart 0.2.0 | A guided, project-local first project for Claude Code beginners. No prior coding or terminal experience required. | pip install agent-kickstart==0.2.0 |
| zer0dex | Dual-layer local memory for agents: a readable markdown index alongside vector retrieval. Reference implementation. | Read the repository before adoption |
| agent-gorgon 0.1.8 | Deterministic runtime policy decisions for autonomous agents, with forensic evidence for later review. | pip install agent-gorgon==0.1.8 |
Our public research examines distinct reliability problems from empirical, measurement, and conceptual perspectives. Each paper addresses a different question and should be evaluated on its own evidence.
- Tool Differentia: Relational Static Analysis for AI Agent Tool Descriptions. A bounded deterministic analysis of the distinguishing information neighboring AI-agent tool descriptions do or do not provide; it documents LintLang H1.6 and does not establish semantic distinguishability or runtime-selection correctness.
- Behavioral Canarying for Prompt Injection: Powerless Model Probes with Explicit Coverage Semantics. A technical note on pre-execution prompt-injection sensing that separates routing disposition from evidence that inspection ran; it does not claim universal detection, formal security, or aggregate accuracy for the current release.
- The Generative Horizon. A conceptual paper on model self-report, recursive interpretive conditioning, and the boundary between measured representations and claims about inner states.
- Precise Records, Unstable Meanings. A measurement-validity audit of claims derived from AI agent telemetry.
- A Taxonomy of Epistemic Failure Modes in Large Language Models. A taxonomy of seven structural epistemic failure modes.
- The Asymmetric Burden of Proof. A matched-vignette study of null-result asymmetry.
Read the research index for abstracts, hosted copies, and citation exports.
Machine-readable publication record: JSON · JSON-LD · BibTeX.
Hermes Labs has contributed 32 merged changes across external open-source projects. Two directly removed runtime reliability failures in major agent frameworks:
| Project | Contribution | Reliability effect |
|---|---|---|
| LangChain | #35544 | Stop forcing tool_choice when extended thinking is enabled |
| Microsoft Semantic Kernel | #13610 | Prevent the truncation reducer from silently deleting system prompts |
Additional merged contributions span PyTorch Ignite, Optuna, React Router, Nuxt, Cloudflare Workers, Sentry, Meta jscodeshift, MobX, ngrx, Microsoft TSDoc and Griffel, GraphQL ESLint, and more.
We keep additional reference implementations and research concepts public so others can inspect, test, fork, and develop them.
Public does not automatically mean flagship or production-ready. Browse the full GitHub catalog and each repository's own status, installation instructions, and limitations.
Hermes Labs was founded by Roli Bosch (Rolando Bosch in professional and academic work). roli@hermes-labs.ai · hermes-labs.ai