Agent Failure Archive: 186 real agent post-mortems, and a free tool that checks your claims against them

[HanbeenMoon]

Published to the registry as io.github.HanbeenMoon/agent-failure-archive. Remote, streamable-http, three tools work with no wallet and no configuration.

https://desktop-ai2ata5-1.tailfeb765.ts.net/mcp

What it is

An archive of 186 post-mortems from running a multi-session agent system in production for eight months. Not a curated best-practices list. These are the incidents that were logged as they happened, with the numbers measured at the time.

A few from the corpus:

• A repair routine existed and its runner ticked 26,662 times over 22.7 hours. It performed zero repairs, because a guard condition never matched. Fourteen sessions sat dead the whole time.
• A publishing watcher exited 0 for days after its login expired. Nothing was published, nothing alerted, because the process kept succeeding.
• A conversation store held 4,279 files that were 2,273 unique sessions plus 2,006 re-dumps under new names. Every count was inflated 2.3x and retrieval over-represented the oldest session by up to 57x.

107 of the 186 come from a longer-running attempt to measure one person's language against the general language, which mostly failed. Those are the measurement failures: null results with no positive control, dose-response arms where the same parameter turned out to be two different treatments, a detector whose output was identical whether the signal was present or not.

The tool that is actually useful day to day

precheck is free and needs no wallet. Give it a conclusion, or paste a paragraph of findings, and it returns which of nine documented self-deception patterns each claim trips.

text: "We ran the new ranker on 40 sampled queries and found no significant
       difference. The watcher is healthy: it exits 0 every run. Our corpus
       contains 4279 documents, 2x the previous release."

comes back with three separate holds: the null result has no positive control, the healthy watcher would look identical having done nothing, and the corpus count may be counting re-dumped snapshots. Sentence selection is deterministic and driven by the same check table, so prose with no claims in it returns empty rather than inventing findings. No model is consulted anywhere in that path.

sample and catalog are also free. The paid tools (audit, search, brief, research, archive) settle per call in USDC on Base over x402, from $0.01 to $1.00, with no account and no subscription. Nothing has sold yet, and that is stated in the repo rather than hidden.

Two things that cost me time publishing, in case they save you some

The registry caps description at 100 characters. A longer one fails the publish step with a 422 and the validation detail is only visible in the Actions log, after OIDC auth has already succeeded. Auth working and publish failing look similar from the outside.

x402[mcp] currently breaks on mcp 2.x. The extra declares mcp>=1.0.0 with no upper bound, so a clean install pulls 2.x, but the server-side payment wrapper does a runtime import of mcp.server.fastmcp, which 2.x removed. The import is lazy, so nothing fails until the first paid tool is declared, and the symptom in a client is Connection closed with no cause. Filed as x402-foundation/x402#3261; pin mcp<2 for the process that serves paid tools.

Honest limits

This is one operator's system. It is prior art, not a general sample of agent systems. Coverage skews toward multi-session coordination, hook and cron wiring, retrieval pipelines, and scheduled repair. Some entries record a fix that was later found to be wrong; those are kept with the correction, because the correction is usually the more useful half.

Repo: https://github.com/HanbeenMoon/agent-failure-archive