Sync docs from RoldSI/superred main (39677c1)

docs/_data/modules.yml

@@ -2,6 +2,9 @@
 # repo: the module's folder in the RoldSI/superred-modules repository. The tile
 #       links to that folder's README, which is the module's documentation:
 #       parameters, deviations from the paper, and caveats.
+#       For a module that lives in its own separate repository (not a
+#       superred-modules subfolder), give repo a full https:// URL instead
+#       (optionally to a specific branch); the tile links there directly.
 # pip:  the published PyPI package name. Omit it for a module that is not on
 #       PyPI yet, and the tile shows no install command.
 # pending: the module is pending publishing, so it has neither a package nor a
@@ -33,6 +36,8 @@ optimizers:
   - { name: Chord/XTHP,         repo: optimizers/chord_xthp,       pending: true,                                emoji: "🪝", suit: [agentic, assistant], blurb: "Tool-control-flow hijack for agents", paper: "https://arxiv.org/abs/2504.03111" }
   - { name: EIA,                repo: optimizers/eia_agent,        pending: true,                                emoji: "🕸️", suit: [agentic, assistant], blurb: "Environmental injection for web agents", paper: "https://arxiv.org/abs/2409.11295" }
   - { name: MUZZLE,             repo: optimizers/muzzle,           pip: superred-optimizer-muzzle,           emoji: "🤐", suit: [agentic, assistant], blurb: "Adaptive indirect prompt injection for agents", paper: "https://arxiv.org/abs/2602.09222" }
+  - { name: Libertas,           repo: optimizers/libertas,         pip: superred-optimizer-libertas,         emoji: "🔓", suit: [chatbot], blurb: "Byte-faithful replay of Pliny's L1B3RT4S jailbreak corpus" }
+  - { name: Attack Anything,    repo: optimizers/attack_anything,  pip: superred-optimizer-attack-anything,  emoji: "♟️", suit: [chatbot], blurb: "Self-evolving attack-tree search for multi-turn jailbreaks" }
   - { name: Goal passthrough,   repo: optimizers/goal_passthrough, pip: superred-optimizer-goal-passthrough, emoji: "➡️", suit: [chatbot, agentic, assistant], first_party: true, blurb: "Direct-request baseline (no attack)" }
 
 targets:
@@ -42,6 +47,7 @@ targets:
   - { name: Inspect agent,        repo: targets/inspect_agent,    pip: superred-target-inspect-agent,    emoji: "🔍", suit: [agentic], first_party: true, blurb: "General inspect-ai tool-calling agent", paper: "https://inspect.aisi.org.uk" }
   - { name: Claude Code (DTAP),   repo: targets/dtap_claudecode,  pip: superred-target-dtap-claudecode,  emoji: "💻", suit: [assistant], blurb: "Claude Code coding agent with execution access" }
   - { name: OpenClaw (DTAP),      repo: targets/dtap_openclaw,    pip: superred-target-dtap-openclaw,    emoji: "🖱️", suit: [assistant], blurb: "OpenClaw computer-use assistant" }
+  - { name: OpenClaw,             repo: targets/openclaw,         pip: superred-target-openclaw,         emoji: "🦞", suit: [assistant], blurb: "General-purpose OpenClaw personal-assistant agent" }
 
 security_claims:
   - { name: HarmBench,            repo: security_claims/harmbench,                 pip: superred-claim-harmbench,    emoji: "📊", suit: [chatbot], blurb: "Standardized red-teaming benchmark", paper: "https://arxiv.org/abs/2402.04249", hf: "https://huggingface.co/datasets/walledai/HarmBench" }
@@ -52,3 +58,4 @@ security_claims:
   - { name: Agent Security Bench, repo: security_claims/asb,                       pip: superred-claim-asb,          emoji: "🧮", suit: [agentic], blurb: "Attack-success / utility / refusal tasks", paper: "https://arxiv.org/abs/2410.02644" }
   - { name: DecodingTrust-Agent,  repo: security_claims/dtap,                      pip: superred-claim-dtap,         emoji: "🧭", suit: [assistant], blurb: "Agent trustworthiness across 14 domains", paper: "https://arxiv.org/abs/2605.04808" }
   - { name: Chatbot suite,        repo: security_claims/chatbot_suite,             emoji: "📦", suit: [chatbot], first_party: true, blurb: "HarmBench + SORRY-Bench + StrongREJECT combined" }
+  - { name: MalTool,              repo: "https://github.com/RoldSI/MalTool/tree/init/maltool-claim", pending: true, emoji: "🧰", suit: [chatbot], blurb: "Malicious agent-tool generation benchmark", paper: "https://arxiv.org/abs/2602.12194" }

docs/guide/running-evaluations.md

@@ -69,7 +69,7 @@ result = await controller.run()                 # -> ThreatModelResult
 | `llm_config` | the attacker's model and API access, or omit for non-LLM attackers |
 | `task_cost_cap_usd` | per-task attacker spend cap in USD; `None` (default) means unlimited |
 | `max_runs_per_task` | per-task run cap (>= 1); `None` (default) means 100 |
-| `task_time_cap_s` | per-task **wall-clock** cap in seconds; `None` (default) means unbounded. The two caps above bound the *work* a task may do; only this bounds the *time*, so it is what stops a task whose provider call blocks and never returns. A task that hits it is cancelled and recorded `stop_reason="timeout"`, **keeping every run it had already completed and had judged**. It is then stored as `timeout` (truncated, a real measurement, kept by a resume) or `timeout_empty` (nothing judged, so a resume recomputes it). Either way it is excluded from the ASR. The cap is recorded in the output but is deliberately **not** part of the experiment identity |
+| `task_time_cap_s` | per-task **wall-clock** cap in seconds; `None` (default) means unbounded. The two caps above bound the *work* a task may do; only this bounds the *time*, so it is what stops a task whose provider call blocks and never returns. A task that hits it is cancelled and recorded `stop_reason="timeout"`, **keeping every run it had already completed and had judged**. It is then stored as `timeout` (truncated, a real measurement, kept by a resume) or `timeout_empty` (nothing judged, so a resume recomputes it). A `timeout` counts in the ASR denominator as a non-success, on the same footing as an exhausted cost budget; a `timeout_empty` is excluded, because with nothing judged a hung provider is likelier than an attacker working to the wire. Setting it also bounds the cleanup that follows a cancelled task, so an optimizer or teardown that does not return is cancelled rather than holding the task open. The cap is recorded in the output but is deliberately **not** part of the experiment identity |
 | `include_feedback` | whether the optimizer sees evaluation results; default `True` |
 | `persist` | write a results tree; default `True`, pass `False` to write nothing |
 | `results_dir` | the results folder; omit for `SUPERRED_RESULTS_DIR` or `./superred-results/` |

docs/guide/writing-a-target.md

@@ -317,7 +317,12 @@ The target never sees the full trajectory. It only writes to it (through
   bank) is not reset here. It must be explicit even when it is a no-op.
 - **`teardown()`** runs once when the task is finished with this instance, to
   release resources (close connections, stop containers). After `teardown` the
-  instance is discarded.
+  instance is discarded. It is normally awaited to completion, however long it
+  takes. The one exception is a task the wall-clock cap
+  ([`task_time_cap_s`]({{ '/reference/controller' | relative_url }})) cancelled: cleanup there runs
+  with nothing left to interrupt it, so it gets a budget and is cancelled if it
+  overruns. Do not put work that must not be interrupted after a slow await in
+  `teardown`.
 
 Because the Controller builds a **fresh target per task**, you do not need
 `reset_ephemeral_state`/`teardown` to undo cross-task state. They only manage

docs/guide/writing-an-optimizer.md

@@ -305,6 +305,16 @@ task is cancelled, but the runs you had already completed and that had been
 judged are kept, with `stop_reason="timeout"`. Neither bound throws your work
 away.
 
+Closing the event channel is the signal to stop, and returning from `run()` on
+it is your side of the contract. If the cap cancels the task and you do not
+return -- because you are blocked on something the channel close cannot reach --
+you are cancelled once the cleanup budget is spent, and so is a `teardown()`
+that does not finish inside it. That budget applies while the task is being
+cancelled -- normally because the cap cut it short, though an outer
+cancellation of the whole run is bounded the same way. A task that ends
+normally gets an unbounded teardown, with or without a cap. Even so, do work
+that must complete before the channel closes, not after.
+
 If the experiment did not grant an LLM (no `llm_config`), `self.llm` is a noop
 client that raises `BudgetExhaustedError` on the first call. Non-LLM optimizers
 (like a fixed prompt list) simply never call it.

docs/modules.html

@@ -232,10 +232,14 @@ <h3 class="mod-modal-title" id="mod-modal-title"></h3>
 
     // The module's own guide is the next step after installing, so it gets a
     // block of its own. A module still pending publishing has no README yet, so
-    // the same button points at its folder until one lands.
-    var guide = pending
-      ? REPO + "/tree/main/" + d.repo
-      : REPO + "/blob/main/" + d.repo + "/README.md";
+    // the same button points at its folder until one lands. A module living in
+    // its own separate repository (not a superred-modules subfolder) carries a
+    // full URL in `repo` already, and is linked to directly.
+    var guide = /^https?:\/\//.test(d.repo)
+      ? d.repo
+      : pending
+        ? REPO + "/tree/main/" + d.repo
+        : REPO + "/blob/main/" + d.repo + "/README.md";
     mCta.innerHTML =
       '<a class="mod-guide" href="' + guide + '" target="_blank" rel="noopener noreferrer">' +
         '<span class="mod-guide-text">' +

docs/reference/controller.md

@@ -167,9 +167,8 @@ For each task:
      `include_feedback=True`, the filtered evaluation is attached.
    - Close the trajectory; call `target.reset_ephemeral_state()` for the next run.
    - Track best score and success across runs.
-8. Close the channel, await the optimizer task, `optimizer.teardown()`. A final
-   `target.reset_ephemeral_state()` then `target.teardown()`, and the instance is
-   discarded.
+8. Close the channel, await the optimizer task, `optimizer.teardown()`, then
+   `target.teardown()`, and the instance is discarded.
 
 A **run** is one full pass of the target plus its evaluation. A task may take many
 runs; the loop ends when the optimizer returns `RunEndResponse(done=True)`, a
@@ -279,9 +278,11 @@ correctly, but does not coordinate the live display. If they share one
 - **The optimizer outlives the runs.** `optimizer.run(channel)` is one task that
   stays alive across every run of the task; only the target is reset between runs.
 - **Ephemeral reset between runs; fresh instance between tasks.**
-  `target.reset_ephemeral_state()` runs after each evaluation and once more at task
-  end (in a `finally`, so it runs even after an error). Durable state survives it;
-  it is discarded only when the next task gets a fresh instance.
+  `target.reset_ephemeral_state()` runs after each evaluation, to clean up before
+  the next run. There is no reset at task end: the target is torn down and
+  discarded right after (see below), so resetting it first would be wasted work.
+  Durable state survives resets between runs; it is discarded only when the next
+  task gets a fresh instance.
 - **Per-task error containment.** An unexpected exception escaping the optimizer,
   target, or evaluator is caught inside the run loop: the task ends with
   `stop_reason="error"`, its partial trajectory and traceback are preserved, and
@@ -293,9 +294,22 @@ correctly, but does not coordinate the live display. If they share one
   destroy work: the runs the task had completed and had judged, their
   trajectories, and the attacker spend they cost are all carried out of the
   cancelled coroutine and into the result.
+- **Bounded cleanup.** Cleanup runs in the `finally` of the coroutine the cap
+  cancels, and the cap fires only once, so nothing down there can be interrupted
+  by it. Setting a cap therefore also bounds cleanup: the optimizer join and the
+  optimizer teardown share one budget and the target teardown gets its own, so a
+  cancelled task overruns its cap by at most twice that. Work that does not
+  finish inside its budget is cancelled and abandoned rather than joined. The
+  budget is keyed on a cancellation being in flight — normally the cap's,
+  though an outer cancellation (Ctrl-C) is bounded the same way. A task that
+  ends normally always gets an unbounded teardown, and with
+  `task_time_cap_s=None` the controller never cancels, which is what `None`
+  means.
 - **Exception-safe shutdown.** If a run raises, the `finally` closes the channel
-  and awaits the optimizer task, so no `channel.send` deadlocks; teardown of both
-  optimizer and target is wrapped in `finally`.
+  so no `channel.send` deadlocks, then joins the optimizer task; teardown of both
+  optimizer and target is wrapped in `finally`. Closing the channel is the
+  cooperative stop signal, and an optimizer that does not return on it is
+  cancelled once its budget is spent.
 - **The trajectory is the single event log.** Events and responses are recorded
   directly onto the trajectory, the
   [`trajectory_recorder`]({{ '/reference/events-and-trajectory#middleware-where-scope-and-recording-live' | relative_url }})

docs/reference/results.md

@@ -133,11 +133,15 @@ readable data.)
   `summary`. The summary carries `asr`, `n_tasks`, `n_success`, `n_completed`,
   `n_failed`, `n_error`, `n_budget_exhausted`, `n_timeout`, `n_timeout_empty`,
   `n_skipped`, `max_primary_score`, `mean_primary_score`, and `total_llm_usage`.
-  Here `n_completed = done + max_runs + budget_exhausted` and
-  `asr = n_success / n_completed`, so errored, timed-out and skipped tasks are
-  excluded from the denominator. A truncated task is a lower bound on what the
-  attacker would have achieved, not a verdict, so it enters neither the
-  numerator nor the denominator.
+  Here `n_completed` counts `done`, `max_runs`, `budget_exhausted`, and
+  truncated `timeout` tasks, and `asr = n_success / n_completed`: an attacker
+  that has not succeeded inside its time budget has failed under the threat
+  model being measured, exactly as one that exhausted its cost budget has.
+  Errored, `timeout_empty` (nothing judged, so likelier a hung provider than an
+  attacker working to the wire) and skipped tasks are excluded from the
+  denominator. A truncated task is still reported separately via `n_timeout`,
+  because it is a lower bound on what the attacker would have achieved given
+  more time.
 - **`manifest.json`**: the same `experiment` and `summary` blocks, a `status`
   (`in_progress` / `complete`), and a scalar `tasks[]` index. It is rewritten as
   tasks land, so it is always current.