Evaluation of Agentic IDS Testbed Generation

IDS research testbeds are resource-intensive to build and adapt. This repo contains a benchmark and evaluation harness used to test whether state-of-the-art coding agents can take over parts of this work.

This includes:

1. 19 tasks derived from AIT AttackBed. Tasks focus on configuring vulnerable infrastructure via Ansible roles, and scripting attacks via AttackMate scenarios. Each task consists of: an expert-validated prompt, a repo base state the agent starts working on, and a functional test suite to evaluate generated code.
2. A CLI to trigger code generations and evaluations.
3. Multi-stage evaluation runners providing details on failure classification for syntax, deployment, and behavioural test stages.
4. Configuration for three state-of-the-art agents: Claude-Code with Opus 4.6, OpenCode with Moonshot Kimi K2.6, and Mistral's Vibe with Devstral 2.

Repository layout

Path	Contents
agents/	Each agent provides a run.sh entrypoint script which the code generation harness calls to execute an agent.
benchmark/tasks/	Task instances. Each task has a prompt.md containing its prompt, config, and test suites. Tasks 1–9 are Ansible roles, 10–19 are AttackMate scenarios.
benchmark/runner/	cli.py driver used to start an agent on a task, and to evaluate generated code.
records/main-results/	Recorded results from running claude-code-opus-4-6 and vibe-devstral-2 five times on each task. Contains the agent-generated attackbed.diff, agent trajectory.json, and eval_result.json per task.
records/attackmate-goldens/	AttackMate artefacts produced by executing reference solutions. Used as a reference for writing AttackMate task tests. Only relevant for traceability purposes.
records/task-requirements/	Documents the expert-validation process of task requirements. These are the source from which prompts and tests in benchmark/tasks/ are derived. Only relevant for traceability purposes.

Setup

Prerequisites: Docker, uv, and an OpenStack account.

Manual steps:

1. Complete the AttackBed setup. This includes:
   1. Creating a floating IP on OpenStack with the description mgmt.
   2. Building the packer images for adminpc, attacker, corpdns, firewall, userpc, videoserver, webcam.
2. Populate Packer configuration (defaults.json), Terragrunt configuration (terraform.tfvars), and Ansible hosts with your constants. This can be done:
   1. If you provide your own AttackBed fork as a reference for each task: Directly in the fork. See our fork for reference.
   2. Manually before starting an evaluation run.

Install and build dependencies:

# 1. Install runner deps
cd benchmark/runner
uv sync

# 2. Build the sandbox image (runs both agents and evaluators)
docker build -t atb-sandbox:latest sandbox/

Add your secrets:

cd benchmark/runner
cp .env.example .env
cp attackbed.env.example attackbed.env

cd agents/claude-code
cp .env.example .env

cd agents/opencode-kimi
cp .env.example .env

cd agents/mistral-vibe
cp .env.example .env

Generate code

From benchmark/runner/:

# Run an agent against every task (or a subset via -i)
uv run cli.py run --agent claude-code
uv run cli.py run --agent mistral-vibe --instances 1_simulatedwebcam 4_zoneminder
uv run cli.py run -a opencode-kimi -i 11_pwnkit_privesc

Each run produces the following outputs at benchmark/outputs/<run-id>/<task>/:

• attackbed: the repo containing the agent's edits
• trajectory.json: the agent's trajectory

Evaluate generated code

From benchmark/runner/:

# Evaluate all tasks of a code generation run (or a subset via -i)
uv run cli.py eval ../outputs/<run-output-directory>
uv run cli.py eval ../outputs/<run-output-directory> --instances 1_simulatedwebcam 4_zoneminder

Each evaluation produces the following outputs at benchmark/outputs/<run-id>/<task>/:

• eval_result.json: contains the per-stage evaluation results
• AttackMate tasks also produce AttackMate execution artefacts: output.log, attackmate.log, attackmate.json

Adding a new agent

1. Create agents/<name>/run.sh which is the entrypoint called by the code generation harness. The sandbox provides:
   • /artifacts/prompt.txt: the task prompt
   • /artifacts/attackbed/: the target repo to work on
2. Write any output to save to /artifacts/.
3. Invoke the agent with uv run cli.py run --agent <name>.

Adding a new task

Create benchmark/tasks/<id>_<name>/prompt.md. Configure the repo the agent will receive, the test suite, and the evaluation runner to use (currently ansible-role or attackmate-scenario).