feat(secops): add Detection Engineering Agent MCP tools and coverage evaluation skill

.gitignore

@@ -133,7 +133,9 @@ celerybeat.pid
 
 # Environments
 .env
+.envrc
 .venv
+.gcloud/
 env/
 venv/
 ENV/

docs/agentic_detection_engineering.md

@@ -0,0 +1,190 @@
+# Agentic Detection Engineering in Google SecOps
+
+Agentic Detection Engineering (ADE) enables security teams to automate and accelerate the end-to-end detection engineering lifecycle using Google Security Operations (SecOps) APIs and AI assistants. 
+
+By integrating threat intelligence, automated Threat Detection Opportunity (TDO) extraction, synthetic telemetry simulation, and sandbox rule coverage evaluations, ADE transforms unstructured threat descriptions into tested, production-ready YARA-L 2.0 detection rules.
+
+---
+
+## Overview
+
+Traditional detection engineering requires manual parsing of threat intelligence reports, manual drafting of adversary simulation commands or test logs, tedious cross-referencing against existing rule corpora, and extensive manual tuning to write YARA-L 2.0 detection rules.
+
+Agentic Detection Engineering in Google SecOps streamlines this into a continuous, automated lifecycle:
+
+```
++-----------------------------+
+| Threat Intelligence Ingest  | (Blogs, Reports, CVEs, TTPs)
++--------------+--------------+
+               |
+               v
++-----------------------------+
+| TDO Generation              | (generate_threat_detection_opportunity)
++--------------+--------------+
+               |
+               v
++-----------------------------+
+| Synthetic Simulation        | (generate_synthetic_events)
++--------------+--------------+
+               |
+               v
++-----------------------------+
+| Rule Coverage Evaluation    | (evaluate_rule_coverage_long_running)
++--------------+--------------+
+               |
+               v
++-----------------------------+
+| Operation Polling & Results | (get_operation)
++--------------+--------------+
+               |
+               +-----------------------+
+               |                       |
+       [Coverage Confirmed]      [Coverage Gap]
+               |                       |
+               v                       v
+      (No action needed)       +-----------------------------+
+                               | Candidate Rule Generation   | (generate_rules)
+                               +--------------+--------------+
+                                              |
+                                              v
+                               +-----------------------------+
+                               | Human Review & Deployment   | (test_rule, create_rule)
+                               +-----------------------------+
+```
+
+---
+
+## The Detection Engineering Lifecycle
+
+### 1. Threat Detection Opportunity (TDO) Extraction
+Detection engineers or autonomous agents analyze threat reports, advisories, or post-incident reviews to identify observable adversary behaviors. Using `generate_threat_detection_opportunity`, the input text is transformed into structured TDO objects containing:
+- **TDO ID**: Unique identifier (e.g., `t01`, `t02`).
+- **Summary**: Concise description of the attacker tactic or procedure.
+- **MITRE ATT&CK Mapping**: Specific tactics and techniques (e.g., `T1059.001` PowerShell, `T1071.001` Web Protocols).
+- **Log Types**: Relevant Chronicle log ingestion types (e.g., `WINEVTLOG`, `PROCESS_EXECUTION`, `GCP_CLOUDAUDIT`).
+
+### 2. Synthetic Event Simulation
+To evaluate whether existing detection rules would catch the activity, `generate_synthetic_events` generates high-fidelity synthetic telemetry. This produces:
+- Raw mock log lines matching the targeted log type formats.
+- Structured Unified Data Model (UDM) events with appropriate entity metadata (`principal`, `target`, `network`, `about`).
+- JSON-encoded UDM event strings (`udmJson`) formatted for direct consumption by Chronicle evaluation engines.
+
+### 3. SecOps UI: Synthetic Data Visibility
+Synthetic events and the resulting detections can be displayed directly in the Google SecOps Web UI for interactive inspection and validation.
+
+To view synthetic telemetry in list and detail views:
+1. Navigate to **Google SecOps**.
+2. Click **Settings** (gear icon) in the navigation bar.
+3. Select **User Preferences** > **Synthetic Data Visibility**.
+4. Check **Show synthetic test data**.
+5. Click **Save**.
+
+![Synthetic Data Visibility](img/synthetic_data_visibility.png)
+
+> **Note:** Enabling this setting displays synthetic test data (including events, detections, and alerts) in list and detail views across Chronicle. This does not affect data generated by Security Validation, which remains hidden by default.
+
+### 4. Rule Coverage Evaluation via Long-Running Operations (LRO)
+Evaluating synthetic events against an organization's active ruleset is computationally intensive. The tool `evaluate_rule_coverage_long_running` initiates an asynchronous evaluation job via Chronicle's `:evaluateRuleCoverageLongRunning` API endpoint:
+- **Sandboxed Execution:** Synthetic events are evaluated in an ephemeral sandbox without committing test records to permanent customer log storage.
+- **Composite Coverage Control:** The `exclude_composite_coverage` parameter allows filtering out multi-event composite rules when testing single atomic behaviors.
+- **Asynchronous Operation:** Returns a standard Google Long-Running Operation resource (e.g., `operations/dea-bkFXS0...`).
+
+### 5. Polling Operation Status
+The `get_operation` tool polls the returned operation name until completion:
+- **In-Progress:** Returns operation metadata including progress status and percentages.
+- **Completed:** Returns the final evaluation result containing covered TDO IDs, uncovered TDO IDs, matching rule identifiers, and matched event counts.
+
+### 6. Candidate Rule Synthesis
+For any TDO identified as having a coverage gap, `generate_rules` synthesizes candidate YARA-L 2.0 detection rules. The generated rules include:
+- Informative `meta` section with author, description, severity, and MITRE ATT&CK tags.
+- Precise `events` logic referencing UDM fields.
+- Deduplication and aggregation logic in `match` and `condition` sections.
+
+### 7. Human-in-the-Loop Review and Deployment
+Generated rules must never be automatically activated in production without human validation. Detection engineers follow these verification steps:
+1. **Rule Logic Inspection:** Verify UDM field references and thresholds.
+2. **Backtesting (`test_rule`):** Execute historical test queries over real tenant data to assess alert volume and detect potential false positives.
+3. **Draft Rule Creation (`create_rule`):** Deploy rule in a disabled (`enabled=False`) or alerting-only state for staging observation.
+4. **Activation:** Enable live evaluation once verified (via the SecOps console or rule management tools).
+
+---
+
+## Available MCP Tools
+
+The `secops-mcp` server provides 5 purpose-built tools for Agentic Detection Engineering:
+
+| Tool | Purpose | Key Parameters |
+|------|---------|----------------|
+| `generate_threat_detection_opportunity` | Extracts structured TDOs from threat descriptions | `threat_description`, `log_types` |
+| `generate_synthetic_events` | Synthesizes realistic raw logs and UDM test events | `threat_detection_opportunities` |
+| `evaluate_rule_coverage_long_running` | Starts asynchronous rule coverage evaluation LRO | `threat_detection_opportunity_events`, `exclude_composite_coverage` |
+| `get_operation` | Polls status and retrieves LRO evaluation results | `name` |
+| `generate_rules` | Generates candidate YARA-L 2.0 rules for coverage gaps | `threat_detection_opportunities`, `background_context` |
+
+For detailed parameter schemas and API reference, see [SecOps MCP Tools](servers/secops_mcp.md).
+
+---
+
+## Agent Skill: `detection-engineering-coverage-evaluation`
+
+The **Google SecOps Extension** packages this entire workflow into a turnkey agent skill:
+
+- **Trigger:** `/security:detect`, `"Evaluate coverage for [URL/Text]"`, `"Develop detections for [Threat]"`.
+- **Location:** `extensions/google-secops/skills/detection-coverage/SKILL.md` (exposed via `.agent/skills/detection-coverage/`).
+- **Prompt Injection Safeguards:** Threat intelligence articles and external blog URLs are treated as untrusted data. The skill enforces clear demarcation between ingested threat content and agent execution instructions.
+- **Human Authorization Gate:** Explicit user confirmation is strictly required prior to saving or enabling any detection rules in production.
+
+---
+
+## Example Workflow
+
+### Step 1: Ingest Threat Description
+```python
+tdo_response = generate_threat_detection_opportunity(
+    threat_description="""
+    Adversaries execute encoded PowerShell commands to download secondary stage payloads
+    from external C2 servers and establish persistent scheduled tasks.
+    """,
+    log_types=["WINEVTLOG", "PROCESS_EXECUTION"]
+)
+```
+
+### Step 2: Generate Synthetic UDM Events
+```python
+events_response = generate_synthetic_events(
+    threat_detection_opportunities=tdo_response["threat_detection_opportunities"]
+)
+```
+
+### Step 3: Evaluate Coverage Sandbox
+```python
+lro_response = evaluate_rule_coverage_long_running(
+    threat_detection_opportunity_events=events_response["threat_detection_opportunity_events"],
+    exclude_composite_coverage=True
+)
+operation_name = lro_response["name"]
+```
+
+### Step 4: Poll LRO Until Done
+```python
+status = get_operation(name=operation_name)
+# Poll until status["done"] is True
+# Coverage results indicate uncovered TDOs
+```
+
+### Step 5: Generate YARA-L 2.0 Rule for Gaps
+```python
+rules_response = generate_rules(
+    threat_detection_opportunities=uncovered_tdos,
+    background_context="Enterprise Windows workstations with Defender and Sysmon telemetry."
+)
+```
+
+---
+
+## Related Documentation
+
+- [SecOps MCP Server Reference](servers/secops_mcp.md)
+- [Detection Engineer Persona](personas/detection_engineer.md)
+- [Google SecOps Extension Skills](google_secops_extension.md)
+- [Official Google SecOps ADE Guide](https://docs.cloud.google.com/chronicle/docs/secops/agentic-detection-engineering)

docs/google_secops_extension.md

@@ -81,6 +81,11 @@ You will be prompted for two environment variables for the MCP configuration:
 *   **Trigger**: "Hunt for [Threat]", "Search for TTP [ID]".
 *   **Function**: Assists in proactive threat hunting by generating hypotheses and constructing complex UDM queries for Chronicle.
 
+### 6. Detection Engineering (`detection-engineering-coverage-evaluation`)
+*   **Trigger**: "Develop detections for [Threat]", "Evaluate coverage for [URL/Text]", `/security:detect`.
+*   **Function**: Orchestrates the end-to-end Detection Engineering lifecycle: extracts TDOs from threat intelligence, simulates synthetic UDM events, evaluates existing rule coverage with long-running operations, generates draft YARA-L 2.0 rules to close coverage gaps, and deploys approved rules.
+*   **Guide**: See [Agentic Detection Engineering Guide](agentic_detection_engineering.md).
+
 ## How it Works
 
 These skills act as **Driver Agents** that:
@@ -96,7 +101,7 @@ The skills employ an **Adaptive Execution** strategy to ensure robustness:
 2.  **Prioritize Remote**: If the **Remote MCP Server** is connected, the skill uses remote tools (e.g., `list_cases`, `udm_search`) for maximum capability.
 3.  **Fallback to Local**: If remote tools are unavailable, the skill automatically falls back to **Local Python Tools** (e.g., `search_security_events`).
 
-For a detailed mapping of Remote vs. Local capabilities, see [`TOOL_MAPPING.md`](../TOOL_MAPPING.md).
+For a detailed mapping of Remote vs. Local capabilities, see [`TOOL_MAPPING.md`](https://github.com/google/mcp-security/blob/main/extensions/google-secops/TOOL_MAPPING.md).
 
 
 ## Cross-Compatibility

docs/index.md

@@ -20,6 +20,7 @@ If you're new to this project, we recommend starting with the [Usage Guide](usag
 ## Quick Links
 
 - **[Installation & Setup](usage_guide.md#getting-started)** - Get started quickly with installation instructions
+- **[Agentic Detection Engineering](agentic_detection_engineering.md)** - Automate detection engineering with TDO extraction, synthetic events, and rule coverage evaluations
 - **[Configuration Reference](usage_guide.md#mcp-server-configuration-reference)** - Configure the MCP servers for your environment
 - **[Usage Examples](usage_guide.md#usage-examples)** - See examples of how to interact with the MCP servers
 - **[Development Guide](development_guide.md)** - Learn how to contribute to or extend the project

docs/personas/detection_engineer.md

@@ -33,8 +33,12 @@ The Detection Engineer, sometimes referred to as a Content Developer, is respons
     *   `list_security_rules`: To review existing rules, identify overlaps, and understand current coverage.
     *   `get_security_alerts`: To analyze the performance and triggering patterns of specific rules.
     *   `lookup_entity`: To quickly gather context on entities involved in test alerts or potential FPs/FNs.
-    *   *(Potentially tools for rule creation/modification/deployment if available via MCP, e.g., `create_detection_rule`, `update_detection_rule`)*
-    *   *(Potentially `validate_udm_query` if available)*
+    *   `generate_threat_detection_opportunity`: To generate structured Threat Detection Opportunities (TDOs) including MITRE ATT&CK techniques, procedures, and log types from threat text.
+    *   `generate_synthetic_events`: To simulate high-fidelity raw logs and enriched UDM events for testing detection coverage.
+    *   `evaluate_rule_coverage_long_running`: To evaluate existing rule coverage by simulating synthetic events in an asynchronous operation.
+    *   `get_operation`: To poll and retrieve the results of long-running operations.
+    *   `generate_rules`: To generate draft YARA-L 2.0 rules for identified detection coverage gaps.
+    *   `create_rule` / `validate_rule`: To validate syntax and deploy approved YARA-L rules.
 *   **`gti-mcp` (For Context & Rule Ideas):**
     *   `search_threats`, `get_collection_report`, `get_collection_mitre_tree`, `get_threat_intel`: To research threats, TTPs, and vulnerabilities that require detection coverage.
     *   `get_file_report`, `get_domain_report`, etc.: To understand IOC characteristics for rule development.
@@ -46,10 +50,11 @@ The Detection Engineer, sometimes referred to as a Content Developer, is respons
 *   **`bigquery` (For Large-Scale Testing):**
     *   `execute-query`: For testing rules against large historical datasets in data lakes.
 
-## Relevant Runbooks
+## Relevant Runbooks & Agent Skills
 
 Detection Engineers are central to the detection lifecycle and related processes:
 
+*   **`detection-engineering-coverage-evaluation` (`/security:detect`)**: The official end-to-end skill automating TDO generation, synthetic event simulation, coverage evaluation, and YARA-L rule drafting.
 *   `detection_rule_validation_tuning.md`: Core workflow for analyzing and tuning rules.
 *   `detection_as_code_workflows.md`: Defines the process for developing and deploying rules if using DaC.
 *   `detection_report.md`: Used to document the performance and logic of specific detections.