Azure and GCP mapped at AWS depth, importable exceptions, an isometric report and a MkDocs docs site

.github/workflows/pages.yml

@@ -1,8 +1,9 @@
 name: Docs
 
-# Builds the documentation site (site/build.py) and deploys it to GitHub Pages
-# on every push to main that touches the docs, the content they are generated
-# from (click commands, docstrings, config.yaml, CHANGELOG.md) or the site itself.
+# Builds the documentation site (MkDocs Material, mkdocs.yml) and deploys it to
+# GitHub Pages on every push to main that touches the docs, the content they are
+# generated from (click commands, docstrings, config.yaml, CHANGELOG.md) or the
+# site itself. The site is served at https://morpheuslord.github.io/cloudg/.
 #
 # One-time setup: under Settings -> Pages, set "Build and deployment"
 # source to "GitHub Actions". The GITHUB_TOKEN cannot create the Pages
@@ -18,12 +19,15 @@ on:
       - "cloudg/**"
       - "config.yaml"
       - "CHANGELOG.md"
+      - "README.md"
       - "pyproject.toml"
+      - "mkdocs.yml"
       - ".github/workflows/pages.yml"
   pull_request:
     paths:
       - "docs/**"
       - "site/**"
+      - "mkdocs.yml"
       - ".github/workflows/pages.yml"
   workflow_dispatch:
 
@@ -46,14 +50,19 @@ jobs:
         with:
           python-version: "3.12"
 
-      # The CLI and API reference pages are generated by importing cloudg,
-      # so the package is installed; its core dependencies already include
-      # everything the site generator needs (markdown-it-py, Pygments, Jinja2).
-      - name: Install cloudg
-        run: pip install .
+      # The CLI, config and API reference pages are generated from the code,
+      # so cloudg is installed next to the docs dependency group. pip reads
+      # dependency groups from 25.1 on.
+      - name: Install cloudg and the docs tools
+        run: |
+          python -m pip install --upgrade "pip>=25.1"
+          pip install . --group docs
 
       - name: Build the site
-        run: python site/build.py --strict
+        env:
+          NO_MKDOCS_2_WARNING: "1"
+          DISABLE_MKDOCS_2_WARNING: "true"
+        run: mkdocs build --strict
 
       - uses: actions/upload-pages-artifact@v3
         if: github.event_name != 'pull_request'

CHANGELOG.md

@@ -2,6 +2,24 @@
 
 Notable changes per release. Patch releases are folded into the major entry they belong to.
 
+## 0.7.0 (2026-10-25)
+
+Azure and GCP are mapped as deeply as AWS, scanner and collector failures raise exceptions you can import and catch, report.html opens on an animated isometric map of the estate, and the docs site moves to MkDocs Material.
+
+Added:
+
+- Azure depth. Besides the Resource Graph breadth query, cloudg reads the `policyresources`, `securityresources`, `recoveryservicesresources`, `networkresources`, `maintenanceresources`, `insightsresources` and `kubernetesconfigurationresources` tables, management-group and tenant-scope role assignments, ARM child resources (APIM, Service Bus, Event Hubs, DNS record sets, Front Door, Virtual WAN, Data Factory and Synapse pipelines, AI Search, Azure OpenAI deployments and more), diagnostic settings, Entra ID through Microsoft Graph (users, groups, app registrations, service principals, app-role assignments, OAuth2 grants, directory roles, PIM eligibility) and the objects inside AKS clusters. The ARM type catalog grows from 125 to 278 types. `azure.map_entra`, `azure.child_resources`, `azure.diagnostic_settings` and `azure.data_plane` turn the extra sources off.
+- GCP depth. 350 Cloud Asset Inventory types now have extractors (123 before), and API sources fill what the inventory does not return: Cloud DNS records, Security Command Center, Cloud Identity groups, IAM deny policies and principal access boundaries, PAM, audit configs, Cloud Monitoring, OS Config, Artifact Analysis, Backup and DR, Apigee, Cloud Scheduler and others. `gcp.deep_sources` picks which ones run.
+- Azure and GCP record coverage per task family, as AWS does, show "not enabled" placeholders for Defender plans, Sentinel and Security Command Center services, and follow `--services` / `inventory.exclude_services`.
+- `cloudg.exceptions`: `CloudGError` and its subclasses (`ScannerError`, `ScannerTimeoutError`, `ScannerNotInstalledError`, `ScannerExecutionError`, `ScannerOutputError`, `ReportParseError`, `ScannerFailed`, `CollectionError`, `CredentialError`, `ProviderSDKMissingError`, `ConfigurationError`), also exported from `cloudg`. Every scanner wrapper takes `strict=True` to raise on the first failure, records each failure in `.failures`, and has `raise_for_errors()`. `parse_report(path, strict=True)` raises `ReportParseError` for an unreadable report.
+- report.html: an estate map in 3D (provider plates, account and region platforms, network slabs, subnet tiles, a control deck for identity and governance), with every relation drawn as a trace line on a lane of its own, asset selection, tracing from the internet, turning, zoom and pan. The D3 tree and force views stay one toggle away. The page also gains a posture overview, a top-risks list, an Assets tab, CSV export, light and dark themes and print styles. The Anatomy iso-kit and the Hairline kernel (both MIT) ship in `cloudg/templates/vendor/` and are always embedded.
+- The docs site is built with MkDocs Material, with API pages generated from the docstrings, tabbed examples, an Examples section, an Exceptions page, a Credits page and a sample report at `/sample-report.html`.
+
+Changed:
+
+- Credential, SDK and collection failures raise `CredentialError`, `ProviderSDKMissingError` and `CollectionError`. They keep their old bases (`RuntimeError`, `ImportError`), so existing `except` clauses still match. `ResilienceError` and `LiveOperationRejected` now derive from `CloudGError`.
+- The PyPI project page is a short overview (`docs/PYPI.md`) that links into the docs site. `docs/DOCUMENTATION.md` stays the full reference that the MCP server serves.
+
 ## 0.6.1 (2026-10-10)
 
 A security fix for report.html, a round of fixes found while documenting every command and API, and the new docs site. Upgrade if you open reports built from scanner output you did not produce yourself. Several commands now behave the way their help text always said, so check the Changed list if you script `cloudg report`, `cloudg scan` or the exit codes.

README.md

@@ -24,6 +24,8 @@ Want an AI agent to work with all of it? `cloudg mcp serve` turns cloudg into an
 
 **Full documentation:** the rendered handbook lives at [morpheuslord.github.io/cloudg](https://morpheuslord.github.io/cloudg/), with the same content as markdown in the [feature reference](https://github.com/morpheuslord/cloudg/blob/main/docs/DOCUMENTATION.md) and release notes in the [changelog](https://github.com/morpheuslord/cloudg/blob/main/CHANGELOG.md). The inventory mapper has its own deep-dive documents: the [inventory reference](https://github.com/morpheuslord/cloudg/blob/main/docs/INVENTORY_REFERENCE.md) (every return structure and output file, field by field), the [inventory catalog](https://github.com/morpheuslord/cloudg/blob/main/docs/INVENTORY_CATALOG.md) (all 156 asset types, their metadata and relationships) and the [inventory internals](https://github.com/morpheuslord/cloudg/blob/main/docs/INVENTORY_INTERNALS.md) (how it works and how to extend it). The MCP server is covered by the [MCP guide](https://github.com/morpheuslord/cloudg/blob/main/docs/MCP.md), the [tool reference](https://github.com/morpheuslord/cloudg/blob/main/docs/MCP_TOOLS.md) and the [privacy reference](https://github.com/morpheuslord/cloudg/blob/main/docs/MCP_PRIVACY.md), and throttling by [rate limits and throttling](https://github.com/morpheuslord/cloudg/blob/main/docs/RESILIENCE.md).
 
+**Sample report:** to see what cloudg produces before running anything, [open the sample report](https://morpheuslord.github.io/cloudg/sample-report.html). It is the `report.html` of a synthetic AWS, Azure and GCP estate, so every ID in it is made up, and `scripts/build_sample_report.py` rebuilds it from the test fixtures.
+
 ---
 
 ## Quick start
@@ -375,7 +377,7 @@ Python 3.11 or newer. The moto/aiobotocore incompatibility around async response
 
 ## Credits
 
-cloudg builds on other people's work, and some of it ships inside the package.
+cloudg builds on other people's work, and some of it ships inside the package. The documentation site has the same list on its [credits page](https://morpheuslord.github.io/cloudg/credits/).
 
 [Prowler](https://github.com/prowler-cloud/prowler) (Apache-2.0). The 22 framework files in `cloudg/rules/frameworks/` are generated from Prowler's compliance JSON by `scripts/import_prowler_compliance.py`. That covers the control IDs, the control titles and descriptions, and the mapping from each control to Prowler check names. Every generated file says so in its header and in its `source` field. Prowler check names also appear in `cloudg/rules/check_equivalence.yaml`, and cloudg runs Prowler and parses its ASFF and OCSF output.
 
@@ -385,6 +387,8 @@ Scanners. [Checkov](https://github.com/bridgecrewio/checkov) (Apache-2.0), [Triv
 
 [parliament](https://github.com/duo-labs/parliament) (BSD-3-Clause) from Duo Labs does the IAM policy linting.
 
+Drawing engines. The estate map in report.html is drawn with the iso-kit from [Anatomy](https://github.com/wheresryan22/anatomy) (MIT, by Ryan), and its empty states use the kernel of [Hairline](https://hairline.lucasmarkes.com) 0.5.1 (MIT, by Lucas Marques). Both are copied into `cloudg/templates/vendor/` and embedded in every report, so the map works offline. The iso-kit is wrapped so it can sit inside a script element, and `cloudg/templates/vendor/NOTICE` lists exactly what changed.
+
 [MITRE ATT&CK®](https://attack.mitre.org/). The `mitre_attack_*.yaml` files contain ATT&CK technique IDs and descriptions. © The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation. ATT&CK is a registered trademark of The MITRE Corporation.
 
 Standards. The hand-written rulesets in `cloudg/rules/` reference public control numbering from the CIS Benchmarks for AWS, Azure and GCP, NIST SP 800-53, NIST CSF, PCI DSS, the AICPA SOC 2 Trust Services Criteria, the HIPAA Security Rule, GDPR, ISO/IEC 27001:2022 and the AWS Foundational Security Best Practices. The standards belong to their publishers. CIS Benchmarks is a trademark of the Center for Internet Security, and cloudg is not affiliated with or endorsed by CIS or any other publisher named here.

cloudg/__init__.py

@@ -1,6 +1,6 @@
 """cloudg (cloud graphing): multi-cloud mapping and security intelligence."""
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
 from cloudg.api import (
     AnalysisResult,
@@ -9,13 +9,39 @@
     PipelineResult,
 )
 from cloudg.config import CloudGConfig, load_config
+from cloudg.exceptions import (
+    CloudGError,
+    CollectionError,
+    ConfigurationError,
+    CredentialError,
+    ProviderSDKMissingError,
+    ReportParseError,
+    ScannerError,
+    ScannerExecutionError,
+    ScannerFailed,
+    ScannerNotInstalledError,
+    ScannerOutputError,
+    ScannerTimeoutError,
+)
 from cloudg.inventory import InventoryMapper, InventoryResult
 
 __all__ = [
     "__version__",
     "AnalysisResult",
     "CloudGEngine",
     "CloudGConfig",
+    "CloudGError",
+    "CollectionError",
+    "ConfigurationError",
+    "CredentialError",
+    "ProviderSDKMissingError",
+    "ReportParseError",
+    "ScannerError",
+    "ScannerExecutionError",
+    "ScannerFailed",
+    "ScannerNotInstalledError",
+    "ScannerOutputError",
+    "ScannerTimeoutError",
     "CollectionResult",
     "InventoryMapper",
     "InventoryResult",

cloudg/api_scanners.py

@@ -17,6 +17,7 @@
 from pathlib import Path
 from typing import Any, Callable
 
+from cloudg.exceptions import ScannerFailed
 from cloudg.schema.models import CloudAsset, Finding, NetworkEdge
 
 __all__ = [
@@ -64,24 +65,12 @@ def resolve_iac_dirs(
 # ---------------------------------------------------------------------------
 
 
-class ScannerFailed(RuntimeError):
-    """A scanner started but did not finish (timeout, launch failure).
-
-    ``findings`` holds whatever it produced before that (for example the
-    images Trivy did scan), so callers can keep partial results.
-    """
-
-    def __init__(self, scanner: str, errors: list[str], findings: list[Finding]) -> None:
-        super().__init__(f"{scanner} " + "; ".join(errors))
-        self.errors = list(errors)
-        self.findings = findings
-
-
 def _finish(label: str, scanner: Any, findings: list[Finding]) -> list[Finding]:
     """Return ``findings``, or raise ScannerFailed when the wrapper recorded errors."""
     errors = list(getattr(scanner, "errors", None) or [])
     if errors:
-        raise ScannerFailed(label, errors, findings)
+        failures = [f for f in getattr(scanner, "failures", None) or [] if f.detail in errors]
+        raise ScannerFailed(label, errors, findings, failures=failures)
     return findings
 
 

cloudg/cli_inventory.py

@@ -292,15 +292,16 @@ def _merge_findings(
 @click.option(
     "--services",
     default=None,
-    help="Service families/collectors to map, comma-separated (e.g. containers,serverless,security)",
+    help="Service families or task names to map, comma-separated, for AWS and Azure "
+    "(e.g. containers,serverless,security)",
 )
 @click.option(
     "--exclude-services", default=None, help="Service families/collectors to skip, comma-separated"
 )
 @click.option(
     "--kubernetes/--no-kubernetes",
     default=None,
-    help="Map workloads inside EKS clusters through the Kubernetes API",
+    help="Map workloads inside EKS and AKS clusters through the Kubernetes API",
 )
 @click.option(
     "--cloud-control/--no-cloud-control",

cloudg/collectors/gcp.py

@@ -43,6 +43,7 @@
 )
 from cloudg.config import GCP_DEFAULT_SKIP_ASSET_TYPES
 from cloudg.coverage import CollectionCoverage, ServiceStatus
+from cloudg.exceptions import CollectionError, ProviderSDKMissingError
 from cloudg.resilience.errors import describe_error
 from cloudg.resilience.gcp import RetryCounter, gcp_retry, gcp_scope, paced
 from cloudg.resilience.governor import get_governor
@@ -186,9 +187,7 @@ def _get_client(self) -> Any:
             try:
                 from google.cloud import asset_v1
             except ImportError as exc:
-                raise RuntimeError(
-                    "google-cloud-asset is not installed; install cloudg[gcp] to collect GCP"
-                ) from exc
+                raise ProviderSDKMissingError("gcp", missing="google-cloud-asset") from exc
             self._client = asset_v1.AssetServiceClient(credentials=self._credentials)
         return self._client
 
@@ -301,7 +300,7 @@ async def _collect_from_search(self, error: Exception, start: float) -> list[dic
                 error=describe_error(search_error),
                 start=search_start,
             )
-            raise RuntimeError(
+            raise CollectionError(
                 f"GCP asset enumeration failed for {self._scope}: "
                 f"ListAssets: {describe_error(error)}; "
                 f"SearchAllResources: {describe_error(search_error)}"

cloudg/collectors/multi.py

@@ -563,7 +563,10 @@ async def _azure_edges_and_coverage(
     """Collect one Azure subscription's edges and record its coverage.
 
     An edge failure is recorded as ``azure_edges`` and never discards the
-    assets; per-service collector errors become ``azure_<service>`` entries.
+    assets; per-service collector errors become ``azure_<service>`` entries,
+    and the per-task records of collectors that keep a ``coverage``
+    (:class:`~cloudg.inventory.azure_deep.AzureDeepInventoryCollector`) are
+    copied in. ``azure_full`` is PARTIAL when any of them failed.
     """
     edges: list[NetworkEdge] = []
     try:
@@ -573,12 +576,15 @@ async def _azure_edges_and_coverage(
         coverage.record("azure_edges", ServiceStatus.FAILED, error=describe_error(exc))
 
     duration_ms = int((time.time() - start) * 1000)
+    task_records = list(getattr(getattr(collector, "coverage", None), "services", None) or [])
+    coverage.services.extend(task_records)
+    degraded = any(s.status in (ServiceStatus.FAILED, ServiceStatus.PARTIAL) for s in task_records)
     service_errors = dict(getattr(collector, "service_errors", None) or {})
     for service, error in service_errors.items():
         coverage.record(f"azure_{service}", ServiceStatus.FAILED, error=error)
     coverage.record(
         "azure_full",
-        ServiceStatus.PARTIAL if service_errors else ServiceStatus.SUCCESS,
+        ServiceStatus.PARTIAL if service_errors or degraded else ServiceStatus.SUCCESS,
         asset_count=asset_count,
         duration_ms=duration_ms,
     )

cloudg/config.py

@@ -166,6 +166,49 @@ class AzureConfig(BaseModel):
         description="Map the management group / subscription / Azure Policy hierarchy (inventory)",
     )
 
+    # Azure parity
+    map_entra: bool = Field(
+        default=True,
+        description="Inventory: map Entra ID (users, groups, apps, service principals, "
+        "directory roles, PIM) through Microsoft Graph; needs Graph read permissions",
+    )
+    entra_max_objects: int = Field(
+        default=10000, ge=0, description="Inventory: most objects read per Microsoft Graph listing"
+    )
+    entra_max_expansions: int = Field(
+        default=1000,
+        ge=0,
+        description="Inventory: most groups / apps / service principals whose members, owners, "
+        "federated credentials or app-role assignments are read one by one",
+    )
+    child_resources: bool = Field(
+        default=True,
+        description="Inventory: read child resources Resource Graph does not return (APIM APIs, "
+        "Service Bus queues, DNS records, functions, ...) through ARM REST",
+    )
+    max_child_parents: int = Field(
+        default=500,
+        ge=0,
+        description="Inventory: most parent resources queried per child-resource kind",
+    )
+    diagnostic_settings: bool = Field(
+        default=True, description="Inventory: read diagnostic settings of logged resource types"
+    )
+    max_diagnostic_settings: int = Field(
+        default=1000,
+        ge=0,
+        description="Inventory: most resources whose diagnostic settings are read",
+    )
+    data_plane: bool = Field(
+        default=True,
+        description="Inventory: read AI Search and Synapse data-plane metadata (data sources, "
+        "indexers, pipelines, linked services); needs data-plane reader roles",
+    )
+    rest_concurrency: int = Field(
+        default=8, ge=1, le=32, description="Inventory: concurrent ARM / Graph REST calls"
+    )
+    # end Azure parity
+
 
 # High-churn Cloud Asset Inventory types the GCP collector skips by default
 # (GCPConfig.skip_asset_types; cloudg.collectors.gcp.DEFAULT_SKIP_ASSET_TYPES)
@@ -231,6 +274,33 @@ class GCPConfig(BaseModel):
     vpc_service_controls: bool = Field(
         default=True, description="Map VPC Service Controls perimeters"
     )
+    # GCP parity: API-backed sources of the deep inventory collector
+    deep_sources: bool = Field(
+        default=True,
+        description="Inventory mapping: also read what Cloud Asset Inventory does not return "
+        "(Cloud DNS records, Security Command Center, Cloud Identity groups, deny and access "
+        "boundary policies, PAM, audit-log configs, Monitoring, OS Config, Artifact Analysis, "
+        "Backup and DR, Cloud Scheduler, Apigee, Infrastructure / Deployment Manager)",
+    )
+    rest_timeout_seconds: float = Field(default=60.0, ge=5.0)
+    cloud_identity_customer_id: str | None = Field(
+        default=None,
+        description="Cloud Identity customer ID (Cxxxxxxx) for group membership; "
+        "found from the organization when unset",
+    )
+    max_dns_records_per_zone: int = Field(default=5000, ge=0)
+    max_vulnerability_occurrences: int = Field(
+        default=5000, ge=0, description="Artifact Analysis occurrences read per project"
+    )
+    max_groups: int = Field(default=2000, ge=0)
+    max_group_members: int = Field(default=5000, ge=0)
+    expand_predefined_roles: bool = Field(
+        default=True, description="Read the permissions of bound predefined roles (IAM Roles API)"
+    )
+    max_role_expansions: int = Field(default=500, ge=0)
+    max_scheduler_locations: int = Field(default=25, ge=0)
+    source_concurrency: int = Field(default=4, ge=1, le=32)
+    # end GCP parity
 
 
 class ScannerConfig(BaseModel):