Repository navigation
Azure and GCP mapped at AWS depth, importable exceptions, an isometric report and a MkDocs docs site - #33
Merged
Conversation
…pers cloudg.exceptions collects every error cloudg raises on purpose under CloudGError, keeping the old built-in bases so existing except clauses still match. The scanner wrappers record each failure as a ScannerError in .failures, raise it at once with strict=True, and raise_for_errors() raises after a run. parse_report takes strict=True and raises ReportParseError for unreadable reports. Credential, SDK and collection failures now raise CredentialError, ProviderSDKMissingError and CollectionError; ResilienceError and LiveOperationRejected join the hierarchy.
EXTRACTORS[t] = fn replaced the earlier extractor when a second module registered the same Cloud Asset Inventory type. The registry now keeps a list per type, like the Azure registry, and extract() runs every entry in order. A failing extractor is recorded in extraction_error and the rest still run.
The report opens on a posture overview (severity bar, share of assets with no open findings, top risks) and gains an Assets tab, CSV export, light and dark themes, print styles and a phone layout. The map is laid out in Python (cloudg/renderers/iso_scene.py): a base plate per provider, platforms per account and region, slabs for networks, tiles for subnets and a solid per asset, with identity and governance assets on a control deck held above the plate. Every relation is routed orthogonally in 3D from its own source port, on a lane no other trace shares, to its own target port, so each one can be followed end to end. The page projects the scene with the Anatomy iso-kit, paints it back to front, animates it in, runs a dot along each lit trace, turns, zooms and pans, and traces paths from the internet. The D3 tree and force views stay one toggle away, and are used alone above 2500 assets or without the map engines. The Anatomy iso-kit and the Hairline kernel (both MIT) are vendored and always inlined. The empty states use a Hairline figure of an empty tray.
The custom generator (site/build.py, site/docsgen, site/templates and the old static CSS and JS) is replaced by MkDocs Material, set up like the Startup-SBOM docs: slate by default with a light toggle, navigation tabs, a release banner, code copy and annotations, linked tabs, mermaid and snippets. site/hooks.py and site/gen/ generate the reference pages at build time and never commit them: the CLI usage and options tables from the click commands, the config.yaml reference, the changelog, and the docs/*.md references split at their "##" headings. Split pages keep the anchors GitHub gives each heading, and every URL of the old site is still served. The API pages render their objects with mkdocstrings below the hand-written prose, with source links to GitHub. New pages: a landing page, an API overview, an Exceptions page for cloudg.exceptions, an Examples section whose scripts live in site/examples/ and are included with snippets, Credits, and a Sample report page that embeds the report published at /cloudg/sample-report.html. The docs tools are a "docs" dependency group, and the Pages workflow runs mkdocs build --strict.
The PyPI long description is now docs/PYPI.md: logo, tagline, badges, a
link to the docs site and the sample report, the install extras, a quick
start for the CLI and Python, links into the site and the GitHub
references, and the credits. docs/DOCUMENTATION.md stays the full
reference, because the MCP layer serves its sections as
cloudg://docs/{topic} resources and the wheel ships it.
README.md and docs/DOCUMENTATION.md point at the sample report's direct address, https://morpheuslord.github.io/cloudg/sample-report.html, and the README's credits section at the credits page of the docs site.
The reports guide describes the estate map, the new tabs, the CSV and JSON downloads and the template variables the map adds. Anatomy and Hairline join the credits in the README, on the PyPI page and on the credits page. The JSON download keeps its findings.json name, and the sample report is rebuilt.
Types that were catalog-only (Spanner, Bigtable, Firestore, BigQuery tables, Dataplex, Data Fusion, NCC hubs, Interconnect, external VPN gateways, Certificate Manager, Cloud Run domain mappings, Eventarc channels, Managed Kafka topics, resource policies, identity pools) now have extractors, and the catalog gains about 200 types: Cloud Scheduler, Backup and DR, Apigee, Binary Authorization, Vertex AI agents and vector search, Discovery Engine, Datastream, Storage Transfer, NetApp Volumes, Cloud IDS, Cloud NGFW, Batch, Assured Workloads, tags, Infrastructure Manager, SCC and SDP configs, PAM, deny and access boundary policies. Relations use the names AWS uses for the same concept (SCHEDULED_BY, MONITORED_BY, BACKUP_TO, PROTECTED_BY_NACL, POLICY_DENIES_ACTION, PERMISSION_BOUNDARY_LIMITS, STREAMS_TO, ...). IAM bindings also link the principal to the bound role (USER_HAS_POLICY, GROUP_HAS_POLICY, ROLE_HAS_POLICY). Hierarchy discovery maps access levels. Shared files: four AssetType members in a "GCP parity" block of schema/models.py (CONTAINER_IMAGE, DATA_TABLE, TAG, FLEET) and the gcp-security: identifier scheme in linker_index.py. BigQuery tables move from OTHER to DATA_TABLE; identifiers are unchanged.
The deep GCP collector now runs API-backed sources from its _enrich() hook: Cloud DNS record sets, Security Command Center (tier, services, sources, exports, with "not enabled" placeholders), Cloud Identity groups, IAM deny and Principal Access Boundary policies, PAM entitlements, predefined role permissions, Data Access audit configs, Cloud Monitoring, OS Config, Artifact Analysis, Backup and DR, Firestore backups, Cloud Scheduler, Apigee, Infrastructure Manager, Deployment Manager and the IAM recommender. They use google-auth AuthorizedSession through an injectable GCPRestClient; each one is best-effort with its own gcp_<name> coverage record. Cross-asset passes fold tag bindings into TAGGED_WITH, link rotating secrets to their handlers (ROTATES_SECRET), give instance resource policies their verb, follow goog-dm and Config Connector labels, link boundary bindings and OS Config filters, and build API Gateway authorizers. inventory.services / exclude_services now filter GCP too, by family, API service, asset type or source, with per-family gcp_assets_<family> coverage. The mapper binds them, and the new gcp.* keys (in a "GCP parity" block of config.py), into the GCP collector.
INVENTORY_INTERNALS.md gets the API-backed sources, the REST client, the cross-asset passes, service families and coverage, the new gcp.* keys and the permissions they need, the new extractor modules (350 types with extractors) and the updated recipes and test notes. INVENTORY_REFERENCE.md lists the GCP placeholders, identifier schemes and coverage records. INVENTORY_CATALOG.md describes the four new asset types and adds the relationship matrix recorded from the GCP parity fixtures. Every addition sits in a "GCP parity" block so it merges next to the Azure changes without touching existing text.
Split the Azure inventory into named tasks with service families that mirror the AWS ones, so inventory.services and exclude_services can select Azure work the same way select_tasks does for AWS, and add a TaskRecorder for per-task SUCCESS / FAILED / PARTIAL / SKIPPED coverage. AzureRestClient reads ARM, Microsoft Graph and service data planes with the existing azure-identity credential: GET only (POST for read-only list actions), next links followed on the same host only, 429 / 503 retried after Retry-After, every call paced through the shared limiter. Shared files get Azure parity blocks: two AssetType values (PLACEMENT_GROUP, APP_REGISTRATION) and the azure.* inventory keys.
AKS clusters with Microsoft Entra integration are read the way EKS clusters are: the API server and CA come from the clusters section of the listClusterUserCredential kubeconfig, the bearer token is an Entra token for the AKS server application (what kubelogin requests), and map_cluster_objects issues GET requests only. map_cluster_objects takes a provider. On Azure, service accounts with the azure.workload.identity/client-id annotation assume the matching user-assigned identity, and load balancer IPs resolve to public IP assets. EKS output is unchanged. Clusters with local accounts only are skipped because their credentials are secrets; run-command is not used because it starts a pod in the cluster.
discover_entra reads the tenant once per run with the azure-identity credential and Graph v1.0 GETs: users, groups and direct members, app registrations (credential counts and expiry, owners, federated identity credentials, requested permissions), service principals and owners, app-role assignments (Microsoft Graph application permissions first), delegated OAuth2 grants, directory roles and PIM eligibility. Principals keep the entra:principal/<objectId> identifier the RBAC placeholders use, and carry no discovered_via, so deduplication folds every subscription's role grants into the real Entra object. Each Graph listing is its own coverage record; a 403 is recorded as a missing Graph permission and the rest is still mapped. tests/azure_fakes.py holds the Resource Graph, ARM and Graph fakes the Azure depth tests share.
The deep Azure collector now goes past the breadth query with best-effort tasks, each recorded as its own coverage entry: - More Resource Graph tables: policy assignments, exemptions, definitions and non-compliant states (a deny definition SCP_RESTRICTS the scopes of its enforced assignments), Defender assessments, sub-assessments, secure score and alerts, Recovery Services and Backup items (BACKUP_TO, Site Recovery REPLICATES_TO), Kubernetes configuration (extensions, Flux), Virtual Network Manager, guest configuration, maintenance and data collection rule associations, management-group role assignments and role definitions (ROLE/USER/GROUP_HAS_POLICY). - ARM REST children and subscription listings behind an injectable arm_client_factory: APIM APIs, backends, products, named values, policies (JWT authorizers), Service Bus, Event Hubs, function bindings, Logic Apps, Stream Analytics, Data Factory, Synapse, Kusto, SQL failover groups and auditing, Cosmos regions and data-plane roles, storage object replication and SFTP users, DNS record sets, Private Resolver, Virtual WAN, Front Door, ACR, AKS trusted access, OpenAI deployments and content filters, AML, AI Search, Automation, Sentinel, diagnostic settings, deny assignments, locks, deployments, deployment stacks, Lighthouse, PIM eligibility and the activity log export. - AI Search and Synapse data planes. - Defender plans monitor every covered resource; plans and Sentinel that are not on get "not enabled" placeholders, like AWS _disabled_asset. inventory.services / exclude_services select Azure tasks and filter the breadth rows by family. azure_full is PARTIAL when a task failed, as aws_full is. Secret values, credentials and contact details found in responses are dropped. Existing asset IDs and relations are unchanged; test_mode_and_queries now allows the extra tables.
InventoryMapper now binds the Azure collector to AzureDeepOptions built from inventory.services, exclude_services, kubernetes and the azure.* depth keys, the same way it binds the AWS options, so --services and --exclude-services filter Azure tasks too. Entra ID is discovered once per run next to the management-group hierarchy when azure.map_entra is on (the default) and the entra task is selected, with one coverage record per Graph listing.
INVENTORY_INTERNALS.md gains sections 8.4 to 8.10: the Azure tasks and families, coverage statuses, service filtering, the azure.* keys, the extra Resource Graph tables, ARM REST child specs and relation-only rows, data planes, Entra ID, in-cluster AKS and the AWS to Azure relation map, plus a recipe for a new child kind and the shared fakes. INVENTORY_REFERENCE.md lists the new placeholders, identifiers, coverage records, collector options and the AKS side of the Kubernetes mapper. INVENTORY_CATALOG.md gets the two new asset types, the ARM types added to the catalog and the relationship matrix observed in the Azure parity tests. Every addition sits between Azure parity markers.
Snapshots record the disk they were taken from (the disk BACKUP_TO the snapshot), and disks and snapshots flag export_open when their network access policy is AllowAll with public access on, so a SAS export URL could reach the data. Disk access resources are linked. discover_azure_hierarchy reads Microsoft.Subscription/aliases through ARM REST: each subscription vending alias is a PROVISIONED_PRODUCT that MANAGES its subscription. A credential without vending rights maps none and reports nothing.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
Up to standards ✅🟢 Issues
|
The AKS and GitHub issuer checks looked for the trusted name anywhere in the issuer URL, so an issuer such as https://evil.example/oidc.prod-aks.azure.com was taken for an AKS cluster and its trust mapped as ROLE_ASSUMES_ROLE instead of CROSS_ACCOUNT_TRUST (CodeQL: incomplete URL substring sanitization). issuer_kind() parses the URL and compares the hostname exactly, and both the ARM extractor and the Entra mapping use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch does four things.
Azure at AWS depth. Besides the Resource Graph breadth query, the Azure collector now reads seven more Resource Graph tables, management-group and tenant-scope role assignments, ARM child resources (APIM, Service Bus, Event Hubs, DNS record sets, Front Door, Virtual WAN, Data Factory and Synapse pipelines, AI Search, Azure OpenAI deployments and more), diagnostic settings, Entra ID through Microsoft Graph and the objects inside AKS clusters. The ARM type catalog goes from 125 to 278 types. Each task family gets its own coverage record, Defender plans and Sentinel get "not enabled" placeholders, and
--services/exclude_servicesfilter Azure the way they filter AWS.GCP at AWS depth. 350 Cloud Asset Inventory types have extractors (123 before), several extractors can now share one type, and API sources fill what the inventory does not return: Cloud DNS records, Security Command Center, Cloud Identity groups, IAM deny policies, principal access boundaries, PAM, audit configs, Cloud Monitoring, OS Config, Artifact Analysis, Backup and DR, Apigee and Cloud Scheduler. Same coverage records, placeholders and service filtering as Azure.
cloudg.exceptions. Every error cloudg raises on purpose derives fromCloudGError, and each keeps its old built-in base, so existingexcept ValueError/except RuntimeErrorcode still matches. The scanner wrappers record each failure in.failures, raise the first one withstrict=True, and haveraise_for_errors().parse_report(path, strict=True)raisesReportParseError.report.html. It opens on a posture overview and a 3D estate map: provider plates, account and region platforms, network slabs, subnet tiles and a control deck for identity and governance assets. The layout and every trace route are computed in Python (
cloudg/renderers/iso_scene.py); each relation gets its own ports and a lane no other trace shares, and tests check both. Selecting an asset lights its relations, "Trace from internet" walks the paths out from the internet, and the map turns, zooms and pans. The D3 tree and force views are one toggle away. The page also gains an Assets tab, CSV export, light and dark themes and print styles. The Anatomy iso-kit and the Hairline kernel (both MIT) are vendored and always inlined.Docs. The site is built with MkDocs Material: API pages from the docstrings, tabbed examples, an Examples section, an Exceptions page, a Credits page, and the sample report at
/sample-report.html, linked from the README. Every old URL still resolves. The PyPI page is a new shortdocs/PYPI.md;docs/DOCUMENTATION.mdstays as the reference the MCP server serves.Tests: the full suite passes (2242 passed, 6 skipped).
ruff checkandruff format --checkare clean, andmkdocs build --stricthas no warnings.