Skip to content

Canonical host deduplication (www.*), consolidate Nmap services and tidy reports/Graph Explorer - #6

Merged
Swapshadow merged 1 commit into
mainfrom
codex/fix-deduplication-and-reporting-issues-in-openeasm
Jun 11, 2026
Merged

Swapshadow merged 1 commit into
mainfrom
codex/fix-deduplication-and-reporting-issues-in-openeasm

Conversation

@Swapshadow

Copy link
Copy Markdown
Owner

Motivation

  • Reduce noisy duplicate hosts such as www.X vs X when they are functionally identical, and avoid duplicated Nmap service rows in reports.
  • Preserve raw data while presenting a DNSDumpster-like consolidated view for humans and downstream exports.
  • Make HTML reports readable and the Graph Explorer usable by deduplicating nodes, adding display modes and filters, and making the SVG responsive.

Description

  • Added a new canonicalization service backend/app/services/host_canonicalization.py that folds exact www.X aliases into X when IP/ASN/services/titles/TLS are equivalent, and produces canonical_hosts, alias_map, canonical_nmap_services and a summary object while keeping raw_hosts/raw_nmap_services.
  • Integrated canonicalization into the pipeline by using it in dnsdumpster_like_inventory and the main audit flow; Nmap open ports are consolidated via consolidate_nmap_services to service_scan["canonical_open_ports"] while raw Nmap rows are preserved as raw_open_ports/raw_nmap_services.
  • Updated API payloads to expose canonical_hosts, canonicalization metadata and raw fields (raw_subdomains, raw_nmap_services) and added summary counters (canonical_host_count, aliases_grouped, http_https_service_count, technology_count, etc.).
  • Updated reports to use canonical data: HTML report adds an aliases column, title/TLS CN/technologies columns and a filterable A-records table; PDF and Excel sheets now show aliases and use consolidated Nmap rows while still embedding raw JSON for full detail.
  • Reworked Graph Explorer backend/frontend to use canonical_hosts (store aliases in node properties), avoid separate www.* nodes when they are aliases, and added display modes (synthèse/DNS/services/technologies/risks/complète), filters, fit-to-screen/fullscreen controls and Ctrl+wheel zoom; also made the SVG responsive and reduced edge/node clutter.
  • Small UX fixes: Reporting Center HTML buttons open the generated HTML in a new tab (no broken download behavior), and reports keep raw data for programmatic consumers.

Testing

  • Ran Python syntax/bytecode checks with python -m compileall backend/app which succeeded (no syntax errors).
  • Validated frontend JavaScript syntax with node --check backend/app/static/app.js which succeeded.
  • Performed an automated canonicalization unit smoke test via PYTHONPATH=backend python - <<'PY' ... PY that exercised canonicalize_hosts on representative examples (merging www.follow... into follow..., keeping divergent www.vpn...) and asserted expected results; this test succeeded.

Codex Task

@Swapshadow
Swapshadow merged commit 935f3b4 into main Jun 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant