Skip to content

feat: report phishing domains to digitalocean in x-arf format - #84

Merged
jaspermayone merged 2 commits into
mainfrom
jaspermayone/54-digitalocean-xarf-reports
Sep 7, 2026
Merged

jaspermayone merged 2 commits into
mainfrom
jaspermayone/54-digitalocean-xarf-reports

Conversation

@jaspermayone

Copy link
Copy Markdown
Member

Closes #54.

DigitalOcean processes its abuse mailbox with automated tooling that only accepts an X-ARF attachment, so a prose report is dropped. This adds abuse@digitalocean.com as a contact, makes reports to it X-ARF, and makes "where applicable" actually work.

Sending X-ARF

Report::AbuseContact gains an accepts_xarf flag. When it is set, Report::AbuseReportMailer builds the SMTP envelope that abusix/xarf specifies instead of the plain HTML report.

graph TD
    A["multipart/report<br/>report-type=feedback-report"] --> B["text/plain<br/>the human readable report"]
    A --> C["message/feedback-report<br/>Feedback-Type: xarf"]
    A --> D["application/json<br/>xarf.json"]
    C -.->|"an ARF parser stops here"| E["ARF receiver"]
    C -.->|"an X-ARF parser reads on"| D
Loading

The HTML part is dropped on that path. RFC 6522 puts the human readable part first, and a lone text/plain keeps the structure identical to the published example. Contacts without the flag are untouched.

Finding the sites DigitalOcean hosts

This was the real gap. Hosting was matched on nameserver patterns alone, but a phishing site on a droplet keeps its registrar's nameservers, so DigitalOcean was never matched.

graph LR
    A[phishing domain] --> B[Report::DomainLookupService]
    B --> C[RDAP / WHOIS<br/>registrar]
    B --> D[A and AAAA records]
    C --> E[find_for_registrar]
    D --> F[find_for_ip<br/>matches published CIDR ranges]
    E --> G[Report::Case submissions]
    F --> G
    H[DigitalOceanRangeSyncJob<br/>weekly] --> I[(ip_ranges)]
    I --> F
Loading
  • Report::DomainLookupService resolves A and AAAA records and stores them. The case report PDF already had a slot for A records that nothing filled.
  • Report::AbuseContact.find_for_ip matches those addresses against published CIDR ranges. match_contacts! falls back to it when no nameserver pattern hits.
  • DigitalOceanRangeSyncJob refreshes the ~1200 ranges weekly from DigitalOcean's published CSV. A stale list silently stops reports, so it is not seeded by hand.

Both address families are resolved: DigitalOcean publishes IPv6 allocations too, and an IPv6-only host would otherwise never be matched.

Correcting the X-ARF version

Xarf::ReportGenerator claimed xarf_version: "4.0.0" with snake_case fields. No such schema exists. abusix publishes schemas 1, 2 and 3, all PascalCase and nested under ReporterInfo and Report. The old output would not have validated anywhere.

The generator, parser and category mapper now speak schema 3. Xarf::EmailReportBuilder is folded into Xarf::ReportGenerator, so the admin UI and the wire share one representation rather than two that disagree.

The category mapper changes shape with it: the invented 7 categories and 40-odd snake_case types are replaced by the real taxonomy, which is a ReportClass of Content, Activity or Vulnerability plus one ReportType.

Schema 3 has no type for an unconfirmed site, so suspicious is reported as Phishing too. ReporterSeverity and the notes carry the confidence, so the receiving desk still sees that the finding is not confirmed.

Verification

Generated documents were validated against schemas/3/phishing.schema.json from the abusix repo with ajv: the domain entry point, the submission entry point, and the IPv6-only case all pass.

Tests cover the MIME structure part by part, the document fields, the taxonomy, parser validation, IP matching including IPv6 and malformed input, the range sync, and the hosting fallback. 446 tests pass; RuboCop, Brakeman and bundler-audit are clean.

Note for review

Cached domain lookups written before this change have no A or AAAA records until their 24-hour TTL expires, so hosting matching by address starts working as the cache turns over.

https://claude.ai/code/session_01UJuHu239wz4x62pLNt651V

@jaspermayone jaspermayone added the migration Includes database migrations. label Sep 7, 2026
@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: b46a84a7-4504-42dd-85da-bbbda6aba5c7

📥 Commits

Reviewing files that changed from the base of the PR and between 7adf60e and c655caa.

📒 Files selected for processing (32)
  • app/controllers/admin/report/abuse_contacts_controller.rb
  • app/controllers/xarf_controller.rb
  • app/jobs/digital_ocean_range_sync_job.rb
  • app/mailers/report/abuse_report_mailer.rb
  • app/models/report/abuse_contact.rb
  • app/models/report/domain_lookup.rb
  • app/services/report/digital_ocean_range_service.rb
  • app/services/report/domain_lookup_service.rb
  • app/services/xarf/category_mapper.rb
  • app/services/xarf/public_report_service.rb
  • app/services/xarf/report_generator.rb
  • app/services/xarf/report_parser.rb
  • app/views/admin/phish_domains/show.html.erb
  • app/views/admin/phish_urls/show.html.erb
  • app/views/admin/report/abuse_contacts/_form.html.erb
  • app/views/docs/index.html.erb
  • app/views/reports/case_report.html.erb
  • app/views/xarf/_report.html.erb
  • app/views/xarf/new.html.erb
  • config/recurring.yml
  • db/migrate/20260907120001_add_accepts_xarf_to_report_abuse_contacts.rb
  • db/migrate/20260907120002_add_aaaa_records_to_report_domain_lookups.rb
  • db/seeds/abuse_contacts.rb
  • test/integration/xarf_tool_test.rb
  • test/mailers/report/abuse_report_mailer_test.rb
  • test/models/report/abuse_contact_test.rb
  • test/models/report/domain_lookup_test.rb
  • test/services/report/digital_ocean_range_service_test.rb
  • test/services/xarf/category_mapper_test.rb
  • test/services/xarf/report_generator_test.rb
  • test/services/xarf/report_parser_test.rb
  • test/test_helper.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

DigitalOcean processes its abuse mailbox with automated tooling that only
accepts an X-ARF attachment, so a prose report is dropped. Closes #54.

Contacts carry an accepts_xarf flag. When it is set, the mailer builds the
SMTP envelope from https://github.com/abusix/xarf instead of the HTML
report: multipart/report with a text/plain part, a message/feedback-report
part that carries Feedback-Type: xarf, and the document as xarf.json. The
HTML part is dropped on that path, because RFC 6522 puts the human readable
part first.

"Where applicable" needed hosting detection to work at all. Hosting was
matched on nameserver patterns alone, but a phishing site on a droplet keeps
its registrar's nameservers, so DigitalOcean was never matched. Domain
lookups now resolve A and AAAA records, Report::AbuseContact.find_for_ip
matches them against published CIDR ranges, and DigitalOceanRangeSyncJob
keeps those ranges current from DigitalOcean's published CSV.

Also corrects the X-ARF layer. Xarf::ReportGenerator claimed version 4.0.0
with snake_case fields, and no such schema exists: abusix publishes 1, 2 and
3, all PascalCase. The generator, parser and category mapper now speak
schema 3, and Xarf::EmailReportBuilder is folded into the generator so the
admin UI and the wire share one representation rather than two that
disagree. Output was checked against schemas/3/phishing.schema.json.

Claude-Session: https://claude.ai/code/session_01UJuHu239wz4x62pLNt651V
The public tool landed on main while this branch was open, and the two
disagree about the wire format. The tool asserted an "xarf_version" of
"4.0.0", which appears in no published schema. This branch moves the
generator to schema 3 of https://github.com/abusix/xarf, so the tool
emitted schema 3 while its tests still expected the invented shape.

Update the tool's tests to the schema 3 keys, and correct the "XARF v4"
labels in the views, the docs page and two comments. Those labels named a
version the payload does not carry.

The parser test that refuses the v4 shape stays as it is. Refusing it is
the point.

Claude-Session: https://claude.ai/code/session_018gopCvu5KgRjeWBM4Zw6pm
@jaspermayone
jaspermayone force-pushed the jaspermayone/54-digitalocean-xarf-reports branch from 7b8cbe1 to c655caa Compare September 7, 2026 18:12
@jaspermayone
jaspermayone merged commit aca8758 into main Sep 7, 2026
4 checks passed
@jaspermayone
jaspermayone deleted the jaspermayone/54-digitalocean-xarf-reports branch September 7, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

migration Includes database migrations.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

send reports to Digital Ocean (where applicable)

1 participant