feat: report phishing domains to digitalocean in x-arf format - #84
Merged
Merged
Conversation
|
Warning Review limit reachedNext included review available in 23 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (32)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jaspermayone
force-pushed
the
jaspermayone/54-digitalocean-xarf-reports
branch
from
September 7, 2026 18:09
b52aa6d to
7b8cbe1
Compare
DigitalOcean processes its abuse mailbox with automated tooling that only accepts an X-ARF attachment, so a prose report is dropped. Closes #54. Contacts carry an accepts_xarf flag. When it is set, the mailer builds the SMTP envelope from https://github.com/abusix/xarf instead of the HTML report: multipart/report with a text/plain part, a message/feedback-report part that carries Feedback-Type: xarf, and the document as xarf.json. The HTML part is dropped on that path, because RFC 6522 puts the human readable part first. "Where applicable" needed hosting detection to work at all. Hosting was matched on nameserver patterns alone, but a phishing site on a droplet keeps its registrar's nameservers, so DigitalOcean was never matched. Domain lookups now resolve A and AAAA records, Report::AbuseContact.find_for_ip matches them against published CIDR ranges, and DigitalOceanRangeSyncJob keeps those ranges current from DigitalOcean's published CSV. Also corrects the X-ARF layer. Xarf::ReportGenerator claimed version 4.0.0 with snake_case fields, and no such schema exists: abusix publishes 1, 2 and 3, all PascalCase. The generator, parser and category mapper now speak schema 3, and Xarf::EmailReportBuilder is folded into the generator so the admin UI and the wire share one representation rather than two that disagree. Output was checked against schemas/3/phishing.schema.json. Claude-Session: https://claude.ai/code/session_01UJuHu239wz4x62pLNt651V
The public tool landed on main while this branch was open, and the two disagree about the wire format. The tool asserted an "xarf_version" of "4.0.0", which appears in no published schema. This branch moves the generator to schema 3 of https://github.com/abusix/xarf, so the tool emitted schema 3 while its tests still expected the invented shape. Update the tool's tests to the schema 3 keys, and correct the "XARF v4" labels in the views, the docs page and two comments. Those labels named a version the payload does not carry. The parser test that refuses the v4 shape stays as it is. Refusing it is the point. Claude-Session: https://claude.ai/code/session_018gopCvu5KgRjeWBM4Zw6pm
jaspermayone
force-pushed
the
jaspermayone/54-digitalocean-xarf-reports
branch
from
September 7, 2026 18:12
7b8cbe1 to
c655caa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #54.
DigitalOcean processes its abuse mailbox with automated tooling that only accepts an X-ARF attachment, so a prose report is dropped. This adds
abuse@digitalocean.comas a contact, makes reports to it X-ARF, and makes "where applicable" actually work.Sending X-ARF
Report::AbuseContactgains anaccepts_xarfflag. When it is set,Report::AbuseReportMailerbuilds the SMTP envelope that abusix/xarf specifies instead of the plain HTML report.graph TD A["multipart/report<br/>report-type=feedback-report"] --> B["text/plain<br/>the human readable report"] A --> C["message/feedback-report<br/>Feedback-Type: xarf"] A --> D["application/json<br/>xarf.json"] C -.->|"an ARF parser stops here"| E["ARF receiver"] C -.->|"an X-ARF parser reads on"| DThe HTML part is dropped on that path. RFC 6522 puts the human readable part first, and a lone
text/plainkeeps the structure identical to the published example. Contacts without the flag are untouched.Finding the sites DigitalOcean hosts
This was the real gap. Hosting was matched on nameserver patterns alone, but a phishing site on a droplet keeps its registrar's nameservers, so DigitalOcean was never matched.
graph LR A[phishing domain] --> B[Report::DomainLookupService] B --> C[RDAP / WHOIS<br/>registrar] B --> D[A and AAAA records] C --> E[find_for_registrar] D --> F[find_for_ip<br/>matches published CIDR ranges] E --> G[Report::Case submissions] F --> G H[DigitalOceanRangeSyncJob<br/>weekly] --> I[(ip_ranges)] I --> FReport::DomainLookupServiceresolves A and AAAA records and stores them. The case report PDF already had a slot for A records that nothing filled.Report::AbuseContact.find_for_ipmatches those addresses against published CIDR ranges.match_contacts!falls back to it when no nameserver pattern hits.DigitalOceanRangeSyncJobrefreshes the ~1200 ranges weekly from DigitalOcean's published CSV. A stale list silently stops reports, so it is not seeded by hand.Both address families are resolved: DigitalOcean publishes IPv6 allocations too, and an IPv6-only host would otherwise never be matched.
Correcting the X-ARF version
Xarf::ReportGeneratorclaimedxarf_version: "4.0.0"with snake_case fields. No such schema exists. abusix publishes schemas 1, 2 and 3, all PascalCase and nested underReporterInfoandReport. The old output would not have validated anywhere.The generator, parser and category mapper now speak schema 3.
Xarf::EmailReportBuilderis folded intoXarf::ReportGenerator, so the admin UI and the wire share one representation rather than two that disagree.The category mapper changes shape with it: the invented 7 categories and 40-odd snake_case types are replaced by the real taxonomy, which is a
ReportClassofContent,ActivityorVulnerabilityplus oneReportType.Schema 3 has no type for an unconfirmed site, so
suspiciousis reported asPhishingtoo.ReporterSeverityand the notes carry the confidence, so the receiving desk still sees that the finding is not confirmed.Verification
Generated documents were validated against
schemas/3/phishing.schema.jsonfrom the abusix repo with ajv: the domain entry point, the submission entry point, and the IPv6-only case all pass.Tests cover the MIME structure part by part, the document fields, the taxonomy, parser validation, IP matching including IPv6 and malformed input, the range sync, and the hosting fallback. 446 tests pass; RuboCop, Brakeman and bundler-audit are clean.
Note for review
Cached domain lookups written before this change have no A or AAAA records until their 24-hour TTL expires, so hosting matching by address starts working as the cache turns over.
https://claude.ai/code/session_01UJuHu239wz4x62pLNt651V