feat(cross-scan): pivot a finished scan into username and email scans, and rate each hit

[brunolm]

tl;dr

• A scan can now reach sites its own mode cannot. --cross-scan mines the finished results for usernames, profile links and email addresses, then scans each against the modules for its own kind. On one live authorised address it took attributable accounts from 40 to 65, over 16 sites the email pass could never reach.
• Following an address is what unlocks sites with no username namespace. A -u run gains 20–24% more distinct sites — Dropbox, Office365, Amazon, Facebook and 16 more that no handle sweep can reach, because there is nothing handle-shaped to look up.
• Most raw cross-scan hits are not the target, so every hit is rated. A swept handle only proves the handle is registered. In one run 78% of hits were unattributable and 8 were provably other people; confirmed / likely / candidate / conflicting is what keeps that honest.
• Following an address means scanning a third party's mailbox, so loud modules are dropped, not prompted for. 23 email modules mail whoever they are given, and the address came off somebody else's profile.
• A multi-target run shares one anchor pool, so it can mis-rate. Known limitation, reachable only through -ef / -uf / patterns with two different people.
• It multiplies request volume by roughly 5x at defaults, and far more if pushed. 153 checks became 834. One module went from a clean hit to erroring on every handle after repeated sweeps.
• Some sites give a linked account as a bare name, not a link, and those were being dropped. "twitter": "BrunoLM7" never matched the URL-based extractor, so 132 handles across the corpus were collected and thrown away. A key named after a platform now becomes a pivot directly.
• Five new flags, all opt-in and off by default, and -m / -c narrow the second pass exactly as they narrow the first.

---

A scan can now reach sites its own mode cannot

--cross-scan runs after the first pass and mines its metadata for three shapes:

Shape	Example	Becomes
A handle the site reports for the target	username: johndoe	username johndoe
A link on the profile	https://github.com/johndoe	username johndoe, site github
An address the profile publishes	emails: johndoe7@gmail.com	email target johndoe7@gmail.com

All four directions fall out of one mechanism:

Direction	Mines
-e → username	a handle the address's profile reports, or a link it carries
-u → username	the person's other handles, advertised on the profiles found
-u → email	an address published on a profile the username pass found
-e → email	a second address exposed by the first one's profiles

Links resolve to a (site, username) pair through a route table in core/pivots.py — hosts, path shapes (/in/{user}, /users/{id}/{user}, /@{user}) and {user}.host subdomains. A path that names a site page rather than a person (github.com/settings, youtube.com/channel/UC…) yields nothing.

Measured against a plain email scan of the same address:

	Email only	With --cross-scan
Checks performed	153	834
Email registrations	40	40
Attributable accounts (confirmed + likely)	40	65
Distinct sites	40	56

Both at stock settings — the cross-scan column is --cross-scan with no tuning. The email pass is identical in both, and is counted as such: one module was rate-limited during the email-only run and returned an error rather than its registration, so it is counted for both rather than being credited to the cross-scan.

Following an address is what unlocks sites with no username namespace

Two live runs at --cross-emails all --cross-depth 10 --cross-sweep 10, handles anonymised:

-u johndoe

pass 1   scan "johndoe" across 227 username modules
         └─ 81 accounts

round 1  4 handle pivots + 1 address pivot extracted
         ├─ handle  JohnDoe7               →  26 accounts,  5 new sites
         ├─ handle  JohnathanDoe           →   3 accounts,  0 new sites
         ├─ handle  johndoe_7              →   4 accounts,  0 new sites
         ├─ handle  codecraft              →   7 accounts,  1 new site
         └─ ADDRESS johndoe7@gmail.com     →  41 accounts, 21 new sites   ← new
                    from Gravatar (emails), rated likely

round 2  nothing new to follow → stop
         108 distinct sites

The address alone out-produced all four handle pivots combined: 21 new sites against 6.

The second run needed depth to get there at all — its own handle has no Gravatar, so the profile carrying the address is itself a round-1 discovery:

-u johndoe7

pass 1   scan "johndoe7" across 227 username modules
         └─ 32 accounts        (no Gravatar → no address yet)

round 1  └─ handle  johndoe                →  83 accounts, 57 new sites
                    ↑ this is what exposes the Gravatar profile

round 2  the Gravatar found in round 1 now yields an address
         ├─ handle  JohnathanDoe           →   3 accounts,  1 new site
         ├─ handle  johndoe_7              →   4 accounts,  0 new sites
         ├─ handle  codecraft              →   7 accounts,  1 new site
         └─ ADDRESS johndoe7@gmail.com     →  40 accounts, 20 new sites   ← new

round 3  nothing new to follow → stop
         111 distinct sites

At --cross-depth 1 that address never appears.

	-u johndoe	-u johndoe7
Sites reachable by handle alone	87	91
Sites with the address leg	108	111
Net new sites	+21 (24%)	+20 (22%)
Email hits a handle also found	20	20

Half the address's hits duplicate a site a handle already reached — still useful as a second, independent signal, but not new coverage. The other half are only reachable this way, and the same 20 in both runs:

Adobe, Amazon, Codecademy, Coursera, Dropbox, Espn, Eventbrite, Facebook,
Firefox, Instagram, Medium, Nytimes, Office365, Patreon, Rappi, Redtube,
Spotify, Vimeo, Wix, Zoho

None of these expose a public username namespace to sweep. An email check is the only route to them.

Most raw cross-scan hits are not the target, so every hit is rated

Sweeping a plausible handle returned five different people alongside the real owner. Every hit therefore carries extra.confidence:

Rating	Meaning
confirmed	a pivot named this exact site and handle
likely	metadata matches the confirmed profiles
candidate	the handle is registered; nothing ties it to the target
conflicting	metadata names someone else

likely and conflicting are decided against anchors — names, personal domains, e-mail addresses, profile URLs and confirmed accounts harvested from the confirmed hits. The strongest signal is a link to a confirmed account, matched on the resolved (site, handle) pair rather than URL text, so a renamed host or a different casing still lands:

twitter.com/JohnDoe2  and  x.com/johndoe2   ->  both resolve to ("x", "johndoe2")

Addresses are rated before they are scanned, on how independently they were reported, and the accounts they find inherit that rating — an account is only as well tied to the target as the address that led to it:

Rating	Earned by
confirmed	two or more sites published it in their own email field
likely	one site published it in an email field, or it sits on a domain the target links to
candidate	prose only, with nothing tying it back

conflicting is never used for an address: it carries no name to disagree with, and inferring a mismatch from the local part would mislabel every shared mailbox.

Two rating rules had to be removed during development because they were circular — build_anchors mines addresses out of the very profiles being rated, so an address on a confirmed profile was vouching for itself. Anchors.link_domains exists to fix the second one: only domains the target was seen to link to count, not domains inferred from harvested addresses.

Where the accounts come from, and how much of the cross-scan's yield survives rating:

	Email only	With --cross-scan
Email registrations (not rated)	40	40
confirmed	—	9
likely	—	16
candidate	—	81
conflicting	—	8
Accounts found	40	154

Following an address means scanning a third party's mailbox

LOUD_MODULES["user"] is empty; LOUD_MODULES["email"] has 23 entries that mail whoever they are given — password resets, verification links. In a first pass that address is the one the operator typed, so --allow-loud plus a per-module prompt is the right bar. In a cross-scan it came off somebody else's profile, and a second pass should not be making that call.

Those modules are therefore dropped from a cross-scan email pivot, not prompted for. --allow-loud puts them back for a caller who has already accepted the consequence.

_email_scope(...)                       -> 131 of 153 modules   (23 loud dropped)
_email_scope(..., allow_loud=True)      -> None  (all modules, the full-batch path)

This is stricter than the existing per-module prompt, and it is a deliberate behavioural difference between the two passes rather than an oversight.

An email field is not proof the address belongs to the account

--cross-emails verified keeps only addresses a site published in its own email field. That says the site published it, not that the site was right about whose it is.

PyPI fills extra["email"] from a package's author_email / maintainer_email, so it can name a co-maintainer or a mailing list:

Sampling three real PyPI accounts, two of the three addresses belonged to somebody else:

user A  ->  email: <a company that authored one of their packages>
user B  ->  email: <a co-maintainer of one of their packages>
user C  ->  email: <actually this account holder>

Keys that name the third party outright (author_email, maintainer_email) are read as text and excluded by the default, but a module that folds them into email defeats that. Fixing PyPI is a breaking change to that module's output and is deliberately left out of this diff.

A multi-target run shares one anchor pool, so it can mis-rate

-ef, -uf and pattern expansion produce several targets in one run, and the cross-scan mines them after the loop as a single pool. Confidence anchors are pooled with them, so one target's identity can vouch for another's handle collision:

pooled anchor names : ['alicesmith', 'bobjones']
a stranger on bob's handle, whose profile reads "Alice Smith"  ->  likely

That is a wrong verdict, not merely a missing one. It is scoped to multi-target runs of different people — a single target is unaffected, and so is a multi-target run of one person's own handles. A live -uf pass over four distinct identities did not trigger it: the pooled anchors never cross-matched, and every verdict equalled what per-target anchors would have produced.

Left as a known limitation rather than fixed here, to keep this diff to the feature. The fix is to key anchors per target instead of per run, which changes _apply_confidence's signature and is better reviewed on its own.

Request volume is the real operational risk

Cross-scan turns one pass into many: each swept username costs roughly a full -u scan, and each scanned address roughly a full -e scan. --cross-sweep caps that (default 3) across all rounds and both kinds, and 0 disables sweeping entirely.

Half the budget is offered to addresses, rounded down, and whatever one kind cannot use falls to the other:

Budget	Usernames available	Addresses available	Spent on
3	5	2	2 usernames, 1 address
3	0	2	2 addresses
3	5	0	3 usernames
1	2	2	1 username

A budget of 1 still goes to a username, which is what it did before addresses existed.

The failure mode this creates is repeat traffic per site. In a run with six sweeps, one module went from returning a confirmed hit to erroring on all six handles with a blocked-availability response — so a larger budget can lose you a hit.

Raising the budget also buys very little. Pushed to --cross-sweep 10 --cross-depth 10, the same address yielded 1,515 checks and one additional attributable account over the stock 834 — 82% more traffic for a single hit. Both deep runs above stopped on an exhausted link graph, not an exhausted budget: one used 5 of 10 targets, the other 4.

Ranking is by how well-vouched a handle is, not by how plausible it looks, so an opaque platform id arriving through a verified link can outrank a real handle and spend a sweep on a string no other site will ever hold. One such route (open.spotify.com/user/) is deliberately registered with no path pattern for that reason.

A username pass can pivot too

-u and -uf accept --cross-scan, and the pivot machinery is the same: a username run's profiles advertise the person's other handles, which is exactly what a pivot consumes.

[link] johndoe_2             tiktok     <- Gravatar (links)
[link] JohnDoe2              x          <- Twitch (twitter)
[link] JohnathanDoe          youtube    <- Linktree (showcased_links)

The scanned handle itself is seeded as already-swept, and an email pass seeds its own address as already-scanned. A pass is already a sweep of its own target and most sites report it straight back, so without the seed it would rank first and spend the budget repeating the scan that just finished.

Five new flags, all opt-in

Flag	Default	Purpose
--cross-scan	off	Enable the second pass
--cross-links {all,verified,none}	all	Which link classes may become username pivots
--cross-emails {all,verified,none}	verified	Which addresses may be scanned as emails
--cross-depth N	1	Rounds of link-following
--cross-sweep N	3	Targets swept against every module of their kind, all rounds; 0 disables sweeping

--cross-emails defaults tighter than --cross-links because the cost of being wrong is not symmetric: a stray username pivot wastes a request, a stray address puts a third party into the report.

The two share a vocabulary but not a meaning for none. --cross-links none still yields handle pivots, because a handle is not a link; --cross-emails none yields nothing, because every email class is an address. That also gives address pivoting a real off switch rather than the --cross-sweep 0 side effect.

-m and -c narrow the second pass exactly as they narrow the first — the sweep, the named checks, and now the email modules too. -m github resolves to user_scan/dev/github.py for the sweep and email_scan/dev/github.py for an address. Only a restriction naming neither leaves nothing to cross-scan.

-m gravatar   1 module first pass ->   1 swept   (was 227)
-c dev       44 modules first pass ->  44 swept   (was 227)
unrestricted 227 modules          -> 227 swept

Links are classed by how far the source platform vouches for them

handle (the site named the account itself), verified (the owner proved control of the far side) and link (free text the owner typed). The class is inherited, never independently confirmed: it comes from a key named verified_accounts / verified_links / connected_accounts, or from a (verified) suffix on the URL. Today exactly one module emits either signal, so --cross-links verified is narrower than its name suggests.

Addresses use the same ladder with two rungs — field (a site's own email key) and text (read out of prose) — so --cross-links verified and --cross-emails verified mean the same thing: nothing the owner merely typed.

A bare handle is a link too

Some sites give a linked account as just a name ("twitter": "BrunoLM7") instead of a link. The extractor only looked for values starting with http, so those names were collected and then quietly thrown away.

A key named after a platform now turns its handle into a pivot directly.

Module	Fields	Before	After
gaming/kick	twitter, instagram, youtube, tiktok, facebook	0	5
social/flickr	facebook, instagram, twitter, tumblr, pinterest	0	5
creator/devto	github, twitter	0	2
gaming/monkeytype	github, twitter	0	2
dev/github	twitter	0	1

132 of these were being dropped across the stored corpus.

No URL is built, because guessing one is unsafe — Kick stores BrunoLeonardoMichels for YouTube, and youtube.com/@BrunoLeonardoMichels 404s while /c/ works. Only the site and handle are needed, so the guess is skipped.

Two things are ignored on purpose: discord (sites put a server invite code there, not a username) and any key ending in _id (those are identifiers, not handles). URL values still work exactly as before.

Two things that look like addresses and are not

Replaying 3,158 stored Found results through the extractor caught both, and both would have been scanned:

Raw value	Parsed as	Why
https://www.tiktok.com/@jane.doe	//www.tiktok.com/@jane.doe	a URL is a legal dot-atom address — / is permitted in a local part
Mastodon: @johndoe@mastodon.social	johndoe@mastodon.social	fediverse handles are address-shaped but are not mailboxes

URLs are now blanked before the address scan (links already arrive as username pivots, so nothing is lost) and a match preceded by a second @ is rejected.

Addresses that reach nobody are dropped outright: role mailboxes (noreply@, postmaster@), RFC 2606 placeholders, reserved TLDs and GitHub's @users.noreply.github.com relay. hello@ and contact@ are kept — that is how a freelancer takes mail.

Deliberate omissions

• PyPI is not fixed. See above — breaking change to a module's output, left for its own PR.
• The root-domain fallback is kept despite known noise. A profile link to a bare domain yields the domain label as a username, which recovers a personal handle nothing else would — but on a company domain it names a different entity. It stays because it is classed link (so --cross-links verified drops it), resolves to no site (so it can only ever feed a sweep, never a named check), and the confidence scorer independently rated its collisions conflicting.
• Link shorteners are dead ends. t.co/… yields no pivot and the redirect is not followed, so whatever it points at stays invisible at any depth.
• Confidence ignores location. A country name fits millions of people and people move.
• Scoring runs after the pass finishes, because the anchors come from that same pass's confirmed hits. Ratings appear in the export and the closing summary, not on result lines as they stream past.

Testing, and what was not tested

Live against five identities in both e-mail and username mode, across depth 1/2/10 and sweep 0/1/2/3/4/5/10, all three --cross-links values, all three --cross-emails values, and a control run with the richest metadata source suppressed to prove pivoting still works from other sites.

The address leg specifically: 14 live runs covering both defaults and all/none, two deep runs at 10/10, an -e pass proving the target address is not rescanned, two nonexistent-handle controls (No usernames, links or addresses to pivot from), and three real accounts that yielded no address at all — two with no Gravatar, one with a Gravatar publishing none.

The extractor was also replayed over 3,158 stored Found results from 134 previous exports, which is what caught the two false positives above.

Also exercised end to end: -uf bulk, pattern expansion, -m and -c scoping on both halves, --no-nsfw, -t / -C overrides, and JSON, CSV and PDF export all carrying confidence and pivot_source.

Not tested:

• No confirmed address occurred live. It needs two sites publishing the same address in an email field; across every corpus and live run, only Gravatar ever published one, so every real address rated likely — and so did all 81 accounts they found. The confirmed path is covered only by constructed cases.
• Multi-address profiles are unproven live. Gravatar joins several addresses into one emails value and the extractor splits them correctly against that format and a synthetic value, but no live account in reach had more than one.
• Hackernews bio is the largest real prose source in the replayed corpus (11 of 13 text pivots) and the module returns ERROR live at the moment, so that path is verified only from stored results.
• The pooling limitation did not fire in practice. A live -uf pass over four distinct identities produced verdicts identical to per-target anchors.
• Run-to-run variance is dominated by site flakiness, not by this code — repeated identical runs differed by one to two accounts.

[kaifcodec]

@brunolm Looks clean and better! I will review it once it is finished. However let me know if you have nay questions about the feature.

[brunolm]

@kaifcodec it should be ready for review!

[kaifcodec]

@brunolm I will review it shortly, as it is a big addition!
Just a quick question, does it cross scans from email to username and vice versa.
What I mean is?
If it is run with -u and finds email(s) in extra from any result then will it follow the email and perform email scan on the extracted email/emails and vice-versa when run with -e flag?

[brunolm]

@kaifcodec No, currently it's:

Direction	Today
-e → username	✅
-u → username	✅
-u → e-mail	❌
-e → e-mail	❌

But I think it makes sense to support that, maybe only with emails from platforms where emails are known to be verified. I could try to implement it here or in a follow up PR. Let me know.

[kaifcodec]

Yeah, -u then extract email and run email scan is more reasonable and better because it will be more successful than -e to username because there is around only 3 to 4 sites that currently expose username from an email and still that is rare case like github, GRavatar and etsy.

But when we scan username there is very high chance it will email(s) from it.

So implementing this will be the better and most useful option among these, so I think maybe it will be better if you implement that in this PR as well, so you don't have to push the helper files in a new branch again.

Anyways in this type of cross scan, as you are already using confidence score I think it will be better if you prioritize the extracted emails from usernames as well before running a scan on extracted emails because in most of the cases we can get multiple emails from a single username (as same username in different sites can be different person) so it will be better to create a list of extracted emails and something like prompt user whether they want to run scan on those all emails or select between them and we will present those emails with confidence score beside them on the terminal based on the frequency and they can choose by interacting with terminal in which email(s) to run cross-scan on and another new flag maybe --auto where it will not prompt the user and will start the email scan on the values from the extracted email(s) list automatically.

I didn't fully review the logics yet I have just gone through the changes and file names with a surface level review, so some of my proposals can be overlapping with your already implemented changes here.

[brunolm]

@kaifcodec Implemented in this PR. The table now reads:

Direction	Today	Run it with
-e → username	✅	user-scanner -e johndoe@gmail.com --cross-scan
-u → username	✅	user-scanner -u johndoe --cross-scan
-u → e-mail	✅	user-scanner -u johndoe --cross-scan
-e → e-mail	✅	user-scanner -e johndoe@gmail.com --cross-scan

And emails are classified this way:

Rating	Earned by
confirmed	two or more sites published it in their own e-mail field
likely	one site published it in an e-mail field, or it sits on a domain the target links to
candidate	prose only, nothing tying it back

Args:

Flag	Default	What it does
--cross-scan	off	Runs the second pass. Mines the finished results for usernames, links and addresses, then scans each against the modules for its own kind. Everything below is inert without it.
--cross-links {all,verified,none}	all	Trust floor for username pivots. all = every class · verified = drops links the owner typed · none = only handles the site reported itself
--cross-emails {all,verified,none}	verified	Trust floor for address pivots. all = includes addresses scraped from prose · verified = only ones a site published in its own email field · none = no address is scanned
--cross-depth N	1	Rounds. Each round pivots off what the previous one found, so a target reachable only through a chain is still reached. Stops early when a round finds nothing new.
--cross-sweep N	3	Budget of targets given the every-module treatment, across all rounds and both kinds. 0 leaves only the sites a pivot named.

[kaifcodec]

Okay @brunolm .
I am currently tied up with some work, I will start reviewing and testing the edge cases on this after around 9 to 10 hours.

[kaifcodec]

@brunolm Most of the features in this PR look solid and work fine!

However, I noticed an issue with the pdf export method when using flags like --cross-scan. When pivoting results to generate the dossier report, it treats the completed scan as a pure USERNAME scan (as if started with -u). As a result, it doesn't include the email_scan details in the main table columns, though it still places those email results under the EXTRACTED DEEP INTELLIGENCE header (see attached screenshot (the Npmjs there is the result of username scan after that the email_scan issue starts)). Also I recommend you to run user-scanner yourself with -u ... -o report.pdf in the --cross-scan mode so you will see the issue clearly.

Note that JSON and CSV exports handle this correctly.

If you want to tweak the PDF layout/structure yourself, let me know! Otherwise, I can push the fixes directly using the existing PDF style since I wrote the pdf_generator.py module.

[brunolm]

Feel free to push fixes for this.

If you need me to change anything I'll be available later.

[kaifcodec]

@brunolm Pushed the fix.

I'll keep this PR open a bit longer while I test a couple more edge cases. Really solid work here though, super impressive PR!

[kaifcodec]

@brunolm What I reviewed so far it looks good to me. However if any edge cases I missed that can only be covered by user's issue opening. So, I am MERGING IT!