Skip to content

perf: look up disposable domains without building a Set of them - #326

Open
Spone wants to merge 1 commit into
micke:mainfrom
Spone:sorted-disposable-domains
Open

Spone wants to merge 1 commit into
micke:mainfrom
Spone:sorted-disposable-domains

Conversation

@Spone

@Spone Spone commented Oct 12, 2026 •

Copy link
Copy Markdown

Problem

ValidEmail2.disposable_emails loads the ~180,000 disposable domains into a Set. Each domain becomes its own String plus a hash entry, so the first email checked grows the process by about 35 MB.

Change

The domains file is sorted and has no duplicates: pull_mailchecker_emails.rb writes it with .uniq.sort. So it can be searched without building a Set:

  • ValidEmail2::SortedDomainList keeps the file as a single binary String (about 2.6 MB) and answers include? by bisection over its lines. Each step reads one line around the middle offset, so there's no index to build.

  • ValidEmail2.disposable_domains is what Address#disposable_domain? and #disposable_mx_server? now use:

    • by default, the SortedDomainList;
    • once disposable_emails has been loaded, that same Set.

    Code that adds domains to the Set or removes them (as one of the specs does) keeps working unchanged.

  • disposable_emails still returns the same mutable Set, built on demand. Nothing changes for code calling it, except that it now pays the 14 MB only when it does.

Measured

Against config/disposable_email_domains.txt (178,700 domains). Memory is the process's RSS growth after loading and one lookup, each case in a fresh process:

Before (Set) After (SortedDomainList)
Memory, Ruby 3.3.7 +33.8 MB +2.6 MB
Memory, Ruby 4.0.7 +36.2 MB +2.6 MB
Load time (Ruby 3.3.7) 80 ms 1.1 ms
Lookup (Ruby 3.3.7) 0.05 µs 4.2 µs

A lookup is slower than a Set's but stays well under the existing perform_under(0.0001) benchmark (100 µs), and a few microseconds per validated email is negligible next to the memory saved in every process.

Specs

  • spec/sorted_domain_list_spec.rb:
    • the file is sorted without duplicates, so a future generator change breaking the order fails here;
    • every domain of the file is found;
    • domains not in it aren't.
  • spec/valid_email2_spec.rb:
    • a disposable domain is detected without loading the Set;
    • once the Set is loaded, domains added to it count.
  • spec/benchmark_spec.rb: a lookup benchmark without the Set.

bundle exec rake: 145 examples, 0 failures. RuboCop reports the same 2 offenses as on main (Metrics/ClassLength on Address, Metrics/BlockLength on the main spec).

ValidEmail2.disposable_emails loads the ~180,000 disposable domains into
a Set: one String object per domain plus its hash entry, about 35 MB of
memory once the first email is checked. The file is sorted (see
pull_mailchecker_emails.rb), so lookups now go through
ValidEmail2.disposable_domains, a SortedDomainList keeping the file as a
single String (2.6 MB) searched by bisection.

disposable_emails still returns the same Set, built on demand. Once it
is loaded, disposable_domains returns it, so code adding domains to it
or removing them keeps working.
@Spone
Spone force-pushed the sorted-disposable-domains branch from 2c6cef4 to 9faaf2c Compare October 12, 2026 14:37
@Spone Spone changed the title perf: look up disposable domains without building a 14 MB Set perf: look up disposable domains without building a Set of them Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant