Dictionary pack mechanism + lazy loading (1.0.0) - #7
Merged
Merged
Conversation
SpellKit still bundles no dictionaries, and still knows the name of none. What it
gains is the MECHANISM: a pack lives in its own gem, registers itself on load, and
brings the tuning measured against its own data.
gem "spellkit"
gem "spellkit-general-medical"
SpellKit.enable_dictionary(:general_medical, lazy: true)
Registration rather than a built-in registry is the load-bearing choice. A registry
here would give SpellKit domain knowledge and would make every pack version bump
need a SpellKit release. Self-registration keeps the README's "SpellKit doesn't
bundle dictionaries" promise literally true and lets a pack ship new terms or new
tuning without this gem releasing anything.
LAZY LOADING, because a Rails initializer runs in EVERY process that boots the app
- web, db:migrate, rake, console - while usually only the web server searches. A
~200k-term pack at edit_distance 2 costs ~2.1 GB resident and ~5.5s to index, and
that OOM-killed a memory-constrained migrate init container in production. Eager
stays the default, so this is additive.
stats and healthcheck deliberately do NOT trigger a deferred load, reporting
deferred: true instead. Without that a liveness probe on a health endpoint would
build the whole index in exactly the process lazy loading protects, quietly
reintroducing the bug.
An unregistered pack raises at BOOT even under lazy - almost always a data gem
missing from the Gemfile, and the error says so - rather than failing on a user's
first search. Registration also stats the files, so a data gem whose payload did
not survive packaging fails while the stack still points at that gem.
The load is mutex-guarded, spec'd by counting builds under 8 concurrent first
calls: a threaded server can take two simultaneous first requests and would
otherwise build a multi-gigabyte index twice.
load_dictionary! forces a deferred load for a web-server boot hook, since lazy
MOVES the cost onto the first request rather than removing it.
README documents the initializer hazard with measured numbers, how to write a pack
gem, and that edit_distance dominates footprint (484 MB at 1 vs 2.1 GB at 2).
211 examples, 0 failures. Lint left cleaner than found (28 offenses -> 17); the
remainder is pre-existing in files this branch does not touch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CiDy8MvBncxzGnnmvDJMtL
1.0.0 freezes the public API, so this is the last moment to decide what is part of it. pack_load_options existed only so LazyChecker could resolve a pack name; it was public purely as a side effect of that collaboration, not because a caller wants it. LazyChecker now resolves through the already-public Packs.fetch(...).load_options and the helper is private. Public pack API is therefore exactly: enable_dictionary, dictionary_checker, load_dictionary!, dictionary_loaded?, dictionary_pack, and Packs.register. The thread-safety spec now counts Checker#load! calls rather than the helper. load! IS the expensive operation, so the count cannot drift if the resolution path is refactored again - which is exactly what just happened to it. 211 examples, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiDy8MvBncxzGnnmvDJMtL
cpetersen
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a pack mechanism to SpellKit. It still bundles no dictionaries and knows the name of none — a pack lives in its own gem and registers itself on load.
Why registration, not a registry
A built-in registry would give SpellKit domain knowledge and make every pack version bump require a SpellKit release. Self-registration keeps the README's "SpellKit doesn't bundle dictionaries" promise literally true, and lets a pack ship new terms or new tuning without this gem releasing anything.
Why lazy loading
A Rails initializer runs in every process that boots the app — web,
db:migrate,rake,console— while usually only the web server searches. A ~200k-term pack atedit_distance: 2costs ~2.1 GB resident and ~5.5s to index. That OOM-killed a memory-constrained migrate init container in production.Eager remains the default, so this is purely additive.
enable_dictionary(..., lazy: true)correct()Details worth reviewing
stats/healthcheckdo not trigger a deferred load — a liveness probe must not build the index in the process lazy protects.load_dictionary!warms it explicitly, foron_worker_boot— lazy moves the cost, it doesn't remove it.Verification
211 examples, 0 failures. Lint left cleaner than found (28 → 17 offenses); the rest is pre-existing in untouched files.