Openconceptlab/ocl_issues#2505 | Fix US/British spelling mismatch in semantic concept search

[snyaggarwal]

Refs OpenConceptLab/ocl_issues#2505 (Problem B only — Problem A still open)
Refs OpenConceptLab/ocl_issues#2511 (Lexical Variant Dictionary infrastructure)

Heads up — approach changed mid-flight

Sunny — apologies for the redirect. After deeper review of #2505 with @paynejd, we landed on a substantially different architecture and felt it was faster to push the new approach directly than to iterate via comments. Your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept — only the variant source changed from inline regex to a dictionary-driven lookup. Happy to walk through the diff on a call if useful.

Three reasons for the redirect:

1. False positives in the original regex list. hem/haem matches themselves, anthem, hemisphere, hemp, hemlock, remember — hard to fix defensively in regex without enumerating exclusions for every English word.
2. Reusability. The fix should be available beyond ConceptFuzzySearch. Standard /concepts/?q=… has the same gap, and upcoming abbreviation-expansion work needs the same primitive. A dictionary-as-Source approach makes that reuse trivial.
3. Eat our own dog food. Storing dictionaries as OCL Sources gives versioning, release management, locale support, editability through the OCL UI, and discoverability for free — and aligns with how UMLS / SNOMED / OBO model lexical variants.

Tradeoff worth flagging

This approach puts pressure on oclapi2 to do caching well. The MVP scale is trivial (~40 entries, in-memory dict), but as dictionary types and sizes grow — especially abbreviation dictionaries (UMLS LRABR is ~250k rows) and once variants are enabled by default on high-QPS endpoints — the per-token DB lookup will need to graduate beyond the MVP's module-level cache. Likely Redis-backed token-indexed cache for >1k-entry dictionaries; ES-backed lookup for very large ones. Plan to revisit when (a) a dictionary exceeds ~5k entries, or (b) variants are enabled by default on a high-QPS endpoint. Captured in #2511.

What this PR does

• New helper: core/common/lexical_variants.py — get_lexical_variants(text, source_uri) and get_variant_terms(text, source_uri). Tokenizes input (lowercase + whitespace + ASCII punctuation strip), looks each token up against the dictionary's Names, returns sibling Names from the matching Concept. Module-level cache keyed on (source_uri, version). Returns [] and never raises if the dictionary is missing — graceful degradation.
• Seed data: core/common/data/lexical-variants-en.json — OCL bulk import file creating OCL/lexical-variants-en (Source + 43 vetted Concepts + Source Version v1.0). Each Concept has en-US and en-GB Names with name_type=Fully Specified. False-positive-prone substring entries dropped.
• Wired into $match: MetadataToConceptsListView reads variants from request body, passes the resolved URI (or None) to ConceptFuzzySearch.search(..., variants_repo=...). The kNN sub-query construction and rescore expansion are unchanged in shape; only the variant source changed.
• Empty-synonyms rescore crash fix: preserved from your initial commit.
• Removed: get_spelling_variant() from core/common/utils.py.

API: variants request body field

Lexical variant expansion is OFF by default. Clients opt in. Same shape will apply to standard concept search when that wiring lands as a follow-up.

Value	Behavior
absent / null / false / "false" / "0"	disabled (default)
true / "true" / "1"	enabled, default dictionary (OCL/lexical-variants-en)
URI string (e.g. "/orgs/myorg/sources/my-dictionary/")	enabled, that dictionary

Tests

core/common/tests.py LexicalVariantsTest covers:

• Tokenization
• Multi-token expansion
• Cache lifecycle (per-source-version) + invalidation
• Missing-source graceful degradation
• Regression: themselves, anthem, hemisphere, hemp, hemlock, remember return no variants

Deployment

The dictionary Source has already been seeded on prod (OCL/lexical-variants-en v1.0, 43 concepts, public read) so the wiring will work as soon as this PR merges and deploys. For dev / qa / staging environments, run:

ocl --server ocl-dev import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-qa import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-staging import file --wait core/common/data/lexical-variants-en.json

Verify with: ocl repo get OCL lexical-variants-en (should show 43 concepts, source_type: "Lexical Variants", extras.dictionary_kind: "lexical_variant", version v1.0 released).

The helper degrades gracefully (returns [], never raises) if the Source doesn't exist yet — so a deploy gap on any environment won't break search.

Out of scope (deferred follow-ups)

• ?variants=... on standard concept search (ConceptListView) — touches CustomESSearch / get_raw_search_string layer, smaller blast radius as a separate PR
• $lexical-variants HTTP operation
• Abbreviation dictionary type (Mapping-based)
• Multi-dictionary composition
• Problem A from #2505 (CIEL bridge exact-match scoring via apply_score() highlight) — entirely separate work; this PR no longer auto-closes that part of the ticket

Architecture / design references

• Full design proposal: Lexical Variant Dictionaries — UMLS / SNOMED / OBO precedents, dictionary-type modeling decisions, tokenization-first reasoning, multi-dictionary roadmap
• Infrastructure tracking ticket: #2511 — Phase 2/3 work (standard search wiring, $lexical-variants operation, abbreviation dictionary type, caching strategy, etc.)

[paynejd]

@snyaggarwal — heads up, I pushed two commits on top of yours that redirect this PR to a dictionary-driven approach (full rationale in the updated description above + the proposal doc).

TL;DR — your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept. The variant source changed from inline regex to a dictionary lookup against ocl/lexical-variants-en (a new OCL Source seeded by bulk import). Variant expansion is now off by default; clients opt in via variants: true (or a custom dictionary URI) in the request body. We'll need follow up work to utilize this in Mapper (expose in project config?).

Apologies for the in-flight redirect — the false-positive cases in the regex list (themselves, anthem, hemisphere, hemp, etc. all matching hem/haem) plus the realization that the same primitive will be needed for upcoming synonym/abbreviation/keyword-expansion evaluation that we're starting pushed me to the dictionary-as-Source approach. Phase 2/3 work tracked in #2511.

Happy to walk through the diff or rework anything that doesn't sit right.

[snyaggarwal]

Two missing things:

1. Project config to enable it ("use_lexical_variants") and for core users only
2. Use Redis caching with TTL

@snyaggarwal to take on this

[paynejd]

Reviewing the latest revision after the redis-cache refactor and use_lexical_variants field addition. Four things to address before merge:

1. LEXICAL_VARIANTS_CACHE_TIMEOUT env override is broken — os.environ.get returns a string when set, but django-redis's cache.set(timeout=...) expects an int. Suggestion inline.

2. use_lexical_variants MapProject flag is persistence-only. The field exists, the serializer surfaces it, but MetadataToConceptsListView.filter_queryset still only reads request.data.get('variants') — so the flag has no effect at $match time until oclmap is updated to read the project config and pass variants: true per request. That's consistent with how encoder_model flows (client passes it per call), so the field is correct as persistence; just call out the dependency in the PR body and file an oclmap follow-up ticket. Otherwise reviewers / Mapper users will reasonably assume flipping the flag does something on its own.

3. Fixture path changed. Sunny moved core/common/data/lexical-variants-en.json → core/lookup_fixtures/lexical-variants-en.json, but (a) import_lookup_values.py has a hardcoded list and does not auto-load this file, so the lookup_fixtures/ placement falsely implies auto-load, and (b) the PR body's deploy commands still reference the old core/common/data/... path — anyone running them on dev/qa/staging gets a 404. Either move the file back, or wire it into import_lookup_values and update the PR body.

4. Rebase needed. PR is CONFLICTING (8 commits behind master). Conflicts are in core/map_projects/{models,serializers,tests/tests}.py from master's prompt_output_locale config addition (c883bd0). Should be a clean resolve — just append use_lexical_variants after prompt_output_locale in each CONFIGURATION_FIELDS / serializer field list / prepare_object loop. Verify migration 0036_mapproject_use_lexical_variants.py doesn't collide with master's migration numbering after rebase.

The redis-cache refactor itself looks good — keying on (source.uri, version) cleanly invalidates released-version edits. HEAD edits will be stale up to the TTL (4 days default), which is fine for a 43-entry vetted dictionary; worth a one-line comment so future maintainers know. The is_core_user gate matches the encoder_model precedent.

[paynejd]

Env override returns a string but cache.set(timeout=...) wants an int. Bug if anyone sets LEXICAL_VARIANTS_CACHE_TIMEOUT in env.

Suggested change

	LEXICAL_VARIANTS_CACHE_TIMEOUT = os.environ.get('LEXICAL_VARIANTS_CACHE_TIMEOUT', 60 * 60 * 24 * 4)
	LEXICAL_VARIANTS_CACHE_TIMEOUT = int(os.environ.get('LEXICAL_VARIANTS_CACHE_TIMEOUT', 60 * 60 * 24 * 4))

[paynejd]

Two small notes:

1. Cache key includes version, so released versions invalidate cleanly when bumped. But for HEAD edits the key stays HEAD and stale data persists up to CACHE_TIMEOUT (4 days default). For a vetted 43-entry dictionary that's fine — but a future maintainer who edits the dictionary in-place will be surprised. Worth a one-line note here. Optionally, hook invalidate_cache(source.uri) into Source / SourceVersion save() post-save so HEAD edits flush.

2. CACHE_TIMEOUT = settings.LEXICAL_VARIANTS_CACHE_TIMEOUT is evaluated at class-definition time. Fine in practice (env reads on process boot), but if anyone ever wants @override_settings to take effect in tests, this won't pick it up — they'd have to use cls.CACHE_TIMEOUT via a classmethod instead. Not worth changing unless tests need it.

[paynejd]

Worth a one-line debug log when variants is supplied but dropped because the caller isn't core — silent drops will confuse anyone reading API docs and trying it from a non-core token (worse results with no explanation).

Also — this is where the new MapProject.use_lexical_variants flag would need to feed in if we wanted server-side default behavior (rather than relying on oclmap to pass variants: true per request). Current design pushes that responsibility to the client, which is consistent with encoder_model. Just calling out: this PR makes the flag visible via the API but its only effect right now is that oclmap can persist user intent — it won't change $match behavior until oclmap reads the project config and passes variants. Recommend an explicit follow-up ticket and a line in the PR body so reviewers don't assume merging this enables variants in the Mapper.

[paynejd]

Correcting myself — I missed that Sunny opened the companion oclmap PR (OpenConceptLab/oclmap#20) earlier today, which does exactly the wiring I was asking about: reads use_lexical_variants from project config, renders a core-user-gated checkbox in AdvancedSettings, persists via FormData on save, and sends variants: useLexicalVariants in the $match request body. So the flag is not dormant — it's a sibling-PR situation, not a missing-wiring situation.

Two PRs land safely in either order: if oclapi2 lands first, oclmap master doesn't yet send variants so default-disabled holds; if oclmap lands first, oclapi2 master ignores the unknown variants field. No race risk.

Retracting the "follow-up ticket" ask. Still worth a one-liner in this PR's body cross-linking oclmap#20 (and vice versa) so reviewers see the pair. Debug-log suggestion for silently-dropped non-core variants still applies.

[paynejd]

The rename from core/common/data/ → core/lookup_fixtures/ is misleading: core/common/management/commands/import_lookup_values.py has a hardcoded list of fixture files and does not pick this one up — so placing it here implies an auto-load that doesn't happen. The file is also a full OCL bulk-import payload (Source line + Concept lines + version line), not the JSONL Concept-only format the other fixtures here use, so wiring it into import_lookup_values isn't a one-line change either.

Two options:

• Move back to core/common/data/lexical-variants-en.json (matches the bulk-import-via-CLI deploy path the PR body describes), or
• Keep here and add a real loader path in import_lookup_values for bulk-format fixtures.

Either way, the PR body's deploy commands still reference the old path — update those so dev/qa/staging deploys don't 404.