feat(search): migrate deed search index to OpenSearch client - #149
Draft
mtigas wants to merge 6 commits into
Draft
feat(search): migrate deed search index to OpenSearch client#149mtigas wants to merge 6 commits into
mtigas wants to merge 6 commits into
Conversation
Swap django-elasticsearch-dsl / elasticsearch-py (8.x) for django-opensearch-dsl 0.8.0 + opensearch-py (<3) so the DeedPage search index can target AWS OpenSearch Service. The elasticsearch-py 8.x client enforces an X-Elastic-Product check that raises UnsupportedProductError against OpenSearch, so the prior stack could not connect to a domain at all. - documents.py / views.py: import from django_opensearch_dsl / opensearchpy (the Q DSL and all field types incl. NestedField map 1:1; query logic unchanged). - settings: ELASTICSEARCH_DSL -> OPENSEARCH_DSL; basic_auth -> http_auth with username via OPENSEARCH_USERNAME (default "admin"). Do not reference the abstract BaseSignalProcessor; gate no-op indexing via OPENSEARCH_DSL_AUTOSYNC=False. - env vars: ELASTICSEARCH_URL/PASSWORD -> OPENSEARCH_URL/USERNAME/PASSWORD. - Pipfile/requirements: pin opensearch-py <3 to avoid the grpcio/protobuf pull from the 3.x line. Legacy Haystack/Solr search (DeedSearchView at /deed_search/) is untouched. Validated: manage.py check is clean; offline query-builder exercise (bool / multi_match / wildcard / nested / match_all) builds under opensearch-py. Follow-ups: management command changes from `search_index --rebuild` to `opensearch index create` + `opensearch document index`; regenerate requirements.txt from Pipfile (pipenv lock) before image build; update .env.example and docs/modules/installation.rst. Co-Authored-By: Claude <noreply@anthropic.com>
Update installation.rst steps 7-10 for the OpenSearch migration: - local dev runs an opensearchproject/opensearch:2 container (security plugin disabled for plain-HTTP local use) instead of Elasticsearch start-local. - env vars ELASTICSEARCH_* -> OPENSEARCH_*; note OPENSEARCH_PASSWORD gates real-time indexing on save. - seed commands change from `search_index --rebuild` to `opensearch index create` + `opensearch document index`. requirements.txt: pin opensearch-py==2.8.0 and add its only new transitive dependency, events==0.5 (urllib3/requests/certifi/python-dateutil are already satisfied by existing pins). Other pins are intentionally left untouched to avoid unrelated churn; run `pipenv lock` to fully formalize. Co-Authored-By: Claude <noreply@anthropic.com>
django-opensearch-dsl's get_indexing_queryset paginates with qs[i:i+n]
(SQL OFFSET), O(offset) per chunk -> quadratic over the ~11.7M-row
deed_deedpage table; throughput collapsed from ~30k to ~3.5k docs/min
partway through a full index. Its Document.get_queryset also bypasses the
Django.get_queryset optimization (plain model.objects.all()), so
prepare_workflow / prepare_matched_terms ran N+1 queries per document.
Override DeedPageDocument.get_indexing_queryset to seek by primary key
(pk > last_pk, O(chunk) via the PK index) over a select_related('workflow')
+ prefetch_related('matched_terms') queryset. Honors --filter/--exclude;
falls back to the library default for --count.
Validated locally: pulling 1000 rows deep in the table + accessing related
fields runs in 0.06s with 4 SQL queries (was N+1 ~2000), no OFFSET emitted,
pk-seek confirmed.
Co-Authored-By: Claude <noreply@anthropic.com>
A single bulk request to the OpenSearch domain over the VPC tunnel exceeded opensearch-py's 10s default read timeout and aborted a full index run (ConnectionTimeout, no retry). Raise the connection timeout to 60s and enable retry_on_timeout (max_retries=5) so a transient slow batch doesn't kill a long bulk indexing job. Co-Authored-By: Claude <noreply@anthropic.com>
OpenSearch caps hits.total at 10,000 by default, `track_total_hits=True` forces a "real" count
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Swap django-elasticsearch-dsl / elasticsearch-py (8.x) for django-opensearch-dsl 0.8.0 + opensearch-py (<3) so the DeedPage search index can target AWS OpenSearch Service. The elasticsearch-py 8.x client enforces an X-Elastic-Product check that raises UnsupportedProductError against OpenSearch, so the prior stack could not connect to a domain at all.