Patch delays - #142
Open
YaphetKG wants to merge 19 commits into
Open
Conversation
YaphetKG
requested review from
vladimir2217 and
yskale
and removed request for
vladimir2217
June 16, 2026 20:19
vladimir2217
approved these changes
Jun 17, 2026
Dug's data classes moved into the dug_data_model library, and the module they used to live in (dug.core.parsers._base) no longer imports. jsonpickle does not raise on that -- it returns the raw dict -- so artifacts written by an older dug index as dicts and fail with "'dict' object has no attribute 'id'". With incremental ingestion those artifacts are only rewritten when their source changes, so they can sit in lakefs across dug upgrades indefinitely. scripts/migrate_pickled_classes.py restamps them in place instead of re-annotating: --scan reports which stored classes no longer import and whether any stored field is missing from the current model, --fix decodes through a legacy module alias, fills in fields added since, and re-encodes. Also import random and time in pipelines/base.py -- init_annotator's retry path used both, so a transient annotator failure raised NameError instead of retrying.
README described a codebase that no longer exists: a KGX fork install, a `bin/roger all` entrypoint, `dags/roger/config/`, Python 3.7, and ~450 lines of pasted CLI output. Replaced with an orientation doc -- what Roger produces, the two workflows and how they differ, an annotated repo tour with a reading order, quickstart, how to run tests without host dependencies, config, the concepts that bite early (lakefs as source of truth, incremental runs, ES as a derived index, jsonpickle-encoded artifacts), and a troubleshooting table. Kept the KGX merge/schema type-conflict rules, the bulk-loader CSV grouping, and the k8s/Helm prerequisites. Dropped the stale run log and screenshots. CLAUDE.md covers what contributors need but the README should not carry: the incremental state machine and why the success callback must not re-resolve refs, deletion propagation, lakefs task wiring flags, the DAG shapes, and the dug_data_model migration runbook.
trying out seperating indexing jobs and adding removal steps for new …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Incremental (delta) lakefs ingestion for the annotate/index DAG, plus fixes uncovered while testing it.
Changes
find_sibling_files()lists each changed file's lakefs directory and pulls anyGapExchange_*sibling not in the diff. dbGaP data dicts depend on that sibling for study name/description, so a delta carrying only the data dict would otherwise fail to parse.build_indexer_obj()/build_search_obj()no longer pass index-name lists; dug now reads index names from config (to_dug_conf()already supplies concepts/variables/studies/sections/kg).Tests
tests/unit/test_tasks_incremental.py— state keys, ref resolution, diff bucketing/prefix filter, skip-on-no-change, first-run full download, incremental diff download, GapExchange sibling pull, manual-override precedence, state callback.