Add stratified sampling across block ids and shared block-boundary logic - #85
Open
ahmedmohammed107 wants to merge 1 commit into
Open
Add stratified sampling across block ids and shared block-boundary logic#85ahmedmohammed107 wants to merge 1 commit into
ahmedmohammed107 wants to merge 1 commit into
Conversation
…k-boundary logic - Within sampling_lookback, sample_batch now performs balanced stratified sampling across every unique block_id present, rather than sampling the lookback window as one undifferentiated pool. Remainders are allocated to the most recent (largest) block ids first, so the newest data is favored when the requested batch size doesn't divide evenly across blocks. - agent_inferences gains 'block_id' and 'context' columns (context is a single, generic BLOB shared by every agent type - see init.sql). DBManager tracks the most recently written row (host, context, block_id) via refresh_latest_row_cache()/_load_latest_row() and advances block_id through _compute_block_id() whenever the gap since the previous sample exceeds the configured max_gap_seconds, or the context value changes. - Added smocs/utils/block_boundary.py: is_new_block(), the shared rule for what counts as a block boundary, now used identically by both DBManager (when assigning block_id on write) and the inference thread (when deciding whether a candidate window may be considered at all) - a window is never allowed to cross a block_id boundary in either training or inference. Both the ingest thread's DBManager (which must seed its block-tracking cache from history on startup) and the inference thread (which reads max_gap_seconds to know when a candidate window has crossed a boundary) are wired up to this here. This builds on the already-merged rename-window-terminology and 77-mysql-db-naming-convention branches; the window/flattened_window terminology throughout is inherited from the former rather than introduced here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #<ISSUE_NUMBER>.
Adds a
block_id-aware notion of "distinct collection run" toagent_inferences, and uses it to (a) guarantee sampled training windows never span a gap or context change, and (b) makesample_batch()sample in a stratified, balanced way across blocks within the lookback window instead of treating it as one undifferentiated pool. The boundary rule itself (is_new_block()) is centralized insmocs/utils/block_boundary.pyand shared verbatim betweenDBManager(on write) and the autoencoder inference thread's streaming buffer (in memory), so the two can never drift apart.Also fixes an undocumented bug where
eval_model()racedtraining_loop()'s post-training count bump and silently skipped evaluation on roughly half of all cycles — factored out via a new_fetch_and_preprocess_batch()helper thateval_model()now calls directly instead ofget_training_data().See the linked issue for full details. New unit test coverage added in
test_db_api.py.