Skip to content

Add Rubin ToO filter fields and ScoutHistory models#23

Open
talister wants to merge 11 commits into
devfrom
add-rubin-too-filter-fields
Open

Add Rubin ToO filter fields and ScoutHistory models#23
talister wants to merge 11 commits into
devfrom
add-rubin-too-filter-fields

Conversation

@talister

@talister talister commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

This PR addresses Issue #21 and #12 by doing the following:

  • Adds 5 extra attributes (Vmag, rate, RA, Dec, time of ephemeris computation/element validity) to ScoutDetail that are needed to track pass/fail for the Rubin NEO ToO filters and associated migration
  • Fixes a bug in the earlier implementation where arc length wasn't converted to the desired units
  • Adds a ScoutDetail class to hold the 'active' attribute (Fixes An active attribute may be needed in ScoutDetail for efficient filtering #12) and links through to append-only ScoutDetailHistory class to make records recording the changing values of Scout targets as they receive additional observations and the orbit it recomputed (this data is not available anywhere so needs to be stored in FOMO to allow past history and retrospective analysis of no. of candidates passing the ToO filters)
  • A management command that uses a previously run Scout DataServiceQuery to retire active candidates and update the TargetName with the final designation once it leaves Scout (by querying the Previous NEOCP page as the only source of what happened to it)

talister and others added 11 commits June 13, 2026 09:57
@talister @claude
Add Vmag, rate, ra, dec, t_ephem to ScoutDetail; fix arc hours->days
1ceb749 
Snapshot Scout's contemporaneous ephemeris quantities (V magnitude, sky
motion rate, RA, Dec) plus the ephemeris epoch (t_ephem) they are valid
for, so they need not be recomputed via the heavyweight Ephemeris path.
RA is parsed from sexagesimal HH:MM[:SS] to decimal degrees; arc is
converted from Scout's hours to days for consistency with confirmed-object
arcs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@talister @claude
Add mocked-Scout-API ingest integration tests
6afed63 
Patch requests.get so the real query_service -> query_targets -> to_target
chain runs end to end, guarding the parse/DB seams that the unit tests mock
away (arc hours->days, sexagesimal RA -> degrees, tEphem). Also covers
rejection at the summary-filter stage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@talister @claude
Add ScoutDetailHistory and active flag via shared abstract base
113456c 
Factor the Scout snapshot fields into an abstract BaseScoutDetail so they are
defined once and shared by two concrete models:
- ScoutDetail (OneToOne, current state) gains an `active` boolean: False once an
  object leaves the Scout list (designated, removed, or impacted). Set True on
  every ingest. Addresses #12 (efficient filtering of stale candidates).
- ScoutDetailHistory (FK, append-only) records each Scout recomputation, deduped
  on (target, last_run) so history reflects Scout reruns not fetch cadence.

to_target writes both the current-state row (active=True) and a history row
(get_or_create on last_run) on each ingest. Includes migration 0003 and ingest
tests covering history creation, idempotency, and the active flag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@talister @claude
Add ingest_scout management command for full-list Scout reconciliation
2563a3a 
The interactive Scout flow only creates user-selected targets, so it never sees
the full Scout list and cannot detect when an object leaves. This command re-runs
a saved broad DataServiceQuery (minimal cuts) headless against the whole current
Scout list, upserting every match (ScoutDetail active=True + history row), then
sweeps: any previously-active candidate absent from the response is marked inactive
(designated, removed, or impacted). Completes #12. Suitable for cron.

Passes a synthetic request (superuser + silent message store) because the base
to_target emits an unguarded messages.warning on the already-exists path. The
sweep is skipped when no candidates are returned, to avoid deactivating the whole
population on a transient Scout API error. Supports --query-name/--query-id,
--no-sweep, and --dry-run. Four command tests added (51 total).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@talister
Add .vscode to ignore
a63148c
@talister @claude
Register ScoutDetailHistory in admin; fix plural verbose name
117fc8f 
Add a read-only ScoutDetailHistoryAdmin (has_add/change_permission
disabled) with date_hierarchy on last_run and the same score/choice
filters as ScoutDetailAdmin, so the append-only history trail is
browseable in the Django admin.

Fix the auto-pluralised 'Scout Detail Historys' by adding an explicit
verbose_name_plural = 'Scout Detail Histories' to ScoutDetailHistory.Meta.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@talister @claude
Add MPC designation lookup to ingest_scout
9e7a2a7 
After the Scout sweep, query the MPC 'Previous NEOCP Objects' page
(https://minorplanetcenter.net/mpcops/neocp/neocp_prev_des/) to find
which Scout candidates have received official IAU provisional designations
(e.g. A11Df9S -> 2026 LX) and record the new name as a TargetName alias,
keeping the Scout provisional as the primary Target.name.

Objects that left Scout as lost/dne/ns/na are silently skipped. Network
failures warn but do not abort the sweep. --dry-run reports what aliases
would be added without writing. get_or_create on the alias name field
prevents any target from claiming a designation already held by another.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@talister @claude
Add missing migration for ScoutDetailHistory Meta options change
bf623e0 
verbose_name_plural was added to ScoutDetailHistory.Meta in 0eb129a
without a corresponding migration, failing the check_migrations CI job.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@talister
Merge remote-tracking branch 'origin/dev' into add-rubin-too-filter-f…
9c872c2 
…ields
@talister
Add fix for incorrect related_name lookup. Add some more tests to cat…
b93b16f 
…ch this and making a start on broader coverage for _update_designations (May need some backports from the similar NEOx methods)
@talister @claude
Add tests for MPC designation parsing and the sweep safety guard
e098eb2 
Cover the two highest-value gaps in ingest_scout.py (72% -> 90%):

- TestFetchMpcPrevDesignations exercises the real MPC HTML-table parsing
  in _fetch_mpc_prev_designations (header rows, dash/None placeholders,
  short rows, empty cells) plus the empty-table and HTTP-error paths,
  mocking requests.get so no network call is made.
- test_sweep_with_no_seen_ids_skips_deactivation covers the _sweep guard
  that must not deactivate the whole stored population when a run returns
  no candidates (almost always a Scout API error).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@talister talister requested a review from jchate6 June 15, 2026 23:35
@talister talister added Data Services Data Services Rubin SSSC Solar System Science Collaboration Work enhancement New feature or request labels Jun 15, 2026
@coveralls

coveralls commented Jun 15, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 27583489620

Coverage increased (+4.4%) to 90.355%

Details

  • Coverage increased (+4.4%) from the base build.
  • Patch coverage: 20 uncovered changes across 4 files (360 of 380 lines covered, 94.74%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
tom_jpl/management/commands/ingest_scout.py 145 130 89.66%
tom_jpl/admin.py 12 10 83.33%
tom_jpl/jpl.py 18 16 88.89%
tom_jpl/models.py 22 21 95.45%
Total (6 files) 380 360 94.74%

Coverage Regressions

No coverage regressions found.

Coverage Stats

Coverage Status
Relevant Lines: 1099
Covered Lines: 993
Line Coverage: 90.35%
Coverage Strength: 0.9 hits per line
💛 - Coveralls

@jchate6 jchate6 moved this to Needs Review in TOM Toolkit Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jchate6 jchate6 Awaiting requested review from jchate6

Assignees

No one assigned

Labels

Data Services Data Services enhancement New feature or request Rubin SSSC Solar System Science Collaboration Work

Projects

TOM Toolkit
Status: Needs Review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

An active attribute may be needed in ScoutDetail for efficient filtering

3 participants

@talister @coveralls @jchate6