Finalise grade syncs left stuck in_progress

[santicomp2014]

The problem

A GradingSync only reaches a terminal status when a sync_grades_complete task observes
every one of its grades as complete. That relies on a single delayed message arriving after
the last grade commits. If it is lost — or fires before the commit it needed to see — nothing
re-checks and the sync stays in_progress forever.

That is not cosmetic. A non-terminal sync blocks all further grade syncing for its
assignment:

if self.auto_grading_service.get_in_progress_sync(assignment):
    self.request.response.status_int = 400
    return {"message": "There's already an auto-grade sync in progress"}

and ix__grading_sync_assignment_status_unique enforces the same rule in the database. So one
lost message takes out grade syncing for that assignment permanently, until someone clears it
by hand.

It happened

Production, 2026-08-16, CA region, assignment 9253. A grade got a 403 from the institution's
Brightspace, exhausted its retries, and set success=False inside its transaction:

22:00:01,587  final failure -> success=False set, completion scheduled (countdown=1)
22:00:02,575  sync_grades_complete SUCCEEDED          <- ran here
22:00:02,944  sync_grade SUCCEEDED - success=False COMMITS here
              (nothing runs again)

The finaliser completed 369 ms before the commit it needed to see, found the grade still
pending, declined to finalise, and exited successfully. The sync sat in in_progress for three
days. The instructor could not sync at all — every attempt returned 400.

Full analysis in docs/grade-sync-completion-race.md.

The fix

sweep_stale_grading_syncs, scheduled every 15 minutes from h-periodic
(hypothesis/h-periodic — needs to merge after this one, since it references this task).

It finalises anything left non-terminal past STALE_GRADING_SYNC_TIMEOUT, and marks grades
still incomplete by then as failed. That second part matters: without it, a lost sync_grade
task would leave a grade NULL forever and the sync could never reach a terminal state — the
reaper would keep declining, exactly like the original finaliser did.

Dispatch happens after the transaction commits, since scheduling work from inside a
transaction is the bug being cleaned up after.

15 minutes is a wide margin: a sync normally completes in ~1 s, and the worst legitimate case
is a grade exhausting max_retries=2 with retry_backoff=10, well under a minute.

The test that should have caught this

The existing test for sync_grades_complete could never fail — its last line was an assignment,
not an assertion:

sync_grades_complete(grading_sync_id=grading_sync.id)
grading_sync.status = status      # not `assert ... == status`

Correcting it revealed three of its five expected values were also wrong: the fixture starts at
scheduled, so a sync that is not finalised stays scheduled, not in_progress.

Not included

This guarantees recovery; it does not reduce how often the race fires. Two follow-ups are
worth doing separately, and are written up in the doc:

1. Schedule the completion task on the transaction's commit hook rather than inline.
2. Give sync_grades_complete a retry policy — it is a bare @app.task(), unlike sync_grade.

Testing

⚠️ The new tests have not been run locally — this machine has no pg_config, so psycopg2
will not build in the tox env. Both files pass py_compile and the logic has been reviewed, but
please let CI be the judge, and give the four new tests a careful read.

The production instance of this was cleared separately by re-running sync_grades_complete for
that sync; recovery steps are in the playbook runbook referenced above.

🤖 Generated with Claude Code