Skip to content

daemon: replace exit(1) callbacks with recoverable state machine - #62

Open
silentone12725 wants to merge 2 commits into
WorldObservationLog:mainfrom
silentone12725:main
Open

daemon: replace exit(1) callbacks with recoverable state machine#62
silentone12725 wants to merge 2 commits into
WorldObservationLog:mainfrom
silentone12725:main

Conversation

@silentone12725

@silentone12725 silentone12725 commented Aug 6, 2026

Copy link
Copy Markdown

Summary

This PR changes the wrapper from a process that terminates on Apple playback lease errors into a long-running, self-recovering daemon.

Previously, both SVPlaybackLeaseManager callbacks (endLeaseCb and pbErrCb) called exit(1) unconditionally. A lease termination therefore killed the entire wrapper, requiring an external supervisor to restart the process.

The new implementation keeps the wrapper and its HTTP services alive, moves FairPlay context reacquisition into a dedicated recovery worker, makes access to the shared decryption context thread-safe, exposes lifecycle state to external consumers, and brings the rootless wrapper in line with the same recovery model.

It also adds build/drift tooling to help keep the C and rootless wrapper implementations synchronized.


What changed

Recovery state machine

main.cpp now maintains an explicit recovery lifecycle:

  • Running
  • Scheduled
  • Refreshing
  • Failed

get_recovery_state() is exported as an extern "C" API so external consumers can observe the daemon state:

Value State
0 Running
1 Scheduled
2 Refreshing
3 Failed

is_recovery_active() provides a simple recovery gate for request handlers.

Non-blocking lease callbacks

endLeaseCb and pbErrCb no longer perform recovery work or terminate the process directly.

Instead, callbacks enqueue a recovery event and return immediately.

This keeps FairPlay/library calls out of the lease-manager callback context and reduces the risk of callback reentrancy or deadlocks.

Dedicated recovery worker

Recovery is handled by a persistent worker thread that owns context reacquisition.

The worker:

  • drains and coalesces queued lease events before each recovery attempt;
  • performs one refresh for a burst of related errors instead of repeatedly refreshing for every callback;
  • calls refresh_decrypt_ctx() as the single owner of reacquisition;
  • verifies recovery using is_preshare_ctx_ready();
  • resets the consecutive-failure counter only after a usable preshareCtx exists; and
  • schedules its own retry after a failed refresh, so recovery continues even when Apple sends no additional lease event.

Retry delay uses bounded exponential backoff:

1s -> 2s -> 5s -> 10s -> 30s

The 30-second delay is clamped rather than terminating recovery, allowing the daemon to survive prolonged transient failures.

Thread-safe FairPlay context access

main.c now protects preshareCtx with a PTHREAD_MUTEX_INITIALIZER-backed mutex.

All relevant reads and writes are synchronized between the decrypt path and the recovery worker.

The mutex is deliberately released before FairPlay/network reacquisition work so long-running recovery operations do not hold the context lock.

is_preshare_ctx_ready() also checks the context while holding the mutex.

Request gating during recovery

Decrypt and M3U8 request handling now checks the recovery state before using the FairPlay context.

During recovery:

  • the decrypt endpoint returns immediately/EOF;
  • the M3U8 endpoint returns an empty response and continues serving subsequent requests; and
  • key-delivery work is prevented from racing context reacquisition.

The decrypt, M3U8, and account servers themselves remain running throughout lease recovery.

External supervision is therefore reserved for actual process failures such as crashes, OOM termination, or other unrecoverable failures rather than normal lease lifecycle events.


DRM lifecycle state

The wrapper now publishes its lifecycle through:

<base-dir>/drm-state

The state file allows the surrounding Go/Electron layer to observe wrapper lifecycle changes without inferring state solely from process existence.

Lifecycle states include:

  • STARTING
  • LOGIN
  • WAITING_2FA
  • INITIALIZING_FAIRPLAY
  • RUNNING
  • RECOVERY
  • FAILED
  • STOPPED

This provides a lightweight interface for status reporting and supervisor integration while keeping recovery ownership inside the wrapper.


Rootless wrapper

wrapper-rootless.c has been updated to follow the persistent-daemon recovery model rather than treating lease termination as an unconditional process-exit condition.

This keeps rootless operation aligned with the recovery behavior introduced in the main wrapper implementation.


Build and drift protection

This PR also adds tooling around the wrapper build:

  • adds Dockerfile.build for a reproducible wrapper build environment;
  • adds scripts/check-drift.py;
  • runs the wrapper drift check from the x86_64 GitHub Actions workflow.

The drift check is intended to catch unintended divergence between related wrapper implementations as recovery/lifecycle behavior evolves.


Result

Before

Apple lease error
       |
       v
 endLeaseCb / pbErrCb
       |
       v
     exit(1)
       |
       v
 wrapper process dies
       |
       v
 external supervisor must restart it

After

Apple lease error
       |
       v
 endLeaseCb / pbErrCb
       |
       v
 enqueue recovery event
       |
       v
 callback returns immediately
       |
       v
 recovery worker
       |
       +--> coalesce events
       |
       +--> refresh FairPlay context
       |
       +--> success --> RUNNING
       |
       `--> failure --> backoff --> retry

The wrapper therefore behaves as a persistent service: lease expiration and transient FairPlay recovery failures are handled internally, while the process and HTTP servers remain alive.

Commits

This PR consists of two logical changes:

  1. daemon: replace exit(1) callbacks with recoverable state machine

    • recovery state machine
    • asynchronous recovery worker
    • retry/backoff behavior
    • context synchronization
    • request gating
  2. Add DRM state tracking and wrapper drift check

    • lifecycle state publication
    • rootless wrapper updates
    • build tooling
    • wrapper drift validation and CI integration

Previously, both SVPlaybackLeaseManager callbacks (endLeaseCb, pbErrCb)
called exit(1) unconditionally, making every Apple lease termination a
fatal process death. wrapper-rootless was behaving like a short-lived
utility rather than a persistent background service.

This commit introduces a proper recovery lifecycle:

Recovery state machine (main.cpp)
- RecoveryState enum: Running / Scheduled / Refreshing / Failed
- get_recovery_state() exported as extern C int for status endpoints
  and Electron IPC (0=Running, 1=Scheduled, 2=Refreshing, 3=Failed)
- is_recovery_active() derived from state, used to gate HTTP requests

Non-blocking callbacks
- endLeaseCb / pbErrCb now push a code onto a queue and return
  immediately — no library calls from within the lease manager thread
- Eliminates reentrancy and deadlock risk from callback context

Dedicated recovery worker thread
- Drains the entire queue on each wake (coalescing): a burst of
  3084+3084+PLAYBACK_ERR produces one refresh cycle, not three
- Exponential backoff: 1s -> 2s -> 5s -> 10s -> 30s (clamped, never stops)
- Calls refresh_decrypt_ctx() as the sole owner of reacquisition logic
- Verifies success via is_preshare_ctx_ready() — resets consec_fails
  only when preshareCtx is non-null after the refresh
- Self-schedules retry on failure (kRetryInternal) so the daemon keeps
  retrying even when Apple sends no further lease-end event (e.g.
  transient network failure during refresh)

Thread safety (main.c)
- g_ctx_mutex (PTHREAD_MUTEX_INITIALIZER) protects all preshareCtx
  reads and writes against concurrent access from the decrypt thread
  and the recovery worker
- Lock is released before FairPlay network calls to avoid blocking
  decryption during reacquisition
- is_preshare_ctx_ready() reads preshareCtx under g_ctx_mutex

Client-visible state gating (main.c)
- handle() and handle_m3u8() check is_recovery_active() per request
- Decrypt server: returns immediately (EOF) during recovery
- M3U8 server: writes empty line, continues loop — no hanging requests
- Prevents FairPlay key-delivery calls from racing the recovery worker

HTTP servers (decrypt, m3u8, account) remain alive across all lease
events. The Electron supervisor continues to provide the outer safety
net for true process deaths (segfault, OOM, etc.).
@silentone12725
silentone12725 marked this pull request as ready for review August 7, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant