Skip to content

fix(server): make notify config update transactional - #102

Open
dust-life wants to merge 1 commit into
chainreactors:devfrom
dust-life:fix/notify-config-transaction
Open

dust-life wants to merge 1 commit into
chainreactors:devfrom
dust-life:fix/notify-config-transaction

Conversation

@dust-life

Copy link
Copy Markdown

Summary

MaliceRPC.UpdateNotifyConfig applies a notification configuration change in
two independent steps: it persists the new configuration
(configs.UpdateNotifyConfig) and only then rebuilds the running notification
service (core.EventBroker.InitService). If service initialization fails after
the write, the RPC returns an error — the update looks rejected — but the
configuration on disk is already the new one. The persisted configuration and
the running service are now inconsistent, and the "failed" configuration takes
effect on the next restart.

Root cause

Two compounding problems in server/rpc/rpc-config.go and
server/internal/notify/notify.go:

  1. No transaction boundary. The RPC persists first and initializes second,
    so any initialization failure (e.g. a Telegram bot token the API rejects)
    lands after the point of no return for the on-disk configuration.
  2. In-place mutation. Notifier.InitService mutates the shared notifier
    (n.enable = true, then UseServices per channel), so a mid-build failure
    leaves a half-initialized service set behind, and a send concurrent with a
    rebuild races the mutation. Updating to a disabled configuration returns
    early and silently leaves the previous services active.

Fix

Make the update transactional — build, persist, then atomically swap:

  • notify.Notifier now publishes complete, immutable service sets through an
    atomic.Pointer. The new Prepare validates the configuration and builds
    the full replacement set (all enabled channels are validated before any
    provider is constructed) without touching the active one, and returns a
    commit closure that swaps it in. The swap cannot fail.
  • UpdateNotifyConfig calls core.EventBroker.PrepareNotifyConfig first
    (every fallible step — validation and provider construction — happens here),
    then persists, then commits. A failure anywhere before the commit leaves
    both the persisted configuration and the running service untouched; the
    commit itself is infallible, so the two never diverge.
  • Snapshot sends also fix the stale-service case: replacing the configuration
    with a disabled one now actually stops delivery, and an in-flight send keeps
    the snapshot it loaded.
  • UpdateNotifyConfig rejects a nil request with InvalidArgument.

Boot-time initialization (options.go → EventBroker.InitService) is
unchanged in behavior; InitService is now Prepare + commit.

Tests

  • server/internal/notify — table-driven validation coverage for every
    enabled channel; a failed Prepare keeps the previously active service set;
    re-initializing with a disabled configuration swaps the old set out (verified
    with an httptest receiver counting deliveries).
  • server/internal/core — PrepareNotifyConfig validation/commit behavior
    and the ErrEventBrokerUnavailable guard for an unavailable broker.
  • server/rpc — end-to-end transaction test driving the real persistence hook
    (the set.* dump installed by PrepareConfig): a valid update persists and
    activates; a follow-up update with an invalid webhook URL fails and leaves
    the persisted file byte-identical, the loaded configuration unchanged, and
    the running service still delivering to the previous receiver; a nil request
    is rejected with InvalidArgument.

go vet ./..., go test ./... -count=1 -timeout 300s, and
CGO_ENABLED=0 go build ./... all pass.

Compatibility

No API or schema change. Valid configurations behave exactly as before; invalid
ones now fail the whole update instead of persisting anyway.

UpdateNotifyConfig persisted the new configuration and then replaced the
running notification service in two independent steps. When service
initialization failed after the write, the RPC returned an error while
the configuration on disk was already updated, leaving the persisted
configuration and the running service inconsistent until the next
restart. Rebuilding the service also mutated the shared notifier in
place, so a partial failure could leave a half-initialized service set,
and updating to a disabled configuration left the previous services
active.

Build and validate the complete replacement service set in memory first
(notify.Notifier.Prepare), persist the configuration, then activate the
replacement with an atomic, infallible swap. Any failure before the
swap leaves both the persisted configuration and the running service
untouched. The notifier now hands out immutable service snapshots via
atomic.Pointer, so in-flight sends never observe a half-initialized
set. UpdateNotifyConfig also rejects a nil request with InvalidArgument.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant