Repository navigation
Conversation
UpdateNotifyConfig persisted the new configuration and then replaced the running notification service in two independent steps. When service initialization failed after the write, the RPC returned an error while the configuration on disk was already updated, leaving the persisted configuration and the running service inconsistent until the next restart. Rebuilding the service also mutated the shared notifier in place, so a partial failure could leave a half-initialized service set, and updating to a disabled configuration left the previous services active. Build and validate the complete replacement service set in memory first (notify.Notifier.Prepare), persist the configuration, then activate the replacement with an atomic, infallible swap. Any failure before the swap leaves both the persisted configuration and the running service untouched. The notifier now hands out immutable service snapshots via atomic.Pointer, so in-flight sends never observe a half-initialized set. UpdateNotifyConfig also rejects a nil request with InvalidArgument.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MaliceRPC.UpdateNotifyConfigapplies a notification configuration change intwo independent steps: it persists the new configuration
(
configs.UpdateNotifyConfig) and only then rebuilds the running notificationservice (
core.EventBroker.InitService). If service initialization fails afterthe write, the RPC returns an error — the update looks rejected — but the
configuration on disk is already the new one. The persisted configuration and
the running service are now inconsistent, and the "failed" configuration takes
effect on the next restart.
Root cause
Two compounding problems in
server/rpc/rpc-config.goandserver/internal/notify/notify.go:so any initialization failure (e.g. a Telegram bot token the API rejects)
lands after the point of no return for the on-disk configuration.
Notifier.InitServicemutates the shared notifier(
n.enable = true, thenUseServicesper channel), so a mid-build failureleaves a half-initialized service set behind, and a send concurrent with a
rebuild races the mutation. Updating to a disabled configuration returns
early and silently leaves the previous services active.
Fix
Make the update transactional — build, persist, then atomically swap:
notify.Notifiernow publishes complete, immutable service sets through anatomic.Pointer. The newPreparevalidates the configuration and buildsthe full replacement set (all enabled channels are validated before any
provider is constructed) without touching the active one, and returns a
commitclosure that swaps it in. The swap cannot fail.UpdateNotifyConfigcallscore.EventBroker.PrepareNotifyConfigfirst(every fallible step — validation and provider construction — happens here),
then persists, then commits. A failure anywhere before the commit leaves
both the persisted configuration and the running service untouched; the
commit itself is infallible, so the two never diverge.
with a disabled one now actually stops delivery, and an in-flight send keeps
the snapshot it loaded.
UpdateNotifyConfigrejects a nil request withInvalidArgument.Boot-time initialization (
options.go→EventBroker.InitService) isunchanged in behavior;
InitServiceis nowPrepare+ commit.Tests
server/internal/notify— table-driven validation coverage for everyenabled channel; a failed
Preparekeeps the previously active service set;re-initializing with a disabled configuration swaps the old set out (verified
with an
httptestreceiver counting deliveries).server/internal/core—PrepareNotifyConfigvalidation/commit behaviorand the
ErrEventBrokerUnavailableguard for an unavailable broker.server/rpc— end-to-end transaction test driving the real persistence hook(the
set.*dump installed byPrepareConfig): a valid update persists andactivates; a follow-up update with an invalid webhook URL fails and leaves
the persisted file byte-identical, the loaded configuration unchanged, and
the running service still delivering to the previous receiver; a nil request
is rejected with
InvalidArgument.go vet ./...,go test ./... -count=1 -timeout 300s, andCGO_ENABLED=0 go build ./...all pass.Compatibility
No API or schema change. Valid configurations behave exactly as before; invalid
ones now fail the whole update instead of persisting anyway.