fix: Report an incomplete DynamoDB auto-config cache write - #885
Merged
Merged
Conversation
SetAll called batchWrite purely for effect and then returned nil, and batchWrite returned nothing. A failed chunk, an oversized item, an item that could not be built, and an item DynamoDB left unprocessed were all logged and forgotten. The caller was told the write succeeded, then later read a partial snapshot and treated it as the whole configuration. batchWrite now returns how many items it could not write, and SetAll adds that to the items it dropped before ever reaching the write. If anything was dropped it returns an error naming the count against the total it set out to write. The write still does not abort on a failure, because a partially written cache is worth more than none. What changed is only that the caller finds out. The new tests drive the real AWS SDK against an httptest server that speaks the DynamoDB wire protocol, so they need no local DynamoDB and run in normal CI. Two of them pin the arithmetic rather than the behavior: an unprocessed item must report "1 of 3" and not "1 of 4", and a wholly failed batch "3 of 3" and not "3 of 6", since an item must not be counted once as a write request and again as a drop.
kinyoklion
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SetAllcalledbatchWritepurely for effect and then returnednil, andbatchWritereturned nothing at all. Four different ways of losing an item -- a failed chunk, an oversized item, an item that could not be built, and an item DynamoDB left unprocessed -- were each logged and then forgotten. The caller was told the write succeeded, and a laterGetAllreturned a partial snapshot that it had no way to recognize as partial, so it treated it as the whole configuration.batchWritenow returns how many items it could not write, andSetAlladds that to the count it had already dropped before reaching the write. If anything was dropped, it returns an error naming the count against the total it set out to write.The write still does not abort on a failure, because a partially written cache is worth more than none. The only change in behavior is that the caller finds out.
internal/autoconfig/stream_manager.go:652already logs whateverSetAllreturns, so the new error surfaces as a warning with no caller change.Tests
internal/autoconfigcache/dynamodb_store_test.godid not exist onv9. The tests drive the real AWS SDK client against anhttptestserver that speaks the DynamoDB wire protocol, so they need no local DynamoDB and run in normal CI. The build-tagged tests elsewhere cover a real database; these cover the store's own accounting, which a real database cannot be made to produce on demand.Two of them pin the arithmetic rather than the behavior, because that is the easy thing to get wrong here: an item must not be counted once as a write request and again as a drop. An unprocessed item has to report
1 of 3rather than1 of 4, and a wholly failed batch3 of 3rather than3 of 6.Relationship to v8
Brings
v9to parity withv8#874. Thev8change also handled payload filters; that half is dropped, sincev9removed payload filters in #867 and #869.Note
Overview
DynamoDB AutoConfig cache
SetAllnow fails when the snapshot cannot be fully written, instead of returning success after partial or dropped writes.SetAlltracks items dropped before batching (marshal/build failures and oversize items) and addsbatchWrite's returned count for failed chunks and DynamoDB unprocessed items. When any drops occur, it returns an error likeN of M items could not be writtenwhile still continuing batch writes for resilience.batchWritenow returns anintof items it could not write.Adds
dynamodb_store_test.gowith an httptest fake DynamoDB wire server to exercise success, chunking (25), unprocessed items, failed batches, and oversized items—especially ensuring drop totals are not double-counted against write requests.Reviewed by Cursor Bugbot for commit b7d73bf. Bugbot is set up for automated code reviews on this repo. Configure here.