Skip to content

fix: Report an incomplete DynamoDB auto-config cache write - #885

Merged
keelerm84 merged 1 commit into
v9from
mk/SDK-3159/dynamodb-incomplete-write
Sep 21, 2026
Merged

keelerm84 merged 1 commit into
v9from
mk/SDK-3159/dynamodb-incomplete-write

Conversation

@keelerm84

@keelerm84 keelerm84 commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Summary

SetAll called batchWrite purely for effect and then returned nil, and batchWrite returned nothing at all. Four different ways of losing an item -- a failed chunk, an oversized item, an item that could not be built, and an item DynamoDB left unprocessed -- were each logged and then forgotten. The caller was told the write succeeded, and a later GetAll returned a partial snapshot that it had no way to recognize as partial, so it treated it as the whole configuration.

batchWrite now returns how many items it could not write, and SetAll adds that to the count it had already dropped before reaching the write. If anything was dropped, it returns an error naming the count against the total it set out to write.

The write still does not abort on a failure, because a partially written cache is worth more than none. The only change in behavior is that the caller finds out. internal/autoconfig/stream_manager.go:652 already logs whatever SetAll returns, so the new error surfaces as a warning with no caller change.

Tests

internal/autoconfigcache/dynamodb_store_test.go did not exist on v9. The tests drive the real AWS SDK client against an httptest server that speaks the DynamoDB wire protocol, so they need no local DynamoDB and run in normal CI. The build-tagged tests elsewhere cover a real database; these cover the store's own accounting, which a real database cannot be made to produce on demand.

Two of them pin the arithmetic rather than the behavior, because that is the easy thing to get wrong here: an item must not be counted once as a write request and again as a drop. An unprocessed item has to report 1 of 3 rather than 1 of 4, and a wholly failed batch 3 of 3 rather than 3 of 6.

Relationship to v8

Brings v9 to parity with v8 #874. The v8 change also handled payload filters; that half is dropped, since v9 removed payload filters in #867 and #869.


Note

Overview
DynamoDB AutoConfig cache SetAll now fails when the snapshot cannot be fully written, instead of returning success after partial or dropped writes.

SetAll tracks items dropped before batching (marshal/build failures and oversize items) and adds batchWrite's returned count for failed chunks and DynamoDB unprocessed items. When any drops occur, it returns an error like N of M items could not be written while still continuing batch writes for resilience. batchWrite now returns an int of items it could not write.

Adds dynamodb_store_test.go with an httptest fake DynamoDB wire server to exercise success, chunking (25), unprocessed items, failed batches, and oversized items—especially ensuring drop totals are not double-counted against write requests.

Reviewed by Cursor Bugbot for commit b7d73bf. Bugbot is set up for automated code reviews on this repo. Configure here.

SetAll called batchWrite purely for effect and then returned nil, and
batchWrite returned nothing. A failed chunk, an oversized item, an item that
could not be built, and an item DynamoDB left unprocessed were all logged and
forgotten. The caller was told the write succeeded, then later read a partial
snapshot and treated it as the whole configuration.

batchWrite now returns how many items it could not write, and SetAll adds that
to the items it dropped before ever reaching the write. If anything was
dropped it returns an error naming the count against the total it set out to
write.

The write still does not abort on a failure, because a partially written cache
is worth more than none. What changed is only that the caller finds out.

The new tests drive the real AWS SDK against an httptest server that speaks
the DynamoDB wire protocol, so they need no local DynamoDB and run in normal
CI. Two of them pin the arithmetic rather than the behavior: an unprocessed
item must report "1 of 3" and not "1 of 4", and a wholly failed batch "3 of 3"
and not "3 of 6", since an item must not be counted once as a write request
and again as a drop.
@keelerm84
keelerm84 requested a review from a team as a code owner September 21, 2026 14:15
@keelerm84
keelerm84 merged commit c37db62 into v9 Sep 21, 2026
18 checks passed
@keelerm84
keelerm84 deleted the mk/SDK-3159/dynamodb-incomplete-write branch September 21, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants