| service | lambda | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_module | aws-sdk-go-v2/service/lambda@v1.107.0 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| last_audit_commit | 302aa4e3c | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| last_audit_date | 2026-09-18 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| overall | A | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| protocol | REST-JSON | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| families |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| gaps | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| items_still_open |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| deferred | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| leaks |
|
cmd/zeroguard flagged 19 rows across 5 Update/Put ops. 7 fields were real
bugs, pointer-ified: UpdateCodeSigningConfig.Description,
UpdateEventSourceMapping.KMSKeyArn, UpdateFunctionConfiguration.{Description,
Role,Handler}, UpdateAlias.{FunctionVersion,Description} — each proven in
update_omitted_members_preserve_state_test.go (typed real client). 12 false
positives: identifiers used only for lookup (UUID, RevisionId precondition
fields), or Put/replace-required fields (UpdateFunctionCode's
ImageUri/S3Bucket/S3Key, PutRuntimeManagementConfig.RuntimeVersionArn).
- InvocationType is a type alias (type InvocationType = string) so lambda backend satisfies sns.LambdaInvoker directly.
- ARN-parsing anti-pattern "take last colon segment" recurs — watch for it elsewhere.
- Trap: RemovePermission wire = DELETE /2015-03-31/functions/{name}/policy/{StatementId} (path, not query).
- ce30166a (Parity sweep 3, unrelated commit that swept in a large dependency+datalayer PR) converted most lambda backend maps to pkgs/store Table/Index. eventInvokeConfigs, versions, layers, versionCounters, functionConcurrencies, layerVersionCounters, layerPolicies, activeConcurrencies, fnCodeSigningConfigs, fisFaults, runtimeManagementConfigs, functionRecursionConfigs, functionScalingConfigs, versionIndex, esmByFunctionARN, runtimes, functionURLServers were deliberately left as plain maps (documented per-field in store_setup.go's package doc) — each has a concrete reason (no pure identity in the value, one-to-many shape, or live non-serializable state). Read that doc comment before "fixing" any of them into a Table.
- pkgs/store.Table/Index perform NO internal locking (by design — see pkgs/store package doc); every lambda call site still takes b.mu itself. Index.Get() returns a slice OWNED BY THE INDEX — never return it directly from a public method without copying first (ListAliases/GetPolicy both copy correctly; verified).
- Policy RevisionId (function-policy and layer-version-policy) is deliberately a pure content-hash of the sorted StatementId set (policyRevisionID in permissions.go, layerPolicyRevisionID in layers.go), NOT a stored uuid.New()-per-mutation field like Function/Version/Alias RevisionID. This works because statement content is immutable once added (no UpdatePermission op exists — a StatementId can only be added once, then removed), so the ID set alone detects every real mutation, and it stays correct across Snapshot/Restore without adding new persisted state.
- writeError's return value is NOT a reliable "did this write an error response" signal — c.JSON (which it wraps) returns nil on any successful write, including a written error, so
if xErr := h.writeError(...); xErr != nilcan never trigger. Handler helpers that write an error and need the caller to stop must return bool (true=continue), matching validateMemoryAndTimeout/checkRevisionID/applyFunctionCodeUpdate. A stale!= nilcheck on such a helper is a latent double-write bug (found + fixed in applyFunctionCodeUpdate this sweep) — grep for this pattern before trusting any "returns error, checked with != nil" helper that calls writeError internally. - Durable-execution family spans THREE independent path prefixes, not one — do not assume everything nests under
/2025-12-01/durable-executions/{DurableExecutionArn}/...: GetDurableExecution/History/State + CheckpointDurableExecution + StopDurableExecution do; ListDurableExecutionsByFunction is/2025-12-01/functions/{FunctionName}/durable-executions(a/functionspath, verified against api_op_ListDurableExecutionsByFunction.go); SendDurableExecutionCallback{Success,Failure,Heartbeat} is/2025-12-01/durable-execution-callbacks/{CallbackId}/{succeed|fail|heartbeat}keyed by CallbackId, not DurableExecutionArn (note succeed/fail, not success/failure — trap for anyone guessing the suffix). See handler_paths.go's prefix constants and handler_durable_execution.go'sisDurableExecPath/dispatchDurableExecRoutes. - Lambda's REST API is spread across a dozen+ date-versioned path prefixes (2015-03-31, 2017-03-31, 2017-10-31, 2018-10-31, 2019-09-25, 2019-09-30, 2020-04-22, 2020-06-30, 2021-07-20, 2021-10-31, 2021-11-15, 2024-08-31, 2025-11-30, 2025-12-01 all appear). gopherstack-l5ir found 4 of these constants carrying a wrong date (tags: 2015-03-31 vs real 2017-03-31; recursion-config: 2024-08-28 vs real 2024-08-31; scaling-config: 2023-10-26 vs real 2025-11-30) that made every op under that prefix unreachable. When adding or auditing any lambda op, verify its date prefix against
httpbinding.SplitURI(...)in serializers.go directly -- do not assume a "close enough" date is correct, and do not trust an existing constant's date without checking it against the SDK source at least once. - durable_execution is intentionally NOT wired into Snapshot/Restore (durableExecutionStore isn't touched by persistence.go) — this predates the wire-shape rewrite and is unrelated to it; durable executions were never persisted, only cleared on Reset (lifecycle.go's
b.durableExecs.reset()). Not flagged as a bug: no entry point exists to repopulate FunctionArn/DurableConfig/InputPayload after a restore anyway (see durable_execution family note above), so persisting the store today would only round-trip empty shells. ListLayersandListLayerVersionssummary narrowing:LayerVersion.Contentwas previously populated onListLayersandListLayerVersionsresponses. Inaws-sdk-go-v2/service/lambda@v1.101.2,types.LayerVersionsListItemdoes not containContent(onlyGetLayerVersion/PublishLayerVersionreturnsContent). Fixed:ListLayersandListLayerVersionsomitContent.
2026-08-23: pagination bug sweep (ListLayerVersions, ListProvisionedConcurrencyConfigs, ListCodeSigningConfigs, ListFunctionsByCodeSigningConfig)
Discovered while auditing the pagination bug class found in medialive.
handleListLayerVersions, handleListProvisionedConcurrencyConfigs,
handleListCodeSigningConfigs, and handleListFunctionsByCodeSigningConfig
all ignored the real Marker/MaxItems request members (lambda@v1.101.2:
ListLayerVersionsInput, ListProvisionedConcurrencyConfigsInput,
ListCodeSigningConfigsInput, ListFunctionsByCodeSigningConfigInput) and
always returned every item in one unbounded page with no NextMarker,
despite NextMarker already existing (unused) on all four output structs.
Fixed using the existing parsePaginationParams + pkgs/page.New +
lambdaDefaultMaxItems pattern already used by ListFunctions/ListLayers
in this package. ListLayerVersions, ListProvisionedConcurrencyConfigs,
and ListFunctionsByCodeSigningConfig are unexported *InMemoryBackend
methods (not part of a public interface) but changed return type from a
bare slice to page.Page[T]; go build ./... confirmed clean, and two
pre-existing test call sites (persistence_test.go, layers_test.go) updated
for the new ListLayerVersions signature. Proven with four
Test*_SDKRoundTrip_Pagination tests (list_pagination_ignored_test.go),
each driving the real SDK client across two 10-item pages of 25 seeded
items and asserting the pages are disjoint; all four fail against the
unfixed handlers (should have 10 item(s), but has 25), hand-reverted
and confirmed.
Audited but NOT fixed: handleListFunctionURLConfigs also ignores
Marker/MaxItems, but the route is always called with a non-empty
{name} path segment, and the per-function code path
(GetFunctionURLConfig(name)) can only ever return 0 or 1 items — this
service's data model has no per-qualifier function URL configs, so the
unbounded branch is dead code with zero real blast radius. Not fixed.
Read serializeOpHttpBindings<Op>Input directly for DeleteFunctionInput
(lambda@v1.101.2 serializers.go:1690,
awsRestjson1_serializeOpHttpBindingsDeleteFunctionInput): FunctionName
is URI-bound, Qualifier is query-bound
(encoder.SetQuery("Qualifier")). handleDeleteFunction
(handler_functions.go) never read the query string at all — it called
h.Backend.DeleteFunction(name) unconditionally, so a client asking to
delete one published version (DeleteFunctionInput{FunctionName, Qualifier: "2"}) instead had the entire function deleted: every version,
every alias, every event source mapping. api_op_DeleteFunction.go's doc
comment is explicit: "To delete a specific function version, use the
Qualifier parameter. Otherwise, all versions and aliases are deleted", and
"You can't delete a version that an alias references." The backend already
tracked exactly the state this needed (b.versionIndex/b.versions for
published versions, b.aliasesByFunction for the alias-reference check) —
only DeleteFunction's dispatch ignored the qualifier.
Fixed via the existing QualifierInvoker/QualifierResolver
optional-extension pattern (store.go) rather than changing
StorageBackend.DeleteFunction's existing signature (would have required
touching services/cloudformation/resources.go:2150, the one out-of-package
caller, and running make build-check): added QualifierDeleter with
DeleteFunctionVersion(name, qualifier string) error, implemented on
InMemoryBackend (functions.go). handleDeleteFunction now reads
Qualifier off the query string; when present it type-asserts
QualifierDeleter and calls DeleteFunctionVersion, which deletes only the
targeted b.versionIndex[name][qualifier] entry (and its b.versions[name]
slice element) after checking b.aliasesByFunction for a referencing alias
(ErrVersionReferencedByAlias, new sentinel → 409 ResourceConflictException)
and rejecting Qualifier=$LATEST (ErrInvalidParameterValue → 400 — $LATEST
has no separate version resource; omit Qualifier to delete the whole
function). An empty Qualifier still calls the original unqualified
DeleteFunction path unchanged. Function tags are only released when the
whole function is deleted (qualifier == "").
TestDeleteFunction_Qualifier (delete_function_version_test.go) drives
the real aws-sdk-go-v2 lambda client, table-driven across three cases:
qualified delete removes only the targeted version ($LATEST and the other
version survive, GetFunctionConfiguration(Qualifier: v1) now 404s);
qualified delete is rejected with ResourceConflictException when an alias
still references that version (and the version survives the rejected
delete); unqualified delete still removes the whole function. Hand-reverted
handleDeleteFunction back to its pre-fix unconditional
h.Backend.DeleteFunction(name) call: both the "removes only that version"
and "blocked by alias reference" subtests failed exactly as predicted (the
whole function vanished instead of just the targeted version, so
GetFunctionConfiguration against the survivor 404'd and the
expected-error assertion against the alias-referenced delete saw no error
at all); restored and confirmed byte-identical via md5sum.
Modelling gaps found in the same header sweep, not implemented:
InvokeInput's TenantId (lambda@v1.101.2 serializers.go:3859,
awsRestjson1_serializeOpHttpBindingsInvokeInput) is a real
X-Amz-Tenant-Id header for Lambda's multi-tenant-function feature —
gopherstack has no tenant concept anywhere in this service, so this is a
genuine unmodeled feature, not a discarded-but-tracked field; reported, not
attempted. InvokeInput.DurableExecutionName (request header) and
InvokeOutput.DurableExecutionArn (response header, deserializers.go:8744,
awsRestjson1_deserializeOpHttpBindingsInvokeOutput) are likewise never
wired on the Invoke path — consistent with, not a new instance of, the
already-documented durable_execution family gap above ("gopherstack has no
StartDurableExecution entry point... this emulator's Invoke path does not
model durable-execution semantics").
Gates: go build ./..., go vet ./services/lambda/..., go test -race -count=1 ./services/lambda/..., go fix -diff ./services/lambda/... (no
diff), gofmt -l services/lambda/ (no output), golangci-lint run ./services/lambda/... (1 finding — godot on the new
DeleteFunctionVersion doc comment's closing quoted sentence, fixed by
rewording so the comment's last line ends outside the quote; 0 issues after,
no //nolint added), go test ./pkgs/persistence/... (no persisted struct
changed) all clean. No exported method signature was changed —
StorageBackend.DeleteFunction is untouched — so make build-check was not
required; go build ./... (whole repo) confirmed clean regardless.
gopherstack-huyl (Create-vs-Update precondition sweep). UpdateAlias
(versions_aliases.go) set alias.FunctionVersion = input.FunctionVersion
unconditionally, so an alias could be repointed at a version number that was
never published — CreateAlias validates the target version against
b.versions[name] (or accepts $LATEST), but UpdateAlias had no
equivalent check. lambda@v1.101.2 deserializers.go's
deserializeOpErrorUpdateAlias models ResourceNotFoundException (the same
code ErrVersionNotFound already maps to on the CreateAlias path), so the
fix mirrors CreateAlias's versionInList check and reuses the existing
sentinel error. handleUpdateAlias (handler_versions_aliases.go) previously
had no ErrVersionNotFound case at all — added one, matching handleCreateAlias's.
New real-SDK-client proof: TestUpdateAlias_UnknownVersionSurfacesResourceNotFoundException
($LATEST still exempted, proven by TestUpdateAlias_LatestVersionSucceeds)
in wire_field_fixes_test.go; hand-reverted versions_aliases.go +
handler_versions_aliases.go, confirmed both tests fail
(ResourceNotFoundException never surfaced), restored.
acceptguard flagged PutFunctionScalingConfigInput.MaximumConcurrency (models.go:130, read
in PutFunctionScalingConfig) as matching no member of any real Input in the module. Confirmed
against lambda@v1.101.2's real shape (api_op_PutFunctionScalingConfig.go,
api_op_GetFunctionScalingConfig.go, types/types.go:1614): the real request nests a
FunctionScalingConfig *types.FunctionScalingConfig under the request body key
"FunctionScalingConfig", and that nested type carries MinExecutionEnvironments/
MaxExecutionEnvironments (both *int32) — an unrelated concept (execution-environment
pool sizing for Lambda Managed Instances functions) to the flat concurrency-limit field a
prior version invented. GetFunctionScalingConfigOutput is also a different shape than what
gopherstack emulated: AppliedFunctionScalingConfig/RequestedFunctionScalingConfig/
FunctionArn as three top-level members, not a single flat struct.
Fixed by reshaping FunctionScalingConfig to the real nested type
(MaxExecutionEnvironments/MinExecutionEnvironments), PutFunctionScalingConfigInput to
nest it under FunctionScalingConfig, and adding real PutFunctionScalingConfigOutput
(FunctionState) and GetFunctionScalingConfigOutput (AppliedFunctionScalingConfig/
RequestedFunctionScalingConfig/FunctionArn) types (models.go). Backend methods
(function_settings.go) now return/accept the real Output/Input shapes directly. The
concurrency-throttling logic in invocation.go (acquireConcurrencySlot) that previously read
sc.MaximumConcurrency now reads sc.MaxExecutionEnvironments as its enforcement knob — a
reasonable emulation choice given execution-environment count is the real API's actual
concurrency-shaping lever for this operation, and no other field in the real shape serves an
analogous role.
Proven via a real aws-sdk-go-v2/service/lambda client round trip
(TestPutFunctionScalingConfig_MinMaxExecutionEnvironments, wire_field_fixes_test.go):
PutFunctionScalingConfig with MinExecutionEnvironments/MaxExecutionEnvironments, then
GetFunctionScalingConfig asserts both values round-trip through
AppliedFunctionScalingConfig/RequestedFunctionScalingConfig/FunctionArn. Hand-reverted
function_settings.go/invocation.go/models.go/store_setup.go, confirmed the test fails
(the real client's FunctionScalingConfig was never read; response fields empty), restored.
Test judgement: function_settings_test.go's TestFunctionScalingConfig_PutGet sent a raw
body of {"MaximumConcurrency":10} and asserted it round-tripped — testing the invented field as
correct. Rewrote to send the real wire shape ({"FunctionScalingConfig":{"MaxExecutionEnvironments":10}})
and assert against AppliedFunctionScalingConfig.MaxExecutionEnvironments.
TestScalingConfig_MaximumConcurrency_Enforced/TestScalingConfig_ZeroConcurrency_Blocked
constructed PutFunctionScalingConfigInput{MaximumConcurrency: &n} literals directly — updated
to the nested FunctionScalingConfig{MaxExecutionEnvironments: &n} shape; the concurrency
enforcement behavior itself (a limit of N blocks the N+1th concurrent invocation) was already
correct and is unchanged, only the field it reads moved.
Known gap noted, not fixed (out of scope for this finding): the real
PutFunctionScalingConfigInput/GetFunctionScalingConfigInput mark Qualifier as a required
member (a version/alias-scoped scaling config), but gopherstack's route
(/2025-11-30/functions/{name}/function-scaling-config) has no qualifier segment and the
backend stores one scaling config per function name regardless of qualifier. A real client must
still supply Qualifier (client-side SDK validation requires it), and gopherstack silently
ignores it rather than erroring or scoping by it. Worth a follow-up bd issue.
Gates: go build, go vet, go test -race -count=1, golangci-lint run — all clean
(./services/lambda/...).
Extracted ground truth from all 85 awsRestjson1_deserializeOpError<Op> switches
in lambda@v1.101.2/deserializers.go (REST-JSON, matched via strings.EqualFold
against X-Amzn-ErrorType/body __type) and diffed every literal exception-code
string used across services/lambda/*.go (both errors.go sentinels and
handler-inline h.writeError(...) literals) against both that per-op ground
truth and the 56 real shapes in lambda@v1.101.2/types/errors.go.
Unlike ecs this same sweep, lambda's codes were already disciplined: of 9
distinct literal exception-name strings hardcoded in handler files (outside the
errors.go sentinel table), only MethodNotAllowedException isn't a real
Lambda type -- and that one is a router-level HTTP-405 guard on unsupported
path/method combinations, not tied to any operation's error model (a real SDK
client can never trigger it), so it's out of this bug class and untouched.
2 bugs found and fixed, both the "two distinct exceptions are both modeled
by this exact op, and gopherstack always emits the wrong one" shape (same as
cloudformation's DescribeStackInstance this same sweep):
PutFunctionCodeSigningConfig(code_signing.go): when the function exists but the givenCodeSigningConfigArndoesn't, the backend returnedErrFunctionNotFound("ResourceNotFoundException") -- the function-not-found sentinel -- for the CSC-not-found case too (the handler's own error message literally said "Function or code signing config not found", indicating the two conditions were known but conflated). This op's own deserializer modelsCodeSigningConfigNotFoundExceptionas a distinct shape fromResourceNotFoundException; fixed the backend to returnErrCodeSigningConfigNotFoundfor this branch and the handler to map it to the correct wire code.GetProvisionedConcurrencyConfig(concurrency.go/handler_concurrency.go): the "config not found for this qualifier" branch already used a distinctly-named sentinel (ErrProvisionedConcurrencyConfigNotFound) but the handler mapped it to the genericResourceNotFoundExceptionwire code instead ofProvisionedConcurrencyConfigNotFoundException, which this op's own deserializer models as a separate shape.DeleteProvisionedConcurrencyConfiguses the same sentinel correctly -- its own deserializer does not model the specific exception, onlyResourceNotFoundException, so that call site was left unchanged (verified from its own switch, not assumed from the sibling).
Pre-existing test asserting the wrong behavior as correct (found and fixed,
same shape as the iam InvalidAction test): provisioned_concurrency_test.go's
TestGetProvisionedConcurrencyConfig/config_not_found asserted wantErrType: "ResourceNotFoundException"; updated to "ProvisionedConcurrencyConfigNotFoundException".
New tests: error_code_fixes_lambdasweep_test.go, both driving the real
aws-sdk-go-v2/service/lambda client and asserting via errors.As against the
SDK's own typed exception; both confirmed failing against the pre-fix code.
Gates: go build ./services/lambda/..., go vet ./services/lambda/... and
repo-wide go vet ./... (clean except a pre-existing, unrelated
services/appconfig failure from a concurrently-edited service), go test -race -count=1 ./services/lambda/... (pass), golangci-lint run --fix ./services/lambda/... (0 issues).
cmd/enumcheck was extended to see an enum value carried on a named
response struct's own composite literal, not only a map[string]any entry.
Run against services/lambda, it surfaced 5 needs-review findings, all
under an SDK-wide ambiguous wire key ("Status" or "Type" shared by
OperationStatus/ExecutionStatus/ProvisionedConcurrencyStatusEnum or
KafkaSchemaRegistryAuthType/OperationType/SourceAccessType in
lambda@v1.101.2/types/enums.go). Hand-checked against each site's true
field: ProvisionedConcurrencyConfig.Status = "READY" (legal
ProvisionedConcurrencyStatusEnumReady), DurableExecution.Status = "RUNNING" (legal ExecutionStatusRunning), DurableOperation.Type = "EXECUTION" (legal OperationTypeExecution), DurableOperation.Status = "STARTED" (legal OperationStatusStarted), TracingConfig.Mode = "PassThrough" (legal TracingModePassThrough). Every value is a real
member of its true single candidate; each only fails the ambiguous-key
tier's "legal in every candidate" check because the other enum(s) sharing
the wire key don't declare that member. No bug found; nothing changed in
this service.
Audited eventFilterMatches/patternMatchesObject/fieldMatchesRule/operatorMatches (event_filter.go) -- the FilterCriteria/Filter.Pattern event-pattern matcher shared by SQS/Kinesis/DynamoDB event source mappings -- against the real AWS Lambda "Filter rule syntax" comparison-operator table (docs.aws.amazon.com/lambda/latest/dg/ invocation-eventfiltering.html; the pinned SDK's types.go carries no prose for this family, FilterCriteria.Filters[].Pattern is a bare *string). 2 bugs, both under- matching:
$or("Or (multiple fields)" in AWS's own table, example"$or": [ {"Location":["New York"]}, {"Day":["Monday"]} ]) was not special-cased at all -- patternMatchesObject treated "$or" as a literal record field name, sovalue["$or"]was always absent and the clause could never match, silently discarding an entire documented operator. Fixed: patternMatchesObject now recognizes "$or", evaluating its array of sibling pattern fragments against the same value and ORing the results; a non-"$or" sibling key in the same object still ANDs against it normally.exists: AWS's own doc states plainly "the Exists operator only works on leaf nodes in your event source JSON. It doesn't match intermediate nodes," with a worked example ({"person":{"address":[{"exists":true}]}}does NOT match even thoughaddressis present, because its value is an object, not a leaf). existsMatches previously took only (arg, present bool) and had no way to see the field's value, so it matched purely on key-presence -- exists:true incorrectly matched an intermediate/nested-object field. Fixed: existsMatches now also takes fieldVal and returns false whenever the field is present but its value is a map[string]any (an intermediate node), matching the documented example exactly.
Gaps recorded, not fixed (documentation doesn't state these precisely enough to
implement without guessing): the page's own text says "Lambda supports the Amazon
EventBridge rules and uses the same syntax as EventBridge," but the page's
comparison-operator table lists only Null/Empty/Equals/Equals-ignore-case/And/Or/
$or/Not (anything-but)/Numeric/Exists/prefix/suffix -- no wildcard, no cidr, and
no nested anything-but forms ({"anything-but":{"prefix":...}} etc.) appear in that
table, even though EventBridge itself documents them. Whether Lambda's event
filtering actually honors those beyond the table is not stated on this page, so
wildcard/cidr remain unimplemented (singleOperatorMatches's default case returns
false, i.e. they always fail to match rather than being silently accepted) and a
nested-object arg to anything-but still falls through to
!scalarMatches(...) (always true, i.e. an unconditional match) rather than being
given real prefix/suffix/equals-ignore-case semantics -- left as-is rather than
fabricated.
New/changed tests (event_filter_test.go, table-driven, same TestLambda_EventFilterMatches
func): +4 cases (2 for $or matching/non-matching/AND-with-sibling, 1 for the
intermediate-node exists fix), all confirmed failing against unmodified code first
(2 actually fail pre-fix: "$or matches when second branch matches" and "exists true
does not match an intermediate object node"; the other 2 new $or cases pass either
way since their expected result is false under both the buggy and fixed logic, but
are kept as regression coverage for the AND-with-sibling-key and no-branch-matches
shapes). Assertion count: 26 -> 30 subtests, 0 dropped, all pre-existing cases
unchanged.
Gates: go build ./services/lambda/..., go vet ./services/lambda/... and repo-wide
go vet ./... (clean), go test -race -count=1 ./services/lambda/... (pass),
golangci-lint run ./services/lambda/... (0 issues).
Re-checked for damage from the handler-resolution defect fixed in
ef0eef041. Built the unpatched cmd/reqfieldscan/cmd/reqfielddiff from
ef0eef041~1 in a worktree, ran both five times against this package, and
diffed against HEAD.
cmd/reqfieldscan: byte-identical across all 5 old runs and HEAD.
cmd/reqfielddiff: findings ranged 234-240 across the 5 old runs (234 at
HEAD), 6 op.field keys moving: CreateFunctionUrlConfig/UpdateFunctionUrlConfig
.{AuthType,Cors} and CreateFunctionUrlConfig.InvokeMode, all present in
some old (misresolved) run and absent at HEAD. The collision is
FunctionUrlConfig/FunctionURLConfig: handleCreateFunctionURLConfig/
handleUpdateFunctionURLConfig (the real handlers) each fold onto a
same-named exported *InMemoryBackend method. Read both handler bodies
(handler_function_urls.go:14,160): AuthType and Cors are genuinely
read on both Create and Update; InvokeMode is genuinely read on Create.
Over-reporting, safe direction.
Real bug found and fixed while reading handleUpdateFunctionURLConfig
to settle the above (not itself one of the 6 moved keys -- reqfielddiff
never flagged it, because UpdateFunctionURLConfigInput simply had no
InvokeMode field to be undeclared against): UpdateFunctionUrlConfigInput.InvokeMode
(lambda@v1.101.2 api_op_UpdateFunctionUrlConfig.go:68) was never declared
on this package's UpdateFunctionURLConfigInput (models.go, only had
Cors/AuthType), so a function URL created BUFFERED could never be
switched to RESPONSE_STREAM (or back) after creation -- CreateFunctionUrlConfig
already supported the field correctly, which is why a spot-check of Create
alone would have said parity was fine. Fixed: added InvokeMode to
UpdateFunctionURLConfigInput, threaded it through
handleUpdateFunctionURLConfig into (*InMemoryBackend).UpdateFunctionURLConfig
(new 4th parameter, applied non-destructively like AuthType/Cors: only
overwrites when non-empty), same pattern as the pre-existing AuthType/Cors
fields. Single in-package caller of the backend method; no other callers to
fix.
New test TestUpdateFunctionUrlConfig_InvokeMode
(wire_field_fixes_test.go) drives the real aws-sdk-go-v2/service/lambda
client: Create with InvokeMode: BUFFERED, Update with
InvokeMode: RESPONSE_STREAM, asserts the SDK-decoded UpdateFunctionUrlConfigOutput
and a follow-up GetFunctionUrlConfig both show RESPONSE_STREAM.
Confirmed failing (asserted BUFFERED, i.e. the update was dropped)
against the pre-fix code before applying the fix.
Gates: go build ./services/lambda/..., go vet ./services/lambda/...
(clean), go test -race -count=1 ./services/lambda/... (pass, existing
suite unweakened, one new test/3 new assertions added),
golangci-lint run ./services/lambda/... (0 issues, no --fix used).
Queue derivation: real List* ops in lambda@v1.101.2 (14 total, lambda has zero
Describe* ops) whose full name never appears (case-insensitive, glob-expanded) verbatim
anywhere in this file. Mechanical grep gave 3: ListCapacityProviders,
ListEventSourceMappings, ListVersionsByFunction.
ListCapacityProviders field-diffed clean: types.CapacityProvider (lambda@v1.101.2
types/types.go) has no Name member at all (identity is CapacityProviderArn-only, real
AWS design, matching this file's existing UpdateCapacityProvider URI-label note) --
gopherstack's CapacityProvider model carries all 10 real members and none of the
json:"-"-internal ones leak onto the wire. Recorded, not fixed (different axis): real
ListCapacityProvidersInput declares Marker/MaxItems/State (pagination + a state
filter); Backend.ListCapacityProviders() takes no parameters and always returns every
provider on one page, unfiltered -- same "pagination/filter ignored" class already
catalogued elsewhere in this campaign, not a naming bug.
ListEventSourceMappings and ListVersionsByFunction were NOT clean -- both are the
Get-right/List-wrong sibling shape, and both share the item builder with their respective
singular/publish operations (so the bug reached every caller of that builder, not just the
List op):
-
**
FunctionVersion(shared byListVersionsByFunction/PublishVersion/GetFunction-by-version) silently dropped 8 real, backend-trackedtypes.FunctionConfigurationmembers that the siblingFunctionConfigurationtype (used byGetFunctionConfiguration) already carries correctly:Architectures/EphemeralStorage/LoggingConfig/MasterArn/StateReason/StateReasonCode/LastUpdateStatus/LastUpdateStatusReason. BothfnToVersionandpublishVersion(versions_aliases.go) buildFunctionVersiondirectly from a*FunctionConfigurationthat already has every one of these fields populated -- the source struct had the data, the conversion never copied it. Fixed: added all 8 fields toFunctionVersion(models.go) with the same json tagsFunctionConfigurationuses, and populated them in both builders. (Two realtypes.FunctionConfigurationmembers --LastUpdateStatusReasonCodeand several capacity/signing/tenancy fields -- are absent from gopherstack'sFunctionConfigurationtoo, i.e. a shared gap with no disagreement to detect; left as a recorded gap, not fixed this pass.ReservedConcurrentExecutions, present on gopherstack'sFunctionConfigurationbut not on the realtypes.FunctionConfigurationat all, is a separate, pre-existing possible issue on the Get side, out of this pass's List-sibling scope -- recorded, not touched.)FunctionVersionis part ofbackendSnapshot(persistence.go'sVersions map[string][]*FunctionVersion) -- the same struct serves both the wire and the persisted shape. The 8 new fields are purely additive (omitempty), soTestSnapshotVersionGuardcorrectly demanded a golden bookkeeping update rather than a version bump; ran with-update, confirmed the diff is additive-only, re-ran clean.Test:
TestListVersionsByFunction_SiblingFields_RealClient(wire_field_fixes_test.go), creates a function with all 8 fields set to distinguishable values viabk.CreateFunction, publishes two versions, asserts all 8 round-trip throughListVersionsByFunction's real SDK client for$LATESTand both published versions (3+ items). Verified failing pre-fix (Architectures/EphemeralStoragedecoded nil/empty). -
ListEventSourceMappings/CreateEventSourceMapping/GetEventSourceMapping(all sharingtoJSONESMResponse) never emittedLastModified, despiteEventSourceMapping.LastModified(event_source_mapping.go) being real, tracked state set at creation. Realtypes.EventSourceMappingConfiguration.LastModifieddecodes viasmithytime.ParseEpochSecondson a JSON Number (confirmed against lambda@v1.101.2deserializers.go'sawsRestjson1_deserializeDocumentEventSourceMappingConfigurationcase"LastModified") -- epoch-seconds, not RFC3339, the same timestamp-format bug class documented elsewhere in this campaign. Fixed: addedLastModified float64tojsonESMResponse(event_source_mapping.go), populated viaawstime.Epoch(m.LastModified)intoJSONESMResponse.golangci-lint run --fixadditionally reordered the struct forfieldalignmentand, as a side effect of that reorder, dropped all three pre-existing//nolint:llldirectives on this struct. That drop was WRONG -- all three lines still exceed 120 characters after realignment (128/126/123 chars; the AWS field names themselves are the width, not the column position), and a subsequent fullgolangci-lint run(without--fix, across all three services together) caught the regression: 3lllfindings on exactly those lines. Restored all three//nolint:lll // AWS field namedirectives by hand;golangci-lint runback to 0 issues. Recorded here because it is a small but concrete instance of this session's own "never trust an artefact's prior verification" mandate applying to a tool's own--fixoutput, not just to hand-written notes.Recorded, not fixed (different axis, genuine unmodeled gaps):
EventSourceMappingArn/FilterCriteriaError/KMSKeyArn/LoggingConfig/MetricsConfig/ProvisionedPollerConfig/ScalingConfig/StartingPositionTimestamp/StateTransitionReasonare realtypes.EventSourceMappingConfigurationmembers with no backing state in this backend'sEventSourceMappingmodel at all -- each would need new accept/store/read wiring, not a field-copy fix.Test:
TestListEventSourceMappings_LastModified_RealClient(wire_field_fixes_test.go), creates a mapping via the real SDK client, assertsLastModifiedround-trips (non-nil, after a pre-call timestamp) on bothCreateEventSourceMapping's response andListEventSourceMappings. Verified failing pre-fix (LastModifieddecoded nil on create).
Protocol: lambda is REST-JSON (awsRestjson1, confirmed from deserializers.go's function
prefix) -- no case folding, so any naming mismatch here is a hard failure class.
No wrapper-key mismatches, no hard decode errors/panics, no transpositions, no invented elements found this pass. Pages fetched: 0 (module cache used throughout).
Gates: go build ./... clean; go vet ./... clean;
go test -race -count=1 ./services/lambda/... clean; go test -race -count=1 -run TestSnapshotVersionGuard ./pkgs/persistence/ clean (after -update refreshed the
additive-only golden, confirmed with git diff --stat showing an 8-line addition only);
golangci-lint run ./services/transfer/... ./services/opensearch/... ./services/lambda/...
0 issues (one --fix pass for fieldalignment on event_source_mapping.go, scoped to that
file, plus a hand restoration of 3 nolint:lll directives --fix incorrectly dropped -- see
above). nolint directives in files touched this pass: event_source_mapping.go has 3
(//nolint:lll x3, all pre-existing and now confirmed still necessary). No nolint
directives in models.go, versions_aliases.go, or wire_field_fixes_test.go.
Five-dimension audit (wire compliance, LocalStack parity, cross-service integration,
performance, resource leaks) with explicit focus on dimension 5 given this package's
long-lived event-source pollers and per-function container lifecycle. Read
store.go/containers.go/runtime_api.go/invocation.go/event_source_poller.go/
janitor.go end to end tracing every goroutine's start/stop and every semaphore's
acquire/release. Both bugs found are in the resource-leak dimension; wire/protocol
compliance was largely spot-checked against this file's existing extensive coverage
rather than re-derived from scratch (see caveat in the final report).
-
cleanupTimedOutRuntime(containers.go) dropped cleanup entirely whenb.cleanupSem(cap 64) was saturated. The two sibling call sites with the identical "evict a runtime underb.mu, clean it up outside the lock via a bounded semaphore" shape --lookupOrRegisterRuntime's LRU eviction andUpdateFunction's runtime eviction -- both fall back to runningcleanupRuntimeinline when the semaphore is full, so the container/port/temp-dirs are always released.cleanupTimedOutRuntimeinstead had a baredefault: returnon the saturated branch: since the timed-outrthad already been deleted fromb.runtimesin the same function, nothing else in the codebase would ever revisit it, so the container was never stopped, the port never released, andzipDir/layerDirsnever removed -- a genuine leak of all three, exactly the "concurrency-limit semaphore never released on error paths" class this audit was scoped to look for. Fixed to match the other two call sites' inline fallback. -
A genuine async-invocation timeout never reached the function's DLQ/on-failure destination, and was never retried.
runAsyncInvocationRetryLoop(invocation.go) treatedwaitForAsyncResult'sok=falsereturn (container never responded withintimeout+containerResponseGracePeriod) as an unconditionalreturn-- skipping both the retry loop (ignoringMaximumRetryAttemptsentirely, even on attempt 0) and the terminaldispatchAsyncOutcomecall that delivers toDeadLetterConfig/DestinationConfig.OnFailure. Verified against AWS's own docs (docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html): "Function errors include errors returned by the function's code and errors returned by the function's runtime, such as timeouts" -- and "To capture records of failed invocations (such as timeouts or runtime errors), create an on-failure destination." A real client publishing an async function whose container hangs would see the event silently vanish with no DLQ/destination record and no retry, instead of the documented retry-then-deliver behavior. Fixed by folding the timeout case into the sameisErrorpath already used for a runtime-reported/errorresult, so a timeout is now retried (subject toMaximumRetryAttempts) and, once retries are exhausted, reaches the destination/DLQ withfunctionError: "Unhandled".Known related limitation, not fixed (deeper architecture question, out of this fix's scope): a retry scheduled after a timeout re-enqueues onto the same
runtimeServerwhose runtimecleanupTimedOutRuntimejust evicted, rather than re-resolving a fresh container viagetOrCreateRuntime-- so each retry attempt is guaranteed to also time out (burning a fulltimeout+graceper attempt) rather than potentially succeeding on a fresh execution environment the way real AWS's retry would. The end state (eventual destination/DLQ delivery) is now correct; only the retry attempts' chance of succeeding is not. This pre-existing "retry reuses the same runtimeServer" design applies equally to plain function-error retries and was not introduced or changed by this fix.
Both regression tests drive the real bug via existing repo helpers
(trackingDockerAPI/fakeAsyncDelivery) rather than new mocks:
TestCleanupTimedOutRuntime_SemSaturated_StillCleansUp (container_cleanup_test.go)
saturates b.cleanupSem via a new FillCleanupSem export helper and a new
CleanupTimedOutRuntimeForTest export wrapper, then asserts the container is still
stopped; TestEnqueueAsync_TimeoutDeliversToFailureDestination (async_invoke_test.go)
starts a runtime server that never answers /next and asserts the configured
OnFailure destination is eventually invoked. Both confirmed failing against the
pre-fix code (hand-reverted via git show HEAD:<path>, not git stash, to avoid
disturbing a concurrent agent's uncommitted work elsewhere in the worktree) and passing
after; restored.
No other bugs found this pass. Also reviewed, no bug found: enqueueAsyncInvocation's
fast/slow-path split and asyncEnqueueWaiters bound; dispatchInvocationLog's
logSem; EventSourcePoller's single-goroutine-for-the-whole-service design
(sweepStaleIterators correctly bounds shardIterators/sqsBatchBuffers growth on
delete); Close()'s poller-cancel + URL-server + runtime + asyncWG.Wait() shutdown
sequencing; handleInvoke's status codes (202 Event, 204 DryRun,
X-Amz-Function-Error/X-Amz-Executed-Version/X-Amz-Log-Result headers) against
awsRestjson1_deserializeOpHttpBindingsInvokeOutput/...DocumentInvokeOutput
(lambda@v1.101.2 deserializers.go:8744-8793) -- StatusCode is read verbatim from the
HTTP response and all four headers are read exactly as this package sets them, no
mismatch.
FIXED (gopherstack-neiq, 2026-09-08): the observation below was correct on the
mechanism but the follow-up ("root-causing which specific test(s) leave an idle
connection open ... is a separate, broader change") turned out to be tractable.
http.DefaultClient.Do (iam_enforcement_test.go:166,
handler_runtime_test.go:290,442,471,489,895) and zero-Transport
&http.Client{Timeout: ...} literals (handler_runtime_test.go:757,
store_test.go:696,787,854) all shared http.DefaultTransport's idle-connection
pool; response bodies were closed correctly but the parked
persistConn.readLoop/writeLoop (client side) and (*conn).serve (server side,
observed in this pass's own reproductions) goroutines raced TestMain's
goleak.VerifyTestMain sample. Fixed by giving every such test its own
http.Client{Transport: &http.Transport{}} via a new newHTTPClient(t, timeout)
helper (test_helpers_test.go) that registers t.Cleanup(c.CloseIdleConnections);
iam_enforcement_test.go uses srv.Client() instead (idiomatic since an
httptest.Server is already in play) -- confirmed by reading
net/http/httptest/server.go that Server.Close (already deferred via
t.Cleanup(srv.Close)) calls s.client.Transport.CloseIdleConnections() whenever
s.client is non-nil, so no extra close was needed there. Purely a client
construction swap -- no test assertion changed.
Verified with process-level repeat runs (each iteration a separate go test
process, since the leak is about process teardown): before the fix, 3/25 runs
(12%) failed on a genuine goleak leaked-goroutine report; after the fix, 0/30
runs failed on goleak (30 clean). The remaining flakiness this pass's own
baseline turned up -- a pre-existing, unrelated data race in
(*mockBackend).InvokeFunction (handler_test.go:110, unguarded
m.invokeCount++ raced across TestExtractOperation_SDKRouteTable's parallel
subtests, 4/30 post-fix runs) and one bind: address already in use port
collision -- are untouched by this fix and confirmed present in the unfixed
baseline too; out of this issue's scope, flagged for a follow-up issue.
The issue's suspected explanation for the branch's reported higher rate (60% vs
28% on main, "several hundred lines of new tests ... widening goleak's sampling
window") does NOT hold up under same-session remeasurement: this pass's own
goleak-only rate was 3/25 (12%) on this branch and 4/20 (20%) on origin/main
-- comparable, with main if anything higher, the opposite direction the
hypothesis predicts. The branch's new test files were confirmed (via grep) to
add zero new HTTP client/server call sites, and the branch's total _test.go
line count (23,762 -> 24,106) is only a 1.4% increase -- too small to plausibly
double a sampling-window-width effect even if that mechanism were real. The
60%/28% figures do not reproduce; environmental run-to-run variance (this VM's
load at measurement time) is the more likely explanation, not a structural
branch effect. Refuted, not confirmed.
Gates: go build ./..., go vet ./services/lambda/..., gofmt -l services/lambda/
(no output), golangci-lint run ./services/lambda/... (0 issues), go test -race -count=1 ./services/lambda/... (0/30 goleak failures after the fix; see above for
the 4 unrelated pre-existing failures).
Confirmed real AWS validates an Image package-type function's Code.ImageUri at
CreateFunction/UpdateFunctionCode time, not only at pull time: a nonexistent
repository or tag is rejected immediately with InvalidParameterValueException
("Source image <uri> does not exist. Provide a valid source image."), reproduced
in multiple public bug reports (e.g. aws/aws-cdk#24648, aws/aws-sdk-go#3736). The
issue's own "possibly structural" caveat does not hold.
Added an optional cross-service seam, following services/networkmanager's
EC2Resolver pattern exactly: ECRResolver (crossservice.go, one method,
ResolveImage(imageURI string) bool), a SetECRResolver setter and ecrResolver
field on InMemoryBackend (store.go), and an ImageURIResolver optional
extension of StorageBackend (alongside the existing QualifierResolver etc.)
that Handler.validateImageURIResolves type-asserts h.Backend against before
accepting a Image-type CreateFunction/UpdateFunctionCode call
(handler_functions.go). A nil resolver -- the default, and every existing test's
backend -- accepts every ImageUri unvalidated, matching this repo's convention
for unwired cross-service checks and preserving every pre-existing test that
creates an Image function with an arbitrary ImageUri like "x".
Not wired up: cli.go is out of scope for this pass (owned by a concurrent
agent). Wiring needs a wireLambdaECR(lambdaReg, ecrReg service.Registerable)
function mirroring wireLambdaS3/wireLambdaCWLogs, plus an adapter type (e.g.
lambdaECRResolverAdapter) whose ResolveImage(imageURI string) bool parses the
account/region/repository/tag-or-digest out of the ECR-style URI and calls
services/ecr's already-exported InMemoryBackend.DescribeImages(ctx, repositoryName, []ImageIdentifier{...}), returning true only on a nil error. No changes needed in
services/ecr itself -- DescribeImages already returns ErrRepositoryNotFound/
ErrImageNotFound for exactly this check. Call site: alongside the other
wireLambda* calls in cli.go's service-wiring sequence.
Regression tests (ecr_resolver_test.go): a fake ECRResolver proves both
directions (unknown image rejected with InvalidParameterValueException and not
persisted; known image accepted) for both CreateFunction and
UpdateFunctionCode, plus a no-resolver-wired test documenting the accept-all
default is unchanged. Each guard was neutered individually and confirmed to make
exactly its own regression test fail (and no others).
2026-09-08: Invoke reached the backend on a rejected header (gopherstack-3t96, P2) -- found and fixed
Part of the sweep following elasticache (gopherstack-8haq, P1), pinpoint (gopherstack-246v),
and apigatewayv2 (gopherstack-wsvb, P1). validateInvocationHeaders (handler_invocation.go:18)
rejected an invalid X-Amz-Invocation-Type or X-Amz-Log-Type header by writing the 400 via
h.writeError and returning that call's result, which is nil after a successful write.
handleInvoke stored that nil in valErr and tested if valErr != nil, which never fired, so
the function was invoked anyway and a second response (the invocation result) was written on top
of the already-committed 400.
Tests first. Added invokeCount to mockBackend (handler_test.go) -- incremented on every
InvokeFunction call -- and two new TestInvoke cases, invalid_invocation_type_not_invoked and
invalid_log_type_not_invoked, asserting bk.invokeCount == 0, not just the response status: no
pre-existing test covered either rejection branch at all (only valid Event/DryRun values were
exercised). Confirmed both FAIL against unmodified code (verbatim, go test ./services/lambda/... -run TestInvoke):
{"level":"ERROR","msg":"echo: response already written to client"}
handler_test.go:785:
Error: Not equal:
expected: 0
actual : 1
Test: TestInvoke/invalid_invocation_type_not_invoked
Messages: a rejected invocation header must not reach the backend
handler_test.go:789:
Error: Received unexpected error:
invalid character '{' after top-level value
Test: TestInvoke/invalid_invocation_type_not_invoked
--- FAIL: TestInvoke/invalid_invocation_type_not_invoked (0.00s)
--- FAIL: TestInvoke/invalid_log_type_not_invoked (0.00s)
The second failure is the double write corrupting the wire body: the mock's {"answer":42}-shaped
result (for request_response-style success) got concatenated onto the already-written 400 JSON,
so the response body was no longer valid JSON on its own -- worse than a clean second response,
since a real client's JSON decoder chokes on it outright rather than seeing a plausible-but-wrong
payload.
Fixed with the pinpoint raw-unwritten-error pattern: validateInvocationHeaders no longer writes;
it returns one of two new unexported static errors (errInvalidInvocationType, errInvalidLogType,
handler_invocation.go), and handleInvoke maps any non-nil error to InvalidParameterValueException/
400 via h.writeError(c, http.StatusBadRequest, "InvalidParameterValueException", valErr.Error())
and writes exactly once. errname/err113 (this repo's golangci-lint config) require these as
static package-level sentinels, not inline errors.New at each call site.
Neuter-verified by reverting handleInvoke's call site back to bare return valErr (in effect a
"forgot the write" regression, distinct from the original bug but on the same guard) and separately
back to the exact original write-then-return-nil shape: both still compile, and the latter
reproduces the expected 0 actual 1 failures above verbatim.
Swept the rest of the package for the same shape (a *echo.Context-taking helper whose result is
stored and re-checked by a caller, rather than the handler's own final return h.writeError(...)):
the seven validate* helpers in handler_functions.go (validateQualifier,
validateCreateFunctionInput, validateSnapStartInput, validateEphemeralStorageInput,
validateMemoryAndTimeout, validateImageURIResolves, validateCreateFunctionCode) already use a
different, safe convention: they return bool and explicitly discard h.writeError's result
(_ = h.writeError(...); return false), so their callers' if !h.validateXxx(...) { return nil }
checks are not fooled by a nil error value. dispatchSpecialRoutes (handler_dispatch.go:552)
returns (bool, error) but is a pure pass-through dispatcher whose result is Echo's own terminal
handler return, never stored and re-checked. No other instance found.
go test -race ./services/lambda/... and golangci-lint run ./services/lambda/... both clean
after the fix. Full go test ./services/... also green (see gopherstack-3t96's cross-service
report for the combined blast-radius run covering lambda, securityhub, and organizations).
2026-09-09: mockBackend.InvokeFunction data race (gopherstack-cedf, P2) -- test-only, found and fixed
(*mockBackend).InvokeFunction (handler_test.go:104) did m.invokeCount++ and then read
m.functions[name] while holding no lock, the sole outlier among the struct's methods --
every sibling (CreateFunction, GetFunction, ListFunctions, DeleteFunction,
UpdateFunction) takes m.mu correctly. TestExtractOperation_SDKRouteTable
(handler_paths_sdk_diff_test.go) shares one mockBackend across its parallel subtests and
drives one of them through the invoke path, so concurrent subtests raced on invokeCount.
Pre-existing and unrelated to any recent change; first surfaced (4/30 runs) by the
gopherstack-9zx goleak-fix pass's own baseline measurement, tracked separately as this issue.
Test first. Added TestMockBackend_InvokeFunction_ConcurrentAccess (handler_test.go): 50
goroutines call InvokeFunction on one shared mockBackend, joined by a sync.WaitGroup, then
the final count is asserted -- a deterministic reproducer rather than relying on the existing
parallel subtests to happen to interleave. Confirmed failing against the unmodified method,
verbatim (go test -race -count=1 -run TestMockBackend_InvokeFunction_ConcurrentAccess ./services/lambda/...):
==================
WARNING: DATA RACE
Read at 0x00c000144170 by goroutine 47:
github.com/blackbirdworks/gopherstack/services/lambda_test.(*mockBackend).InvokeFunction()
/home/agbishop/gopherstack/services/lambda/handler_test.go:110 +0x9a
github.com/blackbirdworks/gopherstack/services/lambda_test.TestMockBackend_InvokeFunction_ConcurrentAccess.func1()
/home/agbishop/gopherstack/services/lambda/handler_test.go:825 +0xd9
Previous write at 0x00c000144170 by goroutine 45:
github.com/blackbirdworks/gopherstack/services/lambda_test.(*mockBackend).InvokeFunction()
/home/agbishop/gopherstack/services/lambda/handler_test.go:110 +0xb2
github.com/blackbirdworks/gopherstack/services/lambda_test.TestMockBackend_InvokeFunction_ConcurrentAccess.func1()
/home/agbishop/gopherstack/services/lambda/handler_test.go:825 +0xd9
==================
--- FAIL: TestMockBackend_InvokeFunction_ConcurrentAccess (0.00s)
testing.go:1865: race detected during execution of test
FAIL
20/20 separate process runs failed pre-fix (reliable, not occasional).
Fixed by taking m.mu.Lock() with a deferred Unlock() at the top of InvokeFunction, matching
its siblings -- it both writes invokeCount and reads m.functions, so the write lock is
required, not RLock. The method has several early returns, so the deferred-unlock shape (not a
manual unlock before each return) is what keeps every path covered.
Line 785's assert.Equal(t, 0, bk.invokeCount, ...) (in TestInvoke, unrelated to the shared-
backend race above -- each TestInvoke subtest gets its own mockBackend via newHandler) still
read the field directly with no lock: locking inside InvokeFunction alone does not make an
external, unlocked read of the same field race-free against any in-flight call. Added a locked
accessor, (*mockBackend).InvokeCount() int (m.mu.RLock/RUnlock), and pointed line 785 at it.
Grepped the rest of the package for other direct field access on mockBackend: every bk.functions[...] = ... setup call (handler_test.go, invocation_test.go) runs synchronously in a subtest's own
setup/setupMock callback before the handler is invoked, against a mockBackend not shared with
any other goroutine -- no lock needed there. TestHandleInvokeWithResponseStream_EventStreamEncoding
(invocation_test.go:568-572) already wraps its direct mb.functions[...]/mb.invokeResult writes in
mb.mu.Lock()/Unlock(); left as-is.
No pre-existing assertion was weakened or removed -- line 785 still checks the same invariant (a rejected invocation header must never reach the backend), just through a locked accessor.
Gates after the fix: golangci-lint run ./services/lambda/... 0 issues; 25 separate process
invocations of go test -p 1 -race -count=1 ./services/lambda/... (a shell loop, not
-count=25), 0/25 failures.
lambda: 45/85 (52.9%) -> 76/85 (89.4%) typed-covered (40 -> 9 uncovered),
31 ops newly covered, realclient_config_and_invocation_test.go added (one outer
t.Parallel() test, 13 subtests covering every named priority family:
permissions, aliases, code signing configs, concurrency, event invoke
config, function URL configs, account settings, layers, recursion config,
runtime management config, invoke (dry-run + legacy async; see below),
invoke-with-response-stream, event source mapping update). Three real
bugs found and fixed:
-
GetFunctionConcurrency404'd withResourceNotFoundExceptionwhen a function had no reserved-concurrency configuration set, but realGetFunctionConcurrencyOutputdocuments no such error for this state --ReservedConcurrentExecutionsis simply an optional field that comes back null. A real client'sGetFunctionConcurrencyon any function in its default (no-reservation) state always failed instead of decodingnil. Fixed to return 200 with the key omitted; three pre-existing tests that encoded the wrong 404 expectation (TestGetFunctionConcurrency,TestConcurrency_PutGetDelete,TestDeleteFunction_ClearsSideState) corrected to the real semantics, not weakened. -
STRUCTURAL, HIGH BLAST RADIUS: the entire
FunctionEventInvokeConfigfamily (Put/Get/Update/Delete/List, 5 ops) was completely unreachable by any real SDK client.lambdaFunctionPrefixes/lambdaPathPrefixes(handler_paths.go) hadlambda2019PathPrefix = "/2019-09-30/functions"(the real date forGetFunctionConcurrency/provisioned-concurrency) but no entry at all for/2019-09-25/functions-- the real, distinct date for this whole family (confirmed against each op's ownawsRestjson1_serializeOp*FunctionEventInvokeConfig*, lambda@v1.107.0 serializers.go), five days off. Every real request 404'd before even reaching lambda's own router (isLambdaPathdidn't recognize the path). Fixed by addinglambda2019EventInvokeConfigPathPrefixand registering it in both prefix tables. -
InvokeWithResponseStreamnever readX-Amz-Invocation-Typeat all (a real, optional header supportingRequestResponse/DryRun, same binding as plainInvoke-- confirmed againstawsRestjson1_serializeOpHttpBindingsInvokeWithResponseStreamInput) -- aDryRunrequest from a real client always ran a full invocation instead of validating only. Fixed to honor the header and return 204 immediately forDryRun, matchingInvoke's existing behavior.
Remaining 9 uncovered ops are the entire DurableExecution family
(CheckpointDurableExecution, GetDurableExecution/-History/-State,
ListDurableExecutionsByFunction, SendDurableExecutionCallback{Success,
Failure,Heartbeat}, StopDurableExecution) -- not a named priority family
for this slice, deliberately not attempted; the wire shapes were already
rewritten and field-diffed in an earlier pass (see the durable_execution
family note above), so this is a coverage gap only, not a suspected bug.
Real RequestResponse-type Invoke/InvokeWithResponseStream (i.e. an
actual container execution, not DryRun) could not be typed-covered in
this unit-test harness either: this backend requires a real Docker
port allocator (b.portAlloc/b.docker), which every other unit test in
this package also deliberately constructs as nil ("no real HTTP servers
in unit tests") -- genuine sync execution is test/integration's job, out
of scope for this pass.
Gates: go build ./... (whole module, clean), go vet,
golangci-lint run --new-from-rev=HEAD (0 issues), go test -race -count=1 (all pass, including the three corrected pre-existing tests).
pkgs/persistence's TestSnapshotVersionGuard clean (no persisted-struct
fields changed by any of the three fixes). No version bump.
Drove the entire DurableExecution family (9 ops named as the prior entry's
remaining coverage gap above) through the real aws-sdk-go-v2 client for
the first time (realclient_durable_execution_test.go, 3 subtests: checkpoint
lifecycle, callbacks, list-by-function). CheckpointDurableExecution is
the only creation path available outside a real Docker Invoke, used as
documented rather than a fabricated backdoor.
One real wire-shape bug, found and fixed: StopDurableExecutionInput.Error
and SendDurableExecutionCallbackFailureInput.Error are each the
request's entire top-level JSON body (confirmed against serializers.go's
awsRestjson1_serializeOpStopDurableExecution /
-CallbackFailure, both of which stream
awsRestjson1_serializeDocumentErrorObject's output directly as the
body — ErrorType/ErrorMessage/etc. at the top level, never wrapped).
Both handlers instead unmarshalled into a struct{ Error *ErrorObject }-shaped wrapper expecting a nonexistent "Error" key, so a real
client's Error was silently dropped every time; StopDurableExecution
always fell back to its generic default "Stopped" / "The durable
execution was stopped by StopDurableExecution." message regardless of
what the caller sent. Fixed both handlers to parse the body directly as
*ErrorObject (new isEmptyErrorObject helper distinguishes a real
client's {} no-error body from an actual populated one, preserving the
existing default-message behavior for a bare stop). Two pre-existing
tests (durable_execution_test.go's TestDurableExecution_StopVariants
and TestDurableExecution_CallbackFailure) had hand-crafted the wrong,
wrapped {"Error":{...}} body — only the pre-fix bug made that shape
"work" — corrected to the real unwrapped shape.
SendDurableExecutionCallbackFailure's fix is not independently
observable: CallbackFailedDetails is one of the ~19 type-specific Event
*Details structs this emulator deliberately never populates (see the
durable_execution family note), so the parsed Error still has no
response field to surface in — fixed for wire-shape consistency with
StopDurableExecution's identical bug, not because a test can assert a
difference. Hand-reverted handler_durable_execution.go and
durable_execution.go to HEAD, re-ran the new test — reproduced the
exact "Stopped" / generic-message failure verbatim — then restored the
fix byte-identical (diffed clean) and reconfirmed passing.
Accept-and-drop / architecture-gap finding, disclosed in
items_still_open, not fixed: ListDurableExecutionsByFunction
permanently returns zero DurableExecutions for any function on this
backend. Root cause: DurableExecution.FunctionARN (durable_execution.go)
is declared and read (matchesListFilter) but never assigned anywhere in
the package — confirmed by grepping every FunctionARN reference in
services/lambda, all reads, zero writes. CheckpointDurableExecution,
the only creation path, carries no function identity in its request at
all (DurableExecutionArn alone, deliberately treated as
"client-opaque, server-never-parses-structure-from-it" per
deriveDurableExecutionName's own doc comment), so there is nothing to
derive FunctionARN from without the same Invoke-entry-point rewiring
the existing durable_execution family note already defers as
out-of-scope. The op still round-trips correctly through the real SDK
(empty list, no decode error) — proven in this slice's list by function
subtest — so this is a functional/architecture gap, not a wire-shape bug;
added to items_still_open since it was previously undisclosed there
(the family note only covered GetDurableExecutionOutput.FunctionArn
being empty on one execution, not that the List op can never match any).
Gates: go build ./... (whole module, clean). go vet clean. go test -race -count=1 ./services/lambda/... clean (including the two corrected
pre-existing tests). golangci-lint run --new-from-rev=HEAD 0 issues
(one justified //nolint:exhaustive on a 2-case switch over
types.EventType in the new test — that op family produces ~20 real
enum values, only 2 are relevant here). go run ./cmd/paritylint: 0 FAIL
throughout. No snapshot_inventory.json changes (durable_execution is
not wired into persistence — pre-existing, documented in Notes above).
No version bump.
Worked all 26 tier-1 findings from cmd/reqfielddiff -dir lambda. This
service is restjson1; every finding below was checked against the pinned
aws-sdk-go-v2/service/lambda@v1.107.0 serializer to confirm whether the
field is httpQuery/httpLabel/httpHeader or body before deciding where (and
whether) it should be read.
21 false positives, two distinct shapes:
- Query-param blind spot (gopherstack-99nj):
GetDurableExecution.IncludeExecutionData,GetDurableExecutionHistory.IncludeExecutionData/.MaxItems/.ReverseOrder,GetDurableExecutionState.MaxItems,ListDurableExecutionsByFunction.MaxItems/.ReverseOrderare all httpQuery per the SDK serializer, and this package reads them viac.Request().URL.Query().Get(...)/parsePaginationParams(c.Request())(handler_durable_execution.go,handler_functions.go) -- a different literal shape fromc.QueryParambut the same class of tool blind spot. - httpHeader binding, already read from the exact header:
Invoke.InvocationType(X-Amz-Invocation-Type,handler_invocation.go:257) andInvokeWithResponseStream.InvocationType(same header,handler_invocation.go:257, with its own comment citing the serializer). - Plain tool misses on already-declared-and-applied body fields (type
mismatch likely confuses the detector --
int/boolhere vs*int32/*boolon the real SDK):CreateEventSourceMapping/UpdateEventSourceMapping'sBatchSize/Enabled/MaximumBatchingWindowInSeconds/MaximumRecordAgeInSeconds/MaximumRetryAttempts(event_source_mapping.go, handler_event_source_mappings.go),PutFunctionRecursionConfig.RecursiveLoop,PutRuntimeManagementConfig.UpdateRuntimeOn.
3 dropped parameters fixed (all restjson1 body fields):
CreateEventSourceMapping.KMSKeyArn/UpdateEventSourceMapping.KMSKeyArn-- decoded nowhere;EventSourceMappinghad no field for it at all. AddedEventSourceMapping.KMSKeyArn, threaded throughCreateEventSourceMappingInput/UpdateEventSourceMappingInput/applyESMUpdate, and echoed onjsonESMResponse(KMSKeyArn, matchingtypes.EventSourceMappingConfiguration.KMSKeyArn). Observable via Create/Update's own response and a follow-upGetEventSourceMapping.UpdateFunctionCode.S3ObjectStorageMode-- decoded nowhere. AddedFunctionConfiguration.S3ObjectStorageMode(internal bookkeeping,json:"-", defaults toCOPYwhen omitted per the documented default) and a newFunctionCodeLocation.ResolvedS3Object/ResolvedS3Objectstruct (matching the real, optionaltypes.ResolvedS3Object-- S3Bucket/S3Key -- populated only inREFERENCEmode, confirmed againsttypes.FunctionCodeLocation).UpdateFunctionCodeOutputitself carries noCodemember on the real API (it isFunctionConfiguration-shaped, notGetFunctionOutput-shaped), so the fix is observable via a follow-upGetFunctiononly, not on Update's own response -- confirmed by hand againstapi_op_UpdateFunctionCode.go's output field list before writing the test this way, not assumed.
2 recorded as gaps tied to a single existing, already-disclosed root
cause (extended items_still_open's existing entry rather than
duplicating it): Invoke.DurableExecutionName (an httpHeader,
X-Amz-Durable-Execution-Name, read nowhere -- Invoke has zero
durable-execution awareness, and InvokeOutput.DurableExecutionArn doesn't
exist anywhere in this package's response shape) and
ListDurableExecutionsByFunction.Qualifier (httpQuery, unobservable
because DurableExecution.Version is declared but never assigned, for the
identical "no Invoke entry point" reason PARITY.md already documents for
ListDurableExecutionsByFunction's FunctionARN gap). Both would require
the same out-of-scope Invoke rewiring already deferred there -- not
standalone one-field fixes.
Proof: realclient_event_source_mapping_kms_test.go, driving the real
aws-sdk-go-v2/service/lambda typed client (newTestLambdaClient, shared
with realclient_durable_execution_test.go).
Gates: go build ./services/lambda/..., go vet ./services/lambda/...,
go test -race -count=1 ./services/lambda/..., golangci-lint run --new-from-rev=HEAD ./services/lambda/... (0 issues). No persisted-struct
field changed (EventSourceMapping.KMSKeyArn/
FunctionConfiguration.S3ObjectStorageMode are both new fields on structs
already in the persistence snapshot -- see gate results below for the
pkgs/persistence inventory rows added), no version bump. go build ./.../go vet ./... at repo root currently fail, but only in
services/quicksight (GetDashboardEmbedURL arity mismatch) -- confirmed
via git status as a concurrent sibling agent's uncommitted in-progress
edit (16 modified quicksight files), not touched by this pass.