Skip to content

Improve search validation, date handling, and client-cache headers - #187

Open
afoote-mitre wants to merge 7 commits into
devfrom
af/180-search-consistency
Open

afoote-mitre wants to merge 7 commits into
devfrom
af/180-search-consistency

Conversation

@afoote-mitre

@afoote-mitre afoote-mitre commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Address items 1-4 of #180 for POST /search: consistent error responses, precise page-size validation, consistent boolean inputs, and more useful date formats. Also implement the approved client-cache response policy from item 7 for both search endpoints.

The additional missing-CVE-ID warning feature from item 5 has been removed following team review. Item 6 and the remaining parts of item 7 are not being adopted in this PR. The issue should remain open for tracking the team's decisions rather than being automatically closed by this change.

Why These Changes Were Needed

Change Why it was needed Primary files
Return a consistent single error object Router validation previously returned an array while middleware and server errors returned an object, even for the same field. /search now consistently returns { "error": "CODE", "message": "Description" }, using the first error when multiple validations fail. Case-insensitive and trailing-slash route variants use the same shape. /webSearch error handling is unchanged. routes/index.js, routes/index.test.js, api-docs/openapi.test.js, README.md, test/postman/buildCollection.js
Distinguish invalid page sizes from values above the maximum Fractional or nonnumeric values previously received a misleading maximum-limit error. Invalid types, fractional values, and nonpositive values now return RESULTS_PER_PAGE_INT; positive integers above 500 return MAX_RESULTS_LIMIT. Valid integer strings remain accepted; arrays and booleans are rejected before coercion. routes/index.js, routes/index.test.js, README.md, test/postman/buildCollection.js
Normalize all boolean filters consistently knownExploited and disputed previously rejected string booleans accepted by the other filters. All five accept JSON booleans and lowercase "true"/"false" strings, normalizing strings before downstream checks. Numbers, null, arrays, and other strings remain invalid. Omitted filters retain their existing behavior. routes/index.js, routes/index.test.js, routes/swagger.js, api-docs/openapi.test.js, README.md, test/postman/buildCollection.js
Accept useful date representations without weakening calendar validation Requiring milliseconds and a literal UTC suffix rejected date-only and explicit-offset inputs. A shared normalizer accepts YYYY-MM-DD and timestamps with seconds, an explicit timezone, and up to three fractional-second digits. Date-only starts become midnight UTC; date-only ends include the whole day. Explicit timestamps retain their exact instant. Range ordering is checked after normalization; impossible dates, timezone-less timestamps, leap seconds, and finer precision remain invalid. Applies to publication, modification, SSVC, and KEV boundaries. utils/searchDates.js, utils/searchDates.test.js, routes/index.js, controllers/middleware.js, utils/errors.js, associated tests, routes/swagger.js, README.md
Add configurable client-cache freshness headers Successful HTTP 200 responses from /search and /webSearch, including empty results, now advertise Cache-Control: private, max-age=60. Error and health/readiness responses use no-store. SearchClientCacheMaxAgeSeconds changes the duration; zero or invalid values produce no-store. This provides explicit client guidance without introducing an API result cache or quota counter. utils/clientCachePolicy.js, server.js, controllers/searchController.js, config files, .env.example, README.md, routes/index.js, routes/swagger.js, generated OpenAPI, and associated tests
Remove the additional missing-ID warning feature The team decided not to adopt the existence-verification feature. Remove its helper, query builder, controller lookup, and feature-specific tests/documentation. Searches no longer perform an auxiliary availability lookup or emit CVE_IDS_NOT_FOUND / CVE_ID_AVAILABILITY_UNVERIFIED. Regression coverage confirms unmatched IDs do not prevent matching records from returning or cause an extra request. controllers/searchController.js, utils/searchQueryBuilder.js, deleted utils/searchCveIdWarnings.js and its tests, routes/search.integration.test.js, routes/swagger.js, README.md, Postman source/collection

CVE ID Behavior

  • Syntactically valid IDs with no matching records contribute zero results without an additional missing-ID warning or existence lookup.
  • Other requested IDs can still return their matching records.
  • The pre-existing CVE_IDS_NOT_RETURNED warning for pre-1999 IDs remains unchanged. This is separate from the removed feature.
  • Existing semantic-search and other request warnings remain unchanged and continue to appear before records.

Date Input Examples

{
  "pubStartDate": "2024-01-01",
  "pubEndDate": "2024-12-31",
  "lastModStartDate": "2024-01-01T01:00:00+01:00"
}

The publication boundaries normalize to 2024-01-01T00:00:00.000Z and 2024-12-31T23:59:59.999Z. The modification boundary normalizes to 2024-01-01T00:00:00.000Z. All boundaries are inclusive.

Client-Cache Behavior

Both search endpoints use POST. Ordinary HTTP caches do not automatically reuse a cached POST response for another POST. Integrators must implement application-level caching keyed by API environment, endpoint, and the complete request body, including filters, scope, sorting, and pagination. The freshness header is a hint, not a requirement that forces clients to cache. No server-side cache, queue, request coalescing, rate accounting, or additional OpenSearch operation is introduced.

Cached results can add up to the configured freshness duration of staleness and do not guarantee indexing freshness or consistency across separately cached pages. Clients should not cache errors or reuse historical quota headers as current accounting. Existing ETags do not add conditional POST/304 behavior.

Deferred or Rejected Items

  • Item 5: Additional missing-CVE-ID warnings. The implementation and auxiliary lookup have been removed. Existing pre-1999 warnings remain.
  • Item 6: Input-echoing error messages. No changes to severity errors that repeat caller input.
  • Item 7: Remaining response-header changes. The client-cache policy is now implemented above. Rate enforcement remains in AWS WAF; this PR does not add rate-limit headers or fabricate remaining/reset quotas. X-Powered-By and User-Agent behavior remain unchanged.

@afoote-mitre
afoote-mitre force-pushed the af/180-search-consistency branch from ff274f8 to 6d12fdd Compare October 20, 2026 15:22
@afoote-mitre afoote-mitre self-assigned this Oct 28, 2026
@afoote-mitre afoote-mitre changed the title Improve /search validation consistency and missing-ID warnings Improve /search validation consistency and date handling Oct 28, 2026
@afoote-mitre afoote-mitre changed the title Improve /search validation consistency and date handling Improve search validation, date handling, and client-cache headers Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant