Improve search validation, date handling, and client-cache headers

.env.example

@@ -5,6 +5,7 @@ OpenSearchDomainEndpoint=
 SearchOpenSearchRequestTimeoutMs=20000
 SearchRequestDeadlineMs=30000
 SearchReadinessTimeoutMs=5000
+SearchClientCacheMaxAgeSeconds=60
 RequestLogSuccessSampleRate=0.1
 RequestLogHealthChecks=false
 WebSearchEnabled=true

README.md

@@ -13,6 +13,7 @@
     - [Virtual Match String Search](#virtual-match-string-search)
     - [Virtual Match String Version Ranges](#virtual-match-string-version-ranges)
 - [Raw Query Endpoint](#raw-query-endpoint)
+- [Client Result Caching](#client-result-caching)
 - [Swagger API docs](#swagger-api-docs)
 - [Testing](#testing)
 - [Operational Logging](#operational-logging)
@@ -38,6 +39,7 @@ Copy `.env.example` to an untracked `.env` and replace or check the following va
 - `SearchOpenSearchRequestTimeoutMs` - optional timeout for each OpenSearch request, default `20000`
 - `SearchRequestDeadlineMs` - optional overall `/search` processing deadline, default `30000`
 - `SearchReadinessTimeoutMs` - optional timeout for the OpenSearch index readiness check, default `5000`
+- `SearchClientCacheMaxAgeSeconds` - client result-cache freshness hint in seconds, default `60`; `0` disables reuse with `no-store`. Invalid values also produce `no-store`.
 - `RequestLogSuccessSampleRate` - optional fraction of successful access logs to retain, default `0.1`; includes health checks when they are enabled for logging
 - `RequestLogHealthChecks` - optional flag to include successful health checks in access-log sampling, default `false`
 - `WebSearchEnabled` - optional `/webSearch` availability flag, default `true`
@@ -123,17 +125,21 @@ When the configured index is an alias spanning multiple indices, every reported
 
 JSON request bodies for both search endpoints are limited to 100 KiB (102,400 bytes, measured after decompression). Larger bodies return HTTP 413 `REQUEST_BODY_TOO_LARGE`. Parser errors return sanitized JSON rather than HTML or stack traces: malformed JSON returns HTTP 400 `MALFORMED_JSON`, other unreadable bodies return HTTP 400 `INVALID_REQUEST`, and unsupported encoding returns HTTP 415 (`SEARCH_CONTENT_TYPE` for `/search`, `REQUEST_ENCODING_UNSUPPORTED` for `/webSearch`). Unexpected server errors return a generic `INTERNAL_SERVER_ERROR` response without exception details, including outside production mode.
 
-`POST /search` accepts an optional JSON object request body and validates supported fields, boolean parameters, pagination values, CVSS severity values, required parameter combinations, CPE Name requirements, and date-time formats. Enum-valued fields are accepted case-insensitively and normalized to their documented canonical values. A non-empty body must use the `application/json` media type with UTF-8 encoding; unsupported media types or encodings return HTTP 415, and malformed JSON returns HTTP 400. If OpenSearch is unavailable, the endpoint returns HTTP 503 with a `Retry-After` header.
+`POST /search` accepts an optional JSON object request body and validates supported fields, boolean parameters, pagination values, CVSS severity values, required parameter combinations, CPE Name requirements, and date-time formats. Enum-valued fields are accepted case-insensitively and normalized to their documented canonical values. A non-empty body must use the `application/json` media type with UTF-8 encoding; unsupported media types or encodings return HTTP 415, and malformed JSON returns HTTP 400. Errors use a single `{ "error": "CODE", "message": "Description" }` object; if multiple validation checks fail, the first validation error is returned. If OpenSearch is unavailable, the endpoint returns HTTP 503 with a `Retry-After` header.
 
 Successful `/search` responses use an API-owned response structure rather than exposing search-backend response objects. The top-level fields are `apiVersion`, `status`, `warnings`, `pagination`, and `records`, in that order. `records` contains complete CVE Records directly, without search-backend hit wrappers or infrastructure metadata. `pagination` contains `page`, `pageSize`, `resultsReturned`, `totalResults`, `totalPages`, and `totalIsExact`. Non-fatal conditions are reported in `warnings` before the potentially long `records` array.
 
 The API omits the top-level `opensearch_ops` ingestion metadata from each `/search` record, including ingestion timestamps and signatures, while preserving all other record fields and containers. This targeted removal does not validate or repair records against the CVE JSON schema and does not modify the indexed data.
 
+`resultsPerPage` must be an integer from 1 through 500; integer strings are also accepted. Invalid types, fractional values, and nonpositive values return `RESULTS_PER_PAGE_INT`. Positive integers above 500 return `MAX_RESULTS_LIMIT`.
+
 CVE Records do not exist for years before 1999. A syntactically valid pre-1999 value in `cveIds` is therefore a valid search that contributes no matches rather than a request error. The response includes a `CVE_IDS_NOT_RETURNED` warning listing those IDs. When pre-1999 and supported-year IDs are requested together, matching supported-year records are still returned.
 
+Other syntactically valid IDs with no matches contribute zero results and do not produce a missing-ID warning.
+
 Use the optional `scope` field to control which parts of each record participate in matching. Values are case-insensitive: `FULL` (the default) searches base record fields, CNA data, and every ADP provider; `CNA_ONLY` searches base record fields and CNA data; and `ADP_ONLY` searches base record fields and every ADP provider. Scope changes matching only, so successful responses always contain the complete CVE record. CPE applicability filters use CNA data and return HTTP 400 when combined with `ADP_ONLY`.
 
-Date filters use the API's full UTC timestamp format and inclusive boundaries. When both boundaries for publication, modification, SSVC timestamp, or KEV date-added filters are supplied, the start must be before or equal to the end.
+Date filters accept `YYYY-MM-DD` or timestamps with seconds and an explicit timezone (`Z` or `+/-HH:MM`), with optional fractional seconds of 1 to 3 digits. For example, `2024-01-01`, `2024-01-01T00:00:00Z`, and `2024-01-01T01:00:00+01:00` are accepted. Values are normalized to UTC. A date-only start means `00:00:00.000Z`; a date-only end means `23:59:59.999Z` on that day. Explicit timestamps retain their exact instant, including end boundaries. Impossible dates, timestamps without a timezone, leap seconds, and precision finer than milliseconds return HTTP 400. When both boundaries for publication, modification, SSVC timestamp, or KEV date-added filters are supplied, the start must be before or equal to the end after normalization. All boundaries are inclusive.
 
 CVSS v3.0, v3.1, combined v3, and v4 severity filters accept `NONE`, `LOW`, `MEDIUM`, `HIGH`, and `CRITICAL`, case-insensitively. They match the stored `baseSeverity` label in scope-selected CNA/ADP metrics; the API does not recalculate the label from scores. CVSS v2 instead maps `LOW`, `MEDIUM`, and `HIGH` to base-score ranges and does not accept `NONE` or `CRITICAL`.
 
@@ -157,13 +163,15 @@ Missing data in selected containers does not make an otherwise compatible reques
 
 `keywordSearch` searches only description text selected by `scope`. By default, the API escapes caller-supplied query syntax, adds a trailing prefix wildcard to each whitespace-delimited term, and combines terms with `AND` using `simple_query_string` with wildcard analysis enabled. For example, `remote code` searches for `remote* AND code*`. Each prefix term is processed by the selected field's text analyzer. With the standard analyzer, `use-after-free` becomes required `use` and `after` tokens plus a `free*` prefix; those tokens need not be adjacent or in order. This also allows hyphenated CVE and CWE IDs to match when present in description text; use `cveIds` or `cweId` to search their dedicated fields. With `keywordExactMatch: true`, the API instead uses an analyzed phrase query in the selected description fields. Phrase matching follows the index's text analyzer; it is not a case-sensitive, character-for-character substring comparison.
 
-SSVC metrics can be filtered with `ssvcExploitation` (`NONE`, `POC`, or `ACTIVE`), `ssvcAutomatable` (`YES` or `NO`), `ssvcTechnicalImpact` (`PARTIAL` or `TOTAL`), `ssvcRole`, `ssvcTimestampStartDate`, and `ssvcTimestampEndDate`. Enum and role comparisons are case-insensitive, timestamps use the API's full UTC timestamp format with inclusive boundaries, and all supplied SSVC conditions must match the same structured SSVC metric entry in a container selected by `scope`.
+SSVC metrics can be filtered with `ssvcExploitation` (`NONE`, `POC`, or `ACTIVE`), `ssvcAutomatable` (`YES` or `NO`), `ssvcTechnicalImpact` (`PARTIAL` or `TOTAL`), `ssvcRole`, `ssvcTimestampStartDate`, and `ssvcTimestampEndDate`. Enum and role comparisons are case-insensitive, date boundaries follow the date-input rules above, and all supplied SSVC conditions must match the same structured SSVC metric entry in a container selected by `scope`.
 
-Use `knownExploited` to include or exclude records with a structured KEV metric in containers selected by `scope`. The optional `kevDateAddedStartDate` and `kevDateAddedEndDate` boundaries are inclusive, require `knownExploited: true`, and use the API's full UTC timestamp format. Stored KEV `dateAdded` values are interpreted as midnight UTC.
+Use `knownExploited` to include or exclude records with a structured KEV metric in containers selected by `scope`. The optional `kevDateAddedStartDate` and `kevDateAddedEndDate` boundaries follow the date-input rules above and require `knownExploited: true`. Stored KEV `dateAdded` values are interpreted as midnight UTC; a date-only end boundary includes that entire day.
 
 Use `referenceURL` for a case-insensitive literal substring search over reference URLs in containers selected by `scope`. The filter accepts one string of at least three characters; user-supplied wildcard syntax is treated literally.
 
-Use the strict boolean `disputed` filter to require (`true`) or exclude (`false`) the exact `disputed` tag in containers selected by `scope`. This does not provide a general-purpose record tag filter.
+Use the boolean `disputed` filter to require (`true`) or exclude (`false`) the exact `disputed` tag in containers selected by `scope`. This does not provide a general-purpose record tag filter.
+
+All boolean filters (`isVulnerable`, `keywordExactMatch`, `noRejected`, `knownExploited`, and `disputed`) accept JSON booleans or the lowercase strings `"true"` and `"false"`. Strings are normalized before filter evaluation; numeric values, null, arrays, and other strings are rejected.
 
 String inputs are bounded before query construction:
 
@@ -310,6 +318,14 @@ Both endpoints use the shared OpenSearch client and reject incomplete upstream r
 
 The `/webSearch` response retains `searchMetadata`, `resultsTotal`, and `data`. Metadata includes `searchStatus`, the response-format `version`, `notes`, `errors`, and `queryTime_ms` when available. The obsolete `cveCore` and `cveMitreCore` package-version fields have been removed. `/search` keeps its existing API-owned response format, warnings, and filter behavior.
 
+## Client Result Caching
+
+Successful HTTP 200 responses from both `/search` and `/webSearch`, including zero-result searches, include `Cache-Control: private, max-age=60` by default. `SearchClientCacheMaxAgeSeconds` (or `appConfig.clientCache.maxAgeSeconds`) accepts a nonnegative safe integer number of seconds. Setting `0`, or an invalid value, produces `Cache-Control: no-store`. Search errors, disabled-endpoint responses, parser errors, and liveness/readiness responses use `no-store`. These headers do not create a cache inside the API, change rate enforcement, or force callers to cache.
+
+Both search endpoints use POST. Ordinary browser and proxy HTTP caches do not automatically satisfy repeated POST requests from stored POST responses. Integrators must implement application-level result caching, using the freshness header as a hint. Key cached results by API environment, endpoint, and the complete request body, including filters, scope, sorting, and pagination. Never share results merely because requests use the same `/search` URL. `private` does not authorize shared HTTP proxy caching, and the API does not provide conditional POST/304 revalidation.
+
+Retain a successful result for no longer than the advertised freshness duration, then issue a fresh request. This permits up to that duration of additional staleness after the result was generated; it does not guarantee indexing freshness or consistency across separately cached pages. A cached `/webSearch` response's `queryTime_ms` describes its original execution, not the current lookup. Clients must not cache errors or treat old rate-limit information as a current quota. Only avoiding an HTTP request reduces traffic counted by the deployment's rate limiter; sending a request still counts even if a future server-side cache answers it.
+
 ## Swagger API docs
 
 Swagger API docs are regenerated by `npm run dev` and can be regenerated manually using: