Skip to content

fix(gateway): log the capture status a request ends with - #5981

Merged
ZaynJarvis merged 2 commits into
volcengine:mainfrom
t0saki:fix/gateway-capture-status-refresh
Oct 15, 2026
Merged

ZaynJarvis merged 2 commits into
volcengine:mainfrom
t0saki:fix/gateway-capture-status-refresh

Conversation

@t0saki

@t0saki t0saki commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Description

/admin/overview can keep reporting a capture as retrying after its delivery recovered.

Each request log row carries the capture_status read when the request was prepared (kernel.prepare / CapturePipeline.confirm), but the row is written only when the request finishes (ProxyRequest.finish). overview_summary takes the newest row per (session, protocol). Here is how it goes wrong:

  1. The capture document has error set, so a request prepared now records retrying.
  2. While that request is still in flight, the worker's retry succeeds. It clears error and logs the kind=capture row active / delivery_recovered.
  3. The request finishes and writes its row with the stale retrying. That row is newer than the recovery row.
  4. The overview reports capture_issues.retrying = 1 until another request on that session logs active. If no request comes, it stays that way.

This PR re-reads the capture document just before the request row is written and recomputes the status with the existing CapturePipeline.metrics. The row now records the status at the time it is logged. A recovery during the request no longer leaves a stale retrying, and a failure during the request no longer leaves a stale active. One small window remains: if the worker inserts its capture row between the re-read and the request row's INSERT, the request row is still the newest. That window is a few milliseconds instead of the whole request duration (25s in the ovtest case). The cost is one capture read per enhanced request.

The re-read has its own try. If it fails, the row keeps the capture fields computed at prepare time, and the request row (token usage, duration, recall metrics) is still logged.

I rejected counting only kind=capture rows in the overview. reset_capture, the deleted-key path and a history_changed restart all clear error without writing a capture row, so the overview would keep showing retrying after any of them.

Found by the ovtest integration run on OpenViking main 92285e5 (2026-10-13), case journey-g13-d-sdk-chat-to-sdk-chat-request-not-reached. After an injected 503, the retried messages/batch returned 200 and the capture log wrote delivery_recovered. A turn slowed by a 25s search delay finished after that, and the overview three seconds later still showed capture_issues{paused:0,retrying:1}.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Related: #5960 also touches proxy.py and moves overview_summary / renames management.log. It does not fix this issue. Whichever PR lands second needs a small rebase.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • openviking_gateway/proxy.py: in finish(), re-read the capture document and recompute the capture fields with CapturePipeline.metrics before management.log. A failed re-read is logged and falls back to the prepare-time fields.
  • tests/gateway/test_app.py: add test_overview_counts_capture_recovered_during_a_request. It seeds a retrying capture, clears it and logs delivery_recovered while the next request is upstream, then asserts that the overview reports no capture issues.
  • tests/gateway/test_app.py: add test_capture_status_refresh_failure_still_logs_request. It makes the re-read fail and asserts that the request row is still written with the prepare-time capture_status.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows
# without the proxy.py change
PYTHONPATH=. pytest tests/gateway/test_app.py::test_overview_counts_capture_recovered_during_a_request --confcutdir=tests/gateway -o addopts=''
E  AssertionError: assert {'paused': 0, 'retrying': 1} == {'paused': 0, 'retrying': 0}
1 failed

# with the fix
1 passed
# test_capture_status_refresh_failure_still_logs_request, with the re-read in the same try as management.log
E  AssertionError: assert [] == ['active']

PYTHONPATH=. pytest tests/gateway --confcutdir=tests/gateway -o addopts=''
596 passed

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

@ZaynJarvis ZaynJarvis added the kernel Core server, runtime, storage, retrieval, SDK, and CLI. label Oct 14, 2026
@t0saki
t0saki marked this pull request as ready for review October 15, 2026 04:26
Copilot AI balanced review requested due to automatic review settings October 15, 2026 04:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

The request log row carried the capture status read when the request was
prepared. A long request that started while delivery was retrying finished
after the worker's delivery_recovered row, so its stale "retrying" became the
newest status for the session and /admin/overview kept counting the capture
in capture_issues until another request arrived.

Re-read the capture document just before the row is written.
@t0saki
t0saki force-pushed the fix/gateway-capture-status-refresh branch from a6343cc to 469fcca Compare October 15, 2026 10:05

@ZaynJarvis ZaynJarvis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved per maintainer request. Rebased on a0c3574; proxy.py conflict resolution keeps the capture status refresh and adopts #5960 user_id= and #6100 import cleanup. Gateway + plugin checks green; API & CLI Integration failure is the CI embedding key (401).

@ZaynJarvis
ZaynJarvis merged commit e05452e into volcengine:main Oct 15, 2026
9 of 10 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Core server, runtime, storage, retrieval, SDK, and CLI.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants