Skip to content

fix(server): port eval soft/hard think-close reply budgets - #958

Open
paride-cmd wants to merge 1 commit into
antirez:mainfrom
paride-cmd:fix/895-server-think-close
Open

fix(server): port eval soft/hard think-close reply budgets#958
paride-cmd wants to merge 1 commit into
antirez:mainfrom
paride-cmd:fix/895-server-think-close

Conversation

@paride-cmd

Copy link
Copy Markdown

Summary

  • Ports `ds4-eval`'s soft/hard reply-budget + forced `` controller into `ds4-server` decode.
  • Adds `--soft-limit-reply-budget` (default 1024), `--hard-limit-reply-budget` (default 512), and `--soft-limit-think-close-rank` (default 3).
  • Soft-close accepts the model's own `` when it is already in the top-N logits; hard-close forces the close marker to leave answer room.
  • Enabled only when request `max_tokens > hard_limit` (same shape as eval's hard < tokens guard). Forced close skips MTP speculation for that step.

Fixes #895

Test plan

  • `make ds4_test && ./ds4_test --server`
  • `./ds4-server --help thinking` shows the new flags
  • Live HTTP: saturating thinking prompt with small `max_tokens` should emit `` and an answer instead of hanging until client timeout

AI usage disclosure: YES — assisted by Cursor agent; logic mirrored from `ds4_eval.c` and covered by new parse/forced-token unit tests.

Made with Cursor

HTTP callers could spend the entire max_tokens budget inside <think>
and never terminate with an answer. Port ds4-eval's soft/hard reply
budget controller to ds4-server with the same defaults and CLI flags so
generation forces </think> while tokens remain for a reply.

Fixes antirez#895
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ds4-server: no reply-budget or forced think-close, so HTTP callers can hang where ds4-eval terminates

1 participant