Skip to content

feat(studio): persist evaluation datasets and feedback in TOS - #1141

Merged
yaozheng-fang merged 3 commits into
volcengine:mainfrom
evanlowe:feat/studio-evaluation-tos
Sep 21, 2026
Merged

yaozheng-fang merged 3 commits into
volcengine:mainfrom
evanlowe:feat/studio-evaluation-tos

Conversation

@evanlowe

Copy link
Copy Markdown
Contributor

Studio evaluation sets and samples now use private TOS storage, scoped only by Runtime ID. The existing evaluation page and feedback controls keep their current appearance and API contract; no frontend source or bundled assets change.

  • Store default Good/Bad Case sets and user-created sets in the same repository, with manual/automatic origin recorded per sample
  • Support set/sample CRUD, moves, filtering, search and pagination; protect updates with ETags and retain deletion markers to prevent automatic retries from restoring deleted data
  • Apply instance limits during Runtime creation, wait for reserved instances before reporting success, and fix structured evaluator initialization under the production lazy-import startup path
  • Select the appropriate default evaluation model for Volcengine and BytePlus

Validation:

  • Deployment/RBAC and instance-readiness regression: 161 passed
  • Evaluation, production startup and feedback regression: 81 passed; one existing startup import test also fails on the unchanged base because requests is already loaded
  • Existing frontend suite: 1,213 passed; production build passed
  • Ruff, formatting and secret scanning passed; changed modules pass Pyright, with no new CLI diagnostics compared with the base
  • Real TOS CRUD and conditional-write conflicts verified on Volcengine and BytePlus
  • Local and deployed Studio end-to-end: Agent creation, conversation, feedback transitions, withdrawal, refresh recovery and automatic evaluation persisted to TOS; automatic samples scored 1.0 in both environments

Existing cloud evaluation data is not migrated. List queries scan the selected Runtime's objects with bounded concurrency. Runtime conversation persistence still follows the Agent's configured memory backend; use database-backed memory to survive instance replacement.

@yaozheng-fang
yaozheng-fang merged commit 5843873 into volcengine:main Sep 21, 2026
16 checks passed
PolarisCzh pushed a commit to PolarisCzh/veadk-python that referenced this pull request Sep 29, 2026
…gine#1141)

* feat(studio): store evaluation datasets and feedback in TOS

* fix(ci): update runtime instance regression selector

* test(studio): allow background recovery in credential reuse checks

Co-authored-by: TRAE CLI <traecli@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants