feat(studio): add preference-driven Agent quality management

frontend/README.md

@@ -519,6 +519,57 @@ See [deployment and operation](service/studio_release_notifier/README.md).
   always use placeholders
   instead of credentials. Long descriptions, names, component summaries, IDs,
   and environment values stay inside the scrollable panel.
+- **Agent quality management**: Agent details include a separate Quality
+  management entry with Evaluation preferences, Evaluation sets, Evaluators, and
+  Evaluation results tabs. Existing Evaluations and feedback workflows are unchanged.
+  - Start with a named evaluation preference: business goals, scenarios, acceptance
+    criteria, unacceptable mistakes, and an emphasis on results, process, or both.
+    Four text areas offer independently generated, selectable suggestions from the
+    Agent context. The model then proposes editable dataset and evaluator preferences;
+    their six text areas also offer independent suggestions aligned with the overall
+    preferences. Selecting an option appends it, and deselecting removes its matching
+    line without replacing other text. Typing does not restart requests. Completed
+    suggestions are cached for the same context and language; retry only requests
+    unfinished fields. No dataset or evaluator is generated before user confirmation.
+  - Confirming the component preferences generates the dataset and evaluator bundle
+    concurrently. Each preference links to its generated components, which can be
+    filtered in their respective tabs. Changing preferences marks previous outputs
+    as needing an update. Partial component successes are retained, and retries skip
+    components already generated with the same preferences. Closing the dialog
+    preserves completed work and the preference draft in page state.
+  - Dataset generation accepts an integer count from 50 to 300, defaulting to 100.
+    Empty, fractional and out-of-range input shows inline errors and blocks saving
+    or generation. Batches contain at most 20 cases, with at most four batches in
+    parallel, a scaled timeout, exact-count checks and normalized duplicate-input
+    detection. Disconnecting cancels dataset generation. Each case includes a concrete
+    input, expected outcome, optional expected actions and verifiable checks, viewable
+    in a drawer. Direct generation is also available with Agent-based brief autofill.
+  - Three evaluator categories—overall, tool capability, and Skill—use concurrent
+    model calls and appear as vertically stacked cards with skeleton loading. The
+    model derives 1–5 dimensions per evaluator from the Agent's documented requirements.
+    Dimension accordions show a localized name, description, scoring basis, minimum
+    and maximum score, and a rationale identifying the relevant Agent instructions or
+    capabilities. Standalone prompts repeat these requirements and specify how to
+    handle unavailable capabilities or missing execution evidence. Regeneration
+    replaces existing definitions only after success.
+  - Generation uses the full available Agent topology, root and child system prompts,
+    tool descriptions, Skill instructions, mounted components including knowledge
+    bases and memory, safe configuration fields, and versioned environment manifests.
+    Resource ownership, configuration versus runtime metadata, and unavailable
+    information remain explicit. Prepared context reuses loaded Agent metadata and
+    shares mounted-resource reads across generation steps; changed metadata invalidates
+    it. Runtime requests use verified app identifiers, and unrelated editor recovery
+    does not restart suggestions. Deployment credentials and raw configuration are
+    excluded from the prepared context.
+  - All generation uses the same provider-specific model as Agent form generation,
+    Pydantic output schemas through Agent/Runner and Ark Responses, with thinking
+    disabled. Volcengine and BytePlus use their respective endpoints and credentials.
+    Failures expose complete captured provider responses, error codes, request IDs and
+    exception traces in expandable details, with credential values redacted. Parallel
+    calls retain diagnostics from failed siblings even if another request times out.
+  - Preferences, datasets and evaluators currently remain in page memory and are lost
+    when leaving Quality management or reloading. Evaluation execution and result
+    analysis are not available yet
 - **Custom-agent workbench**: configure an agent with a rich Markdown
   system-prompt editor (including heading and list shortcuts), choose Harness
   Sidecar optimizations, then debug with expandable, copyable runner error

frontend/server/quality/__init__.py

@@ -0,0 +1,19 @@
+# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Independent, transient evaluation-data generation for Studio quality management."""
+
+from .routes import mount_quality_routes
+
+__all__ = ["mount_quality_routes"]

frontend/server/quality/errors.py

@@ -0,0 +1,91 @@
+# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Preserve generation diagnostics while removing credential values."""
+
+import json
+import os
+import re
+import traceback
+
+
+def generation_diagnostics(error: BaseException, *secrets: str) -> str:
+    sections: list[str] = []
+    seen: set[int] = set()
+    current: BaseException | None = error
+    while current is not None and id(current) not in seen:
+        seen.add(id(current))
+        sections.append(f"{type(current).__name__}: {current}")
+        response = getattr(current, "response", None)
+        status = getattr(current, "status_code", None) or getattr(
+            response, "status_code", None
+        )
+        if status is not None:
+            sections.append(f"HTTP status: {status}")
+        request_id = getattr(current, "request_id", None)
+        if request_id:
+            sections.append(f"Request ID: {request_id}")
+        headers = getattr(response, "headers", {})
+        for name in ("x-request-id", "x-tt-logid", "x-trace-id", "x-amzn-requestid"):
+            if headers.get(name):
+                sections.append(f"{name}: {headers[name]}")
+        body = getattr(current, "body", None)
+        if body is not None:
+            sections.append(
+                "Provider error body:\n"
+                + json.dumps(body, ensure_ascii=False, default=str)
+            )
+        if response is not None:
+            try:
+                response_text = response.text
+            except (AttributeError, RuntimeError, UnicodeError):
+                response_text = ""
+            if response_text:
+                sections.append("Provider response:\n" + str(response_text))
+        current = current.__cause__ or (
+            None if current.__suppress_context__ else current.__context__
+        )
+    sections.append(
+        "Exception trace:\n"
+        + "".join(traceback.format_exception(type(error), error, error.__traceback__))
+    )
+    text = "\n\n".join(sections)
+    environment_secrets = [
+        value
+        for key, value in os.environ.items()
+        if len(value) >= 8
+        and any(
+            marker in key.upper()
+            for marker in ("KEY", "SECRET", "TOKEN", "PASSWORD", "CREDENTIAL")
+        )
+    ]
+    for secret in sorted(set([*secrets, *environment_secrets]), key=len, reverse=True):
+        if secret:
+            text = text.replace(secret, "[REDACTED]")
+    text = re.sub(r"(?i)(\bbearer\s+)[^\s,;\"']+", r"\1[REDACTED]", text)
+    text = re.sub(
+        r"(?i)((?:[\"']?)(?:api[_-]?key|access[_-]?key(?:[_-]?id)?|secret[_-]?(?:access[_-]?)?key|client[_-]?secret|authorization|password|session[_-]?token|security[_-]?token|token|secret|credential|ak|sk|cookie|set-cookie)(?:[\"']?)\s*[:=]\s*)(?:\"[^\"]*\"|'[^']*'|[^\s,;&]+)",
+        r"\1[REDACTED]",
+        text,
+    )
+    return text
+
+
+class QualityGenerationError(Exception):
+    def __init__(self, diagnostics: str, *, timeout: bool = False) -> None:
+        super().__init__(
+            "quality_generation_timeout" if timeout else "quality_generation_failed"
+        )
+        self.diagnostics = diagnostics
+        self.timeout = timeout