I've been reading runtime/sdk.py — run_workflow() returning a WorkflowRunResult(final_message: Message, meta_info: WorkflowMetaInfo), where WorkflowMetaInfo carries session_name, log_id, outputs, token_usage, and output_dir. That's a clean, structured shape for "what happened when this workflow graph ran" — session identity, token accounting, and the final output message all in one place.
I maintain EvalPort, a JSON-Schema-based interchange spec (TestCase/Suite/Grader/Result/ResultSet) so eval data — a run's outcome plus whatever graded it — doesn't stay locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).
WorkflowRunResult maps fairly directly onto an EvalPort Result: final_message as the graded output, token_usage as EvalPort's usage metadata, session_name/log_id as run identifiers. Running the same YAML workflow over a batch of task prompts and collecting the WorkflowRunResults would give a natural ResultSet. What's missing is a way to get that out in a form another framework's grading tooling could read, or to drive run_workflow() from a portable task suite instead of one-off task_prompt strings.
I'd propose a standalone, optional adapter: WorkflowRunResult → EvalPort Result/ResultSet, plus a helper that maps an EvalPort Suite's TestCase.inputs onto run_workflow(..., task_prompt=...) calls. No required dependency, no change to runtime/sdk.py or the graph executor.
Happy to build this as a PR into ChatDev (e.g. a small module beside runtime/sdk.py), or as a standalone package in EvalPort's own adapters/ directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
— Sahi, independent contributor (not affiliated with ChatDev/OpenBMB)
I've been reading
runtime/sdk.py—run_workflow()returning aWorkflowRunResult(final_message: Message, meta_info: WorkflowMetaInfo), whereWorkflowMetaInfocarriessession_name,log_id,outputs,token_usage, andoutput_dir. That's a clean, structured shape for "what happened when this workflow graph ran" — session identity, token accounting, and the final output message all in one place.I maintain EvalPort, a JSON-Schema-based interchange spec (
TestCase/Suite/Grader/Result/ResultSet) so eval data — a run's outcome plus whatever graded it — doesn't stay locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).WorkflowRunResultmaps fairly directly onto an EvalPortResult:final_messageas the graded output,token_usageas EvalPort's usage metadata,session_name/log_idas run identifiers. Running the same YAML workflow over a batch of task prompts and collecting theWorkflowRunResults would give a naturalResultSet. What's missing is a way to get that out in a form another framework's grading tooling could read, or to driverun_workflow()from a portable task suite instead of one-offtask_promptstrings.I'd propose a standalone, optional adapter:
WorkflowRunResult→ EvalPortResult/ResultSet, plus a helper that maps an EvalPortSuite'sTestCase.inputsontorun_workflow(..., task_prompt=...)calls. No required dependency, no change toruntime/sdk.pyor the graph executor.Happy to build this as a PR into ChatDev (e.g. a small module beside
runtime/sdk.py), or as a standalone package in EvalPort's ownadapters/directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md— Sahi, independent contributor (not affiliated with ChatDev/OpenBMB)