Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 38 additions & 4 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import UnoCSS from 'unocss/vite'

import { transformerNotationWordHighlight } from '@shikijs/transformers'
import { extendConfig } from '@voidzero-dev/vitepress-theme/config'
import { defineConfig } from 'vitepress'

Expand Down Expand Up @@ -87,6 +88,9 @@ export default extendConfig(defineConfig({
},
},
markdown: {
codeTransformers: [
transformerNotationWordHighlight(),
],
theme: {
dark: 'github-dark',
light: 'github-light',
Expand Down Expand Up @@ -156,11 +160,26 @@ export default extendConfig(defineConfig({
],
'/en/guide/': [
{
collapsed: false,
items: [
{ link: '/en/guide/', text: 'Introduction' },
{ link: '/en/guide/', text: 'Guide Overview' },
{ link: '/en/guide/why', text: 'Why Vieval' },
{ link: '/en/guide/getting-started', text: 'Getting Started' },
{ link: '/en/guide/core-concepts', text: 'Core Concepts' },
],
text: 'Introduction',
},
{
collapsed: false,
items: [
{ link: '/en/guide/learn/tasks-cases-and-inputs', text: 'Tasks, Cases, and Inputs' },
{ link: '/en/guide/learn/assertions-scores-and-metrics', text: 'Assertions, Scores, and Metrics' },
{ link: '/en/guide/learn/models-and-inference-executors', text: 'Models and Inference Executors' },
{ link: '/en/guide/learn/matrices-and-datasets', text: 'Matrices and Datasets' },
{ link: '/en/guide/learn/reliable-execution', text: 'Reliable Execution' },
{ link: '/en/guide/learn/reports-and-comparisons', text: 'Reports and Comparisons' },
],
text: 'Guide',
text: 'Learn',
},
],
'/zh-hans/api/': [
Expand All @@ -181,11 +200,26 @@ export default extendConfig(defineConfig({
],
'/zh-hans/guide/': [
{
collapsed: false,
items: [
{ link: '/zh-hans/guide/', text: '介绍' },
{ link: '/zh-hans/guide/', text: '指南概览' },
{ link: '/zh-hans/guide/why', text: '为什么选择 Vieval' },
{ link: '/zh-hans/guide/getting-started', text: '快速开始' },
{ link: '/zh-hans/guide/core-concepts', text: '核心概念' },
],
text: '介绍',
},
{
collapsed: false,
items: [
{ link: '/zh-hans/guide/learn/tasks-cases-and-inputs', text: '任务、用例与输入' },
{ link: '/zh-hans/guide/learn/assertions-scores-and-metrics', text: '断言、分数与指标' },
{ link: '/zh-hans/guide/learn/models-and-inference-executors', text: '模型与推理执行器' },
{ link: '/zh-hans/guide/learn/matrices-and-datasets', text: '矩阵与数据集' },
{ link: '/zh-hans/guide/learn/reliable-execution', text: '可靠执行' },
{ link: '/zh-hans/guide/learn/reports-and-comparisons', text: '报告与比较' },
],
text: '指南',
text: '学习',
},
],
},
Expand Down
7 changes: 7 additions & 0 deletions docs/.vitepress/theme/styles.css
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

@source "./**/*.vue";

.vp-doc [class*='language-'] code .highlighted-word {
padding: 1px 2px;
margin: -1px -2px;
background-color: var(--vp-code-line-highlight-color);
border-radius: 4px;
}

/* Brand color: black in light mode, white in dark mode */
/* :root,
:root[data-variant="voidzero"] {
Expand Down
54 changes: 42 additions & 12 deletions docs/content/en/api/index.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,44 @@
# API

This section will document Vieval package exports and runtime entrypoints.

Start with the public package entrypoints:

- `vieval`
- `vieval/config`
- `vieval/core/runner`
- `vieval/core/assertions`
- `vieval/core/inference-executors`
- `vieval/plugins/chat-models`
- `vieval/expect`
- `vieval/testing/expect-extensions`
Vieval publishes focused entrypoints so application code can import the narrowest supported surface for its job. The package's `.` export is imported as `vieval`; every other current export is listed below by its published import path.

## Common authoring

| Import | Import when |
| --- | --- |
| `vieval` | Author eval files or config with the task DSL, config helpers, and the preconfigured assertion API; representative imports are `describeTask`, `defineConfig`, and `expect`. |
| `vieval/core/assertions` | Build and evaluate structured assertion pipelines and convert their outcomes into runner scores with exports such as `evaluateAssertions`, `expectRubric`, and `toRunScores`. |
| `vieval/expect` | Import the standalone `expect` with Vieval matchers pre-installed without importing the rest of the main authoring surface. |

Start with [Tasks, Cases, and Inputs](/en/guide/learn/tasks-cases-and-inputs) and [Assertions, Scores, and Metrics](/en/guide/learn/assertions-scores-and-metrics). The task DSL and the structured core assertion pipeline are both public, but outcomes from `vieval/core/assertions` are not connected to task cases automatically.

## Config and integration

| Import | Import when |
| --- | --- |
| `vieval/config` | Define typed eval or task objects and work with config-owned contracts through exports such as `defineEval`, `defineTask`, and `resolveModelByName`; the CLI-facing `defineConfig` and `loadEnv` helpers remain on `vieval`. |
| `vieval/plugins/chat-models` | Register chat models and resolve matrix selections with exports such as `chatModelFrom`, `ChatModels`, and `modelFromRun`. |

Use the [Config map](/en/config/) for file ownership and scope, and [Models and Inference Executors](/en/guide/learn/models-and-inference-executors) for the boundary between registration, selection, and an actual model call.

## Advanced execution and processing

These are public integration surfaces for hosts that need to assemble or drive execution directly. Most eval authors do not need them.

| Import | Import when |
| --- | --- |
| `vieval/cli` | Embed the import-safe top-level CLI parser and command dispatcher through `parseTopLevelCliArguments` and `runTopLevelCli`. |
| `vieval/core/runner` | Assemble collection, scheduling, and execution directly with exports such as `collectEvalEntries`, `createRunnerSchedule`, and `runScheduledTasks`. |
| `vieval/core/scheduler` | Construct scoped concurrency coordination without the runner's collection APIs by using `createSchedulerRuntime` and `getActiveScopes`. |
| `vieval/core/processors/results` | Apply built-in result policies and inspect the gate through `processRunResults` and the `ResultGateDecision` type. |
| `vieval/core/inference-executors` | Build custom inference integrations with exports such as `createProviderAdapter`, `createRetryPolicy`, and `createOpenAIFromEnv`. |

Read [Reliable Execution](/en/guide/learn/reliable-execution) before composing runner, scheduler, retry, or cache behavior. For aggregation, gating, and persisted evidence, continue with [Reports and Comparisons](/en/guide/learn/reports-and-comparisons).

## Testing support

| Import | Import when |
| --- | --- |
| `vieval/testing/expect-extensions` | Explicitly install Vieval's custom matchers or consume their supporting types through `installVievalExpectMatchers`, `KeywordMatcherOptions`, and `ToolCallContainer`. |

`expect` from the main `vieval` entrypoint and `expect` from `vieval/expect` both arrive with Vieval's matchers installed; the main entrypoint is convenient beside the task DSL, while the subpath is the focused assertion import. `vieval/testing/expect-extensions` exports the installer rather than an `expect` value, so call `installVievalExpectMatchers()` only when explicit installation is the integration you need.
27 changes: 25 additions & 2 deletions docs/content/en/config/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# Config

Vieval configuration lives close to the project that owns the evaluations.
Vieval configuration belongs to the repository or package that owns the evaluations. `vieval run` starts at the command's working directory and uses the nearest readable `vieval.config.ts`, `.mts`, `.cts`, `.js`, `.mjs`, `.cjs`, or `.json`, searching upward through parent directories. Pass `--config <path>` to select a file explicitly; a relative path is resolved from the command's working directory.

This page will document config loading, model provider settings, runner behavior, reporters, and output options.
The config file's directory becomes the base for relative project roots. Each project's `include` and `exclude` patterns are then evaluated within that project root. This keeps discovery ownership explicit when one config describes more than one project.

## Configuration domains

| Domain | What belongs here | Learn more |
| --- | --- | --- |
| Loading and environment | Config discovery or `--config` selection, `defineConfig`, and top-level `env`. Vieval does not load `.env*` files automatically; when those values are needed, call `loadEnv` explicitly from the config file. | [Models and Inference Executors](/en/guide/learn/models-and-inference-executors) |
| Projects and modes | Project names, roots, discovery patterns, executors, and project-scoped settings. Top-level `projects`, `workspaces`, and `comparisons` are mutually exclusive: `run` accepts project mode and currently maps each workspace `{ id, root }` to a project boundary, while `compare` owns comparison mode. Workspace mode does not add another nested orchestration layer. | [Core Concepts](/en/guide/core-concepts) |
| Plugins and models | Top-level or project-local config plugins, model registrations, provider metadata, aliases, and project model overrides. A registration makes a model available to task code; it does not make an inference request by itself. | [Models and Inference Executors](/en/guide/learn/models-and-inference-executors) |
| Run and eval matrices | Project `runMatrix` and `evalMatrix` definitions, plus how eval- and task-level layers extend, override, or disable axes. Matrix expansion schedules variants; task code decides what each selected value does. | [Matrices and Datasets](/en/guide/learn/matrices-and-datasets) |
| Execution policy and concurrency | Project scheduling caps and the task/case attempt, retry, timeout, and concurrency controls that bound execution. The `vieval run` concurrency flags are runtime overrides or caps at their documented scopes; `--attempt` labels report artifacts and does not configure automatic attempts. | [Reliable Execution](/en/guide/learn/reliable-execution) |
| Cache | The task cache is injected through `TaskRunContext`, not declared as a general top-level cache block. CLI runs place project cache data under `.vieval/cache` inside the project root; comparison mode can select a shared benchmark cache namespace. | [Reliable Execution](/en/guide/learn/reliable-execution) |
| Reporters and artifacts | Top-level reporters are inherited by projects unless a project supplies its own list. Local report artifacts are enabled at run time with `--report-out`; `--json` changes standard output and does not create artifacts. | [Reports and Comparisons](/en/guide/learn/reports-and-comparisons) |
| Telemetry | `reporting.openTelemetry` controls runtime OpenTelemetry spans and an end-of-run lifecycle callback. Local reporter events and the OTLP-shaped files produced by `--report-out` do not depend on that switch. | [Reports and Comparisons](/en/guide/learn/reports-and-comparisons) |

## Scope and precedence

Configuration only inherits where the owning types and runtime define inheritance. Top-level `concurrency.workspace` controls workspace scheduling only. For `project`, `task`, `attempt`, and `case`, each top-level concurrency value becomes the project default; a project can override each field independently, and any field it omits continues to inherit the top-level value. Models and reporters behave differently: their top-level arrays feed project defaults, but a project-provided array replaces the complete inherited list. Matrix layers have their own project → eval → task resolution rules. Plugins may transform top-level config, while project-local plugins run against a project-scoped config view. These rules do not imply that every field is available at every scope.

The run command can select projects with `--project`, select a config with `--config`, cap or override the documented concurrency scopes, label report organization with `--workspace`, `--experiment`, and `--attempt`, and choose output behavior with `--json` and `--report-out`. Treat these as command-specific runtime inputs rather than a generic override mechanism for arbitrary config fields.

Each documented field records **Type → Default → Scope → CLI override → Behavior → Example → Interactions**. When a step does not apply to that field, the reference states that explicitly instead of inferring a value or an override.

For the exported config types and helpers, see the [API map](/en/api/).
42 changes: 42 additions & 0 deletions docs/content/en/guide/core-concepts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Core Concepts
prev:
text: Getting Started
link: /en/guide/getting-started
next:
text: Tasks, Cases, and Inputs
link: /en/guide/learn/tasks-cases-and-inputs
---

# Core Concepts

Vieval organizes evaluation work into a small set of nested concepts. The following lifecycle is the map for the rest of the guide:

```text
project
-> task <-> run matrix + eval matrix
-> case -> assertion / score / metric evidence

task -> attempt -> retry

run --json -> machine-readable stdout
run --report-out -> report artifacts -> analyze / compare
```

A **project** defines a discovery boundary and project-level configuration. A discovered eval definition may carry a **task**, while DSL definitions such as `describeTask` register tasks directly. Matrix rows expand the registered tasks into concrete scheduled work. A task contains one or more **cases**, where assertions, normalized scores, and custom metrics provide execution evidence.

Execution reliability has two scopes. An **attempt** reruns a full task and retains the evidence from every attempt. A **retry** lets a failed case run again within an attempt. Output is explicit: `--json` prints a machine-readable result to standard output, while `--report-out <directory>` writes the report artifacts consumed by analysis and comparison commands.

| Concept | Responsibility | Typical scope |
| --- | --- | --- |
| Project | Defines discovery, configuration, executors, and matrix defaults | One product area or evaluation suite |
| Task | Describes evaluation behavior and connects run/eval matrix selections | One behavior evaluated across variants |
| Case | Executes one input or scenario and emits assertion, score, and metric evidence | One check within a task |
| Run matrix | Expands execution-side choices such as models or scenarios | Project, eval, or task definition |
| Eval matrix | Expands evaluation-side choices such as rubrics | Project, eval, or task definition |
| Attempt | Repeats the complete task to measure reliability | One scheduled task |
| Retry | Re-executes a failed case inside an attempt | One case execution |
| Report artifacts | Preserve run summaries, events, and case evidence when `--report-out` is used | One run and its configured report directory |
| Analysis and comparison | Read artifacts to inspect results or compare runs and methods | One or more report artifact sets |

Next, learn how to author [Tasks, Cases, and Inputs](/en/guide/learn/tasks-cases-and-inputs).
118 changes: 112 additions & 6 deletions docs/content/en/guide/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,121 @@
---
title: Getting Started
prev:
text: Why Vieval
link: /en/guide/why
next:
text: Core Concepts
link: /en/guide/core-concepts
---

# Getting Started

Install the package:
This first evaluation is deterministic and runs entirely in your local Node.js process.

## Install Vieval

::: code-group

```sh
```sh [pnpm]
pnpm add -D vieval
```

Create evaluation tasks with the Vieval DSL, configure providers, then run the CLI from your project.
```sh [npm]
npm install --save-dev vieval
```

:::

## Configure a project

Create a config at the root of your project:

```ts [vieval.config.ts]
import { defineConfig } from 'vieval'

export default defineConfig({
projects: [
{
include: ['evals/*.eval.ts'],
inferenceExecutors: [
{
id: 'local',
},
],
models: [
{
aliases: [],
id: 'local:deterministic',
inferenceExecutor: 'local',
inferenceExecutorId: 'local',
model: 'deterministic',
},
],
name: 'getting-started',
root: '.',
},
],
})
```

The project tells Vieval where to discover evaluation files. The `root` is resolved as the project boundary, and `include` selects matching files beneath it. The model entry is local registry metadata used to enable automatic execution of discovered DSL tasks; this evaluation never calls it as a provider.

## Write an evaluation

Create the matching evaluation file:

```ts [evals/arithmetic.eval.ts]
import { caseOf, describeTask, expect } from 'vieval'

describeTask('arithmetic', () => {
caseOf('adds two numbers', () => {
expect(20 + 22).toBe(42)
})
})
```

`describeTask` registers the task, `caseOf` registers one case inside it, and `expect` executes an assertion. If its matcher fails, it throws and the case fails.

## Run it

::: code-group

```sh [pnpm]
pnpm vieval run --config ./vieval.config.ts
```

```sh [npm]
npx vieval run --config ./vieval.config.ts
```

:::

The stable result is one discovered project, one scheduled task, and one passed case. The terminal presentation may change, so this guide does not depend on exact output text or timing.

::: info No credentials required
Automatic DSL execution currently requires a registered model target. The `local:deterministic` entry only satisfies that execution gate: this task makes no provider call, uses no credentials, and incurs no model cost.
:::

## Optional: How to read later examples

Guide examples use an error highlight to identify the line that failed:

```ts [evals/arithmetic.eval.ts]
expect(20 + 22).toBe(41) // [!code error]
```

Removed and added line highlights then show the correction without hiding the original mistake:

```ts [evals/arithmetic.eval.ts]
expect(20 + 22).toBe(41) // [!code --]
expect(20 + 22).toBe(42) // [!code ++]
```

When the edit itself is the subject, the same change may use a diff fence:

```sh
pnpm vieval run
```diff [evals/arithmetic.eval.ts]
- expect(20 + 22).toBe(41)
+ expect(20 + 22).toBe(42)
```

The detailed setup guide will be filled in as the public API and examples settle.
You now have two project files: `vieval.config.ts` defines discovery, while `evals/arithmetic.eval.ts` defines executable evaluation behavior. Continue to [Core Concepts](/en/guide/core-concepts) to see how projects, tasks, cases, attempts, and reports fit together.
Loading
Loading