DatEngine 🚀 Modular AI Agentic Framework written in Nim lang

nimble install datengine

API reference

About

DatEngine is an app-agnostic agentic engine written in Nim. Made to build self-hosted, AI agents via command line and browsers. Designed as a pure library with no HTTP server or UI. A CLI, REST, or WebSocket adapter drives it.

The engine orchestrates an LLM agent loop, manages sessions, executes tools with a strict safety envelope, and automates browsers via CDP (Chrome DevTools). Every wire format (config, sessions, tool schemas) flows through openparser as typed Nim objects.

😍 Key Features

• Any OpenAI-compatible endpoints
• Streaming SSE: token-by-token deltas via ChaChaChat's async HTTP client
• Tool calling: agentic function calling with typed parameter schemas and automatic argument decoding
• Session persistence: Boogie RDBMS with indexed columns, conversation history survives restarts
• Truncation: configurable history window with system-message preservation via replaceMessages
• Cancellation: cooperative cancellation via agent flag (checked between tool iterations)
• Skills: markdown files with YAML frontmatter, fuzzy-matched per turn via floof and injected into the system message (see below)
• Two opt-in sources: global (globalFs disk skills at ~/.<myagent>/skills) and per-session workspace (<skillsDir> relative to Workspace.root); workspace overrides global when names collide. Both empty = skills disabled.
• floof fuzzy matching: SIMD-accelerated subsequence search of user input against keywords and names; skillMinScore threshold (default 0.5), top-N cap via skillMaxPerTurn (default 3).
• flysystem loading: skills are read through flysystem drivers (globalFs.disk("skills") + Workspace.fs.disk("workspace")); newSkillRegistryFromDrivers does driver-level listing, gitignore rules apply only to workspace skills.
• Model-facing tools: skill_list and skill_read let the LLM browse skill content explicitly.

Quick Start

Phase 1 — global init (once at startup): all dirty wiring inside initDatEngine, providers auto-synced:

import datengine

# single call with large param set; everything dirty happens inside:
# newGlobalFs (skills/config/providers at ~/.myagent) + newProviderStore
# + auto syncFromGlobalFs (YAML/JSON at ~/.myagent/providers/*.yml)
# + newSessionStore + baseDir derivation + ensure dirs
var engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  skillsDir = "skills",
  maxIterations = 10,       # max tool-loop iterations per turn (chachachat runToolLoop)
  truncateTo = 50           # keep last N non-system messages in history; 0 = keep all
)
# mode is not set at engine init — set per agent/session below
# or from YAML: let cfg = loadEngineConfig("engine.yml"); var engee = initDatEngine(cfg)

# providers via global single source at ~/.myagent/providers (add via API, not init):
engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")
# name unique: openai -> openai-1/-2 on collision
# users can also add ~/.myagent/providers/ollama.json manually:
# {"name":"ollama","baseUrl":"http://localhost:11434/v1","model":"llama3"}
echo engee.listProviders().len

# model discovery (async, cached in ProviderStore under models:<name>):
import std/asyncdispatch
let models = waitFor engee.fetchProviderModels("openai")          # uses stored baseUrl/apiKeyEnv
# or before provider exists: let models = waitFor engee.fetchProviderModelsForUrl("https://opencode.ai/zen/go/v1", "", "")
# pick and update: var cfg = engee.getProvider("openai").get; cfg.model = models[0].id; engee.upsertProvider(cfg)
# cached access: let cached = engee.getProviderModels("openai")
echo models.len

Phase 2 — per workspace / per agent (per session / per request):

# isolated Workspace at ./storage/workspaces/<sessionId> + artifacts, gitignore-filtered
# AgentMode: amAsk (default, readOnly), amPlan (readOnly), amBuild (writable 10 MB caps)
var agent = engee.newAgent(sessionId) # or engee.newAgentForUser(sessionId, userId)
agent.setMode(amBuild)                # set mode per agent/session (not at engine init)
# mode-aware tools + skills are auto-wired inside (Ask registers no fs_tools)
let resp = waitFor agent.run("Analyze the files in this workspace")
echo resp.text

# runtime mode switch
engee.setMode(amPlan)              # affects default for next workspaces
agent.setMode(amPlan)              # affects this agent's workspace (re-applies PolicyRules)
let ws = engee.getWorkspaceForSession(sessionId)
echo ws.root # ./storage/workspaces/<id>

Tool System

• Allowlisted CLI Binary allowlist, no shell metacharacters, cwd confinement (Workspace.root), per-tool output caps and timeouts

• rtk proxy Token-optimized output for the model (ls, tree, read, grep, find, diff, wc, json)

• Document extraction: pdftotext, pdfinfo, pdftoppm, pdftohtml, vips, sips, convert, ffmpeg; renders/screenshots land on the per-session artifacts disk (Workspace.artifactWrite)

• Per-session gitignore-aware workspace Workspace owns a per-session Filesystem with disks workspace (filtered via pkg/gitignore IgnoreStack; .env, .git, node_modules never reach the model) + artifacts (unfiltered sibling for downloads/renders). Every path is LocalDriver.resolvePath traversal-proof and atomic. Global state lives on a separate host-wide globalFs: Filesystem with named disks skills/config at ~/.myagent. AgentMode (ask/plan/build) enforces flysystem PolicyRules (ask/plan = readOnly=true, build = writable with 10 MB caps).

• Browser automation chopchop CDP: goto, waitForNavigation(NetworkIdle), querySelector, evaluate, screenshot, click, typeText

• Safety envelope per-tool byte/line caps, timeouts (30/60/120s), truncation markers, process kill on timeout, plus PolicyError on AgentMode violations

• Per-session todos Session.todos: seq[TodoItem] (id, content, status: pending|in_progress|completed|cancelled, priority: high|medium|low) persisted via SessionStore (sessions.todosJson). LLM tools todo_create(content, priority?) → id, todo_update(id, content?, status?, priority?) (single-item patch by id), todo_delete(id), todo_read (explicit, not auto-injected). Enforced “plan before build”: in amPlan/amBuild any non-todo tool is blocked until at least one todo exists (ask exempt). Managed per-session, visible across newAgent(sessionId) reloads.

Providers

Providers describe where the engine sends language model requests. Any service with an OpenAI-compatible API can be used by giving its address, the model to use, and where to find the API key.

Providers are global. They are defined once and shared by every session and workspace. They can be managed from inside the app or by editing small text files in the agent home directory.

Each provider has a unique name. If a new provider reuses an existing name, the engine adjusts it automatically so nothing is overwritten.

When adding a new provider, the app can ask the service which models it offers and let the user pick one. That list is remembered for later, and it is only refreshed when explicitly requested.

Plugins

Plugins are optional global extensions that give the assistant extra tools at runtime. They are shared by every session and workspace.

A plugin is a compiled library placed in the agent home directory. The app controls its whole lifecycle: installing, loading, activating, unloading, and uninstalling are all explicit actions. Nothing is loaded automatically, and closing the engine unloads everything that is still active.

Each plugin declares the tools it offers, and the engine attaches the tools of every active plugin to each new session, alongside the built-in tools. Plugin tools run under the same safety rules as built-in tools, including the read-only restrictions of the ask and plan modes.

Plugin authors write plugins with the pluginkit macro DSL, the same style as a regular pluginkit plugin: a manifest block plus lifecycle hooks, with one tool definition per tool. See the documented example in the datengine plugin shim for authors.

Plugin side (myplugin.nim):

import datengine/tools/plugin

plugin myplugin, {
  name: "MyPlugin",
  author: "Example",
  description: "Echo text back to the model",
  license: "MIT",
  url: "https://example.com",
  version: "0.1.0"
}:
  oninit do:
    echo "MyPlugin ready"
  onunload do:
    echo "MyPlugin bye"

datengineTool my_echo, "Echo text",
    """{"type":"object","properties":{"text":{"type":"string"}},"required":["text"]}""":
  "echo:" & args{"text"}.getStr("")

exportDatengineTools()

Compile:

nim c --app:lib --mm:orc --threads:on -o:myplugin.dylib myplugin.nim
# Linux: .so, macOS: .dylib, Windows: .dll

Host side (datengine):

import datengine
import std/asyncdispatch

let engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  mode = amBuild
)
discard engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")

# install persists to ~/.myagent/plugins/ (or custom AgentConfig.pluginsDir)
let dest = engee.installPlugin("/path/to/myplugin.dylib")
# or: let dest = engee.installPlugin("/path/to/myplugin.dylib", "myplugin.dylib")

# load + activate (app-controlled, no auto-load)
let id = engee.loadPlugin(dest)       # → hash id, checks ABI/semver/NimVersion at pluginkit.nim:515
engee.activatePlugin(id)              # calls plugin_init (NimMain), status → pluginStatusActive

# discovery
echo engee.getPluginsDir()            # ~/.myagent/plugins or custom
echo engee.listInstalledPlugins()     # ["…/plugins/myplugin.dylib"]
echo engee.listLoadedPlugins().len    # 1
echo engee.hasPlugin(id)              # true

# per-session attach: engine.bindAgent attaches currently loaded plugins to each newAgent
let agent = engee.newAgent("sess-1")
assert agent.registry.hasTool("my_echo")
assert agent.registry.hasTool("todo_create") # built-ins remain
let res = waitFor agent.registry.getTool("my_echo").get.handler("my_echo", %*{"text":"hello"})
echo res # echo:hello

# unload / uninstall (app-controlled)
engee.unloadPlugin(id)
assert not engee.hasPlugin(id)
engee.uninstallPlugin(id) # not needed if already unloaded: no-op; otherwise unloads then removeFile
# or after re-load:
# let id2 = engee.loadPlugin(dest); engee.activatePlugin(id2); engee.uninstallPlugin(id2)
# engee.close() unloads all remaining plugins

See src/datengine/tools/plugin.nim:1 shim and src/datengine/plugins.nim:60 attachPluginTools.

Storage

• Boogie RDBMS https://github.com/openpeeps/boogie
  Indexed relational store for sessions (indexed columns, structured queries)

• Boogie DocumentStore https://github.com/openpeeps/boogie
  Schemaless JSON store for providers (providers docstore, putObj/getObj, pairs), single global file.

• Flysystem https://github.com/openpeeps/flysystem
  Multi-disk sandbox: per-session Workspace.fs (workspace + artifacts disks, traversal-proof, atomic writes) + host-wide globalFs (skills + config + providers + plugins disks at ~/.myagent). All reads/writes go through StorageDriver; no raw readFile paths escape the engine.

• OpenParser https://github.com/openpeeps/openparser
  Collection parsers/dumpers: Full QR family/JSON/TOML/YAML/FBE/DotEnv/iCal/Regex/SQL/Gettext (po/mo) and more

Skills

Markdown + YAML frontmatter; the engine injects matching raw skill bodies into the system message each turn:

---
name: pdf-analysis
description: How to extract and analyze PDF documents
keywords: [pdf, extract, document, ocr]
---
# PDF Analysis
(instructions for the LLM...)

Testing

• Mock LLM: mock OpenAI-compatible server with streaming SSE and tool_call responses
• ~155 tests: core types, agent lifecycle, tool safety, session round-trip, config parsing, truncation, persistence, boogie storage, skills, providers (DocumentStore, YAML/JSON sync, unique suffix, globalFs)

With Skills

Skills are opt-in via flysystem disks: host-wide globalFs (~/.myagent/skills) and per-session Workspace (<workspace>/skills). Loaded through drivers, not raw paths.

~/.myagent/                    # host globalFs (newGlobalFs)
├── skills/
│   └── pdf-analysis.md       # available in every session
└── config/
    └── engine.yml

<workspace>/                   # per-session Workspace.root via newWorkspace/forSession
└── skills/
    └── project-specific/     # <name>/SKILL.md layout also works
        └── SKILL.md

Legacy newSkillRegistry(fs, skillsDir, globalPath) and newFsTool(root) shims remain for single-workspace scripts, but web apps should use globalFs + Workspace + newSkillRegistryFromDrivers.

On each run, user input is fuzzy-matched against skill keywords and names (floof); matching raw markdown bodies are injected into the system message for that turn. The model can also call skill_list / skill_read explicitly. Agent.setMode and Workspace.setMode can be used to gate plan vs build at runtime.

Architecture

src/datengine/
├── agent.nim          # Agent loop: chachachat Conversation + tools + hooks, holds Workspace, setMode
├── config.nim         # ProviderConfig(name, baseUrl, model, apiKeyEnv) — name globally unique; EngineConfig (agent only, no provider; providers via ProviderStore)
├── models.nim         # Model discovery: LLModel/ModelListResponse via openparser fromJson, async GET {baseUrl}/models (e.g. https://opencode.ai/zen/go/v1/models) + caching
├── mockllm.nim        # OpenAI-compatible mock server for testing
├── prompt.nim         # System prompt builder from tool schemas
├── providers.nim      # Global providers (DocumentStore at ~/.myagent/providers.ddb + globalFs disk providers, OpenAPI-compatible, YAML/JSON via openparser, unique suffix, syncFromGlobalFs) + model cache models:<name>
├── serialization.nim  # openparser glue: fromJsonArgs, jsonOrEmpty, helpers
├── session.nim        # Indexed relational store for sessions (indexed columns, structured queries) + per-session todos (persisted todosJson, LLM-managed via todo_*)
├── skills.nim         # Skill loading via flysystem drivers (globalFs + workspace) + floof matching
├── workspace.nim      # Per-session Workspace (flysystem Filesystem: workspace + artifacts) + globalFs (skills/config/providers at ~/.myagent), AgentMode PolicyRules, gitignore stack, per-session forSession helper
├── engine.nim         # High-level DatEngine (initDatEngine large params, auto sync providers, newAgent factory, getters/setters, fetchProviderModels async wrappers, PluginManager (global plugins at ~/.myagent/plugins), no global agent)
├── plugins.nim        # Host plugin manager (global, tools-only, app-controlled load/activate/install/uninstall, per-session tool attach)
├── tools.nim          # Tool, ToolRegistry, ToolResult, JSON Schema helpers
└── tools/
    ├── cli.nim        # Allowlisted subprocess (threadpool, caps, timeouts): workdir = Workspace.root
    ├── rtk.nim        # rtk output proxy
    ├── document.nim   # poppler/vips/sips/ffmpeg wrappers: outputs to artifacts disk
    ├── fs.nim         # FsTool adapter over Workspace disk (shares LocalDriver + IgnoreStack)
    ├── todo.nim       # Per-session todos (persisted via SessionStore, id-based todo_create/update/delete/read, plan-before-build enforcement)
    ├── plugin.nim     # Author-facing shim over pluginkit (plugin manifest DSL plus datengine tool definitions)
    └── browser.nim    # chopchop CDP browser automation: screenshots to artifacts disk

Dependencies

Package	Version	Role
chachachat	>= 0.1.0	LLM client, SSE streaming, agent loop, tool calling
openparser	>= 0.1.9	JSON/YAML direct-to-object serialization
flysystem	>= 0.1.0	Multi-disk filesystem sandbox
boogie	>= 0.1.2	A suite of WAL-based embedded data stores. RDBMS, KV Store, GraphStore, VectorStore, Columnar and more
gitignore	>= 0.1.0	Spec-compliant ignore stack for workspace sandbox
chopchop	>= 0.1.0	CDP browser automation (goto, evaluate, screenshot, click)
powpow	>= 0.1.9	Event loop, file watcher, HTTP/WS server
marvdown	>= 0.1.4	Markdown parser: YAML frontmatter for skills, HTML output, JSON AST
sweetsyntax	>= 0.2.0	YAML-driven syntax highlighter & AST explorer: ANSI, HTML, JSON renderers, code folds
pluginkit	>= 0.1.1	Plugin manager: macro DSL for dylibs, semantic versioning, permission system, lifecycle hooks
floof	>= 1.0.0	SIMD-accelerated fuzzy search: skill keyword matching against user input

Planned: BU CLI tools

c-blake/bu: ~70 Nim-native CLI tools (pipe-oriented, zero-config, faster than GNU coreutils). Tier 1 integration planned for agent toolchains:

Tool	Purpose
dups	Find duplicate-content files
topn	Top-N rows by any column, single-pass
ndelta	Numeric diff between two reports
cols	Extract columns from delimited text
noc	Strip ANSI escape sequences
ft	Batch file type test
newest	Find N newest/oldest files by timestamp
since	Find files newer than a reference
cstats	Summary stats for numeric columns
catz	Universal decompressor (auto-detect format)
ru	High-precision resource usage measurement
oft	Most-frequent items (count-min sketch)
tails	Unified head+tail with both-ends support

Roadmap

• Core types (Tool, ToolResult, ToolRegistry, serialization)
• Provider layer (chachachat: LLMClient, SSE, streaming hooks)
• Agent loop (Conversation-backed turns, tool calling, truncation, cancellation)
• CLI/RTK tools (allowlist, threadpool subprocess, safety envelope)
• FS tools (flysystem + gitignore workspace sandbox)
• Workspace (per-session flysystem Filesystem: workspace + artifacts, host-wide globalFs skills/config, AgentMode ask/plan/build with PolicyRules, forSession helper)
• Document tools (poppler/vips/sips/ffmpeg)
• Browser tools (chopchop CDP)
• Session persistence (Boogie RDBMS store)
• Skills (marvdown frontmatter, floof fuzzy matching, global + workspace sources via flysystem drivers)
• Config (YAML parsing with defaults, AgentMode, workspace base)
• Mock LLM server (OpenAI-compatible, streaming SSE, tool_call)
• Test suite (140 tests across 7 test files)
• BU CLI tools (dups, topn, ndelta, cols, noc, ft, ...)
• Prompt caching and context window management
• Rate limiting and retry policies
• Multi-agent orchestration
• Vision pipeline (pdftoppm → vips → base64 → model)
• RAG integration (Boogie vector retrieval)
• REST/WebSocket transport layer (powpow HTTP server)
• Web UI for agent interaction

Contributions & Support

• 🐛 Found a bug? Create a new Issue
• 👋 Wanna help? Fork it!

License

LGPLv3 license. Made by Humans from OpenPeeps.
Copyright OpenPeeps & Contributors — All rights reserved.