diff --git a/.env.example b/.env.example index a1de091..03ab457 100644 --- a/.env.example +++ b/.env.example @@ -27,7 +27,7 @@ DECKLE_IMAGE=ghcr.io/authortom/deckle:latest # ---- Server library ---------------------------------------------------------- -# With the server library on, your notes are stored as .md files in the container +# With the server library on, your notes and files are stored in the container # (the "nib-vault" volume in compose.yaml — named before the rename, and kept # that way so upgrades don't strand it) instead of on your device — so Deckle # works from any browser, including Safari and mobile, with nothing kept @@ -66,23 +66,39 @@ DECKLE_SESSION_SECRET= # wrong passwords from anyone lock everyone out for fifteen minutes. # DECKLE_TRUST_PROXY=true -# Optional. Where Deckle keeps its own state — currently the assistant settings -# shared by every device that signs in, API key included. Defaults to -# ".deckle-state" inside the library directory, which is the volume you already -# mount; it is deliberately *not* part of the library, so it never appears in a -# listing, an export or the API. -# -# Sharing those settings requires DECKLE_PASSWORD. Without one the server keeps -# no key at all — it would otherwise hand it to anyone who can reach the port — -# and every browser keeps its own, as it always has. +# The largest single file the library accepts, in megabytes (default 100). An +# agent's reports, spreadsheets and images live here as well as your notes; +# this stops one upload filling the volume. +# DECKLE_MAX_FILE_MB=100 + +# The folder projects live in (default "Projects"). Every folder inside it is a +# project in the app's Projects view, and it is where an agent's +# create_project tool puts new ones. +# DECKLE_PROJECTS_DIR=Projects + +# Log and project-log times use the container's clock. Set your time zone so +# they read as your local time. +# TZ=Europe/London + +# Optional. A folder Deckle keeps to itself: never listed, exported or served. +# Deckle 3 writes nothing there, but Deckle 2.x kept the AI assistant's +# settings in it — provider API key included — so on an upgraded volume it may +# still hold one. Delete ".deckle-state/assistant.json" from the volume once +# you no longer need it. Defaults to ".deckle-state" inside the library. # DECKLE_STATE_DIR=/data/.deckle-state -# ---- API (for agents and scripts) ------------------------------------------ -# Deckle exposes a REST API at /api/v1 so an agent can search, read and write your -# knowledge base. It is OFF unless you set a token below, and it needs the -# server library above — a local folder or in-browser library lives on your device, -# where nothing outside the browser can reach it. +# ---- API and MCP (for agents and scripts) ----------------------------------- +# Deckle exposes a REST API at /api/v1, and an MCP server at /api/v1/mcp, so an +# agent such as Hermes can search, read and write your knowledge base — notes +# and files alike — and keep its work in projects. See +# integrations/hermes/README.md for connecting Hermes. Both are OFF unless you +# set a token below, and both need the server library above — a local folder or +# in-browser library lives on your device, where nothing outside the browser +# can reach it. +# +# Every change made with a token is recorded in the app's Activity feed under +# the token's name, so name tokens after the agent that holds them. # # Comma-separated tokens. Each is either a bare secret (read-write) or # "name:scope:secret", where scope is "r" (read-only) or "rw". Secrets must be @@ -91,8 +107,9 @@ DECKLE_SESSION_SECRET= # DECKLE_API_TOKENS=hermes:rw:REPLACE_WITH_A_LONG_RANDOM_SECRET # # TREAT THESE LIKE PASSWORDS. A read-write token can create, edit and delete -# every note. Give each consumer its own token so you can revoke one without -# disturbing the rest, and prefer "r" for anything that only needs to read. +# every note and file. Give each consumer its own token so you can revoke one +# without disturbing the rest, and prefer "r" for anything that only needs to +# read — over MCP, a read-only token is offered the read tools only. DECKLE_API_TOKENS= # Optional. Origins allowed to call /api/v1 from a browser (comma-separated, or @@ -101,7 +118,3 @@ DECKLE_API_TOKENS= # /api/v1, so a permitted origin still has to present a token. # DECKLE_API_CORS_ORIGINS= - -# ---- AI assistant ---------------------------------------------------------- -# Nothing to set here. API keys are entered in the app's assistant settings and -# stored in your browser — they are never sent to or held by the Deckle server. diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a809ab..0cf9bad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,120 @@ here. Nothing yet. +## [3.0.0] — 2026-09-21 + +A change of direction. Deckle stops being a note app with an AI assistant +built in and becomes the knowledge base for an agent that lives elsewhere — +Hermes, or anything else that speaks MCP or HTTP. The agent does the thinking; +Deckle stores what it produces, shows it, searches it, and keeps a record of +what it did. **Read Removed before upgrading** if you used the assistant. + +### Added + +- **Files, not just notes.** The library holds any file beside the notes — + PDFs, spreadsheets, Word and PowerPoint documents, images, audio, video, + CSV, JSON, code — shown in the tree with their own icons and opened in a tab + like a note. The viewer shows images, PDFs (in the browser's own viewer), + audio and video; renders Word documents, every sheet of an Excel workbook, + PowerPoint slides, CSV tables, JSON and text; and offers anything else as a + download. Files move, rename, delete and restore exactly as notes do, and + byte for byte: moving, binning or restoring one on the server library is a + rename, with nothing downloaded or uploaded again. +- **An MCP server at `/api/v1/mcp`**, behind the same tokens as the REST API. + Tools: `search`, `read`, `list`, `list_projects`, `recent_activity`, and — + for a read-write token — `write_note`, `save_file`, `create_project`, + `update_project`, `log_progress`, `move` and `delete`. Streamable HTTP, + stateless, answering in JSON. A read-only token is shown the read tools + only. +- **Projects.** Every folder inside `Projects/` is a project, with a status, + summary and tags in its `Overview.md` front matter and a dated `Log.md` the + agent keeps as it works. A **Projects** view lists them by what changed last, + filters by status, changes a status in place, and starts new ones. +- **Activity.** Every change made with an API token or over MCP is recorded in + `.deckle/activity.jsonl` under the token's name, with whatever the agent said + about it. The **Activity** feed shows it grouped by day, filterable by + project and agent, and the Activity button in the top bar counts what is new + since you last looked. A file's viewer says which agent saved it, when, and + why. +- **The app notices an agent's changes.** On a server library it checks the + activity log every 15 seconds while the tab is visible, and on focus, and + reloads the tree and any open note when it moves — so a file an agent saves + appears without a refresh. If an agent rewrites the note you are typing in, + your typing still wins, but its version is kept in history first. +- **Front matter is kept, and shown.** A note's YAML front matter appears as a + properties strip above the text, editable as YAML, and is never passed to + the editor — which used to turn the fence into a rule and a heading and save + that back. +- **Images in notes.** `![chart](charts/revenue.png)` shows the image, from the + library or the web. +- **Search inside documents.** `GET /api/v1/search` and MCP's `search` rank + notes and files together, reading the text inside Word, Excel, PowerPoint, + OpenDocument, CSV, JSON, text and code files. PDFs and images are matched by + name. `kind=note|file` narrows a search to one or the other. The sidebar's + quick search finds files by name. +- **`/api/v1/files`** — store any file from a raw request body (`PUT`), fetch it + (`GET`, or `?meta=true` for its details), bin it (`DELETE`), and list files + newest first. Replacing a file moves the old copy to the recycle bin; `.md` + paths are written as notes, with history. +- **`GET /api/v1/activity`** — the activity log, filterable by `since`, + `project`, `actor` and `path`. +- **A Hermes integration kit** in `integrations/hermes/`: setup instructions, a + skill that teaches Hermes how to use Deckle, and `deckle_push.py`, a + standard-library Python script for sending large files and whole folders — + which never sends hidden files, `.env` files, private keys or dependency + caches, and honours a `.deckleignore`. +- **Importing takes any file.** Dropping or picking PDFs, images or whole + folders stores them as they are; a ZIP brings everything inside it, not just + its Markdown. Hidden files and `node_modules`-style caches stay behind. +- `DECKLE_MAX_FILE_MB` (default 100) caps a single file; `DECKLE_PROJECTS_DIR` + (default `Projects`) moves the projects folder. + +### Removed + +- **The AI assistant**, and everything that came with it: Clever Trevor's chat + sheet, the background queue, the assistant's memory, Ask AI on a selection, + the `/ask`, `/summarize`, `/explain` and `/todo` commands, semantic search, + and the model picker. `Ctrl`/`Cmd`+`J` does nothing now. Nothing in your + library is touched: the files the assistant wrote are ordinary notes, and + its own folders — `.deckle/memory/`, `.deckle/runs/` — stay where they are + until you delete them. +- **`/api/assistant-settings`**, and the server's copy of the assistant's + settings. If you used a password-protected server library, **the provider API + key you entered is still in `.deckle-state/assistant.json` on the volume**. + Deckle keeps refusing to serve that folder, but it no longer needs the file: + delete it, and revoke the key with your provider if you are done with it. +- The `@anthropic-ai/sdk` dependency. + +### Changed + +- **The API's folder tree includes files.** `GET /api/v1/folders` (and the app's + own tree) now carries every file, as `kind: "asset"` nodes with `name`, + `title`, `ext` and `size`, beside the notes' `kind: "file"`. Code that + assumed only folders and notes should skip the new kind. `GET /api/v1/notes` + still lists notes only. +- **Deleting a folder bins everything in it.** `DELETE /api/v1/folders/{path}` + and the app used to send a folder's notes to the recycle bin and erase any + other file inside it; files now go to the bin too. +- **Version history made through the API says "agent"** rather than "AI" + (older snapshots keep their label). Every API write is also in the activity + log. +- **The Content-Security-Policy is tighter.** `connect-src` is `'self'` — the + page talked to AI providers, and now talks to nothing but this server — and + `blob:` is allowed for frames, objects and media, which is how a stored PDF, + audio or video file is previewed. +- The side panel is **Projects · Activity · Tasks · Bookmarks**. +- `@types/node` is now a declared dev dependency; it used to arrive by way of + the Anthropic SDK. + +### Fixed + +- **Images vanished from notes.** The editor had no image node, so any + `![alt](src)` in a note was dropped on load and deleted from the file by the + next save. +- **A note's front matter was mangled** by editing the note in the app. +- Downloading a note as `.md` now copies the file itself, front matter + included, rather than what the editor holds. + ## [2.0.0] — 2026-09-15 A hardening release: security fixes, several ways data could be lost, and a @@ -440,7 +554,8 @@ a command palette; light and dark mode with seven palettes; a responsive layout with drawers on a phone; and a token-authenticated REST API at `/api/v1` described by a self-served OpenAPI 3.1 document. -[Unreleased]: https://github.com/authorTom/deckle/compare/v2.0.0...HEAD +[Unreleased]: https://github.com/authorTom/deckle/compare/v3.0.0...HEAD +[3.0.0]: https://github.com/authorTom/deckle/compare/v2.0.0...v3.0.0 [2.0.0]: https://github.com/authorTom/deckle/compare/v1.6.0...v2.0.0 [1.6.0]: https://github.com/authorTom/deckle/compare/v1.5.1...v1.6.0 [1.5.1]: https://github.com/authorTom/deckle/compare/v1.5.0...v1.5.1 diff --git a/README.md b/README.md index c2f9f91..06c0eeb 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,39 @@ # Deckle -**A clean, minimalist, web-based knowledge management platform.** - -Markdown notes with live WYSIWYG editing, a Todoist-style task planner, -bookmarks with collections and comments, and an AI assistant that can search and -edit your knowledge base — plus a keyboard-driven command palette, light and -dark mode, and a fully responsive layout. - -Everything lives as plain Markdown files in a folder you can open, and you -choose where: on your own computer, or — if you self-host with Docker — in a -volume on your own server, which makes Deckle fully web based and leaves nothing -on the device you're using. No third-party account, no lock-in. +**A self-hosted knowledge base for you and your agents.** + +Deckle is where an AI agent's work lives once the conversation is over. Point +an agent such as [Hermes](integrations/hermes/README.md) at it over MCP and it +searches what is already known before it starts, keeps each piece of work in a +project, saves what it produces — notes, reports, PDFs, spreadsheets, images, +code — and logs what it did and why. You review all of it in one place: files +open in the app, an activity feed shows every change under the agent's name, +and projects carry a status and a running log. + +It is also a good place to write yourself: Markdown notes with live WYSIWYG +editing and `[[wikilinks]]`, a Todoist-style task planner, bookmarks with +collections and comments, a keyboard-driven command palette, light and dark +mode, and a fully responsive layout. + +Everything lives as plain files in folders you can open, and you choose where: +on your own computer, or — if you self-host with Docker — in a volume on your +own server, which is what an agent needs to reach it. No third-party account, +no lock-in, and no AI inside Deckle itself: the agent does the thinking, Deckle +keeps the record. **[deckle.redacre.net](https://deckle.redacre.net/)** — the project website: what Deckle does, how to install it, and the questions people ask first. There is no hosted version to sign up for; you run it yourself. -![Markdown notes with a folder tree, editor tabs, live WYSIWYG editing, and a backlinks panel](docs/screenshots/editor.png) +![An agent's project in Deckle: its files in the tree — notes, a PDF report, a spreadsheet, a chart, a CSV — the Projects panel with status and what's new, and a findings note with its properties and an embedded chart](docs/screenshots/projects.png) -| Task planner — Inbox, Today, Upcoming with a mini calendar | Bookmarks — collections and per-bookmark comments | +| Files an agent saved — here a spreadsheet, with who saved it and what links to it | Activity — every change an agent made, by name (dark mode) | | --- | --- | -| ![Todoist-style task planner with due dates, priorities, and a month calendar](docs/screenshots/tasks.png) | ![Bookmarks with coloured collections and a comment box](docs/screenshots/bookmarks.png) | -| **AI assistant — search and edit your library (dark mode)** | **Command palette — everything a keystroke away** | -| ![AI assistant panel in dark mode](docs/screenshots/assistant.png) | ![Command palette in dark mode, listing commands from new note to split editor and themes](docs/screenshots/palette.png) | +| ![A spreadsheet open in Deckle, with its sheets, the agent that saved it, and the notes that link to it](docs/screenshots/files.png) | ![The Activity feed in dark mode beside a PDF report the agent saved](docs/screenshots/activity.png) | +| **Writing — live Markdown, tabs, wikilinks and backlinks** | **Task planner — Inbox, Today, Upcoming with a mini calendar** | +| ![Markdown notes with a folder tree, editor tabs, live WYSIWYG editing, and a backlinks panel](docs/screenshots/editor.png) | ![Todoist-style task planner with due dates, priorities, and a month calendar](docs/screenshots/tasks.png) | +| **Bookmarks — collections and per-bookmark comments** | **Command palette — everything a keystroke away** | +| ![Bookmarks with coloured collections and a comment box](docs/screenshots/bookmarks.png) | ![Command palette in dark mode, listing commands from new note to projects and agent activity](docs/screenshots/palette.png) | ## Why it exists @@ -31,43 +42,88 @@ polished but keep your notes in their database, on their terms, for as long as they stay in business. The local ones keep your files but tie you to one machine, or to a sync service you also have to trust. -Deckle keeps the files as ordinary `.md` on disk — openable in any editor, -syncable, backed up by whatever you already use — while running entirely in the -browser. Self-host it and the same library is reachable from a laptop, a phone or -Safari with nothing stored on the device. +Deckle keeps the files as ordinary files on disk — `.md` notes beside whatever +else you keep, openable in any editor, syncable, backed up by whatever you +already use — while running entirely in the browser. Self-host it and the same +library is reachable from a laptop, a phone or Safari with nothing stored on +the device. -The AI assistant follows the same principle: it can read and rewrite your whole -library, but every change to a note is shown as a diff you approve first, and -your API keys go nowhere except the provider — held in your browser, or, if you -self-host behind a password, on your own server so your other devices are -already set up. The one thing it writes on its own is its memory of how you -work, which is Markdown you can read and delete. +Agents make the same problem worse. An agent that researches, writes and builds +things for you produces a stream of output — reports, datasets, drafts, charts — +that ends up scattered across its working directory and lost with the +conversation. Deckle gives that work a home you own: plain files in a folder, +organised into projects, searchable by the agent next time it needs them, and +reviewable by you, with a record of who changed what. Every change an agent +makes is as recoverable as your own. ## What it does +**For your agent's work** + +- **An MCP server** at `/api/v1/mcp`, so an agent such as Hermes can use + Deckle directly: `search`, `read`, `list`, `list_projects`, + `recent_activity`, `write_note`, `save_file`, `create_project`, + `update_project`, `log_progress`, `move` and `delete`. It uses the same tokens + as the REST API, and a read-only token is offered the read tools only. See + [Connecting Hermes](integrations/hermes/README.md). +- **Files, not just notes** — PDFs, spreadsheets, Word and PowerPoint + documents, images, audio, video, CSV, JSON and code sit in the tree beside the + notes and open in a tab. Images, PDFs, audio and video show in the browser's + own viewers; Word documents, every sheet of a workbook, slides, CSV tables, + JSON and text are rendered in the app; anything else is a download away. + Files move, rename, delete and restore exactly as notes do. +- **Projects** — every folder in `Projects/` is a project, with a status, + summary and tags in its `Overview.md` and a dated `Log.md` the agent keeps as + it works. The Projects view lists them by what changed last, filters by + status, and changes a status in place. +- **Activity** — every change made through the API or MCP is recorded under the + name of the token that made it, with whatever the agent said about it. The + Activity feed groups it by day, the top bar counts what's new since you last + looked, and a file's viewer says who saved it and why. The log itself is + `.deckle/activity.jsonl` in your library. +- **Live** — on a server library, an agent's changes appear in an open app + within about fifteen seconds, without a refresh. +- **Search inside documents** — the API and MCP search rank notes and files + together, reading the text inside Word, Excel, PowerPoint, OpenDocument, CSV, + JSON, text and code files. PDFs and images are found by name, and by the notes + that link to them. +- **Front matter** — YAML at the top of a note (the way agents and Obsidian + describe notes) is shown as a properties strip above the text, editable as + YAML, and never damaged by the editor. +- **A push script** — [`deckle_push.py`](integrations/hermes/skills/deckle/scripts/deckle_push.py) + sends large files and whole folders, never sending hidden files, `.env` files, + private keys or dependency caches. + +**For writing and reading** + - **Live WYSIWYG markdown** — type markdown (`# `, `**bold**`, `- list`) and it - renders inline as you go (powered by TipTap / ProseMirror). + renders inline as you go (powered by TipTap / ProseMirror). Images show in + place, from the library or the web. - **Local folder library** — pick a folder and Deckle reads and writes your notes - there as real `.md` files. Open the same folder in another editor, sync it, or - back it up — it's just Markdown on disk. (Chromium browsers; Safari and Firefox use + there as real files. Open the same folder in another editor, sync it, or + back it up — it's just files on disk. (Chromium browsers; Safari and Firefox use private in-browser storage — see [Where your notes are stored](#where-your-notes-are-stored).) - **Or a server library** — self-host with Docker and your notes can live in a volume on your own server instead, password protected, reachable from any - browser or device with nothing stored locally. + browser or device with nothing stored locally. This is the one an agent can + reach. - **Folder tree** — browse nested subfolders, create folders, and - **drag-and-drop** notes between them: drop onto a folder, onto any note - already inside one, or hold over a shut folder and it springs open so you can - carry a note further down in one go. + **drag-and-drop** notes and files between them: drop onto a folder, onto any + note already inside one, or hold over a shut folder and it springs open so you + can carry a note further down in one go. - **Move to another folder…** — when dragging isn't practical (a long tree, a - collapsed destination, a phone), move a note by naming its destination - instead: from the note's row in the tree, from a search result, from the - editor menu, or from the command palette. Pick a folder from the whole - library, or type a name — including a nested one like `Archive/2026` — and it - is created on the way. + collapsed destination, a phone), move a note or file by naming its destination + instead: from its row in the tree, from a search result, from the editor + menu, or from the command palette. Pick a folder from the whole library, or + type a name — including a nested one like `Archive/2026` — and it is created + on the way. +- **Wikilinks and backlinks** — `[[Note]]` or `[[report.pdf]]` link to notes + and files alike, and every note and file shows what links to it. - **Command palette** — `Ctrl`/`Cmd`+`K` opens a fast, fully keyboard-driven - palette for commands, formatting, and jumping to any note. -- **Search across the library** — find notes by title, path, or file contents. + palette for commands, formatting, and jumping to any note or file. +- **Search across the library** — find notes by title, path, or contents, and + files by name. - **Tasks and planner** — a Todoist-style panel docked on the left: Inbox, Today, Upcoming (agenda and mini month calendar), projects, priorities (P1–P4), recurring tasks, a completed log, and a bin for deleted tasks @@ -79,53 +135,13 @@ work, which is Markdown you can read and delete. with its own comment box. Paste a link to add it, or select a link in a note and use the bookmark button — captured bookmarks link back to their source note. Stored in `.deckle/bookmarks.json` in the library. -- **Version history** — restore points are saved automatically: before every AI - edit, periodically while you type, and before every restore. Open the clock - icon to preview, restore or delete a note's earlier versions (kept in a hidden - `.history` folder, 20 per note). -- **Recycle bin** — deleted notes move to a hidden `.trash` folder and can be - restored or permanently removed. -- **AI assistant** — an optional right-side chat panel that can search and read - your whole library and create, edit, move or delete notes and folders. Every - change to a note is shown as a diff and must be approved before it runs. Ask questions - about your notes and it finds and cites the relevant ones. Works with an - Anthropic, OpenAI or [OpenRouter](https://openrouter.ai) API key, or fully - locally via [LM Studio](https://lmstudio.ai). Reasoning models show a - collapsible "Thought process". The model it is using is named above the - conversation and switches from there — provider included, so moving between a - local model and Claude is one click, not a trip to a settings screen. Your API - key stays in your browser — or, on a password-protected server library, on the - server, so every device you sign in from is configured already. -- **Send it, or queue it** — the assistant's composer has two buttons. *Send* - answers in the conversation; *Queue* hands the same words to the background - and gives you the box back. The panel has a **Queue** tab beside Chat holding - everything in flight — what needs you, what is working, what is waiting its - turn — and one line above the composer keeps you posted while you chat. Jobs run one at a time or several at once, and their - output lands in an **Assistant inbox** folder. Each is pinned to the model you - queued it against — send the long one to Opus and the tidy-up to Haiku, and - see which ran what. That folder is the safety boundary — inside it the - assistant writes freely; anything outside stops the run and asks you to - approve the exact change, and deletions always ask. A run that needs a - decision can ask you a question and wait: only those are badged, on the - assistant button itself, so a parked run is visible with the panel closed. - Finished runs list what they wrote, and any run can be resumed or re-run. - Runs live in `.deckle/runs/` inside the library. -- **Assistant memory** — the assistant keeps what it learns about how you work - as ordinary Markdown in a hidden `.deckle/memory/` folder: one file per fact, - with a one-line index. Only that index is sent with every message; the rest is - fetched when it is relevant to what you asked, inside a token budget, and - recording something it already knows updates that memory rather than adding a - near-duplicate. Read, edit, pin or delete any of it from *Assistant memory* in - the command palette — or open the files in any editor. Off with one tick box. -- **Semantic library search (optional)** — enable embeddings in the assistant - settings (OpenAI or a local LM Studio embedding model) and library search - matches by meaning, not just keywords. Vectors are cached locally and only - changed notes are re-embedded; keyword (BM25) search always works without it. -- **Inline Ask AI** — highlight text and click the brain icon to improve, fix, - shorten, summarise, explain, or run a custom prompt on just that passage, then - Replace / Insert / Copy the result. -- **Quick formatting** — toolbar and floating selection menu for bold, italic, - strikethrough, inline code, headings (H1–H3), lists and quotes. +- **Version history** — restore points are saved automatically: periodically + while you type, before an agent rewrites a note, and before every restore. + Open the clock icon to preview, restore or delete a note's earlier versions + (kept in a hidden `.history` folder, 20 per note). +- **Recycle bin** — deleted notes and files move to a hidden `.trash` folder and + can be restored or permanently removed. A file an agent replaces lands there + too, marked as replaced. - **Focus mode** — hide all chrome for distraction-free writing (`Ctrl`/`Cmd`+`Shift`+`F`, or `Esc` to exit). - **About** — *About Deckle* in the editor menu or the command palette names the @@ -133,26 +149,27 @@ work, which is Markdown you can read and delete. backends is holding it. - **Light and dark mode** — defaults to your system preference; choice persists. - **Responsive** — desktop, tablet and mobile (collapsible note drawer). -- **Import** — drop `.md` files, or a whole folder of them, anywhere on the note - tree; nested folders keep their structure and nothing is ever overwritten. - Importing from the picker asks where the notes should go first — any folder in - the library, or a new one you name — and tells you how many it found before - writing anything. +- **Import** — drop files, or a whole folder of them, anywhere on the tree; + Markdown becomes notes, everything else is stored as it is, nested folders + keep their structure and nothing is ever overwritten. Hidden files and + dependency caches such as `node_modules` stay behind. Importing from the + picker asks where things should go first — any folder in the library, or a + new one you name — and tells you how many files it found before writing + anything. - **Export** — download a note as `.md`, export it to PDF via a clean print - layout, or take the whole knowledge base as a ZIP: every note in its folder - structure plus your tasks, bookmarks and the assistant's memory, optionally + layout, or take the whole knowledge base as a ZIP: every note and file in its + folder structure plus your tasks, bookmarks and the activity log, optionally with the recycle bin and version history for a full backup. - **REST API (optional)** — a token-authenticated API at `/api/v1` so an agent or script can search, read, write and organise your knowledge base, described - by an OpenAPI 3.1 document the server publishes itself. See - [API](#api). + by an OpenAPI 3.1 document the server publishes itself. See [API](#api). ## Where your notes are stored Deckle has three storage backends behind the same library interface. Whichever you -use, your notes are ordinary `.md` files in an identical folder layout — so a -library copied from a disk folder into the server's volume (or the other way -round) just works. Only the in-browser library is awkward to copy, since it lives +use, your notes are ordinary `.md` files, beside your other files, in an +identical folder layout — so a library copied from a disk folder into the +server's volume (or the other way round) just works. Only the in-browser library is awkward to copy, since it lives in browser-managed storage rather than a folder you can open. - **A folder on your computer** — **Chromium desktop browsers** (Chrome, Edge, @@ -168,6 +185,8 @@ in browser-managed storage rather than a folder you can open. - **On your own server** — when you self-host with Docker, notes are stored in a volume on the machine running Deckle. The app then works from any browser, including Safari and mobile, and nothing is kept on the device you're using. + It is also the only one an agent can reach: the API and MCP server serve the + server library. When a server library is available, Deckle asks which you want on first load; you can switch later from the command palette (`Ctrl`/`Cmd`+`K`). Nothing is copied @@ -200,6 +219,23 @@ curl -O https://raw.githubusercontent.com/authorTom/deckle/main/.env.example mv .env.example .env # then edit DECKLE_PORT / DECKLE_IMAGE / DECKLE_PASSWORD ``` +**To let an agent in**, give it a token of its own in the same `.env` — and set +a password, because the agent's token is not the only way into an open port: + +```bash +DECKLE_PASSWORD=a-long-passphrase-for-you +DECKLE_API_TOKENS=hermes:rw:PASTE-A-SECRET-HERE # openssl rand -base64 32 +``` + +```bash +docker compose up -d +``` + +The agent then connects to `http://localhost:8080/api/v1/mcp` (or your +server's address) with that secret as a bearer token — see +[Connecting an agent](#connecting-an-agent), and +[Connecting Hermes](integrations/hermes/README.md) for Hermes step by step. + Updating: ```bash @@ -212,14 +248,18 @@ you, set `DECKLE_IMAGE` in `.env` to whichever tag matches your appetite: | Tag | What you get | | --- | --- | | `:latest` | The newest release. The default | -| `:1` | Newest `1.x` — fixes and new features, never a breaking change | -| `:1.4` | Newest `1.4.x` — fixes only | -| `:1.4.2` | Exactly that release. Never moves, so a rollback is a one-line edit | +| `:3` | Newest `3.x` — fixes and new features, never a breaking change | +| `:3.0` | Newest `3.0.x` — fixes only | +| `:3.0.0` | Exactly that release. Never moves, so a rollback is a one-line edit | | `:edge` | The tip of `main`, unreleased. For trying things, not for deployments | Every release is described in the [changelog](CHANGELOG.md), and what those numbers promise is spelled out under [Versioning](#versioning). +**On `:latest`, the next pull takes you from 2.x to 3.0**, which removes the AI +assistant — read [Upgrading from 2.x](#upgrading-from-2x) first. To stay on 2.x +for now, set `DECKLE_IMAGE=ghcr.io/authortom/deckle:2`. + **Running the image directly**, without compose. Note that the image itself ships with the server library **off**, so this is a static file server with notes on your device: @@ -252,10 +292,6 @@ use as your library — Deckle remembers it for next time (you may be asked to r access on return). In Safari or Firefox, click **Get started** to create the private in-browser library. -To use the AI assistant, open the panel (sparkles icon) → settings (gear) and -pick a provider: Anthropic (Claude), OpenAI, OpenRouter, or a local LM Studio -server. Keys are kept in `localStorage` and sent only to the provider you chose. - There is also a containerised dev server, if you would rather not install Node: ```bash @@ -272,8 +308,8 @@ DECKLE_SERVER_LIBRARY=true DECKLE_LIBRARY_DIR=./library node server/index.mjs & npm run dev # → http://localhost:5173 ``` -To run the tests — the server through its real HTTP handler, the API, every -storage path, the assistant's tools and autosave: +To run the tests — the server through its real HTTP handler, the API and MCP +server, every storage path, file previews and autosave: ```bash npm test # once, as CI runs it @@ -293,53 +329,39 @@ Only relevant when the server library is enabled. | `DECKLE_SESSION_TTL_DAYS` | `30` | How long a sign-in lasts | | `DECKLE_TRUST_PROXY` | *(none)* | `true` behind a reverse proxy (a number for a chain of them), so sign-in throttling sees real client addresses | | `DECKLE_LIBRARY_DIR` | `/data` | Where the notes live inside the container | -| `DECKLE_STATE_DIR` | `/.deckle-state` | Deckle's own state — the shared assistant settings. Never served as part of the library | -| `DECKLE_API_TOKENS` | *(none)* | Bearer tokens for the [API](#api). Blank leaves it switched off | +| `DECKLE_STATE_DIR` | `/.deckle-state` | A folder Deckle keeps to itself, never served. Empty in 3.x — see [Upgrading from 2.x](#upgrading-from-2x) | +| `DECKLE_API_TOKENS` | *(none)* | Bearer tokens for the [API and MCP server](#api). Blank leaves both switched off | | `DECKLE_API_CORS_ORIGINS` | *(none)* | Origins allowed to call `/api/v1` from a browser | +| `DECKLE_MAX_FILE_MB` | `100` | The largest single file the library accepts | +| `DECKLE_PROJECTS_DIR` | `Projects` | The folder projects live in | +| `TZ` | *(UTC)* | Time zone for the activity feed and project logs, e.g. `Europe/London` | | `DECKLE_PORT` | `8080` | Host port (compose only) | -Everything else — theme, AI provider, embeddings — is set in the app itself. On -a server library with a password, the assistant's settings (including the API -key) are kept by the server rather than by one browser, so signing in from a new -device finds the assistant already configured. See -[Assistant settings across devices](#assistant-settings-across-devices). - -### Assistant settings across devices - -The assistant's settings — provider, model, custom instructions and the API key -— normally live in the browser you typed them into. That is the right home for a -library on your own disk, but not for a server library, which exists precisely -so the same notes are reachable from a laptop, a phone and a borrowed machine. - -So a **password-protected** server library keeps one copy of them for everyone -who signs in: - -- Sign in on a new device and the assistant is already configured. Nothing to - turn on. -- The first device to save seeds the server with what it already has, so - switching to a server library never loses a key you had entered. -- Everything travels except the **LM Studio URL** — that usually points at - `localhost`, which is a different machine's localhost on every device. -- Signing out of the server library forgets the settings it lent that browser. - -They are stored in `DECKLE_STATE_DIR` (by default `.deckle-state` inside the -library directory, mode `0600`), which is *not* part of the library: the file -API refuses to read it, it never appears in a tree, a listing, or an export, and -the assistant's own tools cannot reach it. Only a request holding a valid -session cookie can read it back. - -**Without `DECKLE_PASSWORD` the server declines to hold a key at all**, and says -so in the settings panel. An open server would hand it to anyone who could reach -the port, and that is the user's decision to make deliberately, not a default to -inherit. +Everything else — theme, palette — is set in the app itself. + +### Upgrading from 2.x + +Deckle 3 removes the AI assistant. Nothing in your library changes: the notes +the assistant wrote are ordinary notes, and its own folders — `.deckle/memory/` +and `.deckle/runs/` — stay where they are until you delete them. + +**If you used a password-protected server library, your AI provider's API key +is still on the volume**, in `.deckle-state/assistant.json`. Deckle goes on +refusing to serve that folder, but no longer reads it. Delete the file, and +revoke the key with the provider if you have no other use for it: + +```bash +docker compose exec deckle rm /data/.deckle-state/assistant.json +``` ### Upgrading from before the rename This app was called Nib, and its configuration was named `NIB_*` — `NIB_PASSWORD`, `NIB_SERVER_VAULT`, and so on. Those names are still read, so pulling a new image over an existing `.env` keeps working, and the server logs which deprecated names -it honoured at startup. Rename them at your convenience — they are read for the -whole of 1.x and removed no earlier than 2.0.0. +it honoured at startup. Rename them at your convenience — they are still read in +3.x, and dropping them will be a major release of its own, announced in the +changelog. Two things deliberately keep their old names, because changing them would move data rather than rename it: @@ -391,11 +413,31 @@ force a major — each says only what it is about. Every release is in the [changelog](CHANGELOG.md), and on [Releases](https://github.com/authorTom/deckle/releases). +## Connecting an agent + +An agent reaches Deckle two ways, with the same tokens: + +- **MCP**, at `/api/v1/mcp` — the way to connect an agent framework such as + Hermes, Claude Code or anything else that speaks the Model Context Protocol. + Streamable HTTP, stateless, bearer-token authenticated. Its tools are listed + under [What it does](#what-it-does). +- **REST**, at `/api/v1` — for scripts, and for agents that generate tools from + an OpenAPI document. See [API](#api). + +[**Connecting Hermes**](integrations/hermes/README.md) walks through the whole +setup: a token, the `mcp_servers` entry in Hermes's `config.yaml`, the Deckle +skill that teaches Hermes how to organise its work, and the push script for +large files and folders. For any other MCP client the entry is the same shape: +the URL `https://your-deckle/api/v1/mcp` and a header +`Authorization: Bearer `. + ## API Deckle can expose the whole knowledge base over HTTP at `/api/v1`, so an agent can -search your notes, answer from them, write new ones, and file tasks and -bookmarks — everything the app can do, without a browser. +search your notes and documents, answer from them, write new notes, store files +of any type, and file tasks and bookmarks — everything the app can do, without +a browser. Every write is recorded in the activity log under the name of the +token that made it. It is off until you configure a token, and it needs the **server library**: a local folder or in-browser library lives on your device, where nothing outside @@ -443,6 +485,10 @@ curl -H "Authorization: Bearer $TOKEN" \ curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -X POST http://localhost:8080/api/v1/notes \ -d '{"title":"Latency review","folder":"Projects","content":"# Latency review\n\n…"}' + +# 4. Store a file, saying what it is +curl -H "Authorization: Bearer $TOKEN" -T report.pdf \ + "http://localhost:8080/api/v1/files/Projects/Acme/report.pdf?message=final%20report" ``` ### Endpoints @@ -457,15 +503,21 @@ curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ | `PUT` | `/notes/{path}` | Create or replace a note | | `PATCH` | `/notes/{path}` | `append` / `prepend` / `content`, or rename and move | | `DELETE` | `/notes/{path}` | To the recycle bin (`?permanent=true` to erase) | -| `GET` | `/search` | BM25-ranked search with snippets (`q`, `limit`, `folder`) | -| `GET` | `/folders` | The folder and note tree | +| `GET` | `/search` | BM25-ranked search over notes and documents, with snippets (`q`, `limit`, `folder`, `kind`) | +| `GET` | `/files` | List notes and files, newest first (`folder`, `kind`, `recursive`, `limit`, `offset`) | +| `GET` | `/files/{path}` | Download any file (`?meta=true` for its details) | +| `PUT` | `/files/{path}` | Store any file from the raw body; a replaced file goes to the bin (`overwrite`, `message`) | +| `DELETE` | `/files/{path}` | To the recycle bin (`?permanent=true` to erase) | +| `GET` | `/activity` | What agents changed, newest first (`since`, `project`, `actor`, `path`, `limit`) | +| `POST` | `/mcp` | The MCP server — see [Connecting an agent](#connecting-an-agent) | +| `GET` | `/folders` | The folder tree, with its notes and files | | `POST` | `/folders` | Create a folder, including missing parents | -| `DELETE` | `/folders/{path}` | Delete a folder; its notes go to the recycle bin | +| `DELETE` | `/folders/{path}` | Delete a folder; everything in it goes to the recycle bin | | `POST` | `/import` | Create up to 1000 notes in one call | | `GET` | `/export` | The whole library as a ZIP (`?include_hidden=true` for a full backup) | | `GET` `POST` | `/tasks` | List (`filter=inbox\|today\|upcoming\|overdue\|completed`) or add | | `GET` `PATCH` `DELETE` | `/tasks/{id}` | Read, edit, complete or bin a task | -| `GET` `POST` | `/projects` | Task projects | +| `GET` `POST` | `/projects` | Task-planner projects (for knowledge-base projects, see MCP or `/folders`) | | `GET` `POST` | `/bookmarks` | List and save bookmarks | | `GET` `PATCH` `DELETE` | `/bookmarks/{id}` | Read, edit or delete a bookmark | | `GET` `POST` | `/collections` | Bookmark collections | @@ -473,22 +525,24 @@ curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ | `GET` | `/history/{snapshot}` | Read a snapshot's content | | `POST` | `/history/{snapshot}/restore` | Restore a snapshot over its note | | `GET` | `/trash` | What is in the recycle bin | -| `POST` | `/trash/{trashName}/restore` | Restore a deleted note | +| `POST` | `/trash/{trashName}/restore` | Restore a deleted note or file | | `DELETE` | `/trash/{trashName}` | Erase one recycle-bin item | -Paths are library-relative with `/` separators — `Projects/idea.md` — and the -`.md` is added if you leave it off. Hidden dot folders are reserved by Deckle and -rejected; `.trash`, `.history` and `.deckle` have their own endpoints instead. +Paths are library-relative with `/` separators — `Projects/idea.md` — and for +`/notes` the `.md` is added if you leave it off. Hidden dot folders are reserved +by Deckle and rejected; `.trash`, `.history` and `.deckle` have their own +endpoints instead. In `/folders` trees, notes are `kind: "file"` and every other +file is `kind: "asset"`. Errors are always `{ "error": { "code": …, "message": … } }` with a matching HTTP status. ## How it's built React · TypeScript · Vite · TipTap + tiptap-markdown (editor) · lucide-react -(icons) · @anthropic-ai/sdk (Claude; OpenAI-compatible providers via `fetch`). +(icons). No AI SDK: Deckle calls no model. The library — a folder on disk, OPFS, or the server — is the source of truth for -notes; IndexedDB only remembers your chosen folder and caches search embeddings. +notes and files; IndexedDB only remembers your chosen folder. All three backends sit behind the browser's `FileSystemDirectoryHandle` interface, so the rest of the app doesn't know or care which one is in use: the server library is an adapter ([`src/fs/remote.ts`](src/fs/remote.ts)) that @@ -502,15 +556,22 @@ beyond Node itself. server/ # Container runtime (Node built-ins only, no deps) index.mjs # Entry point: boot, listen, graceful shutdown app.mjs # The HTTP app: configuration, routing, error handling - library-api.mjs # Server library file API (tree/read/write/mkdir/delete) + library-api.mjs # Server library file API (tree/read/write/move/delete) library-store.mjs # Library semantics server-side: trash, history, tasks… auth.mjs # Optional password gate + signed session cookies api.mjs # /api/v1 REST API for agents and scripts + mcp.mjs # /api/v1/mcp — the MCP server, over the same library + activity.mjs # The activity log (.deckle/activity.jsonl) + projects.mjs # Projects: Projects//Overview.md and Log.md + extract.mjs # Text out of Word, Excel, PowerPoint, CSV… for search + unzip.mjs # Just enough ZIP reading for office documents + frontmatter.mjs # YAML front matter, read and edited line by line + mime.mjs # Content types for file downloads api-auth.mjs # Bearer tokens for the API, with read-only scopes throttle.mjs # Failed-attempt throttling, and trusted-proxy client addresses dates.mjs # Recurring-task dates, mirroring src/tasks/dates.ts openapi.mjs # The API's self-served OpenAPI 3.1 description - search.mjs # BM25 ranking behind GET /api/v1/search + search.mjs # BM25 ranking over notes and documents zip.mjs # Streaming ZIP writer behind GET /api/v1/export paths.mjs # Library path validation (traversal + symlink escapes) legacy-env.mjs # Honours the pre-rename NIB_* configuration names @@ -521,20 +582,22 @@ src/ fs/ # Library backends: disk, OPFS, remote; version history fs/appData.ts # The hidden .deckle folder, and reading its old name db/notes.ts # IndexedDB store for the chosen folder handle - ai/ # Chat loop, providers, library tools, retrieval, settings + activity/ # Reading and polling the activity log + projects/ # Projects, from the app's side tasks/ # Task state and persistence (.deckle/tasks.json) bookmarks/ # Bookmark state and persistence (.deckle/bookmarks.json) hooks/ # Theme, notes tree, autosave, move, search, history - components/ # Sidebar, editor, palette, assistant, panels, modals - memory/ # Assistant memory (.deckle/memory/*.md) - queue/ # Background run queue (.deckle/runs/), executed in the browser - lib/ # Markdown, PDF and ZIP export; Markdown import; BM25 + components/ # Sidebar, editor, file viewer, palette, panels, modals + editor/ # TipTap extensions: wikilinks, tables, library images + lib/ # Export and import; ZIP; office previews; front matter styles/ # theme / global / editor / print CSS +integrations/hermes/ # Connecting Hermes: setup, a skill, and deckle_push.py + test/ # npm test (Vitest) - server/ # The server through its real HTTP handler: auth, API, storage - client/ # Library, storage, queue, assistant tools, hooks and autosave - helpers/ # An in-memory File System Access API, and a test server + server/ # The server through its real HTTP handler: auth, API, MCP, storage + client/ # Library, storage, previews, hooks and autosave + helpers/ # An in-memory File System Access API, a test server, office files ``` ## Security @@ -561,39 +624,41 @@ Relevant when you enable the server library. otherwise dodge the throttle by naming a new address on every guess. With it, only the entries your proxy appended are believed. - **The page runs only its own scripts.** Every response carries a - Content-Security-Policy with `script-src 'self'`, so script that reaches the - page by any route — a note, a bookmark, a dependency bug — cannot run. It - still allows the browser to call any AI provider, including an LM Studio on - your network. -- **AI keys.** The server never proxies AI requests — your browser always calls - the provider itself. Where the key is *kept* depends on the deployment: with a - password set, the server holds one copy for every device that signs in, stored - outside the library (`DECKLE_STATE_DIR`, mode `0600`) and served only to a - request carrying a valid session. The library API refuses to read it, so it - cannot leave in an export, through `/api/v1`, or via the assistant's own - file-reading tools. With no password set the server declines to hold a key at - all, and each browser keeps its own. + Content-Security-Policy with `script-src 'self'` and `connect-src 'self'`, so + script that reaches the page by any route — a note, a file an agent saved, a + bookmark, a dependency bug — cannot run, and the page talks to nothing but + this server. +- **Stored files are shown, never run.** The app previews images, PDFs, audio + and video from blob URLs it creates itself, typed by extension and never + sniffed; everything else — an HTML page an agent saved included — is shown as + text or offered as a download. `/api/v1/files` serves every file as an + attachment with `nosniff` and a `sandbox` policy. - **API tokens are passwords.** A read-write token can read, rewrite and delete - every note. Give each consumer its own so one can be revoked alone, and start + every note and file, over REST or MCP. Give each consumer its own so one can + be revoked alone — its name is how the Activity feed identifies it — and start anything new on a read-only (`r`) token until its behaviour looks sane. Rotating means editing `DECKLE_API_TOKENS` and restarting; there is no token store to clean up. Failed attempts are throttled (20 per 15 minutes). +- **MCP refuses browser pages it wasn't told about.** A request to + `/api/v1/mcp` carrying an `Origin` header is refused unless + `DECKLE_API_CORS_ORIGINS` names it, as the MCP specification asks. +- **Keep secrets out of the library.** Anything in it can be read by whoever + holds a token, and travels in exports and backups. The Hermes skill tells the + agent never to store credentials, and the push script refuses `.env` files, + private keys and credential files, but nothing inspects what an agent writes + through `save_file`. - **The API and the app share a library, not a login.** `/api/v1` ignores the session cookie and accepts only `Authorization: Bearer`. A browser never attaches that header on its own, so there is no CSRF surface and a stolen session cookie cannot reach the API. -- **Agent edits are as recoverable as yours.** Deleting through the API moves - the note to the same recycle bin, and overwriting snapshots the replaced - version into the same history — so a bad agent run is undone from the app's - own dialogs rather than from a backup. -- **Queued runs are fenced into one folder.** A background run may write freely - only inside the Assistant inbox; anything outside it stops and waits for you - to approve the exact change, and deletions stop wherever they are. Everything - it does is still snapshotted to `.history` and recoverable from `.trash`. -- **The assistant writes without asking in exactly one place.** Its own memory, - under `.deckle/memory/`, which touches no note. Everything that changes a note - still goes through the diff. The memory is plain Markdown you can read, edit - and delete — from *Assistant memory* in the command palette, or in any editor. +- **Agent edits are as recoverable as yours.** Deleting through the API or MCP + moves the note or file to the same recycle bin; overwriting a note snapshots + the replaced version into the same history; replacing any other file moves the + old copy to the bin — so a bad agent run is undone from the app's own dialogs + rather than from a backup. Every such change is in the Activity feed, named. +- **There are no per-folder permissions.** A read-write token reaches the whole + library, including notes you wrote yourself. If an agent should only ever + write in one place, give it a library of its own. > **HTTPS matters in production.** The File System Access API and OPFS require a > secure context — `http://localhost` is fine for local use, but anything served @@ -623,7 +688,7 @@ docker run --rm -v nib-vault:/data -v "$PWD:/out" \ alpine tar czf /out/deckle-backup.tar.gz -C /data . ``` -That includes the hidden `.deckle` (tasks, bookmarks, assistant memory), +That includes the hidden `.deckle` (tasks, bookmarks, the activity log), `.history` and `.trash` folders, so it is a complete library. Or mount a host directory instead of the named volume (`./notes:/data`, which must be writable by uid 1000) and point diff --git a/compose.yaml b/compose.yaml index 1a46ce4..211836f 100644 --- a/compose.yaml +++ b/compose.yaml @@ -52,12 +52,19 @@ services: # number of proxies in a chain — so sign-in throttling sees each visitor # rather than the proxy. Unset, X-Forwarded-For is ignored. DECKLE_TRUST_PROXY: "${DECKLE_TRUST_PROXY:-}" - # Bearer tokens for the /api/v1 REST API, so an agent or script can read - # and write the library. Blank (the default) keeps the API switched off. + # Bearer tokens for the /api/v1 REST API and the MCP server at + # /api/v1/mcp, so an agent such as Hermes can read and write the library. + # Blank (the default) keeps both switched off. # Format: "name:scope:secret" (scope "r" or "rw"), comma-separated. DECKLE_API_TOKENS: "${DECKLE_API_TOKENS:-}" # Origins allowed to call /api/v1 from a browser. Rarely needed. DECKLE_API_CORS_ORIGINS: "${DECKLE_API_CORS_ORIGINS:-}" + # The largest single file the library accepts, in MB (default 100). + DECKLE_MAX_FILE_MB: "${DECKLE_MAX_FILE_MB:-}" + # The folder projects live in (default "Projects"). + DECKLE_PROJECTS_DIR: "${DECKLE_PROJECTS_DIR:-}" + # Activity and project-log times use this zone; unset, they are UTC. + TZ: "${TZ:-}" volumes: # Your notes. A named volume survives `docker compose down` and image diff --git a/docs/screenshots/activity.png b/docs/screenshots/activity.png new file mode 100644 index 0000000..4439d80 Binary files /dev/null and b/docs/screenshots/activity.png differ diff --git a/docs/screenshots/assistant.png b/docs/screenshots/assistant.png deleted file mode 100644 index a4ca47d..0000000 Binary files a/docs/screenshots/assistant.png and /dev/null differ diff --git a/docs/screenshots/bookmarks.png b/docs/screenshots/bookmarks.png index fafce17..776ef10 100644 Binary files a/docs/screenshots/bookmarks.png and b/docs/screenshots/bookmarks.png differ diff --git a/docs/screenshots/capture.mjs b/docs/screenshots/capture.mjs index c1e9a62..2a0ebe4 100644 --- a/docs/screenshots/capture.mjs +++ b/docs/screenshots/capture.mjs @@ -1,4 +1,7 @@ -// Retakes the five README screenshots by driving a real browser. +// Retakes the editor, tasks, bookmarks and palette README screenshots by +// driving a real browser. The files, projects and activity screenshots show an +// agent's work, so they are taken against a library an agent has filled — +// through /api/v1/mcp — rather than the demo library this script expects. // // Pair it with make-demo-library.mjs, which builds the library it shoots. The // app is served through the *server library* backend on purpose: the disk @@ -102,12 +105,13 @@ await shot('tasks') // The first bookmark is expanded so the comment box is visible — that field is // the feature the caption is about. -await page.locator('.panel-tab', { hasText: 'Bookmarks' }).click() +// Inactive tabs show only their icon, so find this one by its name. +await page.locator('.panel-tab[aria-label="Bookmarks"]').click() await settle(300) await page.locator('.bm-title').first().click() await shot('bookmarks') -// ---- Dark mode, for the last two ------------------------------------------- +// ---- Dark mode, for the palette -------------------------------------------- // Dismiss the side panel with its own close button. Clicking the sidebar's // Tasks icon would only switch the panel back to the tasks tab, leaving it open @@ -116,12 +120,6 @@ await page.locator('[aria-label="Close panel"]').click() await page.locator('[aria-label="Toggle theme"]').click() await settle(600) -// ---- Assistant -------------------------------------------------------------- - -await page.locator('[aria-label="AI assistant"]').click() -await shot('assistant') -await page.locator('[aria-label="AI assistant"]').click() // close again - // ---- Command palette -------------------------------------------------------- await page.keyboard.press('ControlOrMeta+k') diff --git a/docs/screenshots/editor.png b/docs/screenshots/editor.png index b6343f2..2af98f0 100644 Binary files a/docs/screenshots/editor.png and b/docs/screenshots/editor.png differ diff --git a/docs/screenshots/files.png b/docs/screenshots/files.png new file mode 100644 index 0000000..49a8ff1 Binary files /dev/null and b/docs/screenshots/files.png differ diff --git a/docs/screenshots/make-demo-library.mjs b/docs/screenshots/make-demo-library.mjs index e30cbf4..a526e7c 100644 --- a/docs/screenshots/make-demo-library.mjs +++ b/docs/screenshots/make-demo-library.mjs @@ -49,7 +49,7 @@ Your notes, tasks, and bookmarks — **plain files, fully local**. ## Why Deckle? - Live WYSIWYG Markdown editing -- An AI assistant that can *search and edit* your library +- A knowledge base your agent fills — notes, PDFs, spreadsheets, images - A Todoist-style planner with due dates and a calendar - Bookmarks with collections and comments diff --git a/docs/screenshots/palette.png b/docs/screenshots/palette.png index fa14cee..ec0b33a 100644 Binary files a/docs/screenshots/palette.png and b/docs/screenshots/palette.png differ diff --git a/docs/screenshots/projects.png b/docs/screenshots/projects.png new file mode 100644 index 0000000..eb20821 Binary files /dev/null and b/docs/screenshots/projects.png differ diff --git a/docs/screenshots/tasks.png b/docs/screenshots/tasks.png index e923b40..7df37d8 100644 Binary files a/docs/screenshots/tasks.png and b/docs/screenshots/tasks.png differ diff --git a/integrations/hermes/README.md b/integrations/hermes/README.md new file mode 100644 index 0000000..416ce9b --- /dev/null +++ b/integrations/hermes/README.md @@ -0,0 +1,153 @@ +# Connecting Hermes to Deckle + +This makes Deckle the knowledge base for a [Hermes](https://hermes-agent.nousresearch.com/) +agent: Hermes searches it before it starts work, keeps each piece of work in a +Deckle project, saves what it produces there — notes, reports, spreadsheets, +images, code — and logs what it did. You review it all in the Deckle app, where +the Activity feed shows every change under the agent's name. + +Three pieces, all in this folder or already in Deckle: + +| Piece | What it does | +| --- | --- | +| Deckle's MCP server, at `/api/v1/mcp` | The tools Hermes calls: `search`, `read`, `list`, `list_projects`, `recent_activity`, `write_note`, `save_file`, `create_project`, `update_project`, `log_progress`, `move`, `delete` | +| [`skills/deckle/SKILL.md`](skills/deckle/SKILL.md) | Teaches Hermes when to use them, how to organise projects, and what never to store | +| [`skills/deckle/scripts/deckle_push.py`](skills/deckle/scripts/deckle_push.py) | Sends large files and whole folders, which MCP is not built for | + +It needs Deckle's **server library** — Hermes reaches Deckle over the network, +so the notes have to live on the server, not in a browser. + +## 1. Give Hermes a token + +Generate a secret and add a token for Hermes to Deckle's `.env`, next to +`compose.yaml`: + +```bash +openssl rand -base64 32 +``` + +```bash +DECKLE_SERVER_LIBRARY=true +DECKLE_PASSWORD=a-long-passphrase-for-you +DECKLE_API_TOKENS=hermes:rw:PASTE-THE-SECRET-HERE +``` + +The name before the first colon (`hermes`) is what the Activity feed calls the +agent. `rw` lets it write; start with `r` (read-only) if you want to watch it +search and read for a while first — a read-only token is offered only the +five read tools. Then restart Deckle: + +```bash +docker compose up -d +``` + +The startup log should say `MCP server at /api/v1/mcp (same tokens)`. Check +the endpoint answers: + +```bash +curl -s -H "Authorization: Bearer PASTE-THE-SECRET-HERE" \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \ + https://deckle.example.com/api/v1/mcp | head -c 400 +``` + +## 2. Point Hermes at it + +Put the token where Hermes can read it — in the environment it runs in, or in +`~/.hermes/.env`: + +```bash +DECKLE_URL=https://deckle.example.com +DECKLE_TOKEN=PASTE-THE-SECRET-HERE +``` + +Then add Deckle to `mcp_servers` in `~/.hermes/config.yaml`: + +```yaml +mcp_servers: + deckle: + url: "https://deckle.example.com/api/v1/mcp" + headers: + Authorization: "Bearer ${DECKLE_TOKEN}" + timeout: 120 +``` + +Deckle's MCP endpoint is Streamable HTTP and answers in plain JSON, which is +Hermes's default for a `url` server — don't set `transport: sse`. Run +`/reload-mcp` in a Hermes session (or restart it) and its tools appear as +`mcp__deckle__search`, `mcp__deckle__save_file` and so on. + +If Hermes runs on the same machine as Deckle, `http://localhost:8080` works +for both `DECKLE_URL` and the MCP `url`. In the same Docker network, use the +Deckle service's name instead (`http://deckle:8080`). + +## 3. Install the skill + +Copy it into Hermes's skills folder: + +```bash +cp -r integrations/hermes/skills/deckle ~/.hermes/skills/deckle +``` + +— or leave it in this repository and list it in `~/.hermes/config.yaml`, so +pulling a new Deckle updates it too: + +```yaml +skills: + external_dirs: + - /path/to/deckle/integrations/hermes/skills +``` + +The skill declares `DECKLE_URL` and `DECKLE_TOKEN` as required, so Hermes +passes them to the push script's terminal. + +## 4. Try it + +Ask Hermes something like: + +> Research the three most popular open-source ELT tools, compare their pricing +> and connector coverage, and put a report in Deckle. + +Then open Deckle. A project appears under **Projects** (the folder icon in the +sidebar, or `Cmd`/`Ctrl`+`K` → *Projects*), and the Activity button in the top +bar counts what Hermes has done since you last looked. Files it saves open in +place: PDFs, images, spreadsheets, Word and PowerPoint documents, CSV, JSON, +text and code. + +## Sending files and folders + +`save_file` takes files up to 10 MB, base64-encoded, which is fine for a chart +or a spreadsheet and wasteful for anything bigger. For those, and for a whole +folder at once, Hermes runs the push script: + +```bash +python3 scripts/deckle_push.py report.pdf --project "Acme research" --message "final report" +python3 scripts/deckle_push.py ./output --project "Acme research" --dry-run +python3 scripts/deckle_push.py ./site --to "Projects/Blog relaunch/site" +``` + +It never sends hidden files and folders (`.git`, `.env`, `.venv`), dependency +and build caches (`node_modules`, `__pycache__`, `target`…), or files that look +like credentials (`*.pem`, `*.key`, `id_rsa`, `credentials.json`, …), and it +honours a `.deckleignore` file of patterns at the top of a pushed folder. A +file that already exists is replaced — the old copy goes to Deckle's recycle +bin — unless `--no-overwrite` is given. Standard-library Python 3.8+, nothing +to install. You can run it yourself too. + +## What to expect, honestly + +- **Deckle has no AI in it.** Hermes does the thinking; Deckle stores, + shows, searches and records. Nothing you store is sent anywhere. +- **Search is by words, not meaning.** It ranks notes and the text inside + Word, Excel, PowerPoint, OpenDocument, CSV, JSON, text and code files. + **PDFs and images are found by name only**, and by the notes that link to + them — which is why the skill asks Hermes to describe every important file + in a note. +- **Every change is recoverable.** A replaced note keeps its old version in + history; a replaced or deleted file goes to the recycle bin; nothing the + agent does is permanent unless it asks for `permanent=true` over REST. +- **A read-write token can change anything in the library.** Deckle has no + per-folder permissions. If that is too much, give Hermes a library of its + own, or a read-only token plus a separate place to write. +- **Changes appear in an open app within about 15 seconds** — it polls the + activity log while the tab is visible. diff --git a/integrations/hermes/skills/deckle/SKILL.md b/integrations/hermes/skills/deckle/SKILL.md new file mode 100644 index 0000000..7f1f382 --- /dev/null +++ b/integrations/hermes/skills/deckle/SKILL.md @@ -0,0 +1,138 @@ +--- +name: deckle +description: Use Deckle as your long-term knowledge base. Search it before starting work, keep each piece of work in a Deckle project, save every deliverable there (notes, reports, PDFs, spreadsheets, images, code), and log progress so the user can review what you did. +version: 1.0.0 +platforms: [macos, linux] +metadata: + hermes: + tags: [knowledge-base, memory, files, projects, deckle] + category: productivity +required_environment_variables: + - name: DECKLE_URL + prompt: Deckle server address, e.g. https://deckle.example.com + help: The address of the Deckle server, without /api/v1. The same server the deckle MCP entry in config.yaml points at. + required_for: sending large files and whole folders with scripts/deckle_push.py + - name: DECKLE_TOKEN + prompt: Deckle API token + help: The secret of a read-write token in the server's DECKLE_API_TOKENS, e.g. the one in "hermes:rw:". + required_for: sending large files and whole folders with scripts/deckle_push.py +--- + +# Deckle — the user's knowledge base + +Deckle is where the user keeps and reviews everything you work on: plain +Markdown notes and ordinary files in folders, shown in the Deckle app with an +activity feed of what you changed. Treat it as your long-term memory. What you +learn in a conversation is lost when the conversation ends; what you put in +Deckle is not. + +You reach it through the `deckle` MCP server (tools named `mcp__deckle__…`), +and, for large files and whole folders, through `scripts/deckle_push.py`. + +## When to use it + +- **Before starting any substantial task** — `search` for what is already + known, and `list_projects` to see whether the work belongs to an existing + project. Read the project's `Overview.md` and `Log.md` before continuing + someone's (or your own earlier) work. `recent_activity` shows what changed + lately. +- **Whenever you produce something worth keeping** — a report, a dataset, a + chart, a script, research notes, a decision and its reasons. +- **After each meaningful step** — `log_progress` with what you did, what you + decided and why, and what is next. +- **When the user asks what you know** about something, or to find a file. + +## How work is organised + +- Every piece of work lives in a **project**: a folder `Projects//`. + Start one with `create_project` (check `list_projects` first so you don't + make a duplicate). Its `Overview.md` holds the status, summary and tags the + user sees in Deckle's Projects view. +- Save everything for the project inside its folder. Suggested layout — use + what fits: + - `Projects//Overview.md` — goal, scope, links to the key outputs + - `Projects//Log.md` — written by `log_progress`, never by hand + - `Projects//.md` — research notes, findings, decisions + - `Projects//Deliverables/` — the finished outputs + - `Projects//Sources/` — downloaded papers, data, reference files +- Work that belongs to no project goes in `Inbox/`. +- When a project is finished, `update_project` its status to `done`. + +## Writing notes + +- `write_note` creates Markdown notes. The default mode never overwrites (a + clash gets a numbered name, which the tool tells you); use `replace` to + rewrite a note deliberately — its old version is kept in history — and + `append` to add to one. +- **Link generously** with `[[wikilinks]]`: `[[Findings]]`, `[[Q3 report.pdf]]`, + `[[Projects/Acme/Overview]]`. Links become backlinks the user can follow. +- Embed images you saved with Markdown, relative to the note: + `![Revenue by quarter](charts/revenue.png)`. +- Front matter is welcome and shown as properties: + ``` + --- + source: https://example.com/report + confidence: medium + tags: [pricing, vendors] + --- + ``` +- Write for a person reading later: a clear title, what it is, where it came + from, what it means. + +## Saving files + +- **Up to about 10 MB**: `save_file`. Text formats (CSV, JSON, code, HTML) + as `content`; binary files (PDF, PNG, XLSX, DOCX) as base64 with + `encoding: "base64"`. Always pass a one-line `message` — it is what the user + reads in the activity feed. +- **Bigger files, or a whole folder**: run the push script from a terminal: + ```bash + python3 scripts/deckle_push.py report.pdf --project "Acme research" --message "final report" + python3 scripts/deckle_push.py ./output --project "Acme research" --message "generated charts" + python3 scripts/deckle_push.py ./output --project "Acme research" --dry-run # see first + ``` + Folders keep their structure. The script never sends hidden files, `.env` + files, private keys, `.git`, `node_modules` or other caches, and honours a + `.deckleignore` file. +- **Describe every important file in a note** that links to it. Deckle can + search inside Word, Excel, PowerPoint, CSV, JSON, text and code files, but + PDFs and images are found only by their name and by the notes that mention + them. A one-paragraph summary note makes a PDF findable forever. +- Replacing a file keeps the old copy in Deckle's recycle bin, so correcting a + deliverable is safe. + +## Reading + +- `read` returns a note's Markdown, or the text inside a document — + spreadsheets come back as one table per sheet. Long content arrives in + windows; follow the `offset` it gives you. +- `search` ranks notes and documents together. Narrow it with `folder` (for + one project) or `kind` (`note` or `file`). + +## Never + +- Never store secrets — API keys, passwords, tokens, private keys, `.env` + files, session cookies — in Deckle, in a note or a file. Deckle is shared + with the user and backed up; treat it as readable by anyone who reads their + notes. +- Never delete the user's own notes to tidy up. `delete` moves things to the + recycle bin, but ask before removing anything you did not create. +- Never write `Log.md` or `Overview.md` front matter by hand — use + `log_progress` and `update_project`. + +## Example + +The user asks for a comparison of three ELT vendors. + +1. `search` "ELT vendors" and `list_projects` — nothing yet. +2. `create_project` name "Data pipeline vendors", summary "Choose an ELT + vendor for the marketing pipeline", tags `[research, vendors]`. +3. Research. `save_file` the pricing table as `Sources/pricing.csv` + (message "pricing scraped from vendor sites"). +4. `write_note` `Projects/Data pipeline vendors/Findings` with the analysis, + linking `[[pricing.csv]]` and embedding the chart you saved. +5. `log_progress`: "Scored three vendors — see [[Findings]]. Next: check + HubSpot connector coverage." +6. Generate the report PDF, then + `python3 scripts/deckle_push.py report.pdf --project "Data pipeline vendors" --message "final report"`. +7. `update_project` status `done` once the user is happy. diff --git a/integrations/hermes/skills/deckle/scripts/deckle_push.py b/integrations/hermes/skills/deckle/scripts/deckle_push.py new file mode 100644 index 0000000..b45ff63 --- /dev/null +++ b/integrations/hermes/skills/deckle/scripts/deckle_push.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +"""Copy files or whole folders into a Deckle knowledge base. + +For what MCP's save_file is not built for: big files (it takes 10 MB of +base64 at most) and whole folders — a project's working directory, a set of +generated reports. Uploads over Deckle's REST API, one file per request, with +the same bearer token Hermes uses for MCP. + + deckle_push.py report.pdf --project "Acme research" + deckle_push.py ./out --project "Acme research" --message "final deliverables" + deckle_push.py ./site --to "Projects/Blog relaunch/site" --dry-run + +Folders keep their structure under the destination. Some things are never +sent: hidden files and folders (.git, .env, .venv…), dependency and build +caches (node_modules, __pycache__…), private keys and credential files, and +anything matching a pattern in a .deckleignore file at the top of a pushed +folder or given with --exclude. A file already in Deckle is replaced — the old +copy goes to Deckle's recycle bin — unless --no-overwrite is given. + +Needs DECKLE_URL (e.g. https://deckle.example.com) and DECKLE_TOKEN in the +environment. Python 3.8+, standard library only. Exits non-zero if anything +failed to upload. +""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + +# Folders that are never anyone's knowledge. +SKIP_DIRS = { + "node_modules", "__pycache__", "venv", "env", ".venv", ".tox", ".mypy_cache", + ".pytest_cache", ".ruff_cache", ".next", ".nuxt", ".cache", ".gradle", ".idea", + ".vscode", "target", "bower_components", ".terraform", +} + +# Files that must never leave the machine, whatever the agent was asked. +SECRET_PATTERNS = [ + ".env", ".env.*", "*.pem", "*.key", "*.p12", "*.pfx", "*.jks", "*.keystore", + "id_rsa*", "id_ed25519*", "id_ecdsa*", "id_dsa*", "*.kdbx", "credentials.json", + "credentials", "*.credentials", "secrets.*", "*.secret", ".netrc", ".pgpass", + "*.tfstate", "*.tfstate.*", "service-account*.json", +] + +# Clutter that only operating systems and editors care about. +CLUTTER_PATTERNS = ["*.pyc", "*.pyo", "*.swp", "*~", "Thumbs.db", "desktop.ini"] + + +def is_secret(name: str) -> bool: + lower = name.lower() + return any(fnmatch.fnmatch(lower, p) for p in SECRET_PATTERNS) + + +def read_ignore_file(folder: Path) -> list[str]: + ignore = folder / ".deckleignore" + if not ignore.is_file(): + return [] + patterns = [] + for line in ignore.read_text(encoding="utf-8", errors="replace").splitlines(): + line = line.strip() + if line and not line.startswith("#"): + patterns.append(line.rstrip("/")) + return patterns + + +def matches(rel: str, patterns: list[str]) -> bool: + """A .gitignore-ish match: a pattern with no slash matches any path segment.""" + parts = rel.split("/") + for pattern in patterns: + if "/" in pattern: + if fnmatch.fnmatch(rel, pattern.lstrip("/")) or fnmatch.fnmatch(rel, pattern.lstrip("/") + "/*"): + return True + elif any(fnmatch.fnmatch(part, pattern) for part in parts): + return True + return False + + +def collect(paths: list[str], excludes: list[str]) -> tuple[list[tuple[Path, str]], list[tuple[str, str]]]: + """Every file to send as (local path, path relative to the destination), and what was skipped.""" + files: list[tuple[Path, str]] = [] + skipped: list[tuple[str, str]] = [] + for raw in paths: + root = Path(raw).expanduser() + if not root.exists(): + skipped.append((raw, "does not exist")) + continue + if root.is_file(): + if root.name.startswith("."): + skipped.append((raw, "hidden file")) + elif is_secret(root.name): + skipped.append((raw, "looks like a secret")) + else: + files.append((root, root.name)) + continue + patterns = excludes + read_ignore_file(root) + base = root.resolve().name + for dirpath, dirnames, filenames in os.walk(root): + here = Path(dirpath) + rel_dir = here.relative_to(root).as_posix() + rel_dir = "" if rel_dir == "." else rel_dir + keep = [] + for d in sorted(dirnames): + rel = f"{rel_dir}/{d}" if rel_dir else d + if d.startswith(".") or d in SKIP_DIRS or matches(rel, patterns): + continue + keep.append(d) + dirnames[:] = keep # prune the walk + for name in sorted(filenames): + rel = f"{rel_dir}/{name}" if rel_dir else name + full = here / name + if name.startswith("."): + continue # hidden: never sent, not worth a line each + if any(fnmatch.fnmatch(name, p) for p in CLUTTER_PATTERNS): + continue + if is_secret(name): + skipped.append((f"{base}/{rel}", "looks like a secret")) + continue + if matches(rel, patterns): + continue + if full.is_symlink() or not full.is_file(): + skipped.append((f"{base}/{rel}", "not a regular file")) + continue + files.append((full, f"{base}/{rel}")) + return files, skipped + + +def human(n: int) -> str: + for unit in ("B", "KB", "MB", "GB"): + if n < 1024 or unit == "GB": + return f"{n:.0f} {unit}" if unit == "B" else f"{n:.1f} {unit}" + n /= 1024 + return f"{n} B" + + +def upload(base_url: str, token: str, dest: str, local: Path, overwrite: bool, message: str | None) -> tuple[int, dict]: + quoted = "/".join(urllib.parse.quote(part, safe="") for part in dest.split("/")) + query = {} + if not overwrite: + query["overwrite"] = "false" + if message: + query["message"] = message + url = f"{base_url}/api/v1/files/{quoted}" + if query: + url += "?" + urllib.parse.urlencode(query) + data = local.read_bytes() + request = urllib.request.Request( + url, + data=data, + method="PUT", + headers={ + "Authorization": f"Bearer {token}", + "Content-Type": "application/octet-stream", + "Content-Length": str(len(data)), + "User-Agent": "deckle-push/1", + }, + ) + try: + with urllib.request.urlopen(request, timeout=300) as response: + return response.status, json.loads(response.read() or b"{}") + except urllib.error.HTTPError as err: + try: + body = json.loads(err.read() or b"{}") + except ValueError: + body = {} + return err.code, body + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Copy files or folders into a Deckle knowledge base.", + epilog="Needs DECKLE_URL and DECKLE_TOKEN in the environment.", + ) + parser.add_argument("paths", nargs="+", help="files and folders to send") + where = parser.add_mutually_exclusive_group() + where.add_argument("--project", help='shorthand for --to "Projects/"') + where.add_argument("--to", help="destination folder in the library (default: Inbox)") + parser.add_argument("--message", help="one line on what this is, shown in Deckle's activity log") + parser.add_argument("--no-overwrite", action="store_true", help="skip files that already exist in Deckle") + parser.add_argument("--exclude", action="append", default=[], metavar="PATTERN", help="skip matching paths (repeatable)") + parser.add_argument("--max-mb", type=float, default=100, help="skip files larger than this (default 100)") + parser.add_argument("--flatten", action="store_true", help="drop a pushed folder's own name from the destination") + parser.add_argument("--dry-run", action="store_true", help="list what would be sent, and send nothing") + args = parser.parse_args() + + base_url = os.environ.get("DECKLE_URL", "").rstrip("/") + token = os.environ.get("DECKLE_TOKEN", "") + if base_url.endswith("/api/v1/mcp"): + base_url = base_url[: -len("/api/v1/mcp")] + if not args.dry_run and (not base_url or not token): + print("DECKLE_URL and DECKLE_TOKEN must be set.", file=sys.stderr) + return 2 + + dest_root = f"Projects/{args.project}" if args.project else (args.to or "Inbox") + dest_root = dest_root.strip("/") + files, skipped = collect(args.paths, args.exclude) + limit = int(args.max_mb * 1024 * 1024) + + sent = replaced = 0 + failed: list[tuple[str, str]] = [] + total_bytes = 0 + for local, rel in files: + if args.flatten and "/" in rel: + rel = rel.split("/", 1)[1] + dest = f"{dest_root}/{rel}" if dest_root else rel + size = local.stat().st_size + if size > limit: + skipped.append((str(local), f"larger than {human(limit)}")) + continue + if args.dry_run: + print(f"would send {dest} ({human(size)})") + continue + status, body = upload(base_url, token, dest, local, not args.no_overwrite, args.message) + if status in (200, 201): + sent += 1 + total_bytes += size + if status == 200: + replaced += 1 + print(f"{'replaced' if status == 200 else 'saved'} {dest} ({human(size)})") + elif status == 409 and args.no_overwrite: + skipped.append((dest, "already in Deckle")) + else: + reason = (body.get("error") or {}).get("message") if isinstance(body, dict) else None + failed.append((dest, reason or f"HTTP {status}")) + print(f"failed {dest}: {reason or f'HTTP {status}'}", file=sys.stderr) + + if args.dry_run: + print(f"\n{len(files)} file(s) would go to {dest_root}/") + else: + extra = f", {replaced} replaced (old copies are in Deckle's recycle bin)" if replaced else "" + print(f"\nSent {sent} file(s), {human(total_bytes)}, to {dest_root}/{extra}.") + for name, reason in skipped: + print(f"skipped {name}: {reason}") + for name, reason in failed: + print(f"FAILED {name}: {reason}") + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/package-lock.json b/package-lock.json index d6028e9..aeca895 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,15 +1,15 @@ { "name": "deckle", - "version": "2.0.0", + "version": "3.0.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "deckle", - "version": "2.0.0", + "version": "3.0.0", "dependencies": { - "@anthropic-ai/sdk": "^0.32.1", "@tiptap/extension-code-block-lowlight": "^2.27.2", + "@tiptap/extension-image": "^2.27.3", "@tiptap/extension-link": "^2.27.2", "@tiptap/extension-placeholder": "^2.11.5", "@tiptap/extension-table": "^2.27.2", @@ -31,6 +31,7 @@ "devDependencies": { "@testing-library/dom": "^10.4.2", "@testing-library/react": "^16.3.3", + "@types/node": "^22.20.4", "@types/react": "^18.3.18", "@types/react-dom": "^18.3.5", "@vitejs/plugin-react": "^4.3.4", @@ -40,21 +41,6 @@ "vitest": "^3.2.7" } }, - "node_modules/@anthropic-ai/sdk": { - "version": "0.32.1", - "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.32.1.tgz", - "integrity": "sha512-U9JwTrDvdQ9iWuABVsMLj8nJVwAyQz6QXvgLsVhryhCEPkLsbcP/MXxm+jYcAwLoV8ESbaTTjnD4kuAFa+Hyjg==", - "license": "MIT", - "dependencies": { - "@types/node": "^18.11.18", - "@types/node-fetch": "^2.6.4", - "abort-controller": "^3.0.0", - "agentkeepalive": "^4.2.1", - "form-data-encoder": "1.7.2", - "formdata-node": "^4.3.2", - "node-fetch": "^2.6.7" - } - }, "node_modules/@asamuzakjp/css-color": { "version": "6.0.7", "resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-6.0.7.tgz", @@ -1748,6 +1734,19 @@ "@tiptap/pm": "^2.7.0" } }, + "node_modules/@tiptap/extension-image": { + "version": "2.27.3", + "resolved": "https://registry.npmjs.org/@tiptap/extension-image/-/extension-image-2.27.3.tgz", + "integrity": "sha512-YCOxC+UOFisHVpowPc3UmUtJDV9tjKJLeHKef3DZ6h1ixOnp2LStuhy2ohcSjISg1mZlYrx+/GoYottjwj7pww==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/ueberdosis" + }, + "peerDependencies": { + "@tiptap/core": "^2.7.0" + } + }, "node_modules/@tiptap/extension-italic": { "version": "2.27.3", "resolved": "https://registry.npmjs.org/@tiptap/extension-italic/-/extension-italic-2.27.3.tgz", @@ -2159,22 +2158,13 @@ "license": "MIT" }, "node_modules/@types/node": { - "version": "18.19.130", - "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz", - "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==", - "license": "MIT", - "dependencies": { - "undici-types": "~5.26.4" - } - }, - "node_modules/@types/node-fetch": { - "version": "2.6.13", - "resolved": "https://registry.npmjs.org/@types/node-fetch/-/node-fetch-2.6.13.tgz", - "integrity": "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw==", + "version": "22.20.4", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.20.4.tgz", + "integrity": "sha512-zJRE40jpHtKqE/C4fgHrAKQLJuSpzEnP9ff9Y7YtoR3Wd2pwqzlekDeEuUQXjRd+QCYnVnNwuJYmhdk9XV8gvA==", + "dev": true, "license": "MIT", "dependencies": { - "@types/node": "*", - "form-data": "^4.0.4" + "undici-types": "~6.21.0" } }, "node_modules/@types/prop-types": { @@ -2326,30 +2316,6 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/abort-controller": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz", - "integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==", - "license": "MIT", - "dependencies": { - "event-target-shim": "^5.0.0" - }, - "engines": { - "node": ">=6.5" - } - }, - "node_modules/agentkeepalive": { - "version": "4.6.0", - "resolved": "https://registry.npmjs.org/agentkeepalive/-/agentkeepalive-4.6.0.tgz", - "integrity": "sha512-kja8j7PjmncONqaTsB8fQ+wE2mSU2DJ9D4XKoJ5PFWIdRMa6SLSN1ff4mOr4jCbfRSsxR4keIiySJU0N9T5hIQ==", - "license": "MIT", - "dependencies": { - "humanize-ms": "^1.2.1" - }, - "engines": { - "node": ">= 8.0.0" - } - }, "node_modules/ansi-regex": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", @@ -2399,12 +2365,6 @@ "node": ">=12" } }, - "node_modules/asynckit": { - "version": "0.4.0", - "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", - "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", - "license": "MIT" - }, "node_modules/baseline-browser-mapping": { "version": "2.11.23", "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.23.tgz", @@ -2472,19 +2432,6 @@ "node": ">=8" } }, - "node_modules/call-bind-apply-helpers": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", - "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/caniuse-lite": { "version": "1.0.30001810", "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001810.tgz", @@ -2533,18 +2480,6 @@ "node": ">= 16" } }, - "node_modules/combined-stream": { - "version": "1.0.8", - "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", - "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==", - "license": "MIT", - "dependencies": { - "delayed-stream": "~1.0.0" - }, - "engines": { - "node": ">= 0.8" - } - }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -2666,15 +2601,6 @@ "node": ">=6" } }, - "node_modules/delayed-stream": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", - "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==", - "license": "MIT", - "engines": { - "node": ">=0.4.0" - } - }, "node_modules/dequal": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", @@ -2704,20 +2630,6 @@ "dev": true, "license": "MIT" }, - "node_modules/dunder-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", - "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.1", - "es-errors": "^1.3.0", - "gopd": "^1.2.0" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/electron-to-chromium": { "version": "1.5.427", "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.427.tgz", @@ -2737,24 +2649,6 @@ "url": "https://github.com/fb55/entities?sponsor=1" } }, - "node_modules/es-define-property": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", - "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-errors": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", - "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, "node_modules/es-module-lexer": { "version": "1.7.0", "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz", @@ -2762,33 +2656,6 @@ "dev": true, "license": "MIT" }, - "node_modules/es-object-atoms": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz", - "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-set-tostringtag": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz", - "integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "get-intrinsic": "^1.2.6", - "has-tostringtag": "^1.0.2", - "hasown": "^2.0.2" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/esbuild": { "version": "0.25.12", "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.12.tgz", @@ -2863,15 +2730,6 @@ "@types/estree": "^1.0.0" } }, - "node_modules/event-target-shim": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz", - "integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/expect-type": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", @@ -2906,41 +2764,6 @@ } } }, - "node_modules/form-data": { - "version": "4.0.6", - "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz", - "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==", - "license": "MIT", - "dependencies": { - "asynckit": "^0.4.0", - "combined-stream": "^1.0.8", - "es-set-tostringtag": "^2.1.0", - "hasown": "^2.0.4", - "mime-types": "^2.1.35" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/form-data-encoder": { - "version": "1.7.2", - "resolved": "https://registry.npmjs.org/form-data-encoder/-/form-data-encoder-1.7.2.tgz", - "integrity": "sha512-qfqtYan3rxrnCk1VYaA4H+Ms9xdpPqvLZa6xmMgFvhO32x7/3J/ExcTd6qpxM0vH2GdMI+poehyBZvqfMTto8A==", - "license": "MIT" - }, - "node_modules/formdata-node": { - "version": "4.4.1", - "resolved": "https://registry.npmjs.org/formdata-node/-/formdata-node-4.4.1.tgz", - "integrity": "sha512-0iirZp3uVDjVGt9p49aTaqjk84TrglENEDuqfdlZQ1roC9CWlPk6Avf8EEnZNcAqPonwkG35x4n3ww/1THYAeQ==", - "license": "MIT", - "dependencies": { - "node-domexception": "1.0.0", - "web-streams-polyfill": "4.0.0-beta.3" - }, - "engines": { - "node": ">= 12.20" - } - }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -2956,15 +2779,6 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, - "node_modules/function-bind": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", - "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/gensync": { "version": "1.0.0-beta.2", "resolved": "https://registry.npmjs.org/gensync/-/gensync-1.0.0-beta.2.tgz", @@ -2975,94 +2789,6 @@ "node": ">=6.9.0" } }, - "node_modules/get-intrinsic": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", - "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.2", - "es-define-property": "^1.0.1", - "es-errors": "^1.3.0", - "es-object-atoms": "^1.1.1", - "function-bind": "^1.1.2", - "get-proto": "^1.0.1", - "gopd": "^1.2.0", - "has-symbols": "^1.1.0", - "hasown": "^2.0.2", - "math-intrinsics": "^1.1.0" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/get-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", - "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", - "license": "MIT", - "dependencies": { - "dunder-proto": "^1.0.1", - "es-object-atoms": "^1.0.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/gopd": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", - "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/has-symbols": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", - "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/has-tostringtag": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz", - "integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==", - "license": "MIT", - "dependencies": { - "has-symbols": "^1.0.3" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/hasown": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", - "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", - "license": "MIT", - "dependencies": { - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/highlight.js": { "version": "11.11.1", "resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-11.11.1.tgz", @@ -3085,15 +2811,6 @@ "node": "^20.19.0 || ^22.12.0 || >=24.0.0" } }, - "node_modules/humanize-ms": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/humanize-ms/-/humanize-ms-1.2.1.tgz", - "integrity": "sha512-Fl70vYtsAFb/C06PTS9dZBo7ihau+Tu/DNCk/OyHhea07S+aeMWpFFkUaXRa8fI+ScZbEI8dfSxwY7gxZ9SAVQ==", - "license": "MIT", - "dependencies": { - "ms": "^2.0.0" - } - }, "node_modules/is-potential-custom-element-name": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/is-potential-custom-element-name/-/is-potential-custom-element-name-1.0.1.tgz", @@ -3353,15 +3070,6 @@ "integrity": "sha512-TxFAc76Jnhb2OUu+n3yz9RMu4CwGfaT788br6HhEDlvWfdeJcLUsxk1Hgw2yJio0OXsxv7pyIPmvECY7bMbluA==", "license": "ISC" }, - "node_modules/math-intrinsics": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", - "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, "node_modules/mdn-data": { "version": "2.27.1", "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz", @@ -3375,31 +3083,11 @@ "integrity": "sha512-Lf+9+2r+Tdp5wXDXC4PcIBjTDtq4UKjCPMQhKIuzpJNW0b96kVqSwW0bT7FhRSfmAiFYgP+SCRvdrDozfh0U5w==", "license": "MIT" }, - "node_modules/mime-db": { - "version": "1.52.0", - "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", - "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/mime-types": { - "version": "2.1.35", - "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", - "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", - "license": "MIT", - "dependencies": { - "mime-db": "1.52.0" - }, - "engines": { - "node": ">= 0.6" - } - }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true, "license": "MIT" }, "node_modules/nanoid": { @@ -3421,46 +3109,6 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, - "node_modules/node-domexception": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", - "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", - "deprecated": "Use your platform's native DOMException instead", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/jimmywarting" - }, - { - "type": "github", - "url": "https://paypal.me/jimmywarting" - } - ], - "license": "MIT", - "engines": { - "node": ">=10.5.0" - } - }, - "node_modules/node-fetch": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz", - "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==", - "license": "MIT", - "dependencies": { - "whatwg-url": "^5.0.0" - }, - "engines": { - "node": "4.x || >=6.0.0" - }, - "peerDependencies": { - "encoding": "^0.1.0" - }, - "peerDependenciesMeta": { - "encoding": { - "optional": true - } - } - }, "node_modules/node-releases": { "version": "2.0.55", "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.55.tgz", @@ -4134,12 +3782,6 @@ "node": ">=16" } }, - "node_modules/tr46": { - "version": "0.0.3", - "resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz", - "integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==", - "license": "MIT" - }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", @@ -4171,9 +3813,10 @@ } }, "node_modules/undici-types": { - "version": "5.26.5", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz", - "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==", + "version": "6.21.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", + "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", + "dev": true, "license": "MIT" }, "node_modules/update-browserslist-db": { @@ -4433,21 +4076,6 @@ "node": ">=18" } }, - "node_modules/web-streams-polyfill": { - "version": "4.0.0-beta.3", - "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-4.0.0-beta.3.tgz", - "integrity": "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug==", - "license": "MIT", - "engines": { - "node": ">= 14" - } - }, - "node_modules/webidl-conversions": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz", - "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==", - "license": "BSD-2-Clause" - }, "node_modules/whatwg-mimetype": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz", @@ -4458,16 +4086,6 @@ "node": ">=20" } }, - "node_modules/whatwg-url": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz", - "integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==", - "license": "MIT", - "dependencies": { - "tr46": "~0.0.3", - "webidl-conversions": "^3.0.0" - } - }, "node_modules/why-is-node-running": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", diff --git a/package.json b/package.json index 2fca04e..6e19fb6 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "deckle", "private": true, - "version": "2.0.0", + "version": "3.0.0", "type": "module", "scripts": { "dev": "vite", @@ -11,8 +11,8 @@ "test:watch": "vitest" }, "dependencies": { - "@anthropic-ai/sdk": "^0.32.1", "@tiptap/extension-code-block-lowlight": "^2.27.2", + "@tiptap/extension-image": "^2.27.3", "@tiptap/extension-link": "^2.27.2", "@tiptap/extension-placeholder": "^2.11.5", "@tiptap/extension-table": "^2.27.2", @@ -34,6 +34,7 @@ "devDependencies": { "@testing-library/dom": "^10.4.2", "@testing-library/react": "^16.3.3", + "@types/node": "^22.20.4", "@types/react": "^18.3.18", "@types/react-dom": "^18.3.5", "@vitejs/plugin-react": "^4.3.4", diff --git a/server/activity.mjs b/server/activity.mjs new file mode 100644 index 0000000..9f41c10 --- /dev/null +++ b/server/activity.mjs @@ -0,0 +1,127 @@ +// The activity log: what agents (and scripts) did to the library, and when. +// +// Every write that arrives through /api/v1 or MCP appends one line here, named +// after the token that made it. It is how a person reviews an agent's work +// after the fact — "what did Hermes change while I was out?" — and it is the +// signal the app polls to know the library moved underneath it. +// +// It lives in the library, as JSON Lines in `.deckle/activity.jsonl`, for the +// same reason tasks do: it is part of the record of the work, so it travels in +// the export and the backup, and any tool that reads a line of JSON can read +// it. Edits made in the app itself are not logged — they are the person's own, +// and version history already keeps them. +// +// Append-only, so a write costs one line rather than rewriting the file. When +// it passes MAX_BYTES it is rolled to `activity.1.jsonl` (replacing any older +// roll), which bounds it at twice that on disk. + +const DATA_DIR = '.deckle' +export const ACTIVITY_FILE = `${DATA_DIR}/activity.jsonl` +const ROLLED_FILE = `${DATA_DIR}/activity.1.jsonl` + +/** Roughly ten thousand events. */ +const MAX_BYTES = 2 * 1024 * 1024 + +/** Anything longer is somebody's essay, not a log line. */ +const MAX_MESSAGE = 500 + +/** + * @param {ReturnType} library + * @param {{ projectsDir?: string, now?: () => number, maxBytes?: number }} [options] + */ +export function createActivityLog( + library, + { projectsDir = 'Projects', now = Date.now, maxBytes = MAX_BYTES } = {}, +) { + // One append at a time, so a roll can't interleave with a write. + let chain = Promise.resolve() + + /** The project a path belongs to, if it sits inside the projects folder. */ + function projectOf(path) { + if (!path || !projectsDir) return undefined + const prefix = `${projectsDir}/` + if (!path.startsWith(prefix)) return undefined + const rest = path.slice(prefix.length) + const slash = rest.indexOf('/') + // "Projects/Acme" is the project folder itself; "Projects/x.md" is not in one. + if (slash === -1) return undefined + return rest.slice(0, slash) || undefined + } + + /** + * Record one event. Never throws: a failed log line must not fail the write + * it describes, which has already happened. + * + * @param {{ actor: string, action: string, path?: string, to?: string, + * kind?: 'note' | 'file' | 'folder' | 'project' | 'task' | 'bookmark', + * size?: number, message?: string, project?: string, count?: number }} event + */ + function record(event) { + const entry = { at: now(), ...event } + const project = event.project ?? projectOf(event.to ?? event.path) + if (project) entry.project = project + if (typeof entry.message === 'string') { + entry.message = entry.message.replace(/\s+/g, ' ').trim().slice(0, MAX_MESSAGE) + if (!entry.message) delete entry.message + } + for (const key of Object.keys(entry)) { + if (entry[key] === undefined) delete entry[key] + } + + const run = chain.then(async () => { + try { + const { size } = await library.appendText(ACTIVITY_FILE, `${JSON.stringify(entry)}\n`) + if (size > maxBytes) { + await library.remove(ROLLED_FILE, false).catch(() => {}) + await library.rename(ACTIVITY_FILE, ROLLED_FILE) + } + } catch (err) { + console.warn('[deckle] could not write the activity log:', err?.message ?? err) + } + }) + chain = run + return run + } + + async function readLines(path) { + let text + try { + text = await library.readText(path) + } catch { + return [] + } + const out = [] + for (const line of text.split('\n')) { + if (!line.trim()) continue + try { + const parsed = JSON.parse(line) + if (parsed && typeof parsed.at === 'number') out.push(parsed) + } catch { + // A torn last line (a crash mid-append) is skipped, not fatal. + } + } + return out + } + + /** + * Events newest first, optionally filtered. `since` is exclusive, in + * milliseconds, so a caller can pass the `at` of the last event it saw. + */ + async function list({ since, limit = 100, project, actor, path } = {}) { + await chain + const events = [...(await readLines(ROLLED_FILE)), ...(await readLines(ACTIVITY_FILE))] + const filtered = events.filter( + (e) => + (since === undefined || e.at > since) && + (!project || e.project === project) && + (!actor || e.actor === actor) && + (!path || e.path === path || e.to === path || e.path?.startsWith(`${path}/`)), + ) + // Reversed before the (stable) sort, so events stamped in the same + // millisecond still come back newest first — in the order they were written. + filtered.reverse().sort((a, b) => b.at - a.at) + return filtered.slice(0, limit) + } + + return { record, list, projectOf } +} diff --git a/server/api.mjs b/server/api.mjs index fded056..9097227 100644 --- a/server/api.mjs +++ b/server/api.mjs @@ -11,19 +11,27 @@ // It requires the server library. A local-folder or in-browser library lives on the // user's device and there is nothing here for an agent to reach. // +// Every write is recorded in the activity log under the name of the token that +// made it (see activity.mjs), which is how a person reviews what an agent did. +// The same library is also offered to agents over MCP, at /api/v1/mcp — see +// mcp.mjs — behind the same tokens. +// // The full surface is documented by the OpenAPI 3.1 document this serves at // GET /api/v1/openapi.json — that spec and this router must be changed together. import { randomUUID } from 'node:crypto' -import { SECURITY_HEADERS } from './static.mjs' +import { SECURITY_HEADERS, pipeFile } from './static.mjs' import { BadPathError } from './paths.mjs' import { TooLargeError } from './library-api.mjs' import { ApiError, isDataDir, + isNotePath, + normalizeFilePath, normalizeFolderPath, normalizeNotePath, } from './library-store.mjs' +import { contentTypeFor } from './mime.mjs' import { buildOpenApi } from './openapi.mjs' import { VERSION } from './version.mjs' import { ClientGoneError, writeZip } from './zip.mjs' @@ -108,20 +116,55 @@ function applyPrefix(nodes, prefix) { ) } +/** The notes in a tree — `kind: 'file'` — leaving other files out. */ function flattenFiles(nodes, out = []) { for (const node of nodes) { if (node.kind === 'folder') flattenFiles(node.children, out) + else if (node.kind === 'file') out.push(node) + } + return out +} + +/** Every entry in a tree that isn't a folder: notes and other files alike. */ +function flattenEntries(nodes, out = []) { + for (const node of nodes) { + if (node.kind === 'folder') flattenEntries(node.children, out) else out.push(node) } return out } +/** A file's public description, from a tree node. */ +function describeEntry(node) { + const folder = node.id.includes('/') ? node.id.slice(0, node.id.lastIndexOf('/')) : '' + if (node.kind === 'file') { + return { path: node.id, name: node.name, folder, kind: 'note', ext: 'md', updatedAt: node.updatedAt } + } + return { + path: node.id, + name: node.name, + folder, + kind: 'file', + ext: node.ext, + size: node.size, + updatedAt: node.updatedAt, + } +} + +/** RFC 5987 filename for Content-Disposition, safe for any Unicode name. */ +function dispositionFor(name) { + const ascii = name.replace(/[^\x20-\x7e]|["\\]/g, '_') + return `attachment; filename="${ascii}"; filename*=UTF-8''${encodeURIComponent(name)}` +} + // ---- Router ---------------------------------------------------------------- export function createApi({ library, store, search, + activity, + mcp, auth, libraryEnabled, libraryName, @@ -162,7 +205,7 @@ export function createApi({ sendJson(req, res, status, { error: { code, message } }, extraHeaders) } - async function readBody(req) { + async function readText(req) { const chunks = [] let size = 0 for await (const chunk of req) { @@ -170,8 +213,12 @@ export function createApi({ if (size > MAX_BODY_BYTES) throw new TooLargeError('request body too large') chunks.push(chunk) } - if (!chunks.length) return {} - const text = Buffer.concat(chunks).toString('utf8') + return Buffer.concat(chunks).toString('utf8') + } + + async function readBody(req) { + const text = await readText(req) + if (!text) return {} let parsed try { parsed = JSON.parse(text) @@ -244,7 +291,7 @@ export function createApi({ return await store.createNote({ path, content }) } - async function patchNote(path, body) { + async function patchNote(path, body, log) { const existing = await store.readNote(path) // Validate the whole request before touching the disk. The move used to @@ -280,6 +327,9 @@ export function createApi({ // at its new home rather than leaving the old one updated. const current = target !== existing.path ? await store.moveNote(existing.path, target) : existing + if (current.path !== existing.path) { + await log({ action: 'moved', kind: 'note', path: existing.path, to: current.path }) + } if (!edit) return current let content @@ -293,7 +343,12 @@ export function createApi({ content = `${edit.text}${separator}${current.content}` } - const { note } = await store.writeNote(current.path, content, 'ai') + const { note } = await store.writeNote(current.path, content, 'agent') + await log({ + action: edit.kind === 'content' ? 'replaced' : edit.kind === 'append' ? 'appended to' : 'prepended to', + kind: 'note', + path: note.path, + }) return note } @@ -318,7 +373,7 @@ export function createApi({ const content = requireString(item, 'content', { max: 4_000_000 }) const path = folder ? `${folder}/${relative}` : relative if (overwrite) { - const { note, created } = await store.writeNote(path, content, 'ai') + const { note, created } = await store.writeNote(path, content, 'agent') imported.push({ path: note.path, created }) } else { const note = await store.createNote({ path, content }) @@ -597,6 +652,82 @@ export function createApi({ /** Colours the app already uses for projects and collections. */ const PALETTE = ['#e5484d', '#f76808', '#ffb224', '#30a46c', '#0091ff', '#8e4ec6'] + // ---- Files --------------------------------------------------------------- + + async function listFiles(url) { + const folder = normalizeFolderPath(url.searchParams.get('folder') ?? '') + const recursive = url.searchParams.get('recursive') !== 'false' + const kind = url.searchParams.get('kind') + const limit = parseIntParam(url.searchParams.get('limit'), 200, { min: 1, max: 2000 }) + const offset = parseIntParam(url.searchParams.get('offset'), 0) + if (folder && (await library.exists(folder)) !== 'directory') { + throw new ApiError(404, 'not_found', `no folder at "${folder}"`) + } + const tree = applyPrefix(await library.tree(folder), folder) + let entries = (recursive ? flattenEntries(tree) : tree.filter((n) => n.kind !== 'folder')).map( + describeEntry, + ) + if (kind === 'note' || kind === 'file') entries = entries.filter((e) => e.kind === kind) + entries.sort((a, b) => b.updatedAt - a.updatedAt) + return { total: entries.length, limit, offset, files: entries.slice(offset, offset + limit) } + } + + async function downloadFile(req, res, path) { + const info = await store.readFileInfo(path) + const { handle, size } = await library.openForRead(path) + res.writeHead(200, { + ...SECURITY_HEADERS, + ...corsHeaders(req), + 'Content-Type': contentTypeFor(info.name), + 'Content-Length': size, + 'Content-Disposition': dispositionFor(info.name), + 'Cache-Control': 'no-store', + 'Last-Modified': new Date(info.updatedAt).toUTCString(), + // Whatever an agent stored, a browser that lands here renders nothing live. + 'Content-Security-Policy': "sandbox; default-src 'none'", + }) + pipeFile(res, handle) + } + + async function uploadFile(req, url, path, log) { + const declared = Number(req.headers['content-length'] || 0) + if (declared > library.maxFileBytes) { + throw new ApiError( + 413, + 'too_large', + `files may be at most ${Math.round(library.maxFileBytes / (1024 * 1024))} MB (DECKLE_MAX_FILE_MB)`, + ) + } + const existing = await library.exists(path) + if (existing === 'directory') throw new ApiError(409, 'conflict', `"${path}" is a folder`) + if (existing === 'file' && url.searchParams.get('overwrite') === 'false') { + throw new ApiError(409, 'conflict', `a file already exists at "${path}"`) + } + const message = url.searchParams.get('message') ?? req.headers['x-deckle-message'] + + // A note keeps its text history; any other file keeps its old copy in the bin. + if (isNotePath(path)) { + const { note, created } = await store.writeNote(path, await readText(req), 'agent') + await log({ action: created ? 'created' : 'replaced', kind: 'note', path: note.path, message }) + return { status: created ? 201 : 200, body: { ...describeNoteFile(note) } } + } + const { file, created } = await store.putFile(path, req, { replacedBy: log.actor }) + await log({ action: created ? 'saved' : 'replaced', kind: 'file', path: file.path, size: file.size, message }) + return { status: created ? 201 : 200, body: file } + } + + function describeNoteFile(note) { + return { + path: note.path, + name: note.path.slice(note.path.lastIndexOf('/') + 1), + folder: note.folder, + kind: 'note', + ext: 'md', + size: note.size, + updatedAt: note.updatedAt, + } + } + // ---- Dispatch ------------------------------------------------------------ async function route(req, res, url, segments, token) { @@ -604,6 +735,35 @@ export function createApi({ const method = req.method const readOnly = token.scope === 'r' + /** Record a write under this token's name. */ + const log = (event) => activity.record({ actor: token.name, via: 'api', ...event }) + log.actor = token.name + + // MCP carries reads and writes alike in POSTs, and decides per tool what a + // read-only token may do — so it is routed before the method check below. + if (head === 'mcp' && !rest.length) { + // The MCP spec asks servers to check Origin, against DNS rebinding. The + // bearer token already stops that; this refuses any browser page that + // DECKLE_API_CORS_ORIGINS hasn't named, whatever it holds. Agents send + // no Origin at all. + const origin = req.headers.origin + if (origin && !allowAllOrigins && !corsOrigins.includes(origin)) { + throw new ApiError(403, 'forbidden_origin', `requests from ${origin} are not allowed`) + } + await mcp.handleHttp(req, token, { + readBody: () => readText(req), + send: (status, body, headers = {}) => { + if (body === undefined) { + res.writeHead(status, { ...SECURITY_HEADERS, ...corsHeaders(req), ...headers }) + res.end() + return + } + sendJson(req, res, status, body, headers) + }, + }) + return { handled: true } + } + if (readOnly && WRITE_METHODS.has(method)) { throw new ApiError( 403, @@ -640,6 +800,7 @@ export function createApi({ if (method === 'GET') return { status: 200, body: await listNotes(url) } if (method === 'POST') { const note = await createNote(await readBody(req)) + await log({ action: 'created', kind: 'note', path: note.path }) return { status: 201, body: note, @@ -663,20 +824,23 @@ export function createApi({ const { note, created } = await store.writeNote( path, requireString(body, 'content', { max: 4_000_000 }), - 'ai', + 'agent', ) + await log({ action: created ? 'created' : 'replaced', kind: 'note', path: note.path }) return { status: created ? 201 : 200, body: note } } if (method === 'PATCH') { - return { status: 200, body: await patchNote(path, await readBody(req)) } + return { status: 200, body: await patchNote(path, await readBody(req), log) } } if (method === 'DELETE') { await store.readNote(path) // 404 before doing anything if (boolParam(url.searchParams.get('permanent'))) { await library.remove(path, false) + await log({ action: 'erased', kind: 'note', path }) return { status: 200, body: { deleted: path, permanent: true } } } - const entry = await store.trashNote(path) + const entry = await store.trashEntry(path, { by: token.name }) + await log({ action: 'deleted', kind: 'note', path }) return { status: 200, body: { deleted: path, permanent: false, trash: entry } } } throw methodNotAllowed(method) @@ -686,9 +850,11 @@ export function createApi({ if (head === 'search' && method === 'GET') { const q = url.searchParams.get('q') ?? url.searchParams.get('query') ?? '' if (!q.trim()) throw new ApiError(400, 'invalid_query', '"q" is required') + const kind = url.searchParams.get('kind') const results = await search.search(q, { limit: parseIntParam(url.searchParams.get('limit'), 10, { min: 1, max: 100 }), folder: normalizeFolderPath(url.searchParams.get('folder') ?? ''), + kind: kind === 'note' || kind === 'file' ? kind : undefined, }) return { status: 200, body: { query: q, count: results.length, results } } } @@ -708,6 +874,7 @@ export function createApi({ const path = normalizeFolderPath(requireString(body, 'path', { max: 1024 })) if (!path) throw new ApiError(400, 'invalid_path', '"path" is required') await library.mkdir(path) + await log({ action: 'created', kind: 'folder', path }) return { status: 201, body: { path } } } throw methodNotAllowed(method) @@ -719,18 +886,85 @@ export function createApi({ if ((await library.exists(path)) !== 'directory') { throw new ApiError(404, 'not_found', `no folder at "${path}"`) } - // Match the app: notes inside go to the recycle bin, not oblivion. - const notes = flattenFiles(applyPrefix(await library.tree(path), path)) - for (const note of notes) await store.trashNote(note.id) - await library.remove(path, true) - return { status: 200, body: { deleted: path, trashed: notes.length } } + // Match the app: everything inside — notes and files — goes to the + // recycle bin, not oblivion. + const trashed = await store.trashFolder(path, { by: token.name }) + await log({ action: 'deleted', kind: 'folder', path, count: trashed }) + return { status: 200, body: { deleted: path, trashed } } } throw methodNotAllowed(method) } // -- import / export -------------------------------------------------- if (head === 'import' && method === 'POST') { - return { status: 200, body: await importNotes(await readBody(req)) } + const body = await readBody(req) + const result = await importNotes(body) + if (result.imported) { + await log({ + action: 'imported', + kind: 'note', + path: normalizeFolderPath(body?.folder ?? '') || undefined, + count: result.imported, + }) + } + return { status: 200, body: result } + } + + // -- files (any type) ------------------------------------------------ + if (head === 'files') { + if (!rest.length) { + if (method === 'GET') return { status: 200, body: await listFiles(url) } + throw methodNotAllowed(method) + } + const path = normalizeFilePath(rest.join('/')) + if (method === 'GET') { + if ((await library.exists(path)) === 'directory') { + throw new ApiError(409, 'conflict', `"${path}" is a folder — list it with GET /files?folder=`) + } + if (url.searchParams.get('meta') === 'true') { + return { status: 200, body: await store.readFileInfo(path) } + } + await downloadFile(req, res, path) + return { handled: true } + } + if (method === 'PUT') return await uploadFile(req, url, path, log) + if (method === 'DELETE') { + const kind = await library.exists(path) + if (kind === 'directory') { + throw new ApiError(409, 'conflict', `"${path}" is a folder — use DELETE /folders/{path}`) + } + if (!kind) throw new ApiError(404, 'not_found', `no file at "${path}"`) + const fileKind = isNotePath(path) ? 'note' : 'file' + if (boolParam(url.searchParams.get('permanent'))) { + await library.remove(path, false) + await log({ action: 'erased', kind: fileKind, path }) + return { status: 200, body: { deleted: path, permanent: true } } + } + const entry = await store.trashEntry(path, { by: token.name }) + await log({ action: 'deleted', kind: fileKind, path }) + return { status: 200, body: { deleted: path, permanent: false, trash: entry } } + } + throw methodNotAllowed(method) + } + + // -- activity -------------------------------------------------------- + if (head === 'activity' && !rest.length && method === 'GET') { + const sinceRaw = url.searchParams.get('since') + let since + if (sinceRaw) { + since = /^\d+$/.test(sinceRaw) ? Number(sinceRaw) : Date.parse(sinceRaw) + if (!Number.isFinite(since)) { + throw new ApiError(400, 'invalid_query', '"since" must be a date or a time in milliseconds') + } + } + const events = await activity.list({ + since, + limit: parseIntParam(url.searchParams.get('limit'), 100, { min: 1, max: 1000 }), + project: url.searchParams.get('project') || undefined, + actor: url.searchParams.get('actor') || undefined, + path: url.searchParams.get('path') || undefined, + }) + return { status: 200, body: { count: events.length, events } } } if (head === 'export' && method === 'GET') { await exportZip(req, res, url) @@ -746,7 +980,9 @@ export function createApi({ return { status: 200, body: { count: tasks.length, tasks } } } if (method === 'POST') { - return { status: 201, body: await createTask(await readBody(req)) } + const task = await createTask(await readBody(req)) + await log({ action: 'added', kind: 'task', message: task.title }) + return { status: 201, body: task } } throw methodNotAllowed(method) } @@ -759,7 +995,14 @@ export function createApi({ return { status: 200, body: task } } if (method === 'PATCH') { - return { status: 200, body: await patchTask(id, await readBody(req)) } + const body = await readBody(req) + const task = await patchTask(id, body) + await log({ + action: body.completed === true ? 'completed' : 'updated', + kind: 'task', + message: task.title, + }) + return { status: 200, body: task } } if (method === 'DELETE') { const permanent = boolParam(url.searchParams.get('permanent')) @@ -774,6 +1017,7 @@ export function createApi({ data.tasks[index].deletedAt = Date.now() return { id, permanent: false } }) + await log({ action: 'deleted', kind: 'task', message: id }) return { status: 200, body: { deleted: result.id, permanent: result.permanent } } } throw methodNotAllowed(method) @@ -814,7 +1058,9 @@ export function createApi({ return { status: 200, body: { count: bookmarks.length, bookmarks } } } if (method === 'POST') { - return { status: 201, body: await createBookmark(await readBody(req)) } + const bookmark = await createBookmark(await readBody(req)) + await log({ action: 'saved', kind: 'bookmark', message: `${bookmark.title} — ${bookmark.url}` }) + return { status: 201, body: bookmark } } throw methodNotAllowed(method) } @@ -883,6 +1129,7 @@ export function createApi({ if (!entry) throw new ApiError(404, 'not_found', 'no such snapshot') const content = await store.readSnapshot(rest[0]) const { note } = await store.writeNote(entry.noteId, content, 'restore') + await log({ action: 'restored a version of', kind: 'note', path: note.path }) return { status: 200, body: note } } throw methodNotAllowed(method) @@ -895,10 +1142,17 @@ export function createApi({ return { status: 200, body: { count: items.length, items } } } if (rest.length === 2 && rest[1] === 'restore' && method === 'POST') { - return { status: 200, body: await store.restoreTrash(rest[0]) } + const restored = await store.restoreTrash(rest[0]) + await log({ + action: 'restored', + kind: isNotePath(restored.path) ? 'note' : 'file', + path: restored.path, + }) + return { status: 200, body: restored } } if (rest.length === 1 && method === 'DELETE') { await store.deleteTrashItem(rest[0]) + await log({ action: 'erased from the bin', message: rest[0] }) return { status: 200, body: { deleted: rest[0] } } } throw methodNotAllowed(method) diff --git a/server/app.mjs b/server/app.mjs index 84a274d..6ef032f 100644 --- a/server/app.mjs +++ b/server/app.mjs @@ -19,9 +19,11 @@ import { createApiAuth } from './api-auth.mjs' import { createApi } from './api.mjs' import { createSearch } from './search.mjs' import { createStaticHandler, pipeFile, SECURITY_HEADERS } from './static.mjs' -import { createLibraryApi, TooLargeError } from './library-api.mjs' +import { createLibraryApi, DEFAULT_MAX_FILE_BYTES, TooLargeError } from './library-api.mjs' import { createLibraryStore } from './library-store.mjs' -import { createSettingsStore, InvalidSettingsError, MAX_SETTINGS_BYTES } from './settings-store.mjs' +import { createActivityLog } from './activity.mjs' +import { createProjects } from './projects.mjs' +import { createMcp } from './mcp.mjs' import { BadPathError } from './paths.mjs' import { resolveLegacyEnv } from './legacy-env.mjs' import { parseTrustProxy } from './throttle.mjs' @@ -32,6 +34,25 @@ const here = path.dirname(fileURLToPath(import.meta.url)) /** A body that parses as JSON but isn't the object every endpoint expects. */ class InvalidBodyError extends Error {} +/** DECKLE_MAX_FILE_MB, as bytes. Blank means the default. */ +function readMaxFileBytes(raw) { + if (raw === undefined || raw === '') return DEFAULT_MAX_FILE_BYTES + const mb = Number(raw) + if (!Number.isFinite(mb) || mb <= 0 || mb > 10_240) { + throw new Error(`DECKLE_MAX_FILE_MB must be a number of megabytes between 1 and 10240, got "${raw}"`) + } + return Math.round(mb * 1024 * 1024) +} + +/** DECKLE_PROJECTS_DIR: a plain library-relative folder, no dot segments. */ +function readProjectsDir(raw) { + const value = String(raw ?? '').trim().replace(/^\/+|\/+$/g, '') || 'Projects' + if (value.split('/').some((s) => !s || s === '..' || s.startsWith('.'))) { + throw new Error(`DECKLE_PROJECTS_DIR must be a folder inside the library, such as "Projects", got "${raw}"`) + } + return value +} + function readPort(raw) { const port = Number(raw || 8080) if (!Number.isInteger(port) || port < 0 || port > 65535) { @@ -56,33 +77,48 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons libraryDir, libraryName: env.DECKLE_LIBRARY_NAME || 'My Notes', libraryEnabled: env.DECKLE_SERVER_LIBRARY === 'true', - // Deckle's own state — not notes, and not part of the library, but kept - // under the library directory by default because that is the volume - // people mount. The library API refuses to serve it wherever it lives. + // Deckle's own state — not notes, and not part of the library. Nothing is + // written there any more, but Deckle 2.x kept the AI assistant's settings + // (provider API key included) in it, so a volume that has been upgraded + // may still hold one. The library API refuses to serve it wherever it + // lives, and must go on refusing: an old key must never leave in an + // export, through /api/v1, or over MCP. stateDir: path.resolve(env.DECKLE_STATE_DIR || path.join(libraryDir, '.deckle-state')), trustProxy: parseTrustProxy(env.DECKLE_TRUST_PROXY), + maxFileBytes: readMaxFileBytes(env.DECKLE_MAX_FILE_MB), + projectsDir: readProjectsDir(env.DECKLE_PROJECTS_DIR), legacyEnv, } const auth = createAuth(env) - const library = createLibraryApi(config.libraryDir, { reservedPaths: [config.stateDir] }) - const settingsStore = createSettingsStore(config.stateDir) - - // Shared assistant settings are offered only behind a password. Without one - // the API is open by design (the deployment guide assumes a VPN or an - // authenticating proxy in front), and an open endpoint handing out a provider - // key is not a trade this server gets to make on the user's behalf. - const sharedSettings = config.libraryEnabled && auth.required + const library = createLibraryApi(config.libraryDir, { + reservedPaths: [config.stateDir], + maxFileBytes: config.maxFileBytes, + }) const serveStatic = createStaticHandler(config.publicDir) // The machine API (/api/v1): off unless tokens are configured, and useless // without the server library, since a local-folder library never reaches this // process at all. const apiAuth = createApiAuth(env, warn) + const store = createLibraryStore(library) + const search = createSearch(library) + const activity = createActivityLog(library, { projectsDir: config.projectsDir }) + const projects = createProjects({ library, store, projectsDir: config.projectsDir }) const handleApi = createApi({ library, - store: createLibraryStore(library), - search: createSearch(library), + store, + search, + activity, + mcp: createMcp({ + library, + store, + search, + activity, + projects, + libraryName: config.libraryName, + maxFileBytes: config.maxFileBytes, + }), auth: apiAuth, libraryEnabled: config.libraryEnabled, libraryName: config.libraryName, @@ -140,9 +176,9 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons name: config.libraryName, authRequired: auth.required, authenticated: config.libraryEnabled && auth.isAuthenticated(req), - // Whether this server will hold the assistant's settings for every - // device, so the app can say why it won't when it won't. - sharedSettings, + // Where projects live, so the app's Projects view looks in the same + // folder the MCP tools write to. + projectsDir: config.projectsDir, }) return true } @@ -199,71 +235,6 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons return false } - // ---- /api/assistant-settings ----------------------------------------------- - - /** - * The assistant's settings, shared by every device signed in to this server. - * - * Gated exactly like the library routes — same-origin app header, then the - * session cookie — plus the password requirement above. The stored object is - * whatever the app sent; this endpoint is a shelf, not a schema. - */ - async function handleAssistantSettingsRoutes(req, res, url) { - if (url.pathname !== '/api/assistant-settings') return false - - if (!config.libraryEnabled) { - sendJson(res, 404, { error: 'server library disabled' }) - return true - } - if (!hasAppHeader(req)) { - sendJson(res, 403, { error: 'forbidden' }) - return true - } - if (!sharedSettings) { - sendJson(res, 409, { - error: 'password_required', - message: - 'Set DECKLE_PASSWORD to share assistant settings between devices. ' + - 'Without it this server is open, and anyone who can reach it could read the key.', - }) - return true - } - if (!auth.isAuthenticated(req)) { - sendJson(res, 401, { error: 'not authenticated' }) - return true - } - - if (req.method === 'GET') { - sendJson(res, 200, { settings: await settingsStore.read() }) - return true - } - - if (req.method === 'PUT') { - let body - try { - body = await readJsonBody(req, MAX_SETTINGS_BYTES) - } catch (err) { - if (err instanceof TooLargeError) throw err - sendJson(res, 400, { error: 'invalid request' }) - return true - } - try { - await settingsStore.write(body.settings) - } catch (err) { - if (err instanceof InvalidSettingsError) { - sendJson(res, 400, { error: err.message }) - return true - } - throw err - } - sendJson(res, 200, { ok: true }) - return true - } - - sendJson(res, 405, { error: 'method not allowed' }) - return true - } - // ---- /api/library ---------------------------------------------------------- async function handleLibraryRoutes(req, res, url) { @@ -328,6 +299,13 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons sendJson(res, 200, { ok: true }) return true + // A rename, so moving a large file — into a folder, or into the recycle + // bin — costs one request and no transfer. The File System Access API + // has no portable move, which is why the handle shim can't express it. + case 'POST /api/library/move': + sendJson(res, 200, await library.rename(rel, url.searchParams.get('to') ?? '')) + return true + case 'DELETE /api/library/entry': await library.remove(rel, url.searchParams.get('recursive') === '1') sendJson(res, 200, { ok: true }) @@ -356,7 +334,6 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons // never be reachable with the app's session cookie. if (await handleApi(req, res, url)) return if (await handleServerLibraryRoutes(req, res, url)) return - if (await handleAssistantSettingsRoutes(req, res, url)) return if (await handleLibraryRoutes(req, res, url)) return if (url.pathname.startsWith('/api/')) { @@ -443,11 +420,6 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons ? '[deckle] server library is password protected' : '[deckle] WARNING: server library has no password (DECKLE_PASSWORD unset) — anyone who can reach this port can read and write your notes', ) - log( - sharedSettings - ? `[deckle] assistant settings shared across devices, stored in ${config.stateDir}` - : '[deckle] assistant settings stay in each browser (sharing them needs DECKLE_PASSWORD)', - ) } else { log('[deckle] server library disabled (set DECKLE_SERVER_LIBRARY=true to enable)') } @@ -466,6 +438,12 @@ export function createApp(rawEnv = process.env, { log = console.log, warn = cons ) } else { log(`[deckle] API enabled at /api/v1 — tokens: ${apiAuth.describe().join(', ')}`) + log('[deckle] MCP server at /api/v1/mcp (same tokens)') + } + if (config.libraryEnabled) { + log( + `[deckle] projects live in ${config.projectsDir}/; files up to ${Math.round(config.maxFileBytes / (1024 * 1024))} MB`, + ) } } diff --git a/server/extract.mjs b/server/extract.mjs new file mode 100644 index 0000000..59719f4 --- /dev/null +++ b/server/extract.mjs @@ -0,0 +1,269 @@ +// Text out of the files an agent stores, for search and for MCP's `read`. +// +// Notes are text already. Of everything else, three families give up their +// words without a dependency: +// +// * plain-text formats — CSV, JSON, YAML, source code, logs — read as UTF-8; +// * Office Open XML — .docx, .xlsx, .pptx — which are ZIPs of XML; +// * OpenDocument — .odt, .ods, .odp — the same idea, one content.xml. +// +// PDFs are not on the list. Getting reliable text out of a PDF means font +// encodings, CMaps and content-stream parsing — a library's worth of work — +// and half-right text is worse than none in a search index. A PDF is still +// found by its name, and by any note that links to it; the Hermes skill asks +// the agent to write that note. +// +// The output is Markdown-flavoured plain text: headings keep a `#`, sheets and +// slides get their own heading, and table cells are separated by ` | `, so an +// agent reading it gets the structure as well as the words. + +import { readZipEntries } from './unzip.mjs' + +/** Formats that are text on disk, by extension. */ +export const TEXT_EXTENSIONS = new Set([ + 'txt', 'text', 'md', 'markdown', 'csv', 'tsv', 'json', 'jsonl', 'ndjson', 'yaml', 'yml', + 'xml', 'html', 'htm', 'log', 'toml', 'ini', 'cfg', 'conf', 'rst', 'org', 'tex', 'bib', + 'py', 'js', 'mjs', 'cjs', 'ts', 'tsx', 'jsx', 'sh', 'bash', 'zsh', 'fish', 'ps1', 'sql', + 'rb', 'go', 'rs', 'java', 'kt', 'kts', 'swift', 'c', 'h', 'cc', 'cpp', 'hpp', 'cs', 'php', + 'r', 'lua', 'pl', 'scala', 'dart', 'css', 'scss', 'less', 'vue', 'svelte', 'graphql', + 'proto', 'dockerfile', 'makefile', 'gradle', 'properties', 'srt', 'vtt', 'ipynb', +]) + +const OOXML = new Set(['docx', 'xlsx', 'pptx']) +const ODF = new Set(['odt', 'ods', 'odp']) + +/** Anything bigger is read for its name only. */ +export const MAX_EXTRACT_SOURCE_BYTES = 25 * 1024 * 1024 +/** And no file contributes more than this much text. */ +export const MAX_EXTRACT_CHARS = 400_000 + +const MAX_SHEET_ROWS = 2000 + +export function extensionOf(name) { + const dot = name.lastIndexOf('.') + return dot > 0 ? name.slice(dot + 1).toLowerCase() : '' +} + +/** Whether `extractText` has anything to say about a file with this name. */ +export function isExtractable(name) { + const ext = extensionOf(name) + return TEXT_EXTENSIONS.has(ext) || OOXML.has(ext) || ODF.has(ext) +} + +function decodeEntities(text) { + return text.replace(/&(#x[0-9a-f]+|#\d+|amp|lt|gt|quot|apos);/gi, (whole, code) => { + const lower = code.toLowerCase() + if (lower === 'amp') return '&' + if (lower === 'lt') return '<' + if (lower === 'gt') return '>' + if (lower === 'quot') return '"' + if (lower === 'apos') return "'" + const n = lower.startsWith('#x') ? parseInt(lower.slice(2), 16) : parseInt(lower.slice(1), 10) + try { + return String.fromCodePoint(n) + } catch { + return whole + } + }) +} + +function tidy(text) { + return text + .replace(/[ \t]+\n/g, '\n') + .replace(/\n{3,}/g, '\n\n') + .trim() + .slice(0, MAX_EXTRACT_CHARS) +} + +// ---- Word ------------------------------------------------------------------ + +/** One paragraph's visible text: runs, tabs and breaks. */ +function wordRuns(xml) { + let out = '' + const re = /]*)?>([^<]*)<\/w:t>||]*)?\/>/g + let m + while ((m = re.exec(xml))) { + if (m[1] !== undefined) out += decodeEntities(m[1]) + else if (m[0].startsWith('/, '').replace(/<\/w:body>[\s\S]*$/, '') + const lines = [] + // Tables first become rows of cells, so their paragraphs aren't read twice. + const withTables = body.replace(/[\s\S]*?<\/w:tbl>/g, (table) => { + const rows = table.match(/][\s\S]*?<\/w:tr>/g) ?? [] + const text = rows + .map((row) => + (row.match(/][\s\S]*?<\/w:tc>/g) ?? []) + .map((cell) => wordRuns(cell).replace(/\s+/g, ' ').trim()) + .join(' | '), + ) + .join('\n') + return `${text.replace(/[<&]/g, (c) => (c === '<' ? '<' : '&'))}` + }) + for (const paragraph of withTables.split(/<\/w:p>/)) { + const text = wordRuns(paragraph) + if (!text.trim()) { + lines.push('') + continue + } + const heading = //.test(paragraph) + if (heading) lines.push(`${'#'.repeat(Math.min(6, Number(heading[1] || 1)))} ${text.trim()}`) + else if (bullet) lines.push(`- ${text.trim()}`) + else lines.push(text) + } + return tidy(lines.join('\n')) +} + +// ---- Excel ----------------------------------------------------------------- + +function columnIndex(ref) { + const letters = /^[A-Z]+/.exec(ref)?.[0] ?? 'A' + let n = 0 + for (const ch of letters) n = n * 26 + (ch.charCodeAt(0) - 64) + return n - 1 +} + +function sharedStrings(xml) { + if (!xml) return [] + return (xml.match(/[\s\S]*?<\/si>/g) ?? []).map((si) => + decodeEntities((si.match(/]*)?>([^<]*)<\/t>/g) ?? []).map((t) => t.replace(/<[^>]+>/g, '')).join('')), + ) +} + +function sheetRows(xml, strings) { + const rows = [] + for (const row of xml.match(/][\s\S]*?<\/row>/g) ?? []) { + if (rows.length >= MAX_SHEET_ROWS) break + const cells = [] + // Self-closing (empty) cells first, or the lazy match would run on into + // the next cell's content. + for (const cell of row.match(/]*\/>|][\s\S]*?<\/c>/g) ?? []) { + const ref = /\sr="([A-Z]+\d+)"/.exec(cell)?.[1] + const type = /\st="([^"]+)"/.exec(cell)?.[1] + const raw = /([^<]*)<\/v>/.exec(cell)?.[1] + let value = '' + if (type === 's' && raw !== undefined) value = strings[Number(raw)] ?? '' + else if (type === 'inlineStr') value = decodeEntities((/]*)?>([^<]*)<\/t>/.exec(cell) ?? [])[1] ?? '') + else if (type === 'b') value = raw === '1' ? 'TRUE' : 'FALSE' + else if (raw !== undefined) value = decodeEntities(raw) + const index = ref ? columnIndex(ref) : cells.length + while (cells.length < index) cells.push('') + cells[index] = value.replace(/\s+/g, ' ').trim() + } + while (cells.length && cells[cells.length - 1] === '') cells.pop() + if (cells.length) rows.push(cells.join(' | ')) + } + return rows +} + +function xlsxText(entries) { + const text = (name) => entries.get(name)?.toString('utf8') + const strings = sharedStrings(text('xl/sharedStrings.xml')) + const workbook = text('xl/workbook.xml') ?? '' + const rels = text('xl/_rels/workbook.xml.rels') ?? '' + const targets = new Map() + for (const rel of rels.match(/]*>/g) ?? []) { + const id = /\sId="([^"]+)"/.exec(rel)?.[1] + const target = /\sTarget="([^"]+)"/.exec(rel)?.[1] + if (id && target) targets.set(id, target.replace(/^\/?(xl\/)?/, 'xl/')) + } + const parts = [] + for (const sheet of workbook.match(/]*>/g) ?? []) { + const name = decodeEntities(/\sname="([^"]*)"/.exec(sheet)?.[1] ?? 'Sheet') + const rid = /\sr:id="([^"]+)"/.exec(sheet)?.[1] + const xml = rid ? text(targets.get(rid) ?? '') : undefined + if (!xml) continue + const rows = sheetRows(xml, strings) + parts.push(`## ${name}\n\n${rows.join('\n')}`) + } + return tidy(parts.join('\n\n')) +} + +// ---- PowerPoint ------------------------------------------------------------ + +function pptxText(entries) { + const slides = [...entries.keys()] + .map((name) => ({ name, n: Number(/^ppt\/slides\/slide(\d+)\.xml$/.exec(name)?.[1]) })) + .filter((s) => Number.isFinite(s.n)) + .sort((a, b) => a.n - b.n) + const parts = slides.map(({ name, n }) => { + const xml = entries.get(name).toString('utf8') + const paragraphs = xml + .split(/<\/a:p>/) + .map((p) => + decodeEntities( + (p.match(/]*)?>[^<]*<\/a:t>/g) ?? []).map((t) => t.replace(/<[^>]+>/g, '')).join(''), + ), + ) + .filter((p) => p.trim()) + return `## Slide ${n}\n\n${paragraphs.join('\n')}` + }) + return tidy(parts.join('\n\n')) +} + +// ---- OpenDocument ---------------------------------------------------------- + +function odfText(contentXml) { + const body = contentXml.replace(/^[\s\S]*?/, '') + return tidy( + decodeEntities( + body + .replace(//g, '\t') + .replace(//g, '\n') + .replace(//g, (_, n) => ' '.repeat(Number(n ?? 1))) + .replace(/<\/table:table-cell>/g, ' | ') + .replace(/<\/(?:text:p|text:h|table:table-row)>/g, '\n') + .replace(/<[^>]+>/g, ''), + ), + ) +} + +/** + * The text in a file's bytes, or null when it has none Deckle can read. + * + * @param {Buffer} buffer + * @param {string} name the file name, for its extension + */ +export function extractText(buffer, name) { + const ext = extensionOf(name) + try { + if (TEXT_EXTENSIONS.has(ext)) { + // A NUL in the first few KB means this "text" file is really binary. + if (buffer.subarray(0, 8192).includes(0)) return null + return buffer.toString('utf8').replace(/^\uFEFF/, '').slice(0, MAX_EXTRACT_CHARS) + } + if (ext === 'docx') { + const entries = readZipEntries(buffer, { want: (n) => n === 'word/document.xml' }) + const xml = entries.get('word/document.xml') + return xml ? docxText(xml.toString('utf8')) : null + } + if (ext === 'xlsx') { + return xlsxText( + readZipEntries(buffer, { + want: (n) => + n === 'xl/workbook.xml' || + n === 'xl/sharedStrings.xml' || + n === 'xl/_rels/workbook.xml.rels' || + /^xl\/worksheets\/sheet\d+\.xml$/.test(n), + }), + ) + } + if (ext === 'pptx') { + return pptxText(readZipEntries(buffer, { want: (n) => /^ppt\/slides\/slide\d+\.xml$/.test(n) })) + } + if (ODF.has(ext)) { + const xml = readZipEntries(buffer, { want: (n) => n === 'content.xml' }).get('content.xml') + return xml ? odfText(xml.toString('utf8')) : null + } + } catch { + // A damaged or unusual document has no text, rather than failing a search. + return null + } + return null +} diff --git a/server/frontmatter.mjs b/server/frontmatter.mjs new file mode 100644 index 0000000..8bd8e17 --- /dev/null +++ b/server/frontmatter.mjs @@ -0,0 +1,150 @@ +// A deliberately small YAML front-matter reader and editor. +// +// Mirrors src/lib/frontmatter.ts — the server has no dependencies, so it can't +// share the client's module, and the two must agree on what a front-matter +// block is. Change them together. +// +// Agents write front matter freely (Hermes, Obsidian and most static-site +// tools all do), so Deckle has to carry it without damage. What it *reads* is +// a flat block of `key: value` pairs, where a value is a string, a number, a +// boolean, or a list — bracketed (`[a, b]`) or as indented `- item` lines. What +// it *edits* it edits line by line, so any YAML it doesn't understand — nested +// maps, multi-line strings, comments — is left exactly as it was written. + +const FENCE = /^---[ \t]*\r?\n([\s\S]*?)\r?\n---[ \t]*(?:\r?\n|$)/ + +/** + * The front-matter block at the top of `text`, if there is one. A note that + * merely opens with a horizontal rule (`---`, some text, `---`) has none: the + * block must start with a `key:` line, as YAML front matter always does. + */ +function matchFence(text) { + const match = FENCE.exec(text) + if (!match) return null + const first = match[1].split(/\r?\n/).find((line) => line.trim() && !line.trim().startsWith('#')) + return first !== undefined && /^[A-Za-z0-9_][\w .-]*:(\s|$)/.test(first) ? match : null +} + +function unquote(value) { + if (value.startsWith('"')) { + try { + return JSON.parse(value) + } catch { + return value.replace(/^"|"$/g, '') + } + } + if (value.startsWith("'") && value.endsWith("'") && value.length > 1) { + return value.slice(1, -1).replace(/''/g, "'") + } + return value +} + +function parseValue(raw) { + const value = raw.trim() + if (!value) return '' + if (value.startsWith('[')) { + return value + .slice(1, value.endsWith(']') ? -1 : undefined) + .split(',') + .map((s) => unquote(s.trim())) + .filter(Boolean) + } + if (value.startsWith('"') || value.startsWith("'")) return unquote(value) + if (value === 'true') return true + if (value === 'false') return false + if (/^-?\d+(\.\d+)?$/.test(value)) return Number(value) + return value +} + +/** + * Split a document into its front-matter block (fences included, or '' when + * there is none) and the Markdown beneath it, verbatim. + */ +export function splitFrontmatter(text) { + const match = matchFence(text) + if (!match) return { frontmatter: '', body: text } + return { frontmatter: match[0], body: text.slice(match[0].length) } +} + +/** The key/value pairs in a document's front matter ({} when it has none). */ +export function parseFrontmatter(text) { + const match = matchFence(text) + if (!match) return {} + const data = {} + const lines = match[1].split(/\r?\n/) + for (let i = 0; i < lines.length; i++) { + const line = lines[i] + // Only top-level keys; indented lines belong to the key above them. + if (!line || /^\s/.test(line) || line.trimStart().startsWith('#')) continue + const colon = line.indexOf(':') + if (colon <= 0) continue + const key = line.slice(0, colon).trim() + const rest = line.slice(colon + 1) + if (!rest.trim()) { + // A block list: `key:` followed by indented `- item` lines. + const items = [] + while (i + 1 < lines.length && /^\s+-\s/.test(lines[i + 1])) { + items.push(unquote(lines[++i].replace(/^\s+-\s+/, '').trim())) + } + data[key] = items.length ? items.filter(Boolean) : '' + continue + } + data[key] = parseValue(rest) + } + return data +} + +/** Quote only where a bare value would parse back as something else. */ +function serializeValue(value) { + if (Array.isArray(value)) return `[${value.map((v) => serializeScalar(String(v))).join(', ')}]` + if (typeof value !== 'string') return String(value) + return serializeScalar(value) +} + +function serializeScalar(value) { + const needsQuotes = + value === '' || + value !== value.trim() || + /^[[\]{}"'#&*!|>%@`,-]/.test(value) || + value.includes(': ') || + value.includes(' #') || + value.includes(',') || + value.includes('\n') || + /^(true|false|null|~|-?\d+(\.\d+)?)$/i.test(value) + return needsQuotes ? JSON.stringify(value) : value +} + +/** + * Set (or, with `undefined`, remove) top-level keys in a document's front + * matter, creating the block if there isn't one. Every other line — other + * keys, comments, nested YAML — is kept as written, in its place. + */ +export function setFrontmatterFields(text, patch) { + const match = matchFence(text) + const lines = match ? match[1].split(/\r?\n/) : [] + const body = match ? text.slice(match[0].length) : text + const pending = new Map(Object.entries(patch)) + + const out = [] + for (let i = 0; i < lines.length; i++) { + const line = lines[i] + const colon = line.indexOf(':') + const key = !/^\s/.test(line) && colon > 0 ? line.slice(0, colon).trim() : null + if (key === null || !pending.has(key)) { + out.push(line) + continue + } + // Drop the key's own continuation lines along with it. + while (i + 1 < lines.length && /^\s+\S/.test(lines[i + 1])) i++ + const value = pending.get(key) + pending.delete(key) + if (value !== undefined && value !== null) out.push(`${key}: ${serializeValue(value)}`) + } + for (const [key, value] of pending) { + if (value !== undefined && value !== null) out.push(`${key}: ${serializeValue(value)}`) + } + + const block = `---\n${out.join('\n')}\n---\n` + if (match) return `${block}${body}` + return `${block}\n${body.replace(/^\s*\n/, '')}` +} diff --git a/server/index.mjs b/server/index.mjs index 08aa236..218670d 100644 --- a/server/index.mjs +++ b/server/index.mjs @@ -19,7 +19,10 @@ // DECKLE_TRUST_PROXY reverse proxies in front, for client addresses // ("true" = one; a number = that many; default none) // DECKLE_API_TOKENS bearer tokens enabling the /api/v1 machine API +// and the MCP server at /api/v1/mcp // DECKLE_API_CORS_ORIGINS origins allowed to call /api/v1 from a browser +// DECKLE_MAX_FILE_MB largest single file the library accepts (default 100) +// DECKLE_PROJECTS_DIR folder holding projects (default Projects) // // The NIB_* names these replaced are still honoured; see legacy-env.mjs. diff --git a/server/library-api.mjs b/server/library-api.mjs index b098558..10ed264 100644 --- a/server/library-api.mjs +++ b/server/library-api.mjs @@ -4,16 +4,17 @@ // It mirrors the handful of operations the browser's File System Access API // offers, because the client adapter (src/fs/remote.ts) presents these // endpoints *as* a FileSystemDirectoryHandle. Keeping the two in step is what -// lets the rest of the app stay backend-agnostic — notes, history, tasks, -// bookmarks and the AI tools all go through the same handle interface. +// lets the rest of the app stay backend-agnostic — notes, files, history, +// tasks and bookmarks all go through the same handle interface. // // Endpoints (all library-relative paths in the ?path= query parameter): -// GET /api/library/tree recursive note tree (one round trip per refresh) +// GET /api/library/tree recursive tree of notes and files (one round trip) // GET /api/library/list shallow directory listing, including dotfiles // GET /api/library/stat entry kind + mtime, 404 when missing // GET /api/library/file file contents // PUT /api/library/file write file contents (creates parent folders) // POST /api/library/dir create a directory (mkdir -p) +// POST /api/library/move rename a file or folder (&to= the destination) // DELETE /api/library/entry remove a file or directory import fs from 'node:fs/promises' @@ -34,14 +35,14 @@ const MD_EXT = /\.md$/i /** * A folder inside the library root that this API pretends does not exist. * - * The server keeps its own state there (the shared assistant settings, and the - * API key in them), and it sits under the library directory only because that - * is the volume a self-hoster actually mounts — not because it is part of the - * library. Every way out of this module passes through `safePath`, `walk` or - * `list`, so denying it in those places is what keeps it out of: the file API, - * the client's export walk (which lists its way through the remote handle), the - * server's own export walk, the machine API, and the assistant's read_file - * tool — none of which have any business reading a key. + * It is the server's own state, not part of the library. Deckle 2.x kept the + * AI assistant's settings there — a provider API key among them — and an + * upgraded volume may still hold that file, so the folder stays invisible even + * though nothing is written to it any more. Every way out of this module + * passes through `safePath`, `walk` or `list`, so denying it in those places is + * what keeps it out of: the file API, the client's export walk (which lists its + * way through the remote handle), the server's own export walk, the machine + * API and MCP — none of which have any business reading a key. * * Set DECKLE_STATE_DIR to move the state somewhere else entirely; this name * stays reserved either way, so a library can't grow a folder that would @@ -49,10 +50,27 @@ const MD_EXT = /\.md$/i */ export const RESERVED_DIR = '.deckle-state' -/** Notes are text; this cap stops a single request filling the volume. */ -const MAX_FILE_BYTES = 32 * 1024 * 1024 +/** + * The largest single file the library accepts, unless DECKLE_MAX_FILE_MB says + * otherwise. Notes are small, but the library also holds what an agent + * produced — PDFs, spreadsheets, images — so this is sized for documents + * rather than prose. It exists so one request cannot fill the volume. + */ +export const DEFAULT_MAX_FILE_BYTES = 100 * 1024 * 1024 + +/** + * Operating-system clutter that is never anyone's work. Dotfiles (.DS_Store, + * ._resource forks) are already skipped by the leading-dot rule. + */ +const CLUTTER = new Set(['thumbs.db', 'desktop.ini', 'icon\r']) -/** Thrown when an upload exceeds MAX_FILE_BYTES mid-stream. */ +/** The lower-case extension of a file name, without the dot ('' if none). */ +export function extensionOf(name) { + const dot = name.lastIndexOf('.') + return dot > 0 ? name.slice(dot + 1).toLowerCase() : '' +} + +/** Thrown when an upload exceeds the size cap mid-stream. */ export class TooLargeError extends Error {} /** Abort a stream once it has passed `limit` bytes (Content-Length can lie). */ @@ -90,11 +108,15 @@ async function statOrNull(abs) { /** * @param {string} rootDir the library directory - * @param {{ reservedPaths?: string[] }} [options] - * extra absolute paths to hide — the state directory, when DECKLE_STATE_DIR - * puts it somewhere inside the library other than RESERVED_DIR + * @param {{ reservedPaths?: string[], maxFileBytes?: number }} [options] + * reservedPaths: extra absolute paths to hide — the state directory, when + * DECKLE_STATE_DIR puts it somewhere inside the library other than + * RESERVED_DIR. maxFileBytes: the per-file size cap. */ -export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { +export function createLibraryApi( + rootDir, + { reservedPaths = [], maxFileBytes = DEFAULT_MAX_FILE_BYTES } = {}, +) { const root = path.resolve(rootDir) // Compared case-insensitively. A library bind-mounted from macOS or Windows @@ -134,10 +156,12 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { /** * Recursive walk producing exactly the shape `buildTree` in src/fs/library.ts - * builds by hand: folders first, then files, both sorted by display name, - * with dotfiles skipped and non-.md files ignored. Ids are relative to the - * directory being walked; the client re-prefixes them if it asked for a - * subfolder. + * builds by hand: folders first, then notes and files together, each group + * sorted by what the tree shows (a note's title, a file's name), with + * dotfiles skipped. A `.md` file is a note; anything else is a file — an + * image, a PDF, a spreadsheet — carried as `kind: 'asset'` so code that only + * understands notes can tell the two apart. Ids are relative to the directory + * being walked; the client re-prefixes them if it asked for a subfolder. */ async function walk(abs, prefix) { const folders = [] @@ -164,24 +188,38 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { name: entry.name, children: await walk(childAbs, id), }) - } else if (entry.isFile() && MD_EXT.test(entry.name)) { - // A note deleted mid-walk is simply not in the tree, rather than a + } else if (entry.isFile()) { + if (CLUTTER.has(entry.name.toLowerCase())) continue + // A file deleted mid-walk is simply not in the tree, rather than a // 404 for the whole library. const stat = await statOrNull(childAbs) if (!stat) continue - files.push({ - kind: 'file', - id, - name: entry.name, - title: entry.name.replace(MD_EXT, ''), - updatedAt: Math.round(stat.mtimeMs), - }) + if (MD_EXT.test(entry.name)) { + files.push({ + kind: 'file', + id, + name: entry.name, + title: entry.name.replace(MD_EXT, ''), + updatedAt: Math.round(stat.mtimeMs), + }) + } else { + files.push({ + kind: 'asset', + id, + name: entry.name, + // What the tree and tabs show: a file is known by its whole name. + title: entry.name, + ext: extensionOf(entry.name), + size: stat.size, + updatedAt: Math.round(stat.mtimeMs), + }) + } } // Symlinks and other entry kinds are ignored rather than followed. } folders.sort((a, b) => a.name.localeCompare(b.name)) - files.sort((a, b) => a.title.localeCompare(b.title)) + files.sort((a, b) => labelOf(a).localeCompare(labelOf(b))) return [...folders, ...files] } @@ -273,7 +311,7 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { await fs.mkdir(path.dirname(abs), { recursive: true }) const tmp = tempNameFor(abs, 'upload') try { - await pipeline(stream, limitBytes(MAX_FILE_BYTES), createWriteStream(tmp)) + await pipeline(stream, limitBytes(maxFileBytes), createWriteStream(tmp)) await fs.rename(tmp, abs) } catch (err) { await fs.rm(tmp, { force: true }) @@ -296,8 +334,11 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { return await fs.readFile(abs) }, - /** Write UTF-8 text, creating parent folders. Same atomic rename as - * `writeFile`, so an interrupted write can't truncate an existing note. */ + /** + * Write UTF-8 text — or a Buffer, written as-is — creating parent folders. + * Same atomic rename as `writeFile`, so an interrupted write can't truncate + * an existing note. + */ async writeText(rel, text) { const abs = await safePath(rel) await fs.mkdir(path.dirname(abs), { recursive: true }) @@ -329,6 +370,43 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { await fs.mkdir(abs, { recursive: true }) }, + /** + * Move a file within the library, creating the destination's folders. + * + * A rename, not a copy: the library is one volume, so moving a 90 MB PDF + * into the recycle bin costs the same as moving a note, and a file can + * never exist half-copied in two places. Refuses to replace anything + * already at `toRel` — the callers pick a free name first. + */ + async rename(fromRel, toRel) { + const from = await safePath(fromRel) + const to = await safePath(toRel) + if (from === root || to === root) throw new BadPathError('cannot move the library root') + if (holdsReserved(from)) throw new BadPathError(`cannot move ${RESERVED_DIR}`) + const stat = await fs.stat(from) + if (from.toLowerCase() !== to.toLowerCase() && (await statOrNull(to))) { + const err = new Error('destination exists') + err.code = 'EEXIST' + throw err + } + await fs.mkdir(path.dirname(to), { recursive: true }) + await fs.rename(from, to) + return { kind: stat.isDirectory() ? 'directory' : 'file' } + }, + + /** + * Append UTF-8 text to a file, creating it and its folders if need be. + * Used for append-only logs, where rewriting the whole file per line + * would be quadratic. + */ + async appendText(rel, text) { + const abs = await safePath(rel) + await fs.mkdir(path.dirname(abs), { recursive: true }) + await fs.appendFile(abs, text, 'utf8') + const stat = await fs.stat(abs) + return { lastModified: Math.round(stat.mtimeMs), size: stat.size } + }, + async remove(rel, recursive) { const abs = await safePath(rel) if (abs === root) throw new BadPathError('cannot remove the library root') @@ -349,6 +427,11 @@ export function createLibraryApi(rootDir, { reservedPaths = [] } = {}) { } }, - maxFileBytes: MAX_FILE_BYTES, + maxFileBytes, } } + +/** What the tree shows for an entry — a note's title, a file's full name. */ +function labelOf(node) { + return node.title +} diff --git a/server/library-store.mjs b/server/library-store.mjs index c53656a..5de9480 100644 --- a/server/library-store.mjs +++ b/server/library-store.mjs @@ -11,6 +11,11 @@ // `.deckle/tasks.json` the app reads, or the two halves would quietly disagree. // So this module mirrors those rules. The formats are the contract — if one // side changes, the other must follow. +// +// A library holds files as well as notes — whatever an agent produced: PDFs, +// spreadsheets, images. They follow the same rules where the rules make +// sense: deleting one moves it to the recycle bin, and replacing one moves the +// old copy there too, since a binary file has no text to snapshot. const MD_EXT = /\.md$/i const ILLEGAL = /[\\/:*?"<>|]/g @@ -51,6 +56,17 @@ function baseName(fileName) { return fileName.replace(MD_EXT, '') } +/** A `.md` file is a note; anything else is a file. */ +export function isNotePath(path) { + return MD_EXT.test(path) +} + +/** "report.final.pdf" → { stem: "report.final", ext: ".pdf" }; no ext → "". */ +function splitExt(name) { + const dot = name.lastIndexOf('.') + return dot > 0 ? { stem: name.slice(0, dot), ext: name.slice(dot) } : { stem: name, ext: '' } +} + function splitPath(id) { const idx = id.lastIndexOf('/') if (idx === -1) return { parentPath: '', name: id } @@ -108,6 +124,17 @@ export function normalizeNotePath(raw) { return joinPath(segments.join('/'), name) } +/** + * Normalise the path of any file — a note or not — from a request. The same + * rules as a note path, except that the extension is the caller's: an agent + * saving `report.pdf` means exactly that. + */ +export function normalizeFilePath(raw) { + const folder = normalizeFolderPath(raw) + if (!folder) throw new ApiError(400, 'invalid_path', 'path is required') + return folder +} + /** Same rules, but for a folder path (no extension forced, '' means the root). */ export function normalizeFolderPath(raw) { const value = String(raw ?? '') @@ -202,16 +229,37 @@ export function createLibraryStore(library) { // ---- Notes --------------------------------------------------------------- - /** Find a free file name in `folder`, appending " 1", " 2", … on collision. */ + /** + * Find a free name in `folder`, appending " 1", " 2", … on collision — + * before the extension, so "report.pdf" becomes "report 1.pdf". + */ async function uniqueName(folder, desired) { if (!(await library.exists(joinPath(folder, desired)))) return desired - const base = baseName(desired) + const { stem, ext } = splitExt(desired) for (let i = 1; ; i++) { - const candidate = `${base} ${i}.md` + const candidate = `${stem} ${i}${ext}` if (!(await library.exists(joinPath(folder, candidate)))) return candidate } } + /** What the API says about a file that isn't a note: everything but its bytes. */ + async function readFileInfo(path) { + const kind = await library.exists(path) + if (kind !== 'file') throw new ApiError(404, 'not_found', `no file at "${path}"`) + const stat = await library.stat(path) + const { parentPath, name } = splitPath(path) + const dot = name.lastIndexOf('.') + return { + path, + name, + folder: parentPath, + kind: isNotePath(path) ? 'note' : 'file', + ext: dot > 0 ? name.slice(dot + 1).toLowerCase() : '', + size: stat.size, + updatedAt: stat.lastModified, + } + } + async function readNote(path) { const kind = await library.exists(path) if (kind !== 'file') throw new ApiError(404, 'not_found', `no note at "${path}"`) @@ -242,7 +290,7 @@ export function createLibraryStore(library) { * Create or replace a note. An overwrite snapshots the version it replaces * into `.history` first, so an agent's edits are as recoverable as the app's. */ - async function writeNote(path, content, reason = 'ai') { + async function writeNote(path, content, reason = 'agent') { const existing = await library.exists(path) if (existing === 'directory') { throw new ApiError(409, 'conflict', `"${path}" is a folder`) @@ -263,13 +311,49 @@ export function createLibraryStore(library) { if (await library.exists(to)) { throw new ApiError(409, 'conflict', `a note already exists at "${to}"`) } - const content = await library.readText(from) - await library.writeText(to, content) - await library.remove(from, false) + await library.rename(from, to) await retargetHistory(from, to) return await readNote(to) } + /** Move or rename any file. Notes keep their history; others just move. */ + async function moveFile(from, to) { + if (from === to) return await readFileInfo(from) + if ((await library.exists(from)) !== 'file') { + throw new ApiError(404, 'not_found', `no file at "${from}"`) + } + if (await library.exists(to)) { + throw new ApiError(409, 'conflict', `something already exists at "${to}"`) + } + await library.rename(from, to) + if (isNotePath(from)) await retargetHistory(from, to) + return await readFileInfo(to) + } + + /** + * Store a file's bytes from a stream, keeping whatever it replaces. + * + * The upload lands under a hidden name first, so a connection that drops + * half way leaves the old file exactly where it was. Only once every byte + * has arrived does the old copy move to the recycle bin — marked as + * replaced, and restorable like anything deleted — and the new one take its + * place. + */ + async function putFile(path, stream, { replacedBy } = {}) { + const existing = await library.exists(path) + if (existing === 'directory') throw new ApiError(409, 'conflict', `"${path}" is a folder`) + const incoming = `${DATA_DIR}/incoming/${Date.now()}-${Math.random().toString(36).slice(2)}` + try { + await library.writeFile(incoming, stream) + if (existing === 'file') await trashEntry(path, { reason: 'replaced', by: replacedBy }) + await library.rename(incoming, path) + } catch (err) { + await library.remove(incoming, false).catch(() => {}) + throw err + } + return { file: await readFileInfo(path), created: existing !== 'file' } + } + // ---- Recycle bin --------------------------------------------------------- async function listTrash() { @@ -282,27 +366,56 @@ export function createLibraryStore(library) { return valid } - /** Move a note to the recycle bin, recording where it came from. */ - async function trashNote(path) { - const content = await library.readText(path) + /** + * Move a note or a file to the recycle bin, recording where it came from. + * + * A rename, so a large file costs no more than a note and arrives intact. + * `reason` is 'replaced' when a newer version took its place, so the bin can + * say why an item is there; `by` names the token that did it. + */ + async function trashEntry(path, { reason, by } = {}) { return await withLock(TRASH_INDEX, async () => { const { name } = splitPath(path) const trashName = await uniqueName(TRASH_DIR, name) - await library.writeText(joinPath(TRASH_DIR, trashName), content) - await library.remove(path, false) + await library.rename(path, joinPath(TRASH_DIR, trashName)) const items = await readJson(TRASH_INDEX, []) const entry = { trashName, originalPath: path, - title: baseName(name), + title: isNotePath(name) ? baseName(name) : name, deletedAt: Date.now(), } + if (!isNotePath(name)) entry.kind = 'file' + if (reason) entry.reason = reason + if (by) entry.by = by await writeJson(TRASH_INDEX, [...(Array.isArray(items) ? items : []), entry]) return entry }) } + /** Kept under its old name: every existing caller trashes notes. */ + const trashNote = (path) => trashEntry(path) + + /** + * Delete a folder the way the app does: everything in it — notes and files — + * goes to the recycle bin, then the emptied folder goes. Returns how many + * items were binned. + */ + async function trashFolder(folder, { by } = {}) { + const entries = [] + const collect = (nodes) => { + for (const node of nodes) { + if (node.kind === 'folder') collect(node.children) + else entries.push(joinPath(folder, node.id)) + } + } + collect(await library.tree(folder)) + for (const path of entries) await trashEntry(path, { by }) + await library.remove(folder, true) + return entries.length + } + async function restoreTrash(trashName) { return await withLock(TRASH_INDEX, async () => { const items = await readJson(TRASH_INDEX, []) @@ -311,18 +424,16 @@ export function createLibraryStore(library) { ) if (!entry) throw new ApiError(404, 'not_found', 'no such item in the recycle bin') - const content = await library.readText(joinPath(TRASH_DIR, trashName)) const { parentPath, name } = splitPath(entry.originalPath) const target = await uniqueName(parentPath, name) const path = joinPath(parentPath, target) - await library.writeText(path, content) - await library.remove(joinPath(TRASH_DIR, trashName), false) + await library.rename(joinPath(TRASH_DIR, trashName), path) await writeJson( TRASH_INDEX, items.filter((i) => i.trashName !== trashName), ) - return await readNote(path) + return isNotePath(path) ? await readNote(path) : await readFileInfo(path) }) } @@ -475,9 +586,15 @@ export function createLibraryStore(library) { writeNote, moveNote, uniqueName, + // files + readFileInfo, + moveFile, + putFile, // bin listTrash, trashNote, + trashEntry, + trashFolder, restoreTrash, deleteTrashItem, // history diff --git a/server/mcp.mjs b/server/mcp.mjs new file mode 100644 index 0000000..9048a2c --- /dev/null +++ b/server/mcp.mjs @@ -0,0 +1,698 @@ +// Deckle as an MCP server, at /api/v1/mcp. +// +// The Model Context Protocol is how agents such as Hermes discover and call +// tools. Pointing one at this endpoint gives it Deckle as a knowledge base: it +// can search and read what is already known, keep its work in projects, save +// what it produces — notes and files alike — and log what it did, all through +// the same library rules the REST API follows (the recycle bin, version +// history, the activity log). +// +// Transport: Streamable HTTP, stateless. Every request is a POST of a JSON-RPC +// message (or a batch), answered with a JSON body — the spec allows a server +// to reply with plain JSON instead of an event stream, and nothing here needs +// to push messages unprompted. So there is no session id, and GET (the +// optional server-to-client stream) is answered 405, as the spec provides. +// +// Authentication is the API's own: a bearer token from DECKLE_API_TOKENS, +// checked before this module sees the request. A read-only (`r`) token is +// shown the read tools only, and refused the rest if it asks anyway. + +import { Readable } from 'node:stream' +import { VERSION } from './version.mjs' +import { BadPathError } from './paths.mjs' +import { TooLargeError } from './library-api.mjs' +import { + ApiError, + isNotePath, + normalizeFilePath, + normalizeFolderPath, + normalizeNotePath, +} from './library-store.mjs' +import { STATUSES } from './projects.mjs' + +/** Newest first. The one a client asks for is used when we know it. */ +const PROTOCOL_VERSIONS = ['2025-11-25', '2025-06-18', '2025-03-26', '2024-11-05'] +const FALLBACK_VERSION = '2025-06-18' + +/** Base64 inflates by a third; keep a decoded file under the JSON body cap. */ +const MAX_INLINE_FILE_BYTES = 10 * 1024 * 1024 +const DEFAULT_READ_CHARS = 40_000 +const MAX_LIST_ENTRIES = 500 + +const INSTRUCTIONS = `Deckle is the user's knowledge base: plain Markdown notes and ordinary files in folders, which the user reads and reviews in the Deckle app. Treat it as long-term memory for your work. + +- Before starting a task, search for what is already known, and list_projects to see where the work belongs. +- Keep each piece of work in a project: create_project once, then save everything for it under Projects//. +- Save deliverables (reports, spreadsheets, images, code) with save_file or write_note, and write findings, decisions and sources as notes rather than leaving them only in the conversation. +- After each meaningful step, log_progress with what you did, what you decided and why, and what is next. +- Link related notes and files with [[wikilinks]] — [[Q3 report.pdf]], [[Research notes]] — and describe every important file in a note, because PDFs and images are only searchable by name and by the notes that mention them. +- Never store secrets: API keys, passwords, tokens, .env files, private keys. + +Deleting moves things to a recycle bin, and replacing a file keeps the old copy there, so the user can undo anything you do.` + +// ---- Formatting ------------------------------------------------------------ + +function iso(ms) { + return ms ? new Date(ms).toISOString() : 'unknown' +} + +function formatBytes(bytes) { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + +function text(value) { + return { content: [{ type: 'text', text: value }] } +} + +function toolError(message) { + return { content: [{ type: 'text', text: message }], isError: true } +} + +/** Arguments arrive untyped; these keep a malformed call from becoming a 500. */ +function str(args, key, { optional = false, max = 100_000 } = {}) { + const value = args?.[key] + if (value === undefined || value === null || value === '') { + if (optional) return undefined + throw new ApiError(400, 'invalid_params', `"${key}" is required`) + } + if (typeof value !== 'string') throw new ApiError(400, 'invalid_params', `"${key}" must be a string`) + if (value.length > max) throw new ApiError(400, 'invalid_params', `"${key}" is too long`) + return value +} + +function int(args, key, fallback, { min = 0, max = Number.MAX_SAFE_INTEGER } = {}) { + const value = args?.[key] + if (value === undefined || value === null) return fallback + const n = Number(value) + if (!Number.isFinite(n)) throw new ApiError(400, 'invalid_params', `"${key}" must be a number`) + return Math.min(max, Math.max(min, Math.floor(n))) +} + +function stringList(args, key) { + const value = args?.[key] + if (value === undefined || value === null) return undefined + if (typeof value === 'string') return value.split(',').map((s) => s.trim()).filter(Boolean) + if (Array.isArray(value)) return value.filter((v) => typeof v === 'string') + throw new ApiError(400, 'invalid_params', `"${key}" must be a list of strings`) +} + +/** A window onto long text, with directions for reading on. */ +function windowed(content, offset, maxChars) { + const slice = content.slice(offset, offset + maxChars) + const end = offset + slice.length + const more = + end < content.length + ? `\n\n[… ${content.length - end} more characters. Call read again with offset=${end} to continue.]` + : '' + return `${slice}${more}` +} + +// ---- The server ------------------------------------------------------------ + +export function createMcp({ library, store, search, activity, projects, libraryName, maxFileBytes }) { + const inlineLimit = Math.min(MAX_INLINE_FILE_BYTES, maxFileBytes) + + const record = (token, event) => activity.record({ actor: token.name, via: 'mcp', ...event }) + + // Each tool: name, whether a read-only token may use it, the schema the + // model sees, and what it does. Descriptions are written for the model. + const tools = [ + { + name: 'search', + readOnly: true, + title: 'Search the knowledge base', + description: + 'Search the Deckle knowledge base: notes by title and text, and files by name and by their contents (Word, Excel, PowerPoint, OpenDocument, CSV, JSON, text and code). Returns ranked paths with snippets. Use it before starting work to find what is already known. PDFs and images are matched by file name and by the notes that link to them.', + inputSchema: { + type: 'object', + properties: { + query: { type: 'string', description: 'Words to look for.' }, + folder: { type: 'string', description: 'Only search inside this folder, e.g. "Projects/Acme".' }, + kind: { type: 'string', enum: ['note', 'file'], description: 'Only notes, or only other files.' }, + limit: { type: 'integer', minimum: 1, maximum: 50, default: 10 }, + }, + required: ['query'], + }, + async run(args) { + const query = str(args, 'query', { max: 1000 }) + const kind = args?.kind === 'note' || args?.kind === 'file' ? args.kind : undefined + const results = await search.search(query, { + limit: int(args, 'limit', 10, { min: 1, max: 50 }), + folder: normalizeFolderPath(args?.folder ?? ''), + kind, + }) + if (!results.length) return text(`No matches for "${query}".`) + return text( + results + .map( + (r, i) => + `${i + 1}. ${r.path} (${r.kind}${r.size !== undefined ? `, ${formatBytes(r.size)}` : ''}, updated ${iso(r.updatedAt)})` + + (r.snippet ? `\n ${r.snippet}` : ''), + ) + .join('\n'), + ) + }, + }, + { + name: 'read', + readOnly: true, + title: 'Read a note or file', + description: + 'Read a note, or the text of a file, by its path. Word, Excel, PowerPoint, OpenDocument, CSV, JSON, text and code files come back as text (Excel as one table per sheet); images, PDFs and other binary files return their details only. Long content is cut at max_chars — call again with the offset it gives to read on.', + inputSchema: { + type: 'object', + properties: { + path: { type: 'string', description: 'Library-relative path, e.g. "Projects/Acme/Overview.md".' }, + offset: { type: 'integer', minimum: 0, default: 0 }, + max_chars: { type: 'integer', minimum: 1000, maximum: 200000, default: DEFAULT_READ_CHARS }, + }, + required: ['path'], + }, + async run(args) { + const raw = str(args, 'path', { max: 1024 }) + const offset = int(args, 'offset', 0) + const maxChars = int(args, 'max_chars', DEFAULT_READ_CHARS, { min: 1000, max: 200_000 }) + let path = normalizeFilePath(raw) + let kind = await library.exists(path) + // Let "Projects/Acme/Overview" find the note, as the REST API does. + if (!kind && !isNotePath(path)) { + const asNote = normalizeNotePath(raw) + if ((await library.exists(asNote)) === 'file') { + path = asNote + kind = 'file' + } + } + if (kind === 'directory') return toolError(`"${path}" is a folder — use list to see what is in it.`) + if (!kind) return toolError(`Nothing at "${path}". Use search or list to find the right path.`) + + if (isNotePath(path)) { + const note = await store.readNote(path) + return text( + `Note: ${note.path}\nUpdated: ${iso(note.updatedAt)} · ${note.content.length} characters\n\n` + + windowed(note.content, offset, maxChars), + ) + } + const info = await store.readFileInfo(path) + const header = `File: ${info.path}\nType: ${info.ext || 'unknown'} · ${formatBytes(info.size)} · updated ${iso(info.updatedAt)}` + const content = await search.textOf(path) + if (content === null) { + return text( + `${header}\n\nDeckle can't read the contents of this kind of file. Look for a note that describes it (search for its name), or read it with your own tools after downloading it from ${`/api/v1/files/${path.split('/').map(encodeURIComponent).join('/')}`}.`, + ) + } + return text(`${header} · ${content.length} characters of text\n\n${windowed(content, offset, maxChars)}`) + }, + }, + { + name: 'list', + readOnly: true, + title: 'List a folder', + description: + 'List the notes, files and folders in a folder of the knowledge base — the top of the library when folder is omitted. Set recursive to include everything below it.', + inputSchema: { + type: 'object', + properties: { + folder: { type: 'string', description: 'Library-relative folder, e.g. "Projects/Acme". Omit for the root.' }, + recursive: { type: 'boolean', default: false }, + }, + }, + async run(args) { + const folder = normalizeFolderPath(args?.folder ?? '') + if (folder && (await library.exists(folder)) !== 'directory') { + return toolError(`No folder at "${folder}".`) + } + const recursive = args?.recursive === true + const lines = [] + let truncated = false + const walk = (nodes, prefix, depth) => { + for (const node of nodes) { + if (lines.length >= MAX_LIST_ENTRIES) { + truncated = true + return + } + const path = prefix ? `${prefix}/${node.id.slice(node.id.lastIndexOf('/') + 1)}` : node.id + const indent = ' '.repeat(depth) + if (node.kind === 'folder') { + lines.push(`${indent}${path}/`) + if (recursive) walk(node.children, path, depth + 1) + } else if (node.kind === 'file') { + lines.push(`${indent}${path} (note, updated ${iso(node.updatedAt)})`) + } else { + lines.push(`${indent}${path} (${node.ext || 'file'}, ${formatBytes(node.size)}, updated ${iso(node.updatedAt)})`) + } + } + } + walk(await library.tree(folder), folder, 0) + if (!lines.length) return text(folder ? `"${folder}" is empty.` : 'The library is empty.') + return text( + `${folder || libraryName}:\n${lines.join('\n')}${truncated ? `\n[… stopped at ${MAX_LIST_ENTRIES} entries; list a subfolder to see more]` : ''}`, + ) + }, + }, + { + name: 'list_projects', + readOnly: true, + title: 'List projects', + description: `List projects — the folders inside ${projects.root}/ — with their status, summary, tags, size and when they last changed, most recent first. Use it to find where work belongs before saving anything.`, + inputSchema: { + type: 'object', + properties: { status: { type: 'string', enum: STATUSES } }, + }, + async run(args) { + const status = STATUSES.includes(args?.status) ? args.status : undefined + const list = await projects.list({ status }) + if (!list.length) { + return text( + status + ? `No ${status} projects.` + : `No projects yet. create_project starts one in ${projects.root}/.`, + ) + } + return text( + list + .map( + (p) => + `- ${p.name} [${p.status ?? 'no status'}] — ${p.summary || 'no summary'}\n` + + ` ${p.path}/ · ${p.notes} notes, ${p.files} files · last change ${iso(p.updatedAt)}` + + (p.tags.length ? ` · tags: ${p.tags.join(', ')}` : '') + + (p.overview ? `\n overview: ${p.overview}` : ''), + ) + .join('\n'), + ) + }, + }, + { + name: 'recent_activity', + readOnly: true, + title: 'Recent activity', + description: + 'What has been created, changed, moved or deleted in the knowledge base through the API and MCP — by you or by other agents — newest first. Use it to pick up where earlier work stopped.', + inputSchema: { + type: 'object', + properties: { + since: { type: 'string', description: 'An ISO date or time; only newer events are returned.' }, + project: { type: 'string', description: 'Only events inside this project.' }, + limit: { type: 'integer', minimum: 1, maximum: 200, default: 30 }, + }, + }, + async run(args) { + const sinceRaw = str(args, 'since', { optional: true, max: 64 }) + let since + if (sinceRaw) { + since = Date.parse(sinceRaw) + if (!Number.isFinite(since)) return toolError(`"since" must be a date, such as 2026-09-21.`) + } + const events = await activity.list({ + since, + project: str(args, 'project', { optional: true, max: 300 }), + limit: int(args, 'limit', 30, { min: 1, max: 200 }), + }) + if (!events.length) return text('No activity recorded yet.') + return text( + events + .map( + (e) => + `${iso(e.at)} · ${e.actor} ${e.action} ${e.kind ?? ''} ${e.path ?? ''}${e.to ? ` → ${e.to}` : ''}`.replace(/\s+/g, ' ').trim() + + (e.message ? ` — ${e.message}` : ''), + ) + .join('\n'), + ) + }, + }, + { + name: 'write_note', + readOnly: false, + title: 'Write a note', + description: + "Create or update a Markdown note. mode 'create' (the default) never overwrites — a clash gets a numbered name, which is returned; 'replace' overwrites the note, keeping the old version in its history; 'append' adds to the end, creating the note if it is missing. Link other notes and files with [[wikilinks]]. '.md' is added to the path if missing.", + inputSchema: { + type: 'object', + properties: { + path: { type: 'string', description: 'e.g. "Projects/Acme/Competitor research.md".' }, + content: { type: 'string', description: 'Markdown.' }, + mode: { type: 'string', enum: ['create', 'replace', 'append'], default: 'create' }, + message: { type: 'string', description: 'One line on why, shown in the activity log.' }, + }, + required: ['path', 'content'], + }, + async run(args, token) { + const path = normalizeNotePath(str(args, 'path', { max: 1024 })) + const content = str(args, 'content', { max: 4_000_000 }) + const mode = args?.mode ?? 'create' + const message = str(args, 'message', { optional: true, max: 1000 }) + if (!['create', 'replace', 'append'].includes(mode)) { + return toolError(`"mode" must be create, replace or append.`) + } + if ((await library.exists(path)) === 'directory') return toolError(`"${path}" is a folder.`) + + if (mode === 'create') { + const note = await store.createNote({ path, content }) + await record(token, { action: 'created', kind: 'note', path: note.path, message }) + return text( + note.path === path + ? `Created ${note.path}.` + : `A note already existed at ${path}, so this was saved as ${note.path}.`, + ) + } + let next = content + if (mode === 'append' && (await library.exists(path)) === 'file') { + const current = await library.readText(path) + const separator = current && !current.endsWith('\n') ? '\n\n' : current ? '\n' : '' + next = `${current}${separator}${content}` + } + const { note, created } = await store.writeNote(path, next, 'agent') + await record(token, { + action: created ? 'created' : mode === 'append' ? 'appended to' : 'replaced', + kind: 'note', + path: note.path, + message, + }) + return text(`${created ? 'Created' : mode === 'append' ? 'Appended to' : 'Replaced'} ${note.path}.`) + }, + }, + { + name: 'save_file', + readOnly: false, + title: 'Save a file', + description: `Save a file of any type — a report, spreadsheet, image, PDF, dataset or source file. For text formats pass the text as content; for binary files pass base64 with encoding "base64" (at most ${formatBytes(inlineLimit)}; for bigger files, or a whole folder, run the deckle_push.py script from the Deckle skill instead). A file already at the path is replaced and the old copy moved to the recycle bin, unless overwrite is false. Describe important files in a note that links to them with [[wikilinks]].`, + inputSchema: { + type: 'object', + properties: { + path: { type: 'string', description: 'e.g. "Projects/Acme/Deliverables/Q3 analysis.xlsx".' }, + content: { type: 'string' }, + encoding: { type: 'string', enum: ['utf8', 'base64'], default: 'utf8' }, + overwrite: { type: 'boolean', default: true }, + message: { type: 'string', description: 'One line on what this is, shown in the activity log.' }, + }, + required: ['path', 'content'], + }, + async run(args, token) { + const path = normalizeFilePath(str(args, 'path', { max: 1024 })) + const encoding = args?.encoding ?? 'utf8' + if (encoding !== 'utf8' && encoding !== 'base64') return toolError(`"encoding" must be utf8 or base64.`) + const raw = str(args, 'content', { max: Math.ceil(inlineLimit * 1.4) + 16 }) + const bytes = encoding === 'base64' ? Buffer.from(raw, 'base64') : Buffer.from(raw, 'utf8') + if (bytes.length > inlineLimit) { + return toolError(`That is ${formatBytes(bytes.length)}; save_file takes at most ${formatBytes(inlineLimit)}. Use the deckle_push.py script for large files.`) + } + const message = str(args, 'message', { optional: true, max: 1000 }) + const existing = await library.exists(path) + if (existing === 'directory') return toolError(`"${path}" is a folder.`) + if (existing === 'file' && args?.overwrite === false) { + return toolError(`A file already exists at "${path}" and overwrite is false.`) + } + + if (isNotePath(path)) { + const { note, created } = await store.writeNote(path, bytes.toString('utf8'), 'agent') + await record(token, { action: created ? 'created' : 'replaced', kind: 'note', path: note.path, message }) + return text(`${created ? 'Saved' : 'Replaced'} ${note.path}.`) + } + const { file, created } = await store.putFile(path, Readable.from([bytes]), { replacedBy: token.name }) + await record(token, { + action: created ? 'saved' : 'replaced', + kind: 'file', + path: file.path, + size: file.size, + message, + }) + return text( + `${created ? 'Saved' : 'Replaced'} ${file.path} (${formatBytes(file.size)}).` + + (created ? '' : ' The previous copy is in the recycle bin.'), + ) + }, + }, + { + name: 'create_project', + readOnly: false, + title: 'Start a project', + description: `Start a project: creates ${projects.root}// with an Overview.md holding its status, summary and tags, which the user sees in Deckle's Projects view. Save everything produced for the project inside that folder. Check list_projects first so you don't start a duplicate.`, + inputSchema: { + type: 'object', + properties: { + name: { type: 'string', description: 'Short and human, e.g. "Acme market research". Becomes the folder name.' }, + summary: { type: 'string', description: 'One or two sentences on the goal.' }, + tags: { type: 'array', items: { type: 'string' } }, + status: { type: 'string', enum: STATUSES, default: 'active' }, + body: { type: 'string', description: 'Optional Markdown for the rest of the overview: goal, scope, sources.' }, + }, + required: ['name', 'summary'], + }, + async run(args, token) { + const project = await projects.create({ + name: str(args, 'name', { max: 120 }), + summary: str(args, 'summary', { max: 2000 }), + tags: stringList(args, 'tags'), + status: args?.status, + body: str(args, 'body', { optional: true, max: 200_000 }), + createdBy: token.name, + }) + await record(token, { action: 'created', kind: 'project', path: project.path, project: project.name }) + return text(`Created project "${project.name}" at ${project.path}/ (overview: ${project.overview}).`) + }, + }, + { + name: 'update_project', + readOnly: false, + title: 'Update a project', + description: `Change a project's status (${STATUSES.join(', ')}), summary or tags. Mark a project done when its work is finished.`, + inputSchema: { + type: 'object', + properties: { + name: { type: 'string' }, + status: { type: 'string', enum: STATUSES }, + summary: { type: 'string' }, + tags: { type: 'array', items: { type: 'string' } }, + }, + required: ['name'], + }, + async run(args, token) { + const project = await projects.update(str(args, 'name', { max: 300 }), { + status: args?.status, + summary: str(args, 'summary', { optional: true, max: 2000 }), + tags: stringList(args, 'tags'), + }) + await record(token, { + action: 'updated', + kind: 'project', + path: project.path, + project: project.name, + message: args?.status ? `status: ${project.status}` : undefined, + }) + return text(`Updated "${project.name}": ${project.status ?? 'no status'} — ${project.summary || 'no summary'}.`) + }, + }, + { + name: 'log_progress', + readOnly: false, + title: 'Log progress', + description: + "Add a dated entry to a project's Log.md: what you did, what you decided and why, and what is next. Write one after each meaningful step, so the user can follow and review the work.", + inputSchema: { + type: 'object', + properties: { + project: { type: 'string' }, + entry: { type: 'string', description: 'Markdown; may link files and notes with [[wikilinks]].' }, + }, + required: ['project', 'entry'], + }, + async run(args, token) { + const entry = str(args, 'entry', { max: 20_000 }) + const { project, path } = await projects.log(str(args, 'project', { max: 300 }), entry, token.name) + await record(token, { action: 'logged', kind: 'note', path, project, message: entry.split('\n')[0] }) + return text(`Logged to ${path}.`) + }, + }, + { + name: 'move', + readOnly: false, + title: 'Move or rename', + description: 'Move or rename a note or file. Never overwrites: fails if something is already at the destination.', + inputSchema: { + type: 'object', + properties: { + from: { type: 'string' }, + to: { type: 'string', description: 'The full new path, including the file name.' }, + }, + required: ['from', 'to'], + }, + async run(args, token) { + const from = normalizeFilePath(str(args, 'from', { max: 1024 })) + let to = normalizeFilePath(str(args, 'to', { max: 1024 })) + // A note stays a note: "Archive/Plan" means "Archive/Plan.md". + if (isNotePath(from) && !isNotePath(to)) to = normalizeNotePath(to) + const moved = await store.moveFile(from, to) + await record(token, { action: 'moved', kind: moved.kind, path: from, to: moved.path }) + return text(`Moved ${from} → ${moved.path}.`) + }, + }, + { + name: 'delete', + readOnly: false, + title: 'Move to the recycle bin', + description: + 'Move a note or file to the recycle bin, where the user can restore it. Does not delete folders.', + inputSchema: { + type: 'object', + properties: { + path: { type: 'string' }, + message: { type: 'string', description: 'One line on why, shown in the activity log.' }, + }, + required: ['path'], + }, + async run(args, token) { + const path = normalizeFilePath(str(args, 'path', { max: 1024 })) + const kind = await library.exists(path) + if (kind === 'directory') return toolError(`"${path}" is a folder; this tool only deletes files.`) + if (!kind) return toolError(`Nothing at "${path}".`) + await store.trashEntry(path, { by: token.name }) + await record(token, { + action: 'deleted', + kind: isNotePath(path) ? 'note' : 'file', + path, + message: str(args, 'message', { optional: true, max: 1000 }), + }) + return text(`Moved ${path} to the recycle bin.`) + }, + }, + ] + + const byName = new Map(tools.map((t) => [t.name, t])) + + function describeTool(tool) { + return { + name: tool.name, + title: tool.title, + description: tool.description, + inputSchema: tool.inputSchema, + annotations: { + title: tool.title, + readOnlyHint: tool.readOnly, + destructiveHint: tool.name === 'delete' || tool.name === 'save_file' || tool.name === 'write_note', + idempotentHint: tool.readOnly, + openWorldHint: false, + }, + } + } + + function rpcError(id, code, message) { + return { jsonrpc: '2.0', id: id ?? null, error: { code, message } } + } + + /** Answer one JSON-RPC message; null for a notification or a response. */ + async function handleMessage(message, token) { + if (!message || typeof message !== 'object' || message.jsonrpc !== '2.0') { + return rpcError(null, -32600, 'invalid JSON-RPC message') + } + const { id, method, params } = message + // A notification (no id) or a response to something we never asked: nothing to say. + if (id === undefined || id === null || typeof method !== 'string') return null + + const canWrite = token.scope === 'rw' + switch (method) { + case 'initialize': { + const requested = params?.protocolVersion + return { + jsonrpc: '2.0', + id, + result: { + protocolVersion: PROTOCOL_VERSIONS.includes(requested) ? requested : FALLBACK_VERSION, + capabilities: { tools: { listChanged: false } }, + serverInfo: { name: 'deckle', title: `Deckle — ${libraryName}`, version: VERSION }, + instructions: canWrite + ? INSTRUCTIONS + : `${INSTRUCTIONS}\n\nThis connection uses a read-only token: search, read, list, list_projects and recent_activity are available; nothing can be saved.`, + }, + } + } + case 'ping': + return { jsonrpc: '2.0', id, result: {} } + case 'tools/list': + return { + jsonrpc: '2.0', + id, + result: { tools: tools.filter((t) => canWrite || t.readOnly).map(describeTool) }, + } + case 'tools/call': { + const tool = byName.get(params?.name) + if (!tool) return rpcError(id, -32602, `unknown tool: ${params?.name}`) + if (!tool.readOnly && !canWrite) { + return { + jsonrpc: '2.0', + id, + result: toolError(`The token "${token.name}" is read-only, so ${tool.name} is not available.`), + } + } + try { + return { jsonrpc: '2.0', id, result: await tool.run(params?.arguments ?? {}, token) } + } catch (err) { + // What went wrong goes back to the model as a tool error, so it can + // correct itself; only the unexpected is logged. + if (err instanceof ApiError) return { jsonrpc: '2.0', id, result: toolError(err.message) } + if (err?.code === 'ENOENT') return { jsonrpc: '2.0', id, result: toolError('Not found.') } + if (err instanceof BadPathError) { + return { jsonrpc: '2.0', id, result: toolError(err.message) } + } + if (err?.code === 'EEXIST') { + return { jsonrpc: '2.0', id, result: toolError('Something already exists at that path.') } + } + if (err instanceof TooLargeError) { + return { jsonrpc: '2.0', id, result: toolError('That file is larger than this library accepts.') } + } + console.error('[deckle] MCP tool failed:', tool.name, err) + return { jsonrpc: '2.0', id, result: toolError('Deckle hit an internal error running that tool.') } + } + } + default: + return rpcError(id, -32601, `method not found: ${method}`) + } + } + + /** + * Handle one HTTP request to the MCP endpoint. `readBody` returns the raw + * text of the request; `send(status, body?, headers?)` writes the response. + */ + async function handleHttp(req, token, { readBody, send }) { + if (req.method === 'GET' || req.method === 'DELETE') { + // No server-initiated stream and no sessions to end: the spec's answer + // for both is 405. + send(405, { error: { code: 'method_not_allowed', message: 'this MCP server answers POST only' } }, { Allow: 'POST' }) + return + } + if (req.method !== 'POST') { + send(405, { error: { code: 'method_not_allowed', message: `${req.method} is not supported` } }, { Allow: 'POST' }) + return + } + + let payload + try { + payload = JSON.parse(await readBody()) + } catch (err) { + if (err instanceof TooLargeError) throw err + send(400, rpcError(null, -32700, 'parse error')) + return + } + + if (Array.isArray(payload)) { + if (!payload.length) { + send(400, rpcError(null, -32600, 'empty batch')) + return + } + const replies = [] + for (const message of payload) { + const reply = await handleMessage(message, token) + if (reply) replies.push(reply) + } + if (!replies.length) send(202) + else send(200, replies) + return + } + + const reply = await handleMessage(payload, token) + if (!reply) send(202) + else send(200, reply) + } + + return { handleHttp, handleMessage, tools } +} diff --git a/server/mime.mjs b/server/mime.mjs new file mode 100644 index 0000000..53b26c4 --- /dev/null +++ b/server/mime.mjs @@ -0,0 +1,57 @@ +// Content types for files served from the library. +// +// Only ever used on a response that is also `attachment`, `nosniff` and +// sandboxed, so a type here is a label for the receiving program — never an +// invitation for a browser to render. Anything not listed is octet-stream. + +const TYPES = { + md: 'text/markdown; charset=utf-8', + markdown: 'text/markdown; charset=utf-8', + txt: 'text/plain; charset=utf-8', + log: 'text/plain; charset=utf-8', + csv: 'text/csv; charset=utf-8', + tsv: 'text/tab-separated-values; charset=utf-8', + json: 'application/json', + jsonl: 'application/x-ndjson', + ndjson: 'application/x-ndjson', + yaml: 'application/yaml', + yml: 'application/yaml', + xml: 'application/xml', + html: 'text/html; charset=utf-8', + htm: 'text/html; charset=utf-8', + pdf: 'application/pdf', + png: 'image/png', + jpg: 'image/jpeg', + jpeg: 'image/jpeg', + gif: 'image/gif', + webp: 'image/webp', + avif: 'image/avif', + svg: 'image/svg+xml', + bmp: 'image/bmp', + ico: 'image/x-icon', + mp3: 'audio/mpeg', + wav: 'audio/wav', + ogg: 'audio/ogg', + m4a: 'audio/mp4', + mp4: 'video/mp4', + webm: 'video/webm', + mov: 'video/quicktime', + zip: 'application/zip', + gz: 'application/gzip', + tar: 'application/x-tar', + docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', + xlsx: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', + pptx: 'application/vnd.openxmlformats-officedocument.presentationml.presentation', + doc: 'application/msword', + xls: 'application/vnd.ms-excel', + ppt: 'application/vnd.ms-powerpoint', + odt: 'application/vnd.oasis.opendocument.text', + ods: 'application/vnd.oasis.opendocument.spreadsheet', + odp: 'application/vnd.oasis.opendocument.presentation', +} + +export function contentTypeFor(name) { + const dot = name.lastIndexOf('.') + const ext = dot > 0 ? name.slice(dot + 1).toLowerCase() : '' + return TYPES[ext] ?? 'application/octet-stream' +} diff --git a/server/openapi.mjs b/server/openapi.mjs index 9066117..6c75e15 100644 --- a/server/openapi.mjs +++ b/server/openapi.mjs @@ -59,13 +59,21 @@ export function buildOpenApi(libraryName) { version: VERSION, summary: 'Read and write a Deckle knowledge base over HTTP.', description: - 'Deckle stores a knowledge base as plain Markdown files in folders, plus a task planner ' + - 'and bookmark collections. This API exposes all of it: search and read notes, create ' + - 'and edit them, move them between folders, manage tasks and bookmarks, browse version ' + - 'history and the recycle bin, and export the whole thing as a ZIP.\n\n' + - 'Deleting a note moves it to the recycle bin rather than erasing it, and overwriting a ' + - 'note snapshots the version it replaces into version history — so edits made through ' + - 'this API are as recoverable as edits made in the app.\n\n' + + 'Deckle is a knowledge base for agentic work: plain Markdown notes and ordinary files ' + + '(PDFs, spreadsheets, images, documents, code) in folders, plus a task planner and ' + + 'bookmark collections. This API exposes all of it: search notes and documents, read ' + + 'and write notes, store and fetch files of any type, move things between folders, ' + + 'review the activity log, manage tasks and bookmarks, browse version history and the ' + + 'recycle bin, and export the whole thing as a ZIP.\n\n' + + 'Work usually lives in projects — folders inside `Projects/` with an `Overview.md` ' + + '(front matter: status, summary, tags) and a `Log.md`. Agents that speak MCP get ' + + 'project tools for these at `/api/v1/mcp`; over REST they are ordinary folders and ' + + 'notes.\n\n' + + 'Deleting moves a note or file to the recycle bin rather than erasing it; overwriting ' + + 'a note snapshots the version it replaces into version history, and replacing any ' + + 'other file moves the old copy to the recycle bin — so changes made through this API ' + + 'are as recoverable as changes made in the app. Every write is recorded in the ' + + 'activity log under the name of the token that made it.\n\n' + 'Authenticate every request with `Authorization: Bearer `. Tokens are ' + 'configured server-side and may be read-only, in which case any write returns 403.', license: { name: 'MIT' }, @@ -74,7 +82,10 @@ export function buildOpenApi(libraryName) { security: [{ bearerAuth: [] }], tags: [ { name: 'notes', description: 'Markdown notes — the knowledge base itself.' }, - { name: 'search', description: 'Find notes by meaning of the words they contain.' }, + { name: 'files', description: 'Files of any type: reports, spreadsheets, images, PDFs.' }, + { name: 'search', description: 'Find notes and documents by the words they contain.' }, + { name: 'activity', description: 'What agents and scripts changed, and when.' }, + { name: 'mcp', description: 'The same library as a Model Context Protocol server.' }, { name: 'folders', description: 'The folder tree notes are organised into.' }, { name: 'transfer', description: 'Bulk import and whole-library export.' }, { name: 'tasks', description: 'The task planner and its projects.' }, @@ -265,12 +276,15 @@ export function buildOpenApi(libraryName) { get: withErrors({ tags: ['search'], operationId: 'searchNotes', - summary: 'Find the notes most relevant to a query.', + summary: 'Find the notes and files most relevant to a query.', description: - 'BM25 ranking over note titles, paths and content, with titles weighted heavily. ' + - 'Returns the best matches with a snippet around the match — read the full note with ' + - 'GET /notes/{path} once you know which one you want. This is the right first call ' + - 'for answering a question from the knowledge base.', + 'BM25 ranking over notes (title, path and content) and other files (name, path, and ' + + 'the text inside Word, Excel, PowerPoint, OpenDocument, CSV, JSON, text and code ' + + 'files), with names weighted heavily. PDFs and images are matched by name only. ' + + 'Returns the best matches with a snippet around the match — read a note with ' + + 'GET /notes/{path} or fetch a file with GET /files/{path} once you know which one ' + + 'you want. This is the right first call for answering a question from the ' + + 'knowledge base.', parameters: [ { name: 'q', @@ -290,6 +304,12 @@ export function buildOpenApi(libraryName) { schema: { type: 'string' }, description: 'Restrict the search to one folder and its subfolders.', }, + { + name: 'kind', + in: 'query', + schema: { type: 'string', enum: ['note', 'file'] }, + description: 'Only notes, or only other files.', + }, ], responses: { 200: jsonResponse('Ranked results, best first.', { @@ -308,7 +328,7 @@ export function buildOpenApi(libraryName) { get: withErrors({ tags: ['folders'], operationId: 'getTree', - summary: 'Get the folder and note tree.', + summary: 'Get the folder tree, with its notes and files.', parameters: [ { name: 'path', @@ -350,19 +370,197 @@ export function buildOpenApi(libraryName) { delete: withErrors({ tags: ['folders'], operationId: 'deleteFolder', - summary: 'Delete a folder, sending every note inside it to the recycle bin.', + summary: 'Delete a folder, sending every note and file inside it to the recycle bin.', + responses: { + 200: jsonResponse('Deletion result.', { + type: 'object', + properties: { + deleted: { type: 'string' }, + trashed: { type: 'integer', description: 'Notes and files moved to the recycle bin.' }, + }, + }), + }, + }), + }, + + '/files': { + get: withErrors({ + tags: ['files'], + operationId: 'listFiles', + summary: 'List notes and files, most recently changed first.', + description: + 'Every note and file under a folder, with type, size and modification time. ' + + 'Use `kind=file` for just the non-note files an agent has stored.', + parameters: [ + { name: 'folder', in: 'query', schema: { type: 'string' }, description: 'Omit for the whole library.' }, + { + name: 'recursive', + in: 'query', + schema: { type: 'boolean', default: true }, + description: 'false lists only the folder itself, not its subfolders.', + }, + { name: 'kind', in: 'query', schema: { type: 'string', enum: ['note', 'file'] } }, + { name: 'limit', in: 'query', schema: { type: 'integer', default: 200, minimum: 1, maximum: 2000 } }, + { name: 'offset', in: 'query', schema: { type: 'integer', default: 0, minimum: 0 } }, + ], + responses: { + 200: jsonResponse('A page of files.', { + type: 'object', + properties: { + total: { type: 'integer' }, + limit: { type: 'integer' }, + offset: { type: 'integer' }, + files: { type: 'array', items: ref('FileInfo') }, + }, + }), + }, + }), + }, + + '/files/{path}': { + parameters: [ + { + name: 'path', + in: 'path', + required: true, + schema: { type: 'string' }, + description: + 'File path relative to the library root, e.g. "Projects/Acme/Q3 report.pdf". ' + + 'Percent-encode each segment. Hidden (dot) folders are rejected.', + }, + ], + get: withErrors({ + tags: ['files'], + operationId: 'getFile', + summary: 'Download a file (or its details, with meta=true).', + parameters: [ + { + name: 'meta', + in: 'query', + schema: { type: 'boolean' }, + description: 'Return the file\'s details as JSON instead of its bytes.', + }, + ], + responses: { + 200: { + description: 'The file, as an attachment — or its details, with meta=true.', + content: { + 'application/octet-stream': { schema: { type: 'string', format: 'binary' } }, + 'application/json': { schema: ref('FileInfo') }, + }, + }, + }, + }), + put: withErrors( + { + tags: ['files'], + operationId: 'putFile', + summary: 'Store a file of any type from the raw request body.', + description: + 'Send the file\'s bytes as the body (curl: `--data-binary @report.pdf` or ' + + '`-T report.pdf`). Parent folders are created. A file already at the path is ' + + 'replaced only once the upload has fully arrived, and the old copy moves to the ' + + 'recycle bin; a `.md` path is written as a note instead, with the old version kept ' + + 'in its history. Files are limited to DECKLE_MAX_FILE_MB (100 MB by default).', + parameters: [ + { + name: 'overwrite', + in: 'query', + schema: { type: 'boolean', default: true }, + description: 'false answers 409 rather than replace an existing file.', + }, + { + name: 'message', + in: 'query', + schema: { type: 'string' }, + description: 'One line on what this is, shown in the activity log. Also accepted as an X-Deckle-Message header.', + }, + ], + requestBody: { + required: true, + content: { 'application/octet-stream': { schema: { type: 'string', format: 'binary' } } }, + }, + responses: { + 200: jsonResponse('Replaced an existing file.', ref('FileInfo')), + 201: jsonResponse('Created a new file.', ref('FileInfo')), + }, + }, + { 413: jsonResponse('Larger than DECKLE_MAX_FILE_MB.', ref('Error')) }, + ), + delete: withErrors({ + tags: ['files'], + operationId: 'deleteFile', + summary: 'Move a file to the recycle bin.', + parameters: [ + { name: 'permanent', in: 'query', schema: { type: 'boolean' }, description: 'Erase instead.' }, + ], responses: { 200: jsonResponse('Deletion result.', { type: 'object', properties: { deleted: { type: 'string' }, - trashed: { type: 'integer', description: 'Notes moved to the recycle bin.' }, + permanent: { type: 'boolean' }, + trash: ref('TrashItem'), }, }), }, }), }, + '/activity': { + get: withErrors({ + tags: ['activity'], + operationId: 'listActivity', + summary: 'What was created, changed, moved or deleted through the API and MCP.', + description: + 'Newest first. Every write made with a token is recorded under the token\'s name; ' + + 'changes made in the app itself are not. Use `since` with the `at` of the last ' + + 'event you saw to fetch only what is new.', + parameters: [ + { + name: 'since', + in: 'query', + schema: { type: 'string' }, + description: 'Unix milliseconds, or an ISO date — only later events.', + }, + { name: 'project', in: 'query', schema: { type: 'string' } }, + { name: 'actor', in: 'query', schema: { type: 'string' }, description: 'A token name.' }, + { name: 'path', in: 'query', schema: { type: 'string' }, description: 'A file, or a folder and everything in it.' }, + { name: 'limit', in: 'query', schema: { type: 'integer', default: 100, minimum: 1, maximum: 1000 } }, + ], + responses: { + 200: jsonResponse('Events, newest first.', { + type: 'object', + properties: { + count: { type: 'integer' }, + events: { type: 'array', items: ref('ActivityEvent') }, + }, + }), + }, + }), + }, + + '/mcp': { + post: { + tags: ['mcp'], + operationId: 'mcp', + summary: 'The Model Context Protocol endpoint (Streamable HTTP, stateless).', + description: + 'Point an MCP client at this URL with the same bearer token. It answers JSON-RPC ' + + 'POSTs with JSON (no event stream, no session id), and offers the tools search, ' + + 'read, list, list_projects and recent_activity — plus, for a read-write token, ' + + 'write_note, save_file, create_project, update_project, log_progress, move and ' + + 'delete. Requests carrying an Origin header must come from DECKLE_API_CORS_ORIGINS.', + requestBody: jsonBody({ type: 'object', description: 'A JSON-RPC 2.0 message, or a batch of them.' }), + responses: { + 200: jsonResponse('A JSON-RPC response (or a batch of them).', { type: 'object' }), + 202: { description: 'The message was a notification; there is nothing to answer.' }, + 401: ERRORS[401], + 403: jsonResponse('An Origin that is not allowed.', ref('Error')), + }, + }, + }, + '/import': { post: withErrors({ tags: ['transfer'], @@ -729,7 +927,7 @@ export function buildOpenApi(libraryName) { get: withErrors({ tags: ['trash'], operationId: 'listTrash', - summary: 'List notes in the recycle bin.', + summary: 'List notes and files in the recycle bin.', responses: { 200: jsonResponse('Recycle bin contents, newest first.', { type: 'object', @@ -766,8 +964,13 @@ export function buildOpenApi(libraryName) { post: withErrors({ tags: ['trash'], operationId: 'restoreTrashItem', - summary: 'Restore a note from the recycle bin to where it was deleted from.', - responses: { 200: jsonResponse('The restored note.', ref('Note')) }, + summary: 'Restore a note or file from the recycle bin to where it was deleted from.', + description: 'If something now occupies that path, the item comes back under a numbered name.', + responses: { + 200: jsonResponse('The restored note, or the restored file\'s details.', { + oneOf: [ref('Note'), ref('FileInfo')], + }), + }, }), }, }, @@ -810,22 +1013,59 @@ export function buildOpenApi(libraryName) { type: 'object', properties: { path: { type: 'string' }, - title: { type: 'string' }, + title: { type: 'string', description: 'A note\'s title, or a file\'s name.' }, folder: { type: 'string' }, + kind: { type: 'string', enum: ['note', 'file'] }, score: { type: 'number', description: 'BM25 relevance; higher is better.' }, - snippet: { type: 'string', description: 'Text around the best match.' }, + snippet: { + type: 'string', + description: 'Text around the best match; empty for a file matched only by name.', + }, + size: { type: 'integer', description: 'Files only.' }, updatedAt: { type: 'integer' }, }, }, + FileInfo: { + type: 'object', + properties: { + path: { type: 'string', example: 'Projects/Acme/Q3 report.pdf' }, + name: { type: 'string', example: 'Q3 report.pdf' }, + folder: { type: 'string', example: 'Projects/Acme' }, + kind: { type: 'string', enum: ['note', 'file'] }, + ext: { type: 'string', example: 'pdf' }, + size: { type: 'integer', description: 'Bytes.' }, + updatedAt: { type: 'integer', description: 'Unix milliseconds.' }, + }, + }, + ActivityEvent: { + type: 'object', + properties: { + at: { type: 'integer', description: 'Unix milliseconds.' }, + actor: { type: 'string', description: 'The name of the token that made the change.' }, + via: { type: 'string', enum: ['api', 'mcp'] }, + action: { type: 'string', example: 'saved' }, + kind: { type: 'string', enum: ['note', 'file', 'folder', 'project', 'task', 'bookmark'] }, + path: { type: 'string' }, + to: { type: 'string', description: 'Moves only: where it went.' }, + project: { type: 'string', description: 'The project folder the change was in, if any.' }, + size: { type: 'integer' }, + count: { type: 'integer', description: 'Bulk changes: how many items.' }, + message: { type: 'string', description: 'What the agent said about it.' }, + }, + }, TreeNode: { type: 'object', - description: 'A folder (with children) or a note.', + description: + 'A folder (with children), a note (kind "file", a .md file), or any other file ' + + '(kind "asset" — a PDF, image, spreadsheet…).', properties: { - kind: { type: 'string', enum: ['folder', 'file'] }, + kind: { type: 'string', enum: ['folder', 'file', 'asset'] }, id: { type: 'string', description: 'Library-relative path.' }, name: { type: 'string' }, - title: { type: 'string', description: 'Files only: the name without ".md".' }, - updatedAt: { type: 'integer', description: 'Files only.' }, + title: { type: 'string', description: 'A note\'s name without ".md"; a file\'s full name.' }, + ext: { type: 'string', description: 'Assets only: the lower-case extension.' }, + size: { type: 'integer', description: 'Assets only: bytes.' }, + updatedAt: { type: 'integer', description: 'Notes and assets.' }, children: { type: 'array', description: 'Folders only.', @@ -922,7 +1162,7 @@ export function buildOpenApi(libraryName) { snapName: { type: 'string', description: 'Identifier for /history/{snapshot}.' }, noteId: { type: 'string', description: 'The note this version belongs to.' }, savedAt: { type: 'integer' }, - reason: { type: 'string', enum: ['edit', 'ai', 'restore'] }, + reason: { type: 'string', enum: ['edit', 'agent', 'ai', 'restore'] }, }, }, TrashItem: { @@ -930,8 +1170,15 @@ export function buildOpenApi(libraryName) { properties: { trashName: { type: 'string', description: 'Identifier for /trash/{trashName}.' }, originalPath: { type: 'string' }, - title: { type: 'string' }, + title: { type: 'string', description: 'A note\'s title, or a file\'s name.' }, deletedAt: { type: 'integer' }, + kind: { type: 'string', enum: ['file'], description: 'Present for non-note files.' }, + reason: { + type: 'string', + enum: ['replaced'], + description: 'Present when a newer version took this one\'s place.', + }, + by: { type: 'string', description: 'The token that deleted or replaced it.' }, }, }, }, diff --git a/server/projects.mjs b/server/projects.mjs new file mode 100644 index 0000000..f10bda7 --- /dev/null +++ b/server/projects.mjs @@ -0,0 +1,227 @@ +// Projects: a convention over folders, not a new kind of storage. +// +// A project is any folder directly inside the projects folder (`Projects/` by +// default, DECKLE_PROJECTS_DIR to change it). Whatever an agent produces while +// working on it — notes, PDFs, spreadsheets, images, source files — lives in +// that folder like any other file, so it opens in any editor and survives +// without Deckle. +// +// Two files give a project its shape, and both are optional: +// +// Overview.md front matter (status, summary, tags, created) above a normal +// note describing the project. README.md or index.md stand in +// when there is no Overview.md, so a pushed code project is +// described by its own README. +// Log.md an append-only, dated record of what was done and why, +// written by the agent as it works. +// +// A folder with neither is still a project — it just has no status yet. + +import { parseFrontmatter, setFrontmatterFields } from './frontmatter.mjs' +import { ApiError } from './library-store.mjs' + +export const OVERVIEW = 'Overview.md' +export const LOG = 'Log.md' +const OVERVIEW_CANDIDATES = [OVERVIEW, 'README.md', 'Readme.md', 'readme.md', 'index.md'] + +export const STATUSES = ['active', 'paused', 'done', 'archived'] + +/** "2026-09-21" in the server's local time (set TZ on the container). */ +function dateStamp(date) { + const pad = (n) => String(n).padStart(2, '0') + return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}` +} + +function timeStamp(date) { + const pad = (n) => String(n).padStart(2, '0') + return `${dateStamp(date)} ${pad(date.getHours())}:${pad(date.getMinutes())}` +} + +function asTags(value) { + if (value === undefined) return undefined + const list = Array.isArray(value) ? value : String(value).split(',') + return list + .map((t) => String(t).trim().replace(/^#/, '')) + .filter(Boolean) + .slice(0, 20) +} + +function validateStatus(status) { + if (status === undefined) return undefined + if (!STATUSES.includes(status)) { + throw new ApiError(400, 'invalid_body', `"status" must be one of ${STATUSES.join(', ')}`) + } + return status +} + +/** Quote a one-line summary for front matter. */ +function oneLine(text, max = 300) { + return String(text ?? '') + .replace(/\s+/g, ' ') + .trim() + .slice(0, max) +} + +export function createProjects({ library, store, now = () => new Date(), projectsDir = 'Projects' }) { + const root = projectsDir + + function folderFor(name) { + const clean = store.sanitizeName(String(name ?? ''), '') + if (!clean || clean.startsWith('.')) { + throw new ApiError(400, 'invalid_body', '"name" must be a usable folder name') + } + return { name: clean, path: `${root}/${clean}` } + } + + /** Counts and the newest change inside a project folder. */ + function summarise(nodes) { + let notes = 0 + let files = 0 + let updatedAt = 0 + const walk = (list) => { + for (const node of list) { + if (node.kind === 'folder') walk(node.children) + else { + if (node.kind === 'file') notes++ + else files++ + updatedAt = Math.max(updatedAt, node.updatedAt ?? 0) + } + } + } + walk(nodes) + return { notes, files, updatedAt } + } + + async function describe(folder) { + const { name, id, children } = folder + const path = `${root}/${id}` + const overviewName = OVERVIEW_CANDIDATES.find((candidate) => + children.some((c) => c.kind === 'file' && c.name === candidate), + ) + let meta = {} + if (overviewName) { + try { + meta = parseFrontmatter(await library.readText(`${path}/${overviewName}`)) + } catch { + // Unreadable overview: the project still lists, just without metadata. + } + } + const counts = summarise(children) + return { + name, + path, + status: STATUSES.includes(meta.status) ? meta.status : meta.status ? String(meta.status) : null, + summary: typeof meta.summary === 'string' ? meta.summary : meta.description ? String(meta.description) : '', + tags: asTags(meta.tags) ?? [], + created: meta.created ? String(meta.created) : null, + overview: overviewName ? `${path}/${overviewName}` : null, + log: children.some((c) => c.kind === 'file' && c.name === LOG) ? `${path}/${LOG}` : null, + ...counts, + } + } + + async function list({ status } = {}) { + if ((await library.exists(root)) !== 'directory') return [] + const tree = await library.tree(root) + const projects = [] + for (const node of tree) { + if (node.kind !== 'folder') continue + projects.push(await describe(node)) + } + const filtered = status ? projects.filter((p) => p.status === status) : projects + // Most recently touched first: that is the project someone is looking for. + filtered.sort((a, b) => b.updatedAt - a.updatedAt || a.name.localeCompare(b.name)) + return filtered + } + + async function get(name) { + const { name: clean, path } = folderFor(name) + if ((await library.exists(path)) !== 'directory') { + throw new ApiError(404, 'not_found', `no project called "${clean}" (looked in ${path})`) + } + return await describe({ name: clean, id: clean, children: await library.tree(path) }) + } + + /** Create a project folder with an Overview.md. Refuses to reuse a name. */ + async function create({ name, summary, status, tags, body, createdBy }) { + const { name: clean, path } = folderFor(name) + if (await library.exists(path)) { + throw new ApiError(409, 'conflict', `a project called "${clean}" already exists`) + } + const meta = [ + '---', + 'type: project', + `status: ${validateStatus(status) ?? 'active'}`, + `summary: ${JSON.stringify(oneLine(summary))}`, + `tags: [${(asTags(tags) ?? []).map((t) => JSON.stringify(t)).join(', ')}]`, + `created: ${dateStamp(now())}`, + ...(createdBy ? [`created_by: ${JSON.stringify(createdBy)}`] : []), + '---', + '', + ].join('\n') + const text = String(body ?? '').trim() + const content = `${meta}\n# ${clean}\n\n${oneLine(summary, 2000)}\n${text ? `\n${text}\n` : ''}` + await library.mkdir(path) + await store.writeNote(`${path}/${OVERVIEW}`, content.replace(/\n{3,}/g, '\n\n'), 'agent') + return await get(clean) + } + + /** + * Change a project's status, summary or tags in its Overview.md's front + * matter. Always Overview.md, never a README standing in for one: a pushed + * code project's README belongs to the code. A project without an + * Overview.md gets one, carrying over what its README said. + */ + async function update(name, { status, summary, tags }) { + const project = await get(name) + const patch = {} + if (status !== undefined) patch.status = validateStatus(status) + if (summary !== undefined) patch.summary = oneLine(summary) + if (tags !== undefined) patch.tags = asTags(tags) + if (!Object.keys(patch).length) { + throw new ApiError(400, 'invalid_body', 'nothing to change: give a status, summary or tags') + } + const overview = `${project.path}/${OVERVIEW}` + let current + let fields = patch + if (project.overview === overview) { + current = await library.readText(overview) + } else { + current = `# ${project.name}\n` + fields = { + type: 'project', + status: project.status ?? undefined, + summary: project.summary || undefined, + tags: project.tags.length ? project.tags : undefined, + ...patch, + } + } + await store.writeNote(overview, setFrontmatterFields(current, fields), 'agent') + return await get(project.name) + } + + /** Append a dated line to the project's Log.md, creating it if need be. */ + async function log(name, entry, actor) { + const project = await get(name) + const text = String(entry ?? '').trim() + if (!text) throw new ApiError(400, 'invalid_body', '"entry" cannot be empty') + if (text.length > 20_000) throw new ApiError(400, 'invalid_body', '"entry" is too long') + const path = `${project.path}/${LOG}` + const exists = (await library.exists(path)) === 'file' + // Continuation lines are indented so a multi-line entry stays one list item. + const [first, ...rest] = text.split(/\r?\n/) + const line = + `- **${timeStamp(now())}**${actor ? ` · ${actor}` : ''} — ${first}` + + rest.map((l) => `\n ${l}`).join('') + + '\n' + if (!exists) { + await library.writeText(path, `# Log — ${project.name}\n\n${line}`) + } else { + const current = await library.readText(path) + await library.appendText(path, current.endsWith('\n') ? line : `\n${line}`) + } + return { project: project.name, path } + } + + return { list, get, create, update, log, root } +} diff --git a/server/search.mjs b/server/search.mjs index ecf8dc4..5ce366c 100644 --- a/server/search.mjs +++ b/server/search.mjs @@ -1,13 +1,17 @@ -// BM25 ranking for GET /api/v1/search. +// BM25 ranking for GET /api/v1/search and MCP's `search` tool. // -// The browser assistant has a richer retrieval layer (src/ai/retrieval.ts) that -// can fuse in embeddings, but embeddings need an API key and the key lives in -// the user's browser, never on the server. So the API offers the lexical half: -// BM25 over note titles, paths, and content — no key, no network, no config. +// Lexical search over everything in the library: notes by title, path and +// content, and files by name and path — plus their contents where there is +// text to be had (plain-text formats and office documents; see extract.mjs). +// No key, no network, no config. // // The index is rebuilt whenever the library's contents change, detected by a -// cheap signature over every note's path and mtime, so repeated searches -// against an unchanged library cost nothing but the scoring pass. +// cheap signature over every entry's path, mtime and size, so repeated +// searches against an unchanged library cost nothing but the scoring pass. +// Extracted document text is cached per file version, so a rebuild after one +// new note doesn't re-read every spreadsheet in the library. + +import { extractText, isExtractable, MAX_EXTRACT_SOURCE_BYTES } from './extract.mjs' const K1 = 1.2 const B = 0.75 @@ -49,10 +53,16 @@ function snippetAround(content, terms) { return snippet } +function folderOf(id) { + return id.includes('/') ? id.slice(0, id.lastIndexOf('/')) : '' +} + export function createSearch(library) { let cache = null // { signature, docs, df, avgLength } + /** Extracted text by `id:mtime:size`, so an unchanged file is read once. */ + const extracted = new Map() - /** Flatten the tree the library API already builds into a list of note files. */ + /** Every note and file in the tree the library API builds. */ function flatten(nodes, out = []) { for (const node of nodes) { if (node.kind === 'folder') flatten(node.children, out) @@ -61,25 +71,48 @@ export function createSearch(library) { return out } + /** + * The searchable text of a file that isn't a note, or '' when it has none. + * Cached by version; exposed so MCP's `read` shares the work. + */ + async function fileText(node) { + if (!isExtractable(node.name) || node.size > MAX_EXTRACT_SOURCE_BYTES) return '' + const key = `${node.id}:${node.updatedAt}:${node.size}` + const hit = extracted.get(key) + if (hit !== undefined) return hit + let text = '' + try { + text = extractText(await library.readBuffer(node.id), node.name) ?? '' + } catch { + // Gone, or unreadable: nothing to index. + } + extracted.set(key, text) + return text + } + async function buildIndex() { const files = flatten(await library.tree('')) - const signature = files.map((f) => `${f.id}:${f.updatedAt}`).join('|') + const signature = files.map((f) => `${f.id}:${f.updatedAt}:${f.size ?? ''}`).join('|') if (cache && cache.signature === signature) return cache const docs = [] const df = new Map() let totalLength = 0 + const live = new Set() for (const file of files) { + const isNote = file.kind === 'file' let content = '' try { - content = await library.readText(file.id) + content = isNote ? await library.readText(file.id) : await fileText(file) } catch { // Disappeared between the walk and the read — skip it. continue } + if (!isNote) live.add(`${file.id}:${file.updatedAt}:${file.size}`) - const titleTokens = tokenize(`${file.title} ${file.id.replace(/\//g, ' ')}`) + const title = isNote ? file.title : file.name + const titleTokens = tokenize(`${title} ${file.id.replace(/[/._-]/g, ' ')}`) const tokens = [ ...Array.from({ length: TITLE_WEIGHT }, () => titleTokens).flat(), ...tokenize(content), @@ -89,10 +122,13 @@ export function createSearch(library) { for (const token of tokens) tf.set(token, (tf.get(token) ?? 0) + 1) for (const token of tf.keys()) df.set(token, (df.get(token) ?? 0) + 1) - docs.push({ file, content, tf, length: tokens.length }) + docs.push({ file, title, isNote, content, tf, length: tokens.length }) totalLength += tokens.length } + // Forget the text of files that were replaced or removed. + for (const key of extracted.keys()) if (!live.has(key)) extracted.delete(key) + cache = { signature, docs, @@ -103,10 +139,11 @@ export function createSearch(library) { } /** - * Rank notes against `query`. `folder` restricts the search to one subtree. - * Returns `{ path, title, folder, score, snippet, updatedAt }` objects. + * Rank notes and files against `query`. `folder` restricts the search to one + * subtree; `kind` to notes ('note') or other files ('file'). + * Returns `{ path, title, folder, kind, score, snippet, updatedAt }` objects. */ - async function search(query, { limit = 10, folder = '' } = {}) { + async function search(query, { limit = 10, folder = '', kind } = {}) { const terms = tokenize(query) if (!terms.length) return [] @@ -116,6 +153,8 @@ export function createSearch(library) { for (const doc of index.docs) { if (folder && !doc.file.id.startsWith(`${folder}/`)) continue + if (kind === 'note' && !doc.isNote) continue + if (kind === 'file' && doc.isNote) continue let score = 0 for (const term of terms) { @@ -129,21 +168,30 @@ export function createSearch(library) { } if (score <= 0) continue - scored.push({ + const result = { path: doc.file.id, - title: doc.file.title, - folder: doc.file.id.includes('/') - ? doc.file.id.slice(0, doc.file.id.lastIndexOf('/')) - : '', + title: doc.title, + folder: folderOf(doc.file.id), + kind: doc.isNote ? 'note' : 'file', score: Number(score.toFixed(4)), - snippet: snippetAround(doc.content, terms), + snippet: doc.content ? snippetAround(doc.content, terms) : '', updatedAt: doc.file.updatedAt, - }) + } + if (!doc.isNote) result.size = doc.file.size + scored.push(result) } scored.sort((a, b) => b.score - a.score) return scored.slice(0, limit) } - return { search } + /** The extracted text of one file, for MCP's `read` (null when there's none). */ + async function textOf(path) { + const stat = await library.stat(path) + const name = path.slice(path.lastIndexOf('/') + 1) + const text = await fileText({ id: path, name, size: stat.size, updatedAt: stat.lastModified }) + return text || null + } + + return { search, textOf } } diff --git a/server/settings-store.mjs b/server/settings-store.mjs deleted file mode 100644 index 4242857..0000000 --- a/server/settings-store.mjs +++ /dev/null @@ -1,94 +0,0 @@ -// The assistant's settings, kept for the whole server rather than per browser. -// -// Deckle's settings normally live in localStorage, which is the right home for -// them: they describe how *this browser* talks to a provider. The server -// library breaks that assumption — it exists so the same library is reachable -// from any device, and asking for the API key again on every one of them is a -// poor answer to "your notes are wherever you are". -// -// So a server library keeps one copy of them here, and every device that signs -// in reads it. Two properties matter, both of them about the key: -// -// * It lives *outside* the library. The notes folder travels — exports, -// backups, the machine API, the assistant's own read_note tool — and a -// provider key must not travel with it. See RESERVED_DIR in -// library-api.mjs, which is what keeps this folder invisible to all of -// that. -// * It is only served to a request that has already passed the password -// gate. With no DECKLE_PASSWORD the API is open by design, and the caller -// refuses to store keys at all in that case. -// -// It is stored in plain text, readable by this process and by whoever can read -// the volume. Encrypting it would only move the problem — the server would -// still have to hand the key to the browser, which is what actually calls the -// provider. - -import fs from 'node:fs/promises' -import path from 'node:path' -import { randomBytes } from 'node:crypto' - -const FILE = 'assistant.json' - -/** Generous enough for a long system prompt, small enough to stay a settings file. */ -export const MAX_SETTINGS_BYTES = 64 * 1024 - -/** Thrown for a body that isn't settings-shaped. */ -export class InvalidSettingsError extends Error {} - -export function createSettingsStore(stateDir) { - const file = path.join(stateDir, FILE) - - return { - /** The stored settings, or null when nothing has been saved yet. */ - async read() { - try { - return JSON.parse(await fs.readFile(file, 'utf8')) - } catch (err) { - // Missing is the normal first-run answer. A corrupt file is treated the - // same way: the device that saves next replaces it, which is a better - // outcome than refusing to load settings forever. - if (err.code !== 'ENOENT') { - console.warn('[deckle] could not read assistant settings:', err.message) - } - return null - } - }, - - /** - * Replace the stored settings. - * - * Written to a temporary name and renamed into place, so a device that - * disconnects mid-write can't leave a half-written file behind — the same - * pattern the library API uses for note uploads. - * - * What's inside is the client's business: this stores the object it is - * given rather than modelling every field, so a new setting needs no - * change here. It checks only that it *is* an object, and that it is small. - */ - async write(value) { - if (!value || typeof value !== 'object' || Array.isArray(value)) { - throw new InvalidSettingsError('settings must be an object') - } - // Measured compact, the way the request carried it. Measuring the - // indented copy refused settings that had arrived comfortably inside - // the same limit, once the indentation pushed them over. - if (Buffer.byteLength(JSON.stringify(value)) > MAX_SETTINGS_BYTES) { - throw new InvalidSettingsError('settings too large') - } - const payload = JSON.stringify(value, null, 2) - await fs.mkdir(stateDir, { recursive: true, mode: 0o700 }) - // Random, not just pid + time: two devices saving in the same - // millisecond would otherwise share a temp file, and one rename would - // then fail on a file the other had already moved. - const tmp = `${file}.${process.pid}.${Date.now()}.${randomBytes(6).toString('hex')}.tmp` - try { - // 0600 from the moment it exists: never briefly world-readable. - await fs.writeFile(tmp, payload, { mode: 0o600 }) - await fs.rename(tmp, file) - } catch (err) { - await fs.rm(tmp, { force: true }) - throw err - } - }, - } -} diff --git a/server/static.mjs b/server/static.mjs index 864077f..626cc39 100644 --- a/server/static.mjs +++ b/server/static.mjs @@ -39,14 +39,21 @@ const GZIP_MIN_BYTES = 1024 /** * What the page may load and run. * - * The point is `script-src 'self'`: the app keeps provider API keys in - * localStorage and can read and write the whole library, so a script that got - * into the page by any route — a note, a bookmark, a dependency bug — must not - * be able to run. Everything else is as open as the app genuinely needs: + * The point is `script-src 'self'`: the app can read and write the whole + * library, so a script that got into the page by any route — a note, a file an + * agent saved, a bookmark, a dependency bug — must not be able to run. + * Everything else is as open as the app genuinely needs: * - * - connect-src allows any http(s) origin, because the browser calls the AI - * provider directly and LM Studio lives at whatever URL the user typed — - * usually http://localhost or a LAN address. + * - connect-src is 'self'. The app talks to nothing but this server; it used + * to allow any origin so the browser could call an AI provider, and that + * assistant is gone. + * - blob: for images, media, frames and objects is how a stored file is + * previewed: the app reads the file through the library, wraps the bytes in + * a blob URL, and shows it in an ,