Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
node_modules/
dist/
/cases/
*.local
.DS_Store
90 changes: 74 additions & 16 deletions PLAN.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Inkstone(砚)— ChatGPT 对话导出
# Inkstone(砚)— ChatGPT / Claude 对话导出

> 名取「砚」:把 GPT 的原始输出研磨成能写进笔记的墨;石对石(砚 ↔ Obsidian)。

> 2026-07-06 brainstorm 定稿。形态:**油猴脚本(Tampermonkey userscript)**,TypeScript 编写,Bun + Vite + vite-plugin-monkey 构建,产物为单个 `.user.js`。

## 目标

- 在 chatgpt.com 页内一键**批量导出全部对话**为 Obsidian 等笔记软件友好的 Markdown
- 在 chatgpt.com / claude.ai 页内一键导出对话为 Obsidian 等笔记软件友好的 Markdown
(两站均支持当前、选择、批量与增量导出;Claude 使用更保守的独立风控)
- 高保真:公式、引用链接、代码、图片/附件、思维链、Canvas 不丢不乱
- 增量同步:重跑只导出有变化的对话
- 全程本地处理,不经任何第三方服务
Expand Down Expand Up @@ -41,7 +42,7 @@
- `thoughts`(推理模型思维链)→ 折叠 callout,默认带、可关
- Canvas textdoc → MVP 整块嵌入,后续做 patch 重放还原终稿
- **未知类型 → 原始 JSON 塞进折叠 callout,永不静默丢内容**
6. **附件管道**:`file-service://` 与 `sediment://` 指针 → files 接口下载 → 笔记同目录下的附件子文件夹(默认 `conversations/attachments/`)→ 链接改写(wikilink `![[...]]` / 标准相对链接,可配;链接相对 .md 严格成立);下载失败留占位说明并在完成文案里报数。笔记/附件子文件夹名均可设置(可 `a/b` 嵌套、可留空;`sanitizeSubdir` 逐段净化防逃逸),油猴设置面板与 CLI `--notes-dir`/`--attachments-dir` 同规则。⚠️ 直写 vault 时注意 macOS 大小写不敏感:目录名撞上 vault 已有目录(如 `Attachments`)会直接写进去——2026-07-10 排查的"附件丢失"即此叠加 fast-note-sync 云预览删本地所致,非代码 bug。
6. **附件管道**:`file-service://` 与 `sediment://` 指针 → files 接口下载 → 笔记同目录下的附件子文件夹(默认 `conversations/attachments/`)→ 链接改写(wikilink `![[...]]` / 标准相对链接,可配;标准 Markdown 路径相对当前 `.md`,Obsidian Wikilink 的目录路径相对 vault 根);下载失败留占位说明并在完成文案里报数。笔记/附件子文件夹名均可设置(可 `a/b` 嵌套、可留空;`sanitizeSubdir` 逐段净化防逃逸),油猴设置面板与 CLI `--notes-dir`/`--attachments-dir` 同规则。⚠️ 直写 vault 时注意 macOS 大小写不敏感:目录名撞上 vault 已有目录(如 `Attachments`)会直接写进去——2026-07-10 排查的"附件丢失"即此叠加 fast-note-sync 云预览删本地所致,非代码 bug。
7. **Frontmatter**:title、chat_id、url(`chatgpt.com/c/<id>`)、created、updated、model(实际生成消息的 model_slug,最后一条为准;多模型另列 models)、tags —— 配 Dataview 即全库索引。
8. **文件名**:`标题-短id.md` 防重名(无空格/波浪线,非法字符与空白归一为 `-`),id 保证增量覆盖稳定。

Expand All @@ -57,30 +58,53 @@

## 项目结构

依赖方向单向收敛:`sites/* → core/*`。core 不认识任何站点,sites 不认识编排。

```
inkstone/
package.json # bun(scripts: dev/build/test/typecheck/offline)
vite.config.ts # vite-plugin-monkey
vite.config.ts # vite-plugin-monkey(match: chatgpt.com / claude.ai)
cli/
export.ts # 官方导出 zip → vault 的离线 CLI(bun 直跑,P4.2)
docs/
claude-adapter-feasibility.md # Claude 移植可行性评估(P5 的依据)
claude-probe.js # 贴进 claude.ai 控制台的结构 / 限流探针
src/
main.ts # UI 注入 + 流程编排 + 输出 Sink(zip/直写)
api.ts # backend-api 客户端(token/列表/全文/附件)
convert/
main.ts # 流程编排 + 输出 Sink(zip/直写)——站点无关
ui.ts # 浮动面板 + 进度——站点无关,锚点与配色问 adapter
state.ts # 增量水位线(按站点分表)+ 设置持久化
core/ # ← 站点无关内核
ir.ts # 中间表示:IRConversation / IRTurn / IRBlock
render.ts # IR → Markdown(轮次标题、callout、围栏、frontmatter)
fetcher.ts # 限速 / 退避 / 并发池 / 取消 / 限流观测(每站点一个实例)
batch-safety.ts # 站点级并发/重试策略 + 429/请求预算/失败率熔断
sites/
types.ts # SiteAdapter 契约(取数 + 转换 + 界面锚点 + 批量能力)
index.ts # 按 location.host 分派
chatgpt/
index.ts # adapter 实装
convert.ts # backend-api JSON → IR(content_type 分发、canmore 语义)
claude/
index.ts # adapter 实装 + Claude 专属批量风控策略
api.ts # 内部 API 客户端 + 保守限流参数 + 有界分页器
types.ts # 从宽的字段类型,[待测] 处已标注
convert.ts # 内部 API JSON → IR(块级分发、主线回溯、附件两处来源)
artifacts.ts # artifact create/update/rewrite 折叠成终稿
convert/ # ChatGPT 专属转换 + 通用文本工具(历史路径,测试直接引用)
markdown.ts # 兼容壳:conversationToIR + renderConversation
linearize.ts # mapping 树 → 线性消息
markdown.ts # 消息 → md(content_type 分发;assetLink 链接风格)
math.ts # 公式定界符转换(代码块感知)
citations.ts # 引用标记还原(matched_text 通道 + 官方导出 token/顺序配对通道)
headings.ts # 标题降级 / 剥离为加粗
canvas.ts # Canvas textdoc patch 重放
citations.ts # 私有区引用标记还原
math.ts # 公式定界符转换(代码块感知)—— 两站点共用
headings.ts # 标题降级 / 剥离为加粗 —— 两站点共用
codeaware.ts # 代码块感知的文本变换基础设施 —— 两站点共用
api.ts # ChatGPT backend-api 客户端(端点与字段,节奏交给 core)
output/
zip.ts # fflate 打包
fsaccess.ts # File System Access 直写 vault(句柄存 IndexedDB)
state.ts # 增量水位线 + 设置持久化
ui.ts # 浮动面板 + 进度
test/
fixtures/*.json # 真实对话 JSON(脱敏
*.test.ts # bun test
fixtures/*.json # 对话 JSON(ChatGPT 真实脱敏 / Claude 合成
*.test.ts # bun test 回归套件
```

## 阶段
Expand All @@ -94,7 +118,41 @@ inkstone/
- **P4 脱离油猴(用户明确期望)**:转换层(convert/)零浏览器依赖、api.ts 只依赖 fetch,天然可复用到:
1. **MV3 浏览器扩展**——同一套 src,加 manifest + content script 打包目标(vite 多入口);不再依赖 Tampermonkey,可上架商店(用户拍板:等功能完善后再做/上架)
2. **官方导出 zip 的离线 CLI** ✅(2026-07-10,`bun run offline <zip|目录> [-o 输出]`)——完全不碰 backend-api,零限流风险;432 对话 + 248 附件 ~8s 转完;支持 `--link-style/--heading-mode/--no-thoughts/--no-assets`
3. Claude/Gemini adapter
3. Claude adapter ✅ 骨架(2026-08-28,见下)/ Gemini 待做

- **P5 多站点架构 + Claude adapter**(2026-08-28):见 `docs/claude-adapter-feasibility.md`(可行性评估)
- **架构**:引入站点无关的中间表示(IR)与适配器契约,依赖方向变成
`sites/{chatgpt,claude}/ → core/{ir,render,fetcher}`。编排(main.ts)与界面(ui.ts)
不再认识任何一家的端点、字段或 DOM;新增站点 = 新增一个 adapter,不改编排。
重构以「现有 102 个测试断言一字不改地通过」为验收标准,行为逐字节等价。
- **两侧数据模型的结构性差异**:ChatGPT 一条消息一种 content_type(消息级分发),
Claude 一条消息多个 typed block 按序交错(块级分发)。主线定位则同构——
两边都是「叶子 + parent 链回溯后反转」。
- **Claude 侧的脏活更少**:Canvas 的正则 patch 重放(150 行)与私有区 Unicode 引用
还原(178 行)在 Claude 都不需要——artifact 的 update 是字面量 `old_str`→`new_str`,
引用是结构化数组。artifact 折叠约 40 行。
- **批量分阶段开放**:首版因限流画像未知而只开放当前对话;2026-08-29 恢复选择、
全部与增量导出,但不照搬 ChatGPT 参数。Claude 固定单并发、不做失败项整批二次重试;
一次带 `Retry-After` 的 429、累计 3 次 429、单批 1000 次 HTTP 尝试,或至少 5 条失败
且失败率超过 25%,都会保护性中止。成功条目才推进水位线,未尝试部分由下次增量补齐。
列表分页另设 250 请求 / 10000 条硬上限,并检测重复页与缺失 uuid,防接口漂移后空转。
- **Claude 限流起步参数**(保守,待实测调整):间距 1500ms(ChatGPT 侧的两倍慢)、
上限 8000ms、每 40 请求歇 30s、首次失败后最多重试 1 次。每个站点持有独立的 fetcher 实例,
一边的限流不拖累另一边。吃到 429 时导出完成文案会报出次数、被推大的间距与
服务端要求的最长等待——未知站点的节奏只能靠实测看清,先让它可见再谈调参。
- **待实测**:`docs/claude-probe.js` 可直接粘进 claude.ai 控制台,打印字段骨架
(不打印对话内容)并做一次 ≤8 请求的保守限流试探。清单见可行性文档第五节:
分页是否生效、thinking 字段名、citations 挂载形态、普通附件地址能否直取字节、
公式定界符、Projects 字段名。FAB 锚点已在 2026-08-28 的真实会话页确认:
顶栏动作为 `[data-testid="wiggle-controls-actions-group"]`(需锚定整个组的左边界,
不能只锚定 Share,否则会盖住 Files),输入框为
`[data-testid="chat-input"][contenteditable="true"]`。同一轮实测还确认 Claude 的
图片 `preview_url` 可能把上传 PNG 转码为 WebP,附件落盘扩展名必须服从响应 MIME。
生成文件卡片则不在消息的 `files[]`:`present_files` 给出展示路径,实际文件需先从
`.../conversations/{id}/wiggle/list-files` 取清单,再经 `wiggle/download-file` 下载。
默认 Markdown 只链接最终沙箱文件,不再重复嵌入每次 `create_file` 的完整中间版本。
- **未做**:Claude 的行内引用锚定(citations 的字符级定位字段未实测,首版只把
来源汇总进文末 Sources,正文一字不动);Claude 官方导出 zip 的离线 CLI 通道。

## 实战经验(2026-07-08 E2E,344 对话实测)

Expand Down
42 changes: 34 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

# Inkstone · 「砚」

**Batch-export your entire chatgpt.com history to Obsidian-friendly Markdown — one click, in the page, fully local.**
**Export your chatgpt.com and claude.ai conversations to Obsidian-friendly Markdown — one click, in the page, fully local.**

*An inkstone grinds raw pigment into ink for writing. Inkstone helps you grind GPT's raw output into ink for your notes.*
*An inkstone grinds raw pigment into ink for writing. Inkstone helps you grind raw model output into ink for your notes.*

[![release](https://img.shields.io/github/v/release/ZhenHuangLab/inkstone)](https://github.com/ZhenHuangLab/inkstone/releases/latest)
[![downloads](https://img.shields.io/github/downloads/ZhenHuangLab/inkstone/total)](https://github.com/ZhenHuangLab/inkstone/releases)
Expand All @@ -20,7 +20,7 @@
&nbsp;→&nbsp;
<a href="https://github.com/ZhenHuangLab/inkstone/releases/latest/download/inkstone.user.js"><b>② Install Inkstone</b></a>
&nbsp;→&nbsp;
<b>③ Open chatgpt.com, hit ⤓</b>
<b>③ Open chatgpt.com or claude.ai, hit ⤓</b>
</p>

---
Expand All @@ -29,7 +29,25 @@

Your ChatGPT history holds real work, but getting it into a vault is painful. The official export is a raw JSON dump: tool and system messages are stripped (Canvas and code-interpreter output simply aren't there), some attachments have already expired server-side, math arrives in `\( \)` delimiters Obsidian won't render, and web-search citations turn into private-use Unicode garbage. Copy-pasting by hand doesn't scale past ten conversations, let alone a thousand.

Inkstone runs inside chatgpt.com and fetches conversations through the same backend API the app itself uses, then converts everything locally in your browser — nothing ever leaves the page. The result is Markdown that reads natively in Obsidian: real headings per turn, `$` / `$$` math, resolved citations, downloaded images, clean frontmatter.
Inkstone runs inside the page and fetches conversations through the same backend API the app itself uses, then converts everything locally in your browser — nothing ever leaves the page. The result is Markdown that reads natively in Obsidian: real headings per turn, `$` / `$$` math, resolved citations, downloaded images, clean frontmatter.

## Supported sites

| | ChatGPT | Claude |
| --- | --- | --- |
| Export current conversation | ✅ | ✅ |
| Batch / export-all | ✅ | ✅ conservative safeguards |
| Incremental sync | ✅ | ✅ |
| Rich documents | Canvas patch replay | Artifact fold-up to final version |
| Thoughts / tool traces | ✅ opt-in | ✅ opt-in |
| Attachments | images and files downloaded | images, documents, and generated files downloaded; text extractions inlined |

**Claude batch export uses a separate, deliberately conservative policy.** Its rate-limit
profile is still not backed by a large real-world sample, so ChatGPT's settings are not
reused: one worker, spacing from 1500 ms, a 30-second rest every 40 requests, and no second
pass over failed items. One global `Retry-After` signal, three 429s, 1000 HTTP attempts, or
an abnormal failure ratio stops the batch. Successful conversations are still written;
unfinished ones do not advance the watermark and are picked up by the next incremental run.

## Screenshots

Expand Down Expand Up @@ -97,11 +115,12 @@ bun run build # → dist/inkstone.user.js, drag it into Tampermonkey

## Usage

Open chatgpt.com (logged in) → click the **⤓ button left of the Share button** in the top bar → pick **Markdown zip** or **raw JSON zip** → unzip into your Obsidian vault.
Open chatgpt.com or claude.ai (logged in) → click the **⤓ button** in the top bar → pick **Markdown zip** or **raw JSON zip** → unzip into your Obsidian vault.

- The button position is switchable (panel → advanced settings): next to Share in the top bar, or a glass button beside the input box
- The UI follows ChatGPT's appearance settings automatically (light/dark + accent color)
- The UI follows the host page's light/dark appearance; ChatGPT follows its selected accent, while Claude uses its brand orange
- Exports are cancelable; a single failed conversation never aborts the run — failures are summarized in `_failures.json`
- Claude batches run with one worker; after a protective abort, wait as instructed instead of restarting immediately

## Offline CLI

Expand All @@ -126,14 +145,21 @@ bun run typecheck
bun run build
```

Note for claude.ai: its CSP may block the dev-server script, so verify Claude-side changes
against a real `bun run build` artifact loaded into Tampermonkey rather than `bun run dev`.

Architecture: `src/core/` is site-agnostic (IR, renderer, throttled fetcher) and
`src/sites/<site>/` holds everything that knows one provider's endpoints, fields and DOM.
Adding a site means adding an adapter, not touching the orchestration. See `PLAN.md` § P5.

## Roadmap

MV3 browser extension (no Tampermonkey, store release) and Claude / Gemini support. Already done: incremental sync, direct-write to an Obsidian vault, settings panel, Canvas patch replay, and the offline CLI. Details in [PLAN.md](./PLAN.md) (Chinese).
An MV3 browser extension (no Tampermonkey, store release) and Gemini support. Already done: the multi-site adapter architecture, Claude batch and incremental export, direct-write to an Obsidian vault, settings panel, Canvas patch replay, Artifact fold-up, and the offline CLI. Details in [PLAN.md](./PLAN.md) (Chinese).

## License

[GPL-3.0](./LICENSE)

## Related Links

[LINUX DO](https://linux.do)
[LINUX DO](https://linux.do)
Loading