Handwriting Synthesis + Live Page Writer

A handwriting synthesis system, served as a streaming API, with a browser app on top that watches it write — line by line, onto a rendered notebook page, live.

You give it text. It writes it back to you, by hand.

---

What's in here

Two parts:

1. The model + serving backend — a recurrent neural network that generates pen strokes (not pixels) from text, trained on the IAM On-Line Handwriting Database, plus a from-scratch numpy reimplementation of the forward pass for fast, lightweight serving.
2. The live page-writer app — a browser frontend that streams every generated point over a WebSocket the moment it's produced and animates it directly onto a notebook-style page, packing lines and wrapping pages the same way a person filling out a page would.

---

How it works

The model

Each generation step predicts the next pen movement — a small (dx, dy, pen_up) triplet — rather than predicting pixels or characters directly. Stack enough of those steps and you get a stroke; stack enough strokes and you get a full sentence in something that actually looks handwritten, wobble and inconsistency included. A temperature setting controls how clean versus messy the output comes out — low temperature gives tight, controlled strokes; higher temperature gives looser, more human-feeling ones.

Training used the IAM On-Line Handwriting Database, plus additional fine-tuning on collected data.

Serving

The model is small, but importing the usual deep learning libraries in a serverless environment is the real latency cost — five to ten seconds just to load the library before any inference happens. So the entire forward pass (LSTM cells, attention window, mixture sampling) was rewritten by hand in pure numpy and validated against the original until outputs matched to within 2×10⁻⁵ absolute error. That took cold starts from several seconds down to under half a second, and shrank the deployed image from ~334MB to ~98MB.

The live page-writer app

Calling the API and getting back a JSON blob of coordinates isn't very satisfying — the interesting part is watching it happen. The app:

1. Takes your typed paragraph and splits it into line-sized chunks, the same way you'd naturally break a paragraph into lines.
2. Sends each chunk to the model and streams every generated point back over a WebSocket the instant it's produced (not after the whole chunk finishes).
3. Reconstructs pen strokes from the raw point stream (cumulative sum + baseline correction, since raw output tends to drift slightly over long sequences).
4. Animates each chunk directly onto a rendered notebook page — ruled lines, red margin line — scaling it to sit naturally on the line, then packs it next to the previous chunk, wrapping to a new line or a new page once the current one fills up.
5. Mirrors the same chunks in the input panel, color-coded, with each chunk's highlight sweeping in right as it's actually being written onto the page.

The model gives you raw ink. Getting that ink to look like it's actually sitting on a page someone wrote on — consistent line heights, straight baselines, natural line wrapping — was its own separate engineering problem.

---

Live demo / API

curl -X POST https://handwriting-api-687921010800.europe-central2.run.app/generate/stream \
  -H "Content-Type: application/json" \
  -d '{"text": "hello world", "temperature": 0.5}'

Returns NDJSON — one {"type": "point", "x": ..., "y": ..., "pen_up": 0|1} per pen step, streamed live.

Endpoint	Description
POST /generate/stream	Streamed NDJSON points (recommended)
POST /generate	Full JSON response
GET /vocab	List of drawable characters (77 total)
GET /health	Liveness check + backend info

Polish characters (ą, ę, ó, …) are automatically transliterated to ASCII. Anything else outside the vocab returns 400.

---

Running the live page-writer app locally

handwriting_app/
├── backend/
│   ├── server.py          # FastAPI app + WebSocket endpoint (/ws/write)
│   ├── model_np.py        # numpy forward pass
│   ├── text_layout.py     # chunking logic
│   ├── weights.npz        # model weights
│   ├── stoi.json          # char -> index map
│   └── std.json           # normalization std
└── frontend/
    ├── index.html         # textarea + canvas UI
    └── app.js              # WebSocket client + page-packing + animation

1. Place weights.npz, stoi.json, std.json in backend/.

2. cd backend
   pip install -r requirements.txt
   uvicorn server:app --reload --port 8000

3. Open http://localhost:8000 — the backend also serves the frontend.
4. Type text, click Write on page, watch it get handwritten live.

---

Architecture reference

Component	Detail
Model	RNN handwriting synthesis, stacked LSTM layers, soft attention window, mixture density output
Training	IAM On-Line Handwriting DB, fine-tuned on additional collected data
Serving	Pure numpy (no torch in production), validated to <2×10⁻⁵ error against the original
Deployment	Google Cloud Run, scale-to-zero
Image size	~98MB (vs ~334MB with CPU torch)
Cold start	~0.1–0.5s (vs 5–10s with torch import)
Live app	FastAPI WebSocket backend + canvas-based frontend, streaming point-by-point

---

Repo layout

model.py              # model definition (training)
model_np.py           # numpy forward pass (serving)
app_np.py             # API server (deployed)
train.py              # training loop
generate.py            # sample from a checkpoint
export_weights.py     # checkpoint -> weights.npz for serving
data.py               # dataset + dataloader

handwriting_app/      # live page-writer web app
  backend/             #   FastAPI + WebSocket streaming
  frontend/            #   canvas page renderer + animation

models/                # checkpoints (gitignored, on Hugging Face)
scripts/               # dev / demo tools
docs/                  # images, deployment notes, dev logs

---

References

• IAM On-Line Handwriting Database