MAHE · Distributed AI Training Framework

Turn idle computers in dorms and offices into an AI training cluster.

MAHE organizes ordinary computers on a LAN into a training cluster using a parameter-server architecture — no GPU budget required, starting from 2-core CPUs. The master only aggregates parameters and schedules work; workers train on local shards with their own compute. The core is C++17 with hand-written SIMD assembly, Python drives training and orchestration, and a self-contained web terminal (HTML/CSS/JS served by Python) provides the control console.

If this project helps you, a star is appreciated.

Why MAHE

Start without GPUs. The hard part of distributed training is usually not FLOPs but the engineering cost of coordinating machines. MAHE needs 2 CPU cores and a LAN — retired office desktops become usable compute.
Friendly to weak networks. Workers train K local steps and upload only the parameter delta Δ; the master merges deltas weighted by shard size. Communication drops by 1–2 orders of magnitude versus per-step gradient sync.
No platform or compiler lock-in. One build script covers Windows (MinGW/MSVC), Linux and macOS. x86-64 uses AVX2, Apple Silicon/ARM64 uses NEON, and missing assembler toolchains fall back to scalar code automatically.
Distribute as one installer. The Python engine, C++ kernels and web terminal are packed into a single installer that creates desktop shortcuts — end users install no Python, numpy or JDK.

Key features

Capability	Description
Parameter-server training	Master owns global params and optimizer state; workers hold replicas and data shards
Compute-weighted data parallelism	Workers report a micro-benchmark score at join time; shard size ∝ `score / Σscore`, so fast nodes don't wait on slow ones
Local multi-step aggregation	FedAvg-style: K local steps per round, only Δ is uploaded — big communication savings
Auto-discovery and direct connect	Master broadcasts over UDP periodically; use `--master <IP>` across subnets
Fault-tolerant scheduling	Heartbeat watchdog reclaims dead nodes' shards; each round waits for `--min-workers`
SIMD kernels	AVX2 (x86-64) / NEON (aarch64) for delta reduction and SGD update, dispatched at runtime
Checkpoints	`.npz` written at end/interruption, resumable
Clean start	No demo workers and no auto-training by default; developers use `--test` / `-t`
Beginner training page	Pick dataset, training preset and model location (native file dialog), then start
Training-volume estimate	Live steps / samples per round / equivalent epochs, with under-/over-training hints
Dataset catalog	Built-in datasets plus an optional community catalog (`--community` / `MAHE_COMMUNITY_URL`)
Node group management	Remove any node from the group, or clear all; removed workers receive `BYE` and exit
Observable UI	Browser web terminal on `127.0.0.1:47830`; boot subtitles finish before training init; logs stream as `[KIND] message`
LAN scan & deploy	Discover LAN hosts, filter by remote-management ports, then pull source from GitHub and build workers
One-click installer	`build.py --bundle` produces a single-file installer that creates desktop shortcuts
Cluster packaging	`--build-cluster` generates master/worker bundles by device count and OS
Assistant (reserved)	DeepSeek protocol and UI entry are wired; no network request without an API key

Quick start

Build, then launch the web terminal:

python3 -m pip install numpy pybind11

python3 build.py --test        # build and run C++ tests (kernel/protocol/scheduler/handshake)
python3 build.py               # build the Python extension

PYTHONPATH=python python3 -m mahe.demo_train            # clean start (default)

The default is a clean environment: master + web terminal only, no nodes and no automatic training. Choose a dataset and model location on the Training page and press Start; add nodes later from the Deploy page.

For developer self-tests use --test / -t, which spawns 2 local workers and starts training immediately:

PYTHONPATH=python python3 -m mahe.demo_train --test --rounds 20

The browser opens http://127.0.0.1:47830/. The boot animation runs first; training modules are initialized only after the page sends ui_ready. Initialization logs appear in the left info panel as [INIT] .... Use --no-browser to suppress auto-open and --no-ui for headless servers.

Real terminal output:

INFO mahe.master: round=1  workers=2 eval_loss=0.2816 eval_acc=0.9080
INFO mahe.master: round=2  workers=2 eval_loss=0.2356 eval_acc=0.9140
...

A checkpoint is written to checkpoints/mahe_model.npz (or ~/.mahe/checkpoints/ in the packaged app). On Windows, replace PYTHONPATH=python with set PYTHONPATH=%CD%\python; everything else is identical.

Full dependency, compiler and troubleshooting notes are in the user guide (Chinese).

Architecture

Web terminal (browser, separate process, crash-isolated)
        │  127.0.0.1:47830 · HTTP + SSE + JSON
Python control plane — master  (aggregation / eval / checkpoints / UI events)
        │  pybind11
C++17 core — transport · discovery · scheduler · kernel
        │  TCP 47820 · 16-byte frame header (magic MAHE / v1)
   ┌────┴─────┐
 Worker A   Worker N   (Python + C++ extension)

How it works

Join. Workers listen for the master's UDP broadcast (47821), then open a TCP connection carrying their compute score and token. The master verifies the token and assigns a node_id.
Shard. The scheduler splits this round's data into weighted shards by compute score and dispatches them to workers.
Local training. Each worker runs K steps (--k-steps) on its shard and computes only the parameter delta Δ.
Aggregate. The master averages Δ weighted by shard sample counts and applies the global update with a SIMD kernel.
Recover. Nodes whose heartbeat times out are marked dead; their shards return to the pending pool for the next round.

Communication per round (same model, same round):

Strategy	Upload per round	Relative traffic
Per-step gradient sync	K × params	baseline
Local K steps + Δ (MAHE)	1 × params	~1/K

SIMD kernels

Kernel	Platform	Implementation
`mahe_reduce_add_f32`	x86-64	NASM / AVX2, 8×f32 per instruction
`mahe_reduce_add_f32`	aarch64	GAS / NEON, 16×f32 per iteration (4×q registers)
`mahe_sgd_update_f32`	x86-64	AVX2 FMA
`mahe_sgd_update_f32`	aarch64	`dup + fneg + fmla` for `param -= lr*grad`
all	any	scalar C++ fallback, element-wise identical

Dispatch is automatic:

python -c "import mahe; print(mahe.mahe_core.kernel_impl())"

prints avx2, neon or scalar. Use --no-asm to force the scalar path.

Cross-platform builds

Platform	Compiler	Assembly	Artifact
Windows x64	MinGW-w64 (recommended) / MSVC	NASM `-f win64`	`mahe_core.cp3XX-win_amd64.pyd`
Linux x86-64	g++ / clang++	NASM `-f elf64`	`mahe_core.cpython-3XX-x86_64-linux-gnu.so`
Linux aarch64	g++ / clang++	NEON (assembled `.S`)	`mahe_core.cpython-3XX-aarch64-linux-gnu.so`
macOS Apple Silicon	clang++	NEON (assembled `.S`)	`mahe_core.cpython-3XX-darwin.so`
macOS Intel	clang++	NASM	same

python build.py                     # core + Python extension
python build.py --test              # build and run C++ tests
python build.py --bundle            # one-click installer (engine + web UI)
python build.py --no-asm            # force scalar kernels
python build.py --compiler g++-13   # pick a compiler
python build.py --clean             # remove artifacts

On Windows, keep Python and the compiler both 64-bit. MinGW setup is described in the user guide · Windows.

LAN cluster

Master (listens on TCP 47820 / UDP 47821):

PYTHONPATH=python python3 -m mahe.demo_train --role master --min-workers 2 --rounds 30

Worker (same subnet, discovered via UDP broadcast):

PYTHONPATH=python python3 -m mahe.demo_train --role worker

Across subnets, or when broadcast is blocked:

PYTHONPATH=python python3 -m mahe.demo_train --role worker --master 192.168.1.10

The demo token mahe-demo-token is hardcoded in demo_train.py. Replace it for any real deployment; connections with a wrong token are rejected.

Generate deployment bundles by device count and OS:

python3 build.py --build-cluster \
  --systems "linux-x86_64,linux-aarch64,windows-x86_64,mac-arm64" \
  --master-index 0

Output lands in build/cluster/device_XX_<role>_<os>-<arch>/, each containing sources, build scripts, launch scripts and device_info.json. Devices matching the build host get a prebuilt extension; others build on first launch on the target machine.

Web console

The web terminal is served by the Python control plane on 127.0.0.1:47830 and opens automatically (HTML/CSS/JS, no external dependencies).

python3 -m mahe.demo_train                    # clean start + console (default)
python3 -m mahe.demo_train --test             # self-test: 2 local workers + auto-train
python3 -m mahe.demo_train --no-browser       # do not open a browser
python3 -m mahe.demo_train --no-ui            # headless server

Flow:

The browser plays the boot subtitles.
On completion it posts ui_ready; only then does the backend initialize the C++ core, scheduler and training modules.
Initialization messages appear in the left info panel as [INIT] …, [WARN] …, [ERROR] ….
After initialization, a deployment panel offers LAN auto-deploy, manual scripts, or "later".

The layout is an intentionally restrained eDEX-style terminal: dark background, hairline grid, monospace type and a single teal accent. Views: dashboard, training, cluster, deploy, assistant and logs. Closing the browser does not affect training; reopening resyncs a full snapshot.

Training page (beginner flow)

Three steps:

Pick data — the dropdown lists built-in and community datasets with dimensions, classes, sample count, a one-line description and recommended parameters. "Refresh" re-fetches the catalog.
Model location — "Browse…" opens a native save dialog on the engine host (PowerShell on Windows, osascript on macOS, zenity/kdialog on Linux); typing a path also works, and missing directories are created.
Volume and parameters — Quick / Standard / Thorough presets fill rounds / k_steps / batch; you can override them. The live estimate shows total steps · samples per round · equivalent epochs with an under-/over-training hint.

Model files are .npz (global parameter vector), saved every 5 rounds and at the end.

Definitions: rounds = parameter aggregation rounds; k_steps = mini-batches each worker runs per round; equivalent epochs = global samples ÷ dataset size (1–50 is a healthy range).

Datasets and the community catalog

Three reproducible datasets ship in the box (three-class spiral, three-class Gaussians, four-class rings). All nodes regenerate the same data locally from the dataset descriptor, so no data files cross the network. The community catalog is optional: set --community <base> or MAHE_COMMUNITY_URL and the engine tries <base>/catalog.json; failures silently fall back to the built-in catalog. Entries must reference a locally registered generator — unknown generators are skipped because the engine never executes untrusted generator code.

Node group management

Each node row in the cluster view has a ✕: confirming removes that node from the group. The master sends BYE and then drops it from the scheduler; a worker built from this version exits on BYE. "Clear nodes" removes them all at once. Removed nodes take part in no further sharding or aggregation.

Assistant (reserved)

The Assistant view is wired to the same command channel. Set DEEPSEEK_API_KEY and restart the engine to enable it; without a key the UI shows "disabled" and makes no network requests. MAHE_DEEPSEEK_MODEL / MAHE_DEEPSEEK_BASE override the model and endpoint.

Sub-node deployment

Scan: the master probes its private /24 (or a user-supplied CIDR), resolves hostnames and checks SSH/SMB/RDP/WinRM/AFP/VNC ports. Results show device name, type and whether auto-deploy is possible.
Deploy: for SSH-capable hosts, the master runs a generated bootstrap script that shallow-clones the configured GitHub repository/branch, installs numpy/pybind11, runs build.py, and starts a worker in the background.
Manual: download the POSIX or Windows script and run it on the target machine. Automatic deployment uses SSH key auth (BatchMode); passwords are never stored.

One-click installer

Pack the frozen Python engine and the web terminal into a single-file installer:

python -m pip install pyinstaller
python build.py --bundle

Platform	Artifact
Windows	`build/installer/MAHE-Setup.exe`
macOS	`build/installer/MAHE-Setup`
Linux	`build/installer/MAHE-Setup`

The installer runs per-user (no admin rights) and always creates a desktop shortcut; on Windows it also creates a Start Menu entry and an uninstall registry item. Launching the shortcut starts mahe-engine (clean start by default; set MAHE_ENGINE_ROLE=worker to run it as a sub-node) and opens the web terminal.

PyInstaller only produces artifacts for the host platform, so build each target on its own OS. Size is roughly 100–130 MB. Public macOS distribution additionally requires signing and notarization.

Project layout

core/
  include/mahe/         protocol, transport, discovery, scheduler, kernel headers
  src/                  protocol / transport / discovery / scheduler / kernel / node
  asm/reduce_avx2.asm   AVX2 delta reduction and SGD update (NASM, x86-64)
  asm/reduce_neon.S     NEON delta reduction and SGD update (GAS, aarch64)
  bindings/pybind.cpp   Python extension (zero-copy NumPy buffers)
  tests/test_core.cpp   C++ unit tests (including end-to-end handshake)
python/mahe/
  model.py              NumPy MLP (CPU-only reference backend)
  data.py               dataset registry: descriptors, generators, volume estimate
  community.py          community dataset catalog (configurable, falls back to built-ins)
  worker_node.py        local multi-step training, Δ upload
  master_node.py        parameter server: weighted aggregation, eval, checkpoints, node removal
  web_server.py         HTTP + SSE + command channel
  webui/                static web terminal (HTML/CSS/JS)
  lan_scan.py           LAN device discovery and deployability filter
  deployer.py           GitHub source pull + remote build/deploy scripts
  platform_dialogs.py   cross-platform native file dialog (model location)
  agent.py              DeepSeek assistant protocol (reserved)
  demo_train.py         default / --test / worker entry point
packaging/
  engine_entry.py       PyInstaller entry for mahe-engine
  installer_entry.py    installer: copy files + desktop shortcuts
docs/USER_GUIDE.md      user guide (Chinese)
build.py                cross-platform build / test / cluster bundling / installer

Testing

python build.py --test

Coverage:

SIMD kernels vs scalar implementation, element-wise comparison (1e-6 tolerance)
Frame codec: length, magic, header parsing
Weighted sharding 3:1 → 750/250
Task dispatch, completion, node removal
Master/worker handshake, token rejection, bidirectional messages, graceful shutdown
Web terminal: static assets, ui_ready handshake and /api/status snapshot

End-to-end: --test really spawns 2 worker subprocesses, trains the requested rounds and writes a checkpoint. python -m mahe.demo_train --test --no-ui --dataset builtin:rings4 verifies that datasets with different dimensions/classes propagate correctly from master to workers.

Roadmap

v0.1 (current): C++ core + AVX2/NEON kernels + Python parameter-server training + web terminal + LAN deployment flow + beginner training page and dataset catalog
v0.2: real multi-machine LAN validation, checkpoint resume, int8 gradient compression
v0.3: DeepSeek assistant enabled, PyTorch backend, richer web console views
v0.4: TLS, web monitoring endpoint

Known limitations

The training backend is a NumPy MLP (CPU); PyTorch is not wired up yet.
int8 gradient compression is designed but not implemented.
Multi-machine validation is mostly local multi-process; real LAN deployment follows the guide.
Automatic sub-node deployment requires SSH key authentication; Windows hosts that only expose WinRM must use the manual PowerShell script.
The community catalog currently accepts generator descriptors only (each node reproduces the data locally); real data-file distribution is not implemented.
The assistant stays disabled until DEEPSEEK_API_KEY is configured.

Contributing

Issues and PRs are welcome. Before submitting:

Make python build.py --test pass.
Keep the C++ and Python sides in sync; new kernels must ship with a scalar fallback.
For assembly changes, test on the target architecture and state the kernel_impl() output in the PR description.

If this project helps you, a star is the best support.

License

GPL-3.0 · compatible with pybind11 (BSD) and NASM (BSD-2).

Name		Name	Last commit message	Last commit date
.trae-html-share-packages/python/mahe/webui		.trae-html-share-packages/python/mahe/webui
core		core
docs		docs
packaging		packaging
python/mahe		python/mahe
.gitignore		.gitignore
DESIGN.md		DESIGN.md
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
build.py		build.py
requirements.txt		requirements.txt

Navigation Menu

Repository navigation

About

Repository files navigation

MAHE · Distributed AI Training Framework

Contents

Why MAHE

Key features

Quick start

Architecture

How it works

SIMD kernels

Cross-platform builds

LAN cluster

Web console

Training page (beginner flow)

Datasets and the community catalog

Node group management

Assistant (reserved)

Sub-node deployment

One-click installer

Project layout

Testing

Roadmap

Known limitations

Contributing

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

MAHE · Distributed AI Training Framework

Contents

Why MAHE

Key features

Quick start

Architecture

How it works

SIMD kernels

Cross-platform builds

LAN cluster

Web console

Training page (beginner flow)

Datasets and the community catalog

Node group management

Assistant (reserved)

Sub-node deployment

One-click installer

Project layout

Testing

Roadmap

Known limitations

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages